If you think RDP performance is good, you should try Sunshine+Moonlight. I regularly stream at 4k 120hz from my gaming PC to my laptop and it's basically indistinguishable from running locally. I've even run it over Tailscale from a thousand miles away (though at 1080p60) and it was still markedly better than RDP.
Caveats - You really want a Windows host for fully accelerated on-GPU hardware video encoding, the server setup is _slightly_ more involved than RDP which is usually preinstalled, and Sunshine+Linux+NVidia requires an annoying driver patch. But overall it's _amazing_.
I'd still pick RDP for remote managing any day, its server and client are so much better integrated (clipboard, peripherals, etc.) for this task than Sunshine/Moonlight. It also avoids the whole issue of setting up a virtual display output[1] as is required by Sunshine. I use both for different tasks.
On a related note, as of recent versions, Sunshine and Moonlight stable releases support 4:4:4 chroma subsampling and bitrates up to 500Mbit/s with HEVC, which results in almost indistinguishable image quality compared to native output[2]. Bitrates that high are unusual in normal content, but at least Apple's Media Engine (on M-series Macs) appears to be capable of decoding it.
[1] Here is a pretty good solution that piggybacks off of Parsec's driver, which is fully signed, though its EDID lacks HDR support. The project also includes a C header file for custom implementations: https://github.com/nomi-san/parsec-vdd
[2] Note: I found that I had to enable 10-bit color streaming (which is available when using Parsec's driver) to get rid of some gradient banding that isn't present in native 8-bit. I suspect it's some encoder and/or color space issue.
I wonder how Apple's ProRes would work for remote desktop over wired networks, especially now that consumer equipment is starting to move beyond 1GbE. They have hardware encode and decode on everything newer than the original M1, and ProRes only does intra-frame compression so a fairly low-latency implementation should be possible.
TBH, After trying both I'd still pick RDP (I used xRDP server on linux) for anything text/GUI related. RDP(not VNC or VNC larpin as RDP) doesn't make image compression artifacts when the connection gets bad it's just add some lag keeping the text crisp.
Sunshine/Moonlight on the other hand reduce image quality keeping the latency low, which is really good when streaming games for example.
> I regularly stream at 4k 120hz from my gaming PC to my laptop and it's basically indistinguishable from running locally
Sunshine+Moonshine is great but lets not exaggerate. On controller it feels fine, on mouse it is still a noticeable difference from local play. I'd compare it to turning on classic v-sync (so not fast sync).
Tested this out again with Overwatch - latency is single-digit milliseconds even at 4k120. I can feel a _slight_ bit of lag at those settings, anything below that is as good as local.
I use Sunshine and Moonlight when I'm out away from my Windows laptop in my home office and back near the router in the front room over my Macbook air, it's great when you can get directly on the LAN with it. But then, RDP is also better over the LAN, and casual disconnection is easier.
But if I want to do graphics-intensive work on the headless tower, I'll use Sunshine and Moonlight.
There are a few disadvantages, in sunshine/moonlight such as you still have monitor output, so you are stuck to the resolutions supported by your monitor and someone can see what you are doing. Of course there are workarounds like dummy monitors.
I do not have clipboard sharing working (I’m not sure if it is even supported) and there is a bug where some specific keys (including \) do not work on some keyboard layouts (Japanese for example), although there is already a pull request that supposedly fix that.
Still it is an amazing solution. The snappiness of having full 3D acceleration is amazing, and as another plus it works in the home edition of Windows.
> in sunshine/moonlight such as you still have monitor output
Apollo[1] fixes this problem really neatly - there's a "Virtual Desktop" option that adds a virtual desktop, and you can disable all the local monitors while in this session so that a local person doesn't see your desktop while you're remoted into it (just remember to lock after you end your session!).
I believe it also preserves monitor layouts when starting/ending sessions.
I used Sunshine and had a bunch of hacks in the startup/teardown scripts to get the same behavior but it was really brittle. Apollo makes this work out of the box.
That doesn't compare to RDP where remote windows act just like local windows in that you can resize them and drag them between monitors, they're not constrained to just being on a virtual desktop.
RemoteApp is basically RDP for one single desktop app running on the remote server but launched and run to appear like a local app. It's improved a lot over the past decade+
There's an extra interesting feature there where the remoteapp windows are aware of each other. It's a niche use case, but the remote side counts as one session, so apps can interact and automation/accessibility mostly works as expected. Although the local system still sees the remote apps as opaque rectangle rather than widgets.
I haven't use them, so please bear my illiteracy around these. Does this mean, it actually creates a local session, not a remote and headless session to serve? If that's the case, it feels like it's just TeamViewer or Remote Assist session where you hop in to an existing session. Or do I misunderstand the concept?
So Sunshine and Apollo are continuations of Nvidia's EOL'd GameStream tech, which is designed to mirror the local display to a device. It doesn't really have the concept of a "user session", it allows you to mirror the currently running local session remotely.
> If that's the case, it feels like it's just TeamViewer or Remote Assist session where you hop in to an existing session
Yes, it's pretty much that, but optimized for A/V latency and game inputs (games that "trap" the mouse in fullscreen are well supported and controller inputs are passed through).
I haven't really used TeamViewer/Remote Assist heavily, but I wager if you wanted to game with those tools it would be a worse experience than something like Sunshine.
>Apollo[1] fixes this problem really neatly - there's a "Virtual Desktop" option that adds a virtual desktop, and you can disable all the local monitors while in this session so that a local person doesn't see your desktop while you're remoted into it (just remember to lock after you end your session!).
Does that mean someone with physical access can take control while you're logged in?
Yes - afaik all of the "game streaming" solutions (Sunshine, Apollo, and probably others) derive from Nvidia's GameStream tech which just mirrors the local display remotely.
My buddy was absolutely delighted by Sunshine+Moonlight and basically forced me to try it. Long story short, it is not nearly as responsive for dev tasks, amazing for gamings or streaming, however.
In your experience, how's Moonlight at audio streaming, particularly in terms of latency?
This is something that most mainstream remote access solutions struggle with, hence why e.g. many screen readers come integrated with some bespoke remote access solution of their own. That's not the only reason, but definitely a very important one, and getting audio right would go a long way towards making a single solution work well for both SR and non-SR users.
Disclaimer, I know nothing of the internals of Microsoft RDP, but it does not appear to use an image representation like VLC, etc. do. My suspicion is that Windows APIs are intercepted and used to build the desktop experience. Don't get me wrong, I know A LOT about Windows internals, but I never looked into that part, sadly. What I do know is based upon what I see, and what I see leads me to the comment above.
The thing about Microsoft RDP is that even at high resolutions it's NOT just streaming 4k video. It uses way way way less bandwidth (and requires way less of a performance GPU / CPU). It uses a ton of tricks like having the local client render as much as possible instead of just streaming video. This results in it feeling way more responsive and native desktop like.
Caveats - You really want a Windows host for fully accelerated on-GPU hardware video encoding, the server setup is _slightly_ more involved than RDP which is usually preinstalled, and Sunshine+Linux+NVidia requires an annoying driver patch. But overall it's _amazing_.