My dream is to have a VR environment where I can summon additional screens into existence (ala xrdesktop) and also create a decent-ish streaming setup out of it. Ideally I’d be able to have a 3d avatar for myself and a bunch of props (IE: coffee cup, desk toys, etc.). It should have a few main components: the monitor world with a desk and some props, an avatar model that uses body tracking to draw its location relative to where the user would be in that VR environment, a window that a view of the scene would be drawn to (so that you can stream it) and finally a fake webcam view that maps to the head of your avatar so that you can use it for video conferencing.
For extra credit this would be a Wayland compositor using Vulkan for rendering 3d.
How hard would it be to make this?
@cadey Have you looked into VRChat and OVR Toolkit for this? At least to test the viability of such a setup?
@jaycie Yes that's how I know that the basic premise works out, but for productive things I use Linux and I don't want the possibility of this setup to be open to the internet. I'd rather have Linux-native tools above all else.
@cadey Incredibly valid. I can ping my girlfriend later. She knows VR tech and VR dev in Unity much better than I do.
The various views into the scene would probably be easy to implement by rendering from multiple cameras to render textures, and streaming from textures can't be very hard.
Vtubers have shown that there certainly is some kind of usable body tracking out there, but it'd probably be a little fiddly on Linux. I hear they use iPhones.