From Neos Wiki
Jump to navigation Jump to search

Geenz's graphics office hours

Here are the notes from 2nd December 2021

These are rough notes typed by ProbablePrime. If there are errors please edit away!


Please do remember to check our Roadmaps as a lot of questions here are partially covered there. Do continue to ask them but the roadmaps are there for you to read:

These are really rough, we're very busy right now. Feel free to edit/update these notes as you see fit.


3x1t_5tyl3: Question in Absentia: What's the current state of affairs in terms of graphical updates and what can we expect before the end of 2021, what can we expect for Q1 2022? Is there anything you specifically (Geenz) can tell us about Future VR Headsets that might be potentially interesting for neos as a platform?

  • A multi-layer question
  • Before the end of the year
    • A bit of a ?
    • Color management requires some Froox time
    • Shader consolidation could be by the end of the Year. We want more consistent options for example Dual Sided "Anything". PBS is the main target. Its easier to ship.
  • Vjaro is interesting.
  • Some of the upcoming AR headsets could be too.
  • Geenz won't name them but we'll see how they progress and go.

Ballooonie: Why is shader compilation different for Neos? Why can't i say: import a custom unity shader freely? Why do shaders affect iteration times? I don't understand why VRchat can do this and Neos cannot.

  • A part of this is, that if we let everyone use custom shaders today under unity it would break when we move to the new Graphics Engine.
    • They're pretty incompatible with our new engine and the future of Neos.
  • The current compilation path for our existing shaders is not that good. We have to compile variants for each shader for each platform for various configurations.
    • It takes about a day to compile even on high end computers
  • Its just simply a matter of it breaking your content later that we want to avoid.

Lexevo: Is there anything you're focusing on specifically to improve performance for the new graphics engine? And is there any performance targets you're aiming towards for the new engine?

  • Lighting is one item:
    • Lighting rendering, using deferred rendering. All opaque objects we encode all the surface details to a geometry buffer which is a screenspace texture with geometry information. We send this to a later area to encode lighting.
      • its great for a monitor, as it lets lights happen really cheap
      • But for VR this is very intensive on your GPU.
      • Shooting these textures back and forth has a high a cost.
      • We're going to move to Forward+ and a derivative such as the one used by Doom Eternal which reduces the bandwidth constraints and costs.
    • Triangle Level culling would also be useful for higher density worlds which would run alongside object culling.
      • Its not us looking at culling each and every triangle its on top of our additional culling stuff.
  • With the new engine we'll be on .NET 6 which gives us so many more optimizations and features. We'll be unstuck from Unity which runs an older version of Mono
    • More optimizations, platform specific optimizations etc.

Earthmark: I recall shader blocks being considered, is there an estimate priority on that kind of thing? Can we get an idea on what the general priority for graphics things is? There is 'look at roadmap' but it often changes, or at least feels like a bucket. What are your general priorities?

  • what are shader blocks, what do you mean?
    • They mean stacking parts of shaders together, I guess
  • our priroties are getting color management out, so that's stalling.
  • Shader consolidation is what Geenz will be focusing on
  • After that is Libvlc which needs some color management stuff
  • After that is a coin flip, between material layering and light probes.
  • Shader blocks would be after that.
  • Prime Reminder here: the roadmaps are constantly changing yes, they should be up to date though. Our current priorities should match the roadmap on GitHub, if this isn't the case let us know and we can get things synced up :)

epicEaston197: Q: what is the ETA of the graphics engine? I've been hearing that it might come next year

  • We don't provide ETAs but Maybe sometime next year.
  • Prime reminder here: The roadmaps are here for a reason, double check them and see what they say. You should be able to see what's going on on them before we can tell you.
  • Additional prime reminder: Easton, we don't provide ETA's you've asked quite a few times. If this is an issue let's chat :D.

jiink: Q: What particular aspect or feature of your graphics work are you personally most excited to see the fruits of? If that makes any sense.

  • New graphics engine is very interesting. Geenz has worked on Second Life and other platforms and seen how those platforms need to shift and change with user generated content.
  • Taking almost a decades worth of knowledge from Geenz and producing a new engine based on that knowledge, designed for a metaverse will be exciting to see.
  • Being able to create something that has the level of modularity and flexibility to enable our users to make more customized content is exciting.
  • Going from a PS1 style in-Neos game to a AAA in-Neos game is really exciting.

Duff: what concepts or "places to start" would you recommend for people unfamiliar/trying to understand graphics programming better? (im not very in-tune with the concept myself, but id like to get a started on a foundation of trying to understand)

  • When Geenz was 10/11 they got into it by getting into 3D modelling and then they couldn't show it off so they got into graphics to show it off.
  • To get the spark going, watch some graphic tech demos from the major engine creators such as Cryengine, Unreal etc. Unreal has some good videos showing how things work
  • SIGGRAPH has put lots of videos online from some talks that you can take a look at what's out there.

Toxic_Cookie: How possible would it be to have a UIX image type that allows you to use a sprite sheet but individually specify which sprite to use? Eg. I could make a single texture that has 16 icons and have 16 "sprite sheet image" UIX components that all point to the same texture but use a separate sprite depending on what the parameter is on their component. (I think atlas info might play a role here)

  • This goes into some of the Neos 2D stuff which is not often talked about.
    • Prime Note: Neos 2D is a rarely mentioned concept of running Neos in a 2D mode like a 2D game and making games like that. Think of it like Unity's 2D stuff. Its not on the roadmap right now but sometimes we talk about it.
  • It warrants some investigation on the team's part to explore as it could provide another optimization technique for all of our UI

Earthmark: Are uix questions graphics questions, or better suited for other office hours?

  • Kinda sorta technically...
  • It depends on the specifics.
  • Toxic's question is Graphics as it deals with optimizations to the graphics pipeline. Its part of the system Geenz wants to get to know more.
  • If its about Layout tools or using UIX, Geenz can't really help you.

epicEaston197Q: why do mirrors not render HSV materials for VR?

  • Geenz isn't sure. Submit a GitHub.

AshtonSparx — Today at 6:14 PM Is the infamous UIX update flash fixable? Regarding textures updating (Like toggle images)

  • Geenz thinks that's maybe a material issue
  • Geenz might be able to look at it

Max: Q: just asking will there be a testing version of the new engine like how danger nightly build or something ?

  • Yes there will be.
  • We'll be running these side by side the old engine to compare and see what needs fixing and breaks etc.

Deretto: It's quite often that I have issues with having my avatar's camera obstructed by hair or a hat. The current culling available unfortunately too potent and removes the ability for anything to get close if enabled. Are there plans for specifically first person "on avatar" culling that'll still allow us to appear as intended in a photo.

  • this is the first time this has come up
  • Right now if we did something like this it would not be the default.
  • We might give you a special culling component that would scale stuff down for your viewpoint
    • VRChat might be doing something like this we aren't sure.
  • Its a maybe based on other priorities

Gizmo: My question is on issue #322, I saw this was given lower priority recently. Could you talk on that if you haven't already? Neos's current SSAO makes it difficult me to use for extended periods.

  • Geenz reduced its priority for a number of reasons.
  • There are some mitigations that are going into place for a number of other reasons for example Motion Blur at lower frames is very strong.
  • There is a mitigation for this that is in review.
  • Otherwise we want to wait for the settings UI re-work so we can provide more fine-grained control
  • We also want post-process volumes, which will allow you to control how post-processing works within areas of a world
  • this would be for multiple things.
    • People should be aware of parenting a Sphere to people's heads to achieve affects we want to put an end to that using these volumes
  • Its lower priority as a bunch of other work needs to happen before we ge to this such that its easier to use for everyone and customizable.

Towneh:In other games, there's popular examples of heavy lifting being done with hlsl shader code, such as sound reactive effects, dmx lighting fixtures based on camera pixel sampling, without the current performance penalties seen in neosvr, are there any plans for allowing interim precompiled shaders that can do this?

  • We're open for proposals to do this for specific shaders.
  • If you have a precompiled shader we can't let you inject that right now.
  • That will change with the new engine but it still won't be HLSL, you'll be able to string Logix together to do this.
  • This also lets us control things such that things not suitable for a shader stay out of the shader land.

Sarra_Kitty: I'm curious about how cross-platform the graphics programming is, Neos has a native Windows and Linux version, how much effort is it maintaining both vs just Windows? Are there specific things you have to do to support linux or is it sharing a lot of the same code? I don't know much of anything about graphics programming, but I'm happy to see that you guys support Linux btw. An "open metaverse" is hardly open unless it supports running on an open-source operating system, and this does!

  • so Firstly talking about Unity:
    • Unity is supported across every platform
    • To maintain a linux and windows version is make sure shaders are compiled.
    • Unity does a lot of the heavy lifting for you.
  • For the new engine:
    • We're uisng a rendering API framework which allows us to not worry about platforms as it abstracts them
    • It therefore won't be a whole lot of extra work due to the design.
    • it lets us deliver features etc.

Earthmark: How do you plan to compete with more staffed engines in terms of graphics power? For instance engines like unreal have a lot of engineering resources neos doesn't have, how is neos going to be competitive against engines like that?

  • Geenz hopes that in time we'll hire more engineers to help.
  • Don't expect us to go on any hiring sprees overnight.
  • We need to cater to that in our team's structure.
  • We might need one extra person in the interim.
  • You are right there are other organizations that have a lot more people.
  • Getting everything together is quite the undertaking so to accelerate that is important
  • Its open-ended as we need to figure out how to expand to do this.

Towneh: Without sampling say RGB macroblocks embedded on a video stream for light data, how else can you guarantee that a video stream's audio is perfectly in sync with data sent via a websocket instead, such as beat drops

  • Geenz isn't sure they can answer that at this time.
  • They'll note this down as having better time sampling.
  • We could potentially have better timecode support. We understand that this isn't that good but with the libvlc update synchronization should be better and then you can test against this.
  • There are other latency concerns.
  • We don't have very robust sampling of video textures
  • There's a bunch of possible solutions to this that we need to look into.

jiink: When should I be using "Alpha" vs "Transparent" modes on materials? Will anything about transparency be treated differently in the new gfx engine?

  • Use alpha when you want the object to be completely invisible.
  • Use transparent when you need edges/parts to be visible such as a bubble where its edge is more visible.

epicEaston197: Q: are the filter materials grabpass shaders?

  • Yes.

ohzee: Is there any plans to do more đŸŽ„devlog stuff in terms of showing progress? It's always welcome to see lil glimpses of the development process

  • Yes.
  • Geenz just hasn't had the time recently as they've been putting time in within other areas.
  • Lookout for more devlog stuff from them soon.

Robyn (QueenHidi): Wait is it possible to read timecode with streams currently?

  • not quite graphics focused.
  • You can get the current progress but I'm not sure you can get the timecode currently.
  • Geenz will look into it

Duskitten: Will the new engine be using gles or vulkan, or what will the main render portion be based on (im late so idk if this has been asked

  • depends on the platform. Linux and windows is likely Vulkan. They can't predict we'll use DirectX
  • For mobile we might be using GLes or vulkan depending on the device.
  • For Mac etc we'll be using Metal.
  • Specific thing we're using here is "The Forge". Link in the chat.

AshtonSparx: How heavy is it having a grab-pass shader as your avatars shader? Like the HUV shader or pixel shader (Example: AFK material)

  • grabpasses can be cached inside Unity. It just depends on what material is using it. This is usually done at the material level and not the shader level there are things you can do to make it faster.
  • These usually aren't super heavy.
  • Once we're more friendly to mobile platforms this might become a little more heavy. But we should have some opmtimizations in place for this.

Duskitten: Will we get visual shader building, or atleast is it something being considered

  • Yes, visual shader building is happening.
  • Shader building will be a subset of Logix so Logix features will be included such as Meta-Programming etc.(text based stuff)
  • Think of this of sort-of like blender
  • You'll be able to collaborate as you build out shaders in real time.

SNERFOIL: will this new graphics thingy allow for stencil buffer shenanigans? or similar?

  • Yes, it will we think so.
  • Stencils are quite useful for a variety of things and effects and experiences.

AshtonSparx: Will we be able to do "painting" like shaders or what ever? Like quesiontable patreon VR games?

  • yes, seeing as you'll be able to interact with shaders via logix. You'll be able to talk to them to setup custom renderning so it should be possible. Yes.

Jam10o: do you plan to support every platform supported by the forge?

  • Within reason yes.
  • Some of the more closed platforms are tougher e.g. Xbox, Playstation etc.
  • The reason we're using Forge is due to the platforms it supports.
  • That makes things like Playstation VR possible IF sony would let us.

Max: I do not know if it is a question you can answer but With the newest engine will we be able to import different models besides fbx .blend files XPS n such?

  • not a graphics question.
  • We should be able to support anything we can currently support.
  • Prime note: Remember the graphics engine is GRAPHICS it won't change most of Neos' Coree features, you can important a ton of model types. Just try :)

ohzee: Is the new rendering engine currently "playable" with Neos? (in the loosest sense of the word)

  • Not yet, due to other priorities.
  • You can use super basic stuff to render objects
  • Its a head without a body.

h33t: Is it currently possible to add and run an additional separate emissive texture to an avatar?

  • Not presently, Geenz wants to support things like emissive masking.