Is there a danger of overfitting if you train something on a physics sim? How do you prevent the model to exploit the differences to real life? Surely there are some numerical errors and other idealizations that result in some stuff being a good solution but not working in real life, or is the sim that accurate?
This (and a global pandemic) are central to the plot of The Talos Principle, a 2014 puzzle game. Can't say much more without spoilers, but we'd better hope its other predictions for the future don't come true.
Vulkan is a rendering technology, Unreal is an engine, that can render using Vulkan under the hood. You can absolutely do this in Unreal. I implemented this at Cruise using UE4 for integration testing, and it worked great for inference (we weren't doing training on sim at that point, but I was pushing for it! There was a paper out in 2018 or so that showed mixing in a bit of simulated data had an outside positive impact on the outputted model). There are companies out there right now doing this with even more modern renderers. I can't comment on how much the rendering realism gap matters here. I think there's some people out there using a variant of lower quality rendering + some kind of diffusion to get "better" images without having to do detailed modeling/lighting for their sims (fuzzy memory, I don't have a source on this).
Unreal will definitely get better results out of the box, but it's also possible to do photorealism with significantly less overhead (particularly UE shader compilation overhead) - useful for single purpose platforms. If you don't need to support lots of specific editor or game features, it may be a valuable investment.
UE is definitely used to obtain simulation data in other domains (this is coming from first hand experience in big tech), but usually through scripting UE handmade levels in python which also needed convoluted server systems at the time (hopefully this has gotten better now).
This is the company that Yan Chernikov - aka TheCherno has cofounded. This engine is based on the engine (Hazel) he built as part of his game engine series on YouTube. If you're interested there is probably at least 100 hours of devlogs about this engine which is kind of cool.
I was the Principal architect for ML-Agents at Unity for a while and this looks like it could be a more elegant version of what we were doing (cause it’s not a sidecar to an engine).
I’m going to try it to see if I can make my Go-1 edu do some work around the house finally
What's the difference of a "robot game engine" to a simulator like <https://gazebosim.org>
nothing
Haven’t Unreal engine and Unity been used for robotics for over a decade?
Hasn’t the Bullet physics engine been used for robotics for over a decade?
I don’t understand this “first game engine for robotics” messaging.
As an aside, this website crashes for me on safari on iOS.
Their claim is actually: "the world's first game engine purpose-built for robotics."
Idk if that's true or not, but it does exclude all the engines you mentioned.
Is there a danger of overfitting if you train something on a physics sim? How do you prevent the model to exploit the differences to real life? Surely there are some numerical errors and other idealizations that result in some stuff being a good solution but not working in real life, or is the sim that accurate?
You add variance to the sim parameters with the hope that the model learns to adapt to changes in environment
This (and a global pandemic) are central to the plot of The Talos Principle, a 2014 puzzle game. Can't say much more without spoilers, but we'd better hope its other predictions for the future don't come true.
Wondering if it would be better to abstract Vulcan away by Unreal, get even better photo-real quality.
I guess the better question is how much does photo-realism quality matter for this kind of sim2real work? A lot I would wager.
Vulkan is a rendering technology, Unreal is an engine, that can render using Vulkan under the hood. You can absolutely do this in Unreal. I implemented this at Cruise using UE4 for integration testing, and it worked great for inference (we weren't doing training on sim at that point, but I was pushing for it! There was a paper out in 2018 or so that showed mixing in a bit of simulated data had an outside positive impact on the outputted model). There are companies out there right now doing this with even more modern renderers. I can't comment on how much the rendering realism gap matters here. I think there's some people out there using a variant of lower quality rendering + some kind of diffusion to get "better" images without having to do detailed modeling/lighting for their sims (fuzzy memory, I don't have a source on this).
Is there a good Unreal + Physics simulator OOB now that people can try?
I guess Isaac Sim is king?
Unreal will definitely get better results out of the box, but it's also possible to do photorealism with significantly less overhead (particularly UE shader compilation overhead) - useful for single purpose platforms. If you don't need to support lots of specific editor or game features, it may be a valuable investment.
UE is definitely used to obtain simulation data in other domains (this is coming from first hand experience in big tech), but usually through scripting UE handmade levels in python which also needed convoluted server systems at the time (hopefully this has gotten better now).
What about Karel the robot?
This is the company that Yan Chernikov - aka TheCherno has cofounded. This engine is based on the engine (Hazel) he built as part of his game engine series on YouTube. If you're interested there is probably at least 100 hours of devlogs about this engine which is kind of cool.
https://www.youtube.com/@TheCherno
I was the Principal architect for ML-Agents at Unity for a while and this looks like it could be a more elegant version of what we were doing (cause it’s not a sidecar to an engine).
I’m going to try it to see if I can make my Go-1 edu do some work around the house finally