Multi-world support

[djeedai]

I'm opening this as an "Idea" type of item to discuss the advantages of having multiple worlds. This is very similar to #2237, but that other item was opened as a Question, and focuses on a single aspect of multi-worlds, while I believe there's more to it. So I'd like to expand the scope and change the format into a discussion.

The idea proposed here would be to add support for (or, enhance and polish the existing support of) multiple World in the same Bevy application.

Current status

Currently the entire Bevy application revolves around a single World. All features are added to that world: from low-level management systems like asset loading and raw inputs, to intermediate level ones like UI interactions, all the way to the actual game logic.

Note: I use the term game here as usual as a terminology shortcut for the core application logic, so this applies equally to, say, a CAD application.

To be exhaustive, I should mention that internally Bevy also has a "render world". However, users rarely have to deal with it, and that world is used as a container for entities and components (for example, there's not component ticking nor time involved in the render world, and it's also cleared each frame), which semantically is different from what the main World is used for. Therefore I'm ignoring it in the discussion below.

Scenarios

This section describes a variety of scenarios where using a separate dedicated World provides some net advantages.

State management

Most applications have several layers of systems, from low-level hardware interaction to high-level "business logic" and user interaction. Often the low-level systems are provided by external libraries, although in a game engine like Bevy it's also frequent to write those systems directly inside the engine codebase itself. Those various layers generally have state, and the lifetime of those states dependent on the level.

• Low-level system are more prone to have runtime-only state which disappears once the application exit. Excluding things like settings, low-level systems generally maintain a state to perform their runtime function, but there's rarely any use for persisting that state across application runs.
• Game or business logic often make use of a persistent game state, and the application is designed to manage that state. In video games, this is the game itself, which is saved and reloaded on next run. For professional applications, this is the "project" (a CAD file, a 3D model, a media file, etc.) that is being edited. The core function of the application is to load-edit-save that state.

Those two categories of state have very different requirements and lifetimes. Because of that, they are best separated into different worlds. Game logic state needs to be serialized as a whole for consistency, but do not need nor care about lower-level subsystem state. Saving the position of the player is critical, but saving the current time of the flashing animation of a menu is generally useless, as no user cares if the menu highlight will restart its animation on next run (and in fact, most users expects it to do so). Separating those states into different World containers ensures that all the game logic and only the game logic is persisted, without the risk to either forget about a game logic state or serialize extra unused state that doesn't need to be.

As an example of this, Unity uses a common state resource for both the game serialization and the Editor-only serialization. This means often games are shipping with Editor-only data. Alternatively, the developers have to write complex systems to parse and strip Editor data before shipping.

Simulation time

Many simulation games (and apps) require very strict time / tick management for simulation accuracy. For example, the Factorio game has a separate dedicated time concept for the game simulation which is entirely decoupled from the "real" time (FPS / framerate) the application is running at. This ensures accuracy in simulation by decoupling the (always inaccurate and varying) application and rendering framerate from the precise ticks of the simulation of the game logic.

This simulation time often can be paused (the simulation stops) for the user/player to be able to inspect the game logic state. During that period however the rest of the application (asset management, UI, .etc) must continue to execute because the game needs to remain interactive; only the simulation is paused. In particular, things like UI animations continue to execute. Translated to Bevy, this means that any system associated with the game logic needs to be paused and not ticked anymore, while any system associated with other non-game-state functionalities need to continue ticking as usual.

Note: Time scaling (speed up / slow down) can be seen as a generalizing of pausing, and is affected in the same way.

Both features (separate simulation time, and pausing) can be achieved with a single World. However, there's a few major drawbacks in doing so:

• The world requires 2 different time resources, and systems must use the appropriate one depending on their function. Any mistake by the developer and the simulation could deviate from its intended time in imperceptible and hard to debug ways (e.g. using the application's own main loop time instead of the simulation time when framerate is sustained at the same value as the simulation one will only produce a tiny drift, whereas under a lower-spec system that framerate can drop and the simulation drift become more consequent). Any Unity developer will have experience mistaking Time.time (the simulation time affected by time scaling and pausing) and Time.unscaledTime (the application time, which cannot be paused) when designing pausable components.
• Pausing the simulation not only requires again that all simulation-related systems use the proper time resource, but also means that all simulation-related system probably need an appropriate driver that skips them when the simulation is paused. Although not strictly required, this ensures that during pause we don't have a hundred calls to paused systems that will return immediately upon checking a pause condition. This is particularly critical when the game/app is not focused or minimized to try to reduce the power consumption (although there are more drastic ways to pause the entire application when minimized; but again, no one expects a game on the pause menu to use 80+% CPU, it's generally something frowned upon by players, so developers have to try and minimize CPU load when the player is not actively using the game).

Advantages of multi-world

The advantage of having multiple worlds for the above scenarios can be summarized into the concept of isolating states. Having a separate simulation world by design ensures that:

• State is separated and cannot leak into another world's state. This offers peace of mind for developers, as systems cannot inadvertently access multiple worlds at once. The design removes an entire class of bugs.
• Ease of use for state persistence, as the World container maps 1:1 with the state to persist. There is no risk to persist extra state, nor forgetting to persist some state.

Note that this isolation would provide strong guarantees by design that a future Bevy Editor would not mix its own state with the state of the game it edits. This is particularly valuable when thinking about a live editing ("play mode") scenario where both the Game application and the Editor application are running simultaneously, and their state must remain separated.

Relation to scenes

To some extent, the state persistence aspect relates to Scenes by the fact that deserializing scenes offers the same level of state persistence loading than deserializing a World. Scenes can be used as "prefabs" to store a part of the game state. However scenes are more about reusability than state isolation, especially as they tend to "merge" into a World when deserialized and do not (currently?) retain an understanding of their content so do not offer any way to easily (re-)serialize a subset of a single World. They also don't help with pausing simulation or simulating only a subset of a World, as systems are associated with Worlds and not Scenes.

Relation to the Subworlds RFC

The Subworlds RFC covers a similar overall idea, and the discussion on the GitHub item goes through similar use cases as above. However there's a few distinctions:

• The main goal here is state isolation, so any reference to moving entities between worlds is out of scope for the current proposal. Same for resouces, they're inherently attached to a single World and should not be shared, or that would defeat the main design goal of isolating state.
• The API to "set the current world" makes few sense in the context of the above proposal. The objective here is to have all worlds used in parallel. The usage described in the RFC of setting the current world is only relevant where the World is used as the main game state to persist, and in that case is best left to the management of the application itself, as there's little incentive for the Bevy systems to know about this.
• Interaction with stages (then sets from v0.10) is trivial and natural, as each system operates on a single World, like is currently the case for the main and render worlds. The unclear part is the API to extract data of a world from a system of another world; this is was Extract does, but is quite annoying/limited at the minute.

Open questions

• What changes are needed to the current World API to achieve this?
• How can Extract be improved for better usability?
• Can we leverage the current app / sub-app feature? Or do we need a new framework to manipulate multiple worlds?
• Do we need a way to mutate a World from another World? Possibly not; the render world's systems have read-only access to the main world. And it might be in fact desirable to uphold this to enforce the isolation guarantee.

[lloydparlee]

I am working on a turn based project where I use a separate ecs world for the logical state. My use case is pretty simple and probably different from what you're thinking of, but I just want to add it here so people are aware of it.

I store my game's logical state in a resource that contains a world and has an api to get and perform actions. Instead of writing systems with queries, I have functions that use the world's get<Component> or get_entity to read or modify the world.

The only systems I have for the logical world are things that happen between player turns. I have a schedule that I run once between each turn that runs these systems.

I also have many kinds of relationships, such as mapping locations to entities, mapping logical entities to app entities, or keeping track of many different kinds of parent child relationships. These were all solvable, but some built in relationship support would have helped.

So for a turn-based use case, the current world api already works well enough. For real time multiple worlds, the complexity is in scheduling systems that run on the other world so that they can run in parallel, and not just be called from inside an app-world system.

For my use case it would be great if I could run a schedule on the logical world and not have to worry about that schedule finishing within a single app-world frame.

[djeedai]

The new stageless Schedule will probably help, but I didn't try it yet.

[djeedai]

For my use case it would be great if I could run a schedule on the logical world and not have to worry about that schedule finishing within a single app-world frame.

Re-reading this, it sounds like this implies the schedule running in parallel on different threads, which then poses some issue for synchronisation. If I take a parallel with Unity and its physics stepping, which is done with constant timestep for stability, there's still some synchronisation happening each frame because e.g. the physics position is needed to render objects. So I think this is a more complex superset feature of what is described here.

[minecrawler]

Doesn't turn-based also mean that you'd require to access the same entities from different worlds (like position data) - or implement syncing? I'm also working on turn-based games (using my custom TS-based engine), and I usually use events to trigger turn processing and keep the rest of the logic (e.g. rendering) running in my ECS.

Maybe having multiple worlds isn't a good idea for turn-based games because of the complexity of syncing, but instead having multiple parallel schedulers would help, which are triggered on different conditions...

[andyy-garcia]

I don't know if it's a good practice but as I got no answer on my topic yet, and I think that it can be related to yours, I reference it here : #9732

(I was/am looking for a normal way to isolate plugins input/output data, and maybe having different worlds for different plugin logics as you suggest might be a solution to that)

[ilyas-taouaou]

This may be useful for the Editor too, where the editor would have a separate world instead of polluting the game world with editor related functionality.

[hcabel]

Funny you mentioned that now, I was researching this topic exactly for this purpose.

My biggest problem with multi world atm, is queries. There isn't (to my current knowledge) a way for queries/resources/commands to be linked to a specific World. (Might be done using a resource world that you always reference, but that would be very hacky and verbose)

For this to work we need a way to specified the world on which a system is being runned on, or do this on a per query basis (With<GameWorld>?, Commands<GameWorld>?)

That being said, I'm not sure if it's a good idea or not. I'm going to experiment with SubApp and Scene now see if that can help me accomplish what I want

[ilyas-taouaou]

Funny you mentioned that now, I was researching this topic exactly for this purpose.

My biggest problem with multi world atm, is queries. There isn't (to my current knowledge) a way for queries/resources/commands to be linked to a specific World. (Might be done using a resource world that you always reference, but that would be very hacky and verbose)

For this to work we need a way to specified the world on which a system is being runned on, or do this on a per query basis (With<GameWorld>?, Commands<GameWorld>?)

That being said, I'm not sure if it's a good idea or not. I'm going to experiment with SubApp and Scene now see if that can help me accomplish what I want

Yea I just tried bevy_editor_pls and I found it annoying that the editor business like gizmos components are also included into the mix.
Also the fact that the editor add some plugins which the user could rely on but then be surprised that the game is not working anymore when he strips out the editor.

[ilyas-taouaou]

A solution for now may be having the editor work similarly to bevy_render where it have its own separate copy of the world that are kept synchronized to avoid interfering with the runtime of the game.

[ilyas-taouaou]

Either way the editor should ideally have no effect on the main app except for its performance overhead.

[hackerzhuli]

I am researching a RTS project and I think this will be very important in the case of RTS, that is, a separate world for game state.

Each match that the player starts must have a separate world, which is where the real game state is, which starts from an new world when the match starts. So we can guarantee that different clients will simulate the match deterministically including entity ids. That way we can do a simple hash on the state of the world and know that different clients have the same exact world. Of course we also run on custom fixed update when we receive a frame of input from the server(which should not be run if the server haven't sent the frame of input regardless of the real time)(this approach is known as lockstep networking).

This way, players can rewind the game, go to a previous point of the game and resume from there. How it's done is we just clone the world at fixed time span (eg. 1 minute), and don't update those worlds(basically, their time is frozen). When a player need to rewind the game. We just clone the nearest world from the past, put all input frames after that frame into the world and manually call fixed update on them. Once we get to the point of time we need, we can resume the game.

This is useful when the player need to rewind when watching a replay(we don't want to start from scratch, of course). Of course the player can resume the game from a replay at any point of that replay(this is useful when players want to practice a particular scenario with others, by not playing the game from scratch, they save a lot of time, this is not a save game, so that player can pick any time they want to resume from). By the way, these features are present in Starcraft 2.

We don't want any UI stuff inside of the world, no app level stuff inside of the world, this world only pertains to a particular match. The world can be totally destroyed when the match is finished and the player quit and go back to a menu screen.

Edit: It looks like sub app is the way to go.

[Architector4]

Stumbled upon this randomly... I wonder if sub-apps is what you all want? They are effectively "sub-worlds" for an app (i.e. second-class to the main singular world), with an extra layer of an "App" over them, which can help with some extra organization (plugins, etc).

You could totally put all the critical game logic inside a sub-app with an otherwise empty world, make its main schedule have a system that rigidly advances time in exact time steps, and sync to/from it as needed.

[djeedai]

There's no mechanism to "extract" from that sub-app back into the main world. The "extract" mechanism of the render world is directly hooked into internal systems, there's no way to reproduce it neither in reverse direction nor (if I remember correctly) for other sub-apps.

[jwright159]

The subapp extract method has mutable access to both worlds, so what's keeping an extraction function from also working in the other direction?

[djeedai]

Last I checked this was not possible. It looks like this changed, so maybe now one can use a subapp. I didn't try again recently.