Skip to content
Back to Blog

Under the Hood: Per-Mod VM Isolation

By Rawframe Team
technical architecture modding

The Problem with Shared State

In many moddable game engines, all mods share a single scripting environment. One global Lua state, one set of globals, one namespace. This is simple to implement, but it creates serious problems:

  • One bad mod crashes everything. An infinite loop or memory leak in Mod A takes down Mod B and Mod C along with it.
  • Name collisions are invisible. Two mods defining a global update() function silently overwrite each other.
  • Hot-reload is all-or-nothing. You can’t reload one mod without tearing down the entire script state.
  • Resource tracking is impossible. You can’t tell which mod is using how much memory or CPU time.

This is a proven approach used by major modding platforms.

One VM Per Mod

When the Rawframe server loads a mod, it creates a dedicated lua_State* for that mod. Each VM gets:

  • Its own global table — no name collisions between mods
  • Its own sandboxed environment — file system access, os.execute, and other dangerous APIs are blocked
  • Its own memory budget — a mod that allocates too much memory gets terminated, not the entire server
  • Its own error boundary — a runtime error in one mod is caught and logged without affecting others

The ModVMManager orchestrates all of this. It owns a map of mod IDs to their VM instances and handles the lifecycle: creation, script loading, tick dispatch, hook routing, and teardown.

Cross-VM Communication

Isolated VMs can’t share Lua values directly — a table in VM-A is meaningless to VM-B. Rawframe solves this with LuaValue serialization.

When Mod A calls mod.call("mod-b", "getScore", playerId), the engine:

  1. Serializes the arguments into a LuaValue tree (a C++ variant type that can represent nil, bool, number, string, and tables)
  2. Looks up Mod B’s exported function in its VM
  3. Deserializes the arguments into Mod B’s Lua state
  4. Calls the function
  5. Serializes the return value back
  6. Deserializes it into Mod A’s state

This is not free — there’s a serialization cost for every cross-mod call. But in practice, inter-mod calls are infrequent (game logic runs within a single mod), and the safety guarantee is worth the overhead.

The serializer handles nested tables with circular reference detection. If a table refers to itself (directly or indirectly), the serializer stops with an error rather than looping forever.

Hook Dispatch and Re-Entrance

Rawframe uses a priority-based hook system. When the engine fires a hook like "Think" or "PlayerConnect", the ModVMManager iterates over all mod VMs and calls each mod’s registered handler for that hook.

This raises a subtle problem: re-entrance. What if Mod A’s Think handler calls mod.call() into Mod B, which fires another hook, which calls back into Mod A? Without protection, you get infinite recursion.

Rawframe prevents this with a re-entrance guard. Each mod VM tracks whether it’s currently executing. If a hook dispatch tries to enter a VM that’s already running, the call is skipped (or deferred, depending on the hook type). This is a simple boolean flag per VM, but it prevents an entire class of hard-to-debug stack overflows.

Hot-Reload Without Downtime

Per-mod VMs make hot-reload straightforward:

  1. Detect that a mod’s files have changed (file watcher or manual trigger)
  2. Call all cleanup hooks in the mod’s VM (OnUnload)
  3. Destroy the mod’s lua_State*
  4. Create a fresh lua_State*
  5. Re-register the engine API functions
  6. Load and execute the mod’s scripts
  7. Fire the OnLoad hook

The entire process takes milliseconds. Other mods keep running uninterrupted. Players on the server don’t disconnect. This is essential for mod development — change a script, hit reload, see the result instantly.

Memory Limits and Profiling

Each mod VM is created with a custom allocator that tracks total memory usage. The engine enforces per-mod memory limits (configurable by the server operator). When a mod exceeds its limit, the allocator returns NULL, the Lua state throws an out-of-memory error, and the mod is terminated cleanly.

The ScriptProfiler can also track per-mod CPU usage by hooking into Luau’s interrupt system. Server operators can see exactly which mod is consuming the most resources and act accordingly.

Why This Matters

Per-mod VM isolation isn’t glamorous, but it’s the foundation that makes everything else possible. It means:

  • Server operators can run untrusted community mods without fear
  • Mod developers can iterate with instant hot-reload
  • One buggy mod can’t ruin the experience for everyone
  • The engine can enforce fair resource usage across mods

It’s the kind of infrastructure that, when it works well, you never notice. And that’s exactly the point.

Back to Blog