Upgrading our 15-year-old engine

Daemon, the Unvanquished engine is a descendant of Quake3 (1998) through ioQuake3, Xreal, ET:Xreal and OpenWolf. That's 15 years of tweaks, fixes and features added at random places in the code.

What used to be the “final evolution” of John Carmack's C style has grown into a mess of interdependent modules and special case handling. As a result, Daemon is less maintainable and the apparent lack of clear separation in the codebase scares new developers. Because Unvanquished is a long-term project we need to make our engine friendlier for new developers while becoming competitive with other modern engines.

Modernizing & cleaning the codebase

The Quake 3 family of engines is single-threaded, although it can optionally use a second thread for the renderer. The multithreading that would be the least intrusive to implement would be to use worker threads like Doom3 BFG does. However we eventually want to get close to TTimo's experimental engine in which the inputs, the client side gamelogic and the renderer are in separate threads. Yet, this is not possible without extensive changes, with most modules working like state machines and exposing inner data.

This is why we decided to incrementally rewrite the engine starting from the base framework, decoupling modules and taking multithreading into consideration. In addition, we are switching from C89 to C++11, or more like C with classes and some templates. Our hope is that by rewriting systems one at a time and keeping compatibility with the older engine, we can easily test changes and eventually switch engines seamlessly. I have already been playing all my games using the engine-upgrade branch.

Changing the virtual machines

The virtual machines containing the gamelogic still use QVMs compiled with a LCC (Light C Compiler) that hasn't been updated since the release of Quake 3. The gamelogic is written in C89, comes with its own standard C library, is under-optimized, and can barely do dynamic allocations. Experiments have been made on using LLVM bytecode compiled at runtime but they have security issues and many limitations.

Most new games and engines would use some scripting language like Lua or AngelScript, but we cannot afford to rewrite all of our gamelogic. Instead, we are looking at PNaCl (Portable Native Client) that is used to sandbox native applications in Google Chrome. It solves all of our problems, and it is a proven technology that will hopefully be maintained by Google for a long time. PNaCl basically compiles C/C++ to an instrumented LLVM bytecode that is in turn compiled at runtime into a sandboxed executable. Another blog post will describe in more details the implementation of PNaCl.

Progress and prospects

Work has started in the engine-upgrade branch, first by making all the engine compile and work in C++. We have replaced some C modules with empty shells that translate and proxy calls to their new C++ equivalent. The command system, base commands, the console data-structures and cvars are ported, soon to be followed by the filesystem API and the logging API. We chose these modules because they are at the basis of the engine and should only depend on each other. In its current state, the engine-upgrade branch is fully functional with minor differences in comparison to the older engine. The wiki page tracks these changes. In addition, we implemented PNaCl sandboxes for the gamelogic and successfully played on a PNaCl server with QVM clients.

The next step will be to stabilize the engine-upgrade branch and merge it into master, along with experimental support for PNaCl sandboxes. Overall I'm very hopeful about this project, making incremental changes to allow the engine upgrade to have visible results very quickly, and PNaCl can make both the engine architecture better and gameplay developers more productive.

As a final note, we are not professional engine developers so comments and criticism are very much welcomed.

Refresh Posts
Re: Upgrading our 15-year-old engine -- majkifajki
Monday, 16 September 2013 08:49
Are you going to replace old school VIS with some kind of portal culling? Are you going to switch to dynamic lights like in Doom3?
| Quote |
Re: Upgrading our 15-year-old engine -- kangz
Monday, 16 September 2013 16:02
Maybe. Both of these are barely related to the big picture of the engine, the article was not about the renderer.
| Quote |
Re: Upgrading our 15-year-old engine -- arQon (guest)
Tuesday, 17 September 2013 04:02
As far as cleanup etc goes, the CNQ3 engine updates we did for CPMA a few years ago were migrated to c and build cleanly at W4. We also refactored quite a lot of the code to improve maintainability and modernise a few things; and fixed a few significant bugs like dynamic lighting failing to "notice" certain surfaces depending on view angle etc. It would probably be quite helpful to you. I don't know exactly where you'd FIND it online these days (since promode.org got hijacked a few years ago) but it must be out there somewhere still.
| Quote |
Re: Upgrading our 15-year-old engine -- arQon (guest)
Tuesday, 17 September 2013 04:10
oh - and it's got much better mouse support (i.e. better sens/accel logic, raw input rather than DI, etc) and much better sound code, and a few other areas as well (I really can't remember all the bits we improved, sorry). Not sure how much of that will be meaningful if you're using SDL's godawful mouse code and OAL for sound, but all of the "non-OS" internal stuff we reworked is substantially better than ioq3 was at the time even if you're going to use "third party" libs for the OS-ish parts.
| Quote |
Re: Upgrading our 15-year-old engine -- ishq
Thursday, 19 September 2013 00:26
Right. We're hoping SDL2 will help with input latency, but if it doesn't, we could implement per OS workarounds, though I'm not particularly enthused by the idea. In terms of sound code, OpenAL does the job well enough such that you can hear enough to have map awareness of approaching enemies and be on your guard. Might be interesting to look into checking promode physics though. Would you happen to have the source lying around somewhere?
| Quote |
Re: Upgrading our 15-year-old engine -- arQon (guest)
Monday, 30 September 2013 17:43
yeah, avoiding os-specific stuff is the whole point of using something like sdl in the first place. but sdl input really was absolutely terrible.iirc the problem i had with sdl/openal sound was more that it didn't cull anything sensibly, so it burned huge amounts of cpu. not really a problem these days, i suppose. :)i'll dig out the cpma code: it was several drives ago, so it might take a while, but it's, erm, on ONE of these dvds in here...
| Quote |
Username :
What is the name of the game that we are developing?: