Introduction

Having started working on games around two and a half years ago, I've already established that designing game-specific systems and tools is the part of development I find the most fulfilling. In my mind, this also extends to the underlying engine features which allow the various sub-systems to come together in a meaningful way for users.

Nine months ago I've switched to using Godot after getting increasingly frustrated with Unity's overall design mis-aligning with what I'd personally find more effective, both in terms of workflow and performance. Whilst I much prefer Godot for making games themselves, I still have a growing itch to work towards an engine of my own. My current motivations are mostly to learn and experiment more deeply with different approaches and ideas, such as Entity-Component Systems (often referred to as 'ECS'), GPU-driven rendering, world state rollback, hot-reloading and much more, all of which I'd like to explore more.

And so I've decided to dive down the rabbit hole of game engine development (I genuinely can't wait!), starting with the creation of the "Polymer Engine" (inspired by its flexible + modular design goals) as a springboard to learn more about modern CPU + GPU architectures, highly-threaded computing, data-oriented design and various other related topics. It's currently closed source whilst I settle on a good basis, but then I'll continue development openly as an open source project!

Backstory + Reasons

Ever since I was a kid, I always had this dream of working on video games in some form. Now that I've (finally) tried to make some games all by myself for a while, I've realized the monumental amount of work required, especially for the sorts of larger games I've always strived to realize. All this requires a lot more than just programming (damn it!).

So, the next best thing then? Finding and reaching out to like-minded people, collaborating and/or working for one of the game studios I truly admire and generally finding ways to contribute in the areas I currently feel I excel the most at: systems and tools programming.

I want to become proficient enough to create awesome tooling that make game dev more joyful, efficient and painless. So in my mind, working on an engine would be the perfect fit for reaching these goals and gaining said proficiency.

Design Goals

This engine is initially focused on getting a solid-enough set of core features in place that'll inform the architecture of the engine from there onwards. Once at that point, I'll start work towards getting a prototype demo working for the types of games I want to make, so that I can identify and prioritize development towards features I actually need, rather than creating assumptions on what I believe I might need. And even whilst developing those features, I'll be writing the code with the expectation that it'll be refactored multiple times as I figure out my goals and iterate on improving the interface for reaching those goals once they're more defined.

I'm using this project to try out ECS game development, since it feels far more inline with my ideal workflow of development so far. I also want the engine to be highly scalable, so that it can run at performant framerate targets on a wide range of hardware, not just the latest and greatest. I'm also putting significant effort into reducing iteration times, since it really hurts development in all aspects, especially more so as a team gets larger. I'm hoping to incorporate hot-reloading for all sorts of assets such as shaders, models, textures, scenes, config, resources, etc. I've also made it a mission to have C++ code hot-reloadable via runtime-linked modular libraries, which I'm not expecting to handle all kinds of code changes, but enough to make various iterative changes significantly quicker. Iteration can also be improved by reducing friction for the developer(s) by making an understandable and intuitive API for creating all sorts of features within the engine.

Current Work

Much of this work was actually already done a few months ago before I decided I'll work on my current Godot game project some more instead to gain more game development experience to better understand what my engine's requirements should be. After having dived into Godot's internals as a result of using GDExtension extensively, I felt I should just give it a try and iterate instead of indefinitely waiting till I'm ready to do so.

Module System

Note: By "modules", I'm not referring to C++20's Modules, but rather more traditional libraries/packages that you include/link.

I've already gotten an initial version on the module system working, which I'll go over in a future blog post once I've worked out most of the kinks (it is heavily based on how Project Island (by Tim Gfrerer, @tgfrerer) approaches its module system!). It separates most sub-systems into their own modules with the main benefits being how it encourages decoupling of sub-systems from one another, and as a consequence of that, you can actually optionally dynamically link these modules at runtime, allowing for hot-reloadable C/C++ code during development. For distributed builds, the libraries are statically linked together.

The only downside to this approach is that the modules have to use a C-based API (which I'm starting to enjoy in some ways) for cross-module communications since C++ scrambles symbols (each compiler does so differently + typically non-deterministically as well), and the compiler can't do various optimizations such as inlining since my approach relies on function pointers a bunch (I'm working to change this!). An extra bonus of this system is that it can be repurposed to offer extensive modding support, with user-created modules for games and/or tools themselves, with a direct API to all the other modules. There are likely security concerns that I'm simply inexperienced enough to know about, but then again most modding isn't exactly safe in most games but built around an honor system (although recent events with Minecraft modding has proven this isn't ideal).

Testbed Project

I have created a testbed project that serves as a testing ground for various features such as the module system above but also third-party libraries integration to be setup. I've managed to get quite a few core systems working already, such as:

• Windowing + Input (using SDL3)
• Logging (using spdlog)
• Testing (using Catch2)
• ECS (using Flecs)
• Very Basic Graphics (using OpenGL 3.3 via glad)
• Physics (using JoltPhysics for 3D and Box2D for 2D)
• Profiling (using tracy)
• Immediate Mode GUI (using Dear ImGui)
• Math (using GLM)

I have a basic first-person shooter (FPS) controller driven by physics, working in a barebones environment that utilizes all these with some nice integration already. It has a fixed timestep for physics, and the variable timestep interpolates transforms to support smooth motion on high-refresh rate monitors. I have shader and module hot-reloading working fully in this example, even with all the third-party libraries (only on Mac + Linux, since Windows needs more work thanks to file locking...).

Here's a small code snippet of the current API, showing how I spawn the thrown physics bodies:

// This is called by the host application continuously until it returns false, which indicates that the app
// should exit. Module reloads are checked and handled in-between updates by the host application.
static bool fps_demo_main_update(fps_demo_main_o *state) {
	// The state object stores all the module state, it's passed as a pointer since the host application
    // must manage the memory so that it survives reloads and we have to use a C-based API for standard ABI
    // compatibility reasons.

    // ...

	if (state->input->keys[SDL_SCANCODE_T].IsJustPressed()) {
        // The transform is decomposed from the matrix instead of using individual components since we want
        // the global transform, because the individual components are local-only. The camera is a child of
        // the player entity with a fixed-offset position + independent rotation. The transform API needs
        // far more work since there is no indication of what is global or local + it's messy when the global
        // and/or local transform should be updated (I'll use Flecs' pairs and sync points to improve both!).
		const Core::Transform3* cam_trans = camera_entity.get<Core::Transform3>();
		vec3 cam_pos = vec3(cam_trans->mat[3]);
		vec3 cam_forward = MathUtils::Vec3::FORWARD * quat_cast(cam_trans->mat);
		
		CreateThrowableBox(state->ecs, cam_pos + cam_forward * 2.0f, cam_forward * 100.0f, vec3(2.0f));
	}

	// ...
}

void CreateThrowableBox(flecs::world *ecs, vec3 pos, vec3 velocity, vec3 size) {
	flecs::entity entity = ecs->entity()
			.emplace<Core::Position3>(pos)
			.emplace<Core::Rotation3>(MathUtils::Quat::IDENTITY)
			.emplace<Core::Transform3>(MathUtils::Mat4x4::IDENTITY)
			.emplace<Renderer::SolidColor>(vec4(0.38f, 1.0f, 0.0f, 1.0f))
            .set<Renderer::MeshPrimitiveCube>({.dimensions = vec3(size)})
			.add<Renderer::VertexArray>();

    // In substitute of an asset manager, I am temporarily using an 'ASSETS_PATH' define to the
    // assets folder path, which changes based on build mode
	Renderer::AddShaderToEntity(entity, ASSETS_PATH "shaders/lit.vs", ASSETS_PATH "shaders/lit.fs");

	const JPH::BodyID& body_id = Physics::AddBoxPhysicsBodyToEntity(entity, vec3(size), true);

    JPH::BodyInterface* body_interface = ecs_world->get<Physics::ServerBodyInterface>()->ptr;
    body_interface->SetLinearVelocity(body_id, MathUtils::Vec3::ToJPH(velocity));
}

Near Future Goals

Dual Update Loop (Fixed + Variable)

A diagram attempting to visualize my mental model of a dual update loop timeline. It's based on Unity's blog post about fixing delta time

I've been planning on paper a different main loop approach that tries to achieve having both a fixed interval, "deterministic" (in the sense that the same input stream should result in the same output) update loop and a variable presentation update loop at the same time. The main difference of having both updates run concurrently as opposed to one after another, is to reduce frame time spikes caused by the logic update, since it almost always takes longer than the variable update. It's practically a local version of a client-server architecture, where the variable update makes predictions to reduce input latency and corrects them via a rollback-like system. I'm still figuring out if this is just over complicating things (it almost definitely is!), but I'll do a blog post about it regardless of the outcome eventually.

Multi-Threading Focus

One thing that I've slowly become saddened by is the lack of full threading in many of the largely available game engines, mostly because it's a very hard problem on it's own, and especially when adding it to a legacy codebase without breaking compatibility too much, because it simply wasn't a priority/consideration yet due to the hardware available at the time. Now, since I'm making this in an age where 6-core CPUs are the most common for gamers (according to the Steam Hardware Survey) and core/thread counts are increasing far more than clock speeds (though clock speeds are "stagnating", Instructions-Per-Clock (IPC) is often making up for this currently), it just makes sense to design the engine to best utilize the available cores in as many ways as possible. The common approach is to use a job system and I'll likely end up doing so too, but I'll try and design most sub-systems to utilize all the threads (within reason, I still am learning a bunch of stuff for this first!).

Future Blog Series + Outro

So as previously mentioned, I'm hoping to go into more depth with each section of the engine as I get to them, hopefully providing insights into design decisions and details related to the implementation itself! And once again, I'm incredibly excited to really kick this project off now that I've found myself far more time.

If you have any feedback, want to see more regular updates and/or just generally chat about this kind of stuff, I'm readily available on Mastodon!