Configuration I/O Plumbing Trifecta

Alright, long time no update, because updates have been few and very backend-oriented. I'll keep visual updates for another post, as I'll keep this focussed on ... innards and plumbing.

The Plumbing Trifecta

You're making a game, and it uses some configuration database for instantiating entities etc. How do we represent and handle that?

• Just hardcode things, e.g. instantiate a config class in your code: that's ok for jams and tiny projects
• Load from text file, e.g. have it in some form of JSON/XML/etc: that's far better, and needs a bit of infrastructure

But this is where the journey starts really. Now that we have the configuration in the game, many scenarios open up:

• We might want to view and modify our database with in-game/engine GUI: that's useful if you want to tweak things and see results immediately in-game
• We might want to save our changes back to the text file
• Text files will grow in size, and parsing will be slow: oops, now you need to think if there's a faster way of reading/writing this database. And there is, using binary! But now you need to support another import/export "path" for this database

This is the plumbing trifecta, which is very useful for bigger projects at least. If you're using a proper game engine, chances are this is already supported. In Unity, ScriptableObjects offer exactly what I described above. But although using engine-provided solutions is dead convenient, it also comes with gotchas, for example coupling with an engine (what if I want to move engines? I moved from Unity to Godot, not doing that again though) or facing limitations in what types can be serialized (e.g. Unity had trouble with general dictionaries).

I started with the JSON -> game path (one direction), then I added the binary path bidirectionally (binary blob <-> game) and now I wanted to add the in-game editing part, using Dear ImGui, which I did. The ImGui part was implemented mostly using reflection, which makes it nice and scalable for future types. Of course, we might want to save changes to disk, so naturally saving data back is essential. But this meant I had to implement the game -> json direction too, which means go through all JSON converters (I'm using Newtonsoft Json) and write an export path that matches the input. Finally, I had to validate that everything works, so I set up a test that does the following:

• tests that json roundtrip is correct
• tests that binary roundtrip is correct (that's via MemoryPack, so I'm more confident)
• tests that (json -> game -> binary) is byte-wise identical to (binary->game->binary)

What work is needed for new types? The ImGui inspector is automatic for classes that are collections of types that are already handled. The JSON import/export is similarly automatic unless I need special behaviour. And for binary serialization I need to decorate the class members. It's pretty painless!

That's it! There were a lot of issues along the way, that eventually got resolved. The weirdest and worst was the ...

Serialization Heisenbug

During a sanity check transformation cycle between json and binary, I realised that there was some difference between a particular data structure. After lots of time spent, and almost being convinced that the issue was not with my code, I discovered that ... the issue was with my code. Surprise eh? It was mainly because of some lazy initialization/allocation, and because the debugger forced that initialization to happen, I was getting different results if I was or was not inspecting the code ( ergo: heisenbug). After copious amounts of printing and data dumping and inspecting what happens to data when being serialized, it became obvious that in one version of the data an array was null, and in another version it was initialised to zeroes. That was the megahint for pointing to the lazy initialization code. Anyway, that's now found and all the data can transform in all directions happily, and if at some point they don't, some sanity code will start shouting, so all good I suppose.

Object pools, short and long term

I couldn't resist and I dealt with some optimisation issue that has been bugging me, GC related. Reason such a refactor is essential because of scalability. Issue was that I was using reflection and dynamically built params object[] arrays in C# for creating commands that creatures execute. Long story short, performance actually got worse. Looking a bit more into it, I remembered that I'm actually pressuring the GC quite a bit in the turn management system, which is responsible for setting up an ordered container with actions to be executed by entities. All of these were either allocated using an old/limited/context-specific way of doing object reuse, so I put a bit of work to set everything up with my current generic object pool solution. After the dust settled, I realised that I know had what looked like "memory leaks" some objects were never freed.. After a bit of debugging and head-scratching, I realised that I was using the object pool with two distinct patterns: "give me something temporary, and I'll return it ASAP" and "give me an object that I've used before - I just don't want to call new()". So I put a bit of effort to differentiate into two object pools, one for short-term "rentals" and one for long-term rentals. After this, it became clear that all my memory leaks were because I was actually still using the "leaked" objects, as they were actions that were yet to happen; they were back in the turn manager list.