DreamTalon

Maciej Latocha programming and rendering blog. Everything is easy, simple and obvious, when you know the answer.

Decentralized cooking

Cooking – aka Baking – a procedure of converting source data into platform optimized assets.

This often involves (but not limited to) data compression, text binarizing and packaging.
In games, cooking revolves of having one large (even gigabyte size) executable, capable of doing everything. With accompanying dozens of command line arguments, and dozens of shell scripts for convenience of not remembering these arguments in a first place.

Typical problems and challenges of having 1 executable for cooking mostly involves:

  • duplication of convenience_scripts.bat
  • silent deprecation of convenience_scripts.bat
  • initialization of cooker services
  • memory management
  • multi-threading
  • data dependency scanning
  • data packaging

Trying to keep it all both fast and bug-free is indeed very challenging task. A situation where after several hours of run results in:

  • memory leak with 64GiB of RAM being not enough
  • crash on a pointer
  • thread race somewhere
  • memory corruption somewhere
  • incorrect output data

Is the most distressful and heart wrenching experience as a programmer. Especially when you also need several hours to even get to the issue reproduction point. Let alone iterating on fixing it and testing.

Therefore I wanted to write about decentralized cooking, which mostly alleviates mentioned issues. By decentralizing I mean instead of having 1 do-everything executable, you have several ultra tiny executables that do 1 and only 1 thing. A 1000 lines of code for executable is way more manageable and easy to understand and bug free than 1000000 (million) lines of code of a doohickey. Once you have your micro executables, you can plug them into existing build system of your choice. Be it for example CMake, GNU Make, Meson or Ninja.

Now the problem of the following goes away due to:

  • memory management and leaking – all of the tasks are isolated into separate processes, all used resources are released upon process exit – no cumulative leaking
  • multi-threading – execs are invoked in parallel by build system
  • dependency scanning – managed by build system, can be run in parallel. File timestamps are equally reliable as custom hashing algorithms – without the can of worms of hashing algorithms
  • crashing mid run – can be restarted from crash point due to dependency scanning of build system
  • bug fixing and debugging – a single tiny process with few arguments can be precisely iterated and investigated directly from visual studio

Also, build systems are proven to be working efficiently and reliably, by being used world-wide by a millions (empiric value) of users.

As a cherry on top of it, you can now run cooking with network distributed execution out of the box, as network compilation broker would not care whether the executable is named clang++.exe or cooker_texture_dds.exe.

Concerns:

Q: Wouldn’t it be overall slower, to invoke a million of executables for every assets than 1 run of cooker?
A: In principle yes – however there are things at the operating system level to greatly mitigate this, like running from ramdisk or OS RAM caching. Without having possibility of comparing ultra_cooker.exe and cooking via build system, I could not give precise answer which would be faster.

Q: Do I need now to list all of the million of assets in build system? So far I only needed to type wildcard for path matching.
A: Certain build systems do permit you listing a wildcard for file and target matching.

Q: Wouldn’t I now need another several convenience_scripts.bat and keep them up to date?
A: Not really, you are now dealing with build system targets – which are expected to be always up to date.
Plus you can run them directly from visual studio solution explorer.

Example of CMake asset cooking:

add_custom_target( shaders )
function( compileShader file )
    set( OPTIMIZE_LEVEL "-O" )
    set( file_out "${CMAKE_CURRENT_BINARY_DIR}/${file}.spv" )
    add_custom_target( "shader.${file}" DEPENDS "${file_out}" )
    add_dependencies( shaders "shader.${file}" )
    add_custom_command(
        OUTPUT "${file_out}"
        DEPENDS "${CMAKE_CURRENT_SOURCE_DIR}/${file}"
        COMMAND Vulkan::glslc ${OPTIMIZE_LEVEL} "${CMAKE_CURRENT_SOURCE_DIR}/${file}" -o "${file_out}"
    )
endfunction()

compileShader( vehicle.frag )
compileShader( vehicle.vert )
compileShader( aliasing.comp )
compileShader( gamma.comp )
compileShader( gui.frag )
compileShader( gui.vert )

Now when running > make help, you could see the following targets:

... shaders
... shader.vehicle.frag
... shader.vehicle.vert
... shader.aliasing.comp
... shader.gamma.comp
... shader.gui.frag
... shader.gui.vert

And to compile all of the shaders via > make shaders.
Or directly specifying which individual shader you want to compiler via > make shader.gamma.comp.

A reference of using build system for cooking can be found under my little tiny game:
https://github.com/xmaciek/starace/tree/main/assets

More Posts