WIP: embed media files

[illwieckz]

Work in progress: embed media files from daemon_src.dpkdir.

I deeply reworked the GLSL include generator to make it reusable, I also renamed DaemonBuildInfo.cmake as DaemonGeneratedSource.cmake and put everything about generating sources there, and revamped it a lot.

It doesn't work yet because:

• The current code uses strings and then files are truncated on first 0, but the sound.wav file contains some bytes like that and this is valid.
• Files aren't listed, so there is no way the engine.shader file gets parsed.

[illwieckz]

So it now loads the whole file.

For a funny reason, it works because since the unvanquished dpk lists those names, the engine looks for them, and then the code to read them works. Of course we should be able to list them even if they aren't in a dpk already.

[illwieckz]

It almost works but…

• This error:

Warn: libpng error: PNG file corrupted by ASCII conversion Warn: PNG image 'gfx/2d/bigchars.png' has first exception handler called [libpng v.'1.6.50'] 

• The files are embedded in all engines (including the server and the tty client).
• I guess the server now requires the client to have that pak loaded just because the engine loaded it.

[illwieckz]

This error

Now fixed, there was a source beautifier stripping \r, now that we not only embed GLSL files, we better drop that.
Also we better beautify the sources in the repository directly.

[slipher]

Now fixed, there was a source beautifier stripping \r, now that we not only embed GLSL files, we better drop that.

Don't drop it. Add an option to specify text or binary mode.

[illwieckz]

Hmm, right now there is something corrupting something, the engine is slow, and the console has some things printed everyframe, actually it prints some PK string so it looks like the console is attempting to print the content of a zip every frame… that's very weird.

[illwieckz]

Don't drop it. Add an option to specify text or binary mode.

I added an option.

[illwieckz]

I now should have filtered out the builtin pak name from the list of required paks sent to clients.

I still don't know from where comes that weird bug slowing down the engine and polluting the console…

[illwieckz]

So I stop there for now, for people around, don't hesitate to look at it, especially for tracking bugs! 😁️

[slipher]

Hmm, in #71 you said we "don't have to think about some generic dpk embedding mechanism", but it looks like you have implemented precisely that. I was thinking we would avoid that and just use the byte arrays generated by the existing embed mechanism directly where needed. For example if we wanted the "bigchars" image, we could add that to the builtin images: create a *consoleFallbackFont image which would upload in the embedded byte array as an image. No need to make it a VFS file. Similarly, for q3shaders, we could just have a string in the source code and feed it into the shader parsing function.

[illwieckz]

Hmm, in #71 you said we "don't have to think about some generic dpk embedding mechanism", but it looks like you have implemented precisely that.

Yes, life is funny. 😄

I started wiring the VFS to test the loading of files, because in fact it's also a good way to make sure the path is exposed and that all the decoding functions are used. For example the PNG image has to be decoded by PNG, etc.

I don't mind the implementation, what I wanted to do there is to plumb the actual file embedding, which seems to now work.

I don't know how to properly make those files usable once they are embedded, and that's a task anyone can pick up.

[illwieckz]

Another thing funny is that I tried to create the smallest null.wav file possible, but then I get this error:

Warn: Unhandled exception (St12length_error): cannot create std::vector larger than max_size() 

My “smaller file” is using such format: signed 16-bit mono 8000 Hz, likely one sample only.

[illwieckz]

Ok, I had to keep enough samples, something like 0.3ms, the file now weights 90 bytes instead of 20KB, here is the generated file:

const unsigned char sound_null_wav[] = { 0x52,0x49,0x46,0x46,0x52,0x00,0x00,0x00,0x57,0x41,0x56,0x45,0x66,0x6d,0x74, 0x20,0x10,0x00,0x00,0x00,0x01,0x00,0x01,0x00,0x40,0x1f,0x00,0x00,0x80,0x3e, 0x00,0x00,0x02,0x00,0x10,0x00,0x64,0x61,0x74,0x61,0x2e,0x00,0x00,0x00,0x00, 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00, 0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,0x00,};

Edit: I also made sure it was truly zeroed.

[illwieckz]

I FOUND THE BUG, IT WORKS.

The bug was that I forgot to set loadedPak.fd = -1 for the loaded builtin pak, meaning later some code was attempting to close the zip for real! I don't know how it did not crashed, and why what looks like some other pak got printed in the terminal (I got PK strings…), but that was that!

[illwieckz]

[FS] Loading builtin pak '*daemon_0.55.5'... [FS] Loading builtin pak '*daemon-client_0.55.5'... [FS] Loading pakdir '…/UnvanquishedAssets/build/_pakdir/pkg/unvanquished_test.dpkdir/'... [FS] Loading pakdir '…/UnvanquishedAssets/build/_pakdir/pkg/tex-common_test.dpkdir/'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-players_0.55.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-legacy_0.55.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-weapons_0.55.3.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-weapons_0.55.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-buildables_0.55.4.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-buildables_0.55.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-voices_0.55.3.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-voices_0.54.dpk'... [FS] Loading pak '~/.local/share/unvanquished/base/pkg/res-soundtrack_0.54.dpk'... ]/which fonts/unifont.ttf File "fonts/unifont.ttf" found in "*daemon-client_0.55.5" ]/which sound/null.wav File "sound/null.wav" found in "*daemon-client_0.55.5" 

[illwieckz]

One can actually run a custom dll game code without any dpk by doing:

./daemonded -set fs_basepak '*daemon' -set vm.cgame.type 3 -set vm.sgame.type 3

[illwieckz]

The annoying downside of embedding a pak in the engine is that, unless I'm wrong, files from a builtin pak are loaded from that builtin pak, i.e. copied in memory. So you get the file taking twice the amount of memory: first in the executable itself, and then as a loaded file.

[illwieckz]

This code may probably be useless because we have solutions to embed all the files we have if I'm right, but I'm using this as a test bed for #1849:

• Genericize the CMake source generator for embedding files #1849

Because this code is much more complex than the GLSL embedding code and also embeds non-text files, so it's very good to test for the robustness of the design and catch pitfalls.