6

In my C++ project I'm using a source generator to embed some resources into the binary.

I use CMake to build my project and my code works but had some issues. I am pretty sure that what I want to accomplish is possible but I didn't find any answer online.

The current problems I have are:

  • The generator runs every time, even if the input files did not change. This is not too big of a deal because it is really fast, but I hopped there was a better way to do it

  • While using Ninja the generator runs at every build (as described above) without rebuilding every time. I think that Ninja sees that the file has not changed and does not build it again, but when I make changes in the resources change it still uses the old version. It takes another build to "realize" that the generated file has changed and rebuild it

  • While using Make the code rebuilds every time, even when the generated file does not change, resulting in wasted build time

In both cases (looking at the output) the generator runs before the compiler.

This situation is not unsustainable but I was wondering if a better solution was possible. Here's a code snippet from my CMakeLists.txt

add_subdirectory(Generator) file(GLOB RESOURCES Resources/*) add_custom_command( OUTPUT src/Resources/Generated.hpp src/Resources/Generated.cpp COMMAND Generator ${RESOURCES} DEPENDS ${RESOURCES} DEPENDS Generator ) add_custom_target(Generated DEPENDS src/Resources/Generated.hpp src/Resources/Generated.cpp) add_dependencies(${PROJECT_NAME} Generated)

Here's a minimal reproducible example, sorry for not having it before.

EDIT: I implemented the functional solution from the correct answer in the fix branch, on the same repo. It might be useful for future reference :)

Improve this question
2
  • 1
    If you've got a file that is generated properly (as described in the answer), you don't actually need a separate target+add a dependency, unless you want to make it possible to generate only the files: simply list both generated files (absolute paths) in the list of sources for ${PROJECT_NAME} and cmake should realize those files may need updating. Commented Apr 18, 2022 at 8:14
  • Furthermore the use of PROJECT_SOURCE_DIR should be avoided: It relies on the presence of the project command in a specific cmake file putting restrictions on other parts of the file you must be aware of when modifying the file. Use CMAKE_CURRENT_SOURCE_DIR which always refers to the dir containing the CMakeLists.txt file containing the variable reference. Commented Apr 18, 2022 at 8:19

1 Answer 1

Reset to default
7

This one is interesting, because there are multiple errors and stylistic issues, which partially overlap each other.

First off:

file(GLOB_RECURSE SRC src/*.cpp src/*.hpp) add_executable(${PROJECT_NAME} ${SRC})

While convenient in the beginning, globbing your sources is not a good idea. At some point you will have a testme.cpp in there that should not be built with the rest, or a conditionally_compiled.cpp that should only be compiled if a certain option is set. You end up compiling sources that you really did not intended to.

In this case, the file src/Generated.hpp from your git repository. That file is supposed to be generated, not checked out from repo. How did it even get in there?

add_custom_command( OUTPUT src/Generated.hpp COMMAND ${PROJECT_SOURCE_DIR}/generator.sh ${PROJECT_SOURCE_DIR}/Resources/data.txt > ${PROJECT_SOURCE_DIR}/src/Generated.hpp DEPENDS Resources/data.txt DEPENDS ${PROJECT_SOURCE_DIR}/generator.sh )

Do you see the output redirection there? You wrote to ${PROJECT_SOURCE_DIR}. That is not a good idea. Your source tree should never have anything compiled or generated in it. Because these things end up being committed with the rest... like it happened to you.

Next issue:

add_custom_target(Generated DEPENDS src/Generated.hpp)

This creates a make target Generated. Try it: make Generated. You keep getting the following output:

[100%] Generating src/Generated.hpp [100%] Built target Generated

Obviously it does not realize that Generated.hpp is already up-to-date. Why not?

Let's look at your custom command again:

add_custom_command( OUTPUT src/Generated.hpp COMMAND ${PROJECT_SOURCE_DIR}/generator.sh ${PROJECT_SOURCE_DIR}/Resources/data.txt > ${PROJECT_SOURCE_DIR}/src/Generated.hpp DEPENDS Resources/data.txt DEPENDS ${PROJECT_SOURCE_DIR}/generator.sh )

What if I told you that your OUTPUT is never actually generated?

Quoting from CMake docs on add_custom_command, emphasis mine:

OUTPUT

Specify the output files the command is expected to produce. If an output name is a relative path it will be interpreted relative to the build tree directory corresponding to the current source directory.

So your output claims to be to the binary tree, but your command's redirection is to the source tree... no wonder the generator keeps getting re-run.

Having your header generated in the wrong location does not give you a compiler error, because that header from your source tree gets picked up by your GLOB_RECURSE. As that one keeps getting re-generated, your executable keeps getting recompiled as well.

Try this from your build directory:

mkdir src && touch src/Generated.hpp && make Generated

Output:

[100%] Built target Generated

You see that make has nothing to do for the Generated target, because it now sees an OUTPUT of your custom command that is newer than its dependencies. Of course, that touched file isn't the generated one; we need to bring it all together.

Solution

Don't write to your source tree.

Since ${PROJECT_BINARY_DIR}/src does not exist, you need to either create it, or live with the created files on the top dir. I did the latter for simplicity here. I also removed unnecessary ${PROJECT_SOURCE_DIR} uses.

add_custom_command( OUTPUT Generated.hpp COMMAND ${PROJECT_SOURCE_DIR}/generator.sh \ ${PROJECT_SOURCE_DIR}/Resources/data.txt \ > Generated.hpp DEPENDS Resources/data.txt DEPENDS generator.sh )

Don't glob, keep control over what actually gets compiled:

add_executable( ${PROJECT_NAME} src/main.cpp )

Add the binary tree to your target's include path. After add_executable:

target_include_directories( ${PROJECT_NAME} PRIVATE ${PROJECT_BINARY_DIR} )

That's it, things work as expected now.

Improve this answer
Sign up to request clarification or add additional context in comments.

4 Comments

I added an example, sorry for not having it before
@Billy: See updated answer.
I'll give it a try tomorrow, but thank you so much for your time.
It did fix my problem! I'm still a newbie (in case you didn't notice...), your help means a lot, thank you :)

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.