How exactly does CMake work? Why are so many files generated?

Question

What exactly was going on behind the scenes when for such a small CMakeLists.txt file,

cmake_minimum_required (VERSION 2.6) project(Tutorial) add_executable(Tutorial tutorial.cpp)

and such a small tutorial.cpp

int main() { return 0; }

there are so many files generated:

CMakeCache.txt cmake_install.cmake Makefile CMakeLists.txt tutorial.cpp

And a CMakeFiles folder with so many files and folders:

CMakeCCompiler.cmake CMakeOutput.log Makefile.cmake cmake.check_cache CMakeSystem.cmake progress.marks CMakeCXXCompiler.cmake CMakeTmp TargetDirectories.txt CMakeDetermineCompilerABI_C.bin CompilerIdC Tutorial.dir CMakeDetermineCompilerABI_CXX.bin CompilerIdCXX CMakeDirectoryInformation.cmake Makefile2

Not understanding what was going on behind the scenes (i.e: why so may files had to be generated and what their purpose was), was the biggest obstacle in being able to learn CMake.

What is the purpose of these files, and when I type cmake ., what exactly is CMake configuring and generating before it builds the project?

I'm aware of out-of-source builds. In case anyone did not do an out of source build and is still looking for a way to clean generated files, this technique works well: stackoverflow.com/a/12055610/453673 — Nav
– Nav, Commented Jul 9, 2013 at 3:13
There's wonderful description at : aosabook.org/en/cmake.html and probably in-depth answer to question (which cannot be summarized in short here). — parasrish
– parasrish, Commented Jun 8, 2016 at 11:08
@Nav Yap, messed up the markup syntax, sorry. Thus, here the corrected version: For people new to cmake I really recommend reading architecture of cmake. It provides just enough information to get a feeling how cmake works without loosing itself into details. — clickMe
– clickMe, Commented Mar 2, 2017 at 10:17

atakli · Accepted Answer · 2021-07-13 14:36:01Z

The secret is that you don't have to understand what the generated files do.

CMake introduces a lot of complexity into the build system, most of which only pays off if you use it for building complex software projects.

The good news is that CMake does a good job of keeping a lot of this messiness away from you: Use out-of-source builds and you don't even have to look at the generated files. If you didn't do this so far (which I guess is the case, since you wrote cmake .), please check them out before proceeding. Mixing the build and source directory is really painful with CMake and is not how the system is supposed to be used.

In a nutshell: Instead of

cd <source_dir> cmake .

always use

cd <build_dir_different_from_source_dir> cmake <source_dir>

I usually use an empty subfolder build inside my source directory as build directory.

To ease your pain, let me give a quick overview of the relevant files which CMake generates:

Project files/Makefiles - What you are actually interested in: The files required to build your project under the selected generator. This can be anything from a Unix Makefile to a Visual Studio solution.
CMakeCache.txt - This is a persistent key/value string storage which is used to cache value between runs. Values stored in here can be paths to library dependencies or whether an optional component is to be built at all. The list of variables is mostly identical to the one you see when running ccmake or cmake-gui. This can be useful to look at from time to time, but I would recommend to use the aforementioned tools for changing any of the values if possible.
Generated files - This can be anything from autogenerated source files to export macros that help you re-integrate your built project with other CMake projects. Most of these are only generated on demand and will not appear in a simple project such as the one from your question.
Anything else is pretty much noise to keep the build system happy. In particular, I never needed to care about anything that is going on inside the CMakeFiles subdirectory.

In general you should not mess with any of the files that CMake generates for you. All problems can be solved from within CMakeLists.txt in one way or the other. As long as the result builds your project as expected, you are probably fine. Do not worry too much about the gory details - as this is what CMake was trying to spare you of in the first place.

Thanks. Was aware of the out of source builds, but as you'd see, the objective of asking this question was to understand more. The CMakeCache.txt part of your answer really helped. It's this kind of explanation that I was looking for. Also, the last sentence of my question: "what exactly is cmake configuring and generating before it builds the project"
CMakeFiles directory contains CMakeError.log and CMakeOutput.log important for troubleshooting CMake builds.
Two things that sometimes you need to check in the CMakeFiles/target_name.dir directory are the flags.make and link.txt files. I've had to check these in complex project, which inherited flags/linked-stuff from upstream projects. E.g,: I had some errors because TPL A had a dependency on TPL B. Also my project had a dependency on B, which I installed locally; but A used a version of B from my system (rather than my local install) which had different enabled options, and it came first, causing errors in my project. Only looking at link.txt I was able to find out this was happening.
I found this to be a cool resource on this topic: aosabook.org/en/cmake.html (from "the architecture of open source applications")
@bartgol Thanks for this but it works only with Makefile generator. Say if you use Ninja the generated files are different. CMake doesn't have a command to do this yet; would be nice to have it though.

Peter Mortensen · Accepted Answer · 2023-05-24 20:45:20Z

As stated on its website:

CMake is cross-platform, open-source build system for managing the build process of software using a compiler-independent method

In most cases, it is used to generate project/make files. In your example, it has produced Makefile which are used to build your software (mostly on Linux/Unix platform).

CMake allows to provide cross-platform build files that would generate platform-specific project/make files for a particular compilation/platform.

For instance, you may to try to compile your software on Windows with Visual Studio, and then with proper syntax in your CMakeLists.txt file you can launch

cmake .

inside your project's directory on the Windows platform. CMake will generate all the necessary project/solution files (.sln, etc.).

If you would like to build your software on the Linux/Unix platform, you would simply go to source directory where you have your CMakeLists.txt file and trigger the same cmake . and it will generate all files necessary for you to build software via a simple make or make all.

Here you have some very good presentation about key CMake functionalities: Learning CMake (Pau Garcia i Quiles)

If you'd like to make platform-dependent library includes / variable definitions, etc., you can use this syntax in the CMakeLists.txt: file:

IF(WIN32) ...do something... ELSE(WIN32) ...do something else... ENDIF(WIN32)

There are also a lot of commands with use of which you are able to prevent the build from failing and in place CMake will notify you that, for instance, you do not have Boost libraries filesystem and regex installed on your system. To do that you can use the following syntax:

find_package(Boost 1.45.0 COMPONENTS filesystem regex)

Having checked that it will generate the makefiles for appropriate system/IDE/compiler.

Thanks Patryk. Helpful answer, but the pdf you linked to, explains mostly the second stage of learning CMake, which is basically implementation of CMake. I'm asking about the explanation of the first stage, where a person has to understand what on earth is going on in CMake!
@Nav I have expanded my answer a little. You can still search the web for more information.

Peter Mortensen · Accepted Answer · 2023-05-24 20:59:40Z

Exactly how CMake works is a question for the developers, so this question can't be answered here.

However, we can give a touch of useful guidance as far as when you should use CMake and when you therefore need to worry about how it works. I'm not a fan of "oh, it just works" answers either—because, especially in software, nothing ever "just works" and you always have to get into the nitty-gritty details at some point.

CMake is an industrial-strength tool. It automates several very complex process and takes into account many variables of which you may not be aware, especially as a fairly new developer, probably working with limited knowledge of all the operating systems and build tools CMake can handle. The reason so many files are generated and why things seem so complex is because all of those other systems are complex and must be accounted for and automated.

Additionally, there are the issues of "caching" and other time-saving features of the tool. To understand everything in CMake would mean understanding everything in these build tools and operating systems and all the possible combinations of these variables, which as you can imagine is impossible.

It's important to note that if you're not in charge of managing a large cross-platform build system, and your code base is a few KLOC, maybe up to 100 KLOG, using CMake seems a little bit like using a 100,000 dollar forestry tree removal machine to remove weeds from your 2 feet by 2 feet (60 x 60 cm) flower garden. (By the way, if you've never seen such a machine, you should look for one on YouTube; they're amazing.)

If your build system is small and simple it's likely to be better to just write your own makefiles by hand or script them yourself. When your makefiles become unwieldy or you need to build a version of your system on another platform, then you can switch over to CMake. At that point, you'll have lots of problems to solve and you can ask more focused questions about it. In the meantime, check out some of the great books that have been written about CMake, or even better, write one yourself! 8)

Collectives™ on Stack Overflow

How exactly does CMake work? Why are so many files generated?

3 Answers 3

5 Comments

3 Comments

Comments

Linked

Hot Network Questions