28

On a native C++ project, linking right now can take a minute or two. Yet, during this time CPU drops from 100% during compilation to virtually zero. Does this mean linking is primarily a disk activity?

If so, is this the main area an SSD would make big changes? But, why aren't all my OBJ files (or as many as possible) kept in RAM after compilation to avoid this? With 4 GB of RAM I should be able to save a lot of disk access and make it CPU-bound again, no?

Update: so the obvious follow-up is, can the VC++ compiler and linker talk together better to streamline things and keep OBJ files in memory, similar to how Delphi does it?

Improve this question
1
  • I speculate it's left up to the OS to cache them in RAM to avoid this, which it does, if there's enough RAM to do so on top of compiling. Since compiling takes a lot of RAM, that probably causes the OS to flush the OBJ files to disk. If you force it to keep the OBJ files in memory to make linking faster, it would therefore probably make compiling far slower still. Commented Nov 3, 2014 at 18:36

5 Answers 5

Reset to default
14

Linking is indeed primarily a disk-based activity. Borland Pascal (back in the day) would keep the entire program in memory, which is why it would link so fast.

Your OBJ files aren't kept in RAM because the compiler and linker are separate programs. If your development environment had an integrated compiler and linker (instead of running them as a separate processes), it could indeed keep everything in RAM.

But you would lose the ability to separate the development environment from the compilers and/or linkers - you would have to use the same compiler/linker, and you wouldn't be able to run the compiler outside the environment.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

I thought it might, but with the various incarnations of Delphi, I wasn't sure if it still did.
If you're running on any reasonable OS, the information will already be cached in memory, lessening the need to have the entire object set in memory for linking.
I've run it on XP, Vista and W7 and it doesn't seem to make much difference. I don't suppose W7 provides any way to see what files are being RAM-cached?
The obj and especially pdb files are often huge and simply will not fit into the memory (at least not into the part of the memory system is willing to use as a cache).
I don't believe I end up with 4Gb of temp files when I build... if I did then a couple of years ago I'd keep running out of disk space. Besides, Windows can use wht it has and has near-infinite memory when VM is included.
7

You can try installing some of those RAM disks utilities and keep your obj directory on the RAM disk or even whole project directory. That should speed it up considerably.

Don't forget to make it permanent afterwards :-D

Improve this answer

1 Comment

worth a go... I think you can set intermediate files to go some place separate so it wouldn't need to be permanent except that if the ONJ files go, you'd have to do full builds all the time. Not sure if it's worth the hassle, depends how helpful a RAM disk might be in automating all this
6

The Visual Studio linker is largely I/O bound, but how much so depends on a few variables.

  1. Incremental linking (common in Debug builds) generally requires a lot less I/O.

  2. Writing a PDB file (for symbols) can consume a lot of the time. It's a specific bottleneck that Microsoft targeted in VS 2010. The PDB writing is now done asynchronously. I haven't tried it, but I've heard it can help link times quite a bit.

  3. If you using link-time code generation (LTCG) (common in Release builds), you have all the usual I/O initially. Then, the linker re-invokes the compiler to re-generate code for sections that can be further optimized. This portion is generally much more CPU-intensive. Off hand, I don't know if the linker actually spins up the compiler in a separate process and waits (in which case you'll still see low CPU usage for the linker process), or if the compilation is done in the linker process (in which case you'll see the linker go through phases of heavy-I/O then heavy-CPU).

Using an SSD can help with the I/O bound portions. Simply having a second drive can help, too. For example, if your source and objects are all on one drive, and you write your PDB to a separate drive, the linker should spend less time waiting for the PDB writer. Having a second spinning drive has helped my current team's link times dramatically.

Improve this answer

1 Comment

+1. Moving from spinning-platter to SSD can make an incredible difference with the speed of most build processes. The speed boost usually has to be seen to be believed :-).
6

In debug builds in Visual Studio you can use incremental linking which allows you to usually avoid a lot of the time spent on linking. Basically it means that instead of linking the whole EXE (or DLL) file from scratch it builds upon the one you last linked, replacing only the things that changed.

This is however not recommended for release builds since it adds some overhead in runtime and can result in an EXE file that is several times larger than the usual.

Improve this answer

5 Comments

Sorry, Doesn't address the question.
It does address the issue of making a "big change" in linking performance.
huh? he asked if a ssd would speed up linking if it is io bound.
Offering alternate approaches is just as valid as answering the literal question (unless the question says the alternate approach has already been considered and discarded).
I view Incremental linking (/Gm?) as a valid answer. Sadly I have turned it off anyway in favour of /MP (multi-thread compilation) to use my cores better.
3

It's hard to say what exactly is taking the linker so long without knowing how it is interacting with the OS. Thankfully, Microsoft provides Process Monitor so you can do just that.

It's helped me diagnose bugs with the Visual Studio IDE and debugger without access to source.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.