Clion Unity

Posted : admin On 1/26/2022
  1. Clinton United States
  2. Clion Unity Point
  3. Clion Unit Testing

A unity build can cut down build times dramatically and is HIGHLY underrated and easily dismissed by many senior software engineers (just like precompiled headers). In this post we will go over what it is, all its pros and cons, and why a “dirty hack” might be worth it if it speeds up your builds by at least a factor of 2 or perhaps even in the double-digits.

Unity allows building for mobiles, PCs, consoles and even more. But, programming in Unity is done in C#, so that’s out of consideration. Unreal Engine on the other hand builds for all the mentioned platforms above and programming is done in C. So it’s your best bet! Check out the Course: dive into the 1 piece of advice that I ignored for years. Why and when I decided to follow it, an.

Why care about build times in the first place

Well… time is money! Let’s do the math: assuming an annual salary of 80k $ - waiting for 30 minutes extra a day for builds is 1/16 of the time of a developer > 5k $ per year. 200 such employees and we reach 1 million $ annually. But employees bring more value to the company than what they get as a salary (usually atleast x3 - if the company is adequate) - so the costs to the employer are actually bigger.

Now let’s consider the facts that waiting for long builds discourages refactoring and experimentation and leads to mental context switches + distractions which are always expensive - so we reach the only possible conclusion: any time spent reducing build times is worthwhile!

Introduction - a short summary to unity builds

A unity build is when a bunch of source files are #include‘d into a single file which is then compiled:

Also known as: SCU (single compilation unit), amalgamated or jumbo.

The main benefit is lower build times (compile + link) because:

  • Commonly included headers get parsed/compiled only once.
  • Less reinstantiation of the same templates: like std::vector<int>.
  • Less work for the linker (for example not having to remove N-1 copies of the same weak symbol - an inline function defined in a header and included in N source files).
  • less compiler invocations.

Note that we don’t have to include all sources in one unity file - as an example: 80 source files can be split in 8 unity files with 10 of the original sources included in each of them and then they can be built in parallel on 8 cores.

United

Why redundant header parsing/compilation is slow:

Here is what happens after including a single header with 2 popular compilers and running only the preprocessor (in terms of file size and lines of code):

headerGCC 7 sizeGCC 7 locMSVC 2017 sizeMSVC 2017 loc
cstdlib43 kb1k loc158 kb11k loc
cstdio60 kb1k loc251 kb12k loc
iosfwd80 kb1.7k loc482 kb23k loc
chrono180 kb6k loc700 kb31k loc
variant282 kb10k loc1.1 mb43k loc
vector320 kb13k loc950 kb45k loc
algorithm446 kb16k loc880 kb41k loc
string500 kb17k loc1.1 mb52k loc
optional660 kb22k loc967 kb37k loc
tuple700 kb23k loc857 kb33k loc
map700 kb24k loc980 kb46k loc
iostream750 kb26k loc1.1 mb52k loc
memory760 kb26k loc857 kb40k loc
random1.1 mb37k loc1.4 mb67k loc
functional1.2 mb42k loc1.4 mb58k loc
all of them2.2 mb80k loc2.1 mb88k loc

And here are some (common) headers from Boost (version 1.66):

headerGCC 7 sizeGCC 7 locMSVC 2017 sizeMSVC 2017 loc
hana857 kb24k loc1.5 mb69k loc
optional1.6 mb50k loc2.2 mb90k loc
variant2 mb65k loc2.5 mb124k loc
function2 mb68k loc2.6 mb118k loc
format2.3 mb75k loc3.2 mb158k loc
signals23.7 mb120k loc4.7 mb250k loc
thread5.8 mb188k loc4.8 mb304k loc
asio5.9 mb194k loc7.6 mb513k loc
wave6.5 mb213k loc6.7 mb454k loc
spirit6.6 mb207k loc7.8 mb563k loc
geometry9.6 mb295k loc9.8 mb448k loc
all of them18 mb560k loc16 mb975k loc

The point here is not to discredit Boost - this is an issue with the language itself when building zero-cost abstractions.

So if we have a few 5 kb source files with a 100 lines of code in each (because we write modular code) and we include some of these - we can easily get hundreds of thousands of lines of code (reaching megabyte sizes) for the compiler to go through for each source file of our tiny program. If some headers are commonly included in those source files then by employing the unity build technique we will compile the contents of each header just once - and this is where the biggest gains from unity builds come from.

A common misconception is that unity builds offer gains because of the reduced disk I/O - after the first time a header is read it is cached by the filesystem (they cache very aggressively since a cache miss is a huge hit).

The PROS of unity builds:

  • Up to 90+% faster (depends on modularity - stitching a few 10k loc files together wouldn’t be much beneficial) - the best gains are with short sources and lots of (heavy) includes.
  • Same as LTO (link-time optimizations - also LTCG) but even faster than normal full builds! Usually LTO builds take tremendously more time (but there are great improvements in that area such as clang’s ThinLTO).
  • ODR (One Definition Rule) violations get caught (see this) - there are still no reliable tools for that. Example - the following code will result in a runtime bug since the linker will randomly remove one of the 2 methods and use the other one since they seem to be identical:
  • Enforces code hygiene such as include guards (or #pragma once) in headers

Clinton United States

Clion Unity

Clion Unity Point

The CONS:

  • Not all valid C++ continues to compile:
    • Clashes of symbols with identical names and internal linkage (in anonymous namespaces or static)
    • Overload ambiguities (also non-explicit 1 argument constructor…?)
    • Using namespaces in sources can be a problem
    • Leaked preprocessor identifiers after some source which defines them
  • Might slow down some workflows:
    • Minimal rebuilds - but if a source file can be excluded from the unity ones for faster iteration it should all be OK
    • Might interfere with parallel compilation - but that can be tuned by better grouping of the sources to avoid “long poles” in compilation
  • Might need a lot of RAM depending on how many sources you combine.
  • One scary caveat is a miscompilation - when the program compiles successfully but in a wrong way (perhaps a better matching overload got chosen somewhere - or something to do with the preprocessor). Example:

    If b.cpp ends up before a.cpp then we would get 84 instead of 42. However I haven’t seen this mentioned anywhere - people don’t run that much into it. Also good tests will definitely help.

How to maintain

We can manually maintain a set of unity source files - or automate that:

  • CMake: either cotire or this

It is desirable to have control on:

  • how many unity source files there are
  • the order of source files in the unity files
  • the ability to exclude certain files (if problematic or for iterating over them)

Projects using this technique

Clion Unity

Unity builds are used in Ubisoft for almost 14 years! Also WebKit! And Unreal…

Clion Unit Testing

There are also efforts by people who build chrome often (and have gotten very good results - parts of it get built in 30% of the original time) to bring native support for unity builds into clang to minimize the code changes needed for the technique (gives unique names to objects with internal linkage (static or in anonymous namespaces), undefines macros between source files, etc.):