Yocto Project Build Times

Last month our friends at Codethink were guests on FLOSS Weekly, talking about Baserock. Baserock is a new embedded build system with some interesting features/quirks (depending on your point of view) that I won’t go into. What caught my attention was the discussion about build times for various embedded build systems.

Yocto, again, if you want to do a clean build it will take days to build your system, even if you do an incremental build, even if you just do a single change and test it, that will take hours.

(source: FLOSS Weekly #230, timestamp 13:21, slightly edited for clarity)

Now “days” for a clean build and “hours” for re-building an image with a single change is quite excessive for the Yocto Project, but also quite specific. I asked Rob Taylor where he was getting these durations from, and he corrected himself on Twitter:

I’m not sure if he meant “hours” for both a full build and an incremental build, or whether by “hours” for incremental he actually meant “minutes”, but I’ll leave this for now and talk about real build times.

Now, my build machine is new but nothing special. It’s built around an Intel Core i7-3770 CPU (quad-core, 3.4GHz) with 16GB of RAM (which is overkill, but more RAM means more disk cache which is always good), and two disks: a 250GB Western Digital Blue for /, and a 1TB Western Digital Green for /data (which is where the builds happen). This was built by PC Specialist for around £600 (the budget was $1000 without taxes) and happily sits in my home study running a nightly build without waking the kids up. People with more money stripe /data across multiple disks, use SSDs, or 10GB tmpfs filesystems, but I had a budget to stick to.

So, let’s wipe my build directory and do another build from scratch (with sources already downloaded). As a reference image I’ll use core-image-sato, which includes an X server, GTK+, the Matchbox window manager suite and some demo applications. For completeness, this is using the 1.3 release – I expect the current master branch to be slightly faster as there’s some optimisations to the housekeeping that have landed.

$ rm -rf /data/poky-master/tmp/
$ time bitbake core-image-sato
Pseudo is not present but is required, building this first before the main build
Parsing of 817 .bb files complete (0 cached, 817 parsed). 1117 targets, 18 skipped, 0 masked, 0 errors.
...
NOTE: Tasks Summary: Attempted 5393 tasks of which 4495 didn't need to be rerun and all succeeded.

real 9m47.289s

Okay, that was a bit too fast. What happened is that I wiped my local build directory, but it’s pulling build components from the “shared state cache”, so it spent six minutes reconstructing a working tree from shared state, and then three minutes building the image itself. The shared state cache is fantastic, especially as you can share it between multiple machines. Anyway, by renaming the sstate directory it won’t be found, and then we can do a proper build from scratch.

$ rm -rf /data/poky-master/tmp/
$ mv /data/poky-master/sstate /data/poky-master/sstate-old
$ time bitbake core-image-sato
Pseudo is not present but is required, building this first before the main build
...
NOTE: Tasks Summary: Attempted 5117 tasks of which 352 didn't need to be rerun and all succeeded.

real 70m37.298s
user 326m45.417s
sys 37m13.304s

That’s a full build from scratch (with downloaded sources, we’re not benchmarking my ADSL) in just over an hour on affordable commodity hardware. As I said this isn’t some “minimal” image that boots straight to busybox, this is building a complete cross-compiling toolchain, the kernel, X.org, GTK+, GStreamer, the Matchbox window manager/panel/desktop, and finally several applications. In total, 431 source packages were built and packaged, numerous QA tests executed and flashable images generated.

My configuration is to build for Intel Atom but a build for an ARM, MIPS, or PowerPC target would also take a similar amount of time, as even what could be considered “native” targets (targeting Atom, building on i7) doesn’t always turn out to be native: for example carrier-grade Xeon’s have instructions that my i7 doesn’t have, and if you were building carrier-grade embedded software you’d want to ensure they were used.

So, next time someone claims Yocto Project/OpenEmbedded takes “days” or even “hours” to do a build, you can denounce that as FUD and point them here!

11 thoughts on “Yocto Project Build Times

    1. Depends on what the incremental build involves. If you patched a leaf package (so no other packages will need to be rebuilt) and will be using the generated packages on the target, then just the compile time plus a few seconds for the packaging. If you want a new image the add on the image generation time (around 3 minutes for me).

  1. Obviously they don’t use yocto so they don’t know yocto and that’s why they mislead people so much. It would be just as easy for an yocto advocate to go around talking the same non-sense about baserock. But that just wouldn’t be nice, to say stuff like that about something you don’t have a clue about, would it?

  2. In my defence re accusations of FUD, this example does take hours on my laptop (X200s) and takes longer if I run inside a VM to avoid any host dependency issues.

    In response to the content of the post – that’s really interesting and good to know. I shall be sure to use these figures when discussing yocto in future.

    I’ll also be sure to test a baserock build on a similar system ;)

    1. Sure, the laptop that I SSH to my build machine from is a X200s. I used it briefly as a build machine before digging out the corporate laptop (i7 instead of Core2, SSD instead of HDD, less than a year old instead of three years old) while waiting for the proper machine to arrive. I wouldn’t recommend using an ultra-portable laptop as a build machine, yet alone an old one, when £600 will get you a reasonable machine. Of course if it’s all you have then fine – I know people doing Yocto builds inside a VM on a MacBook Air – it’s good that the barrier to entry is just “a computer”.

      For what it’s worth, you only need to run Yocto inside a VM if the host isn’t Linux. Otherwise it bootstraps without a problem on almost every distribution (if the distribution can’t build gcc, there are bigger problems), and there are sanity checks to ensure that the build isn’t being contaminated by the host.

    2. Hi there,
      are you going to post the build time stats for the same scenario on baserock inside the VM on the same laptop, on here anytime soon?

  3. We use VMs on a blade server in corporate environments. Works well, but with only 4 cores dedicated (4 of 32 per blade) a vanilla Yocto complete build including QT takes over 4 hours. I’ve done a lot of experimentation in this area. Since the majority of the time is spent compiling code and packaging files, the number of cores running makes a huge difference. the i7’s 8 cores does things approx 4 times faster than on old Xeon 2 core standalone tower server. I’m arguing for more cores to support our Yocto build system, but it’s a tough battle with IT because blade servers are incredibly expensive for the enterprise environment ($60K+ per 32-core blade)!!!

Comments are closed.