Back to list

efficient distributed package builds in OpenBSD

Marc Espie
Author Description:
OpenBSD senor developer.

Maintainer of make, m4, our ports (packages) tree, author of our pkg_* suite and various related perl/shell scripts.
Over the last six months, I've written a new tool (dpb3) to greatly simplify the way packages are built in OpenBSD.
Going beyond sequential builds require efficient dependency computation, since packages must obey some kind of topological sorting.

dpb3 improves over previous tools thanks to several practical ideas:
* the dependency computation is viewed as just another job;
* actual job building starts as soon as we find out packages that don't depend on anything else.
* dpb3 has a model of distant hosts such that it can detect base system discrepancies
* dpb3 computes a lot of logs and shows what's going on in real time. It can also resume builds after manual fixes of build failures.

A few builds have shown that simple heuristics are often enough to avoid build starvation. By feeding back previous build logs to dpb3, one can optimize a new build: in general, similar packages builds will tend to require the same quantity of cpu power, and thus, by scheduling long-running jobs preferentially, dpb3 is able to minimize total build time over a number of cpu cores.

Effectively, this does solve an NP hard problem, due to some specificities of our ports tree:
- there are lots of ports with no dependencies
- there are lots of small ports that can be used to pad the end of the build and ensure every core finishes at the same time.
Target Audience:
Target OS:
Paper Download:
View PDF
Back to list

EuroBSDCon Organizer

see all Sponsors

©2015 · All rights reserved. Imprint