2. misc
  3. How to write a good build system ?

How to write a good build system ?

This is not a HOWTO, but more a RFC. I'm in the need for a portable, fast build-system. For lighttpd, for MySQL Proxy, ...

I need what autotools + gmake do, just in a portable (yes, including windows) and fast way. Up to now I tried SCons and cmake as alternatives and both fell short in important categories (like make dist).

If you are reading this blog since a while you know what comes next: Let's write a full replacement in LUA.

Requirements

I like what the autotools (automake, autoconf and libtool) do. From the users point of view it is:

$ ./configure ...
$ make 
$ make install

With automake you also get:

$ make uninstall
$ make dist
$ make distcheck

Especially the distcheck target is very neat if you are a packager and want to make sure all files needed for the build are also in the final package. It builds the src-tar and builds a test-builds against it. It has rescued me from severval broken releases due to missed header files already.

Alternatives

The problem with autotools is that it is unix shell based.

  • slow due to many fork()s
  • doesn't run on windows natively (run a configure script in cygwin and you will understand why this is a problem)
  • running the configure script takes longer than the build afterwards.
  • 1.7MByte of shell-scripts in the source-package compared to 370kbyte real source (MySQL Proxy)

SCons

SCons was my first attempt a few years ago. It looked very promising. Written all in Python you can add what ever you needed.

Well, at that time you had to extend it to get the basic stuff going. And my python foo isn't good enough to add in proper install-targets and packaging. It also got really confused when I wanted to use the same source-file in 2 different build targets: the binary and the unit-test. Simple things should be simple.

CMake

cmake was the next candidate. It is a bit different as it is a frontend to makefile generator as the autotools are. The build is still done by make. Nice fast builds, simple scripting language for the pre-checks.

They also went the way of declaring the build-structure (binaries, libs, ...) in a DSL. That makes it a lot easier for maintainers to write and read their build-files. No deep long python-stacktraces as in python. Additional macros can be easily added to the builds and be shipped around.

CPack is its packaging support, but seems to have a different focus then I have. Having to provide a blacklist from a src-dir SET(CPACK_SOURCE_STRIP_FILES "") to say what should end up in there the src-tar is the wrong way around. At least I do test-builds from a folder than also contains a bunch of files that the world should never see. If you src-tree is always clean (like svn export) this might work.

And the syntax of their scripting language ... well. No.

Build your own

As a matter of fact here I stand without build-system that meets my needs. My current target is building the MySQL Proxy on Linux, MacOS X and win32.

Throwing lua into the game as a nice side-effect: the scripting language is clean and simple. You can also think about adding the lua-src itself to the build-system to do a full bootstrap in case the user doesn't have it install. Others are doing that already.

Where are we now ?

The overall project is split into 4 stages

  • configure (configure)
  • build (make)
  • install (make install and uninstall)
  • packaging (make dist)
  • testing (make distcheck)

Configure

On the configure side we have:

env = BuildEnv:new()

env:checkPkgConfig("LUA", "lua5.1")

env:checkLib("event", "event_base_set")
env:checkHeader("event.h", { "#include <sys/types.h>" })

env:checkHeaders({
        "arpa/inet.h",
        "netinet/in.h",
})

We have pkgconfig support and simple header and library checks. Loading and saving the results is next and mostly done.

Build

The build-side is pretty simple. I need a dependency graph anyway. What has to be built and what order. Let the user create himself for now, it doesn't hurt:

libmysqlproxy = instsharedlib("libmysql-proxy.so", {
                        obj("network-mysqld.c"),
                        obj("network-mysqld-proto.c"),
                        obj("network-conn-pool.c"),
                        obj("network-socket.c"),
                        obj(nil, { lex("sql-tokenizer.l") } ),
                }, { CPPFLAGS = env.MYSQL_CPPFLAGS .. " " .. env.GMODULE_CPPFLAGS .. " " .. env.LUA_CPPFLAGS}

tree = {
        all = {
                instbinary("mysql-proxy", {
                        obj("chassis.c"),
                        libchassis,
                        libmysqlproxy
                }, { LDFLAGS = env.MYSQL_LDFLAGS .. " " ..env.GMODULE_LDFLAGS .. " " .. env.LUA_LDFLAGS}),
        }
}

We have a few builders already:

  • binary() links object files and libs into a binary
  • obj() compiles C code
  • sharedlib() builds a .so
  • lex() calls flex on a .l file

All of them have the same parameters:

builder(targetname, table of sources, local buildenv)

Some also have a short version in case we can figure out the target name ourself like with obj().

The environment is propagated downwards in the tree. We have a global env and that gets overwritten if needed in the sub-nodes of our dependency tree.

Installation

The example above also shows my current approach on installation. Using instbinary() instead of binary() adds to target to the pool of files to be installed. Binaries go to BINDIR, sharedlibs to LIBDIR like in automake.

Clean

As we know what we generate we also know how to cleanup. It just deletes all the files in the target-nodes.

Packaging

This is the open task for now.

My goal is to have a clean tar.gz builder for source and binary tar.gz. I wanted the same as make dist in automake. Plus, I want a MSI support as in CPack.

When ?

This project is going along at the side and I'll release it when I think it does what I need. There is a target date nor a clear roadmap.

To verify my ideas I'll also try to push the MySQL Server through this build-system and see if it is flexible enough. It will bring yacc and C++ support and funky copy-around of source-files. We'll see.

How to write a build system

Back the initial question: how to write a good build-system ? What do you need ?

ccache, distcc support are on the list, perhaps some cross-platform ?

Comments

Enable javascript to load comments.