A Simple Build System for Full-stack Web Applications

Setting up a build system for a full-stack web application presents some unique challenges. Builds for these projects not only include compiling and testing executables for the back-end, but the build system also needs to be flexible enough to integrate with various tools for processing web assets, packaging the build, etc. Since these systems can get relatively large, build times are also important.

A Phased, Parallel Build System

A relatively simple build system can be setup as follows:

The build is organized into phases. There is a specific order to the phases and commands within a phase can only depend on previous phases.
A top-level build script runs the full build and orchestrates parallel build tasks.
Each subdirectory includes a makefile. The makefile is invoked once per phase by the top-level build script.
The makefile uses project-specific makefile commands to compile code, minify JavaScript files, run tests, etc. These makefile commands provide the flexibility to integrate with almost any build or testing tool.

More information and details are presented below, but the basic system described above is all you will likely need for full-stack web apps. Resultra is a new type of web application for project tracking, and uses this system. Using Resultra’s build system as a case study, this type of build system has proven to be flexible, robust and scalable. Although the system itself is not very sophisticated, it can be used with projects that leverage a full complement of front-end and back-end build and testing tools.

Build Phases

A primary concept for this build system is that builds are organized into phases. Build phases are performed in a specific order. For example, there are phases to compile the code, export the resources, then perform tests.

Another important concept here is that a build command within a given phase can only depend on previous phases; for example, a test can only be run after the code has been compiled.

Top-level Build Script

In general, Python is the glue which ties the entire build system together. All it takes for Python to implement the core functionality of a phased, parallel build system is a simple 100 line build script! To see for yourself, here’s what the code listing currently looks like for Resultra’s top-level build script:

#!/usr/bin/env python
# This script implements a phased build based upon the makefiles in the development tree.
# Within each build phase, make is run on each directory in no particular order. So,
# the build process from each directory is expected to not depend on other directories
# within a a given phase.

# By default a debug build is performed. However to perform a release build, pass the — release
# option on the command line.
import os
import sys
import argparse
import time
from multiprocessing import Pool

parser = argparse.ArgumentParser(description=’Main build script.’)
parser.add_argument(‘ — release’,default=False,action=’store_true’,
help=’perform a release build’)
parser.add_argument(‘ — realcleanonly’,default=False,action=’store_true’,
help=’only run the clean and realclean targets across the build’)
parser.add_argument(‘ — windows’,default=False,action=’store_true’,
help=’cross-compile the Windows Electron client.’)
parser.add_argument(‘ — procs’,default=4,type=int,
help=’number of build tasks to run in parallel build on (default = 4)’)
args = parser.parse_args()

failedDirs = []

debugBuild = 1
if(args.release):
debugBuild = 0

class buildDirResult:
def __repr__(self):
return “(dir = %s, err = %d) “ % (self.dirName,self.errCode)

def __init__(self, dirName,errCode):
self.dirName = dirName
self.errCode = errCode

class buildDirSpec:
def __init__(self, dirName,targetName,debugBuild):
self.dirName = dirName
self.targetName = targetName
self.debugBuild = debugBuild

def buildOneDir(buildSpec):
print “Building: dir=”, buildSpec.dirName, “ phase=”, buildSpec.targetName, “ debug=”, buildSpec.debugBuild
bldCmd = “make -C %s — jobs=2 DEBUG=%s %s” % (buildSpec.dirName, buildSpec.debugBuild, buildSpec.targetName)
print “Build cmd: %s “ % (bldCmd)
retCode = os.system(bldCmd)
if retCode != 0:
print “FAIL: failure building dir = %s, target= %s, err = %d” % (buildSpec.dirName,buildSpec.targetName,retCode)
return buildDirResult(buildSpec.dirName,retCode)

def runMakePhase(makeTargetName):

print “Build: Starting phase = “, makeTargetName
makeDirs = []

for root, dirs, files in os.walk(“..”):
for file in files:
if (file == ‘Makefile’) and (not “node_modules” in root):
makeDirs.append(buildDirSpec(root,makeTargetName,debugBuild))
buildPool = Pool(processes=args.procs)
results = buildPool.map(buildOneDir,makeDirs)
buildPool.close()
buildPool.join()
print “Build: Done with phase = “, makeTargetName
for res in results:
if res.errCode != 0:
failedDirs.append(makeTargetName + “:” + res.dirName)

startTime = time.time()

if args.realcleanonly:
runMakePhase(“clean”)
runMakePhase(“realclean”)
else:
runMakePhase(“install”)
runMakePhase(“prebuild”)
runMakePhase(“build”)
runMakePhase(“export”)
runMakePhase(“package”)
runMakePhase(“test”)
runMakePhase(“systest”)
if args.windows:
runMakePhase(“windows”)
runMakePhase(“winpkg”)

endTime = time.time()

print “\n\n — — — — — — — — — — — — — — — — — — — — — — — — — — — — “
print “Build complete: parallel build tasks = %d, elapse time = %d secs “ % (args.procs, endTime-startTime)
print “\nBuild Results:\n”

if len(failedDirs) > 0:
print “Build failed on following directories:\n”
print “\n”.join(failedDirs)
sys.exit(255)
else:
print “Build succeeded”
sys.exit(0)

You’re welcome to use the code snippet above as a starting point for your own projects. This type of phased, parallel build system is quite generic; it will not only work for full-stack web applications, but almost any type of software development project.

Makefile per Subdirectory

The second component to the build system is a makefile per subdirectory of the project. Each subdirectory’s makefile only implements targets for the build phases which are applicable to that directory, with the other phases being no-ops. Since dependencies are between phases, there are no recursive makefiles.

Makefiles effectively integrate with the shell to invoke individual commands for compiling, testing, minifying JavaScript files, etc. In a build environment for a full-stack application, a variety of different commands are needed, so makefiles are a very good way to automate this.

Besides the parallel build support provided by the top-level build script, Makefiles also provide a second level of parallel execution using the make tool’s ‘ — jobs’ option. This option allows individual commands (jobs) within the makefile to be executed in parallel.

Each individual makefile is oriented to the root project directory using a DEPTH makefile variable. This allows individual makefiles to reuse common makefile rules or utility scripts. For example, if a directory only contains front-end HTML, Javascript and CSS files, all that is needed is a simple 2 line makefile:

DEPTH = ../../..

include $(DEPTH)/webui/build/autoExportAssets.mk

Once a full build has been completed, development and testing can can occur by running make on the subdirectory where changes have occured.

Project-specific Makefile Commands

Besides the top-level build script and a makefile per subdirectory, the third major component of this system is project-specific makefile commands. These commands are needed to integrate the system with various build and testing tools.

In Resultra’s build environment, there are a few other Python scripts to perform utility functions, such as exporting CSS and Javascript files to the build destination directory. Resultra also integrates with gulp to process front-end files, so there are utility scripts and makefile commands specific to this integration. Similarly, Resultra’s back-end is implemented in Go (Golang), so there are makefile commands to invoke compilation and testing commands in Go’s toolchain.

Parallel Builds

Parallel builds are done per phase. Since individual build tasks can only depend upon previous phases, each task within a phase can be run in parallel. So, for example, tests can be run in parallel, or CSS and Javascript files can be processed in parallel.

Using Resultra’s build as an example, below is a sampling of build results with different numbers of parallel build tasks enabled in the top-level build script. In addition to the parallel build tasks launched by the top-level build script, the individual makefiles are invoked with the — jobs=2 option for the make command to internally perform parallel execution of jobs/commands inside the makefile:

These builds were performed on a late 2012 Mac Mini with a quad-core i7 processor (8 virtual cores) and a solid-state drive.

Increasing the number of parallel tasks does improve build times, but the improvements are not earth shattering. In particular, going from 1 to 8 parallel build tasks, the build times improved by 33%.

As one more data point, if make’s jobs option is set to 1 and parallel execution is disabled in the top-level build script, the build time increases to 496 seconds. This data point represents the slowest possible build with no parallel execution at all. Versus this slowest possible data point, a build with 8 parallel build tasks and make’s jobs option set to 2, the build is 43% faster.

Conclusions

It’s possible to create a relatively simple build system for full-stack web apps. This system consists of only a single top-level build script, individual makefiles per subdirectory, and makefile commands to integrate with project-specific build and testing tools. Using Resultra as a case study, this simple type of build system has now been proven to be flexible, robust and scalable for relatively large, full-stack web applications.

The benefits of this system not only include simplicity, but also a good separation of concerns. The top-level build script and individual makefiles provide a lightweight scaffolding to run the build. With this scaffolding in place, custom utility scripts and makefile commands are used to integrate the build system with a full complement of front-end and back-end tools.

Nothing is necessarily new or innovative with this type of build system. This system is similar to build systems I’ve encountered on other projects. Nonetheless, for the benefit of other projects, this case study hopefully illustrates the key concepts behind a phased, parallel build system.