diff --git a/doc/build.md b/doc/build.md new file mode 100644 index 0000000..d9e140e --- /dev/null +++ b/doc/build.md @@ -0,0 +1,180 @@ +# Riju build system + +Riju's build system is complex and takes some time to explain. Bear +with me. (If you just want to add or modify a language, you can read +the [tutorial](tutorial.md) instead.) + +To get a quick overview, run `make help`. + +## Build artifacts + +We have two kinds of artifacts: Docker images (`I=` in the Makefile) +and Debian packages (`L=` and `T=` in the Makefile). + +### Docker images + +* `admin`: The first thing you build, and then everything else + (including building other Docker images) is done from inside. +* `ci`: Same as `admin` but for CI, so it has only the minimum number + of dependencies. +* `packaging`: Provides an environment to build Debian packages. +* `runtime`: Base runtime environment for Riju into which Debian + packages are installed and in which the server is expected to run. +* `composite`: Based on `runtime`, but with all languages' Debian + packages installed. +* `compile`: Compiles the Riju application code (i.e. everything + that's not per-language). +* `app`: Based on `composite`, but with compiled code copied over from + `compile`. This container serves traffic in production. + +Docker images are built by running `make image I= [NC=1]`, and +run by `make shell I= [E=1]`. + +Riju source code and build directories are typically mounted at `/src` +inside the container, so there is generally no need to rebuild and/or +restart containers when making changes. (Exception: `compile` and +`app`.) + +* `NC=1`: pass `--no-cache` to `docker build`. Note that caching is + always disabled for `composite` due to the unique way in which the + build process is implemented for that image (to ensure good + performance). +* `E=1`: map ports to the host. Generally desired for `runtime`, not + needed for `admin`. + +Note that `admin` uses `--network=host` and maps a number of +directories such as `~/.ssh` and `~/.aws`, plus the Docker socket, +inside the container, so you can treat an admin shell more or less the +same as your external development environment. + +Note also that Docker builds do not pull new base images. For that, +use `make pull-base`. + +### Debian packages + +There are three types of Debian packages: + +* `lang`, e.g. `riju-lang-python` (`T=lang L=python`): Installs the + actual language and any associated tools. May declare dependencies + on other Ubuntu packages, and may include files directly. +* `config`, e.g. `riju-config-python` (`T=config L=python`): Installs + a JSON configuration file into `/opt/riju/langs`. The server looks + in this directory to find which languages are supported. +* `shared`, e.g. `riju-shared-pandoc` (`T=shared L=pandoc`): Shared + dependency. This is for when multiple different languages need the + same tool, and there's no Ubuntu package for it. + +There are three basic actions for any particular Debian package: + +* From any container, run `make script L= T=` to generate + the build script for a package. This is placed in + `build///build.bash`. +* From a packaging container, run `make pkg L= T=` to + (re)build a Debian package by executing its build script in a fresh + directory. This is placed in + `build///riju--.deb`. +* From a runtime container, run `make install L= T=` to + install it. + +Each language consists of a `lang` and `config` package, so you need +to follow the above steps for both. The `make scripts L=`, `make +pkgs L=`, and `make installs L=` commands automate this. + +For further convenience, if you already have a runtime container up, +from the admin shell you can use `make repkg L= T=` and/or +`make repkgs L=` to automate the three steps above (run `make +script`, run `make pkg` inside a fresh packaging container, and then +run `make install` inside the existing runtime container). + +Some `lang` packages declare `shared` dependencies, in which case they +won't install until the `shared` package is built and installed +already. This can't be done with `make scripts`, `make pkgs`, `make +installs`, or `make repkgs`: use `make script T=shared L=`, +`make pkg T=shared L=`, `make install T=shared L=`, or +`make repkg T=shared L=`, respectively. (Check the +`install.riju` key in a language's YAML configuration to see if it +declares any such dependencies.) + +#### Package build details + +The build script is executed with a working directory of +`build///src`, and it installs package files into +`build///pkg`. + +If `make pkg` is too high-level, there are more specific commands: + +* `make pkg-clean`: Wipe and recreate the `src` and `pkg` directories. +* `make pkg-build`: Just run the package build script (you also need + to run `make script` if the language configuration has changed). +* `make pkg-deb`: Build the `pkg` directory into the actual Debian + package. + +All Makefile targets with `pkg` in the name take an optional `Z` +parameter for the `.deb` compression level, defaulting to `none`. This +can be increased to `gzip` or even further to `xz`. Increasing the +compression massively increases build time, but massively decreases +the resulting package size. + +## Artifact caching + +All artifacts can be cached on remote registries to avoid being +rebuilt in CI unnecessarily. + +* Docker images are cached on Docker Hub. Push with `make push + I=` and pull with `make pull I=`. +* Debian packages are cached on S3. Push with `make upload T= + L=` and pull with `make download T= L=`. + +CI will take care of managing the remote registries automatically. It +is generally recommended to let CI handle this, and not push anything +yourself. + +## Application build + +We have two compiled parts of Riju: + +* Frontend assets (compiled with Webpack) +* Setuid binary used for privilege deescalation (compiled with LLVM) + +For development: + +* `make frontend-dev` (compile frontend assets, auto recompile on + change) +* `make system-dev` (compile setuid binary, auto recompile on change) +* `make server-dev` (run server, auto restart on change) +* `make dev` (all three of the above) + +For production: + +* `make frontend` (compile frontend assets) +* `make system` (compile setuid binary) +* `make build` (both of the above) +* `make server` (run server) + +## Incremental builds and hashing + +CI is set up so that artifacts are only rebuilt when changes have +occurred. This is done through an extensive hashing algorithm which +produces a consistent hash for each artifact based on its inputs. We +can then check whether the hash has changed, meaning the artifact +should be rebuilt. + +This is implemented mostly behind the scenes, but you can run `make +plan` to execute the hashing algorithm and dump a plan of the minimal +set of actions that would be run if this were CI: + +* If local artifact is missing, but remote artifact is up to date: + download remote to local. +* If remote artifact is missing or outdated, but local artifact is up + to date: upload local to remote. +* If neither local nor remote artifact is up to date: rebuild local + and upload to remote. + +You can run `make sync` to execute this plan, excepting the upload +part (that should, for safety, generally be done only in CI). So, in +principle, `make sync` should bring all your local artifacts up to +date with the latest source (rebuilding some if needed). + +To run a full deployment, use `make publish`. This should definitely +be done only from CI, and with the `Z=xz` flag to enable Debian +package compression.