# Riju build system Riju's build system is complex and takes some time to explain. Bear with me. (If you just want to add or modify a language, you can read the [tutorial](tutorial.md) instead.) To get a quick overview, run `make help`. ## Build artifacts We have two kinds of artifacts: Docker images (`I=` in the Makefile) and Debian packages (`L=` and `T=` in the Makefile). ### Docker images * `admin`: The first thing you build, and then everything else (including building other Docker images) is done from inside. * `ci`: Same as `admin` but for CI, so it has only the minimum number of dependencies. * `packaging`: Provides an environment to build Debian packages. * `runtime`: Base runtime environment for Riju into which Debian packages are installed and in which the server is expected to run. * `composite`: Based on `runtime`, but with all languages' Debian packages installed. * `compile`: Compiles the Riju application code (i.e. everything that's not per-language). * `app`: Based on `composite`, but with compiled code copied over from `compile`. This container serves traffic in production. Docker images are built by running `make image I= [NC=1]`, and run by `make shell I= [E=1]`. Riju source code and build directories are typically mounted at `/src` inside the container, so there is generally no need to rebuild and/or restart containers when making changes. (Exception: `compile` and `app`.) * `NC=1`: pass `--no-cache` to `docker build`. Note that caching is always disabled for `composite` due to the unique way in which the build process is implemented for that image (to ensure good performance). * `E=1`: map ports to the host. Generally desired for `runtime`, not needed for `admin`. Note that `admin` uses `--network=host` and maps a number of directories such as `~/.ssh` and `~/.aws`, plus the Docker socket, inside the container, so you can treat an admin shell more or less the same as your external development environment. Note also that Docker builds do not pull new base images. For that, use `make pull-base`. ### Debian packages There are three types of Debian packages: * `lang`, e.g. `riju-lang-python` (`T=lang L=python`): Installs the actual language and any associated tools. May declare dependencies on other Ubuntu packages, and may include files directly. * `config`, e.g. `riju-config-python` (`T=config L=python`): Installs a JSON configuration file into `/opt/riju/langs`. The server looks in this directory to find which languages are supported. * `shared`, e.g. `riju-shared-pandoc` (`T=shared L=pandoc`): Shared dependency. This is for when multiple different languages need the same tool, and there's no Ubuntu package for it. There are three basic actions for any particular Debian package: * From any container, run `make script L= T=` to generate the build script for a package. This is placed in `build///build.bash`. * From a packaging container, run `make pkg L= T=` to (re)build a Debian package by executing its build script in a fresh directory. This is placed in `build///riju--.deb`. * From a runtime container, run `make install L= T=` to install it. Each language consists of a `lang` and `config` package, so you need to follow the above steps for both. The `make scripts L=`, `make pkgs L=`, and `make installs L=` commands automate this. For further convenience, if you already have a runtime container up, from the admin shell you can use `make repkg L= T=` and/or `make repkgs L=` to automate the three steps above (run `make script`, run `make pkg` inside a fresh packaging container, and then run `make install` inside the existing runtime container). Some `lang` packages declare `shared` dependencies, in which case they won't install until the `shared` package is built and installed already. This can't be done with `make scripts`, `make pkgs`, `make installs`, or `make repkgs`: use `make script T=shared L=`, `make pkg T=shared L=`, `make install T=shared L=`, or `make repkg T=shared L=`, respectively. (Check the `install.riju` key in a language's YAML configuration to see if it declares any such dependencies.) #### Package build details The build script is executed with a working directory of `build///src`, and it installs package files into `build///pkg`. If `make pkg` is too high-level, there are more specific commands: * `make pkg-clean`: Wipe and recreate the `src` and `pkg` directories. * `make pkg-build`: Just run the package build script (you also need to run `make script` if the language configuration has changed). * `make pkg-deb`: Build the `pkg` directory into the actual Debian package. All Makefile targets with `pkg` in the name take an optional `Z` parameter for the `.deb` compression level, defaulting to `none`. This can be increased to `gzip` or even further to `xz`. Increasing the compression massively increases build time, but massively decreases the resulting package size. ## Artifact caching All artifacts can be cached on remote registries to avoid being rebuilt in CI unnecessarily. * Docker images are cached on Docker Hub. Push with `make push I=` and pull with `make pull I=`. * Debian packages are cached on S3. Push with `make upload T= L=` and pull with `make download T= L=`. CI will take care of managing the remote registries automatically. It is generally recommended to let CI handle this, and not push anything yourself. ## Application build We have two compiled parts of Riju: * Frontend assets (compiled with Webpack) * Setuid binary used for privilege deescalation (compiled with LLVM) For development: * `make frontend-dev` (compile frontend assets, auto recompile on change) * `make system-dev` (compile setuid binary, auto recompile on change) * `make server-dev` (run server, auto restart on change) * `make dev` (all three of the above) For production: * `make frontend` (compile frontend assets) * `make system` (compile setuid binary) * `make build` (both of the above) * `make server` (run server) ## Incremental builds and hashing CI is set up so that artifacts are only rebuilt when changes have occurred. This is done through an extensive hashing algorithm which produces a consistent hash for each artifact based on its inputs. We can then check whether the hash has changed, meaning the artifact should be rebuilt. This is implemented mostly behind the scenes, but you can run `make plan` to execute the hashing algorithm and dump a plan of the minimal set of actions that would be run if this were CI: * If local artifact is missing, but remote artifact is up to date: download remote to local. * If remote artifact is missing or outdated, but local artifact is up to date: upload local to remote. * If neither local nor remote artifact is up to date: rebuild local and upload to remote. You can run `make sync` to execute this plan, excepting the upload part (that should, for safety, generally be done only in CI). So, in principle, `make sync` should bring all your local artifacts up to date with the latest source (rebuilding some if needed). To run a full deployment, use `make publish`. This should definitely be done only from CI, and with the `Z=xz` flag to enable Debian package compression.