New file: doc/build.md

doc/build.md

@@ -0,0 +1,180 @@
+# Riju build system
+
+Riju's build system is complex and takes some time to explain. Bear
+with me. (If you just want to add or modify a language, you can read
+the [tutorial](tutorial.md) instead.)
+
+To get a quick overview, run `make help`.
+
+## Build artifacts
+
+We have two kinds of artifacts: Docker images (`I=` in the Makefile)
+and Debian packages (`L=` and `T=` in the Makefile).
+
+### Docker images
+
+* `admin`: The first thing you build, and then everything else
+  (including building other Docker images) is done from inside.
+* `ci`: Same as `admin` but for CI, so it has only the minimum number
+  of dependencies.
+* `packaging`: Provides an environment to build Debian packages.
+* `runtime`: Base runtime environment for Riju into which Debian
+  packages are installed and in which the server is expected to run.
+* `composite`: Based on `runtime`, but with all languages' Debian
+  packages installed.
+* `compile`: Compiles the Riju application code (i.e. everything
+  that's not per-language).
+* `app`: Based on `composite`, but with compiled code copied over from
+  `compile`. This container serves traffic in production.
+
+Docker images are built by running `make image I=<image> [NC=1]`, and
+run by `make shell I=<image> [E=1]`.
+
+Riju source code and build directories are typically mounted at `/src`
+inside the container, so there is generally no need to rebuild and/or
+restart containers when making changes. (Exception: `compile` and
+`app`.)
+
+* `NC=1`: pass `--no-cache` to `docker build`. Note that caching is
+  always disabled for `composite` due to the unique way in which the
+  build process is implemented for that image (to ensure good
+  performance).
+* `E=1`: map ports to the host. Generally desired for `runtime`, not
+  needed for `admin`.
+
+Note that `admin` uses `--network=host` and maps a number of
+directories such as `~/.ssh` and `~/.aws`, plus the Docker socket,
+inside the container, so you can treat an admin shell more or less the
+same as your external development environment.
+
+Note also that Docker builds do not pull new base images. For that,
+use `make pull-base`.
+
+### Debian packages
+
+There are three types of Debian packages:
+
+* `lang`, e.g. `riju-lang-python` (`T=lang L=python`): Installs the
+  actual language and any associated tools. May declare dependencies
+  on other Ubuntu packages, and may include files directly.
+* `config`, e.g. `riju-config-python` (`T=config L=python`): Installs
+  a JSON configuration file into `/opt/riju/langs`. The server looks
+  in this directory to find which languages are supported.
+* `shared`, e.g. `riju-shared-pandoc` (`T=shared L=pandoc`): Shared
+  dependency. This is for when multiple different languages need the
+  same tool, and there's no Ubuntu package for it.
+
+There are three basic actions for any particular Debian package:
+
+* From any container, run `make script L=<lang> T=<type>` to generate
+  the build script for a package. This is placed in
+  `build/<type>/<lang>/build.bash`.
+* From a packaging container, run `make pkg L=<lang> T=<type>` to
+  (re)build a Debian package by executing its build script in a fresh
+  directory. This is placed in
+  `build/<type>/<lang>/riju-<type>-<lang>.deb`.
+* From a runtime container, run `make install L=<lang> T=<type>` to
+  install it.
+
+Each language consists of a `lang` and `config` package, so you need
+to follow the above steps for both. The `make scripts L=<lang>`, `make
+pkgs L=<lang>`, and `make installs L=<lang>` commands automate this.
+
+For further convenience, if you already have a runtime container up,
+from the admin shell you can use `make repkg L=<lang> T=<type>` and/or
+`make repkgs L=<lang>` to automate the three steps above (run `make
+script`, run `make pkg` inside a fresh packaging container, and then
+run `make install` inside the existing runtime container).
+
+Some `lang` packages declare `shared` dependencies, in which case they
+won't install until the `shared` package is built and installed
+already. This can't be done with `make scripts`, `make pkgs`, `make
+installs`, or `make repkgs`: use `make script T=shared L=<lang>`,
+`make pkg T=shared L=<lang>`, `make install T=shared L=<lang>`, or
+`make repkg T=shared L=<lang>`, respectively. (Check the
+`install.riju` key in a language's YAML configuration to see if it
+declares any such dependencies.)
+
+#### Package build details
+
+The build script is executed with a working directory of
+`build/<type>/<lang>/src`, and it installs package files into
+`build/<type>/<lang>/pkg`.
+
+If `make pkg` is too high-level, there are more specific commands:
+
+* `make pkg-clean`: Wipe and recreate the `src` and `pkg` directories.
+* `make pkg-build`: Just run the package build script (you also need
+  to run `make script` if the language configuration has changed).
+* `make pkg-deb`: Build the `pkg` directory into the actual Debian
+  package.
+
+All Makefile targets with `pkg` in the name take an optional `Z`
+parameter for the `.deb` compression level, defaulting to `none`. This
+can be increased to `gzip` or even further to `xz`. Increasing the
+compression massively increases build time, but massively decreases
+the resulting package size.
+
+## Artifact caching
+
+All artifacts can be cached on remote registries to avoid being
+rebuilt in CI unnecessarily.
+
+* Docker images are cached on Docker Hub. Push with `make push
+  I=<image>` and pull with `make pull I=<image>`.
+* Debian packages are cached on S3. Push with `make upload T=<type>
+  L=<lang>` and pull with `make download T=<type> L=<lang>`.
+
+CI will take care of managing the remote registries automatically. It
+is generally recommended to let CI handle this, and not push anything
+yourself.
+
+## Application build
+
+We have two compiled parts of Riju:
+
+* Frontend assets (compiled with Webpack)
+* Setuid binary used for privilege deescalation (compiled with LLVM)
+
+For development:
+
+* `make frontend-dev` (compile frontend assets, auto recompile on
+  change)
+* `make system-dev` (compile setuid binary, auto recompile on change)
+* `make server-dev` (run server, auto restart on change)
+* `make dev` (all three of the above)
+
+For production:
+
+* `make frontend` (compile frontend assets)
+* `make system` (compile setuid binary)
+* `make build` (both of the above)
+* `make server` (run server)
+
+## Incremental builds and hashing
+
+CI is set up so that artifacts are only rebuilt when changes have
+occurred. This is done through an extensive hashing algorithm which
+produces a consistent hash for each artifact based on its inputs. We
+can then check whether the hash has changed, meaning the artifact
+should be rebuilt.
+
+This is implemented mostly behind the scenes, but you can run `make
+plan` to execute the hashing algorithm and dump a plan of the minimal
+set of actions that would be run if this were CI:
+
+* If local artifact is missing, but remote artifact is up to date:
+  download remote to local.
+* If remote artifact is missing or outdated, but local artifact is up
+  to date: upload local to remote.
+* If neither local nor remote artifact is up to date: rebuild local
+  and upload to remote.
+
+You can run `make sync` to execute this plan, excepting the upload
+part (that should, for safety, generally be done only in CI). So, in
+principle, `make sync` should bring all your local artifacts up to
+date with the latest source (rebuilding some if needed).
+
+To run a full deployment, use `make publish`. This should definitely
+be done only from CI, and with the `Z=xz` flag to enable Debian
+package compression.