From a2d889860d9edcdfd70a23b84f86ac9879b99fec Mon Sep 17 00:00:00 2001 From: Radon Rosborough Date: Sat, 6 Feb 2021 21:29:16 -0800 Subject: [PATCH] Start writing tutorial --- doc/tutorial.md | 132 +++++++++++++++++ doc/tutorial/install-debugging.md | 0 doc/tutorial/install.md | 232 ++++++++++++++++++++++++++++++ doc/tutorial/run.md | 1 + 4 files changed, 365 insertions(+) create mode 100644 doc/tutorial/install-debugging.md create mode 100644 doc/tutorial/install.md create mode 100644 doc/tutorial/run.md diff --git a/doc/tutorial.md b/doc/tutorial.md index 8683421..3f6a4ef 100644 --- a/doc/tutorial.md +++ b/doc/tutorial.md @@ -1 +1,133 @@ # Adding your own language to Riju + +Hello and welcome! This tutorial guides you through the basics of +adding a new language to Riju, or modifying an existing language. The +other documentation in this repo has reference material that may be +helpful for advanced use cases, but this page should get you started. + +If you run into any trouble following the guide, please do not +hesitate to open an issue! + +## Project setup + +Fork this repository to your account on GitHub, and clone it locally: + +``` +$ git clone https://github.com/yourname/riju.git +$ cd riju +``` + +Install [Docker](https://www.docker.com/). Then you can build and +start the admin shell: + +``` +$ make image shell I=admin +``` + +All future operations can be done inside the admin shell, where all +dependencies are installed automatically. + +## Start tmux + +Start a tmux session: + +``` +$ make tmux +``` + +If you don't know how to use tmux, see [a +cheatsheet](https://danielmiessler.com/study/tmux/). The useful +keybindings are: + +* `control-b c`: open new tab +* `control-b p/n`: previous/next tab +* `control-b "`: split tab into top and bottom panes +* `control-b %`: split tab into left and right panes +* `control-b `: move between panes +* `control-b control-b `: if you have two tmuxes nested, + use `control-b` twice to do a command on the inner one instead of + the outer one + +## Configure local project + +Using your regular text editor (the Riju repository is synchronized +inside and outside of the container, so you can use whatever editor +you would like, it doesn't have to be something in the terminal), +create a file `.env` in the Riju repository with the following +contents: + +``` +DOCKER_REPO=raxod502/riju +S3_BUCKET=riju +``` + +This tells Riju to pull assets from the official registries that I +maintain, so that you don't have to build them yourself. + +## Set up Docker images + +Download the two Docker images needed for testing a new language: + +``` +$ make pull I=packaging +$ make pull I=runtime +``` + +Create a new tab in tmux (`control-b c`) and start the runtime image +with ports exposed: + +``` +$ make shell I=runtime E=1 +``` + +Inside that shell, start another instance of tmux: + +``` +$ make tmux +``` + +Now within that tmux, start Riju in development mode: + +``` +$ make dev +``` + +You should now be able to navigate to and see +that Riju is running, although it does not have any languages +installed. + +Finally, switch back to the admin shell (`control-b p`). We are ready +to start creating your new language. + +## Create a language configuration + +Create a file `langs/mylanguage.yaml` with the following contents: + +```yaml +id: "mylanguage" +name: "My Language" + +main: "TODO" +template: | + # Fill this in later + +run: | + echo "Hello, world!" +``` + +Now from the admin shell, run `make repkgs L=mylanguage`. Once that +completes, you should see your language at . +Furthermore, you can switch to the runtime image (`control-b n`) and +run `make sandbox L=mylanguage` to test your language at the command +line (e.g. type `run` to print `Hello, world!`). Each time you modify +the language configuration, run `make repkgs` to reinstall the +language. + +Follow these steps to augment your language configuration: + +* [Install your language](tutorial/install.md) +* [Provide run commands](tutorial/run.md) +* [Configure tests](tutorial/tests.md) +* [Provide metadata](tutorial/metadata.md) +* [Add code formatter](tutorial/formatter.md) +* [Add language server](tutorial/lsp.md) diff --git a/doc/tutorial/install-debugging.md b/doc/tutorial/install-debugging.md new file mode 100644 index 0000000..e69de29 diff --git a/doc/tutorial/install.md b/doc/tutorial/install.md new file mode 100644 index 0000000..1184ff1 --- /dev/null +++ b/doc/tutorial/install.md @@ -0,0 +1,232 @@ +# Tutorial: install your language + +Presumably, your language isn't installed by default in Ubuntu. If +not, you'll need to add an `install:` block to your language +configuration describing how to install it. + +The easiest case is system packages, which are supported out of the +box, for example: + +```yaml +install: + apt: + - ruby + npm: + - pug-cli + pip: + - bython + gem: + - solargraph + cpan: + - Acme::Chef + opam: + - ocamlformat +``` + +For more advanced configuration you need to understand how Debian +packaging works. To build a package, we place files into a `${pkg}` +directory mirroring how they should be installed on the system, so for +example a binary intended for `/usr/local/bin/prettier` would be +placed at `${pkg}/usr/local/bin/prettier`. Then the `${pkg}` directory +can be turned into a `.deb`, which can be installed on any system. + +*If you have trouble, see the tutorial on [debugging package +installation](install-debugging.md).* + +## Download a binary or script + +We prefer to put all binaries in `/usr/local/bin` rather than +`/usr/bin` where they might conflict with default Ubuntu packages. + +```yaml +install: + manual: | + install -d "${pkg}/usr/local/bin" + + wget http://www.blue.sky.or.jp/grass/grass.rb + chmod +x grass.rb + cp -T grass.rb "${pkg}/usr/local/bin/grass" +``` + +## Get the latest version from GitHub + +There is a predefined `latest_release` function that can be used to +get the name of the latest tag in a GitHub repository. + +```yaml +install: + manual: | + install -d "${pkg}/usr/local/bin" + + ver="$(latest_release snoe/clojure-lsp)" + wget "https://github.com/snoe/clojure-lsp/releases/download/${ver}/clojure-lsp" + chmod +x clojure-lsp + cp clojure-lsp "${pkg}/usr/local/bin/" +``` + +## Get the latest version from some other website + +In the case that there's no "download the latest version" URL, we +typically use disgusting `grep` pipelines to extract it in a +semi-reliable way from a stable-seeming webpage. + +```yaml +install: + manual: | + install -d "${pkg}/usr/local/bin" + + path="$(curl -fsSL http://static.red-lang.org/download.html | grep -Eo '/dl/linux/[^"]+' | head -n1)" + wget "http://static.red-lang.org${path}" -O red + chmod +x red + cp red "${pkg}/usr/local/bin/" +``` + +## Unpack a tarball + +Sometimes tarballs can be extracted directly to `/usr/local`. However, +unless there's an obviously correct way to do that, we typically put +language distributions in `/opt/` and then put symlinks in +`/usr/local/bin` for the binaries. The `-C` and `--strip-components` +flags are very helpful for controlling the extraction. + +Also, one thing that might be unintuitive is that we use system-global +paths for the targets of symlinks (`/opt/swift/bin/swiftc`) but +`${pkg}` paths for the link names (`${pkg}/usr/local/bin/swiftc`). +This is because while we are putting all files into `${pkg}` during +build, the eventual place they will be installed by the package is +into the root filesystem, so any references to paths *within* files +(including symlink targets) must not mention `${pkg}`. + +```yaml +install: + manual: | + install -d "${pkg}/opt/swift" + install -d "${pkg}/usr/local/bin" + + ver="$(latest_release apple/swift | grep -Eo '[0-9.]+')" + wget "https://swift.org/builds/swift-${ver}-release/ubuntu2004/swift-${ver}-RELEASE/swift-${ver}-RELEASE-ubuntu20.04.tar.gz" -O swift.tar.gz + tar -xf swift.tar.gz -C "${pkg}/opt/swift" --strip-components=2 + ln -s /opt/swift/bin/swiftc /opt/swift/bin/sourcekit-lsp "${pkg}/usr/local/bin/" +``` + +## Compile from source + +Sometimes there is no binary distribution. In this case you want to +clone/download the source code and run the appropriate compilation +command. GCC and LLVM plus a number of scripting languages are already +installed into the `packaging` image by default, but you might have to +include a `prepare` block listing additional things to install to be +able to compile the software. + +For example, in this case the `build.sh` script provided by Zot +happens to require some Qt development dependencies at compile time, +and a non-development dependency at runtime. + +```yaml +install: + prepare: + apt: + - qt5-qmake + - qtscript5-dev + apt: + - libqt5script5 + manual: | + install -d "${pkg}/usr/local/bin" + + git clone https://github.com/manyoso/zot.git + pushd zot + ./build.sh + cp build/bin/zot "${pkg}/usr/local/bin/" + popd +``` + +## Use a shared dependency + +Your language may need to use software such as Prettier which is also +used by other languages but does not have an Ubuntu package. In that +case we package it as a Riju shared dependency (check the `shared` +directory for a list), and you declare the dependency using the `riju` +key. + +```yaml +install: + riju: + - prettier +``` + +## Install custom scripts or config files + +Sometimes you may need to add a wrapper script or config file +somewhere on the filesystem. Riju has the `scripts` and `files` keys +for this. The keys of these maps are the filesystem paths (`scripts` +defaults to putting things in `/usr/local/bin` if relative paths are +given). + +```yaml +install: + scripts: + teco-encode: | + #!/usr/bin/env -S python3 -u + + import re + import sys + + for line in sys.stdin: + line = re.sub(r"\^(.)", lambda m: chr(ord(m.group(1)) ^ 0b1000000), line) + line = line.replace("$", chr(27)) + print(line, end="") + files: + "/opt/sqlite/sqls.yml": | + connections: + - driver: sqlite3 + dataSourceName: db.sqlite3 +``` + +## Setting up skeleton files + +Sometimes your language may require some configuration files in +addition to the actual file to be compiled. Or alternatively, your +language may do a bunch of caching work the first time it starts up, +which you don't want to happen every time someone launches it on Riju. +Both of these problems can be solved by preparing files to be copied +into the user's home directory at the start of a new Riju session +(this is done using the `setup` key, which is explained later). + +In the following example, we ask .NET to create a project template for +us, and then compile the project. Then we take the files that were +created for the project, plus cache directories in `$HOME`, and copy +them into `/opt` so that they can be copied back later using `setup`. +By convention, we name such directories `skel`. + +```yaml +install: + prepare: + apt: + - $(grep-aptavail -wF Package "dotnet-sdk-3\.[0-9.]+" -s Package -n | sort -Vr | head -n1) + manual: | + install -d "${pkg}/opt/qsharp/skel" + + dotnet new -i Microsoft.Quantum.ProjectTemplates + dotnet new console -lang Q# -o main + dotnet run --project main + + shopt -s dotglob + cp -R * "${HOME}/.dotnet" "${HOME}/.nuget" "${pkg}/opt/qsharp/skel/" + rm "${pkg}/opt/qsharp/skel/main/Program.qs" + chmod -R a=u,go-w "${pkg}/opt/qsharp/skel" +``` + +## Dealing with versioned APT packages + +For some reason, some APT packages have version numbers in their +names. For example, you can't `sudo apt install lua`; you have to +`sudo apt install lua5.4`. The best way to deal with these situations +is to use various `grep-aptavail` hacks to identify the latest +available version programmatically. Check the man page as well as uses +of `grep-aptavail` in Riju to understand the options. + +```yaml +install: + apt: + - $(grep-aptavail -XF Provides lua -a -XF Version "$(grep-aptavail -XF Provides lua -s Version -n | sort -Vr | head -n1)" -s Package -n | head -n1) +``` diff --git a/doc/tutorial/run.md b/doc/tutorial/run.md new file mode 100644 index 0000000..dd22fcb --- /dev/null +++ b/doc/tutorial/run.md @@ -0,0 +1 @@ +# Tutorial: provide run commands