2daa11c7aa
This was an absolutely disgusting operation. Here's the commands I used to do it (see <https://docs.gomplate.ca/> for the tool): $ node > require('fs').writeFile('langs.json', JSON.stringify(require('./langs').langs), console.log) $ ./gomplate -c .=langs.json -f template.yaml > langs.yaml $ cat langs.json | jq 'keys[]' -r L | while read lang; do cat langs.yaml | sed -n "/^ #START $lang\$/,/^ #END $lang\$/p" | tail -n+2 | head -n-1 | sed 's/^ //' > langs/$lang.yaml; done I had some difficulty getting valid YAML to be generated, so I found the tool at <http://www.yamllint.com/> to be helpful in identifying errors. For validation that nothing important had changed in the transformation, I used <http://www.jsondiff.com/> to compare langs.json against the output of this command: $ ./gomplate -c .=langs.json -f template.yaml | node yaml2json.js | jq '(.. | select(type == "string")) |= sub("\n+$";"") | (.. | select(.template?) | .template?) |= (.+"\n") | (.. | select(.format?.input?) | .format?.input?) |= (.+"\n")' Here's yaml2json.js: import fs from "fs"; import YAML from "yaml"; console.log(JSON.stringify(YAML.parse(fs.readFileSync(0, "utf-8")), null, 2)); And here's the template.yaml, which is approximately the worst thing ever: {{ range . }} "{{ .id }}": #START {{ .id }} id: "{{ .id }}" {{ if has . "aliases" }} aliases: {{ range .aliases }} - "{{ . }}" {{ end }}{{ end }} name: "{{ .name }}" {{ if has . "monacoLang" }} monacoLang: {{ .monacoLang }} {{ end }}{{ if has . "info" }} info: {{ if has .info "impl" }} impl: "{{ .info.impl }}" {{ end }}{{ if has .info "version" }} version: "{{ .info.version }}" {{ end }} year: {{ .info.year }} desc: "{{ .info.desc }}" {{ if eq "string" (printf "%T" .info.ext) }} ext: {{ .info.ext }} {{ else if eq 0 (len .info.ext) }} ext: [] {{ else }} ext: {{ range .info.ext }} - {{ . }} {{ end }}{{ end }} web: {{ if has .info.web "wiki" }} wiki: "{{ .info.web.wiki }}" {{ end }}{{ if has .info.web "esolang" }} esolang: "{{ .info.web.esolang }}" {{ end }}{{ if has .info.web "home" }} home: "{{ .info.web.home }}" {{ end }}{{ if has .info.web "impl" }} impl: "{{ .info.web.impl }}" {{ end }} source: {{ if eq "string" (printf "%T" .info.web.source) }}"{{ .info.web.source }}" {{ else }}null {{ end }}{{ if eq "string" (printf "%T" .info.category) }} category: {{ .info.category }} {{ else if eq 0 (len .info.category) }} category: [] {{ else }} category: {{ range .info.category }} - {{ . }} {{ end }}{{ end }}{{ if eq "string" (printf "%T" .info.mode) }} mode: {{ .info.mode }} {{ else if eq 0 (len .info.mode) }} mode: [] {{ else }} mode: {{ range .info.mode }} - {{ . }} {{ end }}{{ end }}{{ if eq "string" (printf "%T" .info.platform) }} platform: {{ .info.platform }} {{ else if eq 0 (len .info.platform) }} platform: [] {{ else }} platform: {{ range .info.platform }} - {{ . }} {{ end }}{{ end }}{{ if eq "string" (printf "%T" .info.syntax) }} syntax: {{ .info.syntax }} {{ else if eq 0 (len .info.syntax) }} syntax: [] {{ else }} syntax: {{ range .info.syntax }} - {{ . }} {{ end }}{{ end }}{{ if eq "string" (printf "%T" .info.typing) }} typing: {{ .info.typing }} {{ else if eq 0 (len .info.typing) }} typing: [] {{ else }} typing: {{ range .info.typing }} - {{ . }} {{ end }}{{ end }}{{ if eq "string" (printf "%T" .info.paradigm) }} paradigm: {{ .info.paradigm }} {{ else if eq 0 (len .info.paradigm) }} paradigm: [] {{ else }} paradigm: {{ range .info.paradigm }} - {{ . }} {{ end }}{{ end }}{{ if eq "string" (printf "%T" .info.usage) }} usage: {{ .info.usage }} {{ else if eq 0 (len .info.usage) }} usage: [] {{ else }} usage: {{ range .info.usage }} - {{ . }} {{ end }}{{ end }}{{ end }}{{ if or (has . "daemon") (has . "setup") }} {{ end }}{{ if has . "daemon" }} daemon: | {{ .daemon | indent 4 }} {{ end }}{{ if has . "setup" }} setup: | {{ .setup | indent 4 }} {{ end }}{{ if has . "repl" }} repl: | {{ .repl | indent 4 }} {{ end }}{{ if has . "input" }}{{ if .input | strings.Contains "\x1b" }} input: "{{ .input | strings.ReplaceAll "\x1b" "\\x1b" }}" {{ else }} input: |{{ if .input | regexp.Match "^\\s" }}2{{ end }} {{ .input | indent 4 }} {{ end }}{{ end }}{{ if has . "output" }} output: | {{ .output | indent 4 }} {{ end }} main: "{{ .main }}" template: {{ if eq .id "whitespace" }}{{ .template | strings.Quote }} {{ else }}|{{ if .template | regexp.Match "^\\s" }}2{{ end }} {{ .template | indent 4 }}{{ end }}{{ if has . "prefix" }} prefix: | {{ .prefix | indent 4 }} {{ end }}{{ if has . "suffix" }} suffix: | {{ .suffix | indent 4 }} {{ end }}{{ if has . "createEmpty" }} createEmpty: "{{ .createEmpty }}" {{ end }} {{ if has . "compile" }} compile: | {{ .compile | indent 4 }} {{ end }} run: | {{ .run | indent 4 }} {{ if has . "helloInput" }} helloInput: | {{ .helloInput | indent 4 }} {{ end }}{{ if has . "hello" }} hello: | {{ .hello | indent 4 }} {{ end }}{{ if has . "helloMaxLength" }} helloMaxLength: {{ .helloMaxLength }} {{ end }}{{ if has . "runReplInput" }} runReplInput: | {{ .runReplInput | indent 4 }} {{ end }}{{ if has . "runReplOutput" }} runReplOutput: | {{ .runReplOutput | indent 4 }} {{ end }}{{ if has . "scope" }} scope: code: | {{ .scope.code | indent 6 }} {{ if has .scope "after" }} after: "{{ .scope.after }}" {{ end }}{{ if has .scope "input" }}{{ if .scope.input | strings.Contains "\x1b" }} input: "{{ .scope.input | strings.ReplaceAll "\x1b" "\\x1b" }}" {{ else }} input: | {{ .scope.input | indent 6 }} {{ end }}{{ end }}{{ if has .scope "output" }} output: | {{ .scope.output | indent 6 }} {{ end }}{{ end }}{{ if has . "format" }} format: run: | {{ .format.run | indent 6 }} {{ if has .format "input" }} input: | {{ .format.input | indent 6 }}{{ end }}{{ if has .format "output" }} output: | {{ .format.output | indent 6 }}{{ end }}{{ end }}{{ if has . "pkg" }} pkg: install: | {{ .pkg.install | indent 6 }} {{ if has .pkg "uninstall" }} uninstall: | {{ .pkg.uninstall | indent 6 }} {{ end }}{{ if has .pkg "all" }} all: | {{ .pkg.all | indent 6 }} {{ end }}{{ if has .pkg "search" }} search: | {{ .pkg.search | indent 6 }} {{ end }}{{ end }}{{ if has . "lsp" }} lsp: {{ if has .lsp "setup" }} setup: | {{ .lsp.setup | indent 6 }} {{ end }} start: | {{ .lsp.start | indent 6 }} {{ if has .lsp "disableDynamicRegistration" }} disableDynamicRegistration: {{ .lsp.disableDynamicRegistration }} {{ end }}{{ if has .lsp "init" }} init: {{ .lsp.init | data.ToYAML | indent 6 }}{{ end }}{{ if has .lsp "config" }} config: {{ .lsp.config | data.ToYAML }}{{ end }}{{ if has .lsp "lang" }} lang: "{{ .lsp.lang }}" {{ end }}{{ if has .lsp "code" }} code: {{ .lsp.code | strings.Quote }} {{ end }}{{ if has .lsp "after" }} after: {{ .lsp.after | strings.Quote }} {{ end }}{{ if has .lsp "item" }} item: {{ .lsp.item | strings.Quote }} {{ end }}{{ end }}{{ if has . "skip" }} skip: {{ range .skip }} - {{ . }} {{ end }}{{ end }} #END {{ .id }} {{ end }} |
||
|---|---|---|
| .circleci | ||
| backend | ||
| bin | ||
| docker | ||
| frontend | ||
| langs | ||
| packer | ||
| system | ||
| tf | ||
| tools | ||
| .dockerignore | ||
| .gitignore | ||
| Makefile | ||
| README.md | ||
| package.json | ||
| webpack.config.cjs | ||
| yarn.lock | ||
README.md
Riju
For now, this README just has some miscellaneous notes about the build architecture that I'm planning to set up. Later, it will be converted back into a proper README.
Steps to build Riju from scratch locally:
- Build the packaging Docker image.
- Generate a Debian package for each language.
- Build the runtime Docker image.
- For each language, install its Debian package into a fresh copy of the runtime Docker image and run its tests.
- Install every language's Debian package into a single copy of the runtime Docker image and run all the tests.
Build artifacts:
- Packaging image
- Runtime image
- Debian packages
- Application image
Steps to build Riju from cache locally:
- To run: Pull application image.
- To build application image: Pull runtime image and all Debian packages.
- To build Debian packages: Pull packaging image.
- To build runtime image: Build from scratch.
- To build packaging image: Build from scratch.
To manipulate published artifacts we basically want to do atomic updates which keep the integration tests passing. Possible operations:
- Rebuild packaging image: This can be done at any time.
- Rebuild runtime image: Rebuild application image and verify integration tests are still passing. Do not rebuild any Debian packages.
- Rebuild Debian package: Verify unit tests are passing. Rebuild application image and verify integration tests are still passing. If rebuilding multiple Debian packages, then we can run the integration tests only once. If rebuilding enough Debian packages, the probability that at least one will fail is very high. We can then trigger a more targeted update. This process could be automated.
- CI: Rebuild packaging and runtime images if needed. Rebuild Debian packages for any changed languages. Fetch everything unchanged from registry. Rebuild application image and verify integration tests are still passing. If yes and operating on main branch, publish all artifacts and deploy.