# Introduction

Ambient does automated [continuous integration][] for programmers in a safe and secure way.
That means it builds the software and runs its automated tests,
but prevents the software under test from harming the host it runs on.
Ambient achieves this by only running code from the software under test,
or any of its dependencies,
in a virtual machine without network access,
and only the resources configured by the Ambient user.


[continuous integration]: https://en.wikipedia.org/wiki/Continuous_integration


# Installation

See [`INSTALL.md`](https://radicle.network/nodes/radicle.liw.fi/rad:zwPaQSTBX8hktn22F6tHAZSFH2Fh/tree/INSTALL.md)
in the source tree.


# Getting started

## Built-in help

The `ambient` command has extensive builtin help: give the `-h` or `--help` option to the command,
or any subcommand,
or use `help` subcommand:

~~~
ambient -h
ambient --help
ambient run --help
ambient help
ambient help run
~~~

## Projects file {#projects}

To run CI on a project using Ambient, you need to create a "projects file".

~~~yaml
projects:
  dummy:
    image: /scratch/ambient-images/ambient-boot.qcow2
    source: ~/pers/ambient-ci/ambient-ci
    pre_plan:
      - action: cargo_fetch
    plan:
      - action: cargo_clippy
      - action: cargo_build
      - action: cargo_doc
      - action: cargo_test
~~~

The projects file, in YAML, specifies the projects Ambient should know about.
For each project:

* `image` is the virtual machine image to use
  - a custom image is published at [`files.liw.fi`](https://files.liw.fi/ambient/)
* `source` is the path to the source directory of the project
  - it can be, but does not need to be, a Git checkout
* `pre_plan`, `plan`, and `post_plan` are lists of actions to execute for a CI run
  - `pre_plan` actions are executed in a VM with network access
  - `plan` actions are executed inside a VM with no network access
  - `post_plan` actions are executed in a VM with network access, with secrets

In each kind of plan, only specific actions are allowed. See the [Actions](#actions) chapter for
a list.

## Configuration

You should configure Ambient by creating `~/.config/ambient/config.yaml`
(the location obeys the [XDG base directory spec](https://specifications.freedesktop.org/basedir/latest/)):

~~~yaml
tmpdir: /scratch/tmp
projects: ~/liw-dot-files/ambient.yaml
rsync_target: "_ewww@webby:/srv/http"
dput_target: apt.liw.fi
executor: /scratch/cargo-cache/x86_64-unknown-linux-musl/debug/ambient-execute-plan
artifacts_max_size: 1G
cache_max_size: 50G
state: /scratch/ambient-state
qemu:
  cpus: 8
  memory: 16G
secrets:
  - source: /home/ci/.ssh/id_ci-cert.pub
    dest: key-cert.pub
  - source: /home/ci/.ssh/id_ci
    dest: key
~~~

The fields are:

* `tmpdir` - location where Ambient temporary files are put
  - these can get quite large, so it can be necessary to override the default `$TMPDIR`
* `projects` - path to the [projects file](#projects)
* `rsync_target` - where the `rsync` action uploads files over SSH
* `rsync_target_base`, `rsync_target_map` - compute `rsync` target by
  appending a per-project value to a common base value
* `dput_target` - where `deb` packages are upload using `dput`
* `executor` - the `ambient-execute-plan` program suitable for the VM used
* `artifacts_max_size`, `cache_max_size` - how big the artifacts and cache directories can grow
* `state` - location where Ambient keeps pre-project state
* `qemu.cpus`, `qemu.memory` - number of CPUs and memory to give to the VM
* `secrets` - a mapping of files on host to files in `/ci/secrets` for the post-plan VM

Some of those are optional, as they have defaults.
To see the actual run-time configuration used by `ambient`, run:

~~~
ambient config
~~~

You can specify the configuration file to use with the `--config` option.

The `rsync` target fields are a little complicated.

* If `rsync_target` is set, that is used, regardless of the other fields.
* If only `rsync_target_base` is set, it is joined with the project name.
* If both `rsync_target_base` and `rsync_target_map` are set, the project name
  is used to index the map, and the base value is joined with that; if the map
  does not have an entry for the project, the project name is used as a fallback

Example:

~~~
rsync_target_base: ci@example.com:/srv/http
rsync_target_map:
  foo: bar
  bar: yo
~~~

With this configuration the `rsync` target will be as in the table below.

|project|`rsync` target|
|:-|:-|
|`foo`|`ci@example.com:/srv/http/bar`
|`bar`|`ci@example.com:/srv/http/yo`
|`xyzzy`|`ci@example.com:/srv/http/xyzzy`


## Secrets (SSH keys)

The virtual machine that runs post-plan actions will have the secrets
configured (see `secrets` field) under `/ci/secrets`. The SSH client in the
VM is configured to use `/ci/secrets/ssh_key`, if it exists. This allows the
`rsync`, `rsync2`, `dput`, and `dput2` post-plan actions to deliver artifacts
using SSH. If an SSH user certificate is at `/ci/secrets/ssh_key-cert.pub`, it
will be used automatically.

An example configuration file snippet:

~~~yaml
secrets:
  - source: /home/ci/.ssh/id_ci-cert.pub
    dest: key-cert.pub
  - source: /home/ci/.ssh/id_ci
    dest: key
~~~

Note that the secrets files are only copied to the VM where post-plan actions
run. They are not exposed to pre-plan and plan actions. This prevents the code
being built from including a copy of a secret file.


## Run Ambient

Given an image and a [projects file](#projects), run Ambient using a command like this:

~~~
ambient run 
~~~

This runs CI for every project in the projects file, if the project source code has changed.
Add the `--force` option to force CI to run.

Any output to the standard output and error from any of the actions is written to the standard output
and gathered in a "run log".
The run log can be viewed later with

~~~
ambient log dummy
~~~

The run log is, for now, quite messy.


# Actions for use in Ambient CI plans {#actions}

An action is an atomic task executed during a CI run. Actions are specified in
the CI plan. The plan is divided into a pre-plan, actual plan, and post-plan.
All three are executed in fresh, ephemeral virtual machines. The pre- and
post-plan actions are executed with network access and the actual plan is
executed with no network access. The actions allowed in pre- and post-plan are
carefully designed and vetted so that they are safe and secure to run. Most
importantly, they do not execute any code from the software under test.

Actions in the actual plan can do whatever they want, and certainly execute
code from the software under test (that's the point of CI). Because the
virtual machine isolates the host machine, even malicious code can't harm the
host or other machines.

## The workspace in the virtual machine

Ambient sets up a workspace where CI is run. Until version 0.10, this was
`/workspace`, but after that, it is also known as `/ci`, for brevity. The old
name also works to avoid breaking existing CI plans. However, Ambient checks
for uses of the old name and warns about them. If `lint` is set to `true`,
it'll refuse to run CI if there's a warning.

The workspace contains several directories:

* `/ci/src` - the source code; this is the contents of the `source` directory
  in the project specification; changes in the source directory are not
  persisted and vanish when the VM shuts down
* `/ci/deps` - dependencies downloaded by pre-plan actions;
  changes in the dependencies directory are not persisted and vanish when the
  VM shuts down
* `/ci/cache` - a cache that persists between CI runs
* `/ci/artifacts` - files built by CI, for publication by post-plan actions
* `/ci/secrets` - files configured in the configuration file; the directory
  always exists, but the files are there only during post-plan

The `cache` and `artifacts` directories have a maximum size. When the CI
run ends, the contents of those directories are written as tar archives
to the corresponding virtual block devices set up by Ambient. This is what
will fail if the directories are too large. See [`artifacts_max_size` and
`cache_max_size`](#configuration) in the configuration file.

You mostly only need to care about the workspace and its layout in the
[`shell` action](#shell). All other actions know what's where, but your
shell script snippets need to use the right locations. Also any [custom
action](#custom) executables need to do that.

Ambient adds actions to unpack the source and other directories at the start
of a CI run, and exporting the cache and artifacts directories at the end. You
don't need to do that manually.


## Pre-plan actions

The following actions are allowed in the pre-plan.

### `dummy`

~~~yaml
- action: dummy
~~~

Do nothing, except write a message to the standard output. This action is
meant for trouble-shooting pre-plans.

### `pwd`

~~~yaml
- action: pwd
~~~

Write out the path to the current working directory. This action is meant
for trouble-shooting.

### `cargo_fetch`

~~~yaml
- action: cargo_fetch
~~~

Download Rust crate dependencies using `cargo fetch`. The downloaded crates
will be available in the VM in a location where `cargo` will find them. The
downloaded crates will be cached so that they do not need to be downloaded
again for the next CI run.

For safety and security, `cargo fetch` is run on a stripped down copy of
the source tree. This prevents the software under test from reconfiguring
`cargo` using files in the source tree. This does not affect what crates are
downloaded.

The crates will be in `/ci/deps` and the `CARGO_HOME` environment variable is
set to that location.

### `http_get`

~~~yaml
- action: http_get
  items:
  - url: https://files.example.com/big.xz
    filename: data.xz
  - url: https://files.example.com/cat.jpg
    filename: cat.jpg
~~~

Download files over HTTP if they are missing locally or have changed on the
server. The downloaded files are available in the dependencies directory in
the VM, `/ci/deps`. If the file exists locally, the download uses the HTTP
header `If-Modified-Since` to avoid downloading it again, unless the file on
the server has a newer modification time.

The downloads are done using HTTP GET. There is no support for authentication.
There is no automatic cleaning of downloaded files.

Both the `url` and `filename` fields are required. The filename may not contain
a directory.

### `deb_get` {#deb_get}

~~~yaml
- action: deb_get
  packages:
  - liwc
  suite: trixie
~~~

Download `deb` packages from Debian, using the `http://deb.debian.org/debian`
mirror.

Allowed fields:

* `packages` is the list of package names to download.

* `suite` is a release code name (`trixie`, `bookworm`, etc) or
  an alias (`stable`, `oldstable`, etc).

* `cert` is the OpenPGP certificate (i.e. public key) for the key used to sign
  the release. The certificates for `trixie` and `bookworm` are built into
  Ambient. Note that aliases like `stable` can refer to a different release at
  different times, and so a cert isn't know for them.

The action will figure out the dependencies needed (`Depends` and `Pre-Depends`
fields in package metadata). Recommended and suggested packages are not
considered. The entire package dependency graph is downloaded, as the action
is run outside the VM and cannot know what the VM needs.

This action merely downloads the packages. See the plan action
[`deb_install`](#deb_install) to actually install them inside the VM.

The packages are downloaded into the `debian` subdirectory of the project
dependencies directory.

### `npm_get` {#npm_get}

~~~yaml
- action: npm_get
~~~

Download `npm` (or `node`) packages found in `npm-shrinkwrap.json` or
`package-lock.json` (use first file found). They are placed in the `npm`
subdirectory of the project dependencies directory.

It is up to the plan to install them inside the VM.

### `pip_download`

~~~yaml
- action: pip_download
  requirements: requirements.txt
  allow_sdist: false
~~~

Download Python packages from PyPI, using the `pip download` command.
This requires the VM image to have Python with `pip` installed. The
dependencies need to be specified in a way that `pip` understands,
such as with a `requirements.txt` file.

To install the downloaded packages in the plan VM, use a command such as the
following:

~~~
plan:
  - action: shell
    shell: |
      pip install --break-system-packages \
        --no-index --find-links /ci/deps/pip -r requirements.txt
~~~

Allowed fields:

* `requirements` is the name of the file, relative to the source
  directory, where the requirements are specified. Optional. Defaults to
  `requirements.txt`

* `allow_sdist` specifies whether source distributions (not just wheels)
  are allowed. Optional. Defaults to `false`.


### `go_mod_download`

Download Go module dependencies using `go mod download`.

~~~yaml
- action: go_mod_download
~~~

Using this action requires the VM image has the Go toolchain installed.

The downloaded modules are put into `/ci/deps/gomod`. To use them
in a CI plan, set the environment variable before running `go`:

~~~
export GOMODCACHE=/ci/deps/gomod
go build
~~~

(This is not yet set automatically by Ambient.)

### `rustup`

~~~yaml
- action: rustup
  channel: stable
  target: x86_64-unknown-linux-musl
  profile: minimal
~~~

Download a specific Rust toolchain version using `rustup` on the host. The
toolchain will be available in the VM.

Allowed fields:

* `channel` is release channel to use, or version of Rust. Mandatory.

* `target` is the platform to build for. Optional. Default is whatever
  the host running Ambient has.

* `profile` is installation profile and defaults to `minimal`. Optional.

See the [rustup book](https://rust-lang.github.io/rustup/concepts/index.html)
for more information.

The toolchains are downloaded into the `rustup` subdirectory of the
dependencies directory.

## Actual plan actions

These are also known as "unsafe actions". They're allowed to be dangerous
and to run code from the software under test, because they get executed in
a virtual machine that isolates and constrains the code so it can't harm the
host running Ambient, or any other hosts.

### `mkdir`

~~~yaml
- action: mkdir
  pathname: /ci/src
~~~

Ensure a directory exists. If it already does, do nothing, otherwise create
it. This is meant for Ambient internal use, to create the CI workspace when a
CI run starts. User-provided plans can also use this action.

### `tar_create`

~~~yaml
- action: tar_create
  archive: /ci/artifacts/rsync/source.tar
  directory: /ci/src
~~~

Create a [`tar` archive][] with the contents of a directory. This is meant for
Ambient internal use to export data from the virtual machine via virtual block
devices, i.e., the cache and artifacts directories. User-provided plans can
also use this action.

[`tar` archive]: https://en.wikipedia.org/wiki/Tar_(computing)

### `tar_extract`

~~~yaml
- action: tar_extract
  archive: /dev/vdx
  directory: /ci/yummy
~~~

Extract a [`tar` archive][] to a directory. This is meant for Ambient internal
use to import data into the virtual machine via virtual block devices, e.g.,
the source, dependencies, and cache directories. User-provided plans can also
use this action.

### `shell`

~~~yaml
- action: shell
  shell: |
    echo hello, world
    make
~~~

Execute a shell script snippet using `bash`. The Bash setting `set -xeuo
pipefail` will be in effect. This means that if the snippet uses a variable
that hasn't been set, or a command fails, the action fails.

### `deb_install` {#deb_install}

~~~yaml
- action: deb_install
~~~

Install all `deb` packages downloaded by the [`deb_get`](#deb_get) pre-plan action.

### `cargo_fmt`

~~~yaml
- action: cargo_fmt
~~~

Check that Rust source code is formatted in the idiomatic style, by running
`cargo fmt --check`.

### `cargo_clippy`

~~~yaml
- action: cargo_clippy
~~~

Check that Rust code is correct and idiomatic by running `cargo clippy`. Any
warnings are treated as errors.

### `cargo_deny`

~~~yaml
- action: cargo_deny
~~~

Check that Rust code only has dependencies, licenses, and known security
problems that are explicitly allowed, by running `cargo deny`.

### `cargo_doc`

~~~yaml
- action: cargo_doc
~~~

Format documentation from Rust code, using `cargo doc`. The formatted
documentation is put in the `cargo` target directory, `CARGO_TARGET`, which
is in the cache directory. It is not automatically put into the artifacts
directory. You have to do that yourself, if you want it.

### `cargo_build`

~~~yaml
- action: cargo_build
~~~

Build a Rust project, including binaries, tests, and examples ("all targets").
This runs `cargo build` with suitable options.

### `cargo_test`

~~~yaml
- action: cargo_test
~~~

Run tests in a Rust project, using `cargo test`.

### `cargo_install`

~~~yaml
- action: cargo_install
~~~

Install a Rust project into the artifacts directory, `/ci/artifacts`.

### `deb`

~~~yaml
- action: deb
~~~

Build a Debian `deb` package. This first creates an "upstream tar" from the
source tree, using `git archive`, and then running `dpkg-buildpackage`, the
standard tool for building Debian packages. The built files, including the
`.changes` file, get put into the artifacts directory, `/ci/artifacts`.

Note that this assumes the source is in Git, that Debian packaging tools are
installed in the VM, and that the `debian` directory includes the necessary
files for Debian packaging.

Also note that this action does not invent a version number for CI builds.

This action is meant to be used together with the [`dput`](#dput) post-plan
action.

This action differs from the [`deb2` action](#deb2) only in the location of
where built packages are put.

### `deb2`

~~~yaml
- action: deb2
~~~

Build a Debian `deb` package. This first creates an "upstream tar" from the
source tree, using `git archive`, and then running `dpkg-buildpackage`, the
standard tool for building Debian packages. The built files, including the
`.changes` file, get put into `debian` directory in the artifacts directory,
`/ci/artifacts/debian`.

Note that this assumes the source is in Git, that Debian packaging tools are
installed in the VM, and that the `debian` directory includes the necessary
files for Debian packaging.

Also note that this action does not invent a version number for CI builds.

This action is meant to be used together with the [`dput2`](#dput2) post-plan
action.

This action differs from the [`deb` action](#deb) only in the location of
where built packages are put.

### `custom`

~~~yaml
- action: custom
  name: dch
  args:
    debemail: liw@liw.fi
    debfullname: "Lars Wirzenius"
~~~

Execute a "custom action". Custom actions are executables (usually shell
scripts, but any executable is OK), located in the `.ambient` directory at the
root of the source tree. The `custom` action runs the executable and passes it
arguments as specified in the `args` field. The whole `args` field is encoded
as JSON and given to the executable via its standard input. In addition, each
argument `name` is separately encoded as JSON and an environment variable
`AMBIENT_CI_name` is set to contain the JSON. The executable can use either,
whichever is more convenient to it.

Note that Ambient does not try to retrieve the custom action executables
from anywhere. The project that uses the `custom` action has to include
the executable in their source tree. This avoids making Ambient do package
management for the executables, but make using them more tedious. The `custom`
action is best considered a compromise while the Ambient project explores
the needs and wants for custom actions. The Ambient project maintains a small
[repository of custom actions][] as part of the compromise.

[repository of custom actions]: https://radicle.network/nodes/radicle.liw.fi/rad%3Az1LPp3eFHEwG7sZrnc447W1aC5Z

Example custom action script to add a Debian package changelog entry with a
new version suitable for a CI build:

~~~sh
#!/usr/bin/env bash

set -euo pipefail

export DEBEMAIL="$AMBIENT_CI_debemail"
export DEBFULLNAME="$AMBIENT_CI_debfullname"
export CARGO_TARGET_DIR=/ci/cache
export CARGO_HOME=/ci/deps
export HOME=/root
export PATH="/root/.cargo/bin:$PATH"

git reset --hard
git clean -fdx

V="$(dpkg-parsechangelog -SVersion | sed 's/-[^-]*$//')"
T="$(date -u "+%Y%m%dT%H%M%S")"
version="$V.ci$T-1"
dch -v "$version" "CI build under Ambient."
dch -r ''
~~~

(The example comes from the repository of custom actions, script `dch`.)

### `setenv`

~~~yaml
- action: setenv
  set:
    foo: bar
~~~

Set environment variables for later actions.

## Post-plan actions

### `dummy`

This is the same action as in the pre-plan.

### `pwd`

This is the same action as in the pre-plan.

### `rsync`

~~~yaml
- action: rsync
~~~

Publish the contents of the entire artifacts directory (`/ci/artifacts` in
the VM) using the `rsync` program. The sync target is specified in Ambient
configuration (see [`rsync_target`](#configuration)); the software under test cannot affect
that. The SSH credentials for the host running Ambient are used by `rsync`.
Any files on the target that are not in the artifacts directory are deleted.

### `rsync2`

~~~yaml
- action: rsync2
~~~

Publish the contents of the `rsync` subdirectory of the artifacts directory
(`/ci/artifacts/rsync` in the VM) using the `rsync` program. The sync target
is specified in Ambient configuration (see [`rsync_target`](#configuration)); the software under
test cannot affect that. The SSH credentials for the host running Ambient
are used by `rsync`. Any files on the target that are not in the artifacts
directory are deleted.

The difference between the `rsync` and `rsync2` actions is the location of
the files to be published. The `rsync` action publishes the entire artifacts
directory, which may also contain packages built by the `deb` action. The
`rsync2` action only publishes the subdirectory, which contains no packages.

### `dput`

~~~yaml
- action: dput
~~~

Upload Debian `deb` packages from the artifacts directory to an APT
repository, using the `dput` program on the host and the SSH credentials of
the host. The packages are found by looking for `.changes` files anywhere inside
`/ci/artifacts`, including subdirectories. The [`deb` action](#deb) puts them
in `/ci/artifacts`.

### `dput2`

~~~yaml
- action: dput2
~~~

Upload Debian `deb` packages from the artifacts directory to an APT
repository, using the `dput` program on the host and the SSH credentials of
the host. The packages are found by looking for `.changes` files anywhere
inside `/ci/artifacts/debian`, including subdirectories. The [`deb2`
action](#deb2) puts them there.

