spindlet

A Kubernetes execution engine for tangled’s spindle CI
runner. Each workflow becomes a pod, each step becomes an exec into it, and
spindle itself runs as an ordinary unprivileged Deployment — no Docker socket,
no /dev/kvm, no host mounts.

It implements upstream’s models.Engine interface, so it’s a drop-in
alternative to the Docker (nixery) and microVM backends rather than a fork of
spindle.

How a workflow runs

InitWorkflow    parse the workflow yaml, resolve the nixery image, prepend setup steps
SetupWorkflow   create a NetworkPolicy, create the pod, wait for it to accept exec
RunStep   (×N)  exec `bash -s` into the pod, feed it the step script over stdin
DestroyWorkflow delete the pod and the policy

The pod’s PID 1 is sleep infinity. It exists purely to hold the container
open between steps, which is what lets step 3 see the files step 2 wrote — the
workspace is an emptyDir at /tangled shared by every exec.

A few decisions worth knowing about

Secrets never touch argv. The Kubernetes exec API has no Env field, so the
obvious move is env KEY=value cmd — which puts secrets where ps and the API
server’s audit log can see them. Instead the whole step, exports and all, is
generated as a shell script and streamed over stdin. Secrets never become
Kubernetes objects either, so they’re never in etcd. There’s an integration test
asserting a running step can’t find its own secret in /proc/$$/cmdline, so this
stays true.

The script isn’t executed straight off stdin, though: bash reads stdin
incrementally, so a step running ssh, kubectl apply -f - or
docker login --password-stdin would eat the rest of its own script and exit
zero having skipped it. The stream instead stages the step to a file via a
heredoc whose delimiter is derived from the body, runs it with stdin closed, and
deletes it — so a step can’t reach the transport that carried it.

Exec goes over websockets. The v5 subprotocol carries the CLOSE signal that
lets stdin reach EOF, which the script-over-stdin design depends on. SPDY
remains as a fallback for older API servers and header-mangling proxies, but
only on upgrade failures — never on a command failure, or a failed step would
run twice.

Fork code can be sandboxed. InitWorkflow can tell whether the checkout came
from a fork, and untrusted workflows get runtimeClassName from a separate
setting, so you can point them at gVisor or Kata while trusted code runs
normally.

Workflow pods can’t phone home. Each gets a NetworkPolicy that denies all
ingress and blocks egress to RFC1918 and link-local — so no reaching cluster
services or the cloud metadata endpoint at 169.254.169.254 — with DNS allowed
explicitly, since kube-dns lives inside a blocked range.

Engine names

Registered under two names pointing at the same instance:

• kubernetes — the default name, for pipelines that use Kubernetes-only
  features and can’t run anywhere else.
• nixery — a compatibility alias. Tangled is federated and repos move
  between spindles; engine: nixery runs on hosted spindle’s Docker backend
  and here, because the manifest schema and image resolution are identical.

microvm is deliberately not registered. Its manifest accepts services:,
virtualisation: and registry: — NixOS guest configuration with no pod
translation — so those pipelines fail loudly with unknown engine instead of
running with options silently dropped.

The one real gap under the nixery alias: pipelines that build images by
bind-mounting the host Docker socket. Pods have no host socket. Point
workflow.dockerHost at a remote dockerd or buildkitd and they work again.

Getting started

nix develop        # go, kubectl, kind, helm + unittest, buildah, just
just               # list every recipe
just check         # build, vet, race tests, lint, chart tests
just integration   # spins up kind and runs the real thing

Build an image and install the chart into that kind cluster:

just image
just image-kind

helm install spindlet charts/spindlet \
  --namespace spindlet-system --create-namespace \
  --set image.repository=atcr.io/gabe.labrats.cc/spindlet \
  --set spindle.hostname=spindle.example.com \
  --set spindle.owner=did:plc:you

just release does the real thing — checks, builds, verifies the image can do
what spindle needs, then pushes the image and chart and tags the repository. It
shows what it resolved from Chart.yaml and waits for confirmation before
anything leaves the machine.

Configuration

The chart’s values.yaml is the documented
surface, and values.schema.json rejects typos at helm template rather than
at pod start. Underneath, the engine reads SPINDLE_K8S_PIPELINES_* environment
variables alongside upstream’s own SPINDLE_* ones, using the same
go-envconfig upstream does.

For anything the chart doesn’t model — tolerations, affinity, a cache PVC,
sidecars — there’s workflow.podTemplate, a PodSpec used as the base for
every workflow pod. The engine reapplies the fields its execution model depends
on (image, command, mounts, restartPolicy: Never, no service account token)
on top, so a template can change where and how big a workflow runs but not
what runs.

Building images

Point pipelines at a BuildKit daemon and keep the builder out of the workflow
pod, which then needs no privileges at all:

dependencies:
  nixpkgs: [buildkit]
steps:
  - name: build
    command: buildctl build --frontend dockerfile.v0 --local context=. --local dockerfile=.

BUILDKIT_HOST and any client certificates belong in workflow.podTemplate, so
pipelines never name the builder. Registry credentials stay in the workflow pod:
BuildKit takes auth from the client, not from its own config.

Running a builder inside the pod instead means privileged pod security for
the whole namespace — buildah calls unshare(CLONE_NEWUSER) even under
--isolation chroot, RuntimeDefault denies it, and baseline refuses the
Unconfined profile that would allow it. workflow.seccompProfile and
workflow.extraCapabilities exist for that case.

Caching

Workflows start from an empty emptyDir, so dependencies are re-fetched every
run. There is no cache in the engine — mount one through workflow.podTemplate,
remembering that a ReadWriteOnce claim serialises concurrent workflows.

Requirements

• Kubernetes 1.30+ for websocket exec; older clusters fall back to SPDY.
• A workflow image with git ≥ 2.49 and bash. spindle clones with
  --revision=, which is why the floor is that high, and it checks at startup.
• Pod Security baseline in the workflow namespace. restricted is too tight:
  nix profile needs CHOWN and SETUID inside the container.
• Single replica. State is sqlite on a PVC, so the Deployment is Recreate.
  Out-of-tree entrypoints can only be standalone role — mill and executor need
  unexported fields only upstream’s Run can set.

Layout

cmd/spindlet/   entrypoint; mirrors spindle.Run and injects the engine map
engine/         the five Engine methods, pod building, exec, reaping
config/         SPINDLE_K8S_PIPELINES_* config
charts/spindlet chart, values schema, helm-unittest suites
justfile        build, test, image and release recipes
hack/           the container signature policy buildah needs