CI Artifact And Test Temp Disk Full Policy

When CI fails with No space left on device after tests pass or near artifact upload, the root cause is often peak temporary storage: test outputs, binlogs, fixture fanout, Docker image archives, Playwright/browser artifacts, or a scratch directory that is never pruned between shards.

$99 CI artifact policy

Turn one flaky disk-full job into a reusable artifact and temp-storage budget.

Use this for Helix/binlog uploads, Dart or Rust test temp output, Docker/kind image archives, browser artifacts, generated reports, and test fixtures that fill the runner after the useful work already ran.

peak temp -> artifact budget -> prune boundary -> upload gate -> shard policy

Read-only evidence

Capture the peak storage owners before deleting anything.

These checks separate test scratch, upload staging, Docker archives, tool caches, and generated reports. They are safe to paste into most public CI issues after removing private paths.

df -h; du -xhd1 "$PWD" /tmp "$RUNNER_TEMP" "$RUNNER_TOOL_CACHE"

Request $99 CI policy Request $29 incident triage

Runbook: Budget The Upload And Scratch Peaks

Measure disk before tests, before artifact upload, and after upload. Many failures happen after the test result exists.
Track each mount separately: root, workspace, /tmp, Docker root, tool cache, and runner temp.
Classify outputs as reusable cache, generated test scratch, required failure evidence, or upload-only artifacts.
Prune generated scratch only after the shard has uploaded the evidence needed for debugging.
Add a hard free-byte reserve before the next shard, Docker image archive, or artifact upload starts.
When failing early, report the largest bucket and the first safe cleanup action.

Copy-ready issue reply

Use this when CI passes work but fails on storage.

This keeps the discussion focused on peak storage ownership instead of a vague cache cleanup.

I would treat this as a CI peak-storage budget problem, not just generic runner cleanup.

The useful evidence is:
- free bytes before tests
- free bytes before artifact upload
- top workspace/temp/report/artifact directories
- whether Docker/kind/image archives create a second full copy
- which outputs are required for failure diagnosis versus generated scratch

Acceptance rule: every shard should leave enough reserve for the next shard or upload step, and cleanup should only remove generated outputs after required logs/artifacts are uploaded.

Request policy review

Do Not Delete First

Failure logs, binlogs, screenshots, traces, coverage files, or minidumps that are the only evidence for the failed run.
Tool caches on shared or self-hosted runners before confirming they are rebuildable and not shared with active jobs.
Docker volumes or named caches unless the job created them and the cleanup boundary is explicit.
Source workspace state before artifact upload has captured the needed debug output.