Case file 4/5: the layer that arrived late
The case of the config that named a layer its image did not carry, the warning that printed correctly for days, and the diagnosis that deduced because it could not look
🕵️ Show the full case file → 📋 Session protocol →
Case file · Code Homicide Dept.
A whole day turning copies into a shared cascade, and at the end the site down twice for the same thing: a config naming a layer its artifact does not carry. The second outage caused by whoever had just written the guard against the first.
Show glossary
- three-layer cascade
- rustelo (framework) → website-htmx-rustelo (implementation) → consumer site. A file present in a higher layer wins; one absent from it is inherited from below. A site's config imports `htmx-site/
.ncl`, which IS the middle layer, and that layer lives in a directory the binary mounts at startup. - rebuild class
- The classification meaning «these bytes are only valid against an image that resolves them». Its declared behaviour was «they ship inert and are reported as owed», and that is the pathogen: inert only while nothing restarts. The same `publish` restarts.
- read at startup
- A file the server evaluates ONCE, when the process starts. Delivering it changes nothing visible; the next restart activates it. That is what separates a fault you see immediately from one that waits days with the detonator in place.
- delivery over exec
- Content travels to the PV through `kubectl exec`, inside the container. The consequence is cruel and defines this case: when the container stops starting, the tool loses the very path by which it would undo its own change.
- ask the artifact
- The rule this case bought. A gate consults no list, no date and no reasoning about what the image ought to carry: it asks the image. An artifact's contents belong to the artifact and change underneath you.
- I could not look
- The state no check may report as green. With the container down there was no `exec`, and the gap was filled with a chronologically impeccable, false deduction. An «I could not look» is a finding.
The protocol to declare, version and verify this → ontoref.dev
The double ledger — what it cost, and what it left
The uncomfortable part deserves no polish. The second outage was caused by whoever had just written the guard against the first, and the diagnostic error that followed — claiming two sites ran different images — was committed through exactly the vice this repository had spent the day removing from everywhere else: I could not look, so I deduced.
What it cost for the layer to arrive afterwards
- ontoref.dev outages in one day 2 — same class, ~19 h apart
- CrashLoopBackOff restarts (second one) 6 on the first pod, 3 on its replacement
- Layers the config named / the image carried 5 / 4
- Files actually broken / reverted under pressure 1 / 4
- Recovery scale to 0 + temp pod mounting the PVC — twice, because
execdies with the container - False claim issued under pressure «the two sites run different images» — one digest, 097bf919
- Probes discarded while building the gate 3 (ConfigMap, baked MANIFEST,
host) - Final mechanism 105 lines of preflight + 115 of the compatibility gate
What was left standing
- Mechanism copies removed 3 scripts × 3 sites, and 5 config stems 3 sites were writing by hand
- Gates that ask the artifact 2 — one in the dev loop, one at delivery
- Sites born with the defence in place all of them: the gate ships in the generator template
The suspects — the false leads
| The publish script bypassed the classifier | “False: `update-content.sh` calls content.nu, and the classifier graded all four files rebuild, correctly. The guard worked; what failed was what the label meant.” | DISMISSED |
| The two sites run different images | “False, and I claimed it with the container down and no way to look. Both pods ran the same digest. The deduction was chronologically impeccable: the cluster does not run on dates, it runs on digests.” | REFUTED BY THE DIGEST |
| Mount the profile layers with a ConfigMap | “Insufficient: the profile layers import framework base records the image also predated. Two blind overlays onto an image nobody could inspect.” | INSUFFICIENT |
| Bake a profile MANIFEST into the image | “Unnecessary: the `htmx-site/` directory ALREADY is the manifest. A file would have been a second copy of a directory listing — the exact shape I had spent the day deleting.” | REDUNDANT |
| `host $cp_node` to learn whether the target answers | “Wrong: `cp_node` is an ~/.ssh/config alias, not a DNS name. It called the real control plane unresolvable and skipped the check on the one site that could be verified.” | TESTED ON ONE INPUT |
The weapon — Two lines of the same `publish`. The first declares the debt; the second detonates it.
[publish] 4 file(s) still owe a rebuild — bytes ship inert,
the rest of this changeset does not wait on them
[publish] 21 files → class=restart → rolling restart
The warning had been printing correctly for days. A debt that cries on every run is a debt people learn to scroll past, and that is how it was read on the night it mattered.
The turn — Three questions, none of them to a list.
# the gate asks the pod, never a list kubectl exec deploy/$D -- printenv NICKEL_IMPORT_PATH # its order, not mine kubectl exec deploy/$D -- nickel export …/index.ncl # its nickel, its image kubectl exec deploy/$D -- ls …/nickel/htmx-site/ # the directory IS the manifest
Falsified in all three branches: green with the five layers, red naming the missing one, and SKIP said out loud when the target is not configured yet.
The verdict
The instrument that would have answered — config-check — validated against the roots
installed on the laptop, which are always ahead of any image. That is why a site could be
green locally and unstartable in a pod, and why the fault did not feel like carelessness:
every piece answered, and none of them answered that.
What remains written is not a warning. It is that a machine now asks the question, twice — in the dev loop and at delivery — and both times it asks the artifact.
| A config names a layer the image does not carry | profile-compat-check: the imports against the pod's ls, in the dev loop. |
| Unresolvable bytes reach the PV | preflight: exports index.ncl inside the pod. Red delivers nothing. |
| «rebuild» read as a gate when it was a label | The notice is worded from the proof, not from the classification. |
| The list of image-dependent files | A rule derived from the import, never a list: the list is born one entry short. |
| Being unable to look at the artifact | SKIP said out loud · RED if it will not answer · a verdict if it does. No green by ignorance. |
Case law — what enforces the lesson today
- ✓A config importing a shared layer is verified against the TARGET's image, never against the laptop's roots
gate:just profile-compat-check - ✓The site's config must export against the three roots the server mounts, not the developer's
gate:just config-check - ✓Both of the above run on every
check, including the first one a freshly generated site ever runsgate:just check - ✓The delivery preflight refuses to publish bytes the pod cannot load — and now someone can point at it
test:provisioning/tests/test_preflight_refuses.nu
An «I could not look» is a finding. We wrote it into a gate that same afternoon, and I did not apply it to myself.
From the project vocabulary (2)
- Gate
- Typed prerequisites and policies controlling FSM state transitions in a project.
- ontoref
- The protocol itself: a typed, queryable surface on which a project declares WHAT IT IS (ontology) and HOW IT ACTS (reflection), so a claim about the project can be contradicted by a machine and not only by a reader.