Storage reorganisation · detailed structure

Step by Step

Each of the six steps and three gates in full — what it changes, how it is built, how it reverses, and where it stands today. The state on each card is measured, not asserted.

Complete: 2Steps 1 and 3
Partly done: 2Step 2 has no mobile leg; step 4 has no flag
Built, not run: 1Step 5 — the copier
Unbuilt: 1Step 6
Gates passed: 0 of 3Gate 1 substantially met, not signed off

How to read this

Steps 1 to 3 change nothing a user can see and can ship on ordinary release days. Behaviour changes at step 4. Nothing is deleted or moved at any point, in any step — so every reversal below lands on an original that was never touched.

Step 1 — Store the full file path

Step 1

Widen the columns and fill in where every file is now

complete

What it does: Every column and JSON key that holds a file name is retyped to fit a complete path, then backfilled with where the file is now — the legacy flat folder — never where it is going.
Why first: Nothing else can be built on a value that does not say where its file is. Until the row carries the whole answer, every reader has to keep guessing the folder.
How it was built: Nine columns retyped: two widened to varchar(512), two to text, and five converted from a comma-joined string to varchar[] with each element trimmed and the empties dropped. Seventeen columns and thirty-two JSON key paths backfilled. Widths were raised before any value was written, so no statement in the backfill could overflow.
The rule that protects it: A value that already contained a slash was left alone — it is either a complete path already or an address somewhere else, and prefixing either would break it.
Reversal: Each migration has a down. The originals are untouched in the bucket, so a reverse restores bare names against folders that still hold every object.
State: Complete. 1,488,632 values re-counted in clone1; 1,488,594 hold a complete path. The 38 that do not are pre-existing and appear identically in capClone.

Step 2 — One rule finds every file

Step 2

Every path comes out of one table

partial

What it does: Every place that glued a location together by hand now calls one shared rule. A module name goes in, a prefix comes out; nothing else builds a storage path.
Why it matters: A single rule is what makes step 4 a change in one place rather than a change in every upload site, and what stops a value already in the new tree being prefixed a second time.
How it was built: UPLOAD_MODULE_TARGETS in backendApi/src/shared/modules/storage/upload-targets.ts is the table. uploadTargetFor throws when a target whose scope needs a branch does not name one, so a misconfigured module fails at presign rather than quietly writing a level too high. The front end mirrors it in capExpertApp/src/app/core/utils/media-paths.ts.
Reversal: Code-only. Reverting the branch restores the hand-built prefixes; no data changes.
State: Done for the API and the web app. The mobile leg is not: neither mobileApp nor mobileApi carries any of this storage work. mobileApi has CAP-2031 commits, but they are unrelated release fixes on release/v15.

Gate 1 — stop here

Gate 1

The point of no ambiguity

partial

The criteria: Steps 1 and 2 finished and live. Every repo green. Nowhere builds a location by hand. Older records holding only a file name still display. Nothing the website depends on has been lost from the published interface. The step-1 reversal performed once. Zero files moved.
Why a full stop: The system now knows where every file lives and not one file has moved. That is the only moment in the plan where both halves are simultaneously true, and it is the base every later rollback returns to.
State: Substantially met, not signed off. Zero files have moved, and the path rule is in place across the API and web app. Two criteria are open: the mobile leg of step 2, and evidence that the step-1 reversal has been performed once for real.

Step 3 — Thumbnails learn both shapes

Step 3

A thumbnail beside every picture, in either layout

deployed · running

What it does: The thumbnail service watches the whole img/ tree, not one folder, and writes the small copy at <same folder>/thumb/<same file name>.
Why before step 4: A picture uploaded into the new layout with no service watching it gets no thumbnail, and nothing goes back for it later. The service is trigger-only.
How it was built: A separate Node 22 service in thumbnailService/, on a GCS finalize trigger. A staged thumbnail is moved rather than regenerated; an original under 50 KB is copied as its own thumbnail. Public and thumbnail are separate decisions. THUMB_FOLDER_KEYS is derived from the thumb flags on the upload targets, so marking a module thumbnailed is one edit.
Reversal: Stop the trigger. Thumbnails are derived objects that no record references — readers compute the address from the picture — so removing them breaks no row.
State: Deployed and running as a Cloud Function on the bucket's finalize event, and node --test reports 78 of 78 passing. This step is closed.

Step 4 — New uploads change shape

Step 4

New files land in the account folder

partial

What the plan says: Flip one setting. New files land in the account's folder, staged under temp/ until the record is saved. Existing files untouched.
How it was actually built: The staging half is built exactly as described — tempPrefixFor stages, commitUploads files on save and returns the final paths, and the caller writes them back onto the record. The flag was not. finalPrefixFor returns the new prefix unconditionally; no setting gates it.
Why that matters: The setting was the plan's mechanism for ordering step 4 after step 3, and for reversing step 4 without a deploy. Without it, step 4 went live as soon as steps 1–2 shipped rather than on a deliberate flip, and it can now only be reversed by a deploy. In clone1 26 values already sit in the new tree. The ordering held — step 3 is deployed, so those uploads did get their thumbnails — but by circumstance rather than by design.
Reversal: Code-only, and therefore a deploy rather than a setting change. Objects already written to the new layout stay readable, because the value in the row is a complete path either way.
State: Live in effect, but not as the plan specifies. The behaviour is on; the switch that was supposed to control it does not exist. file_migration_enabled is a different flag — it is the copier's run switch, not the upload shape. Since step 3 is deployed, the practical risk is reversibility rather than missing thumbnails.

Gate 2 — stop here

Gate 2

An upload works end to end, and an abandoned one is reclaimed

not started

The criteria: An upload from web and from phone lands in the right folder and gets a thumbnail; five abandoned uploads never leave temp/.
State: Cannot currently be passed. The thumbnail half needs step 3 deployed, and the phone half needs the mobile leg of step 2. The reclaim half is built: TempUploadCleanupJob runs daily and releases a staged object only when two independent answers agree — nothing in the schema names its path, and the ledger accounts for the object.

Step 5 — Copy the existing files

Step 5

Move the pre-cutover objects into the new layout

built · not run

What it does: Account by account, copy each pre-cutover object to where the new layout says it belongs, verify it, rewrite the record, and leave the original in place.
How it was built: CAP-680. Two ledger tables, a seeder that fills the queue with one row per destination object, a runner that copies in chained batches across parallel lanes, and a write-back pass that rewrites the records. Set out in full on CAP-680.
Reversal: Built and symmetric. The ledger keeps both halves of every rewrite, so the rollback is the same SQL with stored_value and destination_path swapped rather than a second implementation that could drift.
State: Built, never run. file_migration_queue and file_migration_batches hold zero rows in clone1. The copier also has no descriptor for 30 of the 73 columns that can hold a path — see CAP-680 Gaps.

Gate 3 — stop here

Gate 3

One account, for real, including the rollback

not started

The criteria: One account copied, verified, and rolled back once for real before anything runs at scale.
Why it is the strictest gate: It is the only gate that tests the reversal on live data. Every later batch depends on the rollback working, and a rollback that has never been run is a claim rather than a fact.
State: Not performed. The queue has never been seeded, so no account has been copied and nothing has been rolled back. A rehearsal mode exists for exactly this — FILE_MIGRATION_TARGET_PREFIX writes every copy under a scratch folder while still copying and writing back consistently, so the whole path can be exercised end to end without touching a real destination.

Step 6 — Keep and watch the old folders

Step 6

Let evidence decide when the old folders can go

not started

What it does: Stop writing to the legacy folders, log which app versions still read them, and let the evidence — not a date — decide when they can be removed.
Why it is last: Customers run mobile versions we cannot update. The log is the only way to know when the last of them has stopped asking.
State: Unbuilt. No legacy-read logging exists anywhere in backendApi. The first half is true by construction — nothing writes to the legacy folders any more, because finalPrefixFor only produces new-layout paths — but nothing is recording who still reads them.

The two unbuilt pieces are new construction

Step 6's logging and step 4's flag are the only items in the plan that are neither repair nor migration. They were deliberately left until the copier existed, on the reasoning that there is no point logging legacy reads before anything has moved. That reasoning holds for step 6. It does not hold for step 4's flag, which was the mechanism for keeping step 4 behind step 3 — and step 4 is already live.