Skip to main content

Versioned immutable asset releases

Day-to-day operation of release channels is covered in the Egress Operations Runbook.

Status: implemented and rolled out. Publishing landed in sync-to-gcs.yml on 24 Aug 2026; the resolver and SDK began reading release paths on 25 Aug; and 140 of 142 router entries were moved onto a release channel on 26 Aug. Sections written before rollout are marked as such rather than rewritten, so the reasoning behind each decision stays readable. Day-to-day operation is in the Egress Operations Runbook. The layout, build-id format, manifest schema, retention window and scope recorded here are the agreed decisions, not open questions. Every fact about current behaviour was verified against live HTTP headers, live bucket listings, and the shipped code on the same date.

One operational step is not automated and must be run once by a maintainer: the retention lifecycle rule (§3) and, after the workflow change merges, ./scripts/deploy-workflow.sh --yes to propagate it to the env branches.

Read Storefront asset delivery and the egress surface first — it is the factual baseline this proposal builds on.

1. What problem this solves​

Storefront scripts/ and css/ are served from a stable URL whose content changes on every deploy:

https://storage.googleapis.com/finmatch-p/scripts/finance-container.js
https://storage.googleapis.com/finmatch-p/css/finmatch.css

Because the bytes at that URL change, the cache header has to be a compromise — public, max-age=900, stale-while-revalidate=3600 on finmatch-p — and every returning shopper re-validates or refetches roughly every 15 minutes per object. A cold visitor pulls ~700 KB of script and CSS (finance-container.js alone is 174,454 B).

Versioned immutable URLs remove the compromise: content at a versioned URL never changes, so public, max-age=31536000, immutable is safe, and freshness moves entirely to the one small mutable resolver response (getMerchantConfig), which already has the right caching model (30 s buckets + ETag/304).

When written, this step only published the immutable copies, and nothing read them until Step 3 teaches the resolver and SDK to point at a version. Existing env-bucket publishing is unchanged and keeps serving all live traffic.

2. What a release is​

A release is an immutable snapshot of the storefront runtime asset tree (scripts/ + css/) from exactly one env-branch commit, plus a manifest describing it.

  • One release per push to finmatch-p that touches scripts/ or css/.
  • Releases are never overwritten, patched, or deleted while any merchant can still resolve to them.
  • Releases contain no config. configs/ is admin/API-owned, mutable, and single-writer; freezing a copy of it into a release would create exactly the mirror-drift class of problem the Jun 2026 cleanup removed (source of truth §1). Config freshness is handled separately by content-hash query strings (Step 5).

Build identifier​

<env>-<YYYYMMDD>-<HHMM>-<short-sha>      e.g.  p-20260824-2137-301e047d

Rationale: sync-to-gcs.yml already computes git rev-parse --short HEAD and a commit timestamp for build-info.json, so this needs no new counter, no new state, and no second writer. It maps one-to-one back to a commit for incident triage.

The time component matters for exactly that triage. With date and sha only, two releases on the same day sort alphabetically by hash, so a directory listing does not tell you which one is newer — the wrong property to discover during an incident. HHMM makes the listing strictly chronological.

Rejected alternatives: a monotonic counter needs stored state and a writer; a bare hash loses all time context; semver implies compatibility guarantees that do not exist for a continuously deployed asset tree.

A re-run of the workflow on an unchanged commit produces the same build id and therefore the same immutable path. The publish step must be idempotent: re-publishing identical bytes to an existing release path is a no-op, and publishing different bytes to an existing release path is a hard failure (see §6, guardrail RELEASE_IMMUTABLE).

3. Bucket and path naming​

Two candidates. Both were checked against the constraints that actually bite; the recommendation is Option B for the first implementation.

Option A — dedicated bucket​

gs://finmatch-releases/<build-id>/scripts/finance-container.js
gs://finmatch-releases/<build-id>/css/finmatch.css
gs://finmatch-releases/<build-id>/release-manifest.json

Option B — versioned prefix inside the existing env bucket​

gs://finmatch-p/releases/<build-id>/scripts/finance-container.js
gs://finmatch-p/releases/<build-id>/css/finmatch.css
gs://finmatch-p/releases/<build-id>/release-manifest.json

What separates them​

Constraint (verified)Option A (new bucket)Option B (env-bucket prefix)
Per-origin CORS allow-list. gs://finmatch-p returns access-control-allow-origin: https://<merchant-domain> per allowed origin, not *. ES module imports are always CORS-mode fetches, so an origin missing from the list breaks the storefront outright. The list is applied by merchant-api to a hardcoded CORS_TARGET_BUCKETS = ['finmatch-finance-marketing-assets', 'finmatch-shared', 'finmatch-p', 'finmatch-s', 'finmatch-t'].Needs a merchant-api code change plus a POST /admin/reconcile-cors run before the first merchant can load a release. Until then every release URL fails CORS for every merchant.Inherited unchanged. No code change, no reconcile.
Bucket creation, IAM, public readNew bucket in europe-west2; grant the GCS_CREDENTIALS service account write; grant allUsers object read; add to the Check bucket permissions loop in sync-to-gcs.yml.Nothing.
Object accumulationNeeds a lifecycle rule from day one.Needs the same lifecycle rule, scoped to the releases/ prefix.
Interaction with existing sync stepsNone.gsutil -m rsync only runs against configs; cp -r targets scripts and css; the setmeta reapply globs are gs://finmatch-p/scripts/** and gs://finmatch-p/css/**. None of these touch or re-header releases/**. Verified against the current workflow.
Later CDN backingCleaner: one backend bucket holding only immutable objects.Workable but the backend bucket would also cover mutable scripts/, css/, configs/.
Blast radius if the publish step misbehaves (as assessed before rollout)Confined to a bucket nothing read at the time.Writes into the bucket that serves production, under a new prefix.

Decision: Option B, with Option A as the end state. Option B removes the CORS blocker, which is the one constraint that can break live storefronts rather than merely delay the programme, and it needs no infrastructure sign-off beyond the workflow diff. Migrating to Option A later is cheap because Step 3 makes the base URL a resolver-supplied value, not something burned into the SDK: repoint the channel, and merchants follow.

Why not gs://finmatch-shared/releases/…?​

Raised during sign-off, since finmatch-shared is already on the CORS allow-list (verified: it returns access-control-allow-origin: https://<merchant-domain> for a real merchant origin) and would therefore also need no code change. It was rejected on a safety property, not on aesthetics.

Env modules take their environment from the resolver and fall back to reading their own module URL when the resolver has not told them. Running that helper against each candidate path, with no runtime block present:

Module served fromresolved envisProductionRuntime
finmatch-p/scripts/… (today)ptrue
finmatch-p/releases/<id>/scripts/…ptrue
finmatch-shared/releases/<id>/scripts/…sharedfalse

isProductionRuntime false switches off the Propensio and Snap shared rate-card top-ups, and Propensio has 23 products in the shared catalogue and 0 in the finmatch-p env rate card — so a Propensio merchant would silently lose every Propensio option. That state is reachable whenever the runtime block is absent: a browser running a cached SDK (max-age=300, stale-while-revalidate=86400), a resolver not yet deployed, or a failed resolver call falling back to cached config. Nesting under the env bucket fails safe in all three; nesting under shared fails silent.

Naming the path gs://finmatch-shared/releases/finmatch-p/<id>/ does not rescue it: the match is unanchored and takes the first hit, so the bucket name still wins.

Environment-neutrality, the actual motivation, is already delivered without the path: channel pointers carry an absolute baseUrl, and a pointer's environment is validated against the merchant's live environment, so a sandbox preview on t can load a release hosted under finmatch-p/releases/. The build id also carries the origin.

If Option A is preferred later, it must be sequenced as: create bucket → add to CORS_TARGET_BUCKETS → deploy merchant-api → reconcile CORS → verify a real merchant origin gets an access-control-allow-origin header on a release URL → only then publish anything Step 3 can consume.

Lifecycle / retention​

Releases are additive, so the bucket grows by roughly one asset tree (~700 KB) per finmatch-p deploy that touches scripts/ or css/.

Agreed rule: delete objects under releases/ older than 90 days.

180 was the first proposal, on the assumption that deleting a pinned release would break a storefront. It does not: if a release is pruned while a merchant still points at it, the critical scripts 404 and the SDK retries them from the env bucket, leaving the page exactly as it is today. That path is covered by an explicit test (automatic fallback when the release 404s, in which all 16 scripts recover from the env bucket with zero redirects), so a mistaken prune degrades rather than breaks. Given that, longer retention buys very little; 90 days is ample history for triage at roughly 700 KB per deploy that touches scripts/ or css/.

Applying it is a one-off manual step, and gsutil lifecycle set REPLACES the bucket's entire lifecycle configuration. Read the current config first and merge, never set blind:

gsutil lifecycle get gs://finmatch-p > /tmp/finmatch-p-lifecycle.json   # keep this
# merge the rule below into the existing "rule" array, then:
gsutil lifecycle set /tmp/finmatch-p-lifecycle-merged.json gs://finmatch-p
{ "action": { "type": "Delete" },
  "condition": { "age": 90, "matchesPrefix": ["releases/"] } }

CI deliberately does not apply this: a workflow that rewrote bucket lifecycle on every push could silently drop an unrelated rule.

4. Release manifest​

One file per release, inside the versioned (therefore immutable) path:

<base-url>/release-manifest.json
{
  "schemaVersion": 1,
  "buildId": "p-20260824-757fb7c2",
  "env": "p",
  "branch": "finmatch-p",
  "commit": "757fb7c2",
  "builtAt": "2026-08-24T13:41:02Z",
  "baseUrl": "https://storage.googleapis.com/finmatch-p/releases/p-20260824-757fb7c2",
  "cacheControl": "public, max-age=31536000, immutable",
  "entrypoints": {
    "container": "scripts/finance-container.js",
    "css": "css/finmatch.css"
  },
  "assets": {
    "scripts/finance-container.js": { "bytes": 174454, "md5": "J3lQ2J91tVgnU6FV2JAmGg==" },
    "css/finmatch.css": { "bytes": 133278, "md5": "RKnPpmn6szfIRkpjthLWGg==" }
  }
}

Field notes:

  • schemaVersion — bumped on any breaking shape change. Readers must ignore unknown fields so the manifest can grow additively.
  • commit / builtAt / branch — the same three facts build-info.json already carries, so a release is traceable to a commit without a second version registry.
  • baseUrl — absolute, so a consumer never has to reconstruct it from a bucket name and prefix. This is the value Step 3 hands to the SDK as runtime.baseUrl.
  • assets — bytes and md5 exactly as GCS reports them (x-goog-hash: md5=…), keyed by path relative to the release root, covering every published directory. This makes a release verifiable after the fact and is what the RELEASE_IMMUTABLE guardrail compares against on a re-run (§6), including the asset count so a file that disappears is caught too.
  • entrypoints — named rather than positional, so the SDK's tiered loader does not have to hardcode filenames that differ between the modern (finance-container.js) and legacy (embed-lender-calc.js) paths.

No mutable pointer file in the bucket​

There is deliberately no latest.json or channels.json in the release bucket. Channel → version pointers live in merchant-router.json, whose single writer is merchant-api (Step 3). A mutable pointer object in a bucket that CI also writes would create a second writer for routing state — the precise failure mode documented in source of truth §4 (Jun 2026 router-drift incident).

5. Cache headers​

PathCache-ControlWhy
releases/<build-id>/**public, max-age=31536000, immutableContent at the URL can never change.
existing scripts/, css/unchanged (public, max-age=900, stale-while-revalidate=3600 on p; public, max-age=10, must-revalidate on s/t)Releases run alongside existing publishing; this step changes nothing about what live traffic is served.
build-info.jsonunchanged (no-cache)Deploy marker, not an asset.

6. CI touchpoints​

All changes are inside .github/workflows/sync-to-gcs.yml on finmatch-shared, then propagated to env branches with ./scripts/deploy-workflow.sh --yes (the file is drift-checked by workflow-drift-check.yml and by an in-workflow guardrail, so an env-branch edit fails the build).

Insertion point​

Inside the existing Sync environment branches to GCS step, after the scripts/ and css/ cp -r and the setmeta reapply, and before the build-info.json upload. That ordering means a release is only published once the mutable env copy for the same commit has already succeeded, so a release can never describe a state the env bucket never reached.

The diff, as implemented​

# ── Versioned immutable release (read by merchants on a release channel) ──
if [ "${{ github.ref_name }}" = "finmatch-p" ] && { [ -d "scripts" ] || [ -d "css" ]; }; then
  BUILD_ID="p-$(git log -1 --format=%cd --date=format:'%Y%m%d')-$(git rev-parse --short HEAD)"
  RELEASE_PREFIX="gs://${{ github.ref_name }}/releases/${BUILD_ID}"
  IMMUTABLE_HEADER="public, max-age=31536000, immutable"
  echo "DEBUG: Publishing release ${BUILD_ID} to ${RELEASE_PREFIX}"

  # Guardrail RELEASE_IMMUTABLE: a re-run on the same commit must not change
  # bytes already published at an immutable URL. The published manifest already
  # records an md5 per asset (§4), so compare against that rather than
  # re-deriving it — and cover every directory the publish step uploads, not
  # just scripts/.
  if gsutil -q stat "${RELEASE_PREFIX}/release-manifest.json" 2>/dev/null; then
    echo "INFO: release ${BUILD_ID} already published; verifying byte-identity"
    gsutil cat "${RELEASE_PREFIX}/release-manifest.json" > /tmp/published-manifest.json
    DRIFT=0

    # Membership first, as an exact set comparison: catches both a new file and
    # a file that has disappeared, without comparing counts as strings.
    jq -r '.assets | keys[]' /tmp/published-manifest.json | sort > /tmp/published-assets.txt
    for DIR in scripts css; do
      [ -d "$DIR" ] && find "$DIR" -type f -printf '%p\n'
    done | sort > /tmp/local-assets.txt
    if ! diff -u /tmp/published-assets.txt /tmp/local-assets.txt > /tmp/asset-set-diff.txt; then
      echo "::error::asset set changed for ${BUILD_ID}:"
      cat /tmp/asset-set-diff.txt
      DRIFT=1
    fi

    # Then content, for every asset present in both.
    while IFS= read -r REL; do
      [ -f "$REL" ] || continue
      LOCAL_MD5="$(openssl dgst -md5 -binary "$REL" | base64)"
      PUBLISHED_MD5="$(jq -r --arg k "$REL" '.assets[$k].md5 // ""' /tmp/published-manifest.json)"
      if [ -n "$PUBLISHED_MD5" ] && [ "$LOCAL_MD5" != "$PUBLISHED_MD5" ]; then
        echo "::error::${REL} differs from the bytes published at ${BUILD_ID} (local ${LOCAL_MD5}, published ${PUBLISHED_MD5})"
        DRIFT=1
      fi
    done < /tmp/local-assets.txt

    if [ "$DRIFT" != "0" ]; then
      echo "::error::Immutable release paths must never change."
      exit 1
    fi
    echo "OK: release ${BUILD_ID} unchanged, nothing to publish"
  else
    [ -d "scripts" ] && gsutil -m -h "Cache-Control:${IMMUTABLE_HEADER}" cp -r scripts "${RELEASE_PREFIX}/"
    [ -d "css" ]     && gsutil -m -h "Cache-Control:${IMMUTABLE_HEADER}" cp -r css     "${RELEASE_PREFIX}/"

    # Manifest is generated from what was actually uploaded, not from the
    # working tree, so it can never describe files that failed to publish.
    python3 - "$BUILD_ID" "$RELEASE_PREFIX" <<'PY' > /tmp/release-manifest.json
    # ... walks `gsutil ls -L` output for the prefix, emits the §4 schema ...
PY
    gsutil -h "Cache-Control:${IMMUTABLE_HEADER}" -h "Content-Type:application/json" \
      cp /tmp/release-manifest.json "${RELEASE_PREFIX}/release-manifest.json"
  fi
fi

Guardrail interactions (all verified against the current workflow)​

GuardrailEffect on this step
Guardrail - block protected runtime config writesNo interaction. Releases contain no configs/ file, and the protected regex is anchored at repo-root paths.
Guardrail - block ghost runtime edits on finmatch-sharedNo interaction. The publish step is gated on github.ref_name == 'finmatch-p'; nothing is added under root scripts/, css/, configs/ on finmatch-shared.
Guardrail - fail if sync-to-gcs workflow driftedRequires ./scripts/deploy-workflow.sh --yes after the finmatch-shared merge, before the next env-branch push.
Guardrail - env branches must not carry scripts/finmatch-sdk.jsNo interaction. The SDK is not part of a release (see §7).
Pre/post Snapshot protected runtime object hashesNo interaction; it only inspects configs/finmatch-merchant-config.json.
Clean up .DS_Store filesGlobs are depth-limited (gs://<bucket>/*/.DS_Store, */*/.DS_Store) and would not reach releases/<build-id>/scripts/. A .DS_Store committed under scripts/ would be published into a release; the existing cp -r has the same exposure today.
snap-rate-card-test.ymlNo interaction; path-filtered to configs/finmatch-rate-card.json and scripts/snap-rate-card-test.js.

Secrets and permissions​

Option B needs no new secret or IAM grant: GCS_CREDENTIALS already writes gs://finmatch-p. Option A needs a write grant for that service account on the new bucket plus allUsers object read, and the bucket added to the Check bucket permissions loop.

7. What is deliberately not in a release​

  • The canonical SDK (finmatch-finance-marketing-assets/s/scripts/finmatch-sdk.js). Its URL is burned into merchant pages and cannot be versioned. It must stay a small, short-cache bootstrap that resolves a version rather than being one (asset delivery §5).
  • configs/ — see §2.
  • The legacy header-loader shims under gs://finmatch-finance-marketing-assets/{p,s,t}/scripts/. Retiring those needs merchant outreach and is out of scope for this programme.

8. Blockers Step 3 must clear before anything reads a release​

These were found while verifying this proposal and are the reason Step 2 and Step 3 are not fully independent. Publishing releases (this step) is safe in isolation; consuming them is not, until these are resolved.

8.1 Env detection by import.meta.url (7 sites, 2 files)​

Storefront modules decide production-only behaviour by sniffing their own URL:

const scriptUrl = new URL(import.meta.url);
const isFinmatchP =
    scriptUrl.hostname === 'storage.googleapis.com' &&
    scriptUrl.pathname.includes('/finmatch-p/');

Under gs://finmatch-p/releases/<build-id>/scripts/… the path still contains /finmatch-p/, so Option B happens to keep these gates working — but that is a coincidence, not a design, and it would break immediately under Option A (gs://finmatch-releases/… has no /finmatch-p/ segment). Current sites, on finmatch-p:

FileLineWhat it gates
scripts/finance-container.js~160Propensio shared rate-card top-up
scripts/finance-container.js~218Snap shared rate-card top-up
scripts/finance-container.js~280shared vs env-local lender-overrides
scripts/finance-container.js~289relative ../configs/lender-overrides.json
scripts/finmatch-finance-app.js~678isFinmatchPRuntime() (rate-card candidates, and one further gate at ~762)
scripts/finmatch-finance-app.js~908shared vs env-local lender-styling
scripts/finmatch-finance-app.js~915relative ../configs/lender-styling.json

The same sniffing also means a sandbox preview with "Preview link loads: Test build" cannot exercise the two rate-card top-ups, because assets then come from gs://finmatch-t/. That is called out in the Step 1 PR.

Required fix (Step 3): the resolver states the runtime environment explicitly and the SDK exposes it (window.finmatch.runtime.environment); env modules prefer that value and fall back to import.meta.url sniffing only when it is absent.

8.2 Relative config URLs resolved against the module URL​

new URL('../configs/lender-overrides.json', import.meta.url) and the lender-styling.json equivalent resolve relative to wherever the module was loaded from. Under a release path that becomes …/releases/<build-id>/configs/lender-overrides.json, which does not exist. Only finmatch-s takes these branches today, so it is latent rather than live — but finmatch-s must not be moved onto releases until the SDK supplies absolute config URLs. Step 5 groundwork (resolver-supplied, content-hashed config URLs) is the fix.

8.3 Per-origin CORS​

See §3. Applies to Option A only.

8.4 Legacy shim cache busting​

finmatch-header-loader.js appends v=${stored.version || Date.now()} to the SDK URL, so any legacy snippet that never stored a version busts the SDK cache on every shim execution. This does not block releases, but it does mean SDK cache-hit rate cannot be assumed until the shim population is audited (asset delivery §2).

9. Verification​

The publish step was extracted from the workflow and executed against the real finmatch-p working tree with a gsutil stand-in backed by a local directory, so the shell logic itself is tested rather than reasoned about. Five runs:

RunScenarioResult
1first publish79 assets published, Cache-Control: public, max-age=31536000, immutable recorded on scripts/, css/ and the manifest; manifest emitted with the §4 shape
2re-run, same commit, unchanged treerelease … already published and unchanged; nothing to do, exit 0, nothing re-uploaded
3re-run with css/finmatch.css modifiedexit 1, naming the file and both md5s
4re-run with a script addedexit 1, set diff showing +scripts/zz-new-module.js
5re-run after reverting bothclean no-op again
6env sync reported a failurerelease skipped with a warning, nothing published

Timestamps are rendered with TZ=UTC git log --date=format-local:…. Plain %cd renders in the commit's own timezone, which would have stamped a commit made at 23:37+05:30 as 23:37Z — out by five and a half hours in both the build id and builtAt.

Cross-checked against production: the manifest's md5 for scripts/finance-container.js is 9EVpPBGdgo8sMDalJOT+ww==, byte-identical to the x-goog-hash on the live gs://finmatch-p/scripts/finance-container.js. So a manifest describes exactly what production serves.

Still to confirm on the first real run, since a local stand-in cannot prove them:

  • fetch a built asset at its immutable URL and assert HTTP 200 plus the immutable cache-control from GCS itself;
  • assert the mutable env paths are untouched — gs://finmatch-p/scripts/** still on max-age=900, stale-while-revalidate=3600, same generations.

No traffic metric was expected at this step, because nothing read these paths until a channel pointer names one.

10. Sign-off record (24 Aug 2026)​

  • Option B — gs://finmatch-p/releases/<build-id>/.
  • Build id <env>-<YYYYMMDD>-<HHMM>-<short-sha>.
  • Manifest schema §4, including no mutable pointer object in the bucket.
  • Retention: delete releases/** after 90 days, applied manually per §3.
  • finmatch-p only. finmatch-s needs the absolute-config-URL work in §8.2 first, and there is no reason to take that on before the mechanism has proven itself on production.
  • ./scripts/deploy-workflow.sh --yes is run by the maintainer after the workflow change merges.

Deploys and pinned merchants​

A merchant on a release channel is pinned to a version. Publishing a new release does not move them, so a deploy would reach the unversioned env bucket — and therefore every merchant not on a channel — while release-channel merchants kept running the previous build from behind a year-long immutable cache. A bug fix would never arrive.

sync-to-gcs therefore advances any channel that already has a pointer to the build it has just published, on pushes to finmatch-p. The rule is narrow by design:

SituationBehaviour
No channel has a pointerExits cleanly, changes nothing. Adopting release channels stays an operator decision made through merchant-api
A channel has a pointerAdvanced to the new BUILD_ID, and the merchant count on release channels is logged
The release was not publishedFails before touching any channel
Any HTTP status other than 200Fails loudly. curl -sS exits 0 on a 500, so the status is checked explicitly
HTTP 200 but no channels objectFails loudly. A JSON error body has no channels key, and defaulting it to {} would be indistinguishable from "none in use"
channels present but emptyExits cleanly — this is the genuine "none in use" case
CI cannot read internal-api-secretFails with the manual PUT to run instead

This requires the CI service account behind GCS_CREDENTIALS to hold roles/secretmanager.secretAccessor on internal-api-secret. Until that grant exists the step fails rather than silently skipping, because a silent skip is exactly the failure mode it is there to prevent.

What this means for the caching win​

Advancing on every deploy means shoppers do re-download assets after a deploy — the same as today, since a deploy changes the bytes either way. The win is between deploys: a returning shopper fetches nothing at all, where previously they revalidated 20-odd files every 15 minutes. Measured on absolutemotocross.co.uk after moving it to a release channel: 895,043 bytes on a cold visit, then zero on every subsequent visit.