Egress Operations Runbook

Every lever used to run the asset-delivery system: moving merchants between release channels, pulling a merchant back, applying cache headers, and measuring the result. The design behind these levers is in Storefront Runtime Contract and Versioned Asset Releases; this page is what to type when something needs doing.

What is live

Piece	State
JSON cache-busters removed from rate-card and overrides fetches	Live on finmatch-p and finmatch-t
Resolver in-memory object cache, 30 s TTL	Live in getMerchantConfig
Content-hashed URLs for the four admin/API-written config objects	Live
public, max-age=31536000, immutable on those four objects	Live
Versioned immutable releases for scripts and CSS	Live, 140 of 142 router entries
sync-to-gcs advances an existing channel pointer on each deploy	Live

The two entries not on a release channel are environment: t. The live channel points at a finmatch-p release, and the resolver rejects a pointer whose environment differs from the merchant's, so assigning a test merchant to it would report success while the resolver quietly fell back. They are excluded deliberately.

Before any command: authentication

Every mutating endpoint requires x-internal-auth. The secret lives in Secret Manager, and Cloud Shell sessions lose their project setting regularly, so pass --project explicitly rather than relying on session state.

SECRET=$(gcloud secrets versions access latest \
  --secret=internal-api-secret --project=finmatch-finance-mkt)
M=https://merchant-api-238644427841.europe-west2.run.app

echo "secret length: ${#SECRET}"   # must print 64

If the length is 0, the session lost its credentials. Run gcloud auth login, then repeat. A zero-length secret produces {"error":"Forbidden"} on every call, which is a confusing way to discover an auth problem — hence the check.

Levers

See the current state

curl -sS "$M/admin/runtime-channels" | jq '{channels, merchants}'

GET needs no auth. Returns the pointer for each channel and a rollout summary: how many merchants are on a release channel, on the env bucket, and kill-switched.

Pull one merchant back

The emergency brake for a single merchant. Takes effect on the next resolver response, about 30 seconds, with no deploy and no cache purge.

curl -sS -X PUT "$M/admin/merchants/<MERCHANT_ID>/runtime-channel" \
  -H "x-internal-auth: $SECRET" -H 'Content-Type: application/json' \
  -d '{"killSwitch":true}' | jq .

Verify, allowing for the 30 s resolver cache:

sleep 32
curl -sS "https://getmerchantconfig-staging-238644427841.europe-west2.run.app/?merchantID=<MERCHANT_ID>" \
  | jq -c '{source: .runtime.source, reason: .runtime.fallbackReason}'

Expect source: "env-bucket" and reason: "kill-switch". Reverse it with {"killSwitch":false}.

Exercised against production on M281700 (the demo site) on 26 Aug 2026: the merchant fell back to finmatch-p within the cache window, its scripts loaded from the unversioned path, and no other merchant was affected.

Pull everyone back

curl -sS -X DELETE "$M/admin/runtime-channels/live" \
  -H "x-internal-auth: $SECRET" | jq .

One write, every merchant on the channel returns to the env bucket within ~30 seconds. This is the preferred rollback for a bad release: it does not require touching merchants individually and cannot hit the rate limit described below.

Point a channel at a release

Normally handled automatically by sync-to-gcs on each deploy. Do it by hand when pinning to a specific build.

V=p-20260825-2238-3177e82f
curl -sS -X PUT "$M/admin/runtime-channels/live" \
  -H "x-internal-auth: $SECRET" -H 'Content-Type: application/json' \
  -d "{\"version\":\"$V\",\"baseUrl\":\"https://storage.googleapis.com/finmatch-p/releases/$V\",\"environment\":\"p\"}" | jq .

The base URL must be a releases/<version> tree in finmatch-p, -s or -t, and the version must be the whole release segment. Anything else is rejected — a channel pointer decides which JavaScript every storefront on the channel executes, so the allow-list is deliberately narrow.

List published releases:

curl -sS "https://storage.googleapis.com/storage/v1/b/finmatch-p/o?prefix=releases/&delimiter=/" \
  | jq -r '.prefixes[]'

Assign merchants to a channel — pace it

Every assignment is a read-modify-write of one file, gs://finmatch-secrets/merchant-router.json. Google Cloud Storage allows roughly one write per second to a single object. An unpaced loop over ~137 merchants fails partway with HTTP 429 rateLimitExceeded. This happened twice during the August 2026 rollout.

Failures are safe — the writes use generation preconditions, so a rejected write changes nothing — but the rollout is left half-applied.

gsutil cat gs://finmatch-secrets/merchant-router.json > /tmp/router.json

jq -r 'to_entries
  | map(select(.key | startswith("_") | not))
  | map(select(.key != "versionStamp"))
  | map(select(.value.environment == "p"))
  | map(select(.value.runtimeChannel == null))
  | .[].key' /tmp/router.json > /tmp/todo.txt

echo "merchants to move: $(wc -l < /tmp/todo.txt)"

while read -r ID; do
  for ATTEMPT in 1 2 3; do
    R=$(curl -sS -X PUT "$M/admin/merchants/$ID/runtime-channel" \
      -H "x-internal-auth: $SECRET" -H 'Content-Type: application/json' \
      -d '{"channel":"live"}' | jq -r '.runtimeChannel // "retry"')
    [ "$R" = "live" ] && break
    sleep 3
  done
  printf '%-22s %s\n' "$ID" "$R"
  sleep 2
done < /tmp/todo.txt

curl -sS "$M/admin/runtime-channels" | jq '.merchants'

The selector excludes _-prefixed reserved keys, versionStamp, non-production merchants and anyone already on a channel — so it is safe to re-run, and it picks up merchants added since the last run.

There is no bulk endpoint. One would let this be a single write; see the follow-up note at the end.

Apply immutable headers to existing config objects

The writers set the header on every new write. This covers copies that already exist, and is idempotent.

./scripts/apply-immutable-config-headers.sh

Run it after any change to which objects are hash-addressed. It verifies itself and prints the resulting Cache-Control for each object.

A caution when spot-checking afterwards. For up to an hour after a setmeta, some GCS frontends keep serving the previous response with the old header. Object metadata is correct throughout. Check x-goog-generation to tell a genuine rewrite from a stale cached response:

curl -sS -D - -o /dev/null "https://storage.googleapis.com/finmatch-shared/finmatch-rate-card.json" \
  | grep -iE 'cache-control|x-goog-generation|last-modified'

Measuring egress

The method

Compare each hour against the same hour on clean baseline days, and prefer bytes per request over raw totals. Same-hour comparison controls for the daily traffic curve; bytes per request controls for traffic volume, so a quiet morning cannot flatter the result.

23 and 24 August 2026 are the clean pre-programme baseline. Do not use 25 August: that window is contaminated by the programme's own activity — browser harnesses, CI runs, release publishing — and runs up to 49% above the adjacent days on finmatch-p.

The report

START="$(date -u -d '5 days ago' +%Y-%m-%dT%H:00:00Z)"
END="$(date -u +%Y-%m-%dT%H:00:00Z)"
TOKEN="$(gcloud auth print-access-token)"
PROJECT="$(gcloud config get-value project)"

for B in finmatch-shared finmatch-p; do
  for METRIC in "network/sent_bytes_count:bytes" "api/request_count:reqs"; do
    M_NAME="${METRIC%%:*}"; KIND="${METRIC##*:}"
    curl -sS -G "https://monitoring.googleapis.com/v3/projects/${PROJECT}/timeSeries" \
      -H "Authorization: Bearer ${TOKEN}" \
      --data-urlencode "filter=metric.type=\"storage.googleapis.com/${M_NAME}\" AND resource.labels.bucket_name=\"${B}\"" \
      --data-urlencode "interval.startTime=${START}" \
      --data-urlencode "interval.endTime=${END}" \
      --data-urlencode "aggregation.alignmentPeriod=3600s" \
      --data-urlencode "aggregation.perSeriesAligner=ALIGN_SUM" \
      -o "/tmp/${B}.${KIND}.json"
  done
done

Follow nextPageToken if present; four days of hourly data can paginate. Then sum per hour across all returned series — there is usually more than one series per bucket, and taking only the first understates everything.

gcloud monitoring time-series list does not exist. Use the REST API as above.

Two traps

A cache-invalidation event ruins the window. Publishing a release changes every asset URL, and republishing the rate card changes its content hash. Either makes every returning shopper re-download once. Both happened on 26 August 2026, and the following morning showed finmatch-p bytes per request up 29% and finmatch-shared up 32% — not a regression, a one-off cost. Wait 24–48 hours after any invalidation before drawing conclusions.

Bytes per request is not comparable across the immutable change. Before, the request stream contained many cheap 304 revalidations that dragged the average down. Immutable headers remove those entirely, so bytes per request can rise while total cost falls. Watch the request count alongside it.

Results to date

Measured on 26 August 2026, 00:00–09:00 UTC, against the clean 23–24 August baseline, before the release rollout:

Bucket	Baseline	After	Change
finmatch-shared	12.49 GB	4.88 GB	−60.9%
finmatch-p	34.14 GB	8.30 GB	−75.7%
Combined	46.63 GB	13.18 GB	−71.7%

Extrapolating to a full day using the observed traffic curve puts the saving at roughly 130 GB/day, or about £350/month at GCS Premium Tier rates ($0.12/GB for the first TB, $0.11/GB to 10 TB). The percentages are measured; the daily and cash figures involve scaling ten quiet hours to twenty-four, so treat them as ±20%.

Browser-measured, on a live storefront:

• config objects: 387,675 bytes on a cold visit, 0 on every subsequent visit
• scripts and CSS on a release channel: 895,043 bytes cold, 0 thereafter, and 0 network round trips after the cache window expires, against 20 for a merchant not on a channel

The release rollout's contribution is not yet isolated; it completed at 18:20 on 26 August and the following day's window was contaminated by that invalidation.

Pricing that saving in GBP

The GB numbers above are from Cloud Monitoring. The £/GiB rate and the ongoing SKU trend live in the private repo guide — not on this public docs site, so invoice figures never get published to gs://finmatch-admin.

• Guide: docs/cost/billing-visibility.md on finmatch-shared (not under admin-docs/).
• Script: tools/billing-sku-report.sh.
• In Cloud Shell, from a clone of this repo:

cd ~/finmatch
git checkout finmatch-shared && git pull origin finmatch-shared
gcloud config set project finmatch-finance-mkt
bash tools/billing-sku-report.sh

The first run may print NO BILLING EXPORT TABLE FOUND. That is the one-time Billing → BigQuery export setup in the private guide. Until that is on, use Console → Billing → Reports → Group by SKU. Do not commit the printed £ figures.

Known follow-ups

No bulk assignment endpoint. Per-merchant opt-in was chosen so a merchant only ever moves because someone moved it. That property is worth keeping, but it left no safe bulk operation, so mass changes are done by repeating the unit — which is what runs into the rate limit. A PUT /admin/runtime-channels/:channel/merchants taking a list would be one read and one write.

Test-environment releases are not published. The release step is gated to finmatch-p, so there is nothing for a t merchant to point at. Serving test merchants from versioned releases needs the publishing step extended and a channel whose pointer names a finmatch-t release.

finmatch-p/configs/finmatch-rate-card.json has no writer. Last written 1 April 2026 and excluded from the sync-to-gcs config sync. It is still read as the runtime's primary rate card, with the shared catalogue topping it up. See ssot-consolidation-plan.md.