Compress embedded web assets with correct HTTP negotiation #82

Open
opened 2026-10-04 18:56:18 +00:00 by kayg · 11 comments
Owner

Problem and evidence

Cold containers JavaScript was 324,670 raw bytes / 117,684 gzip-equivalent bytes but was served with identity encoding. Host detail was 344,432 / 123,055 bytes. Across static assets, raw size was 2,291,200 bytes; JavaScript was 1,744,641 raw / 533,182 gzip bytes. Load1 was 25–28 during the browser capture. Current embedded responses move asset.data into a Vec and set content type/cache control, with no content-encoding negotiation or Vary header (crates/server/src/lib.rs:1002-1020).

Network evidence is in browser-cold-results.json and browser-cold-timeline.json.

Recommendation

Precompress embedded text assets at build time, negotiate accepted encodings, and implement correct Vary and ETag behavior. Serve shared immutable response bytes; avoid retaining a second full raw copy if the identity strategy permits it.

Payoff, effort and risk

Expected gain: reduce the containers response by 206,986 bytes (63.8%). Replacing rather than duplicating raw text storage could save about 1.39 MB in the release payload; measure the net binary size.
Quick win vs deeper change: build-time compression is estimated at ≤½ day; robust content negotiation, validators and cache tests are the deeper part, estimated at 1–2 days.
Risk: low to medium; incorrect Vary or ETag handling can serve the wrong representation or break conditional requests.

Priority: Now (≤1 day for the transfer win).

2026-10-04 owner decisions: Matches the one-binary/static-web product and no-Node-at-runtime decision. Generate compressed assets during the existing build. Related: #69 (keep the production web bundle in the shipped binary).

Report: perf-audit report
Source: perf-audit 2026-10-04

## Problem and evidence Cold containers JavaScript was 324,670 raw bytes / 117,684 gzip-equivalent bytes but was served with identity encoding. Host detail was 344,432 / 123,055 bytes. Across static assets, raw size was 2,291,200 bytes; JavaScript was 1,744,641 raw / 533,182 gzip bytes. Load1 was 25–28 during the browser capture. Current embedded responses move asset.data into a Vec and set content type/cache control, with no content-encoding negotiation or Vary header (crates/server/src/lib.rs:1002-1020). Network evidence is in [browser-cold-results.json](/mnt/hdd/infraternal/jobs/perf-audit/browser-cold-results.json) and [browser-cold-timeline.json](/mnt/hdd/infraternal/jobs/perf-audit/browser-cold-timeline.json). ## Recommendation Precompress embedded text assets at build time, negotiate accepted encodings, and implement correct Vary and ETag behavior. Serve shared immutable response bytes; avoid retaining a second full raw copy if the identity strategy permits it. ## Payoff, effort and risk **Expected gain:** reduce the containers response by 206,986 bytes (63.8%). Replacing rather than duplicating raw text storage could save about 1.39 MB in the release payload; measure the net binary size. **Quick win vs deeper change:** build-time compression is estimated at ≤½ day; robust content negotiation, validators and cache tests are the deeper part, estimated at 1–2 days. **Risk:** low to medium; incorrect Vary or ETag handling can serve the wrong representation or break conditional requests. **Priority:** Now (≤1 day for the transfer win). **2026-10-04 owner decisions:** Matches the one-binary/static-web product and no-Node-at-runtime decision. Generate compressed assets during the existing build. Related: #69 (keep the production web bundle in the shipped binary). **Report:** [perf-audit report](/mnt/hdd/infraternal/jobs/perf-audit/report.md) Source: perf-audit 2026-10-04
Author
Owner

Starting web-perf-now on branch web-perf-now, base head 069059d31fcd91366e76072e6be4871c01e5cde6. I am reading the three issue scopes and existing perf evidence, then will implement the precompressed asset serving, cold-route JS reductions, and normalized container search cache in this worktree.

Starting web-perf-now on branch `web-perf-now`, base head `069059d31fcd91366e76072e6be4871c01e5cde6`. I am reading the three issue scopes and existing perf evidence, then will implement the precompressed asset serving, cold-route JS reductions, and normalized container search cache in this worktree.
Author
Owner

#82 progress (branch web-perf-now, base 069059d)

The static build now emits gzip and Brotli variants plus per-representation SHA-256 ETags. The final browser run measured the /containers JavaScript payload at 98,959 Content-Length bytes across 30 responses (26 encoded), versus the 324,670-byte uncompressed baseline: 225,711 fewer payload bytes (69.5%). The route-wide gzip-equivalent JavaScript check is 111,793 bytes. The full browser gate still fails after its route captures on /api/v1/containers 503/401 responses under load; all cold-route measurements were emitted.

#82 progress (branch web-perf-now, base 069059d) The static build now emits gzip and Brotli variants plus per-representation SHA-256 ETags. The final browser run measured the /containers JavaScript payload at 98,959 Content-Length bytes across 30 responses (26 encoded), versus the 324,670-byte uncompressed baseline: 225,711 fewer payload bytes (69.5%). The route-wide gzip-equivalent JavaScript check is 111,793 bytes. The full browser gate still fails after its route captures on /api/v1/containers 503/401 responses under load; all cold-route measurements were emitted.
Author
Owner

#82 finished
Branch: web-perf-now
Base: 069059d31f
Head: 52fe3ae1da

Built 109 text assets into 102 gzip and 102 Brotli variants. Precompression reported 2,315,378 source bytes, 609,837 gzip-or-identity bytes, and 518,533 Brotli-or-identity bytes.

Before, cold /containers JavaScript was 324,670 raw bytes / 117,684 gzip-equivalent bytes and served with identity. After, the browser measured 98,959 Content-Length payload bytes across 30 JS responses (26 encoded), a 225,711-byte / 69.5% reduction from the raw baseline; route JavaScript is 111,793 gzip-equivalent bytes. Host detail moved from 344,432 raw / 123,055 gzip-equivalent to 102,762 transfer bytes / 116,431 gzip-equivalent. All cold-route measurements are in /mnt/hdd/infraternal/artifacts/verify/web-perf-now-browser-final/browser.

The required fast gate exited 0. Verbatim output:

PASS fast: migrations are append-only (0.278s)
PASS fast: workspace package dependencies (0.205s)
PASS fast: gate target-directory and launch diagnostics tests (0.291s)
PASS fast: JavaScript dependencies (0.004s)
PASS fast: advisory review policy tests (0.195s)
PASS fast: supply-chain checks (5.837s)
PASS fast: build web app (14.103s)
PASS fast: cargo fmt --check (1.425s)
PASS fast: cargo clippy (21.444s)
PASS fast: cargo test (180.428s)
PASS fast: svelte-check (19.463s)
PASS fast: UI package svelte-check (7.484s)
PASS fast: stream package typecheck (7.103s)
PASS fast: Vitest (4.179s)
PASS fast: UI package Vitest (3.852s)
PASS fast: stream package Vitest (1.348s)
PASS fast: OpenAPI and TypeScript client drift (44.707s)
PASS fast: terminal TypeScript (2.703s)
PASS fast: terminal Vitest (1.457s)

The separate browser gate recorded all cold-route budget metrics, then failed its page-clean assertion: FAIL browser: Playwright against release server (324.222s). It reported /api/v1/containers 503/401 console errors while VM load was above the 8-CPU target. I did not weaken that assertion.

#82 finished Branch: web-perf-now Base: 069059d31fcd91366e76072e6be4871c01e5cde6 Head: 52fe3ae1dac77f1294e473d9d647cb93f40624a6 Built 109 text assets into 102 gzip and 102 Brotli variants. Precompression reported 2,315,378 source bytes, 609,837 gzip-or-identity bytes, and 518,533 Brotli-or-identity bytes. Before, cold /containers JavaScript was 324,670 raw bytes / 117,684 gzip-equivalent bytes and served with identity. After, the browser measured 98,959 Content-Length payload bytes across 30 JS responses (26 encoded), a 225,711-byte / 69.5% reduction from the raw baseline; route JavaScript is 111,793 gzip-equivalent bytes. Host detail moved from 344,432 raw / 123,055 gzip-equivalent to 102,762 transfer bytes / 116,431 gzip-equivalent. All cold-route measurements are in /mnt/hdd/infraternal/artifacts/verify/web-perf-now-browser-final/browser. The required fast gate exited 0. Verbatim output: ``` PASS fast: migrations are append-only (0.278s) PASS fast: workspace package dependencies (0.205s) PASS fast: gate target-directory and launch diagnostics tests (0.291s) PASS fast: JavaScript dependencies (0.004s) PASS fast: advisory review policy tests (0.195s) PASS fast: supply-chain checks (5.837s) PASS fast: build web app (14.103s) PASS fast: cargo fmt --check (1.425s) PASS fast: cargo clippy (21.444s) PASS fast: cargo test (180.428s) PASS fast: svelte-check (19.463s) PASS fast: UI package svelte-check (7.484s) PASS fast: stream package typecheck (7.103s) PASS fast: Vitest (4.179s) PASS fast: UI package Vitest (3.852s) PASS fast: stream package Vitest (1.348s) PASS fast: OpenAPI and TypeScript client drift (44.707s) PASS fast: terminal TypeScript (2.703s) PASS fast: terminal Vitest (1.457s) ``` The separate browser gate recorded all cold-route budget metrics, then failed its page-clean assertion: `FAIL browser: Playwright against release server (324.222s)`. It reported /api/v1/containers 503/401 console errors while VM load was above the 8-CPU target. I did not weaken that assertion.
Author
Owner

Correction: I retried scripts/verify browser against committed head 52fe3ae1da after the other active browser job ended. The retry again emitted the passing cold-route budgets and transfer metrics, then reported FAIL browser: Playwright against release server (318.534s) at the light/desktop page-clean assertion, with /api/v1/containers 503/401 console errors. The cause is not established; the previous note attributing it to VM load alone was too strong. The required fast gate remains green (verbatim output in the finish comment above).

Correction: I retried `scripts/verify browser` against committed head 52fe3ae1dac77f1294e473d9d647cb93f40624a6 after the other active browser job ended. The retry again emitted the passing cold-route budgets and transfer metrics, then reported `FAIL browser: Playwright against release server (318.534s)` at the light/desktop page-clean assertion, with /api/v1/containers 503/401 console errors. The cause is not established; the previous note attributing it to VM load alone was too strong. The required `fast` gate remains green (verbatim output in the finish comment above).
Author
Owner

Starting fix-static-check (#82) on branch browser-green, base head a10c3e43a86a2cc2d6a6fca86afc225c906ff390.

I’ll trace integration_http_checks, reproduce the failure against /mnt/hdd/infraternal/gate-target/browser-green2/release/infraternal, fix the product or check based on the failing assertion, then run the requested fast tier. I will not run merge.

Starting fix-static-check (#82) on branch `browser-green`, base head `a10c3e43a86a2cc2d6a6fca86afc225c906ff390`. I’ll trace `integration_http_checks`, reproduce the failure against `/mnt/hdd/infraternal/gate-target/browser-green2/release/infraternal`, fix the product or check based on the failing assertion, then run the requested `fast` tier. I will not run `merge`.
Author
Owner

Finding from the isolated integration_http_checks trace: it returns 1 at the root-page assertion grep -F 'data-sveltekit-preload-data="hover"' "$index_file". The supplied release server is ready and the health/OpenAPI checks pass. select_content_coding chooses Brotli when Accept-Encoding is absent, so the plain curl used for / writes compressed bytes to index.html; the assertion then cannot find the HTML marker. I’m changing the absent-header default to identity and adding coverage for that request case.

Finding from the isolated `integration_http_checks` trace: it returns 1 at the root-page assertion `grep -F 'data-sveltekit-preload-data="hover"' "$index_file"`. The supplied release server is ready and the health/OpenAPI checks pass. `select_content_coding` chooses Brotli when `Accept-Encoding` is absent, so the plain `curl` used for `/` writes compressed bytes to `index.html`; the assertion then cannot find the HTML marker. I’m changing the absent-header default to identity and adding coverage for that request case.
Author
Owner

The expanded rerun reached only the new compressed-representation assertion before returning 1. The first HTML-linked asset, /_app/immutable/entry/start.zjOP-vs_.js, has no .br representation because precompression intentionally omits a variant when it would not reduce bytes. Other linked assets do have .br files. I’m changing the check to select an HTML-linked JS/CSS file whose generated .br variant exists, preserving the production behavior and making the assertion match the build rule.

The expanded rerun reached only the new compressed-representation assertion before returning 1. The first HTML-linked asset, `/_app/immutable/entry/start.zjOP-vs_.js`, has no `.br` representation because precompression intentionally omits a variant when it would not reduce bytes. Other linked assets do have `.br` files. I’m changing the check to select an HTML-linked JS/CSS file whose generated `.br` variant exists, preserving the production behavior and making the assertion match the build rule.
Author
Owner

Finished fix-static-check (#82) on browser-green.

Head SHA: 92a04fb3913940a25f924e4c32dd8921f342c0ba

The failing assertion was the root-page grep for data-sveltekit-preload-data="hover". With no Accept-Encoding, select_content_coding selected Brotli, so plain curl wrote compressed bytes and the HTML assertion failed. Missing Accept-Encoding now selects identity; explicit Brotli and gzip negotiation is preserved. The server unit test covers unnegotiated HTML bytes and immutable asset headers across identity, Brotli, and gzip. The HTTP check compares identity bodies, requires the SPA fallback to match /, and verifies Content-Encoding, Vary, cache policy and content type for identity, Brotli and gzip.

Verification:

  • Isolated HTTP step against /mnt/hdd/infraternal/gate-target/browser-green2/release/infraternal: PASS isolated release server HTTP and static behavior
  • Targeted regression test: cargo test: 1 passed, 367 filtered out (15 suites, 0.00s)
  • Final INFRATERNAL_VERIFY_RUN_ID=fix-static-check scripts/verify fast stage output (verbatim):
PASS fast: migrations are append-only (0.139s)
PASS fast: workspace package dependencies (0.198s)
PASS fast: gate target-directory and launch diagnostics tests (0.335s)
PASS fast: JavaScript dependencies (0.006s)
PASS fast: advisory review policy tests (0.193s)
PASS fast: supply-chain checks (3.740s)
PASS fast: build web app (9.768s)
PASS fast: cargo fmt --check (1.254s)
PASS fast: cargo clippy (12.785s)
PASS fast: cargo test (145.936s)
PASS fast: svelte-check (9.153s)
PASS fast: UI package svelte-check (3.747s)
PASS fast: stream package typecheck (4.738s)
PASS fast: Vitest (3.583s)
PASS fast: UI package Vitest (2.903s)
PASS fast: stream package Vitest (0.816s)
PASS fast: OpenAPI and TypeScript client drift (8.419s)
PASS fast: terminal TypeScript (1.926s)
PASS fast: terminal Vitest (1.087s)

An earlier full workspace test attempt hit an unrelated provider-bollard test once; it passed in isolation and all subsequent complete fast runs passed. No merge gate was run.

Finished fix-static-check (#82) on `browser-green`. Head SHA: `92a04fb3913940a25f924e4c32dd8921f342c0ba` The failing assertion was the root-page `grep` for `data-sveltekit-preload-data="hover"`. With no `Accept-Encoding`, `select_content_coding` selected Brotli, so plain curl wrote compressed bytes and the HTML assertion failed. Missing `Accept-Encoding` now selects identity; explicit Brotli and gzip negotiation is preserved. The server unit test covers unnegotiated HTML bytes and immutable asset headers across identity, Brotli, and gzip. The HTTP check compares identity bodies, requires the SPA fallback to match `/`, and verifies `Content-Encoding`, `Vary`, cache policy and content type for identity, Brotli and gzip. Verification: - Isolated HTTP step against `/mnt/hdd/infraternal/gate-target/browser-green2/release/infraternal`: `PASS isolated release server HTTP and static behavior` - Targeted regression test: `cargo test: 1 passed, 367 filtered out (15 suites, 0.00s)` - Final `INFRATERNAL_VERIFY_RUN_ID=fix-static-check scripts/verify fast` stage output (verbatim): ```text PASS fast: migrations are append-only (0.139s) PASS fast: workspace package dependencies (0.198s) PASS fast: gate target-directory and launch diagnostics tests (0.335s) PASS fast: JavaScript dependencies (0.006s) PASS fast: advisory review policy tests (0.193s) PASS fast: supply-chain checks (3.740s) PASS fast: build web app (9.768s) PASS fast: cargo fmt --check (1.254s) PASS fast: cargo clippy (12.785s) PASS fast: cargo test (145.936s) PASS fast: svelte-check (9.153s) PASS fast: UI package svelte-check (3.747s) PASS fast: stream package typecheck (4.738s) PASS fast: Vitest (3.583s) PASS fast: UI package Vitest (2.903s) PASS fast: stream package Vitest (0.816s) PASS fast: OpenAPI and TypeScript client drift (8.419s) PASS fast: terminal TypeScript (1.926s) PASS fast: terminal Vitest (1.087s) ``` An earlier full workspace test attempt hit an unrelated `provider-bollard` test once; it passed in isolation and all subsequent complete `fast` runs passed. No merge gate was run.
Author
Owner

Started fix-deploy-node on branch fix-deploy-node at base head 851d502909. Scope: make scripts/build-web work with the repository's existing JS runtime, inspect deploy/build paths, and run the requested build-only and fast gates. No production deploy will be run.

Started fix-deploy-node on branch fix-deploy-node at base head 851d50290998f1328b7e5557f9084a5d7d0dff07. Scope: make scripts/build-web work with the repository's existing JS runtime, inspect deploy/build paths, and run the requested build-only and fast gates. No production deploy will be run.
Author
Owner

Finding: deploy/o2/build.sh pins Bun 1.4.2 and puts it on PATH, then calls scripts/build-web; that script's final precompression step invoked node, which the disposable Ubuntu build image never installs. I switched that invocation to Bun and documented the shared pinned runtime in docs/DEPLOY-MONITOR01.md. Bun's current node:zlib reference marks the module fully supported and exposes both gzipSync and brotliCompressSync: https://bun.sh/reference/node/zlib

Path check: scripts/deploy-o2 still exists and uses the same deploy/o2/build.sh; scripts/verify fast prepares the bundle through scripts/build-web; scripts/preview also shares it. No tracked CI/workflow files were present.

Local output evidence: scripts/build-web generated 109 .gz, 109 .br, and 348 ETag metadata files, including _metadata/index.html.identity.etag. A byte check found identity assets and Brotli variants identical to Node output. Bun's gzip bytes differ from Node for 47 variants (and their 47 gzip ETags); the generated ETags match the actual Bun bytes. A decode/hash check verified all 109 gzip and 109 Brotli variants reproduce their source bytes and have matching ETags.

Finding: `deploy/o2/build.sh` pins Bun 1.4.2 and puts it on PATH, then calls `scripts/build-web`; that script's final precompression step invoked `node`, which the disposable Ubuntu build image never installs. I switched that invocation to Bun and documented the shared pinned runtime in `docs/DEPLOY-MONITOR01.md`. Bun's current `node:zlib` reference marks the module fully supported and exposes both `gzipSync` and `brotliCompressSync`: https://bun.sh/reference/node/zlib Path check: `scripts/deploy-o2` still exists and uses the same `deploy/o2/build.sh`; `scripts/verify fast` prepares the bundle through `scripts/build-web`; `scripts/preview` also shares it. No tracked CI/workflow files were present. Local output evidence: `scripts/build-web` generated 109 `.gz`, 109 `.br`, and 348 ETag metadata files, including `_metadata/index.html.identity.etag`. A byte check found identity assets and Brotli variants identical to Node output. Bun's gzip bytes differ from Node for 47 variants (and their 47 gzip ETags); the generated ETags match the actual Bun bytes. A decode/hash check verified all 109 gzip and 109 Brotli variants reproduce their source bytes and have matching ETags.
Author
Owner

Finished fix-deploy-node. Branch: fix-deploy-node; base: 851d50290998f1328b7e5557f9084a5d7d0dff07; head: 5af6c2797a7a955baf06eb5884ba4c8dbac0edb8. Commit: fix(deploy): use Bun for web precompression.

The web build now invokes scripts/precompress-web.mjs with the pinned Bun runtime already installed by deploy/o2/build.sh. docs/DEPLOY-MONITOR01.md records that shared build choice. scripts/deploy-o2 is still present and uses this same builder; scripts/verify fast and scripts/preview call the shared scripts/build-web. No tracked CI/workflow files were present.

rtk scripts/build-web passed locally. scripts/deploy-monitor01 --build-only --ref HEAD passed for x86_64 and aarch64; both builds emitted 109 gzip and 109 Brotli variants and completed bundle checksums and local OCI image validation. The packaged image HTTP smoke checked identity, Brotli, and gzip bodies against each other and verified each response ETag against its embedded bytes:

PASS packaged image serves identity, br and gzip with matching embedded ETags

Fast gate exited 0. Its stage output:

PASS fast: migrations are append-only (0.137s)
PASS fast: workspace package dependencies (0.189s)
PASS fast: gate target-directory and launch diagnostics tests (0.278s)
PASS fast: JavaScript dependencies (0.004s)
PASS fast: advisory review policy tests (0.185s)
PASS fast: supply-chain checks (3.919s)
PASS fast: build web app (9.387s)
PASS fast: cargo fmt --check (1.295s)
PASS fast: cargo clippy (124.010s)
PASS fast: cargo test (204.988s)
PASS fast: svelte-check (9.667s)
PASS fast: UI package svelte-check (3.760s)
PASS fast: stream package typecheck (3.803s)
PASS fast: Vitest (2.992s)
PASS fast: UI package Vitest (3.123s)
PASS fast: stream package Vitest (0.809s)
PASS fast: OpenAPI and TypeScript client drift (47.875s)
PASS fast: terminal TypeScript (2.186s)
PASS fast: terminal Vitest (1.220s)

Build-only output:

PASS build-only 5af6c2797a7a955baf06eb5884ba4c8dbac0edb8 (localhost/infraternal:monitor01-5af6c2797a7a955baf06eb5884ba4c8dbac0edb8)

The fast gate's Bun audit step reported one low cookie@0.6.0 advisory as report-only; it did not fail the gate. No production deployment was run.

Finished fix-deploy-node. Branch: `fix-deploy-node`; base: `851d50290998f1328b7e5557f9084a5d7d0dff07`; head: `5af6c2797a7a955baf06eb5884ba4c8dbac0edb8`. Commit: `fix(deploy): use Bun for web precompression`. The web build now invokes `scripts/precompress-web.mjs` with the pinned Bun runtime already installed by `deploy/o2/build.sh`. `docs/DEPLOY-MONITOR01.md` records that shared build choice. `scripts/deploy-o2` is still present and uses this same builder; `scripts/verify fast` and `scripts/preview` call the shared `scripts/build-web`. No tracked CI/workflow files were present. `rtk scripts/build-web` passed locally. `scripts/deploy-monitor01 --build-only --ref HEAD` passed for x86_64 and aarch64; both builds emitted 109 gzip and 109 Brotli variants and completed bundle checksums and local OCI image validation. The packaged image HTTP smoke checked identity, Brotli, and gzip bodies against each other and verified each response ETag against its embedded bytes: ```text PASS packaged image serves identity, br and gzip with matching embedded ETags ``` Fast gate exited 0. Its stage output: ```text PASS fast: migrations are append-only (0.137s) PASS fast: workspace package dependencies (0.189s) PASS fast: gate target-directory and launch diagnostics tests (0.278s) PASS fast: JavaScript dependencies (0.004s) PASS fast: advisory review policy tests (0.185s) PASS fast: supply-chain checks (3.919s) PASS fast: build web app (9.387s) PASS fast: cargo fmt --check (1.295s) PASS fast: cargo clippy (124.010s) PASS fast: cargo test (204.988s) PASS fast: svelte-check (9.667s) PASS fast: UI package svelte-check (3.760s) PASS fast: stream package typecheck (3.803s) PASS fast: Vitest (2.992s) PASS fast: UI package Vitest (3.123s) PASS fast: stream package Vitest (0.809s) PASS fast: OpenAPI and TypeScript client drift (47.875s) PASS fast: terminal TypeScript (2.186s) PASS fast: terminal Vitest (1.220s) ``` Build-only output: ```text PASS build-only 5af6c2797a7a955baf06eb5884ba4c8dbac0edb8 (localhost/infraternal:monitor01-5af6c2797a7a955baf06eb5884ba4c8dbac0edb8) ``` The fast gate's Bun audit step reported one low `cookie@0.6.0` advisory as report-only; it did not fail the gate. No production deployment was run.
Commenting is not possible because the repository is archived.
No labels
No milestone
No project
No assignees
1 participant
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/infraternal-v0#82
No description provided.