Startup reconciliation overflows the default debug thread stack during Photo upload burst

kayg commented

2026-10-04 11:19:21 +00:00

Owner

During #1051 upload acknowledgement verification on branch head 7d529de84, the debug server built with OPENSSL_NO_VENDOR=1 aborted during the first 120-Photo burst after startup reconciliation began. This is not SLOW and is not classified as an upload-intent failure.

Server output:

thread 'calternal-startup-reconcile' (2294622) has overflowed its stack
fatal runtime error: stack overflow, aborting

The thread at crates/calternal-server/src/wire.rs uses std:🧵:Builder without stack_size; main sets 4 MiB only on the Tokio worker runtime. Run ADVERSARIAL_UPLOAD500_ONLY=1 ADVERSARIAL_KEEP_WORK_DIR=1 ADVERSARIAL_SKIP_WEB_BUILD=1 ADVERSARIAL_SERVER_BIN=<this branch debug binary> UPLOAD500_REPEATS=30 UPLOAD500_WORKERS=8 bash tests/adversarial/run.sh with default RUST_MIN_STACK.

Artifacts are retained in the upload500-1051 worktree: artifacts/upload500-fixed-30.log, artifacts/startup-stack-crash.log, target/tmp/adversarial.aWMxV4. Only 26 PATCH acknowledgements succeeded before server loss in iteration 1. The remaining responses were proxy 502. Readiness stays no. #1051 will continue its focused check with an explicit RUST_MIN_STACK override and retain the default-stack failure.

During #1051 upload acknowledgement verification on branch head `7d529de84`, the debug server built with OPENSSL_NO_VENDOR=1 aborted during the first 120-Photo burst after startup reconciliation began. This is not SLOW and is not classified as an upload-intent failure. Server output: ``` thread 'calternal-startup-reconcile' (2294622) has overflowed its stack fatal runtime error: stack overflow, aborting ``` The thread at `crates/calternal-server/src/wire.rs` uses std::thread::Builder without stack_size; main sets 4 MiB only on the Tokio worker runtime. Run `ADVERSARIAL_UPLOAD500_ONLY=1 ADVERSARIAL_KEEP_WORK_DIR=1 ADVERSARIAL_SKIP_WEB_BUILD=1 ADVERSARIAL_SERVER_BIN=<this branch debug binary> UPLOAD500_REPEATS=30 UPLOAD500_WORKERS=8 bash tests/adversarial/run.sh` with default RUST_MIN_STACK. Artifacts are retained in the upload500-1051 worktree: `artifacts/upload500-fixed-30.log`, `artifacts/startup-stack-crash.log`, `target/tmp/adversarial.aWMxV4`. Only 26 PATCH acknowledgements succeeded before server loss in iteration 1. The remaining responses were proxy 502. Readiness stays no. #1051 will continue its focused check with an explicit RUST_MIN_STACK override and retain the default-stack failure.

kayg referenced this issue

2026-10-04 11:19:47 +00:00

Uploads: Photo burst returned an intermittent 500 despite a complete Calendar projection #1051

kayg referenced this issue

2026-10-04 11:19:48 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 11:28:03 +00:00

Uploads: Photo burst returned an intermittent 500 despite a complete Calendar projection #1051

kayg referenced this issue

2026-10-04 11:39:49 +00:00

Uploads: Photo burst returned an intermittent 500 despite a complete Calendar projection #1051

kayg referenced this issue

2026-10-04 11:39:50 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg commented

2026-10-04 11:42:09 +00:00

Author

Owner

Starting #1054 on job/stack-1054 at base c39ffe5d90126527d7aacf2d8b79507929c80616 (the issue's requested dev base). I am tracing the startup reconciliation stack use and will run the debug/release burst plus deep-tree reproductions before fixing it.

Starting #1054 on `job/stack-1054` at base `c39ffe5d90126527d7aacf2d8b79507929c80616` (the issue's requested `dev` base). I am tracing the startup reconciliation stack use and will run the debug/release burst plus deep-tree reproductions before fixing it.

kayg commented

2026-10-04 12:02:33 +00:00

Author

Owner

Finding from source audit: build_live_app created calternal-startup-reconcile with std::thread::Builder::spawn and no stack size. Its current-thread Tokio runtime polls one combined async startup future. The 4 MiB SERVER_WORKER_STACK_SIZE in main.rs applies to the separate multi-thread server runtime. calternal-search::Indexer::open_inner also starts a scan/index actor with an unsized std::thread::Builder and block_on(actor.run(...)). The Files, Notes, Tags, Search and semantic directory walks I inspected use explicit work stacks and bounded depth, so directory recursion is not the observed path. I am measuring the default-stack burst and a 200-folder startup scan next.

Finding from source audit: `build_live_app` created `calternal-startup-reconcile` with `std::thread::Builder::spawn` and no stack size. Its current-thread Tokio runtime polls one combined async startup future. The 4 MiB `SERVER_WORKER_STACK_SIZE` in `main.rs` applies to the separate multi-thread server runtime. `calternal-search::Indexer::open_inner` also starts a scan/index actor with an unsized `std::thread::Builder` and `block_on(actor.run(...))`. The Files, Notes, Tags, Search and semantic directory walks I inspected use explicit work stacks and bounded depth, so directory recursion is not the observed path. I am measuring the default-stack burst and a 200-folder startup scan next.

kayg commented

2026-10-04 12:12:40 +00:00

Author

Owner

The 30-iteration default-stack debug probe finished on the requested production base c39ffe5d90126527d7aacf2d8b79507929c80616: 3,600/3,600 uploads acknowledged, zero failed iterations, and all Calendar pages returned 200. The crash did not reproduce on this base. The issue's retained crash evidence is from 7d529de84, which contains the later startup path. I am keeping the repair because both startup and Search indexer use dedicated unsized OS threads, and the startup future is polled by a current-thread runtime there.

The 30-iteration default-stack debug probe finished on the requested production base `c39ffe5d90126527d7aacf2d8b79507929c80616`: 3,600/3,600 uploads acknowledged, zero failed iterations, and all Calendar pages returned 200. The crash did not reproduce on this base. The issue's retained crash evidence is from `7d529de84`, which contains the later startup path. I am keeping the repair because both startup and Search indexer use dedicated unsized OS threads, and the startup future is polled by a current-thread runtime there.

kayg commented

2026-10-04 12:25:18 +00:00

Author

Owner

Regression-harness finding: the first draft ran four successful 120-Photo rounds on one server process, so only one startup worker was exercised. I changed the focused runner to restart the server before each burst and am rerunning the requested ten rounds on the debug build with RUST_MIN_STACK unset.

Regression-harness finding: the first draft ran four successful 120-Photo rounds on one server process, so only one startup worker was exercised. I changed the focused runner to restart the server before each burst and am rerunning the requested ten rounds on the debug build with `RUST_MIN_STACK` unset.

kayg commented

2026-10-04 12:42:22 +00:00

Author

Owner

The corrected debug regression probe passed on ten fresh server starts with RUST_MIN_STACK unset. Each round uploaded 120 Photos, returned Calendar HTTP 200, exposed all 120 items, and reported zero duplicate paths; total: 1,200 uploads and zero failed rounds. The 200-level startup scan is also included in the live-app test runner at wire::tests::startup_reconciliation_indexes_photos_written_while_stopped and will run under the calternal-server test gate.

The corrected debug regression probe passed on ten fresh server starts with `RUST_MIN_STACK` unset. Each round uploaded 120 Photos, returned Calendar HTTP 200, exposed all 120 items, and reported zero duplicate paths; total: 1,200 uploads and zero failed rounds. The 200-level startup scan is also included in the live-app test runner at `wire::tests::startup_reconciliation_indexes_photos_written_while_stopped` and will run under the `calternal-server` test gate.

kayg commented

2026-10-04 13:34:21 +00:00

Author

Owner

#1054 final report

Built

The startup worker was a raw std::thread with Rust's default 2 MiB stack. It used block_on to poll the combined startup reconciliation future. The Tokio worker stack setting does not apply to that thread. Source inspection found no directory-depth recursion or large stack array in the startup scans: Search and filesystem walks use explicit task stacks and bounded depth. The stack pressure came from polling the nested async startup work on the small native stack, with debug builds using more stack than release builds.

I boxed the startup and CAS scrub futures and set documented 8 MiB stacks on the startup reconcile, Search indexer, CAS scrub and User Home purge threads. calternal-fs has no production std::thread::Builder for reconcile, scan or index work, and its tree walks are already iterative. The server test adds a 200-folder tagged Note and a Photo, then checks Files, Tags and Photos startup indexes on an explicit 2 MiB thread. The adversarial runner restarts the real server before every burst, so each round exercises a fresh startup thread.

Files: crates/calternal-server/src/wire.rs, crates/calternal-search/src/indexer.rs, tests/adversarial/run.sh, tests/adversarial/setup.mjs, tests/adversarial/startup_stack_1054.py.

Verification

The repaired debug reproduction ran with RUST_MIN_STACK unset: 10 fresh server starts, 1,200 uploads, zero failed rounds. Each round reported 120 visible Photos, zero upload errors, zero duplicate paths and Calendar HTTP 200. The release reproduction passed once with the same checks. The 200-folder startup test passed inside wire::tests::live_apps_run_in_separate_processes.

Gate output:

$ cargo fmt --check
(no output; exit 0)

$ cargo clippy -p calternal-server --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 34s

$ cargo test -p calternal-server
test wire::tests::live_apps_run_in_separate_processes ... ok
test result: ok. 163 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 23.14s

$ cargo clippy -p calternal-fs --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 39.64s

$ cargo test -p calternal-fs
test result: ok. 59 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 19.35s
test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.51s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

$ cargo clippy -p calternal-search --all-targets -- -D warnings
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 52s

$ cargo test -p calternal-search
test result: ok. 41 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 18.39s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.00s
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s
test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 359.62s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s
test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s
test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 4.02s
test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s

$ cargo build --release -p calternal-server
    Finished `release` profile [optimized] target(s) in 20m 24s

The web production build completed before the server build. cargo clean removed 24,312 files (12.4 GiB), and I deleted the web build output.

Known gaps

The pre-fix production-base debug run completed 30 bursts (3,600 uploads) without reproducing the historical crash. The issue reports that crash on 7d529de84; the repaired debug and release probes passed.

Decisions

The design document does not set detached filesystem-thread stack sizes. I chose 8 MiB to give debug-build reconciliation futures headroom while keeping each worker's stack bounded. The regression test uses an explicit 2 MiB stack to cover the reported default-stack condition.

Branch: job/stack-1054
Head: 64f3ad1a0bc1d1bb5b9fbff0d64d137b0d005866

READY FOR MERGE: yes

#1054 final report ## Built The startup worker was a raw `std::thread` with Rust's default 2 MiB stack. It used `block_on` to poll the combined startup reconciliation future. The Tokio worker stack setting does not apply to that thread. Source inspection found no directory-depth recursion or large stack array in the startup scans: Search and filesystem walks use explicit task stacks and bounded depth. The stack pressure came from polling the nested async startup work on the small native stack, with debug builds using more stack than release builds. I boxed the startup and CAS scrub futures and set documented 8 MiB stacks on the startup reconcile, Search indexer, CAS scrub and User Home purge threads. `calternal-fs` has no production `std::thread::Builder` for reconcile, scan or index work, and its tree walks are already iterative. The server test adds a 200-folder tagged Note and a Photo, then checks Files, Tags and Photos startup indexes on an explicit 2 MiB thread. The adversarial runner restarts the real server before every burst, so each round exercises a fresh startup thread. Files: `crates/calternal-server/src/wire.rs`, `crates/calternal-search/src/indexer.rs`, `tests/adversarial/run.sh`, `tests/adversarial/setup.mjs`, `tests/adversarial/startup_stack_1054.py`. ## Verification The repaired debug reproduction ran with `RUST_MIN_STACK` unset: 10 fresh server starts, 1,200 uploads, zero failed rounds. Each round reported 120 visible Photos, zero upload errors, zero duplicate paths and Calendar HTTP 200. The release reproduction passed once with the same checks. The 200-folder startup test passed inside `wire::tests::live_apps_run_in_separate_processes`. Gate output: ```text $ cargo fmt --check (no output; exit 0) $ cargo clippy -p calternal-server --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 6m 34s $ cargo test -p calternal-server test wire::tests::live_apps_run_in_separate_processes ... ok test result: ok. 163 passed; 0 failed; 6 ignored; 0 measured; 0 filtered out; finished in 23.14s $ cargo clippy -p calternal-fs --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 39.64s $ cargo test -p calternal-fs test result: ok. 59 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 19.35s test result: ok. 44 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 5.51s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s $ cargo clippy -p calternal-search --all-targets -- -D warnings Finished `dev` profile [unoptimized + debuginfo] target(s) in 1m 52s $ cargo test -p calternal-search test result: ok. 41 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 18.39s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 1.00s test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.06s test result: ok. 24 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 359.62s test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.05s test result: ok. 2 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.04s test result: ok. 1 passed; 0 failed; 2 ignored; 0 measured; 0 filtered out; finished in 4.02s test result: ok. 4 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s $ cargo build --release -p calternal-server Finished `release` profile [optimized] target(s) in 20m 24s ``` The web production build completed before the server build. `cargo clean` removed 24,312 files (12.4 GiB), and I deleted the web build output. ## Known gaps The pre-fix production-base debug run completed 30 bursts (3,600 uploads) without reproducing the historical crash. The issue reports that crash on `7d529de84`; the repaired debug and release probes passed. ## Decisions The design document does not set detached filesystem-thread stack sizes. I chose 8 MiB to give debug-build reconciliation futures headroom while keeping each worker's stack bounded. The regression test uses an explicit 2 MiB stack to cover the reported default-stack condition. Branch: `job/stack-1054` Head: `64f3ad1a0bc1d1bb5b9fbff0d64d137b0d005866` READY FOR MERGE: yes

kayg referenced this issue

2026-10-04 18:55:26 +00:00

Plan: merge order and conflicts for merge round 7b #867

kayg referenced this issue

2026-10-04 18:55:27 +00:00

SHARING: one Share dialog everywhere + share Notes, Canvases and folders #1034

kayg referenced this issue

2026-10-04 18:55:28 +00:00

SHARING: invite links for people without an account #1035

kayg closed this issue

2026-10-04 18:55:32 +00:00