fuzz: ssh_auth_once_decode found a failing input

kayg commented

2026-11-05 22:21:44 +00:00

Owner

Nightly parser fuzzing (D351, #344).

Target: ssh_auth_once_decode
Source: 521cb8990b720ce751466d2eb81e5881b496bfda
Input SHA-256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855

Input (base64): ``

Reproduce with cargo +nightly fuzz run ssh_auth_once_decode <input>, after unzipping the input attached to the report comment.

Nightly parser fuzzing (D351, #344). Target: `ssh_auth_once_decode` Source: `521cb8990b720ce751466d2eb81e5881b496bfda` Input SHA-256: `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855` Input (base64): `` Reproduce with `cargo +nightly fuzz run ssh_auth_once_decode <input>`, after unzipping the input attached to the report comment.

kayg commented

2026-11-05 22:21:44 +00:00

Author

Owner

Failing input for ssh_auth_once_decode from source 521cb8990b720ce751466d2eb81e5881b496bfda, SHA-256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.

Input (base64): ``

Failing input for `ssh_auth_once_decode` from source `521cb8990b720ce751466d2eb81e5881b496bfda`, SHA-256 `e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855`. Input (base64): ``

ssh_auth_once_decode-e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.zip

270 B

kayg commented

2026-11-05 22:32:23 +00:00

Author

Owner

J264 starting on branch fuzz-fixes-j264, base 159edcedc9978f0370ea8cb02a937082d631c9b6 (origin/dev). Plan: reproduce every reported input on a build VM, add failing regression coverage, fix the demonstrated cause, run each target for 600 seconds, run scripts/verify quick through build-slot in the foreground, and push one atomic commit per issue. Estimate: 90 minutes including compilation/slot queues; no merge, deploy, or issue closure.

J264 starting on branch `fuzz-fixes-j264`, base `159edcedc9978f0370ea8cb02a937082d631c9b6` (origin/dev). Plan: reproduce every reported input on a build VM, add failing regression coverage, fix the demonstrated cause, run each target for 600 seconds, run `scripts/verify quick` through build-slot in the foreground, and push one atomic commit per issue. Estimate: 90 minutes including compilation/slot queues; no merge, deploy, or issue closure.

kayg commented

2026-11-05 22:57:06 +00:00

Author

Owner

J264b (taking over from J264): finding. The decoder is fine. The runner filed an environmental timeout as a parser finding.

What the nightly saw (/var/lib/infraternal-fuzz/.local/state/infraternal-fuzz/ssh_auth_once_decode.log on dev-02, run of 521cb8990, 22:21 VM time):

INFO:     1794 files found in .../corpus/ssh_auth_once_decode
ALARM: working on the last Unit for 10 seconds
MS: 0 ; base unit: 0000000000000000000000000000000000000000
... Test unit written to .../artifacts/ssh_auth_once_decode/timeout-da39a3ee5e6b4b0d3255bfef95601890afd80709
==1979244== ERROR: libFuzzer: timeout after 10 seconds
SUMMARY: libFuzzer: timeout

The artifact is a 0-byte timeout-*, so it's a timeout, not a panic. libFuzzer always runs one empty input first in ReadAndExecuteSeedCorpora, before the seed corpus and before INITED, and the 10 s alarm fired during that first call. The stack is stripped and doesn't symbolize.

The decoder on empty input: I ran the nightly's own binary (same build, copied out read-only) on dev-02:

ssh_auth_once_decode -timeout=10 <empty file>: Executed ... in 0 ms, rc=0, three times out of three
-runs=200000 on the empty input: Done 200000 runs in 6 second(s)

serde_json::from_slice returns Err at once for both OneUseAuth and TargetAuth, so no assertion is reachable. During the replay dev-02 was at load 25–32, with /proc/pressure/io at full avg10=86–95% (IO-starved). The nightly ran on the same host and filed the same input under the same conditions. A 10 s stall in the first call is environmental: page-ins of the 88 MB ASan binary while the disk is saturated.

What is wrong: scripts/nightly-fuzz.py files every artifact as a parser bug, including timeout-* artifacts. Unlike panics, those depend on host load.

Planned fix (on fuzz-fixes-j264):

Before filing a timeout-* artifact, the runner runs it once more, alone, with the same -timeout.
It files the input only if it times out again.
Otherwise it moves the input to unconfirmed/<target>/ and logs it, so it isn't re-checked every night.
Crash artifacts are unchanged.
The regression is a fixture test in scripts/test-nightly-fuzz.py (gated by verify's "nightly fuzz reporter" row), which fails before the fix.

I'm not filing a decoder change, because none is needed. Replay log: /tmp/j264b-370-repro.txt on the control host.

J264b (taking over from J264): finding. **The decoder is fine. The runner filed an environmental timeout as a parser finding.** **What the nightly saw** (`/var/lib/infraternal-fuzz/.local/state/infraternal-fuzz/ssh_auth_once_decode.log` on dev-02, run of 521cb8990, 22:21 VM time): ``` INFO: 1794 files found in .../corpus/ssh_auth_once_decode ALARM: working on the last Unit for 10 seconds MS: 0 ; base unit: 0000000000000000000000000000000000000000 ... Test unit written to .../artifacts/ssh_auth_once_decode/timeout-da39a3ee5e6b4b0d3255bfef95601890afd80709 ==1979244== ERROR: libFuzzer: timeout after 10 seconds SUMMARY: libFuzzer: timeout ``` The artifact is a 0-byte `timeout-*`, so it's a **timeout**, not a panic. libFuzzer always runs one empty input first in `ReadAndExecuteSeedCorpora`, before the seed corpus and before `INITED`, and the 10 s alarm fired during that first call. The stack is stripped and doesn't symbolize. **The decoder on empty input:** I ran the nightly's own binary (same build, copied out read-only) on dev-02: - `ssh_auth_once_decode -timeout=10 <empty file>`: `Executed ... in 0 ms`, rc=0, three times out of three - `-runs=200000` on the empty input: `Done 200000 runs in 6 second(s)` `serde_json::from_slice` returns `Err` at once for both `OneUseAuth` and `TargetAuth`, so no assertion is reachable. During the replay dev-02 was at load 25–32, with `/proc/pressure/io` at `full avg10=86–95%` (IO-starved). The nightly ran on the same host and filed the same input under the same conditions. A 10 s stall in the first call is environmental: page-ins of the 88 MB ASan binary while the disk is saturated. **What is wrong:** `scripts/nightly-fuzz.py` files every artifact as a parser bug, including `timeout-*` artifacts. Unlike panics, those depend on host load. **Planned fix (on `fuzz-fixes-j264`):** - Before filing a `timeout-*` artifact, the runner runs it once more, alone, with the same `-timeout`. - It files the input only if it times out again. - Otherwise it moves the input to `unconfirmed/<target>/` and logs it, so it isn't re-checked every night. - Crash artifacts are unchanged. - The regression is a fixture test in `scripts/test-nightly-fuzz.py` (gated by `verify`'s "nightly fuzz reporter" row), which fails before the fix. I'm not filing a decoder change, because none is needed. Replay log: `/tmp/j264b-370-repro.txt` on the control host.

kayg referenced this issue

2026-11-05 23:02:29 +00:00

fuzz: host_name_parse found a failing input #353

kayg commented

2026-11-05 23:02:29 +00:00

Author

Owner

J264b progress: fuzz-fixes-j264 now has f839112ba (#353 parser fix) and b1e8d164e (#370 runner fix), both pushed.

#370 regression red/green on compute01: before, FAILED (errors=2) with AttributeError: ... no attribute 'confirmed_inputs'. After, Ran 13 tests ... OK (~/.cache/codex-scratch/j264b/370-{red,green}.log).
Queued on dev-02: the fuzz build with the nightly's toolchain, a replay of the #353 inputs and the empty input, the runner's exact re-check command, then 600 s of host_name_parse and 600 s of ssh_auth_once_decode, seeded from copies of the nightly corpora (~/.cache/codex-scratch/j264b/fuzz-run1.log).
Queued on compute01: scripts/verify quick (~/.cache/codex-scratch/j264b/verify-quick.log).

J264b progress: `fuzz-fixes-j264` now has f839112ba (#353 parser fix) and b1e8d164e (#370 runner fix), both pushed. - #370 regression red/green on compute01: before, `FAILED (errors=2)` with `AttributeError: ... no attribute 'confirmed_inputs'`. After, `Ran 13 tests ... OK` (`~/.cache/codex-scratch/j264b/370-{red,green}.log`). - Queued on dev-02: the fuzz build with the nightly's toolchain, a replay of the #353 inputs and the empty input, the runner's exact re-check command, then 600 s of `host_name_parse` and 600 s of `ssh_auth_once_decode`, seeded from copies of the nightly corpora (`~/.cache/codex-scratch/j264b/fuzz-run1.log`). - Queued on compute01: `scripts/verify quick` (`~/.cache/codex-scratch/j264b/verify-quick.log`).

kayg referenced this issue

2026-11-05 23:40:55 +00:00

fuzz: release_channel found a failing input #374

kayg referenced this issue

2026-11-05 23:40:57 +00:00

fuzz: remote_standby_decode found a failing input #358

kayg commented

2026-11-06 00:27:36 +00:00

Author

Owner

J264b finish.

Branch fuzz-fixes-j264, head d1450e1a0b. It has two commits on base 159edcedc:

271913dc0 fix(plugin-api): machine_name refuses numeric first labels, accepts bare 0x (#353)
d1450e1a0 fix(fuzz): re-check timeouts in isolation before filing them (#370). This also covers #374 and #358.

I have not merged, deployed or closed anything.

Gate: scripts/verify quick through build-slot on compute01 at d1450e1a0 (~/.cache/codex-scratch/j264b/verify-quick2.log), verbatim:

quick scope vs merge base with origin/dev: workspace=0 packages=[infraternal-plugin-api] tracer=0
+---------------------------+--------------+
| Rust gate                 | Result       |
+---------------------------+--------------+
| scope guard               | PASS         |
| migration names           | PASS         |
| Nix Cargo vendoring       | PASS         |
| fmt                       | PASS         |
| doc coverage ratchet      | PASS         |
| shape lints off (D351)    | PASS         |
| doc comments on diff      | PASS         |
| fixed test waits          | PASS         |
| clippy (workspace)        | PASS         |
| unit tests (changed)      | PASS         |
+---------------------------+--------------+

QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done.
VERIFY_EXIT=0

An earlier quick run at the pre-amend head b1e8d164e failed only on fmt (one assert_eq! in the #353 test). I folded the fix into the #353 commit and force-with-lease pushed the branch, so each commit passes the gate.
quick scoped only infraternal-plugin-api. The runner's Python tests (the nightly fuzz reporter row, in fast) passed separately on compute01: Ran 13 tests ... OK (370-green.log). The scripts are byte-identical between b1e8d164e and d1450e1a0.
As tasked, I did not run scripts/verify fast.

Fuzz runs: one batched build on dev-02, with the nightly's own toolchain (rustc 1.101.0-nightly 708c94ef0, cargo-fuzz 0.13.2, ASan) and its flags (-timeout=10 -rss_limit_mb=2048 -max_len=1048580), seeded from copies of the nightly corpora. 600 s per target:

target	executions	artifacts
host_name_parse	102 384 284 in 601 s	none
ssh_auth_once_decode	13 828 284 in 601 s	none
release_channel	57 369 130 in 601 s	none
remote_standby_decode	10 143 408 in 601 s	none

Logs on dev-02: ~/.cache/codex-scratch/j264b/fuzz-run1.log, fuzz-run2.log and fuzz-<target>.log.

Runner re-check, per input (fuzz-run2.log). I called the runner's own confirmed_inputs() with the exact cargo fuzz run <target> <input> -- -timeout=10 -rss_limit_mb=2048 -artifact_prefix=... command that main() builds, on copies of the nightly's four timeout artifacts:

re-check release_channel/timeout-3f4bde2f…: rc=0 Executed … in 0 ms  -> set aside (not filed)
re-check remote_standby_decode/timeout-1c839c82…: rc=0 Executed … in 0 ms  -> set aside (not filed)
re-check remote_standby_decode/timeout-941a98f0…: rc=0 Executed … in 0 ms  -> set aside (not filed)
re-check ssh_auth_once_decode/timeout-da39a3ee…: rc=0 Executed … in 0 ms  -> set aside (not filed)

On the fixed build, the four #353 inputs replay with rc=0 (Executed … in 31/5/1/5 ms).

Once this merges and the nightly checkout moves to it, its first run re-checks the four timeout artifacts still in the nightly's artifact directory and moves them to unconfirmed/. These issues can then be closed by the owner.

J264b finish. **Branch** `fuzz-fixes-j264`, **head d1450e1a0bc234a2acf8fc346c6cb521f06eebb9**. It has two commits on base 159edcedc: - 271913dc0 `fix(plugin-api): machine_name refuses numeric first labels, accepts bare 0x (#353)` - d1450e1a0 `fix(fuzz): re-check timeouts in isolation before filing them (#370)`. This also covers #374 and #358. I have not merged, deployed or closed anything. **Gate:** `scripts/verify quick` through build-slot on compute01 at d1450e1a0 (`~/.cache/codex-scratch/j264b/verify-quick2.log`), verbatim: ``` quick scope vs merge base with origin/dev: workspace=0 packages=[infraternal-plugin-api] tracer=0 +---------------------------+--------------+ | Rust gate | Result | +---------------------------+--------------+ | scope guard | PASS | | migration names | PASS | | Nix Cargo vendoring | PASS | | fmt | PASS | | doc coverage ratchet | PASS | | shape lints off (D351) | PASS | | doc comments on diff | PASS | | fixed test waits | PASS | | clippy (workspace) | PASS | | unit tests (changed) | PASS | +---------------------------+--------------+ QUICK pre-check passed. This is not the gate: run scripts/verify fast before reporting done. VERIFY_EXIT=0 ``` - An earlier quick run at the pre-amend head b1e8d164e failed only on `fmt` (one `assert_eq!` in the #353 test). I folded the fix into the #353 commit and force-with-lease pushed the branch, so each commit passes the gate. - `quick` scoped only `infraternal-plugin-api`. The runner's Python tests (the `nightly fuzz reporter` row, in `fast`) passed separately on compute01: `Ran 13 tests ... OK` (`370-green.log`). The scripts are byte-identical between b1e8d164e and d1450e1a0. - As tasked, I did **not** run `scripts/verify fast`. **Fuzz runs:** one batched build on dev-02, with the nightly's own toolchain (rustc 1.101.0-nightly 708c94ef0, cargo-fuzz 0.13.2, ASan) and its flags (`-timeout=10 -rss_limit_mb=2048 -max_len=1048580`), seeded from copies of the nightly corpora. 600 s per target: | target | executions | rc | artifacts | |---|---|---|---| | host_name_parse | 102 384 284 in 601 s | 0 | none | | ssh_auth_once_decode | 13 828 284 in 601 s | 0 | none | | release_channel | 57 369 130 in 601 s | 0 | none | | remote_standby_decode | 10 143 408 in 601 s | 0 | none | Logs on dev-02: `~/.cache/codex-scratch/j264b/fuzz-run1.log`, `fuzz-run2.log` and `fuzz-<target>.log`. **Runner re-check, per input** (`fuzz-run2.log`). I called the runner's own `confirmed_inputs()` with the exact `cargo fuzz run <target> <input> -- -timeout=10 -rss_limit_mb=2048 -artifact_prefix=...` command that `main()` builds, on copies of the nightly's four timeout artifacts: ``` re-check release_channel/timeout-3f4bde2f…: rc=0 Executed … in 0 ms -> set aside (not filed) re-check remote_standby_decode/timeout-1c839c82…: rc=0 Executed … in 0 ms -> set aside (not filed) re-check remote_standby_decode/timeout-941a98f0…: rc=0 Executed … in 0 ms -> set aside (not filed) re-check ssh_auth_once_decode/timeout-da39a3ee…: rc=0 Executed … in 0 ms -> set aside (not filed) ``` On the fixed build, the four #353 inputs replay with rc=0 (`Executed … in 31/5/1/5 ms`). **Once this merges and the nightly checkout moves to it**, its first run re-checks the four timeout artifacts still in the nightly's artifact directory and moves them to `unconfirmed/`. These issues can then be closed by the owner.

kayg referenced this issue

2026-11-06 01:05:07 +00:00

fuzz: host_name_parse found a failing input #353

kayg commented

2026-11-06 01:05:07 +00:00

Author

Owner

J274 review of fuzz-fixes-j264 @ d1450e1a0: REQUEST CHANGES on the runner commit (P2: the tests do not cover the re-check exit-code rule; P2: an input that is slow but under the 10 s bar gets set aside). The parser commit is approved. Full review: #353 (comment)

J274 review of `fuzz-fixes-j264` @ d1450e1a0: **REQUEST CHANGES** on the runner commit (P2: the tests do not cover the re-check exit-code rule; P2: an input that is slow but under the 10 s bar gets set aside). The parser commit is approved. Full review: https://git.kayg.org/kayg/infraternal/issues/353#issuecomment-81439

kayg commented

2026-11-06 01:36:13 +00:00

Author

Owner

J264b round 2: the runner fixes for this issue are on fuzz-runner-recheck, head 0fc6934ede. Details, mutation reds, the real-input proof (this issue's input set aside; a synthetic 6 s input filed) and the gate are in #353 (comment 81484): #353 (comment)

J264b round 2: the runner fixes for this issue are on `fuzz-runner-recheck`, head 0fc6934ede35b31f4313a497d377412ec47410b7. Details, mutation reds, the real-input proof (this issue's input set aside; a synthetic 6 s input filed) and the gate are in #353 (comment 81484): https://git.kayg.org/kayg/infraternal/issues/353#issuecomment-81484

kayg referenced this issue

2026-11-06 01:42:38 +00:00

fuzz: host_name_parse found a failing input #353

kayg referenced this issue

2026-11-06 01:44:49 +00:00

fuzz: host_name_parse found a failing input #353

kayg referenced this issue

2026-11-06 16:54:37 +00:00

Housekeeping 2026-11-06 #402

kayg referenced this issue

2026-11-06 21:38:59 +00:00

fuzz: host_name_parse found a failing input #353