STALE? Ops: rootless Podman storage on o2 had root-owned directories #26

Open
opened 2026-09-24 14:34:02 +00:00 by kayg · 2 comments
Owner

2026-09-24 ~13:39 UTC: /home/kayg/.local/share/containers/storage/overlay/l became root:root (and /tmp/containers-user-1002/containers/overlay has been root:root since boot on Jul 21). All kayg podman commands then failed with 'chown ... operation not permitted'; healthchecks failed, and matrix-mas/synapse/haproxy and calternal-docs went down, which also broke Forgejo SSH key lookups and API calls. Fixed by chowning both directories to kayg and restarting the failed units. Root cause unknown: some process ran podman with kayg's storage paths as root. Find it and stop it.

2026-09-24 ~13:39 UTC: /home/kayg/.local/share/containers/storage/overlay/l became root:root (and /tmp/containers-user-1002/containers/overlay has been root:root since boot on Jul 21). All kayg podman commands then failed with 'chown ... operation not permitted'; healthchecks failed, and matrix-mas/synapse/haproxy and calternal-docs went down, which also broke Forgejo SSH key lookups and API calls. Fixed by chowning both directories to kayg and restarting the failed units. Root cause unknown: some process ran podman with kayg's storage paths as root. Find it and stop it.
Author
Owner

The issue says the immediate outage was recovered after correcting directory ownership and restarting the affected services. It also says the root cause was unknown. The issue has had no activity since 2026-09-24, and this host condition cannot be reproduced from current origin/dev code. Marked STALE? for owner review and left open.

The issue says the immediate outage was recovered after correcting directory ownership and restarting the affected services. It also says the root cause was unknown. The issue has had no activity since 2026-09-24, and this host condition cannot be reproduced from current `origin/dev` code. Marked `STALE?` for owner review and left open.
kayg changed title from Ops: kayg's rootless Podman storage on o2 had root-owned directories to STALE? Ops: rootless Podman storage on o2 had root-owned directories 2026-10-03 11:51:22 +00:00
Author
Owner

The report is dated 2026-09-24 and says the storage ownership and service failures were recovered by correcting directory ownership and restarting units; it records the root cause as unknown. This is host-state evidence, and I found no current origin/dev code path or test that reproduces the historical condition. Keeping #26 open with its existing STALE? title for owner review.

The report is dated 2026-09-24 and says the storage ownership and service failures were recovered by correcting directory ownership and restarting units; it records the root cause as unknown. This is host-state evidence, and I found no current origin/dev code path or test that reproduces the historical condition. Keeping #26 open with its existing STALE? title for owner review.
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
kayg/calternal#26
No description provided.