Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #10464 +/- ##
==========================================
+ Coverage 88.73% 88.78% +0.04%
==========================================
Files 103 103
Lines 19387 19483 +96
Branches 3023 3045 +22
==========================================
+ Hits 17204 17298 +94
- Misses 1514 1515 +1
- Partials 669 670 +1 ☔ View full report in Codecov by Harness. |
cdcd518 to
0ff265e
Compare
0ff265e to
fb72df4
Compare
ThomasWaldmann
left a comment
There was a problem hiding this comment.
Thanks! #10443, #10445 and #10455 are merged now, so what remains here is fb72df4 only; it merges cleanly into current master.
The interaction with the merged salvage and the single index rebuild looks right to me: packs recorded corrupt are never skipped, so the salvage always works on a result of this run, and pack_files + pack_skipped == len(pack_infos) still blocks the salvage after an interrupted loop. A full --repair with a corrupt index leaves the rebuild to the archives phase, which reads every pack anyway.
Bug: a repository-only repair with --max-age can not fix an index fragment that fails the authentication
The max_age = 0 reset is in the if index_errors and repo_only: branch (repository.py:1622), which runs before the index cross-check. But the cross-check can still find an index error: a fragment that matches its name, but fails the authentication of the key's envelope or does not deserialize (CorruptChunkIndexFragment, index_errors += 1 at repository.py:1664). Then max_age stays active, recently checked packs are skipped, pack_files == len(pack_infos) is false and the rebuild never runs. The comment at repository.py:1761-1762 ("max_age was set to 0 above for exactly this path, so no pack was skipped") does not hold in that case.
Repro, based on test_check_repairs_index_fragment_failing_authentication (plaintext fragment), with a plain check() first, so the packs get recorded intact:
assert repository.check(repair=False) is False # records the packs intact
repository.check(repair=True, repo_only=True, validate=accept_all, max_age=3600)Checked 2 index files (1 errors) and 0 packs (0 errors). Reused 1 recent pack check result(s).
Finished full repository check, index still corrupt.
The check returns False and the bad fragment is still in index/. Without max_age the same repair rebuilds the index and succeeds. Every rerun within the --max-age window fails the same way, and nothing hints at --max-age being the reason.
Suggestion: do the reset (and its log line) after the cross-check, conditioned on repair and repo_only and index_errors, and add a test for this fragment case (the new test only covers the store hash case via rot_index). A full --repair is not affected, the archives phase rebuilds there.
Docs that this PR makes stale
check_cmd.py:311, repair epilog item 1: "repair mode verifies every pack" is no longer true with--max-age.repository.py:1490,check()docstring: "With repair=True, every pack is verified, then ..." contradicts the newmax_ageparagraph further down.
docs/internals/data-structures.rst (checked-packs) is still correct.
Description
Point 5 of #10026: use the
cache/checked-packsrecords during repair.--repairrejected--max-ageand re-hashed every pack, so a repair on a large repository paid full verification cost even right after a check. It now reuses intact records the same way a normal check does.--repair+--max-ageis allowed.--repair+--max-durationand--archives-only+--max-agestay rejected.--repository-only --repairwith a corrupt index ignores--max-ageand logs it. It rebuilds the index from the packs it verified in that run, so a skipped pack would feed the rebuild unverified. A full--repairis unaffected - the archives phase rebuilds from every pack.pack_files + pack_skipped == len(pack_infos), so a skipped pack does not block the salvage; an interrupted or time-boxed loop still does.This is option 2 of the three in #10026 (comment), narrowed to
--repository-only: after #10445 the index rebuild insidecheck()runs only on a repository-only repair, so a full repair can keep the reuse.Stacked on #10455.
Checklist
master(or maintenance branch if only applicable there)