Compare commits

...
Author SHA1 Message Date
Gergo MagyarandClaude Opus 4.8 e766cedd1a fix(eval): drop tags from the benchmark's per-arm clone
Every benchmark-arm session failed with "sanitized graph snapshot
preparation failed: clone has more than 1024 references; refusing
incomplete sanitization" (confirmed via a real workflow_dispatch run,
29738099937, after the prior activation fixes let the proposer succeed
end-to-end for the first time).

make_worktree() creates each arm's throwaway clone with a plain `git
clone`, which inherits every tag and branch from the source. This repo's
history has grown to 1144 tags (a v1.6.9-rc.N release-candidate series)
out of 1650 total refs, exceeding oracle_assets.MAX_CLONE_REFS=1024 -- a
fail-closed guard in sanitize_clone_for_hidden_oracles() that refuses to
proceed unless it can enumerate and delete every ref before handing a
sanitized snapshot to a benchmark session (so an agent can never discover
oracle answers via a ref the sanitization missed).

`ref` at every call site (evolve.py, runner.py, sanitized_graph.py) is
always a bare SHA or the literal "HEAD", never a branch name, so
`--single-branch --branch <ref>` isn't viable (git clone's --branch
requires a name). Tags are never used by the checkout fallback or by
sanitization's own delete-everything behavior, so dropping them via
--no-tags removes the 1144-ref majority without touching branch-fetch
behavior or the existing ref/origin-ref checkout fallback, and without
weakening MAX_CLONE_REFS itself.

Verified against the real repository (not just the test fixture): cloning
/workspace (1650 refs, 1144 tags) via the fixed make_worktree() now
produces a clone with 237 total refs and 0 tags.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Va5uu9Ar3e45QZ5xFsG4AZ
2026-07-20 11:36:58 +00:00
2 changed files with 53 additions and 0 deletions
@@ -1025,3 +1025,55 @@ def test_review_phase_rejects_workspace_or_skill_mutation(
assert rec["resolved"] is False
assert rec["error_kind"] == "review-evidence-invalid"
assert expected_detail in rec["error_detail"]
def _git(repo, *args, check=True):
return subprocess.run(["git", "-C", str(repo), *args], check=check, capture_output=True, text=True)
def _git_commit(repo, message):
_git(
repo,
"-c",
"user.name=test",
"-c",
"user.email=test@invalid",
"commit",
"--quiet",
"--allow-empty",
"-m",
message,
)
return _git(repo, "rev-parse", "HEAD").stdout.strip()
def test_make_worktree_clone_has_no_tags_but_keeps_all_branches(tmp_path):
# oracle_assets.MAX_CLONE_REFS refuses to sanitize a clone with more than
# 1024 refs; this repo's own history has 1000+ release-candidate tags, so
# a plain `git clone` of it (inheriting every tag) trips that cap on every
# benchmark session. make_worktree must not carry tags into its throwaway
# clone, but callers pass a bare SHA or "HEAD" as `ref` (never a branch
# name -- see evolve.py:476, runner.py:1037, sanitized_graph.py:345), so
# branch-fetching itself must stay untouched: a commit reachable only from
# a non-default branch must still resolve via the existing
# checkout(ref) -> checkout(origin/{ref}) fallback.
repo = tmp_path / "repo"
repo.mkdir()
_git(repo, "init", "--quiet")
_git(repo, "checkout", "--quiet", "-b", "main")
_git_commit(repo, "base")
_git(repo, "tag", "v1.0.0-rc.1")
_git(repo, "checkout", "--quiet", "-b", "other")
other_sha = _git_commit(repo, "only on other")
_git(repo, "checkout", "--quiet", "main")
clones = tmp_path / "clones"
clones.mkdir()
target = runner.make_worktree(repo, other_sha, clones)
tags = _git(target, "tag").stdout.split()
assert tags == [], f"clone must carry no tags, found: {tags}"
current = _git(target, "rev-parse", "HEAD").stdout.strip()
assert current == other_sha
+1
View File
@@ -240,6 +240,7 @@ def make_worktree(repo: Path, ref: str, parent: Path) -> Path:
"clone",
"--no-local",
"--no-hardlinks",
"--no-tags",
"--quiet",
str(repo),
str(target),