From 505b865e7a3e486c256a50e88bc41512d65763e4 Mon Sep 17 00:00:00 2001 From: KumamuKuma <157353275+KumamuKuma@users.noreply.github.com> Date: Fri, 17 Jul 2026 03:51:39 +0800 Subject: [PATCH] docs(bench): document community monorepo runs --- .gitignore | 1 + CONTRIBUTING.md | 7 ++ docs/benchmarks/large-monorepo.md | 146 ++++++++++++++++++++++++++++++ 3 files changed, 154 insertions(+) create mode 100644 docs/benchmarks/large-monorepo.md diff --git a/.gitignore b/.gitignore index e85c257b..12ebe52d 100644 --- a/.gitignore +++ b/.gitignore @@ -6,6 +6,7 @@ dist .env .env.* coverage/ +benchmark-results/ *.log __pycache__/ .claude/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a6b4386e..e9d09486 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -110,6 +110,13 @@ Then open a Pull Request on GitHub with: ## 🧪 Testing Guidelines +### Large Repository Benchmarks + +Use the [large monorepo benchmark](docs/benchmarks/large-monorepo.md) to collect +reproducible scale, timing, and memory reports from projects that are too large +for CI fixtures. The guide includes a pinned TensorFlow example and explains +which deterministic stages are measured. + ### Writing Tests - Use Vitest for testing diff --git a/docs/benchmarks/large-monorepo.md b/docs/benchmarks/large-monorepo.md new file mode 100644 index 00000000..643f8709 --- /dev/null +++ b/docs/benchmarks/large-monorepo.md @@ -0,0 +1,146 @@ +# Large Monorepo Benchmark + +Use this benchmark to collect reproducible scale and performance evidence from +large repositories without running an LLM. It is intended for local and +community runs on projects that are too large for normal CI fixtures. + +## Scope + +The runner executes the real deterministic helpers used by `/understand`: + +1. file scanning, classification, and line counting; +2. static import-map extraction; +3. semantic batch planning; and +4. Tree-sitter structural extraction for every batch. + +It reports wall-clock time, CPU time where available, peak resident memory, +output size, repository scale, batching statistics, structural coverage, and +deterministic SHA-256 digests. + +For the regular scan, imports, and batching helpers, `peakRssBytes` is the peak +RSS of that helper process. The concurrent structure stage instead reports +`maxWorkerPeakRssBytes`, the maximum peak RSS of any individual worker, not a +sum across workers. Its `userCpuTimeMicros` and `systemCpuTimeMicros` values are +summed across all structure workers. + +It does **not** run an LLM, call an API, count tokens, estimate cost, generate a +knowledge graph, or render the dashboard. Do not present these results as an +end-to-end `/understand` benchmark. `estimatedAgentInputBytes` is the size of +the deterministic batch payload, not a token estimate. + +## Run the benchmark + +From an Understand Anything checkout: + +```bash +corepack pnpm install --frozen-lockfile +corepack pnpm --filter @understand-anything/core build +corepack pnpm benchmark:large-repo /absolute/path/to/repository --label public-repository-name --output /absolute/path/to/benchmark-results/public-repository-name.json +``` + +The command writes two files: + +- `public-repository-name.json`, which conforms to the versioned + [report schema](large-repo-report-1.0.0.schema.json); and +- `public-repository-name.md`, a human-readable summary beside the JSON file. + +The benchmark tests compile that schema and validate normal, empty, degraded, +and partial failed reports against it. The runner does not perform runtime +schema validation. + +The default concurrency is 5. Override it with `--concurrency 1` through +`--concurrency 32`. Use the same concurrency when comparing runs. + +All intermediate files are created under the operating system's temporary +directory and deleted after the run. Existing `.ua/` and +`.understand-anything/` analysis data is excluded from benchmark input. The +only persistent writes are the requested JSON report and its adjacent Markdown +summary. `--output` is mandatory and must resolve outside the subject +repository; the CLI enforces this boundary, including through filesystem +aliases. + +Published warning summaries are bounded. Each entry records the stage, the +total warning `count`, up to five sanitized `messages`, and a `truncated` flag +that reports omitted or shortened detail. + +`--keep-artifacts` preserves intermediate files and prints their location. +Those private files contain absolute paths and detailed structural data; do not +publish them without reviewing and sanitizing them. + +## Reproducibility protocol + +For results that another contributor can reproduce: + +1. Record an immutable commit for both Understand Anything and the subject + repository. The report captures both commits when the directories are Git + worktrees. +2. Use clean worktrees where possible. The report records `dirty: true` when + tracked or untracked changes are present. +3. Keep the operating system, Node.js version, machine, concurrency, and + `.understandignore` rules constant between compared runs. +4. Run at least three times and retain every report. Treat the first run as a + possible cold-cache result instead of silently discarding it. +5. Compare timing and memory only on the same machine. Cross-machine reports + are useful scale evidence, but they are not a fair performance regression + comparison. +6. Confirm matching `inputDigest` and `outputDigest` values before comparing + performance for supposedly identical inputs. + +Share both the JSON and Markdown files. The report intentionally omits the Git +remote URL, hostname, and absolute subject/tool/artifact paths. It does include +the public label, commit hashes, OS release, CPU model, memory size, and project +statistics, so review both files before publishing them. + +## TensorFlow reproduction recipe + +TensorFlow is a useful manual subject because it is a large, multilingual +monorepo. The benchmark is not TensorFlow-specific, and TensorFlow is +intentionally not cloned or benchmarked in CI. The commands below are a +reproduction recipe, not a claim that a TensorFlow benchmark was run. + +This example pins TensorFlow v2.19.0 to commit +`e36baa302922ea3c7131b302c2996bd2051ee5c4`: + +### Bash + +```bash +git clone --depth 1 --branch v2.19.0 https://github.com/tensorflow/tensorflow.git ../tensorflow-v2.19.0 +if [ "$(git -C ../tensorflow-v2.19.0 rev-parse HEAD)" != "e36baa302922ea3c7131b302c2996bd2051ee5c4" ]; then echo "TensorFlow v2.19.0 did not resolve to the pinned commit" >&2; exit 1; fi +corepack pnpm benchmark:large-repo ../tensorflow-v2.19.0 --label tensorflow-v2.19.0 --output ../benchmark-results/tensorflow-v2.19.0.json +``` + +### PowerShell + +```powershell +git clone --depth 1 --branch v2.19.0 https://github.com/tensorflow/tensorflow.git ..\tensorflow-v2.19.0 +if ((git -C ..\tensorflow-v2.19.0 rev-parse HEAD).Trim() -ne 'e36baa302922ea3c7131b302c2996bd2051ee5c4') { throw 'TensorFlow v2.19.0 did not resolve to the pinned commit' } +corepack pnpm benchmark:large-repo ..\tensorflow-v2.19.0 --label tensorflow-v2.19.0 --output ..\benchmark-results\tensorflow-v2.19.0.json +``` + +Both recipes assert the immutable commit before benchmarking and keep report +outputs outside the TensorFlow checkout. Community testers can share the two +report files; the environment block makes hardware and runtime differences +explicit. + +## Community scale plan + +1. Land the reproducible deterministic harness and its validation-tested report + contract. +2. Collect at least three reports with matching input and output digests for + TensorFlow and other community monorepos. +3. Compare timing and memory only across same-machine runs. +4. Use that evidence to choose the next bottleneck to optimize. +5. Keep LLM, token, and cost benchmarking as a separate future layer rather + than inferring end-to-end performance from this deterministic harness. + +## Status and exit codes + +| Exit code | Meaning | Report behavior | +| ---: | --- | --- | +| `0` | Completed with status `ok` or `degraded` | JSON and Markdown are written | +| `1` | A deterministic stage or integrity check failed | Partial reports are written when the output location is writable | +| `2` | Invalid CLI usage | No report is written | + +`degraded` means the deterministic pipeline completed but reported warnings or +skipped files. Inspect `warnings`, `integrity`, and the Markdown summary before +using that run in a comparison.