Compare commits
3
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Gergo Magyar
|
39628adeea | ||
|
jiangnan
|
acda87a247 | ||
|
jiangnan
|
8ca25fe753 |
@@ -17,13 +17,12 @@ npx gitnexus analyze
|
||||
|
||||
Run from the project root. This parses all source files, builds the knowledge graph, writes it to `.gitnexus/`, and generates CLAUDE.md / AGENTS.md context files.
|
||||
|
||||
| Flag | Effect |
|
||||
| ------------------- | ------------------------------------------------------------------------------------------------------- |
|
||||
| `--force` | Force full re-index even if up to date |
|
||||
| `--embeddings` | Enable embedding generation for semantic search (off by default) |
|
||||
| `--drop-embeddings` | Drop existing embeddings on rebuild. By default, an `analyze` without `--embeddings` preserves them. |
|
||||
| Flag | Effect |
|
||||
| -------------- | ---------------------------------------------------------------- |
|
||||
| `--force` | Force full re-index even if up to date |
|
||||
| `--embeddings` | Enable embedding generation for semantic search (off by default) |
|
||||
|
||||
**When to run:** First time in a project, after major code changes, or when `gitnexus://repo/{name}/context` reports the index is stale. In Claude Code, a PostToolUse hook detects staleness after `git commit` and `git merge` and notifies the agent to run `analyze` — the hook does not run analyze itself, to avoid blocking the agent for up to 120s and risking KuzuDB corruption on timeout.
|
||||
**When to run:** First time in a project, after major code changes, or when `gitnexus://repo/{name}/context` reports the index is stale.
|
||||
|
||||
### status — Check index freshness
|
||||
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
plans/
|
||||
@@ -1,21 +0,0 @@
|
||||
---
|
||||
alwaysApply: true
|
||||
---
|
||||
|
||||
# GitNexus — Cursor project rules
|
||||
|
||||
Last reviewed: 2026-03-24
|
||||
|
||||
Canonical agent instructions: **[AGENTS.md](../AGENTS.md)** (GitNexus MCP rules, monorepo commands, Cursor Cloud notes). **[CLAUDE.md](../CLAUDE.md)** adds Claude Code-specific notes and points back to AGENTS.md for GitNexus.
|
||||
|
||||
## Non-negotiables (always apply)
|
||||
|
||||
- NEVER edit a function/class/method without running `gitnexus_impact` first.
|
||||
- NEVER rename symbols with find-and-replace — use `gitnexus_rename`.
|
||||
- NEVER commit without running `gitnexus_detect_changes()`.
|
||||
- NEVER ignore HIGH/CRITICAL risk warnings from impact analysis.
|
||||
- NEVER run `npx gitnexus analyze` without `--embeddings` if `.gitnexus/meta.json` shows stored embeddings.
|
||||
|
||||
Full rules: **[AGENTS.md](../AGENTS.md)** (`gitnexus:start` block, Cursor Cloud section).
|
||||
|
||||
**Rule architecture:** Prefer this file plus optional `.cursor/rules/*.mdc` globs (YAML `globs` in frontmatter). Legacy `.cursorrules` is deprecated; content lives here.
|
||||
@@ -1,12 +0,0 @@
|
||||
---
|
||||
globs:
|
||||
- "gitnexus/**"
|
||||
- "gitnexus-web/**"
|
||||
---
|
||||
|
||||
# GitNexus build/test quick refs
|
||||
|
||||
- CLI (`gitnexus/`): `npm test`; `npm run test:integration`; `npx tsc --noEmit`.
|
||||
- Web (`gitnexus-web/`): `npm test`; `npm run dev`; `npx tsc -b --noEmit`; `E2E=1 npx playwright test` (needs servers).
|
||||
- `npm install` in `gitnexus/` runs `prepare` (tsc build) and `postinstall` (tree-sitter patches); needs `python3`, `make`, `g++`.
|
||||
- LadybugDB locking tests may fail in containerized environments because of `/tmp` file locks (known issue, not a code bug).
|
||||
@@ -1,14 +0,0 @@
|
||||
---
|
||||
globs:
|
||||
- "eval/**"
|
||||
---
|
||||
|
||||
# GitNexus eval harness (Python)
|
||||
|
||||
- **Run tests**: `cd eval && uv run pytest tests/`
|
||||
- **Run with coverage**: `cd eval && uv run coverage run -m pytest tests/ && uv run coverage report`
|
||||
- **Lint**: `cd eval && uv run ruff check .`
|
||||
- **Run eval**: `cd eval && uv run python run_eval.py --config configs/<config>.yaml`
|
||||
- Shared constants live in `eval/constants.py`; tool specs in `eval/tool_registry.py`.
|
||||
- Error logging uses `utils/errors.py` — set `GITNEXUS_EVAL_DEBUG=1` for full tracebacks.
|
||||
- Property-based tests use Hypothesis (`eval/tests/test_property_based.py`).
|
||||
+3
-3
@@ -1,5 +1,5 @@
|
||||
# Deprecated for Cursor Agent Mode
|
||||
# AI Agent Rules
|
||||
|
||||
Use **`.cursor/index.mdc`** (`alwaysApply: true`) for project rules. See [AGENTS.md](AGENTS.md).
|
||||
Follow .gitnexus/RULES.md for all project context and coding guidelines.
|
||||
|
||||
This file is kept only as a breadcrumb for older workflows.
|
||||
This project uses GitNexus MCP for code intelligence. See .gitnexus/RULES.md for available tools and best practices.
|
||||
|
||||
@@ -1,21 +0,0 @@
|
||||
.git
|
||||
.gitignore
|
||||
.DS_Store
|
||||
|
||||
node_modules
|
||||
**/node_modules
|
||||
|
||||
dist
|
||||
**/dist
|
||||
coverage
|
||||
**/coverage
|
||||
|
||||
.env
|
||||
.env.local
|
||||
.env.*.local
|
||||
|
||||
**/*.tsbuildinfo
|
||||
|
||||
.gitnexus
|
||||
gitnexus-web/playwright-report
|
||||
gitnexus-web/test-results
|
||||
@@ -1,19 +0,0 @@
|
||||
# Images (signed Cosign keyless on every push from main / vX.Y.Z tags).
|
||||
# Available from both GHCR (default below) and Docker Hub — pick one:
|
||||
# GHCR: ghcr.io/abhigyanpatwari/gitnexus{,-web}:latest
|
||||
# Docker Hub: akonlabs/gitnexus{,-web}:latest
|
||||
# Both registries receive the same digest from a single signed build.
|
||||
SERVER_IMAGE=ghcr.io/abhigyanpatwari/gitnexus:latest
|
||||
WEB_IMAGE=ghcr.io/abhigyanpatwari/gitnexus-web:latest
|
||||
|
||||
# Container names
|
||||
SERVER_CONTAINER_NAME=gitnexus-server
|
||||
WEB_CONTAINER_NAME=gitnexus-web
|
||||
|
||||
# Host ports — the web UI expects the server on http://localhost:4747 by default.
|
||||
SERVER_HOST_PORT=4747
|
||||
WEB_HOST_PORT=4173
|
||||
|
||||
# Optional read-only mount, exposed to the server as /workspace.
|
||||
# Override with the directory that contains the repos you want to index.
|
||||
WORKSPACE_DIR=./
|
||||
@@ -1,5 +0,0 @@
|
||||
# Prettier initial formatting (2026-03-28)
|
||||
afcc3d1523f99c77ff67c4fd1af12334660113f6
|
||||
|
||||
# ESLint unused import removal (2026-03-28)
|
||||
1491826bc8da5436d3b1eb092d9274f2c9f028a7
|
||||
@@ -1,2 +0,0 @@
|
||||
* text=auto eol=lf
|
||||
.husky/* text eol=lf
|
||||
@@ -1,86 +0,0 @@
|
||||
name: Bug report
|
||||
description: Report unexpected behavior or a regression
|
||||
labels: [bug]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
**Goal:** capture enough context to reproduce and fix the issue quickly.
|
||||
Use **one issue per bug**; split unrelated problems.
|
||||
|
||||
- type: dropdown
|
||||
id: area
|
||||
attributes:
|
||||
label: Area
|
||||
description: Where does the problem show up?
|
||||
options:
|
||||
- gitnexus (CLI / core / indexing / MCP server)
|
||||
- gitnexus-web (browser UI / WASM / workers)
|
||||
- CI / GitHub Actions
|
||||
- Documentation / developer experience
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: summary
|
||||
attributes:
|
||||
label: Summary
|
||||
description: One sentence — what went wrong?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: context
|
||||
attributes:
|
||||
label: Context
|
||||
description: What were you trying to do? Any relevant links, PRs, or commits?
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: expected
|
||||
attributes:
|
||||
label: Expected behavior
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: actual
|
||||
attributes:
|
||||
label: Actual behavior
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: reproduce
|
||||
attributes:
|
||||
label: Steps to reproduce
|
||||
description: Ordered steps, sample repo or minimal case, commands run.
|
||||
placeholder: |
|
||||
1. …
|
||||
2. …
|
||||
3. …
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: environment
|
||||
attributes:
|
||||
label: Environment
|
||||
description: OS, Node version, browser (if web), GitNexus version or commit SHA.
|
||||
placeholder: |
|
||||
- OS:
|
||||
- Node:
|
||||
- Browser (if applicable):
|
||||
- Commit / version:
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: logs
|
||||
attributes:
|
||||
label: Logs / screenshots
|
||||
description: Paste errors, stack traces, or attach screenshots (redact secrets).
|
||||
validations:
|
||||
required: false
|
||||
@@ -1 +0,0 @@
|
||||
blank_issues_enabled: true
|
||||
@@ -1,74 +0,0 @@
|
||||
name: Feature request
|
||||
description: Propose a new capability or improvement
|
||||
labels: [enhancement]
|
||||
body:
|
||||
- type: markdown
|
||||
attributes:
|
||||
value: |
|
||||
**Goal:** describe the problem and desired outcome so maintainers can size and prioritize.
|
||||
Prefer **small, shippable** requests; split large ideas into phases.
|
||||
|
||||
- type: dropdown
|
||||
id: area
|
||||
attributes:
|
||||
label: Area
|
||||
description: Primary part of the monorepo this relates to.
|
||||
options:
|
||||
- gitnexus (CLI / core / indexing / MCP server)
|
||||
- gitnexus-web (browser UI / WASM / workers)
|
||||
- CI / release / packaging
|
||||
- Documentation / developer experience
|
||||
- Other
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: problem
|
||||
attributes:
|
||||
label: Problem or opportunity
|
||||
description: What pain point or gap exists today?
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: proposal
|
||||
attributes:
|
||||
label: Proposed solution
|
||||
description: What should happen instead? User-visible behavior, APIs, or UX.
|
||||
validations:
|
||||
required: true
|
||||
|
||||
- type: textarea
|
||||
id: alternatives
|
||||
attributes:
|
||||
label: Alternatives considered
|
||||
description: Other approaches you considered and why this one is preferred.
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: acceptance
|
||||
attributes:
|
||||
label: Acceptance criteria
|
||||
description: Testable conditions for “done” (bullets or checkboxes in prose).
|
||||
placeholder: |
|
||||
- When … then …
|
||||
- Documentation / tests updated where appropriate
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: textarea
|
||||
id: constraints
|
||||
attributes:
|
||||
label: Constraints
|
||||
description: Compatibility, performance, security, or “must not change” boundaries.
|
||||
validations:
|
||||
required: false
|
||||
|
||||
- type: checkboxes
|
||||
id: willing
|
||||
attributes:
|
||||
label: Contribution
|
||||
options:
|
||||
- label: I am willing to open a PR for this (may need design discussion first).
|
||||
required: false
|
||||
@@ -1,52 +0,0 @@
|
||||
## Summary
|
||||
|
||||
<!-- One or two sentences: what does this PR change? -->
|
||||
|
||||
## Motivation / context
|
||||
|
||||
<!-- Why is this change needed? Link issues, ADRs, or prior discussion. -->
|
||||
|
||||
## Areas touched
|
||||
|
||||
<!-- Check all that apply -->
|
||||
|
||||
- [ ] `gitnexus/` (CLI / core / MCP server)
|
||||
- [ ] `gitnexus-web/` (Vite / React UI)
|
||||
- [ ] `.github/` (workflows, actions)
|
||||
- [ ] `eval/` or other tooling
|
||||
- [ ] Docs / agent config only (`AGENTS.md`, `CLAUDE.md`, `.cursor/`, `llms.txt`, etc.)
|
||||
|
||||
## Scope & constraints
|
||||
|
||||
**In scope**
|
||||
|
||||
- <!-- bullets -->
|
||||
|
||||
**Explicitly out of scope / not done here**
|
||||
|
||||
- <!-- bullets — prevents reviewers assuming missing work is an oversight -->
|
||||
|
||||
## Implementation notes
|
||||
|
||||
<!-- Optional: design choices, tradeoffs, follow-ups -->
|
||||
|
||||
## Testing & verification
|
||||
|
||||
<!-- What you ran; paste commands. Omit sections that do not apply. -->
|
||||
|
||||
- [ ] `cd gitnexus && npm test`
|
||||
- [ ] `cd gitnexus && npm run test:integration` *(if core/indexing/MCP paths changed)*
|
||||
- [ ] `cd gitnexus && npx tsc --noEmit`
|
||||
- [ ] `cd gitnexus-web && npm test` *(if web changed)*
|
||||
- [ ] `cd gitnexus-web && npx tsc -b --noEmit` *(if web changed)*
|
||||
- [ ] Manual / Playwright E2E *(note environment — see `gitnexus-web/e2e/`)*
|
||||
|
||||
## Risk & rollout
|
||||
|
||||
<!-- Breaking changes, migrations, index refresh (`npx gitnexus analyze`), release notes -->
|
||||
|
||||
## Checklist
|
||||
|
||||
- [ ] PR body meets repo minimum length (workflow may label short descriptions)
|
||||
- [ ] If `AGENTS.md` / overlays changed: headers, scope block, and changelog updated per project conventions
|
||||
- [ ] No secrets, tokens, or machine-specific paths committed
|
||||
@@ -1,105 +0,0 @@
|
||||
# Wraps docker/build-push-action with one automatic retry. Upstream explicitly
|
||||
# keeps retry out of the action (docker/build-push-action#1422); a local
|
||||
# composite keeps docker.yml readable and pins the same action SHA in one place.
|
||||
name: Docker build-push (with retry)
|
||||
description: >-
|
||||
Runs docker/build-push-action twice on failure with a configurable backoff,
|
||||
then exposes the digest from whichever attempt succeeded.
|
||||
|
||||
inputs:
|
||||
context:
|
||||
description: Build context path
|
||||
required: false
|
||||
default: '.'
|
||||
file:
|
||||
description: Dockerfile path (relative to repo root)
|
||||
required: true
|
||||
platforms:
|
||||
description: Comma-separated platforms list for buildx
|
||||
required: true
|
||||
push:
|
||||
description: Whether to push (string 'true' or 'false')
|
||||
required: true
|
||||
tags:
|
||||
description: Newline-separated image tags (from docker/metadata-action)
|
||||
required: true
|
||||
labels:
|
||||
description: Labels string (from docker/metadata-action)
|
||||
required: true
|
||||
cache-from:
|
||||
description: buildx cache-from value
|
||||
required: true
|
||||
cache-to:
|
||||
description: buildx cache-to value (include ignore-error=true for GHA cache flakes)
|
||||
required: true
|
||||
retry-wait-seconds:
|
||||
description: Seconds to sleep before the second attempt
|
||||
required: false
|
||||
default: '45'
|
||||
|
||||
outputs:
|
||||
digest:
|
||||
description: Manifest digest from the successful build attempt
|
||||
value: ${{ steps.resolve.outputs.digest }}
|
||||
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- name: Build and push (attempt 1)
|
||||
id: try1
|
||||
continue-on-error: true
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
with:
|
||||
context: ${{ inputs.context }}
|
||||
file: ${{ inputs.file }}
|
||||
platforms: ${{ inputs.platforms }}
|
||||
push: ${{ inputs.push == 'true' }}
|
||||
tags: ${{ inputs.tags }}
|
||||
labels: ${{ inputs.labels }}
|
||||
cache-from: ${{ inputs.cache-from }}
|
||||
cache-to: ${{ inputs.cache-to }}
|
||||
provenance: mode=max
|
||||
sbom: true
|
||||
|
||||
- name: Backoff before Docker build retry
|
||||
if: steps.try1.outcome == 'failure'
|
||||
shell: bash
|
||||
env:
|
||||
RETRY_WAIT_SECONDS: ${{ inputs.retry-wait-seconds }}
|
||||
run: |
|
||||
echo "::warning::Docker build-push attempt 1 failed; retrying in ${RETRY_WAIT_SECONDS}s…"
|
||||
sleep "${RETRY_WAIT_SECONDS}"
|
||||
|
||||
- name: Build and push (attempt 2)
|
||||
id: try2
|
||||
if: steps.try1.outcome == 'failure'
|
||||
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
|
||||
with:
|
||||
context: ${{ inputs.context }}
|
||||
file: ${{ inputs.file }}
|
||||
platforms: ${{ inputs.platforms }}
|
||||
push: ${{ inputs.push == 'true' }}
|
||||
tags: ${{ inputs.tags }}
|
||||
labels: ${{ inputs.labels }}
|
||||
cache-from: ${{ inputs.cache-from }}
|
||||
cache-to: ${{ inputs.cache-to }}
|
||||
provenance: mode=max
|
||||
sbom: true
|
||||
|
||||
- name: Resolve image digest
|
||||
id: resolve
|
||||
if: always()
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
if [ "${{ steps.try1.outcome }}" = "success" ]; then
|
||||
echo "digest=${{ steps.try1.outputs.digest }}" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
if [ "${{ steps.try2.outcome }}" = "success" ]; then
|
||||
echo "::notice::docker-build-push retry succeeded (attempt 2); investigate if this recurs across runs."
|
||||
echo "digest=${{ steps.try2.outputs.digest }}" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
echo "::error::Docker build and push failed after two attempts (registry/cache flake or real build error)."
|
||||
exit 1
|
||||
@@ -1,24 +0,0 @@
|
||||
name: Setup GitNexus Web
|
||||
description: Setup Node.js 20.19+ (vite 7 floor), build gitnexus-shared, install web dependencies
|
||||
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
|
||||
with:
|
||||
# Vite 7 requires Node ^20.19.0 || >=22.12.0 (require(esm) support).
|
||||
# Pin explicitly so we don't depend on the floating "20" alias resolving
|
||||
# to a high enough patch version on every runner image.
|
||||
node-version: '20.19.0'
|
||||
cache: npm
|
||||
cache-dependency-path: gitnexus-web/package-lock.json
|
||||
|
||||
- name: Build gitnexus-shared
|
||||
run: npm install && npm run build
|
||||
shell: bash
|
||||
working-directory: gitnexus-shared
|
||||
|
||||
- name: Install web dependencies
|
||||
run: npm ci
|
||||
shell: bash
|
||||
working-directory: gitnexus-web
|
||||
@@ -10,17 +10,12 @@ inputs:
|
||||
runs:
|
||||
using: composite
|
||||
steps:
|
||||
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 20
|
||||
cache: npm
|
||||
cache-dependency-path: gitnexus/package-lock.json
|
||||
|
||||
- name: Build gitnexus-shared
|
||||
run: npm install && npm run build
|
||||
shell: bash
|
||||
working-directory: gitnexus-shared
|
||||
|
||||
- name: Install dependencies
|
||||
run: npm ci
|
||||
shell: bash
|
||||
|
||||
@@ -1,75 +0,0 @@
|
||||
version: 2
|
||||
updates:
|
||||
# Keep third-party Actions SHA pins current. See CONTRIBUTING.md — when
|
||||
# reviewing these bumps, verify the SHA corresponds to the claimed tag by
|
||||
# running `gh api repos/<owner>/<action>/git/refs/tags/<tag>` before merge.
|
||||
- package-ecosystem: github-actions
|
||||
directory: /
|
||||
schedule:
|
||||
interval: weekly
|
||||
open-pull-requests-limit: 5
|
||||
commit-message:
|
||||
prefix: chore
|
||||
include: scope
|
||||
labels:
|
||||
- dependencies
|
||||
- ci
|
||||
|
||||
# Gitnexus npm deps — tree-sitter grammars checked daily so we catch
|
||||
# new releases that unblock the tree-sitter 0.25 upgrade ASAP. Grammars
|
||||
# are grouped so lockstep bumps produce a single PR. The tree-sitter
|
||||
# RUNTIME is pinned — upgrade deliberately via the drift check workflow.
|
||||
# See .github/scripts/check-tree-sitter-upgrade-readiness.py for
|
||||
# the upgrade readiness tracker.
|
||||
- package-ecosystem: npm
|
||||
directory: /gitnexus
|
||||
schedule:
|
||||
interval: daily
|
||||
open-pull-requests-limit: 10
|
||||
commit-message:
|
||||
prefix: chore(deps)
|
||||
include: scope
|
||||
labels:
|
||||
- dependencies
|
||||
groups:
|
||||
tree-sitter-grammars:
|
||||
patterns:
|
||||
- tree-sitter-*
|
||||
exclude-patterns:
|
||||
- tree-sitter
|
||||
- tree-sitter-cli
|
||||
ignore:
|
||||
# Pin the tree-sitter runtime at 0.21.x until the drift check
|
||||
# reports all grammars are peer-dep compatible with 0.25.
|
||||
- dependency-name: tree-sitter
|
||||
update-types:
|
||||
- version-update:semver-major
|
||||
- version-update:semver-minor
|
||||
# tree-sitter-cli follows the runtime's version cadence. Bump when
|
||||
# regenerating vendor/tree-sitter-proto/src/parser.c, not on a schedule.
|
||||
- dependency-name: tree-sitter-cli
|
||||
|
||||
# gitnexus-web (thin frontend client).
|
||||
- package-ecosystem: npm
|
||||
directory: /gitnexus-web
|
||||
schedule:
|
||||
interval: weekly
|
||||
open-pull-requests-limit: 5
|
||||
commit-message:
|
||||
prefix: chore(deps)
|
||||
include: scope
|
||||
labels:
|
||||
- dependencies
|
||||
- frontend
|
||||
|
||||
# Shared types package.
|
||||
- package-ecosystem: npm
|
||||
directory: /gitnexus-shared
|
||||
schedule:
|
||||
interval: weekly
|
||||
open-pull-requests-limit: 5
|
||||
commit-message:
|
||||
prefix: chore(deps)
|
||||
include: scope
|
||||
labels:
|
||||
- dependencies
|
||||
@@ -1,53 +0,0 @@
|
||||
# release-drafter config — used only for PR autolabeling by
|
||||
# `.github/workflows/pr-labeler.yml` (the workflow passes `disable-releaser: true`,
|
||||
# so the draft-release side of release-drafter never runs).
|
||||
#
|
||||
# The labels applied here are the same ones `.github/release.yml` maps to
|
||||
# categorized release-notes sections.
|
||||
#
|
||||
# `sync-labels: true` removes managed autolabels that no longer match the PR —
|
||||
# critical for the breaking-change case: if a PR title drops the `!` or the body
|
||||
# drops `BREAKING CHANGE:`, the `breaking` label is pulled off automatically.
|
||||
|
||||
# Required by release-drafter; not used because releaser is disabled.
|
||||
name-template: 'unused'
|
||||
tag-template: 'unused'
|
||||
template: |
|
||||
$CHANGES
|
||||
|
||||
sync-labels: true
|
||||
|
||||
autolabeler:
|
||||
- label: enhancement
|
||||
title:
|
||||
- '/^feat(\([^)]+\))?!?:/i'
|
||||
- label: bug
|
||||
title:
|
||||
- '/^fix(\([^)]+\))?!?:/i'
|
||||
- label: performance
|
||||
title:
|
||||
- '/^perf(\([^)]+\))?!?:/i'
|
||||
- label: refactor
|
||||
title:
|
||||
- '/^refactor(\([^)]+\))?!?:/i'
|
||||
- label: documentation
|
||||
title:
|
||||
- '/^docs(\([^)]+\))?!?:/i'
|
||||
- label: test
|
||||
title:
|
||||
- '/^test(\([^)]+\))?!?:/i'
|
||||
- label: ci
|
||||
title:
|
||||
- '/^ci(\([^)]+\))?!?:/i'
|
||||
- label: dependencies
|
||||
title:
|
||||
- '/^(build|deps)(\([^)]+\))?!?:/i'
|
||||
- label: chore
|
||||
title:
|
||||
- '/^(chore|revert)(\([^)]+\))?!?:/i'
|
||||
# Breaking-change marker: either `!` in the type prefix or `BREAKING CHANGE:` in body.
|
||||
- label: breaking
|
||||
title:
|
||||
- '/^[a-z]+(\([^)]+\))?!:/i'
|
||||
body:
|
||||
- '/BREAKING[ -]CHANGE:/i'
|
||||
+1
-1
@@ -35,7 +35,7 @@ changelog:
|
||||
- dependencies
|
||||
- title: "\U0001F4DD Other Changes"
|
||||
labels:
|
||||
- '*'
|
||||
- "*"
|
||||
exclude:
|
||||
labels:
|
||||
- dependencies
|
||||
|
||||
@@ -1,798 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Monitor tree-sitter 0.25 upgrade readiness.
|
||||
|
||||
Tracks two things Dependabot cannot see:
|
||||
|
||||
1. Peer-dep compatibility. Each tree-sitter-* grammar declares a peer
|
||||
dependency on the tree-sitter runtime. We want to know when every
|
||||
grammar's *latest npm release* satisfies tree-sitter@0.25.0 so we
|
||||
can upgrade without --legacy-peer-deps.
|
||||
|
||||
2. Vendored upstream drift. vendor/tree-sitter-proto/ is a snapshot of
|
||||
coder3101/tree-sitter-proto's parser.c. When upstream moves, we want
|
||||
to know whether we can pick it up.
|
||||
|
||||
Invoked from .github/workflows/tree-sitter-upgrade-readiness.yml daily.
|
||||
Runs locally too:
|
||||
|
||||
python3 .github/scripts/check-tree-sitter-upgrade-readiness.py
|
||||
|
||||
Outputs Markdown to stdout. Exit 0 when every grammar is upgrade-ready
|
||||
and the vendored proto is in sync. Exit 1 when blockers remain (the
|
||||
workflow uses this to open or update a tracking issue).
|
||||
|
||||
No external deps -- stdlib only, so it runs on any vanilla runner.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
import urllib.error
|
||||
import urllib.request
|
||||
|
||||
REPO_ROOT = pathlib.Path(__file__).resolve().parents[2]
|
||||
GITNEXUS_DIR = REPO_ROOT / "gitnexus"
|
||||
|
||||
# ── Upgrade target ──────────────────────────────────────────────────────
|
||||
# The runtime version we want to upgrade TO. Update this when the goal
|
||||
# changes (e.g. once 0.25 lands and we target 0.26).
|
||||
TARGET_RUNTIME = "0.25.0"
|
||||
TARGET_RUNTIME_MAJOR_MINOR = ".".join(TARGET_RUNTIME.split(".")[:2])
|
||||
|
||||
# Tree-sitter runtime -> (min_abi, max_abi) it can load. Only the current
|
||||
# and target entries matter; extend when changing TARGET_RUNTIME.
|
||||
RUNTIME_ABI_RANGES: dict[str, tuple[int, int]] = {
|
||||
"0.21": (13, 14),
|
||||
"0.25": (13, 15),
|
||||
}
|
||||
|
||||
assert TARGET_RUNTIME_MAJOR_MINOR in RUNTIME_ABI_RANGES, (
|
||||
f"RUNTIME_ABI_RANGES has no entry for {TARGET_RUNTIME_MAJOR_MINOR!r}. "
|
||||
f"Add the ABI range after auditing the upstream release notes."
|
||||
)
|
||||
|
||||
# Grammars we use. Values are the upstream GitHub repos to check for
|
||||
# unreleased ABI bumps (owner/repo, branch, parser.c path).
|
||||
GRAMMARS: dict[str, tuple[str, str, str]] = {
|
||||
"tree-sitter-c": ("tree-sitter/tree-sitter-c", "master", "src/parser.c"),
|
||||
"tree-sitter-c-sharp": ("tree-sitter/tree-sitter-c-sharp", "master", "src/parser.c"),
|
||||
"tree-sitter-cpp": ("tree-sitter/tree-sitter-cpp", "master", "src/parser.c"),
|
||||
"tree-sitter-dart": ("UserNobody14/tree-sitter-dart", "master", "src/parser.c"),
|
||||
"tree-sitter-go": ("tree-sitter/tree-sitter-go", "master", "src/parser.c"),
|
||||
"tree-sitter-java": ("tree-sitter/tree-sitter-java", "master", "src/parser.c"),
|
||||
"tree-sitter-javascript": ("tree-sitter/tree-sitter-javascript", "master", "src/parser.c"),
|
||||
"tree-sitter-kotlin": ("fwcd/tree-sitter-kotlin", "main", "src/parser.c"),
|
||||
"tree-sitter-php": ("tree-sitter/tree-sitter-php", "master", "php/src/parser.c"),
|
||||
"tree-sitter-python": ("tree-sitter/tree-sitter-python", "master", "src/parser.c"),
|
||||
"tree-sitter-ruby": ("tree-sitter/tree-sitter-ruby", "master", "src/parser.c"),
|
||||
"tree-sitter-rust": ("tree-sitter/tree-sitter-rust", "master", "src/parser.c"),
|
||||
"tree-sitter-swift": ("alex-pinkus/tree-sitter-swift", "main", "src/parser.c"),
|
||||
"tree-sitter-typescript": ("tree-sitter/tree-sitter-typescript", "master", "typescript/src/parser.c"),
|
||||
# Vendored parsers — kept here so the upstream coords for drift
|
||||
# detection are co-located with every other grammar's coords.
|
||||
"tree-sitter-proto": ("coder3101/tree-sitter-proto", "main", "src/parser.c"),
|
||||
}
|
||||
|
||||
# Grammars deliberately held below npm latest. The readiness report surfaces
|
||||
# these so reviewers can tell intentional pins apart from drift, and so the
|
||||
# context for each pin (which issue motivated it) is visible at a glance.
|
||||
# Add an entry whenever you pin a grammar below npm latest.
|
||||
INTENTIONAL_PINS: dict[str, str] = {
|
||||
"tree-sitter-c": (
|
||||
"#1242 — last release built against the tree-sitter@0.21 ABI; "
|
||||
"tree-sitter-c@0.23.x prebuilds segfault on Windows under tree-sitter@0.21.1"
|
||||
),
|
||||
"tree-sitter-cpp": (
|
||||
"#1242 — last 0.23.x release before tree-sitter-cpp added a runtime "
|
||||
"dep on the broken-ABI tree-sitter-c@^0.23.1; pinning here removes "
|
||||
"the need for a transitive override"
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# ── Helpers ─────────────────────────────────────────────────────────────
|
||||
|
||||
def _load_package_json() -> dict:
|
||||
return json.loads((GITNEXUS_DIR / "package.json").read_text())
|
||||
|
||||
|
||||
def read_current_runtime() -> str:
|
||||
"""Return the tree-sitter runtime version pinned in package.json (e.g. '0.21')."""
|
||||
pkg = _load_package_json()
|
||||
raw = pkg["dependencies"]["tree-sitter"]
|
||||
match = re.search(r"(\d+)\.(\d+)", raw)
|
||||
if not match:
|
||||
raise SystemExit(f"could not parse tree-sitter version: {raw!r}")
|
||||
return f"{match.group(1)}.{match.group(2)}"
|
||||
|
||||
|
||||
def read_pinned_grammar_versions() -> dict[str, str]:
|
||||
"""Return the grammar version range pinned in gitnexus/package.json.
|
||||
|
||||
Looks at both runtime and optional dependencies. Returns the raw range
|
||||
string (e.g. '0.21.4', '^0.23.0', 'file:./vendor/...') so the report can
|
||||
expose how flexible each pin is.
|
||||
"""
|
||||
pkg = _load_package_json()
|
||||
pinned: dict[str, str] = {}
|
||||
for section in ("dependencies", "optionalDependencies"):
|
||||
for name, spec in (pkg.get(section) or {}).items():
|
||||
if name.startswith("tree-sitter-"):
|
||||
pinned[name] = spec
|
||||
return pinned
|
||||
|
||||
|
||||
def npm_view_json(pkg: str) -> dict | None:
|
||||
"""Fetch package metadata from the npm registry via HTTPS.
|
||||
|
||||
Uses the registry API directly so we don't depend on the npm CLI
|
||||
being available (it's a batch file on Windows which complicates
|
||||
subprocess calls).
|
||||
"""
|
||||
url = f"https://registry.npmjs.org/{pkg}/latest"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers={"Accept": "application/json"})
|
||||
with urllib.request.urlopen(req, timeout=8) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except (urllib.error.URLError, urllib.error.HTTPError, json.JSONDecodeError):
|
||||
return None
|
||||
|
||||
|
||||
def satisfies_target(peer_range: str | None, target: str) -> bool:
|
||||
"""Check if a semver range like '^0.22.4' or '^0.25.0' satisfies the target.
|
||||
|
||||
Simple heuristic: extract the minimum version from the range and check
|
||||
if target >= min. For caret ranges (^X.Y.Z), the upper bound is the
|
||||
next major (for X>0) or next minor (for X==0). We check both bounds.
|
||||
"""
|
||||
if peer_range is None:
|
||||
# No peer dep declared = no constraint = compatible.
|
||||
return True
|
||||
match = re.search(r"(\d+)\.(\d+)\.(\d+)", peer_range)
|
||||
if not match:
|
||||
return False
|
||||
min_major, min_minor, min_patch = int(match.group(1)), int(match.group(2)), int(match.group(3))
|
||||
|
||||
t_match = re.search(r"(\d+)\.(\d+)\.(\d+)", target)
|
||||
if not t_match:
|
||||
return False
|
||||
t_major, t_minor, t_patch = int(t_match.group(1)), int(t_match.group(2)), int(t_match.group(3))
|
||||
|
||||
# Target must be >= minimum.
|
||||
target_tuple = (t_major, t_minor, t_patch)
|
||||
min_tuple = (min_major, min_minor, min_patch)
|
||||
if target_tuple < min_tuple:
|
||||
return False
|
||||
|
||||
# For caret ranges with major 0: ^0.X.Y allows [0.X.Y, 0.(X+1).0).
|
||||
if peer_range.startswith("^") and min_major == 0:
|
||||
if t_major != 0 or t_minor >= min_minor + 1:
|
||||
return False
|
||||
# For caret ranges with major >0: ^X.Y.Z allows [X.Y.Z, (X+1).0.0).
|
||||
elif peer_range.startswith("^") and min_major > 0:
|
||||
if t_major >= min_major + 1:
|
||||
return False
|
||||
|
||||
return True
|
||||
|
||||
|
||||
_GITHUB_TOKEN = os.environ.get("GITHUB_TOKEN")
|
||||
|
||||
|
||||
def fetch_text(url: str, timeout: int = 8) -> str | None:
|
||||
"""Fetch a URL and return its text, or None on failure.
|
||||
|
||||
Adds an Authorization header for github.com URLs when GITHUB_TOKEN is
|
||||
set (raises the rate limit from 60 to 5 000 requests/hour).
|
||||
"""
|
||||
headers: dict[str, str] = {}
|
||||
if _GITHUB_TOKEN and ("github.com" in url or "githubusercontent.com" in url):
|
||||
headers["Authorization"] = f"Bearer {_GITHUB_TOKEN}"
|
||||
try:
|
||||
req = urllib.request.Request(url, headers=headers)
|
||||
with urllib.request.urlopen(req, timeout=timeout) as resp:
|
||||
return resp.read().decode("utf-8", errors="ignore")
|
||||
except (urllib.error.URLError, urllib.error.HTTPError):
|
||||
return None
|
||||
|
||||
|
||||
def extract_abi_from_text(text: str) -> int | None:
|
||||
"""Extract LANGUAGE_VERSION from parser.c text."""
|
||||
match = re.search(r"#define\s+LANGUAGE_VERSION\s+(\d+)", text[:4096])
|
||||
return int(match.group(1)) if match else None
|
||||
|
||||
|
||||
def extract_language_version(parser_c: pathlib.Path) -> int | None:
|
||||
"""Return the LANGUAGE_VERSION defined in a parser.c, or None if absent."""
|
||||
if not parser_c.is_file():
|
||||
return None
|
||||
with parser_c.open("r", encoding="utf-8", errors="ignore") as fh:
|
||||
head = fh.read(4096)
|
||||
return extract_abi_from_text(head)
|
||||
|
||||
|
||||
def md_h(text: str, level: int = 2) -> str:
|
||||
return f"{'#' * level} {text}\n"
|
||||
|
||||
|
||||
def _first_sentence(text: str) -> str:
|
||||
"""Return the leading sentence of a free-form rationale string.
|
||||
|
||||
Vendor package.json `_vendoredBy` fields often look like
|
||||
"<reason>. <install-script breadcrumb>. Do NOT <warning>." — the
|
||||
first sentence is what reviewers actually want to read; the rest is
|
||||
noise in this context. Match a sentence-ending '.' followed by
|
||||
whitespace; fall back to the whole string if nothing matches.
|
||||
"""
|
||||
text = text.strip()
|
||||
match = re.search(r"\.\s+[A-Z]", text)
|
||||
return text[: match.start() + 1] if match else text
|
||||
|
||||
|
||||
def range_includes(spec: str | None, version: str) -> bool:
|
||||
"""Return True if pinned-range `spec` accepts the concrete `version`.
|
||||
|
||||
Handles the spec shapes we actually use in package.json:
|
||||
- exact pins ('0.21.4')
|
||||
- caret / tilde ranges ('^0.23.0', '~0.23.5')
|
||||
- non-registry pins ('file:./vendor/...', 'git+...') — always False,
|
||||
because there's no meaningful "behind npm latest" comparison.
|
||||
"""
|
||||
if not spec or spec == "—":
|
||||
return False
|
||||
if spec.startswith(("file:", "git", "http")):
|
||||
return False
|
||||
if spec.startswith(("^", "~")):
|
||||
return satisfies_target(spec, version)
|
||||
return spec.strip() == version.strip()
|
||||
|
||||
|
||||
def is_vendored_pin(spec: str | None) -> bool:
|
||||
return bool(spec) and spec.startswith(("file:", "git", "http"))
|
||||
|
||||
|
||||
def vendored_drift_summary(
|
||||
name: str, upstream_repo: str, upstream_branch: str, parser_path: str
|
||||
) -> dict:
|
||||
"""Inspect a vendored grammar under gitnexus/vendor/<name>.
|
||||
|
||||
Returns the vendored package.json's ``version`` and ``_vendoredBy``
|
||||
fields (which carry the human rationale for vendoring), the vendored
|
||||
parser's ABI, and a comparison against upstream main. We deliberately
|
||||
rely on ``_vendoredBy`` rather than a parallel registry in this
|
||||
script: the rationale belongs next to the vendored sources, not in
|
||||
a daily-running CI script.
|
||||
"""
|
||||
vendor_dir = GITNEXUS_DIR / "vendor" / name
|
||||
pkg: dict = {}
|
||||
pkg_path = vendor_dir / "package.json"
|
||||
if pkg_path.is_file():
|
||||
try:
|
||||
pkg = json.loads(pkg_path.read_text(encoding="utf-8", errors="ignore"))
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
vendored_parser = vendor_dir / parser_path
|
||||
if not vendored_parser.is_file():
|
||||
vendored_parser = vendor_dir / "src" / "parser.c"
|
||||
vendored_abi = extract_language_version(vendored_parser)
|
||||
|
||||
upstream_url = (
|
||||
f"https://raw.githubusercontent.com/{upstream_repo}/"
|
||||
f"{upstream_branch}/{parser_path}"
|
||||
)
|
||||
upstream_text = fetch_text(upstream_url)
|
||||
upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None
|
||||
|
||||
sha_text = fetch_text(
|
||||
f"https://api.github.com/repos/{upstream_repo}/commits/{upstream_branch}"
|
||||
)
|
||||
upstream_sha = "?"
|
||||
if sha_text:
|
||||
try:
|
||||
upstream_sha = json.loads(sha_text).get("sha", "?")[:12]
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
local_text = (
|
||||
vendored_parser.read_text(encoding="utf-8", errors="ignore")
|
||||
if vendored_parser.is_file()
|
||||
else ""
|
||||
)
|
||||
in_sync = bool(
|
||||
upstream_text
|
||||
and local_text.replace("\r\n", "\n") == upstream_text.replace("\r\n", "\n")
|
||||
)
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"vendored_version": pkg.get("version", "?"),
|
||||
"vendored_by": pkg.get("_vendoredBy"),
|
||||
"vendored_abi": vendored_abi,
|
||||
"upstream_repo": upstream_repo,
|
||||
"upstream_branch": upstream_branch,
|
||||
"upstream_sha": upstream_sha,
|
||||
"upstream_abi": upstream_abi,
|
||||
"in_sync": in_sync,
|
||||
}
|
||||
|
||||
|
||||
# ── Main ────────────────────────────────────────────────────────────────
|
||||
|
||||
|
||||
def _classify_grammar(
|
||||
*,
|
||||
name: str,
|
||||
pinned_spec: str | None,
|
||||
npm_version: str,
|
||||
peer_range: str | None,
|
||||
fetch_failed: bool,
|
||||
target_compat: bool,
|
||||
current_compat: bool,
|
||||
upstream_progress: str | None,
|
||||
) -> dict:
|
||||
"""Decide a single primary disposition + a separate bump-now hint.
|
||||
|
||||
Buckets are mutually exclusive and ordered by what a reviewer should
|
||||
look at first:
|
||||
- fetch_failed : npm registry fetch failed (treat as blocker, but
|
||||
surface separately so reviewers don't confuse it
|
||||
with an upstream block)
|
||||
- intentional : pinned in INTENTIONAL_PINS — explicit choice
|
||||
- ready : npm-latest peer dep already accepts the target
|
||||
runtime; nothing to do
|
||||
- waiting : main has a fix (ABI 15 or relaxed peer) but no
|
||||
published npm release yet
|
||||
- blocked : peer dep too tight on both npm and main
|
||||
|
||||
Independently of bucket, `bump_now` reports whether reviewers can
|
||||
move the pin forward today without touching the runtime — we only
|
||||
suggest it when npm-latest's peer dep also accepts our *current*
|
||||
runtime, otherwise the bump would break `npm install`.
|
||||
"""
|
||||
is_vendored = is_vendored_pin(pinned_spec)
|
||||
behind_latest = (
|
||||
not is_vendored
|
||||
and npm_version != "?"
|
||||
and not range_includes(pinned_spec, npm_version)
|
||||
)
|
||||
# Intentional pins must never appear as actionable bumps — by definition
|
||||
# we're holding them back on purpose. The pin can only be lifted by
|
||||
# editing INTENTIONAL_PINS and package.json together.
|
||||
bump_now = behind_latest and current_compat and name not in INTENTIONAL_PINS
|
||||
|
||||
if fetch_failed:
|
||||
bucket = "fetch_failed"
|
||||
elif name in INTENTIONAL_PINS:
|
||||
bucket = "intentional"
|
||||
elif target_compat:
|
||||
bucket = "ready"
|
||||
elif upstream_progress:
|
||||
bucket = "waiting"
|
||||
else:
|
||||
bucket = "blocked"
|
||||
|
||||
return {
|
||||
"name": name,
|
||||
"pinned_spec": pinned_spec or "—",
|
||||
"npm_version": npm_version,
|
||||
"peer_range": peer_range,
|
||||
"target_compat": target_compat,
|
||||
"current_compat": current_compat,
|
||||
"upstream_progress": upstream_progress,
|
||||
"behind_latest": behind_latest,
|
||||
"bump_now": bump_now,
|
||||
"bucket": bucket,
|
||||
"is_vendored": is_vendored,
|
||||
}
|
||||
|
||||
|
||||
def main() -> int:
|
||||
blockers: dict[str, str] = {}
|
||||
lines: list[str] = []
|
||||
lines.append(md_h("Tree-sitter 0.25 upgrade readiness", 1))
|
||||
lines.append("")
|
||||
|
||||
current_runtime = read_current_runtime()
|
||||
current_abi_range = RUNTIME_ABI_RANGES.get(current_runtime, (0, 0))
|
||||
target_abi_range = RUNTIME_ABI_RANGES.get(TARGET_RUNTIME_MAJOR_MINOR, (0, 0))
|
||||
pinned_versions = read_pinned_grammar_versions()
|
||||
|
||||
lines.append(
|
||||
f"`tree-sitter@{current_runtime}.x` (ABI {current_abi_range[0]}–{current_abi_range[1]}) "
|
||||
f"→ target `tree-sitter@{TARGET_RUNTIME}` "
|
||||
f"(ABI {target_abi_range[0]}–{target_abi_range[1]})."
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# First pass: gather raw data + classification per grammar. We render
|
||||
# the human-friendly buckets first, then the raw matrix in a <details>
|
||||
# block at the end. Status text in the matrix is preserved verbatim
|
||||
# so the workflow's row-diff change-detection keeps working.
|
||||
grammar_rows: list[dict] = []
|
||||
raw_matrix: list[str] = [
|
||||
"| Grammar | Pinned | npm latest | Peer dep | Satisfies 0.25? | ABI | Upstream ABI | Status |",
|
||||
"|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
|
||||
vendored_grammars: list[dict] = []
|
||||
|
||||
for name, (upstream_repo, upstream_branch, parser_path) in sorted(GRAMMARS.items()):
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
|
||||
# Vendored grammars don't have an "npm latest" we install from —
|
||||
# we ship our own copy under gitnexus/vendor/<name>. Treat them
|
||||
# as a separate kind of artefact: their readiness for the runtime
|
||||
# upgrade depends on the vendored ABI being in the target range,
|
||||
# not on a peer-dep negotiation.
|
||||
if is_vendored_pin(pinned_spec):
|
||||
v = vendored_drift_summary(name, upstream_repo, upstream_branch, parser_path)
|
||||
v["pinned_spec"] = pinned_spec
|
||||
# Three-state classification: in-range, out-of-range, or
|
||||
# not-introspectable (e.g. tree-sitter-swift ships only
|
||||
# prebuilt .node binaries, no parser.c — assume compatible).
|
||||
if v["vendored_abi"] is None:
|
||||
v["target_compat"] = True
|
||||
v["abi_state"] = "prebuilt"
|
||||
status = "Vendored (prebuilt — ABI not introspectable)"
|
||||
elif target_abi_range[0] <= v["vendored_abi"] <= target_abi_range[1]:
|
||||
v["target_compat"] = True
|
||||
v["abi_state"] = "in_range"
|
||||
status = "Vendored (ABI in target range)"
|
||||
else:
|
||||
v["target_compat"] = False
|
||||
v["abi_state"] = "out_of_range"
|
||||
status = "Vendored (ABI out of range)"
|
||||
blockers[name] = (
|
||||
f"vendored `{name}`: ABI {v['vendored_abi']} outside target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]}"
|
||||
)
|
||||
# Keep vendored grammars in the raw matrix so the workflow's
|
||||
# row-diff change-detection picks up status transitions on
|
||||
# them too. npm-only columns get sentinels.
|
||||
raw_matrix.append(
|
||||
f"| `{name}` | {pinned_spec} | (vendored) | (vendored) | "
|
||||
f"{'Yes' if v['target_compat'] else '**No**'} | "
|
||||
f"{v['vendored_abi'] or '?'} | {v['upstream_abi'] or '?'} | {status} |"
|
||||
)
|
||||
vendored_grammars.append(v)
|
||||
continue
|
||||
|
||||
# Fetch latest npm metadata.
|
||||
info = npm_view_json(name)
|
||||
fetch_failed = info is None
|
||||
npm_version = "?"
|
||||
peer_range = None
|
||||
peer_optional = True
|
||||
if info:
|
||||
npm_version = info.get("version", "?")
|
||||
peers = info.get("peerDependencies") or {}
|
||||
peer_range = peers.get("tree-sitter")
|
||||
meta = info.get("peerDependenciesMeta") or {}
|
||||
ts_meta = meta.get("tree-sitter") or {}
|
||||
peer_optional = ts_meta.get("optional", False) if peer_range else True
|
||||
|
||||
if fetch_failed:
|
||||
peer_display = "? (fetch failed)"
|
||||
target_compat = False
|
||||
current_compat = False
|
||||
else:
|
||||
peer_display = peer_range or "none"
|
||||
if peer_range and not peer_optional:
|
||||
peer_display += " (required)"
|
||||
target_compat = satisfies_target(peer_range, TARGET_RUNTIME)
|
||||
current_compat = satisfies_target(peer_range, f"{current_runtime}.0")
|
||||
|
||||
# Check installed ABI using the same parser_path from GRAMMARS.
|
||||
installed_parser = GITNEXUS_DIR / "node_modules" / name / parser_path
|
||||
if not installed_parser.is_file():
|
||||
# Fallback to default location.
|
||||
installed_parser = GITNEXUS_DIR / "node_modules" / name / "src" / "parser.c"
|
||||
installed_abi = extract_language_version(installed_parser)
|
||||
abi_display = str(installed_abi) if installed_abi else "?"
|
||||
|
||||
# Check upstream (main/master branch) ABI for unreleased work.
|
||||
upstream_url = (
|
||||
f"https://raw.githubusercontent.com/{upstream_repo}/"
|
||||
f"{upstream_branch}/{parser_path}"
|
||||
)
|
||||
upstream_text = fetch_text(upstream_url)
|
||||
upstream_abi = extract_abi_from_text(upstream_text) if upstream_text else None
|
||||
upstream_abi_display = str(upstream_abi) if upstream_abi else "?"
|
||||
|
||||
# Status text + upstream-progress detection. The Status column
|
||||
# values are preserved as-is to keep the workflow's row-diff
|
||||
# change-detection working on the raw matrix below.
|
||||
upstream_progress: str | None = None
|
||||
if fetch_failed:
|
||||
status = "Unknown (fetch failed)"
|
||||
blockers[name] = f"`{name}`: npm registry fetch failed — could not verify peer dep"
|
||||
elif name in INTENTIONAL_PINS:
|
||||
# An intentional pin is, by definition, a held-back grammar:
|
||||
# whatever npm-latest's peer dep says, our shipped version is
|
||||
# the one whose ABI/peer must accept the target runtime, and
|
||||
# the pin entry exists precisely because it does not. Treat
|
||||
# it as a blocker until the pin is lifted (entry removed from
|
||||
# INTENTIONAL_PINS), at which point this grammar falls back
|
||||
# to standard classification on the next run.
|
||||
status = "Intentionally pinned"
|
||||
blockers[name] = (
|
||||
f"`{name}` intentionally pinned at `{pinned_spec}` "
|
||||
f"({INTENTIONAL_PINS[name]}) — pin must be lifted "
|
||||
f"before the {TARGET_RUNTIME} runtime upgrade"
|
||||
)
|
||||
elif target_compat:
|
||||
status = "Ready"
|
||||
elif upstream_abi and upstream_abi >= 15:
|
||||
status = "Unreleased (ABI 15 on main)"
|
||||
upstream_progress = f"ABI 15 on `{upstream_repo}@{upstream_branch}` not yet published"
|
||||
blockers[name] = f"`{name}`: ABI 15 on `{upstream_repo}` main but not published to npm"
|
||||
else:
|
||||
status = "Blocking"
|
||||
blockers[name] = f"`{name}@{npm_version}`: peer `{peer_display}` incompatible with 0.25"
|
||||
|
||||
# Also check upstream package.json for relaxed peer dep — beats
|
||||
# the ABI-15 hint when both are true.
|
||||
if not target_compat and not fetch_failed:
|
||||
upstream_pkg_url = (
|
||||
f"https://raw.githubusercontent.com/{upstream_repo}/"
|
||||
f"{upstream_branch}/package.json"
|
||||
)
|
||||
upstream_pkg_text = fetch_text(upstream_pkg_url)
|
||||
if upstream_pkg_text:
|
||||
try:
|
||||
upstream_pkg = json.loads(upstream_pkg_text)
|
||||
upstream_peer = (upstream_pkg.get("peerDependencies") or {}).get("tree-sitter")
|
||||
if upstream_peer and satisfies_target(upstream_peer, TARGET_RUNTIME):
|
||||
status = "Unreleased (peer relaxed on main)"
|
||||
upstream_progress = (
|
||||
f"peer relaxed to `{upstream_peer}` on "
|
||||
f"`{upstream_repo}@{upstream_branch}` not yet published"
|
||||
)
|
||||
blockers[name] = f"`{name}`: peer dep relaxed on `{upstream_repo}` main but not published to npm"
|
||||
except json.JSONDecodeError:
|
||||
pass
|
||||
|
||||
pinned_spec = pinned_versions.get(name, "—")
|
||||
compat_icon = "Yes" if target_compat else "**No**"
|
||||
raw_matrix.append(
|
||||
f"| `{name}` | {pinned_spec} | {npm_version} | {peer_display} | "
|
||||
f"{compat_icon} | {abi_display} | {upstream_abi_display} | {status} |"
|
||||
)
|
||||
|
||||
grammar_rows.append(_classify_grammar(
|
||||
name=name,
|
||||
pinned_spec=pinned_spec,
|
||||
npm_version=npm_version,
|
||||
peer_range=peer_range,
|
||||
fetch_failed=fetch_failed,
|
||||
target_compat=target_compat,
|
||||
current_compat=current_compat,
|
||||
upstream_progress=upstream_progress,
|
||||
))
|
||||
|
||||
# ── Bucketize ────────────────────────────────────────────────────
|
||||
by_bucket: dict[str, list[dict]] = {
|
||||
k: [] for k in ("ready", "intentional", "waiting", "blocked", "fetch_failed")
|
||||
}
|
||||
for row in grammar_rows:
|
||||
by_bucket[row["bucket"]].append(row)
|
||||
bump_now = [r for r in grammar_rows if r["bump_now"]]
|
||||
ready_count = len(by_bucket["ready"])
|
||||
|
||||
# ── TL;DR ────────────────────────────────────────────────────────
|
||||
npm_count = len(grammar_rows)
|
||||
vendored_count = len(vendored_grammars)
|
||||
vendored_ready = sum(1 for v in vendored_grammars if v["target_compat"])
|
||||
|
||||
if not blockers:
|
||||
verdict = "**Ready** — all grammars are 0.25-compatible. The runtime upgrade can proceed."
|
||||
else:
|
||||
moved = "no" if not by_bucket["waiting"] else f"yes — {len(by_bucket['waiting'])} grammars have unreleased fixes on main"
|
||||
verdict = (
|
||||
f"**Blocked** — {len(blockers)} grammars are not yet 0.25-compatible. "
|
||||
f"Upstream movement: {moved}."
|
||||
)
|
||||
|
||||
lines.append(md_h("TL;DR", 2))
|
||||
lines.append(verdict)
|
||||
lines.append("")
|
||||
lines.append(f"- {ready_count}/{npm_count} npm-installed grammars already accept tree-sitter@{TARGET_RUNTIME}")
|
||||
if vendored_count:
|
||||
lines.append(
|
||||
f"- {vendored_ready}/{vendored_count} vendored grammars at an ABI within the target runtime range"
|
||||
)
|
||||
lines.append(f"- {len(by_bucket['intentional'])} intentionally pinned (see below)")
|
||||
lines.append(f"- {len(by_bucket['waiting'])} waiting on an upstream npm release")
|
||||
lines.append(f"- {len(by_bucket['blocked'])} blocked on upstream (no fix even on main)")
|
||||
if by_bucket['fetch_failed']:
|
||||
lines.append(f"- {len(by_bucket['fetch_failed'])} could not be checked (npm registry unreachable)")
|
||||
if bump_now:
|
||||
lines.append(
|
||||
f"- **{len(bump_now)} bump candidate(s) you can take TODAY** (npm-latest "
|
||||
f"is newer than the pin AND its peer dep accepts our current runtime)"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── What you can do today ───────────────────────────────────────
|
||||
if bump_now:
|
||||
lines.append(md_h("What you can do today", 2))
|
||||
lines.append(
|
||||
"These pins lag npm latest and the latest version's peer dep already "
|
||||
"accepts our current `tree-sitter@" + current_runtime + ".x` runtime. "
|
||||
"Bumping is independent of the 0.25 upgrade and should be a quick PR."
|
||||
)
|
||||
lines.append("")
|
||||
for r in sorted(bump_now, key=lambda r: r["name"]):
|
||||
lines.append(
|
||||
f"- `{r['name']}`: `{r['pinned_spec']}` → `{r['npm_version']}` "
|
||||
f"(peer `{r['peer_range'] or 'none'}`)"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── Per-disposition sections ────────────────────────────────────
|
||||
def _emit_bucket(title: str, body_intro: str, rows: list[dict], render) -> None:
|
||||
if not rows:
|
||||
return
|
||||
lines.append(md_h(f"{title} ({len(rows)})", 3))
|
||||
lines.append(body_intro)
|
||||
lines.append("")
|
||||
for r in sorted(rows, key=lambda r: r["name"]):
|
||||
lines.append(render(r))
|
||||
lines.append("")
|
||||
|
||||
lines.append(md_h("Disposition", 2))
|
||||
|
||||
_emit_bucket(
|
||||
"Ready for 0.25",
|
||||
"These grammars' npm-latest peer dep already accepts the target runtime. No action needed for the upgrade.",
|
||||
by_bucket["ready"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}` — pinned `{r['pinned_spec']}`, npm latest `{r['npm_version']}`"
|
||||
+ (" _(also a bump candidate — see above)_" if r["bump_now"] else "")
|
||||
),
|
||||
)
|
||||
|
||||
if by_bucket["intentional"]:
|
||||
lines.append(md_h(f"Intentionally pinned ({len(by_bucket['intentional'])})", 3))
|
||||
lines.append(
|
||||
"Deliberately held below npm latest. These are **not** drift — each entry "
|
||||
"lists the issue motivating the pin and the condition for unpinning."
|
||||
)
|
||||
lines.append("")
|
||||
for r in sorted(by_bucket["intentional"], key=lambda r: r["name"]):
|
||||
reason = INTENTIONAL_PINS.get(r["name"], "(no rationale recorded)")
|
||||
lines.append(
|
||||
f"- `{r['name']}` pinned at `{r['pinned_spec']}` "
|
||||
f"(npm latest `{r['npm_version']}`)\n {reason}"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
_emit_bucket(
|
||||
"Waiting on upstream npm release",
|
||||
"Fixes are merged on the upstream main branch but not yet published to npm. "
|
||||
"We can move forward as soon as upstream cuts a release.",
|
||||
by_bucket["waiting"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`. "
|
||||
f"_{r['upstream_progress']}_"
|
||||
),
|
||||
)
|
||||
|
||||
_emit_bucket(
|
||||
"Blocked on upstream",
|
||||
"Peer dep is too tight on both the latest npm release and on upstream main. "
|
||||
"These need an upstream issue/PR before we can proceed.",
|
||||
by_bucket["blocked"],
|
||||
lambda r: (
|
||||
f"- `{r['name']}@{r['npm_version']}` — peer `{r['peer_range'] or 'none'}`"
|
||||
+ (" _(vendored)_" if r["is_vendored"] else "")
|
||||
),
|
||||
)
|
||||
|
||||
_emit_bucket(
|
||||
"Could not check",
|
||||
"npm registry fetch failed for these grammars. Re-run the workflow to retry.",
|
||||
by_bucket["fetch_failed"],
|
||||
lambda r: f"- `{r['name']}` (pinned `{r['pinned_spec']}`)",
|
||||
)
|
||||
|
||||
# ── Vendored parsers ────────────────────────────────────────────
|
||||
if vendored_grammars:
|
||||
lines.append(md_h(f"Vendored parsers ({len(vendored_grammars)})", 2))
|
||||
lines.append(
|
||||
"These grammars ship from `gitnexus/vendor/` rather than the npm "
|
||||
"registry. Their compatibility is governed by the **vendored "
|
||||
"ABI** (must lie in the target runtime's range), not by a peer-"
|
||||
"dep negotiation. The rationale for each vendored copy lives in "
|
||||
"its own `package.json` `_vendoredBy` field."
|
||||
)
|
||||
lines.append("")
|
||||
for v in sorted(vendored_grammars, key=lambda v: v["name"]):
|
||||
sync_label = (
|
||||
"in sync with upstream" if v["in_sync"] else "diverged from upstream"
|
||||
)
|
||||
if v["abi_state"] == "in_range":
|
||||
abi_label = f"ABI `{v['vendored_abi']}` (in target range)"
|
||||
elif v["abi_state"] == "prebuilt":
|
||||
abi_label = "ABI `prebuilt` (binary-only vendor, source not introspectable)"
|
||||
else:
|
||||
abi_label = (
|
||||
f"ABI `{v['vendored_abi']}` (**outside** target range "
|
||||
f"{target_abi_range[0]}..{target_abi_range[1]})"
|
||||
)
|
||||
upstream_abi_str = (
|
||||
f"ABI `{v['upstream_abi']}`" if v["upstream_abi"] else "ABI `?`"
|
||||
)
|
||||
lines.append(
|
||||
f"- **`{v['name']}`** `{v['vendored_version']}` — {abi_label}, "
|
||||
f"upstream `{v['upstream_repo']}@{v['upstream_sha']}` "
|
||||
f"{upstream_abi_str} · {sync_label}"
|
||||
)
|
||||
if v["vendored_by"]:
|
||||
# Show the first sentence — vendor package.json fields tend
|
||||
# to start with the rationale and tail off into install-
|
||||
# script breadcrumbs that aren't useful in this report.
|
||||
rationale = _first_sentence(v["vendored_by"])
|
||||
lines.append(f" - **Why vendored:** {rationale}")
|
||||
# Action computation: needs regen iff upstream ABI exceeds
|
||||
# vendored AND is still within target range. If upstream ABI
|
||||
# exceeds the target, that's a runtime-side blocker. For
|
||||
# prebuilt-only vendors we can't drive this from source ABI;
|
||||
# the action is a manual upstream-binary refresh, surfaced
|
||||
# via the in-sync flag instead.
|
||||
if v["abi_state"] == "prebuilt":
|
||||
if not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** check whether upstream has shipped a new "
|
||||
"prebuilt release; this vendor ships binary-only artefacts."
|
||||
)
|
||||
elif v["upstream_abi"] and v["vendored_abi"] and v["upstream_abi"] > v["vendored_abi"]:
|
||||
if v["upstream_abi"] <= target_abi_range[1]:
|
||||
lines.append(
|
||||
f" - **Action:** after upgrading to tree-sitter@{TARGET_RUNTIME}, "
|
||||
f"regenerate `parser.c` from upstream `{v['upstream_sha']}`."
|
||||
)
|
||||
else:
|
||||
lines.append(
|
||||
f" - **Action:** wait for a runtime supporting ABI "
|
||||
f"{v['upstream_abi']}; current target ({TARGET_RUNTIME}) only "
|
||||
f"goes up to ABI {target_abi_range[1]}."
|
||||
)
|
||||
blockers[f"vendored-{v['name']}-abi"] = (
|
||||
f"vendored {v['name']}: upstream ABI {v['upstream_abi']} outside target range"
|
||||
)
|
||||
elif not v["in_sync"]:
|
||||
lines.append(
|
||||
" - **Action:** review upstream changes; vendored copy may "
|
||||
"need a refresh (no ABI bump required)."
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
# ── Raw matrix (for completeness + workflow row-diff) ────────────
|
||||
lines.append(md_h("Full grammar matrix", 2))
|
||||
lines.append(
|
||||
"<details><summary>Click to expand the raw per-grammar table "
|
||||
"(used by the workflow's change-detection bot).</summary>\n"
|
||||
)
|
||||
lines.extend(raw_matrix)
|
||||
lines.append("\n</details>")
|
||||
lines.append("")
|
||||
|
||||
print("\n".join(lines))
|
||||
return 1 if blockers else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
# Force UTF-8 output: the report contains em-dashes and arrows that
|
||||
# Windows' default cp1252 codepage can't encode, while Linux runners
|
||||
# default to UTF-8 anyway.
|
||||
try:
|
||||
sys.stdout.reconfigure(encoding="utf-8") # type: ignore[attr-defined]
|
||||
except Exception:
|
||||
pass
|
||||
sys.exit(main())
|
||||
@@ -1,179 +0,0 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Enforce the GitHub Actions concurrency convention.
|
||||
|
||||
See CONTRIBUTING.md -> "GitHub Actions — Concurrency Convention" for the rules.
|
||||
|
||||
Invoked from .github/workflows/ci-quality.yml. Runs locally too:
|
||||
python3 .github/scripts/check-workflow-concurrency.py .github/workflows
|
||||
|
||||
Rules:
|
||||
1. Every entry-point (non-reusable) workflow declares a top-level
|
||||
`concurrency:` block.
|
||||
2. Reusable workflows (on: workflow_call ONLY) do NOT declare one.
|
||||
3. The `concurrency.group` expression MUST reference either
|
||||
`${{ github.workflow }}` or one of the approved hardcoded literal prefixes
|
||||
for workflows that are simultaneously entry-points AND reusable (on: push/
|
||||
workflow_call). Two such exceptions are currently approved:
|
||||
- `CI-` for ci.yml (the original canonical form)
|
||||
- `docker-build-push-` for docker.yml
|
||||
This is checked by substring containment rather than prefix match because
|
||||
the group value is a conditional expression that resolves to a `CI-…` or
|
||||
`docker-build-push-…` literal at runtime.
|
||||
|
||||
We deliberately do not use a YAML library — keeps the script dependency-free
|
||||
on any vanilla runner. `on:` block parsing is line-based and handles both the
|
||||
flat (`on: workflow_call`) and mapping (`on:\n workflow_call:`) forms.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import pathlib
|
||||
import re
|
||||
import sys
|
||||
|
||||
|
||||
REQUIRED_TOKENS = ("${{ github.workflow }}", "CI-", "docker-build-push-")
|
||||
|
||||
|
||||
def is_reusable(lines: list[str]) -> bool:
|
||||
"""Return True iff the workflow's `on:` block names only `workflow_call`."""
|
||||
in_on = False
|
||||
on_indent: int | None = None
|
||||
keys: list[str] = []
|
||||
|
||||
for raw in lines:
|
||||
# Skip blank lines and comments
|
||||
stripped = raw.strip()
|
||||
if not stripped or stripped.startswith("#"):
|
||||
continue
|
||||
|
||||
indent = len(raw) - len(raw.lstrip(" "))
|
||||
|
||||
if not in_on:
|
||||
if raw.startswith("on:"):
|
||||
remainder = raw[len("on:"):].strip()
|
||||
if not remainder:
|
||||
# `on:` followed by indented mapping on next lines
|
||||
in_on = True
|
||||
on_indent = indent
|
||||
continue
|
||||
if remainder.startswith("[") and remainder.endswith("]"):
|
||||
# Flow-style list: on: [workflow_call]
|
||||
items = [
|
||||
item.strip() for item in remainder.strip("[]").split(",")
|
||||
]
|
||||
return items == ["workflow_call"]
|
||||
# Scalar form: on: workflow_call (or a single other event)
|
||||
return remainder == "workflow_call"
|
||||
continue
|
||||
|
||||
# Inside the `on:` block; stop when indentation returns to <= on_indent
|
||||
if on_indent is not None and indent <= on_indent:
|
||||
break
|
||||
|
||||
# Only consider keys at on_indent + indentation step (anything deeper
|
||||
# is nested config like `types:`)
|
||||
if ":" not in stripped:
|
||||
continue
|
||||
# Heuristic: first-level event keys are those with indent == on_indent + 2
|
||||
# (the canonical step for a 2-space YAML doc). We collect all first-level
|
||||
# keys by tracking the smallest indent seen inside the block.
|
||||
keys.append((indent, stripped.split(":", 1)[0].strip()))
|
||||
|
||||
if not keys:
|
||||
return False
|
||||
|
||||
# Take only the outermost-indented keys as the event list
|
||||
min_indent = min(i for i, _ in keys)
|
||||
events = [name for i, name in keys if i == min_indent]
|
||||
return events == ["workflow_call"]
|
||||
|
||||
|
||||
CONCURRENCY_RE = re.compile(r"^concurrency:\s*$")
|
||||
GROUP_RE = re.compile(r"^\s+group:\s*(.+?)\s*$")
|
||||
|
||||
|
||||
def extract_group_key(lines: list[str]) -> str | None:
|
||||
"""Return the `group:` value of the top-level `concurrency:` block, or None."""
|
||||
for idx, raw in enumerate(lines):
|
||||
if CONCURRENCY_RE.match(raw):
|
||||
# Scan forward until we leave the concurrency block (next top-level key
|
||||
# is at column 0 and ends with `:`).
|
||||
for follow in lines[idx + 1:]:
|
||||
if follow and not follow.startswith(" ") and follow.rstrip().endswith(":"):
|
||||
break
|
||||
m = GROUP_RE.match(follow)
|
||||
if m:
|
||||
return m.group(1).strip().strip("'").strip('"')
|
||||
break
|
||||
return None
|
||||
|
||||
|
||||
def has_top_level_concurrency(lines: list[str]) -> bool:
|
||||
return any(CONCURRENCY_RE.match(raw) for raw in lines)
|
||||
|
||||
|
||||
def check(workflows_dir: pathlib.Path) -> int:
|
||||
fail = 0
|
||||
files = sorted(
|
||||
list(workflows_dir.glob("*.yml")) + list(workflows_dir.glob("*.yaml"))
|
||||
)
|
||||
for path in files:
|
||||
lines = path.read_text(encoding="utf-8").splitlines()
|
||||
reusable = is_reusable(lines)
|
||||
has_conc = has_top_level_concurrency(lines)
|
||||
|
||||
if reusable:
|
||||
if has_conc:
|
||||
print(
|
||||
f"::error file={path}::Reusable workflow (on: workflow_call) "
|
||||
"must NOT declare its own concurrency block — it inherits "
|
||||
"from the caller. See CONTRIBUTING.md -> GitHub Actions — "
|
||||
"Concurrency Convention."
|
||||
)
|
||||
fail = 1
|
||||
continue
|
||||
|
||||
if not has_conc:
|
||||
print(
|
||||
f"::error file={path}::Missing top-level concurrency block. "
|
||||
"See CONTRIBUTING.md -> GitHub Actions — Concurrency Convention."
|
||||
)
|
||||
fail = 1
|
||||
continue
|
||||
|
||||
group = extract_group_key(lines)
|
||||
if group is None:
|
||||
print(
|
||||
f"::error file={path}::concurrency block is missing a "
|
||||
"`group:` key."
|
||||
)
|
||||
fail = 1
|
||||
continue
|
||||
|
||||
if not any(token in group for token in REQUIRED_TOKENS):
|
||||
print(
|
||||
f"::error file={path}::concurrency.group `{group}` must "
|
||||
f"reference one of {REQUIRED_TOKENS} (use ${{{{ github.workflow }}}} "
|
||||
"for normal entry-point workflows; use an approved literal prefix "
|
||||
"only for workflows that are both entry-points AND reusable — "
|
||||
"see CONTRIBUTING.md -> GitHub Actions — Concurrency Convention)."
|
||||
)
|
||||
fail = 1
|
||||
|
||||
return fail
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
if len(argv) != 2:
|
||||
print(f"usage: {argv[0]} <workflows-dir>", file=sys.stderr)
|
||||
return 2
|
||||
workflows_dir = pathlib.Path(argv[1])
|
||||
if not workflows_dir.is_dir():
|
||||
print(f"not a directory: {workflows_dir}", file=sys.stderr)
|
||||
return 2
|
||||
return check(workflows_dir)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv))
|
||||
@@ -1,257 +0,0 @@
|
||||
"""Pure math utilities for triage sweep embedding analysis.
|
||||
|
||||
All functions are stateless and perform no I/O (except model loading by FastEmbed).
|
||||
Each function operates on numpy arrays and returns numpy arrays or plain Python types.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import numpy as np
|
||||
from numpy.typing import NDArray
|
||||
from fastembed import TextEmbedding
|
||||
from sklearn.decomposition import PCA
|
||||
from sklearn.covariance import EllipticEnvelope
|
||||
from sklearn.metrics.pairwise import cosine_similarity
|
||||
|
||||
# FastEmbed model — BAAI/bge-small-en-v1.5 produces 384-dimensional embeddings.
|
||||
# ~46MB quantized ONNX, runs on CPU in ~0.5s per batch of 32.
|
||||
EMBEDDING_MODEL: str = "BAAI/bge-small-en-v1.5"
|
||||
|
||||
# Embedding dimensionality (determined by model choice).
|
||||
EMBEDDING_DIM: int = 384
|
||||
|
||||
# Batch size for FastEmbed. 32 balances memory and throughput on
|
||||
# a 2-vCPU GitHub Actions runner with ~7GB RAM.
|
||||
EMBEDDING_BATCH_SIZE: int = 32
|
||||
|
||||
|
||||
def embed_texts(texts: list[str]) -> NDArray[np.float32]:
|
||||
"""Embed a list of texts into dense vectors using FastEmbed.
|
||||
|
||||
Returns an array of shape (len(texts), 384) with dtype float32.
|
||||
Empty input returns a (0, 384) array.
|
||||
"""
|
||||
if not texts:
|
||||
return np.empty((0, EMBEDDING_DIM), dtype=np.float32)
|
||||
|
||||
model = TextEmbedding(model_name=EMBEDDING_MODEL)
|
||||
vectors = list(model.embed(texts, batch_size=EMBEDDING_BATCH_SIZE))
|
||||
return np.vstack(vectors).astype(np.float32)
|
||||
|
||||
|
||||
def normalize_rows(matrix: NDArray[np.float32]) -> NDArray[np.float32]:
|
||||
"""L2-normalize each row to unit length.
|
||||
|
||||
Zero-norm rows (e.g. from empty text) remain zero vectors.
|
||||
Uses eps=1e-10 in the denominator to avoid division by zero.
|
||||
"""
|
||||
if matrix.shape[0] == 0:
|
||||
return matrix
|
||||
|
||||
norms = np.linalg.norm(matrix, axis=1, keepdims=True)
|
||||
return matrix / (norms + 1e-10)
|
||||
|
||||
|
||||
def reduce_dimensions(
|
||||
matrix: NDArray[np.float32],
|
||||
max_components: int,
|
||||
) -> NDArray[np.float32]:
|
||||
"""Reduce dimensionality via PCA.
|
||||
|
||||
Computes n_components = min(max_components, n-1, d). If n_components < 1,
|
||||
returns the matrix unchanged. Logs explained variance for observability.
|
||||
"""
|
||||
n, d = matrix.shape
|
||||
if n <= 1:
|
||||
return matrix
|
||||
|
||||
n_components = min(max_components, n - 1, d)
|
||||
if n_components < 1:
|
||||
return matrix
|
||||
|
||||
pca = PCA(n_components=n_components)
|
||||
reduced = pca.fit_transform(matrix)
|
||||
explained = pca.explained_variance_ratio_.sum()
|
||||
print(f"PCA: {d}d -> {n_components}d, explained variance: {explained:.3f}")
|
||||
return reduced.astype(np.float32)
|
||||
|
||||
|
||||
def detect_outliers(
|
||||
matrix: NDArray[np.float32],
|
||||
contamination: float = 0.1,
|
||||
iqr_multiplier: float = 3.0,
|
||||
max_outlier_pct: float = 0.05,
|
||||
) -> list[tuple[int, float]]:
|
||||
"""Flag items whose Mahalanobis distance exceeds an IQR-based cutoff.
|
||||
|
||||
Uses EllipticEnvelope (robust covariance via MCD) to estimate the
|
||||
multivariate Gaussian, then computes sqrt(squared Mahalanobis distance)
|
||||
for each sample. The cutoff is Q75 + iqr_multiplier * IQR, which
|
||||
adapts to the actual distribution of distances.
|
||||
|
||||
A hard cap ensures no more than max_outlier_pct * n items are flagged;
|
||||
when the cap is hit, only the most extreme items (sorted by distance
|
||||
descending) are kept.
|
||||
|
||||
Returns (index, distance) tuples sorted by index ascending, along with
|
||||
the cutoff value stored as an attribute on the returned list.
|
||||
"""
|
||||
n = matrix.shape[0]
|
||||
if n < 2:
|
||||
return []
|
||||
|
||||
envelope = EllipticEnvelope(contamination=contamination, random_state=42)
|
||||
envelope.fit(matrix)
|
||||
|
||||
# .mahalanobis() returns squared Mahalanobis distances
|
||||
distances = np.sqrt(envelope.mahalanobis(matrix))
|
||||
|
||||
# IQR-based cutoff
|
||||
q25, q75 = np.percentile(distances, [25, 75])
|
||||
iqr = q75 - q25
|
||||
cutoff = q75 + iqr_multiplier * iqr
|
||||
|
||||
outlier_mask = distances > cutoff
|
||||
indices = np.where(outlier_mask)[0]
|
||||
|
||||
# Hard cap: keep at most max_outlier_pct * n items
|
||||
max_count = max(1, int(max_outlier_pct * n))
|
||||
if len(indices) > max_count:
|
||||
# Sort by distance descending, take the most extreme
|
||||
sorted_by_dist = sorted(indices, key=lambda i: distances[i], reverse=True)
|
||||
indices = np.array(sorted_by_dist[:max_count])
|
||||
|
||||
# Sort by index ascending for stable output
|
||||
indices = np.sort(indices)
|
||||
result = [(int(idx), float(distances[idx])) for idx in indices]
|
||||
|
||||
# Attach cutoff as metadata so the report can use it
|
||||
result = _OutlierResult(result) # type: ignore[assignment]
|
||||
result.cutoff = float(cutoff) # type: ignore[attr-defined]
|
||||
return result # type: ignore[return-value]
|
||||
|
||||
|
||||
class _OutlierResult(list):
|
||||
"""A list subclass that carries metadata (cutoff) from outlier detection."""
|
||||
cutoff: float = 0.0
|
||||
|
||||
|
||||
def find_duplicate_pairs(
|
||||
matrix: NDArray[np.float32],
|
||||
threshold: float,
|
||||
) -> list[tuple[int, int, float]]:
|
||||
"""Find pairs of items with cosine similarity above threshold.
|
||||
|
||||
Returns (i, j, similarity) tuples where i < j. The input should be
|
||||
L2-normalized embeddings (full dimensionality, not PCA-reduced) so
|
||||
cosine similarity equals the dot product.
|
||||
"""
|
||||
n = matrix.shape[0]
|
||||
if n <= 1:
|
||||
return []
|
||||
|
||||
sim_matrix = cosine_similarity(matrix)
|
||||
# Upper triangle indices (i < j), excluding diagonal
|
||||
rows, cols = np.triu_indices(n, k=1)
|
||||
similarities = sim_matrix[rows, cols]
|
||||
|
||||
mask = similarities > threshold
|
||||
pairs: list[tuple[int, int, float]] = []
|
||||
for idx in np.where(mask)[0]:
|
||||
pairs.append((int(rows[idx]), int(cols[idx]), float(similarities[idx])))
|
||||
|
||||
return pairs
|
||||
|
||||
|
||||
# ── Label suggestion via z-score normalized embedding similarity ──────
|
||||
|
||||
# Z-score threshold: a label must be this many standard deviations above
|
||||
# the column mean to be considered a match.
|
||||
LABEL_Z_THRESHOLD: float = 1.5
|
||||
|
||||
# Margin gate: the top-1 label must beat the second-best by this many
|
||||
# z-score units to be accepted (subsequent labels don't need a margin).
|
||||
LABEL_Z_MARGIN: float = 0.5
|
||||
|
||||
# Floor for per-column standard deviation to avoid division by near-zero.
|
||||
LABEL_Z_STD_FLOOR: float = 0.01
|
||||
|
||||
# Minimum raw cosine similarity required even if z-score is high.
|
||||
# Prevents suggesting labels that are "relatively best" but still poor.
|
||||
MIN_RAW_SIMILARITY: float = 0.3
|
||||
|
||||
# Maximum number of labels to suggest per item.
|
||||
MAX_LABELS_PER_ITEM: int = 3
|
||||
|
||||
|
||||
def suggest_labels(
|
||||
item_embeddings: NDArray[np.float32],
|
||||
label_embeddings: NDArray[np.float32],
|
||||
label_names: list[str],
|
||||
z_threshold: float = LABEL_Z_THRESHOLD,
|
||||
z_margin: float = LABEL_Z_MARGIN,
|
||||
std_floor: float = LABEL_Z_STD_FLOOR,
|
||||
min_raw_sim: float = MIN_RAW_SIMILARITY,
|
||||
max_per_item: int = MAX_LABELS_PER_ITEM,
|
||||
) -> list[list[tuple[str, float]]]:
|
||||
"""Suggest labels for each item using z-score normalized similarity.
|
||||
|
||||
1. Compute raw cosine similarity matrix (n items x m labels).
|
||||
2. Column-wise z-score: for each label j, normalize across all items.
|
||||
3. For each item, rank labels by z-score descending.
|
||||
4. Accept a label only if z >= z_threshold AND raw_sim >= min_raw_sim.
|
||||
5. Margin gate: the top-1 label must beat #2 by z_margin; subsequent
|
||||
labels don't need a margin.
|
||||
6. Cap at max_per_item.
|
||||
|
||||
Returns a list of length n, where each element is a list of
|
||||
(label_name, raw_similarity) tuples. Empty list if nothing qualifies.
|
||||
"""
|
||||
n = item_embeddings.shape[0]
|
||||
m = label_embeddings.shape[0]
|
||||
if n == 0 or m == 0:
|
||||
return [[] for _ in range(n)]
|
||||
|
||||
# (n, m) raw similarity matrix
|
||||
sim_matrix = cosine_similarity(item_embeddings, label_embeddings)
|
||||
|
||||
# Column-wise z-score normalization
|
||||
col_means = sim_matrix.mean(axis=0) # shape (m,)
|
||||
col_stds = sim_matrix.std(axis=0) # shape (m,)
|
||||
col_stds = np.maximum(col_stds, std_floor)
|
||||
z_matrix = (sim_matrix - col_means) / col_stds
|
||||
|
||||
suggestions: list[list[tuple[str, float]]] = []
|
||||
for i in range(n):
|
||||
z_row = z_matrix[i]
|
||||
raw_row = sim_matrix[i]
|
||||
|
||||
# Rank labels by z-score descending
|
||||
ranked = np.argsort(z_row)[::-1]
|
||||
|
||||
item_labels: list[tuple[str, float]] = []
|
||||
|
||||
# Margin gate: top-1 z-score must beat #2 by z_margin.
|
||||
# If not, the assignment is ambiguous — skip this item entirely.
|
||||
if len(ranked) > 1:
|
||||
top1_z = float(z_row[ranked[0]])
|
||||
top2_z = float(z_row[ranked[1]])
|
||||
if top1_z - top2_z < z_margin:
|
||||
suggestions.append(item_labels)
|
||||
continue
|
||||
|
||||
for rank_pos, idx in enumerate(ranked):
|
||||
if len(item_labels) >= max_per_item:
|
||||
break
|
||||
|
||||
z_val = float(z_row[idx])
|
||||
raw_val = float(raw_row[idx])
|
||||
|
||||
# Must pass both z-threshold and raw similarity floor
|
||||
if z_val < z_threshold or raw_val < min_raw_sim:
|
||||
continue
|
||||
|
||||
item_labels.append((label_names[idx], raw_val))
|
||||
|
||||
suggestions.append(item_labels)
|
||||
|
||||
return suggestions
|
||||
@@ -1,4 +0,0 @@
|
||||
fastembed>=0.5.0
|
||||
numpy>=1.26.0
|
||||
scikit-learn>=1.4.0
|
||||
scipy>=1.10.0
|
||||
@@ -1,600 +0,0 @@
|
||||
"""Triage sweep: fetch open issues/PRs, detect outliers and duplicates, generate a report.
|
||||
|
||||
Entrypoint script for the triage-sweep workflow. Fetches all open items via
|
||||
the GitHub REST API, delegates embedding and analysis to embedding_utils,
|
||||
generates a markdown report, and optionally creates a report issue.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import urllib.request
|
||||
import urllib.parse
|
||||
from typing import TypedDict
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from embedding_utils import (
|
||||
embed_texts,
|
||||
normalize_rows,
|
||||
reduce_dimensions,
|
||||
detect_outliers,
|
||||
find_duplicate_pairs,
|
||||
suggest_labels,
|
||||
LABEL_Z_THRESHOLD,
|
||||
LABEL_Z_MARGIN,
|
||||
LABEL_Z_STD_FLOOR,
|
||||
MIN_RAW_SIMILARITY,
|
||||
MAX_LABELS_PER_ITEM,
|
||||
)
|
||||
|
||||
# ── Thresholds (overridable via workflow_dispatch inputs) ──────────────
|
||||
|
||||
# IQR multiplier for outlier cutoff: cutoff = Q75 + IQR_MULTIPLIER * IQR.
|
||||
IQR_MULTIPLIER: float = float(os.environ.get("INPUT_IQR_MULTIPLIER", "3.0"))
|
||||
|
||||
# Hard cap: at most this fraction of items can be flagged as outliers.
|
||||
MAX_OUTLIER_PCT: float = float(os.environ.get("INPUT_MAX_OUTLIER_PCT", "0.05"))
|
||||
|
||||
# EllipticEnvelope contamination: expected fraction of outliers in the data.
|
||||
# Governs how aggressively the robust covariance downweights extreme points.
|
||||
CONTAMINATION: float = float(os.environ.get("INPUT_CONTAMINATION", "0.1"))
|
||||
|
||||
# Cosine similarity above which two items are flagged as duplicates.
|
||||
# 0.92 catches near-identical issues while tolerating paraphrasing.
|
||||
COSINE_THRESHOLD: float = float(os.environ.get("INPUT_COSINE_THRESHOLD", "0.92"))
|
||||
|
||||
# Hard cap on items to process. Prevents runaway costs on very large repos.
|
||||
MAX_ITEMS: int = int(os.environ.get("INPUT_MAX_ITEMS", "500"))
|
||||
|
||||
# When true, print report to stdout/file but do not create a GitHub issue.
|
||||
DRY_RUN: bool = os.environ.get("INPUT_DRY_RUN", "false").lower() == "true"
|
||||
|
||||
# ── Fixed constants (not user-configurable) ───────────────────────────
|
||||
|
||||
# Minimum number of samples required for EllipticEnvelope to fit
|
||||
# a Gaussian reliably. Must be >= 3 * PCA_MAX_COMPONENTS so the
|
||||
# covariance matrix is estimated from enough data points.
|
||||
PCA_MAX_COMPONENTS: int = 20
|
||||
MIN_SAMPLES_FOR_OUTLIER_DETECTION: int = 100
|
||||
|
||||
# Max character length for embedding input text. bge-small-en-v1.5 has a
|
||||
# 512-token context window (~4 chars/token). We keep title + body under
|
||||
# this limit so the model sees the full text instead of silently truncating.
|
||||
MAX_EMBED_CHARS: int = 2000
|
||||
|
||||
# GitHub REST API page size (max allowed is 100).
|
||||
API_PAGE_SIZE: int = 100
|
||||
|
||||
# Report issue label.
|
||||
REPORT_LABEL: str = "triage-report"
|
||||
|
||||
# Report file path (written for the summary step to pick up).
|
||||
REPORT_FILE: str = "/tmp/triage-report.md"
|
||||
|
||||
|
||||
class TriageItem(TypedDict):
|
||||
"""One open issue or PR, with only the fields we need."""
|
||||
number: int
|
||||
title: str
|
||||
html_url: str
|
||||
is_pr: bool
|
||||
labels: list[str]
|
||||
created_at: str
|
||||
# title + body concatenated, used as embedding input
|
||||
text: str
|
||||
|
||||
|
||||
def github_api_get(path: str) -> list[dict]:
|
||||
"""Make a single authenticated GET request to the GitHub REST API.
|
||||
|
||||
Reads GITHUB_TOKEN and GITHUB_REPOSITORY from env. Raises SystemExit
|
||||
with the HTTP status and response body on any non-2xx response.
|
||||
"""
|
||||
token = os.environ["GITHUB_TOKEN"]
|
||||
repo = os.environ["GITHUB_REPOSITORY"]
|
||||
url = f"https://api.github.com/repos/{repo}{path}"
|
||||
|
||||
req = urllib.request.Request(url)
|
||||
req.add_header("Accept", "application/vnd.github+json")
|
||||
req.add_header("Authorization", f"Bearer {token}")
|
||||
req.add_header("X-GitHub-Api-Version", "2022-11-28")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
return json.loads(resp.read().decode("utf-8"))
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
print(f"::error::GitHub API {e.code}: {body}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def fetch_all_open_items() -> list[TriageItem]:
|
||||
"""Paginate through all open issues and PRs.
|
||||
|
||||
Returns up to MAX_ITEMS TriageItem dicts. Items with a pull_request
|
||||
key are marked is_pr=True. The text field is title + body concatenated.
|
||||
"""
|
||||
items: list[TriageItem] = []
|
||||
page = 1
|
||||
|
||||
while len(items) < MAX_ITEMS:
|
||||
path = (
|
||||
f"/issues?state=open&per_page={API_PAGE_SIZE}"
|
||||
f"&sort=created&direction=desc&page={page}"
|
||||
)
|
||||
data = github_api_get(path)
|
||||
|
||||
if not data:
|
||||
break
|
||||
|
||||
for raw in data:
|
||||
if len(items) >= MAX_ITEMS:
|
||||
break
|
||||
|
||||
body = raw.get("body", "") or ""
|
||||
full_text = f"{raw['title']}\n\n{body}"
|
||||
# Truncate to fit the embedding model's token window.
|
||||
# Title is always preserved; body gets clipped if needed.
|
||||
if len(full_text) > MAX_EMBED_CHARS:
|
||||
full_text = full_text[:MAX_EMBED_CHARS]
|
||||
items.append(TriageItem(
|
||||
number=raw["number"],
|
||||
title=raw["title"],
|
||||
html_url=raw["html_url"],
|
||||
is_pr="pull_request" in raw,
|
||||
labels=[lbl["name"] for lbl in raw.get("labels", [])],
|
||||
created_at=raw["created_at"],
|
||||
text=full_text,
|
||||
))
|
||||
|
||||
if len(data) < API_PAGE_SIZE:
|
||||
break
|
||||
|
||||
page += 1
|
||||
|
||||
return items
|
||||
|
||||
|
||||
class RepoLabel(TypedDict):
|
||||
"""A label from the repo with its embedding text."""
|
||||
name: str
|
||||
description: str
|
||||
# "name: description" concatenated for embedding
|
||||
text: str
|
||||
|
||||
|
||||
def fetch_repo_labels() -> list[RepoLabel]:
|
||||
"""Fetch all labels from the repository, paginating if needed.
|
||||
|
||||
Returns labels with name, description, and a text field suitable
|
||||
for embedding ("name: description"). Labels with no description
|
||||
use just the name.
|
||||
"""
|
||||
labels: list[RepoLabel] = []
|
||||
page = 1
|
||||
|
||||
while True:
|
||||
data = github_api_get(f"/labels?per_page={API_PAGE_SIZE}&page={page}")
|
||||
for raw in data:
|
||||
name = raw["name"]
|
||||
desc = raw.get("description", "") or ""
|
||||
text = f"{name}: {desc}" if desc else name
|
||||
labels.append(RepoLabel(name=name, description=desc, text=text))
|
||||
|
||||
if len(data) < API_PAGE_SIZE:
|
||||
break
|
||||
page += 1
|
||||
|
||||
return labels
|
||||
|
||||
|
||||
def apply_labels_to_item(item_number: int, labels: list[str]) -> None:
|
||||
"""Add labels to a single issue/PR via the GitHub API.
|
||||
|
||||
Skips silently if labels list is empty. Uses POST which adds labels
|
||||
without removing existing ones.
|
||||
"""
|
||||
if not labels:
|
||||
return
|
||||
|
||||
token = os.environ["GITHUB_TOKEN"]
|
||||
repo = os.environ["GITHUB_REPOSITORY"]
|
||||
url = f"https://api.github.com/repos/{repo}/issues/{item_number}/labels"
|
||||
|
||||
payload = json.dumps({"labels": labels}).encode("utf-8")
|
||||
req = urllib.request.Request(url, data=payload, method="POST")
|
||||
req.add_header("Accept", "application/vnd.github+json")
|
||||
req.add_header("Authorization", f"Bearer {token}")
|
||||
req.add_header("X-GitHub-Api-Version", "2022-11-28")
|
||||
req.add_header("Content-Type", "application/json")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
resp.read()
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
# Non-fatal: log warning but don't abort the sweep
|
||||
print(f"::warning::Failed to label #{item_number}: {e.code} {body}")
|
||||
|
||||
|
||||
def _item_age(created_at: str) -> str:
|
||||
"""Compute a human-readable age string from an ISO 8601 created_at timestamp."""
|
||||
try:
|
||||
created = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
|
||||
delta = datetime.now(timezone.utc) - created
|
||||
days = delta.days
|
||||
if days < 1:
|
||||
return "<1d"
|
||||
if days < 30:
|
||||
return f"{days}d"
|
||||
if days < 365:
|
||||
return f"{days // 30}mo"
|
||||
return f"{days // 365}y"
|
||||
except (ValueError, TypeError):
|
||||
return "?"
|
||||
|
||||
|
||||
def _suggested_action(a: TriageItem, b: TriageItem) -> str:
|
||||
"""Determine a suggested action for a duplicate pair based on types and age."""
|
||||
if a["is_pr"] and b["is_pr"]:
|
||||
return "Review for overlap"
|
||||
if not a["is_pr"] and not b["is_pr"]:
|
||||
# Both issues — close the newer one
|
||||
try:
|
||||
a_dt = datetime.fromisoformat(a["created_at"].replace("Z", "+00:00"))
|
||||
b_dt = datetime.fromisoformat(b["created_at"].replace("Z", "+00:00"))
|
||||
newer = b if b_dt > a_dt else a
|
||||
except (ValueError, TypeError):
|
||||
newer = b
|
||||
return f"Close #{newer['number']} as duplicate"
|
||||
# One issue, one PR
|
||||
return "Link PR to issue"
|
||||
|
||||
|
||||
def generate_report(
|
||||
items: list[TriageItem],
|
||||
outlier_results: list[tuple[int, float]],
|
||||
duplicate_pairs: list[tuple[int, int, float]],
|
||||
label_suggestions: list[list[tuple[str, float]]] | None = None,
|
||||
) -> str:
|
||||
"""Generate a structured markdown triage report."""
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
|
||||
repo = os.environ.get("GITHUB_REPOSITORY", "unknown/repo")
|
||||
|
||||
# Compute label suggestion counts early for the health table
|
||||
outlier_set = {idx for idx, _ in outlier_results}
|
||||
suggested_count = 0
|
||||
if label_suggestions is not None:
|
||||
suggested_count = sum(
|
||||
1 for i, s in enumerate(label_suggestions)
|
||||
if s and not items[i]["labels"] and i not in outlier_set
|
||||
)
|
||||
|
||||
# ── Health summary table at the top ──────────────────────────────
|
||||
lines: list[str] = [
|
||||
"## Triage Sweep Report",
|
||||
"",
|
||||
f"**Run:** {now} UTC",
|
||||
f"**Items analyzed:** {len(items)}",
|
||||
f"**Thresholds:** IQR multiplier {IQR_MULTIPLIER}, Cosine > {COSINE_THRESHOLD}",
|
||||
"",
|
||||
"### Health Summary",
|
||||
"",
|
||||
"| Metric | Value |",
|
||||
"|--------|-------|",
|
||||
f"| Items analyzed | {len(items)} |",
|
||||
f"| Outliers flagged | {len(outlier_results)} |",
|
||||
f"| Duplicate pairs | {len(duplicate_pairs)} |",
|
||||
f"| Label suggestions | {suggested_count} |",
|
||||
"",
|
||||
]
|
||||
|
||||
# ── Outlier section ──────────────────────────────────────────────
|
||||
# Determine cutoff for high-confidence split
|
||||
cutoff = getattr(outlier_results, "cutoff", 0.0)
|
||||
high_conf_cutoff = 2 * cutoff if cutoff > 0 else float("inf")
|
||||
|
||||
high_conf = [(idx, d) for idx, d in outlier_results if d > high_conf_cutoff]
|
||||
borderline = [(idx, d) for idx, d in outlier_results if d <= high_conf_cutoff]
|
||||
|
||||
lines.extend([
|
||||
f"### Potential Outliers / Spam ({len(outlier_results)})",
|
||||
"",
|
||||
"Items with unusually high Mahalanobis distance from the distribution center.",
|
||||
"These may be spam, off-topic, or poorly described.",
|
||||
"",
|
||||
])
|
||||
|
||||
if high_conf:
|
||||
lines.append(f"**High Confidence** ({len(high_conf)} items, distance > 2x cutoff)")
|
||||
lines.append("")
|
||||
lines.append("| # | Type | Title | Distance | Age |")
|
||||
lines.append("|---|------|-------|----------|-----|")
|
||||
for idx, distance in high_conf:
|
||||
item = items[idx]
|
||||
kind = "PR" if item["is_pr"] else "Issue"
|
||||
age = _item_age(item["created_at"])
|
||||
title = item["title"][:80] + ("..." if len(item["title"]) > 80 else "")
|
||||
lines.append(
|
||||
f"| [#{item['number']}]({item['html_url']}) "
|
||||
f"| {kind} | {title} | {distance:.2f} | {age} |"
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if borderline:
|
||||
lines.append("<details>")
|
||||
lines.append(f"<summary>Borderline ({len(borderline)} items)</summary>")
|
||||
lines.append("")
|
||||
lines.append("| # | Type | Title | Distance | Age |")
|
||||
lines.append("|---|------|-------|----------|-----|")
|
||||
for idx, distance in borderline:
|
||||
item = items[idx]
|
||||
kind = "PR" if item["is_pr"] else "Issue"
|
||||
age = _item_age(item["created_at"])
|
||||
title = item["title"][:80] + ("..." if len(item["title"]) > 80 else "")
|
||||
lines.append(
|
||||
f"| [#{item['number']}]({item['html_url']}) "
|
||||
f"| {kind} | {title} | {distance:.2f} | {age} |"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("</details>")
|
||||
lines.append("")
|
||||
|
||||
if not outlier_results:
|
||||
lines.append("None found.")
|
||||
|
||||
# ── Duplicate pairs section ──────────────────────────────────────
|
||||
lines.extend([
|
||||
"",
|
||||
f"### Potential Duplicates ({len(duplicate_pairs)} pairs)",
|
||||
"",
|
||||
"Pairs of items with cosine similarity above the threshold.",
|
||||
"",
|
||||
])
|
||||
|
||||
if duplicate_pairs:
|
||||
lines.append("| Item A | Item B | Similarity | Suggested Action |")
|
||||
lines.append("|--------|--------|------------|------------------|")
|
||||
for i, j, sim in duplicate_pairs:
|
||||
a = items[i]
|
||||
b = items[j]
|
||||
kind_a = "PR" if a["is_pr"] else "Issue"
|
||||
kind_b = "PR" if b["is_pr"] else "Issue"
|
||||
action = _suggested_action(a, b)
|
||||
lines.append(
|
||||
f"| [#{a['number']}]({a['html_url']}) {kind_a}: {a['title']} "
|
||||
f"| [#{b['number']}]({b['html_url']}) {kind_b}: {b['title']} "
|
||||
f"| {sim:.3f} | {action} |"
|
||||
)
|
||||
else:
|
||||
lines.append("None found.")
|
||||
|
||||
# ── Label suggestions section ────────────────────────────────────
|
||||
if label_suggestions is not None:
|
||||
# High confidence: top-1 label with raw_sim >= 0.5
|
||||
# Low confidence: top-1 label with raw_sim < 0.5
|
||||
high_conf_labels: list[tuple[int, list[tuple[str, float]]]] = []
|
||||
low_conf_labels: list[tuple[int, list[tuple[str, float]]]] = []
|
||||
for i, sugs in enumerate(label_suggestions):
|
||||
if sugs and not items[i]["labels"] and i not in outlier_set:
|
||||
top1 = sugs[:1]
|
||||
if top1[0][1] >= 0.5:
|
||||
high_conf_labels.append((i, top1))
|
||||
else:
|
||||
low_conf_labels.append((i, top1))
|
||||
|
||||
total_suggestions = len(high_conf_labels) + len(low_conf_labels)
|
||||
lines.extend([
|
||||
"",
|
||||
f"### Suggested Labels ({total_suggestions} unlabeled items)",
|
||||
"",
|
||||
"Labels suggested by z-score normalized embedding similarity against repo label descriptions.",
|
||||
"Only shown for unlabeled items that were not flagged as outliers.",
|
||||
"",
|
||||
])
|
||||
|
||||
# Label concentration warning
|
||||
if total_suggestions > 0:
|
||||
label_counts: dict[str, int] = {}
|
||||
for _, sugs in high_conf_labels + low_conf_labels:
|
||||
for name, _ in sugs:
|
||||
label_counts[name] = label_counts.get(name, 0) + 1
|
||||
for name, count in label_counts.items():
|
||||
if count > total_suggestions * 0.5:
|
||||
lines.append(
|
||||
f"> **Warning:** Label `{name}` accounts for "
|
||||
f"{count}/{total_suggestions} suggestions "
|
||||
f"({count * 100 // total_suggestions}%). "
|
||||
f"Consider reviewing label descriptions for specificity."
|
||||
)
|
||||
lines.append("")
|
||||
|
||||
if high_conf_labels:
|
||||
lines.append("| # | Type | Title | Suggested Label |")
|
||||
lines.append("|---|------|-------|--------------------|")
|
||||
for idx, sugs in high_conf_labels:
|
||||
item = items[idx]
|
||||
kind = "PR" if item["is_pr"] else "Issue"
|
||||
label_strs = [f"`{name}` ({score:.2f})" for name, score in sugs]
|
||||
lines.append(
|
||||
f"| [#{item['number']}]({item['html_url']}) "
|
||||
f"| {kind} | {item['title']} | {', '.join(label_strs)} |"
|
||||
)
|
||||
|
||||
if low_conf_labels:
|
||||
lines.append("")
|
||||
lines.append("<details>")
|
||||
lines.append(f"<summary>Low-confidence suggestions ({len(low_conf_labels)} items)</summary>")
|
||||
lines.append("")
|
||||
lines.append("| # | Type | Title | Suggested Label |")
|
||||
lines.append("|---|------|-------|--------------------|")
|
||||
for idx, sugs in low_conf_labels:
|
||||
item = items[idx]
|
||||
kind = "PR" if item["is_pr"] else "Issue"
|
||||
label_strs = [f"`{name}` ({score:.2f})" for name, score in sugs]
|
||||
lines.append(
|
||||
f"| [#{item['number']}]({item['html_url']}) "
|
||||
f"| {kind} | {item['title']} | {', '.join(label_strs)} |"
|
||||
)
|
||||
lines.append("")
|
||||
lines.append("</details>")
|
||||
|
||||
if not high_conf_labels and not low_conf_labels:
|
||||
lines.append("No unlabeled items need suggestions.")
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"### Summary",
|
||||
"",
|
||||
f"- {len(outlier_results)} outliers flagged for review",
|
||||
f"- {len(duplicate_pairs)} duplicate pairs found",
|
||||
f"- {len(items)} items analyzed in total",
|
||||
])
|
||||
|
||||
if label_suggestions is not None:
|
||||
lines.append(f"- {suggested_count} items suggested for labeling")
|
||||
|
||||
lines.extend([
|
||||
"",
|
||||
"---",
|
||||
f"*Generated by [triage-sweep](https://github.com/{repo}/actions) — no LLM was used.*",
|
||||
])
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def create_report_issue(report_body: str) -> None:
|
||||
"""Create a GitHub issue with the triage report.
|
||||
|
||||
Posts to the issues API with the triage-report label.
|
||||
Raises SystemExit on non-201 response.
|
||||
"""
|
||||
token = os.environ["GITHUB_TOKEN"]
|
||||
repo = os.environ["GITHUB_REPOSITORY"]
|
||||
url = f"https://api.github.com/repos/{repo}/issues"
|
||||
|
||||
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
|
||||
payload = json.dumps({
|
||||
"title": f"Triage Sweep Report — {today}",
|
||||
"body": report_body,
|
||||
"labels": [REPORT_LABEL],
|
||||
}).encode("utf-8")
|
||||
|
||||
req = urllib.request.Request(url, data=payload, method="POST")
|
||||
req.add_header("Accept", "application/vnd.github+json")
|
||||
req.add_header("Authorization", f"Bearer {token}")
|
||||
req.add_header("X-GitHub-Api-Version", "2022-11-28")
|
||||
req.add_header("Content-Type", "application/json")
|
||||
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as resp:
|
||||
resp_body = resp.read().decode("utf-8")
|
||||
if resp.status != 201:
|
||||
print(f"::error::Failed to create issue: {resp.status} {resp_body}")
|
||||
sys.exit(1)
|
||||
result = json.loads(resp_body)
|
||||
print(f"Created issue: {result.get('html_url', 'unknown')}")
|
||||
except urllib.error.HTTPError as e:
|
||||
body = e.read().decode("utf-8", errors="replace")
|
||||
print(f"::error::Failed to create issue: {e.code} {body}")
|
||||
sys.exit(1)
|
||||
|
||||
|
||||
def write_report(report: str) -> None:
|
||||
"""Write the report to the file system for the summary step."""
|
||||
with open(REPORT_FILE, "w", encoding="utf-8") as f:
|
||||
f.write(report)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
"""Orchestrate the full triage sweep."""
|
||||
# 1. Validate environment
|
||||
for var in ("GITHUB_TOKEN", "GITHUB_REPOSITORY"):
|
||||
if not os.environ.get(var):
|
||||
print(f"::error::Missing required environment variable: {var}")
|
||||
sys.exit(1)
|
||||
|
||||
# 2. Fetch all open issues + PRs
|
||||
items = fetch_all_open_items()
|
||||
print(f"Fetched {len(items)} open items")
|
||||
|
||||
if len(items) == 0:
|
||||
report = "## Triage Sweep Report\n\nNo open issues or PRs found."
|
||||
write_report(report)
|
||||
print("No items to analyze.")
|
||||
return
|
||||
|
||||
# 3. Extract texts for embedding
|
||||
texts: list[str] = [item["text"] for item in items]
|
||||
|
||||
# 4. Embed all texts (returns numpy float32 array of shape [n, 384])
|
||||
embeddings = embed_texts(texts)
|
||||
|
||||
# 5. L2-normalize
|
||||
embeddings = normalize_rows(embeddings)
|
||||
|
||||
# 6. Outlier detection (Mahalanobis via EllipticEnvelope)
|
||||
outlier_results: list[tuple[int, float]] = []
|
||||
if len(items) >= MIN_SAMPLES_FOR_OUTLIER_DETECTION:
|
||||
reduced = reduce_dimensions(embeddings, PCA_MAX_COMPONENTS)
|
||||
outlier_results = detect_outliers(
|
||||
reduced,
|
||||
contamination=CONTAMINATION,
|
||||
iqr_multiplier=IQR_MULTIPLIER,
|
||||
max_outlier_pct=MAX_OUTLIER_PCT,
|
||||
)
|
||||
else:
|
||||
print(
|
||||
f"Skipping outlier detection: {len(items)} items < "
|
||||
f"{MIN_SAMPLES_FOR_OUTLIER_DETECTION} minimum"
|
||||
)
|
||||
|
||||
# 7. Duplicate detection (pairwise cosine similarity)
|
||||
duplicate_pairs = find_duplicate_pairs(embeddings, COSINE_THRESHOLD)
|
||||
|
||||
# 8. Label suggestion via embedding similarity
|
||||
label_suggestions: list[list[tuple[str, float]]] | None = None
|
||||
repo_labels = fetch_repo_labels()
|
||||
if repo_labels:
|
||||
label_texts = [lbl["text"] for lbl in repo_labels]
|
||||
label_names = [lbl["name"] for lbl in repo_labels]
|
||||
label_embeddings = embed_texts(label_texts)
|
||||
label_embeddings = normalize_rows(label_embeddings)
|
||||
label_suggestions = suggest_labels(embeddings, label_embeddings, label_names)
|
||||
print(f"Computed label suggestions against {len(repo_labels)} repo labels")
|
||||
|
||||
# NOTE: Auto-labeling is disabled. The report shows suggestions for
|
||||
# human review. To re-enable, uncomment the block below.
|
||||
#
|
||||
# # Apply top label to unlabeled items (unless dry run)
|
||||
# # Skip outliers — flagged items shouldn't get categorized
|
||||
# outlier_set = {idx for idx, _ in outlier_results}
|
||||
# if not DRY_RUN:
|
||||
# applied_count = 0
|
||||
# for i, sugs in enumerate(label_suggestions):
|
||||
# if sugs and not items[i]["labels"] and i not in outlier_set:
|
||||
# # Apply only the top-1 label (highest confidence)
|
||||
# apply_labels_to_item(items[i]["number"], [sugs[0][0]])
|
||||
# applied_count += 1
|
||||
# print(f"Applied labels to {applied_count} unlabeled items")
|
||||
else:
|
||||
print("No repo labels found — skipping label suggestions")
|
||||
|
||||
# 9. Generate report
|
||||
report = generate_report(items, outlier_results, duplicate_pairs, label_suggestions)
|
||||
|
||||
# 10. Write report to file (for summary step)
|
||||
write_report(report)
|
||||
|
||||
# 11. Create report issue (unless dry run)
|
||||
if DRY_RUN:
|
||||
print("Dry run — skipping issue creation and label application.")
|
||||
print(report)
|
||||
else:
|
||||
create_report_issue(report)
|
||||
print("Report issue created.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -1,468 +0,0 @@
|
||||
"""Tests for embedding_utils.py — all embedding model calls are mocked."""
|
||||
from __future__ import annotations
|
||||
|
||||
import sys
|
||||
from unittest.mock import patch, MagicMock
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
# Mock fastembed before importing the module under test (persistent)
|
||||
if "fastembed" not in sys.modules:
|
||||
sys.modules["fastembed"] = MagicMock()
|
||||
|
||||
from embedding_utils import (
|
||||
embed_texts,
|
||||
normalize_rows,
|
||||
reduce_dimensions,
|
||||
detect_outliers,
|
||||
find_duplicate_pairs,
|
||||
suggest_labels,
|
||||
EMBEDDING_DIM,
|
||||
EMBEDDING_MODEL,
|
||||
EMBEDDING_BATCH_SIZE,
|
||||
LABEL_Z_THRESHOLD,
|
||||
LABEL_Z_MARGIN,
|
||||
LABEL_Z_STD_FLOOR,
|
||||
MIN_RAW_SIMILARITY,
|
||||
MAX_LABELS_PER_ITEM,
|
||||
)
|
||||
|
||||
|
||||
class TestEmbedTexts:
|
||||
"""Tests for the embed_texts function."""
|
||||
|
||||
def test_empty_list_returns_empty_array(self):
|
||||
result = embed_texts([])
|
||||
assert result.shape == (0, EMBEDDING_DIM)
|
||||
assert result.dtype == np.float32
|
||||
|
||||
@patch("embedding_utils.TextEmbedding")
|
||||
def test_single_text(self, mock_cls):
|
||||
mock_model = MagicMock()
|
||||
mock_cls.return_value = mock_model
|
||||
vec = np.random.randn(EMBEDDING_DIM).astype(np.float32)
|
||||
mock_model.embed.return_value = iter([vec])
|
||||
|
||||
result = embed_texts(["hello world"])
|
||||
|
||||
mock_cls.assert_called_once_with(model_name=EMBEDDING_MODEL)
|
||||
mock_model.embed.assert_called_once_with(
|
||||
["hello world"], batch_size=EMBEDDING_BATCH_SIZE
|
||||
)
|
||||
assert result.shape == (1, EMBEDDING_DIM)
|
||||
assert result.dtype == np.float32
|
||||
np.testing.assert_array_almost_equal(result[0], vec)
|
||||
|
||||
@patch("embedding_utils.TextEmbedding")
|
||||
def test_multiple_texts(self, mock_cls):
|
||||
mock_model = MagicMock()
|
||||
mock_cls.return_value = mock_model
|
||||
vecs = [
|
||||
np.random.randn(EMBEDDING_DIM).astype(np.float32)
|
||||
for _ in range(5)
|
||||
]
|
||||
mock_model.embed.return_value = iter(vecs)
|
||||
|
||||
result = embed_texts(["a", "b", "c", "d", "e"])
|
||||
assert result.shape == (5, EMBEDDING_DIM)
|
||||
assert result.dtype == np.float32
|
||||
|
||||
|
||||
class TestNormalizeRows:
|
||||
"""Tests for L2 row normalization."""
|
||||
|
||||
def test_empty_matrix(self):
|
||||
m = np.empty((0, 10), dtype=np.float32)
|
||||
result = normalize_rows(m)
|
||||
assert result.shape == (0, 10)
|
||||
|
||||
def test_single_row(self):
|
||||
m = np.array([[3.0, 4.0]], dtype=np.float32)
|
||||
result = normalize_rows(m)
|
||||
# Norm should be ~1.0
|
||||
norm = np.linalg.norm(result[0])
|
||||
assert abs(norm - 1.0) < 1e-5
|
||||
|
||||
def test_multiple_rows(self):
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((10, 50)).astype(np.float32)
|
||||
result = normalize_rows(m)
|
||||
norms = np.linalg.norm(result, axis=1)
|
||||
np.testing.assert_allclose(norms, 1.0, atol=1e-5)
|
||||
|
||||
def test_zero_row_stays_near_zero(self):
|
||||
m = np.array([[0.0, 0.0, 0.0], [1.0, 0.0, 0.0]], dtype=np.float32)
|
||||
result = normalize_rows(m)
|
||||
# Zero row divided by eps -> very small values
|
||||
assert np.linalg.norm(result[0]) < 1e-3
|
||||
# Non-zero row should be unit norm
|
||||
assert abs(np.linalg.norm(result[1]) - 1.0) < 1e-5
|
||||
|
||||
def test_preserves_direction(self):
|
||||
m = np.array([[2.0, 0.0], [0.0, 3.0]], dtype=np.float32)
|
||||
result = normalize_rows(m)
|
||||
np.testing.assert_allclose(result[0], [1.0, 0.0], atol=1e-5)
|
||||
np.testing.assert_allclose(result[1], [0.0, 1.0], atol=1e-5)
|
||||
|
||||
|
||||
class TestReduceDimensions:
|
||||
"""Tests for PCA dimensionality reduction."""
|
||||
|
||||
def test_single_sample_returns_unchanged(self):
|
||||
m = np.random.randn(1, 50).astype(np.float32)
|
||||
result = reduce_dimensions(m, 10)
|
||||
np.testing.assert_array_equal(result, m)
|
||||
|
||||
def test_reduces_dimensions(self):
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((100, 50)).astype(np.float32)
|
||||
result = reduce_dimensions(m, 10)
|
||||
assert result.shape == (100, 10)
|
||||
assert result.dtype == np.float32
|
||||
|
||||
def test_caps_at_n_minus_1(self):
|
||||
rng = np.random.default_rng(42)
|
||||
# 5 samples, 20 features -> max components = 4 (n-1)
|
||||
m = rng.standard_normal((5, 20)).astype(np.float32)
|
||||
result = reduce_dimensions(m, 50)
|
||||
assert result.shape == (5, 4)
|
||||
|
||||
def test_caps_at_d(self):
|
||||
rng = np.random.default_rng(42)
|
||||
# 100 samples, 3 features -> max components = 3
|
||||
m = rng.standard_normal((100, 3)).astype(np.float32)
|
||||
result = reduce_dimensions(m, 50)
|
||||
assert result.shape == (100, 3)
|
||||
|
||||
def test_max_components_respected(self):
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((50, 30)).astype(np.float32)
|
||||
result = reduce_dimensions(m, 5)
|
||||
assert result.shape[1] == 5
|
||||
|
||||
|
||||
class TestDetectOutliers:
|
||||
"""Tests for IQR-based outlier detection."""
|
||||
|
||||
def test_single_sample_returns_empty(self):
|
||||
m = np.random.randn(1, 5).astype(np.float32)
|
||||
result = detect_outliers(m)
|
||||
assert result == []
|
||||
|
||||
def test_empty_returns_empty(self):
|
||||
# n < 2 case
|
||||
m = np.empty((0, 5), dtype=np.float32)
|
||||
result = detect_outliers(m)
|
||||
assert result == []
|
||||
|
||||
def test_finds_outliers_in_synthetic_data(self):
|
||||
rng = np.random.default_rng(42)
|
||||
# Create a tight cluster with one obvious outlier
|
||||
cluster = rng.standard_normal((50, 3)).astype(np.float32) * 0.1
|
||||
outlier = np.array([[100.0, 100.0, 100.0]], dtype=np.float32)
|
||||
m = np.vstack([cluster, outlier])
|
||||
result = detect_outliers(m)
|
||||
# The outlier (index 50) should be detected
|
||||
outlier_indices = [idx for idx, _ in result]
|
||||
assert 50 in outlier_indices
|
||||
|
||||
def test_returns_list_of_index_distance_tuples(self):
|
||||
rng = np.random.default_rng(42)
|
||||
# Tight cluster + outlier to guarantee at least one result
|
||||
cluster = rng.standard_normal((20, 3)).astype(np.float32) * 0.1
|
||||
far_point = np.array([[50.0, 50.0, 50.0]], dtype=np.float32)
|
||||
m = np.vstack([cluster, far_point])
|
||||
result = detect_outliers(m)
|
||||
assert isinstance(result, list)
|
||||
for item in result:
|
||||
assert isinstance(item, tuple)
|
||||
assert len(item) == 2
|
||||
idx, dist = item
|
||||
assert isinstance(idx, int)
|
||||
assert isinstance(dist, float)
|
||||
assert dist > 0
|
||||
|
||||
def test_iqr_cutoff_behavior(self):
|
||||
"""Lower IQR multiplier should flag more items than higher multiplier."""
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((100, 3)).astype(np.float32)
|
||||
low = detect_outliers(m, iqr_multiplier=1.0, max_outlier_pct=0.5)
|
||||
high = detect_outliers(m, iqr_multiplier=5.0, max_outlier_pct=0.5)
|
||||
assert len(low) >= len(high)
|
||||
|
||||
def test_dimension_aware_no_mass_flagging(self):
|
||||
"""High-dimensional clean Gaussian data should not flag everything."""
|
||||
rng = np.random.default_rng(42)
|
||||
# 500 samples, 10 dims — well-conditioned for robust covariance
|
||||
m = rng.standard_normal((500, 10)).astype(np.float32)
|
||||
result = detect_outliers(m)
|
||||
# With IQR-based cutoff on clean Gaussian data,
|
||||
# only a small fraction should be flagged (well under 50%)
|
||||
assert len(result) < 250
|
||||
|
||||
def test_contamination_parameter(self):
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((50, 3)).astype(np.float32)
|
||||
# Should not raise with different contamination values
|
||||
result = detect_outliers(m, contamination=0.05)
|
||||
assert isinstance(result, list)
|
||||
|
||||
def test_max_outlier_pct_hard_cap(self):
|
||||
"""The hard cap should limit outlier count to max_outlier_pct * n."""
|
||||
rng = np.random.default_rng(42)
|
||||
# Create data with many potential outliers (bimodal)
|
||||
cluster = rng.standard_normal((80, 3)).astype(np.float32) * 0.1
|
||||
outliers = rng.standard_normal((20, 3)).astype(np.float32) * 50.0
|
||||
m = np.vstack([cluster, outliers])
|
||||
# Very low IQR multiplier to flag a lot, but cap at 5%
|
||||
result = detect_outliers(m, iqr_multiplier=0.5, max_outlier_pct=0.05)
|
||||
max_allowed = max(1, int(0.05 * 100)) # 5
|
||||
assert len(result) <= max_allowed
|
||||
|
||||
def test_hard_cap_keeps_most_extreme(self):
|
||||
"""When capped, the most extreme items (highest distance) should be kept."""
|
||||
rng = np.random.default_rng(42)
|
||||
cluster = rng.standard_normal((90, 3)).astype(np.float32) * 0.1
|
||||
# Create outliers with increasing extremity
|
||||
outliers = np.array([
|
||||
[10.0, 10.0, 10.0],
|
||||
[20.0, 20.0, 20.0],
|
||||
[50.0, 50.0, 50.0],
|
||||
], dtype=np.float32)
|
||||
m = np.vstack([cluster, outliers])
|
||||
# Cap at ~1 item (0.01 * 93 = 0, but min is 1)
|
||||
result = detect_outliers(m, iqr_multiplier=0.5, max_outlier_pct=0.02)
|
||||
if len(result) > 0:
|
||||
# The most extreme (index 92, distance for [50,50,50]) should be kept
|
||||
indices = [idx for idx, _ in result]
|
||||
assert 92 in indices
|
||||
|
||||
def test_cutoff_attribute(self):
|
||||
"""Returned result should carry a cutoff attribute."""
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((50, 3)).astype(np.float32)
|
||||
result = detect_outliers(m)
|
||||
assert hasattr(result, "cutoff")
|
||||
assert isinstance(result.cutoff, float)
|
||||
assert result.cutoff > 0
|
||||
|
||||
|
||||
class TestFindDuplicatePairs:
|
||||
"""Tests for cosine similarity duplicate detection."""
|
||||
|
||||
def test_single_item_returns_empty(self):
|
||||
m = np.random.randn(1, 10).astype(np.float32)
|
||||
result = find_duplicate_pairs(m, 0.9)
|
||||
assert result == []
|
||||
|
||||
def test_empty_returns_empty(self):
|
||||
m = np.empty((0, 10), dtype=np.float32)
|
||||
result = find_duplicate_pairs(m, 0.9)
|
||||
assert result == []
|
||||
|
||||
def test_identical_vectors_detected(self):
|
||||
vec = np.random.randn(10).astype(np.float32)
|
||||
vec = vec / np.linalg.norm(vec)
|
||||
m = np.vstack([vec, vec, np.random.randn(10).astype(np.float32)])
|
||||
result = find_duplicate_pairs(m, 0.99)
|
||||
# Items 0 and 1 are identical, should be found
|
||||
assert any(i == 0 and j == 1 for i, j, _ in result)
|
||||
|
||||
def test_orthogonal_vectors_not_detected(self):
|
||||
m = np.eye(5, dtype=np.float32)
|
||||
result = find_duplicate_pairs(m, 0.5)
|
||||
assert result == []
|
||||
|
||||
def test_returns_correct_format(self):
|
||||
vec = np.random.randn(10).astype(np.float32)
|
||||
vec = vec / np.linalg.norm(vec)
|
||||
m = np.vstack([vec, vec])
|
||||
result = find_duplicate_pairs(m, 0.5)
|
||||
assert len(result) >= 1
|
||||
for item in result:
|
||||
assert len(item) == 3
|
||||
i, j, sim = item
|
||||
assert isinstance(i, int)
|
||||
assert isinstance(j, int)
|
||||
assert isinstance(sim, float)
|
||||
assert i < j
|
||||
|
||||
def test_i_less_than_j(self):
|
||||
rng = np.random.default_rng(42)
|
||||
# Create some similar vectors
|
||||
base = rng.standard_normal(10).astype(np.float32)
|
||||
m = np.vstack([base + rng.standard_normal(10) * 0.01 for _ in range(5)])
|
||||
result = find_duplicate_pairs(m, 0.5)
|
||||
for i, j, _ in result:
|
||||
assert i < j
|
||||
|
||||
def test_high_threshold_fewer_pairs(self):
|
||||
rng = np.random.default_rng(42)
|
||||
m = rng.standard_normal((10, 20)).astype(np.float32)
|
||||
# Normalize for meaningful cosine similarities
|
||||
norms = np.linalg.norm(m, axis=1, keepdims=True)
|
||||
m = m / norms
|
||||
low = find_duplicate_pairs(m, 0.3)
|
||||
high = find_duplicate_pairs(m, 0.9)
|
||||
assert len(low) >= len(high)
|
||||
|
||||
|
||||
class TestSuggestLabels:
|
||||
"""Tests for z-score normalized label suggestion."""
|
||||
|
||||
def test_empty_items_returns_empty_lists(self):
|
||||
items = np.empty((0, 10), dtype=np.float32)
|
||||
labels = np.random.randn(3, 10).astype(np.float32)
|
||||
result = suggest_labels(items, labels, ["a", "b", "c"])
|
||||
assert result == []
|
||||
|
||||
def test_empty_labels_returns_empty_per_item(self):
|
||||
items = np.random.randn(5, 10).astype(np.float32)
|
||||
labels = np.empty((0, 10), dtype=np.float32)
|
||||
result = suggest_labels(items, labels, [])
|
||||
assert len(result) == 5
|
||||
assert all(s == [] for s in result)
|
||||
|
||||
def test_identical_embedding_gets_that_label(self):
|
||||
"""If an item embedding strongly matches one label, z-score should highlight it."""
|
||||
# Create multiple items so z-score normalization is meaningful
|
||||
rng = np.random.default_rng(42)
|
||||
# 10 random items + 1 item that matches label "bug" exactly
|
||||
random_items = rng.standard_normal((10, 3)).astype(np.float32)
|
||||
bug_vec = np.array([[1.0, 0.0, 0.0]], dtype=np.float32)
|
||||
items = np.vstack([random_items, bug_vec])
|
||||
labels = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32)
|
||||
result = suggest_labels(
|
||||
items, labels, ["bug", "feature", "docs"],
|
||||
z_threshold=0.5, z_margin=0.0, min_raw_sim=0.1,
|
||||
)
|
||||
# The last item (matching bug_vec) should get "bug" as top suggestion
|
||||
last_item_sugs = result[-1]
|
||||
if last_item_sugs:
|
||||
assert last_item_sugs[0][0] == "bug"
|
||||
|
||||
def test_z_score_suppresses_dominant_label(self):
|
||||
"""When all items are similar to one label, z-scores should be low
|
||||
(none stands out) and that label should not be blindly suggested."""
|
||||
# All items identical — z-score for every item on every label is 0
|
||||
items = np.ones((10, 3), dtype=np.float32)
|
||||
labels = np.array([[1.0, 1.0, 1.0], [0.0, 1.0, 0.0]], dtype=np.float32)
|
||||
result = suggest_labels(
|
||||
items, labels, ["catch-all", "specific"],
|
||||
z_threshold=1.5, min_raw_sim=0.3,
|
||||
)
|
||||
# With identical items, std=0 -> z-scores are all 0 -> nothing passes z_threshold
|
||||
for sugs in result:
|
||||
assert sugs == []
|
||||
|
||||
def test_margin_gate_blocks_top1(self):
|
||||
"""Top-1 label must beat #2 by z_margin to be accepted as position 0."""
|
||||
rng = np.random.default_rng(99)
|
||||
# 20 items, each slightly different, 2 labels
|
||||
items = rng.standard_normal((20, 5)).astype(np.float32)
|
||||
# Two labels that are nearly identical -> margin gate should block top-1
|
||||
labels = np.array([[1.0, 0.5, 0.0, 0.0, 0.0],
|
||||
[1.0, 0.5, 0.01, 0.0, 0.0]], dtype=np.float32)
|
||||
result = suggest_labels(
|
||||
items, labels, ["label-a", "label-b"],
|
||||
z_threshold=0.0, z_margin=10.0, min_raw_sim=0.0, max_per_item=1,
|
||||
)
|
||||
# With a huge margin requirement and max_per_item=1, nothing should pass
|
||||
# because the only candidate (top-1) is blocked by margin gate,
|
||||
# and max_per_item=1 prevents falling through to position 2
|
||||
for sugs in result:
|
||||
assert sugs == []
|
||||
|
||||
def test_margin_gate_passes_when_clear_winner(self):
|
||||
"""When top-1 clearly beats #2, it should pass the margin gate."""
|
||||
# Create items where one strongly matches label 0 vs label 1
|
||||
items = np.array([
|
||||
[1.0, 0.0, 0.0, 0.0, 0.0], # strongly matches label-a
|
||||
[0.0, 0.0, 0.0, 0.0, 1.0], # matches neither well
|
||||
] * 5, dtype=np.float32) # 10 items for stable z-scores
|
||||
labels = np.array([
|
||||
[1.0, 0.0, 0.0, 0.0, 0.0], # label-a
|
||||
[0.0, 1.0, 0.0, 0.0, 0.0], # label-b (orthogonal)
|
||||
], dtype=np.float32)
|
||||
result = suggest_labels(
|
||||
items, labels, ["label-a", "label-b"],
|
||||
z_threshold=0.5, z_margin=0.3, min_raw_sim=0.1,
|
||||
)
|
||||
# Items matching label-a should get it suggested (clear z-score advantage)
|
||||
got_label_a = sum(1 for sugs in result if sugs and sugs[0][0] == "label-a")
|
||||
assert got_label_a > 0
|
||||
|
||||
def test_min_raw_similarity_filter(self):
|
||||
"""Even with high z-score, low raw similarity should be filtered out."""
|
||||
# Items are orthogonal to all labels -> raw similarity near 0
|
||||
items = np.array([[1.0, 0.0, 0.0]], dtype=np.float32)
|
||||
labels = np.array([[0.0, 0.0, 1.0]], dtype=np.float32)
|
||||
result = suggest_labels(
|
||||
items, labels, ["irrelevant"],
|
||||
z_threshold=0.0, z_margin=0.0, min_raw_sim=0.9,
|
||||
)
|
||||
# Raw similarity is ~0, which is below min_raw_sim=0.9
|
||||
assert result[0] == []
|
||||
|
||||
def test_max_per_item_respected(self):
|
||||
"""Even if many labels qualify, max_per_item caps the results."""
|
||||
rng = np.random.default_rng(42)
|
||||
# Create items with some variance so z-scores differentiate
|
||||
items = rng.standard_normal((20, 10)).astype(np.float32)
|
||||
base = items[0]
|
||||
# All labels very similar to item 0
|
||||
labels = np.array([base + rng.standard_normal(10) * 0.01 for _ in range(10)])
|
||||
names = [f"label-{i}" for i in range(10)]
|
||||
result = suggest_labels(
|
||||
items, labels, names,
|
||||
z_threshold=0.0, z_margin=0.0, min_raw_sim=0.0, max_per_item=2,
|
||||
)
|
||||
for sugs in result:
|
||||
assert len(sugs) <= 2
|
||||
|
||||
def test_returns_raw_similarity_not_z_score(self):
|
||||
"""Returned scores should be raw cosine similarity, not z-scores."""
|
||||
rng = np.random.default_rng(42)
|
||||
items = rng.standard_normal((15, 5)).astype(np.float32)
|
||||
labels = rng.standard_normal((3, 5)).astype(np.float32)
|
||||
names = ["bug", "feature", "docs"]
|
||||
result = suggest_labels(
|
||||
items, labels, names,
|
||||
z_threshold=0.0, z_margin=0.0, min_raw_sim=-1.0,
|
||||
)
|
||||
# Raw cosine similarity should be in [-1, 1] range
|
||||
for sugs in result:
|
||||
for name, score in sugs:
|
||||
assert -1.0 <= score <= 1.0 + 1e-5
|
||||
assert isinstance(name, str)
|
||||
assert isinstance(score, float)
|
||||
|
||||
def test_returns_correct_format(self):
|
||||
rng = np.random.default_rng(42)
|
||||
items = rng.standard_normal((3, 10)).astype(np.float32)
|
||||
labels = rng.standard_normal((5, 10)).astype(np.float32)
|
||||
names = ["bug", "feature", "docs", "ci", "test"]
|
||||
result = suggest_labels(
|
||||
items, labels, names,
|
||||
z_threshold=0.0, z_margin=0.0, min_raw_sim=-1.0,
|
||||
)
|
||||
assert len(result) == 3
|
||||
for sugs in result:
|
||||
for name, score in sugs:
|
||||
assert isinstance(name, str)
|
||||
assert isinstance(score, float)
|
||||
assert name in names
|
||||
|
||||
def test_text_truncation_in_labels(self):
|
||||
"""Label names should be returned as-is even when very long."""
|
||||
rng = np.random.default_rng(42)
|
||||
items = rng.standard_normal((10, 5)).astype(np.float32)
|
||||
long_name = "a" * 200
|
||||
labels = rng.standard_normal((1, 5)).astype(np.float32)
|
||||
result = suggest_labels(
|
||||
items, labels, [long_name],
|
||||
z_threshold=0.0, z_margin=0.0, min_raw_sim=-1.0,
|
||||
)
|
||||
for sugs in result:
|
||||
if sugs:
|
||||
assert sugs[0][0] == long_name
|
||||
@@ -1,873 +0,0 @@
|
||||
"""Tests for sweep.py — all external calls (API, embedding) are mocked."""
|
||||
from __future__ import annotations
|
||||
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
from io import BytesIO
|
||||
from unittest.mock import patch, MagicMock, mock_open
|
||||
from urllib.error import HTTPError
|
||||
|
||||
import numpy as np
|
||||
import pytest
|
||||
|
||||
# Mock fastembed before importing sweep (which imports embedding_utils)
|
||||
sys.modules["fastembed"] = MagicMock()
|
||||
|
||||
# Set required env vars before importing sweep (module-level constants read env)
|
||||
os.environ.setdefault("GITHUB_TOKEN", "test-token")
|
||||
os.environ.setdefault("GITHUB_REPOSITORY", "owner/repo")
|
||||
|
||||
from sweep import (
|
||||
github_api_get,
|
||||
fetch_all_open_items,
|
||||
fetch_repo_labels,
|
||||
apply_labels_to_item,
|
||||
generate_report,
|
||||
create_report_issue,
|
||||
write_report,
|
||||
main,
|
||||
TriageItem,
|
||||
RepoLabel,
|
||||
REPORT_FILE,
|
||||
REPORT_LABEL,
|
||||
API_PAGE_SIZE,
|
||||
MIN_SAMPLES_FOR_OUTLIER_DETECTION,
|
||||
PCA_MAX_COMPONENTS,
|
||||
MAX_EMBED_CHARS,
|
||||
IQR_MULTIPLIER,
|
||||
MAX_OUTLIER_PCT,
|
||||
_item_age,
|
||||
_suggested_action,
|
||||
)
|
||||
|
||||
|
||||
def _make_api_issue(number: int, title: str = "Test issue", is_pr: bool = False,
|
||||
body: str = "Issue body", labels: list[str] | None = None,
|
||||
created_at: str = "2026-03-21T00:00:00Z") -> dict:
|
||||
"""Helper to build a mock GitHub API issue response object."""
|
||||
result: dict = {
|
||||
"number": number,
|
||||
"title": title,
|
||||
"html_url": f"https://github.com/owner/repo/issues/{number}",
|
||||
"body": body,
|
||||
"created_at": created_at,
|
||||
"labels": [{"name": lbl} for lbl in (labels or [])],
|
||||
}
|
||||
if is_pr:
|
||||
result["pull_request"] = {"url": "..."}
|
||||
return result
|
||||
|
||||
|
||||
class TestGithubApiGet:
|
||||
"""Tests for the github_api_get function."""
|
||||
|
||||
@patch("sweep.urllib.request.urlopen")
|
||||
def test_successful_request(self, mock_urlopen):
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = json.dumps([{"id": 1}]).encode()
|
||||
mock_resp.__enter__ = lambda s: s
|
||||
mock_resp.__exit__ = MagicMock(return_value=False)
|
||||
mock_urlopen.return_value = mock_resp
|
||||
|
||||
result = github_api_get("/issues?state=open")
|
||||
assert result == [{"id": 1}]
|
||||
|
||||
@patch("sweep.urllib.request.urlopen")
|
||||
def test_http_error_exits(self, mock_urlopen):
|
||||
error = HTTPError(
|
||||
url="https://api.github.com/repos/owner/repo/issues",
|
||||
code=403,
|
||||
msg="Forbidden",
|
||||
hdrs=None, # type: ignore[arg-type]
|
||||
fp=BytesIO(b'{"message": "rate limited"}'),
|
||||
)
|
||||
mock_urlopen.side_effect = error
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
github_api_get("/issues")
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
|
||||
class TestConstants:
|
||||
"""Tests for module-level constants."""
|
||||
|
||||
def test_min_samples_is_at_least_3x_pca_max(self):
|
||||
"""MIN_SAMPLES must be >= 3 * PCA_MAX_COMPONENTS for reliable covariance."""
|
||||
assert MIN_SAMPLES_FOR_OUTLIER_DETECTION >= 3 * PCA_MAX_COMPONENTS
|
||||
|
||||
def test_min_samples_is_100(self):
|
||||
assert MIN_SAMPLES_FOR_OUTLIER_DETECTION == 100
|
||||
|
||||
def test_pca_max_components_is_20(self):
|
||||
assert PCA_MAX_COMPONENTS == 20
|
||||
|
||||
def test_iqr_multiplier_default(self):
|
||||
assert IQR_MULTIPLIER == 3.0
|
||||
|
||||
def test_max_outlier_pct_default(self):
|
||||
assert MAX_OUTLIER_PCT == 0.05
|
||||
|
||||
|
||||
class TestFetchAllOpenItems:
|
||||
"""Tests for fetch_all_open_items."""
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_empty_repo(self, mock_get):
|
||||
mock_get.return_value = []
|
||||
items = fetch_all_open_items()
|
||||
assert items == []
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_single_page(self, mock_get):
|
||||
mock_get.return_value = [
|
||||
_make_api_issue(1, "Bug report"),
|
||||
_make_api_issue(2, "Feature request", is_pr=True),
|
||||
]
|
||||
items = fetch_all_open_items()
|
||||
assert len(items) == 2
|
||||
assert items[0]["number"] == 1
|
||||
assert items[0]["is_pr"] is False
|
||||
assert items[1]["is_pr"] is True
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_text_field_constructed(self, mock_get):
|
||||
mock_get.return_value = [
|
||||
_make_api_issue(1, "My Title", body="My Body"),
|
||||
]
|
||||
items = fetch_all_open_items()
|
||||
assert items[0]["text"] == "My Title\n\nMy Body"
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_long_body_truncated(self, mock_get):
|
||||
"""Bodies exceeding MAX_EMBED_CHARS are truncated to fit the token window."""
|
||||
long_body = "x" * (MAX_EMBED_CHARS + 500)
|
||||
mock_get.return_value = [
|
||||
_make_api_issue(1, "Title", body=long_body),
|
||||
]
|
||||
items = fetch_all_open_items()
|
||||
assert len(items[0]["text"]) == MAX_EMBED_CHARS
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_short_body_not_truncated(self, mock_get):
|
||||
"""Bodies under the limit are left intact."""
|
||||
mock_get.return_value = [
|
||||
_make_api_issue(1, "Title", body="Short body"),
|
||||
]
|
||||
items = fetch_all_open_items()
|
||||
assert items[0]["text"] == "Title\n\nShort body"
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_null_body_handled(self, mock_get):
|
||||
issue = _make_api_issue(1, "No body")
|
||||
issue["body"] = None
|
||||
mock_get.return_value = [issue]
|
||||
items = fetch_all_open_items()
|
||||
assert items[0]["text"] == "No body\n\n"
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_labels_extracted(self, mock_get):
|
||||
mock_get.return_value = [
|
||||
_make_api_issue(1, "Labeled", labels=["bug", "high-priority"]),
|
||||
]
|
||||
items = fetch_all_open_items()
|
||||
assert items[0]["labels"] == ["bug", "high-priority"]
|
||||
|
||||
@patch("sweep.MAX_ITEMS", 3)
|
||||
@patch("sweep.github_api_get")
|
||||
def test_max_items_cap(self, mock_get):
|
||||
mock_get.return_value = [_make_api_issue(i) for i in range(100)]
|
||||
items = fetch_all_open_items()
|
||||
assert len(items) == 3
|
||||
|
||||
@patch("sweep.API_PAGE_SIZE", 2)
|
||||
@patch("sweep.github_api_get")
|
||||
def test_pagination(self, mock_get):
|
||||
# First page: 2 items (full page), second page: 1 item (partial -> stop)
|
||||
mock_get.side_effect = [
|
||||
[_make_api_issue(1), _make_api_issue(2)],
|
||||
[_make_api_issue(3)],
|
||||
]
|
||||
items = fetch_all_open_items()
|
||||
assert len(items) == 3
|
||||
assert mock_get.call_count == 2
|
||||
|
||||
|
||||
class TestItemAge:
|
||||
"""Tests for _item_age helper."""
|
||||
|
||||
def test_recent_item(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
recent = (datetime.now(timezone.utc) - timedelta(hours=12)).isoformat()
|
||||
assert _item_age(recent) == "<1d"
|
||||
|
||||
def test_days_old(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
old = (datetime.now(timezone.utc) - timedelta(days=15)).isoformat()
|
||||
assert _item_age(old) == "15d"
|
||||
|
||||
def test_months_old(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
old = (datetime.now(timezone.utc) - timedelta(days=90)).isoformat()
|
||||
assert _item_age(old) == "3mo"
|
||||
|
||||
def test_years_old(self):
|
||||
from datetime import datetime, timezone, timedelta
|
||||
old = (datetime.now(timezone.utc) - timedelta(days=400)).isoformat()
|
||||
assert _item_age(old) == "1y"
|
||||
|
||||
def test_invalid_date(self):
|
||||
assert _item_age("not-a-date") == "?"
|
||||
|
||||
|
||||
class TestSuggestedAction:
|
||||
"""Tests for _suggested_action helper."""
|
||||
|
||||
def test_both_issues_close_newer(self):
|
||||
a = TriageItem(
|
||||
number=1, title="A", html_url="u", is_pr=False, labels=[],
|
||||
created_at="2026-01-01T00:00:00Z", text="t",
|
||||
)
|
||||
b = TriageItem(
|
||||
number=2, title="B", html_url="u", is_pr=False, labels=[],
|
||||
created_at="2026-02-01T00:00:00Z", text="t",
|
||||
)
|
||||
result = _suggested_action(a, b)
|
||||
assert "Close #2 as duplicate" in result
|
||||
|
||||
def test_both_prs_review(self):
|
||||
a = TriageItem(
|
||||
number=1, title="A", html_url="u", is_pr=True, labels=[],
|
||||
created_at="2026-01-01T00:00:00Z", text="t",
|
||||
)
|
||||
b = TriageItem(
|
||||
number=2, title="B", html_url="u", is_pr=True, labels=[],
|
||||
created_at="2026-01-01T00:00:00Z", text="t",
|
||||
)
|
||||
assert _suggested_action(a, b) == "Review for overlap"
|
||||
|
||||
def test_issue_pr_link(self):
|
||||
a = TriageItem(
|
||||
number=1, title="A", html_url="u", is_pr=False, labels=[],
|
||||
created_at="2026-01-01T00:00:00Z", text="t",
|
||||
)
|
||||
b = TriageItem(
|
||||
number=2, title="B", html_url="u", is_pr=True, labels=[],
|
||||
created_at="2026-01-01T00:00:00Z", text="t",
|
||||
)
|
||||
assert _suggested_action(a, b) == "Link PR to issue"
|
||||
|
||||
|
||||
class TestGenerateReport:
|
||||
"""Tests for the markdown report generator."""
|
||||
|
||||
def test_no_findings(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Test", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="Test",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [])
|
||||
assert "## Triage Sweep Report" in report
|
||||
assert "Items analyzed:** 1" in report
|
||||
assert "None found." in report
|
||||
assert "0 outliers flagged" in report
|
||||
assert "0 duplicate pairs found" in report
|
||||
|
||||
def test_health_summary_table(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Test", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="Test",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [])
|
||||
assert "### Health Summary" in report
|
||||
assert "| Metric | Value |" in report
|
||||
assert "| Items analyzed | 1 |" in report
|
||||
|
||||
def test_iqr_multiplier_in_thresholds(self):
|
||||
"""Report should show IQR multiplier, not percentile."""
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Test", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="Test",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [])
|
||||
assert "IQR multiplier" in report
|
||||
assert "percentile" not in report.lower().split("thresholds")[0] # not in thresholds line
|
||||
|
||||
def test_with_outliers_shows_distance_and_age(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=10, title="Spam Issue", html_url="https://example.com/10",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
|
||||
),
|
||||
TriageItem(
|
||||
number=20, title="Good Issue", html_url="https://example.com/20",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="good",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [(0, 12.34)], [])
|
||||
assert "#10" in report
|
||||
assert "Spam Issue" in report
|
||||
assert "12.34" in report
|
||||
assert "1 outliers flagged" in report
|
||||
# Age column should be present
|
||||
assert "| Age |" in report
|
||||
|
||||
def test_outlier_borderline_in_details(self):
|
||||
"""Borderline outliers should be in a <details> section."""
|
||||
from embedding_utils import _OutlierResult
|
||||
items = [
|
||||
TriageItem(
|
||||
number=10, title="Borderline", html_url="https://example.com/10",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
|
||||
),
|
||||
]
|
||||
# Create outlier results with cutoff=10.0, distance=12.0 (< 2*cutoff=20)
|
||||
outlier_results = _OutlierResult([(0, 12.0)])
|
||||
outlier_results.cutoff = 10.0
|
||||
report = generate_report(items, outlier_results, [])
|
||||
assert "<details>" in report
|
||||
assert "Borderline" in report
|
||||
|
||||
def test_outlier_high_confidence(self):
|
||||
"""Items with distance > 2x cutoff should be in high confidence section."""
|
||||
from embedding_utils import _OutlierResult
|
||||
items = [
|
||||
TriageItem(
|
||||
number=10, title="Definite Spam", html_url="https://example.com/10",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
|
||||
),
|
||||
]
|
||||
outlier_results = _OutlierResult([(0, 25.0)])
|
||||
outlier_results.cutoff = 10.0
|
||||
report = generate_report(items, outlier_results, [])
|
||||
assert "High Confidence" in report
|
||||
|
||||
def test_with_duplicates_suggested_action(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="First", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="a",
|
||||
),
|
||||
TriageItem(
|
||||
number=2, title="Second", html_url="https://example.com/2",
|
||||
is_pr=True, labels=[], created_at="2026-02-01T00:00:00Z", text="b",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [(0, 1, 0.954)])
|
||||
assert "#1" in report
|
||||
assert "#2" in report
|
||||
assert "0.954" in report
|
||||
assert "1 duplicate pairs found" in report
|
||||
assert "Suggested Action" in report
|
||||
assert "Link PR to issue" in report
|
||||
|
||||
def test_duplicate_both_issues_close_newer(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="First", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="a",
|
||||
),
|
||||
TriageItem(
|
||||
number=2, title="Second", html_url="https://example.com/2",
|
||||
is_pr=False, labels=[], created_at="2026-02-01T00:00:00Z", text="b",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [(0, 1, 0.95)])
|
||||
assert "Close #2 as duplicate" in report
|
||||
|
||||
def test_duplicate_both_prs_review(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="PR A", html_url="https://example.com/1",
|
||||
is_pr=True, labels=[], created_at="2026-01-01T00:00:00Z", text="a",
|
||||
),
|
||||
TriageItem(
|
||||
number=2, title="PR B", html_url="https://example.com/2",
|
||||
is_pr=True, labels=[], created_at="2026-01-01T00:00:00Z", text="b",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [(0, 1, 0.95)])
|
||||
assert "Review for overlap" in report
|
||||
|
||||
def test_pr_type_label(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=5, title="PR Title", html_url="https://example.com/5",
|
||||
is_pr=True, labels=[], created_at="2026-01-01T00:00:00Z", text="pr",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [(0, 8.5)], [])
|
||||
assert "| PR |" in report
|
||||
|
||||
def test_footer_present(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="T", html_url="u",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="t",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [])
|
||||
assert "no LLM was used" in report
|
||||
|
||||
|
||||
class TestCreateReportIssue:
|
||||
"""Tests for creating the report GitHub issue."""
|
||||
|
||||
@patch("sweep.urllib.request.urlopen")
|
||||
def test_successful_creation(self, mock_urlopen):
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.status = 201
|
||||
mock_resp.read.return_value = json.dumps({
|
||||
"html_url": "https://github.com/owner/repo/issues/99",
|
||||
}).encode()
|
||||
mock_resp.__enter__ = lambda s: s
|
||||
mock_resp.__exit__ = MagicMock(return_value=False)
|
||||
mock_urlopen.return_value = mock_resp
|
||||
|
||||
# Should not raise
|
||||
create_report_issue("# Test Report")
|
||||
|
||||
@patch("sweep.urllib.request.urlopen")
|
||||
def test_http_error_exits(self, mock_urlopen):
|
||||
error = HTTPError(
|
||||
url="https://api.github.com/repos/owner/repo/issues",
|
||||
code=422,
|
||||
msg="Unprocessable",
|
||||
hdrs=None, # type: ignore[arg-type]
|
||||
fp=BytesIO(b'{"message": "validation failed"}'),
|
||||
)
|
||||
mock_urlopen.side_effect = error
|
||||
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
create_report_issue("# Test Report")
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
|
||||
class TestWriteReport:
|
||||
"""Tests for the write_report helper."""
|
||||
|
||||
@patch("builtins.open", mock_open())
|
||||
def test_writes_to_file(self):
|
||||
write_report("# Report Content")
|
||||
from builtins import open as builtin_open # noqa
|
||||
# Verify open was called with the right path
|
||||
from unittest.mock import call
|
||||
open_mock = open # The patched version
|
||||
open_mock.assert_called_once_with(REPORT_FILE, "w", encoding="utf-8") # type: ignore[attr-defined]
|
||||
open_mock().write.assert_called_once_with("# Report Content") # type: ignore[attr-defined]
|
||||
|
||||
|
||||
class TestFetchRepoLabels:
|
||||
"""Tests for fetch_repo_labels."""
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_fetches_and_constructs_labels(self, mock_get):
|
||||
mock_get.return_value = [
|
||||
{"name": "bug", "description": "Something isn't working"},
|
||||
{"name": "enhancement", "description": "New feature or request"},
|
||||
{"name": "docs", "description": ""},
|
||||
]
|
||||
labels = fetch_repo_labels()
|
||||
assert len(labels) == 3
|
||||
assert labels[0]["name"] == "bug"
|
||||
assert labels[0]["text"] == "bug: Something isn't working"
|
||||
assert labels[2]["text"] == "docs" # no description, just name
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_empty_repo_labels(self, mock_get):
|
||||
mock_get.return_value = []
|
||||
labels = fetch_repo_labels()
|
||||
assert labels == []
|
||||
|
||||
@patch("sweep.github_api_get")
|
||||
def test_null_description_handled(self, mock_get):
|
||||
mock_get.return_value = [
|
||||
{"name": "wontfix", "description": None},
|
||||
]
|
||||
labels = fetch_repo_labels()
|
||||
assert labels[0]["text"] == "wontfix"
|
||||
|
||||
@patch("sweep.API_PAGE_SIZE", 2)
|
||||
@patch("sweep.github_api_get")
|
||||
def test_label_pagination(self, mock_get):
|
||||
"""Repos with more labels than one page should fetch all pages."""
|
||||
mock_get.side_effect = [
|
||||
# First page: full (2 items = API_PAGE_SIZE)
|
||||
[
|
||||
{"name": "bug", "description": "Broken"},
|
||||
{"name": "feature", "description": "New"},
|
||||
],
|
||||
# Second page: partial (1 item < API_PAGE_SIZE) -> stop
|
||||
[
|
||||
{"name": "docs", "description": "Documentation"},
|
||||
],
|
||||
]
|
||||
labels = fetch_repo_labels()
|
||||
assert len(labels) == 3
|
||||
assert mock_get.call_count == 2
|
||||
assert labels[0]["name"] == "bug"
|
||||
assert labels[2]["name"] == "docs"
|
||||
|
||||
|
||||
class TestApplyLabelsToItem:
|
||||
"""Tests for apply_labels_to_item."""
|
||||
|
||||
def test_empty_labels_skips(self):
|
||||
# Should not make any API call
|
||||
apply_labels_to_item(1, [])
|
||||
|
||||
@patch("sweep.urllib.request.urlopen")
|
||||
def test_successful_label_application(self, mock_urlopen):
|
||||
mock_resp = MagicMock()
|
||||
mock_resp.read.return_value = b'[{"name": "bug"}]'
|
||||
mock_resp.__enter__ = lambda s: s
|
||||
mock_resp.__exit__ = MagicMock(return_value=False)
|
||||
mock_urlopen.return_value = mock_resp
|
||||
|
||||
# Should not raise
|
||||
apply_labels_to_item(42, ["bug", "enhancement"])
|
||||
|
||||
@patch("sweep.urllib.request.urlopen")
|
||||
def test_http_error_is_non_fatal(self, mock_urlopen):
|
||||
error = HTTPError(
|
||||
url="https://api.github.com/repos/owner/repo/issues/1/labels",
|
||||
code=404,
|
||||
msg="Not Found",
|
||||
hdrs=None, # type: ignore[arg-type]
|
||||
fp=BytesIO(b'{"message": "not found"}'),
|
||||
)
|
||||
mock_urlopen.side_effect = error
|
||||
|
||||
# Should NOT raise — labeling failures are warnings, not fatal
|
||||
apply_labels_to_item(1, ["bug"])
|
||||
|
||||
|
||||
class TestGenerateReportWithLabels:
|
||||
"""Tests for label suggestions in the report."""
|
||||
|
||||
def test_report_includes_label_section_high_confidence(self):
|
||||
"""High-confidence label (raw_sim >= 0.5) should appear in main table."""
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Fix crash", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="crash",
|
||||
),
|
||||
]
|
||||
suggestions = [[("bug", 0.85)]]
|
||||
report = generate_report(items, [], [], label_suggestions=suggestions)
|
||||
assert "Suggested Labels" in report
|
||||
assert "`bug` (0.85)" in report
|
||||
assert "1 items suggested for labeling" in report
|
||||
|
||||
def test_report_low_confidence_in_details(self):
|
||||
"""Low-confidence label (raw_sim < 0.5) should be in <details> section."""
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Something", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="something",
|
||||
),
|
||||
]
|
||||
suggestions = [[("maybe-bug", 0.35)]]
|
||||
report = generate_report(items, [], [], label_suggestions=suggestions)
|
||||
assert "Low-confidence suggestions" in report
|
||||
assert "<details>" in report
|
||||
assert "`maybe-bug` (0.35)" in report
|
||||
|
||||
def test_report_skips_already_labeled_items(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Already labeled", html_url="https://example.com/1",
|
||||
is_pr=False, labels=["bug"], created_at="2026-01-01T00:00:00Z", text="bug",
|
||||
),
|
||||
]
|
||||
suggestions = [[("bug", 0.95)]]
|
||||
report = generate_report(items, [], [], label_suggestions=suggestions)
|
||||
assert "0 items suggested for labeling" in report
|
||||
assert "No unlabeled items" in report
|
||||
|
||||
def test_report_excludes_outliers_from_suggestions(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Spam garbage", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
|
||||
),
|
||||
TriageItem(
|
||||
number=2, title="Real bug", html_url="https://example.com/2",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="bug",
|
||||
),
|
||||
]
|
||||
suggestions = [[("bug", 0.85)], [("bug", 0.90)]]
|
||||
# Item 0 is an outlier (with distance) — should be excluded from label suggestions
|
||||
report = generate_report(items, [(0, 15.2)], [], label_suggestions=suggestions)
|
||||
assert "1 unlabeled items" in report # only item 2
|
||||
assert "#2" in report
|
||||
# Item 0 (outlier) should NOT be in the suggestions table
|
||||
assert "Spam garbage" not in report.split("Suggested Labels")[1]
|
||||
|
||||
def test_report_without_label_suggestions(self):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="T", html_url="u",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="t",
|
||||
),
|
||||
]
|
||||
report = generate_report(items, [], [], label_suggestions=None)
|
||||
assert "Suggested Labels" not in report
|
||||
|
||||
def test_label_concentration_warning(self):
|
||||
"""When >50% of suggestions point to the same label, a warning should appear."""
|
||||
items = [
|
||||
TriageItem(
|
||||
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
|
||||
)
|
||||
for i in range(4)
|
||||
]
|
||||
# 3 out of 4 items get "bug" label -> 75% concentration
|
||||
suggestions = [
|
||||
[("bug", 0.85)],
|
||||
[("bug", 0.80)],
|
||||
[("bug", 0.75)],
|
||||
[("enhancement", 0.90)],
|
||||
]
|
||||
report = generate_report(items, [], [], label_suggestions=suggestions)
|
||||
assert "Warning" in report
|
||||
assert "`bug`" in report
|
||||
assert "3/4" in report
|
||||
|
||||
|
||||
class TestMain:
|
||||
"""Tests for the main orchestration function."""
|
||||
|
||||
@patch.dict(os.environ, {"GITHUB_TOKEN": "", "GITHUB_REPOSITORY": "owner/repo"})
|
||||
def test_missing_token_exits(self):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main()
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
@patch.dict(os.environ, {"GITHUB_TOKEN": "tok", "GITHUB_REPOSITORY": ""})
|
||||
def test_missing_repo_exits(self):
|
||||
with pytest.raises(SystemExit) as exc_info:
|
||||
main()
|
||||
assert exc_info.value.code == 1
|
||||
|
||||
@patch("sweep.write_report")
|
||||
@patch("sweep.fetch_all_open_items", return_value=[])
|
||||
def test_no_items(self, mock_fetch, mock_write):
|
||||
main()
|
||||
mock_write.assert_called_once()
|
||||
report = mock_write.call_args[0][0]
|
||||
assert "No open issues or PRs found" in report
|
||||
|
||||
@patch("sweep.create_report_issue")
|
||||
@patch("sweep.write_report")
|
||||
@patch("sweep.suggest_labels", return_value=[])
|
||||
@patch("sweep.find_duplicate_pairs", return_value=[])
|
||||
@patch("sweep.detect_outliers", return_value=[])
|
||||
@patch("sweep.reduce_dimensions")
|
||||
@patch("sweep.normalize_rows")
|
||||
@patch("sweep.embed_texts")
|
||||
@patch("sweep.fetch_repo_labels")
|
||||
@patch("sweep.fetch_all_open_items")
|
||||
def test_full_flow_with_enough_items(
|
||||
self, mock_fetch, mock_labels, mock_embed, mock_norm, mock_reduce,
|
||||
mock_outliers, mock_dupes, mock_suggest, mock_write, mock_create,
|
||||
):
|
||||
"""Test the full flow with >= MIN_SAMPLES items (outlier detection runs)."""
|
||||
n = MIN_SAMPLES_FOR_OUTLIER_DETECTION
|
||||
items = [
|
||||
TriageItem(
|
||||
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
|
||||
)
|
||||
for i in range(n)
|
||||
]
|
||||
mock_fetch.return_value = items
|
||||
mock_labels.return_value = [
|
||||
RepoLabel(name="bug", description="Something broken", text="bug: Something broken"),
|
||||
]
|
||||
|
||||
embeddings = np.random.randn(n, 384).astype(np.float32)
|
||||
mock_embed.return_value = embeddings
|
||||
mock_norm.return_value = embeddings
|
||||
mock_reduce.return_value = np.random.randn(n, 10).astype(np.float32)
|
||||
|
||||
main()
|
||||
|
||||
mock_fetch.assert_called_once()
|
||||
mock_labels.assert_called_once()
|
||||
# embed_texts called twice: once for items, once for labels
|
||||
assert mock_embed.call_count == 2
|
||||
mock_norm.assert_called()
|
||||
mock_reduce.assert_called_once()
|
||||
mock_outliers.assert_called_once()
|
||||
mock_dupes.assert_called_once()
|
||||
mock_suggest.assert_called_once()
|
||||
mock_write.assert_called_once()
|
||||
mock_create.assert_called_once()
|
||||
|
||||
@patch("sweep.create_report_issue")
|
||||
@patch("sweep.write_report")
|
||||
@patch("sweep.suggest_labels", return_value=[])
|
||||
@patch("sweep.find_duplicate_pairs", return_value=[])
|
||||
@patch("sweep.detect_outliers")
|
||||
@patch("sweep.reduce_dimensions")
|
||||
@patch("sweep.normalize_rows")
|
||||
@patch("sweep.embed_texts")
|
||||
@patch("sweep.fetch_repo_labels", return_value=[])
|
||||
@patch("sweep.fetch_all_open_items")
|
||||
def test_skips_outlier_detection_for_few_items(
|
||||
self, mock_fetch, mock_labels, mock_embed, mock_norm, mock_reduce,
|
||||
mock_outliers, mock_dupes, mock_suggest, mock_write, mock_create,
|
||||
):
|
||||
"""With < MIN_SAMPLES items, outlier detection should be skipped."""
|
||||
n = MIN_SAMPLES_FOR_OUTLIER_DETECTION - 1
|
||||
items = [
|
||||
TriageItem(
|
||||
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
|
||||
)
|
||||
for i in range(n)
|
||||
]
|
||||
mock_fetch.return_value = items
|
||||
|
||||
embeddings = np.random.randn(n, 384).astype(np.float32)
|
||||
mock_embed.return_value = embeddings
|
||||
mock_norm.return_value = embeddings
|
||||
|
||||
main()
|
||||
|
||||
# Outlier detection should not have been called
|
||||
mock_reduce.assert_not_called()
|
||||
mock_outliers.assert_not_called()
|
||||
# But duplicates should still be checked
|
||||
mock_dupes.assert_called_once()
|
||||
|
||||
@patch.dict(os.environ, {"INPUT_DRY_RUN": "true"})
|
||||
@patch("sweep.DRY_RUN", True)
|
||||
@patch("sweep.write_report")
|
||||
@patch("sweep.create_report_issue")
|
||||
@patch("sweep.apply_labels_to_item")
|
||||
@patch("sweep.suggest_labels", return_value=[[("bug", 0.85)]])
|
||||
@patch("sweep.find_duplicate_pairs", return_value=[])
|
||||
@patch("sweep.normalize_rows")
|
||||
@patch("sweep.embed_texts")
|
||||
@patch("sweep.fetch_repo_labels")
|
||||
@patch("sweep.fetch_all_open_items")
|
||||
def test_dry_run_skips_issue_creation_and_labeling(
|
||||
self, mock_fetch, mock_labels, mock_embed, mock_norm,
|
||||
mock_dupes, mock_suggest, mock_apply, mock_create, mock_write,
|
||||
):
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Item", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="text",
|
||||
)
|
||||
]
|
||||
mock_fetch.return_value = items
|
||||
mock_labels.return_value = [
|
||||
RepoLabel(name="bug", description="Broken", text="bug: Broken"),
|
||||
]
|
||||
embeddings = np.random.randn(1, 384).astype(np.float32)
|
||||
mock_embed.return_value = embeddings
|
||||
mock_norm.return_value = embeddings
|
||||
|
||||
main()
|
||||
|
||||
mock_create.assert_not_called()
|
||||
mock_apply.assert_not_called()
|
||||
mock_write.assert_called_once()
|
||||
|
||||
@patch("sweep.create_report_issue")
|
||||
@patch("sweep.write_report")
|
||||
@patch("sweep.apply_labels_to_item")
|
||||
@patch("sweep.suggest_labels")
|
||||
@patch("sweep.find_duplicate_pairs", return_value=[])
|
||||
@patch("sweep.normalize_rows")
|
||||
@patch("sweep.embed_texts")
|
||||
@patch("sweep.fetch_repo_labels")
|
||||
@patch("sweep.fetch_all_open_items")
|
||||
def test_labels_not_auto_applied(
|
||||
self, mock_fetch, mock_labels, mock_embed, mock_norm,
|
||||
mock_dupes, mock_suggest, mock_apply, mock_write, mock_create,
|
||||
):
|
||||
"""Auto-labeling is disabled; labels should appear in report only."""
|
||||
items = [
|
||||
TriageItem(
|
||||
number=1, title="Crash bug", html_url="https://example.com/1",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="crash",
|
||||
),
|
||||
TriageItem(
|
||||
number=2, title="Already labeled", html_url="https://example.com/2",
|
||||
is_pr=False, labels=["enhancement"], created_at="2026-01-01T00:00:00Z", text="feat",
|
||||
),
|
||||
]
|
||||
mock_fetch.return_value = items
|
||||
mock_labels.return_value = [
|
||||
RepoLabel(name="bug", description="Broken", text="bug: Broken"),
|
||||
]
|
||||
mock_suggest.return_value = [
|
||||
[("bug", 0.90)],
|
||||
[("bug", 0.45)],
|
||||
]
|
||||
|
||||
embeddings = np.random.randn(2, 384).astype(np.float32)
|
||||
mock_embed.return_value = embeddings
|
||||
mock_norm.return_value = embeddings
|
||||
|
||||
main()
|
||||
|
||||
# Auto-labeling is disabled — apply_labels_to_item should never be called
|
||||
mock_apply.assert_not_called()
|
||||
|
||||
@patch("sweep.create_report_issue")
|
||||
@patch("sweep.write_report")
|
||||
@patch("sweep.apply_labels_to_item")
|
||||
@patch("sweep.suggest_labels")
|
||||
@patch("sweep.find_duplicate_pairs", return_value=[])
|
||||
@patch("sweep.detect_outliers")
|
||||
@patch("sweep.reduce_dimensions")
|
||||
@patch("sweep.normalize_rows")
|
||||
@patch("sweep.embed_texts")
|
||||
@patch("sweep.fetch_repo_labels")
|
||||
@patch("sweep.fetch_all_open_items")
|
||||
def test_outliers_excluded_from_report_suggestions(
|
||||
self, mock_fetch, mock_labels, mock_embed, mock_norm, mock_reduce,
|
||||
mock_outliers, mock_dupes, mock_suggest, mock_apply, mock_write, mock_create,
|
||||
):
|
||||
"""Items flagged as outliers should not appear in report label suggestions."""
|
||||
n = MIN_SAMPLES_FOR_OUTLIER_DETECTION
|
||||
items = [
|
||||
TriageItem(
|
||||
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
|
||||
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
|
||||
)
|
||||
for i in range(n)
|
||||
]
|
||||
mock_fetch.return_value = items
|
||||
mock_labels.return_value = [
|
||||
RepoLabel(name="bug", description="Broken", text="bug: Broken"),
|
||||
]
|
||||
mock_outliers.return_value = [(0, 12.5), (5, 15.3)]
|
||||
mock_suggest.return_value = [[("bug", 0.85)] for _ in range(n)]
|
||||
|
||||
embeddings = np.random.randn(n, 384).astype(np.float32)
|
||||
mock_embed.return_value = embeddings
|
||||
mock_norm.return_value = embeddings
|
||||
mock_reduce.return_value = np.random.randn(n, 10).astype(np.float32)
|
||||
|
||||
main()
|
||||
|
||||
# Auto-labeling is disabled
|
||||
mock_apply.assert_not_called()
|
||||
# Report should still be generated (outliers excluded from suggestions in report)
|
||||
mock_write.assert_called_once()
|
||||
report = mock_write.call_args[0][0]
|
||||
# Outlier items 0 and 5 should not appear in the label suggestions section
|
||||
assert "Item 0" not in report.split("Suggested Labels")[1] if "Suggested Labels" in report else True
|
||||
@@ -1,83 +0,0 @@
|
||||
name: E2E Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
check-changes:
|
||||
name: Check web module changes
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
web_changed: ${{ steps.filter.outputs.web }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: dorny/paths-filter@fbd0ab8f3e69293af611ebaee6363fc25e6d187d # v3
|
||||
id: filter
|
||||
with:
|
||||
filters: |
|
||||
web:
|
||||
- 'gitnexus-web/**'
|
||||
|
||||
e2e:
|
||||
name: e2e (chromium)
|
||||
needs: check-changes
|
||||
if: needs.check-changes.result == 'success' && needs.check-changes.outputs.web_changed == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
|
||||
- uses: ./.github/actions/setup-gitnexus-web
|
||||
|
||||
- name: Install Playwright browsers
|
||||
run: npx playwright install --with-deps chromium
|
||||
working-directory: gitnexus-web
|
||||
|
||||
- name: Install backend dependencies
|
||||
run: npm ci
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Build backend
|
||||
run: npm run build
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Analyze repository (index for backend)
|
||||
run: |
|
||||
node gitnexus/dist/cli/index.js analyze || true
|
||||
if [ ! -d ".gitnexus" ]; then
|
||||
echo "::error::No .gitnexus index created"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Start backend server
|
||||
run: node dist/cli/index.js serve &
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Wait for backend readiness
|
||||
run: npx wait-on http://localhost:4747/api/repos --timeout 30000
|
||||
working-directory: gitnexus-web
|
||||
|
||||
- name: Start Vite dev server
|
||||
run: npm run dev &
|
||||
working-directory: gitnexus-web
|
||||
|
||||
- name: Wait for Vite dev server
|
||||
run: npx wait-on http://localhost:5173 --timeout 30000
|
||||
working-directory: gitnexus-web
|
||||
|
||||
- name: Run E2E tests
|
||||
run: npx playwright test
|
||||
working-directory: gitnexus-web
|
||||
env:
|
||||
E2E: '1'
|
||||
|
||||
- name: Upload test results
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: e2e-results
|
||||
path: |
|
||||
gitnexus-web/test-results/
|
||||
gitnexus-web/playwright-report/
|
||||
retention-days: 5
|
||||
@@ -0,0 +1,101 @@
|
||||
name: Integration Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
# ── Integration test matrix ─────────────────────────────────────────
|
||||
# Each test-group runs on a SEPARATE runner per OS, giving full process
|
||||
# isolation for the KuzuDB native C++ addon.
|
||||
# 3 OS x 4 groups = 12 parallel jobs.
|
||||
#
|
||||
# Groups:
|
||||
# kuzu-db — 7 files using withTestKuzuDB / kuzu-adapter (native addon)
|
||||
# Each file runs as its own `vitest run` invocation for full
|
||||
# process isolation. KuzuDB's native N-API addon registers
|
||||
# persistent handles that prevent fork workers from exiting
|
||||
# on Linux, and its C++ destructors segfault during
|
||||
# process.exit(). Running each file in its own process lets
|
||||
# the OS reclaim all resources cleanly.
|
||||
# pipeline — 3 files: ingestion pipeline + csv, each creates own temp DB
|
||||
# e2e — 2 files: child-process only (spawnSync), no in-process kuzu
|
||||
# standalone — 4 files: pure logic, no kuzu, no child processes
|
||||
test-matrix:
|
||||
name: integration (${{ matrix.os }} / ${{ matrix.test-group }})
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest, macos-latest]
|
||||
test-group: [kuzu-db, pipeline, e2e, standalone]
|
||||
include:
|
||||
- test-group: kuzu-db
|
||||
# Marker — actual files are listed in the run step below
|
||||
test-glob: ''
|
||||
- test-group: pipeline
|
||||
test-glob: >-
|
||||
test/integration/pipeline.test.ts
|
||||
test/integration/csv-pipeline.test.ts
|
||||
test/integration/parsing.test.ts
|
||||
- test-group: e2e
|
||||
test-glob: >-
|
||||
test/integration/cli-e2e.test.ts
|
||||
test/integration/hooks-e2e.test.ts
|
||||
- test-group: standalone
|
||||
test-glob: >-
|
||||
test/integration/filesystem-walker.test.ts
|
||||
test/integration/enrichment.test.ts
|
||||
test/integration/tree-sitter-languages.test.ts
|
||||
test/integration/worker-pool.test.ts
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 15
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
with:
|
||||
build: 'true'
|
||||
|
||||
# kuzu-db: run each file in its own vitest process for full isolation.
|
||||
# KuzuDB's native addon hangs fork workers on Linux — process isolation
|
||||
# is the only reliable fix boundary.
|
||||
- name: Run integration tests — kuzu-db (process-isolated)
|
||||
if: matrix.test-group == 'kuzu-db'
|
||||
working-directory: gitnexus
|
||||
shell: bash
|
||||
run: |
|
||||
set -e
|
||||
files=(
|
||||
test/integration/kuzu-core-adapter.test.ts
|
||||
test/integration/kuzu-pool.test.ts
|
||||
test/integration/local-backend.test.ts
|
||||
test/integration/local-backend-calltool.test.ts
|
||||
test/integration/search-core.test.ts
|
||||
test/integration/search-pool.test.ts
|
||||
test/integration/augmentation.test.ts
|
||||
)
|
||||
for f in "${files[@]}"; do
|
||||
echo "::group::$f"
|
||||
npx vitest run --reporter=verbose --pool=forks "$f"
|
||||
echo "::endgroup::"
|
||||
done
|
||||
|
||||
# Non-kuzu groups: run all files in a single vitest invocation
|
||||
- name: Run integration tests — ${{ matrix.test-group }}
|
||||
if: matrix.test-group != 'kuzu-db'
|
||||
run: npx vitest run --reporter=verbose ${{ matrix.test-glob }}
|
||||
working-directory: gitnexus
|
||||
|
||||
# ── Unified status gate ──────────────────────────────────────────────
|
||||
# Branch protection should require THIS job, not the matrix jobs directly.
|
||||
status:
|
||||
name: integration (all groups)
|
||||
needs: test-matrix
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- name: Check all matrix jobs passed
|
||||
run: |
|
||||
result="${{ needs.test-matrix.result }}"
|
||||
if [[ "$result" != "success" ]]; then
|
||||
echo "::error::Integration matrix failed or cancelled: $result"
|
||||
exit 1
|
||||
fi
|
||||
@@ -4,69 +4,10 @@ on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
format:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 20
|
||||
cache: npm
|
||||
cache-dependency-path: package-lock.json
|
||||
- run: npm ci
|
||||
- run: npx prettier --check .
|
||||
|
||||
lint:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 20
|
||||
cache: npm
|
||||
cache-dependency-path: package-lock.json
|
||||
- run: npm ci
|
||||
- run: npx eslint .
|
||||
|
||||
typecheck:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: actions/checkout@v4
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
- run: npx tsc --noEmit
|
||||
working-directory: gitnexus
|
||||
|
||||
typecheck-web:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: ./.github/actions/setup-gitnexus-web
|
||||
- run: npx tsc -b --noEmit
|
||||
working-directory: gitnexus-web
|
||||
|
||||
# Enforces the convention documented in CONTRIBUTING.md → "GitHub Actions —
|
||||
# Concurrency Convention":
|
||||
# 1. Every entry-point (non-reusable) workflow declares a top-level
|
||||
# `concurrency:` block.
|
||||
# 2. Reusable workflows (`on: workflow_call` only) do NOT declare one —
|
||||
# they inherit concurrency from the caller.
|
||||
# 3. The concurrency group key starts with `${{ github.workflow }}` or
|
||||
# the literal `CI-` prefix (the documented ci.yml exception for
|
||||
# reusable-workflow-safe grouping).
|
||||
# Reusability is detected by parsing each workflow's `on:` block, not an
|
||||
# allowlist, so new reusable workflows never produce false positives.
|
||||
workflow-convention:
|
||||
name: Workflow concurrency convention
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- name: Validate workflow concurrency convention
|
||||
shell: bash
|
||||
run: |
|
||||
set -euo pipefail
|
||||
python3 .github/scripts/check-workflow-concurrency.py .github/workflows
|
||||
|
||||
@@ -1,423 +0,0 @@
|
||||
name: CI Report
|
||||
|
||||
# Triggered after the CI workflow completes. Because workflow_run
|
||||
# always runs code from the *default branch*, it receives a read/write
|
||||
# GITHUB_TOKEN — even when the triggering PR comes from a fork.
|
||||
|
||||
on:
|
||||
workflow_run:
|
||||
workflows: ['CI']
|
||||
types: [completed]
|
||||
|
||||
permissions:
|
||||
actions: read # needed to list/download workflow run artifacts
|
||||
contents: read # needed for sparse checkout of vitest.config.ts
|
||||
pull-requests: write # needed to post sticky PR comment
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Serialize sticky-comment writes per PR so two rapid CI completions don't race.
|
||||
# Internal PRs surface in `pull_requests[0].number`. Fork PRs leave that array empty,
|
||||
# so we fall back to `<head-repo-full-name>/<head-branch>`, which is stable across
|
||||
# reruns and subsequent pushes for the same fork PR (unlike `workflow_run.id` which
|
||||
# is unique per run and therefore does not serialize anything).
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.workflow_run.pull_requests[0].number || format('{0}/{1}', github.event.workflow_run.head_repository.full_name, github.event.workflow_run.head_branch) }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
pr-report:
|
||||
name: PR Report
|
||||
# Only run for pull-request CI runs
|
||||
if: >-
|
||||
github.event.workflow_run.event == 'pull_request' &&
|
||||
github.event.workflow_run.conclusion != 'cancelled'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
# ── Download artifacts from the CI run ────────────────────────
|
||||
- name: Download artifacts
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v7
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const runId = context.payload.workflow_run.id;
|
||||
|
||||
const allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
run_id: runId,
|
||||
});
|
||||
|
||||
async function downloadArtifact(name, dest) {
|
||||
const match = allArtifacts.data.artifacts.find(a => a.name === name);
|
||||
if (!match) {
|
||||
core.warning(`Artifact "${name}" not found`);
|
||||
return false;
|
||||
}
|
||||
const zip = await github.rest.actions.downloadArtifact({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
artifact_id: match.id,
|
||||
archive_format: 'zip',
|
||||
});
|
||||
fs.mkdirSync(dest, { recursive: true });
|
||||
fs.writeFileSync(path.join(dest, `${name}.zip`), Buffer.from(zip.data));
|
||||
return true;
|
||||
}
|
||||
|
||||
const temp = process.env.RUNNER_TEMP;
|
||||
await downloadArtifact('pr-meta', path.join(temp, 'dl'));
|
||||
await downloadArtifact('test-reports', path.join(temp, 'dl'));
|
||||
|
||||
- name: Extract artifacts
|
||||
shell: bash
|
||||
run: |
|
||||
cd "$RUNNER_TEMP/dl"
|
||||
# Extract each artifact into its own directory to avoid filename collisions
|
||||
for z in *.zip; do
|
||||
[ -f "$z" ] || continue
|
||||
name="${z%.zip}"
|
||||
mkdir -p "$RUNNER_TEMP/artifacts/$name"
|
||||
unzip -o "$z" -d "$RUNNER_TEMP/artifacts/$name"
|
||||
done
|
||||
|
||||
- name: Read PR metadata
|
||||
id: meta
|
||||
shell: bash
|
||||
run: |
|
||||
DIR="$RUNNER_TEMP/artifacts/pr-meta"
|
||||
if [ ! -f "$DIR/pr_number" ]; then
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
echo "::warning::pr_number artifact missing — skipping report"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Validate PR number is a positive integer (artifact comes from
|
||||
# untrusted fork code, so treat contents defensively).
|
||||
PR_NUM=$(cat "$DIR/pr_number" | tr -d '[:space:]')
|
||||
if ! [[ "$PR_NUM" =~ ^[0-9]+$ ]]; then
|
||||
echo "skip=true" >> "$GITHUB_OUTPUT"
|
||||
echo "::error::Invalid PR number in artifact: '$PR_NUM'"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
echo "skip=false" >> "$GITHUB_OUTPUT"
|
||||
echo "pr_number=$PR_NUM" >> "$GITHUB_OUTPUT"
|
||||
# Validate job-result strings against known GitHub Actions values.
|
||||
# Artifact contents come from the PR workflow (potentially untrusted
|
||||
# fork code), so we whitelist to prevent newline injection into
|
||||
# GITHUB_OUTPUT.
|
||||
validate_result() {
|
||||
local val
|
||||
val=$(cat "$1" | tr -d '[:space:]')
|
||||
case "$val" in
|
||||
success|failure|cancelled|skipped) echo "$val" ;;
|
||||
*) echo "unknown" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
echo "quality=$(validate_result "$DIR/quality_result")" >> "$GITHUB_OUTPUT"
|
||||
echo "tests=$(validate_result "$DIR/tests_result")" >> "$GITHUB_OUTPUT"
|
||||
echo "e2e=$(validate_result "$DIR/e2e_result")" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Checkout (for vitest config)
|
||||
if: steps.meta.outputs.skip != 'true'
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
sparse-checkout: gitnexus/vitest.config.ts
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
# ── Fetch base branch coverage for delta reporting ───────────
|
||||
- name: Fetch base branch coverage
|
||||
if: steps.meta.outputs.skip != 'true'
|
||||
id: base-coverage
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v7
|
||||
with:
|
||||
script: |
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
|
||||
// Find the latest successful CI run on main
|
||||
const runs = await github.rest.actions.listWorkflowRuns({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
workflow_id: 'ci.yml',
|
||||
branch: 'main',
|
||||
status: 'success',
|
||||
per_page: 1,
|
||||
});
|
||||
|
||||
if (runs.data.workflow_runs.length === 0) {
|
||||
core.setOutput('found', 'false');
|
||||
core.info('No successful main branch CI runs found');
|
||||
return;
|
||||
}
|
||||
|
||||
const mainRunId = runs.data.workflow_runs[0].id;
|
||||
const artifacts = await github.rest.actions.listWorkflowRunArtifacts({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
run_id: mainRunId,
|
||||
});
|
||||
|
||||
const testReports = artifacts.data.artifacts.find(a => a.name === 'test-reports');
|
||||
if (!testReports) {
|
||||
core.setOutput('found', 'false');
|
||||
core.info('No test-reports artifact on main branch');
|
||||
return;
|
||||
}
|
||||
|
||||
const zip = await github.rest.actions.downloadArtifact({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
artifact_id: testReports.id,
|
||||
archive_format: 'zip',
|
||||
});
|
||||
|
||||
const dest = path.join(process.env.RUNNER_TEMP, 'base-coverage');
|
||||
fs.mkdirSync(dest, { recursive: true });
|
||||
fs.writeFileSync(path.join(dest, 'base.zip'), Buffer.from(zip.data));
|
||||
core.setOutput('found', 'true');
|
||||
core.setOutput('dir', dest);
|
||||
|
||||
- name: Extract base coverage
|
||||
if: steps.meta.outputs.skip != 'true' && steps.base-coverage.outputs.found == 'true'
|
||||
shell: bash
|
||||
run: |
|
||||
cd "${{ steps.base-coverage.outputs.dir }}"
|
||||
mkdir -p base
|
||||
unzip -o base.zip -d base
|
||||
|
||||
- name: Build report
|
||||
if: steps.meta.outputs.skip != 'true'
|
||||
id: report
|
||||
shell: bash
|
||||
env:
|
||||
QUALITY: ${{ steps.meta.outputs.quality }}
|
||||
TESTS: ${{ steps.meta.outputs.tests }}
|
||||
E2E: ${{ steps.meta.outputs.e2e }}
|
||||
BASE_FOUND: ${{ steps.base-coverage.outputs.found }}
|
||||
BASE_DIR: ${{ steps.base-coverage.outputs.dir }}
|
||||
RUN_URL: ${{ github.event.workflow_run.html_url }}
|
||||
run: |
|
||||
DIR="$RUNNER_TEMP/artifacts"
|
||||
|
||||
# ── Helper: read coverage summary into prefixed vars ──
|
||||
read_cov() {
|
||||
local prefix=$1 file=$2
|
||||
if [ -n "$file" ] && [ -f "$file" ]; then
|
||||
local val
|
||||
val=$(jq -r '.total.statements.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
|
||||
printf -v "${prefix}_STMTS" '%s' "$val"
|
||||
val=$(jq -r '.total.branches.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
|
||||
printf -v "${prefix}_BRANCH" '%s' "$val"
|
||||
val=$(jq -r '.total.functions.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
|
||||
printf -v "${prefix}_FUNCS" '%s' "$val"
|
||||
val=$(jq -r '.total.lines.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
|
||||
printf -v "${prefix}_LINES" '%s' "$val"
|
||||
val=$(jq -r '"\(.total.statements.covered)/\(.total.statements.total)"' "$file" 2>/dev/null) || val=""
|
||||
printf -v "${prefix}_STMTS_COV" '%s' "$val"
|
||||
val=$(jq -r '"\(.total.branches.covered)/\(.total.branches.total)"' "$file" 2>/dev/null) || val=""
|
||||
printf -v "${prefix}_BRANCH_COV" '%s' "$val"
|
||||
val=$(jq -r '"\(.total.functions.covered)/\(.total.functions.total)"' "$file" 2>/dev/null) || val=""
|
||||
printf -v "${prefix}_FUNCS_COV" '%s' "$val"
|
||||
val=$(jq -r '"\(.total.lines.covered)/\(.total.lines.total)"' "$file" 2>/dev/null) || val=""
|
||||
printf -v "${prefix}_LINES_COV" '%s' "$val"
|
||||
return 0
|
||||
else
|
||||
printf -v "${prefix}_STMTS" '%s' "N/A"
|
||||
printf -v "${prefix}_BRANCH" '%s' "N/A"
|
||||
printf -v "${prefix}_FUNCS" '%s' "N/A"
|
||||
printf -v "${prefix}_LINES" '%s' "N/A"
|
||||
printf -v "${prefix}_STMTS_COV" '%s' ""
|
||||
printf -v "${prefix}_BRANCH_COV" '%s' ""
|
||||
printf -v "${prefix}_FUNCS_COV" '%s' ""
|
||||
printf -v "${prefix}_LINES_COV" '%s' ""
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Read coverage reports ──
|
||||
UNIT_SUMMARY=$(find "$DIR/test-reports" -name "coverage-summary.json" -type f 2>/dev/null | head -1)
|
||||
|
||||
read_cov "U" "$UNIT_SUMMARY"
|
||||
|
||||
# ── Read base branch coverage (main) ──
|
||||
BASE_SUMMARY=""
|
||||
if [ "$BASE_FOUND" = "true" ] && [ -n "$BASE_DIR" ]; then
|
||||
BASE_SUMMARY=$(find "$BASE_DIR/base" -name "coverage-summary.json" -type f 2>/dev/null | head -1)
|
||||
fi
|
||||
read_cov "B" "$BASE_SUMMARY"
|
||||
|
||||
# ── Locate test results ──
|
||||
RESULTS_FILE=$(find "$DIR/test-reports" -name "test-results.json" -type f 2>/dev/null | head -1)
|
||||
WEB_RESULTS_FILE=$(find "$DIR/test-reports" -name "web-test-results.json" -type f 2>/dev/null | head -1)
|
||||
|
||||
sum_results() {
|
||||
local file=$1
|
||||
if [ -n "$file" ] && [ -f "$file" ]; then
|
||||
jq -r '"\(.numTotalTests) \(.numPassedTests) \(.numFailedTests) \(.numPendingTests) \(.numTotalTestSuites) \(((.testResults | map(.endTime) | max) - (.startTime)) / 1000 | floor)"' "$file" 2>/dev/null || echo "0 0 0 0 0 0"
|
||||
else
|
||||
echo "0 0 0 0 0 0"
|
||||
fi
|
||||
}
|
||||
|
||||
read CLI_T CLI_P CLI_F CLI_S CLI_SU CLI_D <<< "$(sum_results "$RESULTS_FILE")"
|
||||
read WEB_T WEB_P WEB_F WEB_S WEB_SU WEB_D <<< "$(sum_results "$WEB_RESULTS_FILE")"
|
||||
|
||||
TOTAL=$((CLI_T + WEB_T))
|
||||
PASSED=$((CLI_P + WEB_P))
|
||||
FAILED=$((CLI_F + WEB_F))
|
||||
SKIPPED=$((CLI_S + WEB_S))
|
||||
SUITES=$((CLI_SU + WEB_SU))
|
||||
DURATION=$((CLI_D > WEB_D ? CLI_D : WEB_D))
|
||||
|
||||
# ── Status helpers ──
|
||||
status_icon() {
|
||||
case "$1" in
|
||||
success) echo "✅" ;;
|
||||
failure) echo "❌" ;;
|
||||
cancelled) echo "⏭️" ;;
|
||||
*) echo "❓" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# Validate a value looks like a number (integer or decimal, optional
|
||||
# leading minus). Returns 1 for anything else — guards against awk
|
||||
# injection when artifact values come from untrusted fork code.
|
||||
is_numeric() { [[ "$1" =~ ^-?[0-9]+(\.[0-9]+)?$ ]]; }
|
||||
|
||||
cov_delta() {
|
||||
local pct=$1 base=$2
|
||||
if [ "$pct" = "N/A" ] || [ "$base" = "N/A" ]; then echo "—"; return; fi
|
||||
if ! is_numeric "$pct" || ! is_numeric "$base"; then echo "—"; return; fi
|
||||
local diff
|
||||
diff=$(awk -v p="$pct" -v b="$base" 'BEGIN { printf "%.1f", p - b }')
|
||||
if [ "$(awk -v p="$pct" -v b="$base" 'BEGIN { print (p > b) ? 1 : 0 }')" = "1" ]; then
|
||||
echo "📈 +${diff}"
|
||||
elif [ "$(awk -v p="$pct" -v b="$base" 'BEGIN { print (p < b) ? 1 : 0 }')" = "1" ]; then
|
||||
echo "📉 ${diff}"
|
||||
else
|
||||
echo "＝ ${diff}"
|
||||
fi
|
||||
}
|
||||
|
||||
cov_bar() {
|
||||
local pct=$1 base=$2
|
||||
if [ "$pct" = "N/A" ] || ! is_numeric "$pct"; then echo "—"; return; fi
|
||||
local filled
|
||||
filled=$(awk -v p="$pct" 'BEGIN { printf "%d", p / 5 }')
|
||||
(( filled < 0 )) && filled=0
|
||||
(( filled > 20 )) && filled=20
|
||||
local empty=$((20 - filled))
|
||||
local bar=""
|
||||
for ((i=0; i<filled; i++)); do bar+="█"; done
|
||||
for ((i=0; i<empty; i++)); do bar+="░"; done
|
||||
# Green if >= base (or base unavailable), red if dropped
|
||||
if [ "$base" = "N/A" ] || ! is_numeric "$base" || [ "$(awk -v p="$pct" -v b="$base" 'BEGIN { print (p >= b) ? 1 : 0 }')" = "1" ]; then
|
||||
echo "🟢 ${bar}"
|
||||
else
|
||||
echo "🔴 ${bar}"
|
||||
fi
|
||||
}
|
||||
|
||||
# ── Overall status ──
|
||||
if [[ "$QUALITY" == "success" && "$TESTS" == "success" && ("$E2E" == "success" || "$E2E" == "skipped") ]]; then
|
||||
OVERALL="✅ **All checks passed**"
|
||||
else
|
||||
OVERALL="❌ **Some checks failed**"
|
||||
fi
|
||||
|
||||
# ── Build markdown ──
|
||||
{
|
||||
echo "body<<GITNEXUS_CI_REPORT_EOF_7f3a"
|
||||
echo "## CI Report"
|
||||
echo ""
|
||||
echo "${OVERALL}"
|
||||
echo ""
|
||||
echo "### Pipeline Status"
|
||||
echo ""
|
||||
echo "| Stage | Status | Details |"
|
||||
echo "|-------|--------|---------|"
|
||||
echo "| $(status_icon "$QUALITY") Typecheck | \`${QUALITY}\` | tsc --noEmit |"
|
||||
echo "| $(status_icon "$TESTS") Tests | \`${TESTS}\` | unit tests, 3 platforms |"
|
||||
echo "| $(status_icon "$E2E") E2E | \`${E2E}\` | gitnexus-web changes only |"
|
||||
echo ""
|
||||
|
||||
if [ "$TOTAL" -gt 0 ] 2>/dev/null; then
|
||||
echo "### Test Results"
|
||||
echo ""
|
||||
echo "| Tests | Passed | Failed | Skipped | Duration |"
|
||||
echo "|-------|--------|--------|---------|----------|"
|
||||
echo "| ${TOTAL} | ${PASSED} | ${FAILED} | ${SKIPPED} | ${DURATION}s |"
|
||||
echo ""
|
||||
|
||||
if [ "$FAILED" = "0" ]; then
|
||||
echo "✅ All **${PASSED}** tests passed"
|
||||
else
|
||||
echo "❌ **${FAILED}** failed / **${PASSED}** passed"
|
||||
fi
|
||||
if [ "$SKIPPED" != "0" ]; then
|
||||
echo ""
|
||||
echo "<details>"
|
||||
echo "<summary>${SKIPPED} test(s) skipped — expand for details</summary>"
|
||||
echo ""
|
||||
for rf in "$RESULTS_FILE" "$WEB_RESULTS_FILE"; do
|
||||
if [ -n "$rf" ] && [ -f "$rf" ]; then
|
||||
jq -r '
|
||||
.testResults[]
|
||||
| .assertionResults[]?
|
||||
| select(.status == "pending" or .status == "skipped")
|
||||
| "- \(.ancestorTitles | join(" > ")) > \(.title)"
|
||||
' "$rf" 2>/dev/null || true
|
||||
fi
|
||||
done
|
||||
echo ""
|
||||
echo "</details>"
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# ── Coverage table helper ──
|
||||
cov_table() {
|
||||
local label=$1 s=$2 b=$3 f=$4 l=$5 sc=$6 bc=$7 fc=$8 lc=$9
|
||||
shift 9
|
||||
local bs=$1 bb=$2 bf=$3 bl=$4
|
||||
echo "#### ${label}"
|
||||
echo ""
|
||||
echo "| Metric | Coverage | Covered | Base | Delta | Status |"
|
||||
echo "|--------|----------|---------|------|-------|--------|"
|
||||
echo "| Statements | **${s}%** | ${sc} | ${bs}% | $(cov_delta "$s" "$bs") | $(cov_bar "$s" "$bs") |"
|
||||
echo "| Branches | **${b}%** | ${bc} | ${bb}% | $(cov_delta "$b" "$bb") | $(cov_bar "$b" "$bb") |"
|
||||
echo "| Functions | **${f}%** | ${fc} | ${bf}% | $(cov_delta "$f" "$bf") | $(cov_bar "$f" "$bf") |"
|
||||
echo "| Lines | **${l}%** | ${lc} | ${bl}% | $(cov_delta "$l" "$bl") | $(cov_bar "$l" "$bl") |"
|
||||
echo ""
|
||||
}
|
||||
|
||||
if [ "$U_STMTS" != "N/A" ]; then
|
||||
echo "### Code Coverage"
|
||||
echo ""
|
||||
cov_table "Tests" \
|
||||
"$U_STMTS" "$U_BRANCH" "$U_FUNCS" "$U_LINES" \
|
||||
"$U_STMTS_COV" "$U_BRANCH_COV" "$U_FUNCS_COV" "$U_LINES_COV" \
|
||||
"$B_STMTS" "$B_BRANCH" "$B_FUNCS" "$B_LINES"
|
||||
else
|
||||
echo "### Code Coverage"
|
||||
echo ""
|
||||
echo "⚠️ Coverage data unavailable - check the [unit test job](${RUN_URL}) for details."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
echo "---"
|
||||
echo "<sub>📋 [View full run](${RUN_URL}) · Generated by CI</sub>"
|
||||
echo "GITNEXUS_CI_REPORT_EOF_7f3a"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Comment on PR
|
||||
if: steps.meta.outputs.skip != 'true'
|
||||
uses: marocchino/sticky-pull-request-comment@0ea0beb66eb9baf113663a64ec522f60e49231c0 # v2
|
||||
with:
|
||||
header: ci-report
|
||||
number: ${{ steps.meta.outputs.pr_number }}
|
||||
message: ${{ steps.report.outputs.body }}
|
||||
@@ -1,110 +0,0 @@
|
||||
name: Scope Resolution Parity
|
||||
|
||||
# Reusable workflow — called from ci.yml. Does NOT declare concurrency;
|
||||
# it inherits the caller's concurrency group per the convention documented
|
||||
# in CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
#
|
||||
# ── Purpose (RFC #909 Ring 3, §6.4 "Observability gates") ──────────────
|
||||
# For every language in `MIGRATED_LANGUAGES` (exported from
|
||||
# `gitnexus/src/core/ingestion/registry-primary-flag.ts`), run the
|
||||
# resolver integration test at `test/integration/resolvers/<slug>.test.ts`
|
||||
# TWICE on every PR:
|
||||
#
|
||||
# 1. `REGISTRY_PRIMARY_<LANG>=0` — legacy DAG path (guarantees we haven't
|
||||
# broken the old path while migrating). Known legacy gaps may be skipped
|
||||
# through the resolver test helper's expected-failure list.
|
||||
# 2. `REGISTRY_PRIMARY_<LANG>=1` — registry-primary path (guarantees the
|
||||
# new path carries the same behavior — the parity gate).
|
||||
#
|
||||
# BOTH must pass. The source of truth is the TypeScript constant — adding
|
||||
# a language to that `Set` is the ONLY contributor action; CI auto-
|
||||
# discovers it, runs parity, and the language's default production path
|
||||
# flips to registry-primary in the same change.
|
||||
#
|
||||
# When the set is empty (e.g. mid-Ring-3 for every language), the parity
|
||||
# matrix is skipped and the workflow reports success — no-op until a
|
||||
# language is explicitly claimed migrated.
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
discover:
|
||||
name: Discover migrated languages
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
outputs:
|
||||
languages: ${{ steps.read.outputs.languages }}
|
||||
has-any: ${{ steps.read.outputs.has-any }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
|
||||
- name: Extract MIGRATED_LANGUAGES from registry-primary-flag.ts
|
||||
id: read
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
run: |
|
||||
set -euo pipefail
|
||||
# `tsx` evaluates the TS source directly (no build step), imports
|
||||
# the exported `Set`, and emits a GH-Actions-friendly JSON matrix.
|
||||
LANGS=$(npx tsx scripts/ci-list-migrated-languages.ts)
|
||||
COUNT=$(printf '%s' "$LANGS" | jq 'length')
|
||||
HAS_ANY="false"
|
||||
if [[ "$COUNT" -gt 0 ]]; then HAS_ANY="true"; fi
|
||||
echo "languages=$LANGS" >> "$GITHUB_OUTPUT"
|
||||
echo "has-any=$HAS_ANY" >> "$GITHUB_OUTPUT"
|
||||
echo "Discovered $COUNT migrated language(s): $LANGS"
|
||||
echo "Parity matrix will run: $HAS_ANY"
|
||||
|
||||
parity:
|
||||
name: ${{ matrix.lang.slug }} parity
|
||||
needs: discover
|
||||
if: needs.discover.outputs.has-any == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
strategy:
|
||||
# One language failing must not abort the others — we want the full
|
||||
# parity matrix result on a single CI run so a reviewer sees every
|
||||
# regression at once rather than one-at-a-time.
|
||||
fail-fast: false
|
||||
matrix:
|
||||
lang: ${{ fromJSON(needs.discover.outputs.languages) }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
with:
|
||||
build: 'true'
|
||||
|
||||
- name: Verify resolver test file exists
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
run: |
|
||||
set -euo pipefail
|
||||
TEST_FILE="test/integration/resolvers/${{ matrix.lang.slug }}.test.ts"
|
||||
if [[ ! -f "$TEST_FILE" ]]; then
|
||||
echo "::error title=Missing resolver test::\
|
||||
Expected $TEST_FILE for '${{ matrix.lang.slug }}' (listed in \
|
||||
MIGRATED_LANGUAGES). Either fix the slug or add the test file \
|
||||
before listing this language as migrated."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
- name: Resolver tests — legacy DAG (REGISTRY_PRIMARY_${{ matrix.lang.envvar }}=0)
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
env:
|
||||
FLAG_NAME: REGISTRY_PRIMARY_${{ matrix.lang.envvar }}
|
||||
# Explicitly force the flag to `0` even though it also defaults to
|
||||
# `MIGRATED_LANGUAGES.has(lang)` — once a language is in the set,
|
||||
# the default flips to registry-primary, so an unset env var would
|
||||
# silently re-run the same path as step #2. `env FOO=0 cmd` spawns
|
||||
# `cmd` with the override scoped to just this invocation.
|
||||
run: env "$FLAG_NAME=0" npx vitest run "test/integration/resolvers/${{ matrix.lang.slug }}.test.ts"
|
||||
|
||||
- name: Resolver tests — registry-primary (REGISTRY_PRIMARY_${{ matrix.lang.envvar }}=1)
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
env:
|
||||
FLAG_NAME: REGISTRY_PRIMARY_${{ matrix.lang.envvar }}
|
||||
run: env "$FLAG_NAME=1" npx vitest run "test/integration/resolvers/${{ matrix.lang.slug }}.test.ts"
|
||||
@@ -1,74 +0,0 @@
|
||||
name: Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
tests:
|
||||
name: ubuntu / coverage
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
with:
|
||||
build: 'true'
|
||||
|
||||
- name: Run all tests with coverage
|
||||
run: >-
|
||||
npx vitest run
|
||||
--reporter=default
|
||||
--reporter=json
|
||||
--outputFile=test-results.json
|
||||
--coverage
|
||||
--coverage.reporter=json-summary
|
||||
--coverage.reporter=json
|
||||
--coverage.reporter=text
|
||||
--coverage.thresholdAutoUpdate=false
|
||||
--coverage.reportOnFailure=true
|
||||
working-directory: gitnexus
|
||||
|
||||
# gitnexus-shared already built by setup-gitnexus action above
|
||||
- name: Install gitnexus-web dependencies
|
||||
run: npm ci
|
||||
working-directory: gitnexus-web
|
||||
|
||||
- name: Run gitnexus-web unit tests
|
||||
run: >-
|
||||
npx vitest run
|
||||
--reporter=default
|
||||
--reporter=json
|
||||
--outputFile=web-test-results.json
|
||||
working-directory: gitnexus-web
|
||||
|
||||
- name: Run docker-server integration tests
|
||||
run: node --test docker-server.test.mjs
|
||||
|
||||
- name: Upload test reports
|
||||
if: always()
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
with:
|
||||
name: test-reports
|
||||
path: |
|
||||
gitnexus/coverage/coverage-summary.json
|
||||
gitnexus/coverage/coverage-final.json
|
||||
gitnexus/test-results.json
|
||||
gitnexus-web/web-test-results.json
|
||||
retention-days: 5
|
||||
|
||||
cross-platform:
|
||||
name: ${{ matrix.os }}
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
# Ubuntu already covered by the coverage job above
|
||||
os: [windows-latest, macos-latest]
|
||||
runs-on: ${{ matrix.os }}
|
||||
timeout-minutes: 25
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
with:
|
||||
build: 'true'
|
||||
- run: npx vitest run
|
||||
working-directory: gitnexus
|
||||
@@ -0,0 +1,55 @@
|
||||
name: Unit Tests
|
||||
|
||||
on:
|
||||
workflow_call:
|
||||
|
||||
jobs:
|
||||
unit-tests:
|
||||
name: unit (ubuntu / coverage)
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
|
||||
- name: Run unit tests with coverage
|
||||
run: >-
|
||||
npx vitest run test/unit
|
||||
--reporter=default
|
||||
--reporter=json
|
||||
--outputFile=test-results.json
|
||||
--coverage
|
||||
--coverage.reporter=json-summary
|
||||
--coverage.reporter=json
|
||||
--coverage.reporter=text
|
||||
--coverage.thresholdAutoUpdate=false
|
||||
--coverage.reportOnFailure=true
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Coverage report
|
||||
if: always()
|
||||
uses: davelosert/vitest-coverage-report-action@v2
|
||||
with:
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Upload test reports
|
||||
if: always()
|
||||
uses: actions/upload-artifact@v4
|
||||
with:
|
||||
name: test-reports
|
||||
path: |
|
||||
gitnexus/coverage/coverage-summary.json
|
||||
gitnexus/test-results.json
|
||||
retention-days: 5
|
||||
|
||||
cross-platform:
|
||||
name: unit (${{ matrix.os }})
|
||||
strategy:
|
||||
fail-fast: true
|
||||
matrix:
|
||||
os: [ubuntu-latest, windows-latest, macos-latest]
|
||||
runs-on: ${{ matrix.os }}
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
- run: npx vitest run test/unit
|
||||
working-directory: gitnexus
|
||||
+174
-104
@@ -1,139 +1,209 @@
|
||||
name: CI
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
pull_request:
|
||||
branches: [main]
|
||||
paths-ignore: ['**.md', 'docs/**', 'LICENSE']
|
||||
workflow_call:
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Hardcoded `CI-` prefix (not `${{ github.workflow }}`) because this workflow is
|
||||
# invoked as a reusable workflow from publish.yml and release-candidate.yml. In
|
||||
# called-workflow context `github.workflow` evaluation is ambiguous across GitHub
|
||||
# Actions versions, and a prefix that could resolve to the caller's name would
|
||||
# share a concurrency group with the caller → deadlock. A literal prefix is
|
||||
# immune. Direct `pull_request` invocations use `CI-<ref>`; invocations from a
|
||||
# reusable-workflow caller fall into a per-run-unique group that never serializes
|
||||
# with the caller. `push` to main is handled by release-candidate.yml, which
|
||||
# calls this workflow once before publishing.
|
||||
concurrency:
|
||||
group: ${{ github.event_name == 'pull_request' && format('CI-{0}', github.ref) || format('CI-nested-{0}', github.run_id) }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
# ── Reusable workflow orchestration ─────────────────────────────────
|
||||
# Each concern lives in its own workflow file for maintainability:
|
||||
# ci-quality.yml — typecheck (tsc --noEmit)
|
||||
# ci-tests.yml — unit + integration tests with coverage + cross-platform
|
||||
# ci-e2e.yml — E2E tests (only when gitnexus-web/ changes)
|
||||
# ci-scope-parity.yml — RFC #909 Ring 3 parity gate: legacy DAG + registry-primary
|
||||
# both pass, per migrated language in the JSON registry
|
||||
# ci-unit-tests.yml — unit tests with coverage + cross-platform
|
||||
# ci-integration.yml — integration test matrix (3 OS x 4 groups)
|
||||
#
|
||||
# Shared setup is DRY via .github/actions/setup-gitnexus composite action.
|
||||
|
||||
jobs:
|
||||
quality:
|
||||
uses: ./.github/workflows/ci-quality.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
tests:
|
||||
uses: ./.github/workflows/ci-tests.yml
|
||||
permissions:
|
||||
contents: read
|
||||
unit-tests:
|
||||
uses: ./.github/workflows/ci-unit-tests.yml
|
||||
|
||||
e2e:
|
||||
uses: ./.github/workflows/ci-e2e.yml
|
||||
permissions:
|
||||
contents: read
|
||||
integration:
|
||||
uses: ./.github/workflows/ci-integration.yml
|
||||
|
||||
scope-parity:
|
||||
uses: ./.github/workflows/ci-scope-parity.yml
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# ── Save PR metadata for the reporting workflow ─────────────────
|
||||
# The ci-report.yml workflow (triggered by workflow_run) needs the
|
||||
# PR number and job results to post a comment. We save them as an
|
||||
# artifact because workflow_run context doesn't reliably carry PR
|
||||
# info for fork PRs.
|
||||
save-pr-meta:
|
||||
name: Save PR Metadata
|
||||
# ── PR test & coverage report ────────────────────────────────────
|
||||
# Downloads coverage artifacts from unit tests and posts a summary
|
||||
# comment on the PR with test results and coverage metrics.
|
||||
pr-report:
|
||||
name: PR Report
|
||||
if: always() && github.event_name == 'pull_request'
|
||||
needs: [quality, tests, e2e, scope-parity]
|
||||
needs: [quality, unit-tests, integration]
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
pull-requests: write
|
||||
steps:
|
||||
- name: Write metadata
|
||||
shell: bash
|
||||
env:
|
||||
PR_NUMBER: ${{ github.event.number }}
|
||||
QUALITY: ${{ needs.quality.result }}
|
||||
TESTS: ${{ needs.tests.result }}
|
||||
E2E: ${{ needs.e2e.result }}
|
||||
SCOPE_PARITY: ${{ needs.scope-parity.result }}
|
||||
run: |
|
||||
mkdir -p pr-meta
|
||||
echo "$PR_NUMBER" > pr-meta/pr_number
|
||||
echo "$QUALITY" > pr-meta/quality_result
|
||||
echo "$TESTS" > pr-meta/tests_result
|
||||
echo "$E2E" > pr-meta/e2e_result
|
||||
echo "$SCOPE_PARITY" > pr-meta/scope_parity_result
|
||||
# TODO(post-merge): remove backward-compat copies once ci-report.yml
|
||||
# on main reads underscore names.
|
||||
# Backward-compat: ci-report.yml on main still reads hyphenated
|
||||
# names. workflow_run always executes from the default branch, so
|
||||
# the main-branch reader won't find the underscore variants until
|
||||
# this PR is merged. Write both until then.
|
||||
cp pr-meta/pr_number pr-meta/pr-number
|
||||
cp pr-meta/quality_result pr-meta/quality-result
|
||||
cp pr-meta/tests_result pr-meta/tests-result
|
||||
cp pr-meta/e2e_result pr-meta/e2e-result
|
||||
|
||||
- name: Upload PR metadata
|
||||
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
|
||||
- name: Download test reports
|
||||
uses: actions/download-artifact@v4
|
||||
with:
|
||||
name: pr-meta
|
||||
path: pr-meta/
|
||||
retention-days: 1
|
||||
name: test-reports
|
||||
path: reports
|
||||
continue-on-error: true
|
||||
|
||||
- name: Debug artifact contents
|
||||
run: find reports -type f 2>/dev/null || echo "No reports directory"
|
||||
continue-on-error: true
|
||||
|
||||
- name: Build report
|
||||
id: report
|
||||
run: |
|
||||
# ── Locate coverage file (artifact path may vary) ──
|
||||
COV_FILE=$(find reports -name "coverage-summary.json" -type f 2>/dev/null | head -1)
|
||||
if [ -n "$COV_FILE" ]; then
|
||||
STMTS=$(jq -r '.total.statements.pct' "$COV_FILE")
|
||||
BRANCH=$(jq -r '.total.branches.pct' "$COV_FILE")
|
||||
FUNCS=$(jq -r '.total.functions.pct' "$COV_FILE")
|
||||
LINES=$(jq -r '.total.lines.pct' "$COV_FILE")
|
||||
STMTS_COV=$(jq -r '"\(.total.statements.covered)/\(.total.statements.total)"' "$COV_FILE")
|
||||
BRANCH_COV=$(jq -r '"\(.total.branches.covered)/\(.total.branches.total)"' "$COV_FILE")
|
||||
FUNCS_COV=$(jq -r '"\(.total.functions.covered)/\(.total.functions.total)"' "$COV_FILE")
|
||||
LINES_COV=$(jq -r '"\(.total.lines.covered)/\(.total.lines.total)"' "$COV_FILE")
|
||||
else
|
||||
STMTS="N/A"; BRANCH="N/A"; FUNCS="N/A"; LINES="N/A"
|
||||
STMTS_COV=""; BRANCH_COV=""; FUNCS_COV=""; LINES_COV=""
|
||||
fi
|
||||
|
||||
# ── Locate test results ──
|
||||
RESULTS_FILE=$(find reports -name "test-results.json" -type f 2>/dev/null | head -1)
|
||||
if [ -n "$RESULTS_FILE" ]; then
|
||||
TOTAL=$(jq -r '.numTotalTests' "$RESULTS_FILE")
|
||||
PASSED=$(jq -r '.numPassedTests' "$RESULTS_FILE")
|
||||
FAILED=$(jq -r '.numFailedTests' "$RESULTS_FILE")
|
||||
SKIPPED=$(jq -r '.numPendingTests' "$RESULTS_FILE")
|
||||
SUITES=$(jq -r '.numTotalTestSuites' "$RESULTS_FILE")
|
||||
DURATION=$(jq -r '((.testResults | map(.endTime) | max) - (.startTime)) / 1000 | floor' "$RESULTS_FILE" 2>/dev/null || echo "N/A")
|
||||
else
|
||||
TOTAL="N/A"; PASSED="N/A"; FAILED="N/A"; SKIPPED="N/A"
|
||||
SUITES="N/A"; DURATION="N/A"
|
||||
fi
|
||||
|
||||
# ── Coverage thresholds (from vitest.config.ts P0 settings) ──
|
||||
THRESH_STMTS=26; THRESH_BRANCH=23; THRESH_FUNCS=28; THRESH_LINES=27
|
||||
|
||||
# ── Status helpers ──
|
||||
status_icon() {
|
||||
case "$1" in
|
||||
success) echo "✅" ;;
|
||||
failure) echo "❌" ;;
|
||||
cancelled) echo "⏭️" ;;
|
||||
*) echo "❓" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
cov_bar() {
|
||||
local pct=$1 thresh=$2
|
||||
if [ "$pct" = "N/A" ]; then echo "—"; return; fi
|
||||
local filled=$(echo "$pct / 5" | bc 2>/dev/null || echo 0)
|
||||
local empty=$((20 - filled))
|
||||
local bar=""
|
||||
for ((i=0; i<filled; i++)); do bar+="█"; done
|
||||
for ((i=0; i<empty; i++)); do bar+="░"; done
|
||||
if [ "$(echo "$pct >= $thresh" | bc 2>/dev/null)" = "1" ]; then
|
||||
echo "🟢 ${bar}"
|
||||
else
|
||||
echo "🔴 ${bar}"
|
||||
fi
|
||||
}
|
||||
|
||||
QUALITY="${{ needs.quality.result }}"
|
||||
UNIT="${{ needs.unit-tests.result }}"
|
||||
INTEG="${{ needs.integration.result }}"
|
||||
|
||||
# ── Overall status ──
|
||||
if [[ "$QUALITY" == "success" && "$UNIT" == "success" && "$INTEG" == "success" ]]; then
|
||||
OVERALL="✅ **All checks passed**"
|
||||
else
|
||||
OVERALL="❌ **Some checks failed**"
|
||||
fi
|
||||
|
||||
# ── Build markdown ──
|
||||
{
|
||||
echo "body<<REPORT_EOF"
|
||||
echo "## CI Report"
|
||||
echo ""
|
||||
echo "${OVERALL}"
|
||||
echo ""
|
||||
echo "### Pipeline Status"
|
||||
echo ""
|
||||
echo "| Stage | Status | Details |"
|
||||
echo "|-------|--------|---------|"
|
||||
echo "| $(status_icon "$QUALITY") Typecheck | \`${QUALITY}\` | tsc --noEmit |"
|
||||
echo "| $(status_icon "$UNIT") Unit Tests | \`${UNIT}\` | 3 platforms |"
|
||||
echo "| $(status_icon "$INTEG") Integration | \`${INTEG}\` | 3 OS × 4 groups = 12 jobs |"
|
||||
echo ""
|
||||
|
||||
if [ "$TOTAL" != "N/A" ]; then
|
||||
echo "### Test Results"
|
||||
echo ""
|
||||
if [ "$FAILED" = "0" ]; then
|
||||
echo "✅ **${PASSED}** passed"
|
||||
else
|
||||
echo "❌ **${FAILED}** failed / **${PASSED}** passed"
|
||||
fi
|
||||
if [ "$SKIPPED" != "0" ]; then
|
||||
echo " · ${SKIPPED} skipped"
|
||||
fi
|
||||
echo " · ${SUITES} suites · ${TOTAL} total"
|
||||
if [ "$DURATION" != "N/A" ]; then
|
||||
echo " · ⏱️ ${DURATION}s"
|
||||
fi
|
||||
echo ""
|
||||
fi
|
||||
|
||||
if [ "$STMTS" != "N/A" ]; then
|
||||
echo "### Code Coverage"
|
||||
echo ""
|
||||
echo "| Metric | Coverage | Covered | Threshold | Status |"
|
||||
echo "|--------|----------|---------|-----------|--------|"
|
||||
echo "| Statements | **${STMTS}%** | ${STMTS_COV} | ${THRESH_STMTS}% | $(cov_bar "$STMTS" "$THRESH_STMTS") |"
|
||||
echo "| Branches | **${BRANCH}%** | ${BRANCH_COV} | ${THRESH_BRANCH}% | $(cov_bar "$BRANCH" "$THRESH_BRANCH") |"
|
||||
echo "| Functions | **${FUNCS}%** | ${FUNCS_COV} | ${THRESH_FUNCS}% | $(cov_bar "$FUNCS" "$THRESH_FUNCS") |"
|
||||
echo "| Lines | **${LINES}%** | ${LINES_COV} | ${THRESH_LINES}% | $(cov_bar "$LINES" "$THRESH_LINES") |"
|
||||
echo ""
|
||||
echo "<details>"
|
||||
echo "<summary>Coverage thresholds are auto-ratcheted — they only go up</summary>"
|
||||
echo ""
|
||||
echo "Vitest \`thresholds.autoUpdate\` bumps the floor whenever local coverage exceeds it."
|
||||
echo "CI enforces the current thresholds; developers commit the ratcheted values."
|
||||
echo "</details>"
|
||||
echo ""
|
||||
else
|
||||
echo "### Code Coverage"
|
||||
echo ""
|
||||
echo "⚠️ Coverage data unavailable — check the [unit test job](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) for details."
|
||||
echo ""
|
||||
fi
|
||||
|
||||
echo "---"
|
||||
echo "<sub>📋 [View full run](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }}) · Generated by CI</sub>"
|
||||
echo "REPORT_EOF"
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Comment on PR
|
||||
uses: marocchino/sticky-pull-request-comment@v2
|
||||
with:
|
||||
header: ci-report
|
||||
message: ${{ steps.report.outputs.body }}
|
||||
|
||||
# ── Unified CI gate ──────────────────────────────────────────────
|
||||
# Single required check for branch protection.
|
||||
ci-status:
|
||||
name: CI Gate
|
||||
needs: [quality, tests, e2e, scope-parity]
|
||||
needs: [quality, unit-tests, integration]
|
||||
if: always()
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Check all jobs passed
|
||||
shell: bash
|
||||
env:
|
||||
QUALITY: ${{ needs.quality.result }}
|
||||
TESTS: ${{ needs.tests.result }}
|
||||
E2E: ${{ needs.e2e.result }}
|
||||
SCOPE_PARITY: ${{ needs.scope-parity.result }}
|
||||
run: |
|
||||
echo "Quality: $QUALITY"
|
||||
echo "Tests: $TESTS"
|
||||
echo "E2E: $E2E"
|
||||
echo "Scope parity: $SCOPE_PARITY"
|
||||
if [[ "$QUALITY" != "success" ]] ||
|
||||
[[ "$TESTS" != "success" ]]; then
|
||||
echo "::error::Quality or test jobs failed"
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$E2E" != "success" && "$E2E" != "skipped" ]]; then
|
||||
echo "::error::E2E job failed"
|
||||
exit 1
|
||||
fi
|
||||
# scope-parity is a reusable workflow. With an empty migrated-
|
||||
# languages list, its parity matrix is skipped and the outer
|
||||
# workflow still reports `success`. If any entry's legacy-DAG or
|
||||
# registry-primary run fails, the workflow reports `failure`.
|
||||
# Accept only `success`; `skipped` would mean the entire
|
||||
# discover job was skipped too (upstream failure), which should
|
||||
# still block.
|
||||
if [[ "$SCOPE_PARITY" != "success" ]]; then
|
||||
echo "::error::Scope-resolution parity gate failed (RFC #909 Ring 3)"
|
||||
echo "Quality: ${{ needs.quality.result }}"
|
||||
echo "Unit Tests: ${{ needs.unit-tests.result }}"
|
||||
echo "Integration: ${{ needs.integration.result }}"
|
||||
if [[ "${{ needs.quality.result }}" != "success" ]] ||
|
||||
[[ "${{ needs.unit-tests.result }}" != "success" ]] ||
|
||||
[[ "${{ needs.integration.result }}" != "success" ]]; then
|
||||
echo "::error::One or more CI jobs failed"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -1,96 +1,44 @@
|
||||
name: Claude Code Review
|
||||
|
||||
# Uses pull_request_target so the workflow runs as defined on the default branch,
|
||||
# which allows access to secrets for posting review comments on fork PRs.
|
||||
# SECURITY: The checkout pins the fork's HEAD SHA (not the branch name) to
|
||||
# prevent TOCTOU races (force-push between trigger and checkout). The
|
||||
# claude-code-action sandboxes execution — it does NOT run arbitrary code
|
||||
# from the checked-out source.
|
||||
|
||||
on:
|
||||
# Trigger only when explicitly requested:
|
||||
# - Add the "claude-review" label to a PR, OR
|
||||
# - Comment "@claude" or "/review" on a PR
|
||||
pull_request_target:
|
||||
types: [labeled]
|
||||
issue_comment:
|
||||
types: [created]
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Serialize per-PR to avoid racing review comments.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.issue.number || github.event.pull_request.number }}
|
||||
cancel-in-progress: false
|
||||
pull_request:
|
||||
types: [opened, synchronize, ready_for_review, reopened]
|
||||
# Optional: Only run on specific file changes
|
||||
# paths:
|
||||
# - "src/**/*.ts"
|
||||
# - "src/**/*.tsx"
|
||||
# - "src/**/*.js"
|
||||
# - "src/**/*.jsx"
|
||||
|
||||
jobs:
|
||||
claude-review:
|
||||
# Run only when:
|
||||
# 1. The "claude-review" label is added to a non-draft PR by a trusted contributor, OR
|
||||
# 2. A trusted contributor comments "@claude" or "/review" on a PR
|
||||
if: |
|
||||
(
|
||||
github.event_name == 'pull_request_target' &&
|
||||
github.event.label.name == 'claude-review' &&
|
||||
github.event.pull_request.draft == false &&
|
||||
(github.event.pull_request.author_association == 'OWNER' ||
|
||||
github.event.pull_request.author_association == 'MEMBER' ||
|
||||
github.event.pull_request.author_association == 'COLLABORATOR')
|
||||
) ||
|
||||
(
|
||||
github.event_name == 'issue_comment' &&
|
||||
github.event.issue.pull_request &&
|
||||
(contains(github.event.comment.body, '@claude') ||
|
||||
contains(github.event.comment.body, '/review')) &&
|
||||
(github.event.comment.author_association == 'OWNER' ||
|
||||
github.event.comment.author_association == 'MEMBER' ||
|
||||
github.event.comment.author_association == 'COLLABORATOR')
|
||||
)
|
||||
# Optional: Filter by PR author
|
||||
# if: |
|
||||
# github.event.pull_request.user.login == 'external-contributor' ||
|
||||
# github.event.pull_request.user.login == 'new-developer' ||
|
||||
# github.event.pull_request.author_association == 'FIRST_TIME_CONTRIBUTOR'
|
||||
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
pull-requests: read
|
||||
issues: read
|
||||
id-token: write
|
||||
|
||||
steps:
|
||||
# For issue_comment triggers, resolve the PR number, head SHA, and fork repo
|
||||
- name: Resolve PR context
|
||||
id: pr
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v7
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
script: |
|
||||
let pr;
|
||||
if (context.eventName === 'issue_comment') {
|
||||
const resp = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: context.payload.issue.number,
|
||||
});
|
||||
pr = resp.data;
|
||||
} else {
|
||||
pr = context.payload.pull_request;
|
||||
}
|
||||
core.setOutput('number', pr.number);
|
||||
core.setOutput('sha', pr.head.sha);
|
||||
core.setOutput('repo', pr.head.repo.full_name);
|
||||
core.setOutput('branch', pr.head.ref);
|
||||
|
||||
- name: Checkout PR head
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
repository: ${{ steps.pr.outputs.repo }}
|
||||
ref: ${{ steps.pr.outputs.sha }}
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Run Claude Code Review
|
||||
id: claude-review
|
||||
uses: anthropics/claude-code-action@9469d113c6afd29550c402740f22d1a97dd1209b # v1
|
||||
uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
allowed_non_write_users: '*'
|
||||
show_full_output: true
|
||||
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
|
||||
plugins: 'code-review@claude-code-plugins'
|
||||
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ steps.pr.outputs.number }}'
|
||||
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ github.event.pull_request.number }}'
|
||||
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
|
||||
# or https://code.claude.com/docs/en/cli-reference for available options
|
||||
|
||||
|
||||
@@ -10,102 +10,41 @@ on:
|
||||
pull_request_review:
|
||||
types: [submitted]
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Serialize per-PR/issue to avoid racing comments.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.issue.number || github.event.pull_request.number || github.event.issue.id }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
claude:
|
||||
if: |
|
||||
(
|
||||
github.event_name == 'issue_comment' &&
|
||||
contains(github.event.comment.body, '@claude') &&
|
||||
(github.event.comment.author_association == 'OWNER' ||
|
||||
github.event.comment.author_association == 'MEMBER' ||
|
||||
github.event.comment.author_association == 'COLLABORATOR')
|
||||
) ||
|
||||
(
|
||||
github.event_name == 'pull_request_review_comment' &&
|
||||
contains(github.event.comment.body, '@claude') &&
|
||||
(github.event.comment.author_association == 'OWNER' ||
|
||||
github.event.comment.author_association == 'MEMBER' ||
|
||||
github.event.comment.author_association == 'COLLABORATOR')
|
||||
) ||
|
||||
(
|
||||
github.event_name == 'pull_request_review' &&
|
||||
contains(github.event.review.body, '@claude') &&
|
||||
(github.event.review.author_association == 'OWNER' ||
|
||||
github.event.review.author_association == 'MEMBER' ||
|
||||
github.event.review.author_association == 'COLLABORATOR')
|
||||
) ||
|
||||
(
|
||||
github.event_name == 'issues' &&
|
||||
(contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
|
||||
(github.event.issue.author_association == 'OWNER' ||
|
||||
github.event.issue.author_association == 'MEMBER' ||
|
||||
github.event.issue.author_association == 'COLLABORATOR')
|
||||
)
|
||||
(github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
|
||||
(github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
|
||||
(github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
permissions:
|
||||
contents: read
|
||||
pull-requests: write
|
||||
issues: write
|
||||
pull-requests: read
|
||||
issues: read
|
||||
id-token: write
|
||||
actions: read # required for Claude to read CI results on PRs
|
||||
actions: read # Required for Claude to read CI results on PRs
|
||||
steps:
|
||||
# For PR-related triggers, resolve the fork repo so we can checkout correctly.
|
||||
- name: Resolve PR context
|
||||
id: pr
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v7
|
||||
with:
|
||||
script: |
|
||||
// Determine if this event is PR-related
|
||||
let prNumber = null;
|
||||
if (context.eventName === 'issue_comment' && context.payload.issue.pull_request) {
|
||||
prNumber = context.payload.issue.number;
|
||||
} else if (context.eventName === 'pull_request_review_comment') {
|
||||
prNumber = context.payload.pull_request.number;
|
||||
} else if (context.eventName === 'pull_request_review') {
|
||||
prNumber = context.payload.pull_request.number;
|
||||
}
|
||||
|
||||
if (!prNumber) {
|
||||
core.setOutput('is_pr', 'false');
|
||||
return;
|
||||
}
|
||||
|
||||
const resp = await github.rest.pulls.get({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
pull_number: prNumber,
|
||||
});
|
||||
const pr = resp.data;
|
||||
|
||||
core.setOutput('is_pr', 'true');
|
||||
core.setOutput('number', String(prNumber));
|
||||
core.setOutput('sha', pr.head.sha);
|
||||
core.setOutput('repo', pr.head.repo.full_name);
|
||||
core.setOutput('branch', pr.head.ref);
|
||||
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
uses: actions/checkout@v4
|
||||
with:
|
||||
repository: ${{ steps.pr.outputs.is_pr == 'true' && steps.pr.outputs.repo || github.repository }}
|
||||
ref: ${{ steps.pr.outputs.is_pr == 'true' && steps.pr.outputs.sha || '' }}
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Run Claude Code
|
||||
id: claude
|
||||
uses: anthropics/claude-code-action@9469d113c6afd29550c402740f22d1a97dd1209b # v1
|
||||
uses: anthropics/claude-code-action@v1
|
||||
with:
|
||||
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
|
||||
github_token: ${{ secrets.GITHUB_TOKEN }}
|
||||
allowed_non_write_users: '*'
|
||||
show_full_output: true
|
||||
|
||||
# This is an optional setting that allows Claude to read CI results on PRs
|
||||
additional_permissions: |
|
||||
actions: read
|
||||
|
||||
# Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
|
||||
# prompt: 'Update the pull request description to include a summary of changes.'
|
||||
|
||||
# Optional: Add claude_args to customize behavior and configuration
|
||||
# See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
|
||||
# or https://code.claude.com/docs/en/cli-reference for available options
|
||||
# claude_args: '--allowed-tools Bash(gh pr:*)'
|
||||
|
||||
|
||||
@@ -1,259 +0,0 @@
|
||||
name: Docker Build & Push
|
||||
|
||||
on:
|
||||
push:
|
||||
tags:
|
||||
- 'v*'
|
||||
pull_request:
|
||||
# workflow_dispatch is allowed for dry-run testing only. Publishing is still
|
||||
# exclusively tag-driven so that every signed image corresponds 1:1 to a
|
||||
# published `gitnexus@X.Y.Z` on npm. dry_run:true (the default) skips all
|
||||
# push, sign, and attestation steps — the build runs but nothing is published.
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
dry_run:
|
||||
description: 'Build only — skip push, signing, and attestations'
|
||||
required: false
|
||||
default: true
|
||||
type: boolean
|
||||
workflow_call:
|
||||
inputs:
|
||||
tag:
|
||||
description: >-
|
||||
The full v-prefixed tag to build (e.g. v1.2.3-rc.1).
|
||||
The tag must already exist in the repo and its tree must contain
|
||||
a gitnexus/package.json whose version matches the tag.
|
||||
required: true
|
||||
type: string
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Tag refs are unique per release, so distinct tags run in parallel.
|
||||
# Re-pushes of the same tag serialize. cancel-in-progress: false — never cancel a publish mid-flight.
|
||||
# Hardcoded `docker-build-push-` prefix (not `${{ github.workflow }}`) when invoked as a reusable
|
||||
# workflow: in called-workflow context `github.workflow` is ambiguous and could resolve to the
|
||||
# caller's name, sharing a concurrency group with the caller → deadlock.
|
||||
# Direct tag-push invocations use `docker-build-push-<ref>`; workflow_call invocations get a
|
||||
# per-run-unique group (they are already serialized by the caller's own concurrency group).
|
||||
concurrency:
|
||||
group: ${{ (github.event_name == 'push') && format('docker-build-push-{0}', github.ref) || format('docker-build-push-nested-{0}', github.run_id) }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build-push:
|
||||
name: Build & Push ${{ matrix.image.name }}
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 60
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
# Required for Cosign keyless signing via the OIDC token exchange,
|
||||
# and for build provenance / SBOM attestations.
|
||||
id-token: write
|
||||
attestations: write
|
||||
|
||||
strategy:
|
||||
fail-fast: false
|
||||
matrix:
|
||||
image:
|
||||
# Static UI bundle. Small, fast image. Drop-in replacement for the
|
||||
# legacy single-image setup at the same `gitnexus` repository slug
|
||||
# is intentionally avoided — the UI now lives at `gitnexus-web` and
|
||||
# the CLI/server takes the canonical `gitnexus` slug below.
|
||||
- name: gitnexus-web
|
||||
dockerfile: Dockerfile.web
|
||||
slug: gitnexus-web
|
||||
# CLI / `gitnexus serve` backend. Heavy native deps (tree-sitter,
|
||||
# onnxruntime-node) live only in this image.
|
||||
- name: gitnexus
|
||||
dockerfile: Dockerfile.cli
|
||||
slug: gitnexus
|
||||
|
||||
steps:
|
||||
# Only the workflow_call path requires a non-empty `inputs.tag` — callers
|
||||
# (e.g. release-candidate.yml) must pass the RC tag explicitly. On direct
|
||||
# tag pushes the tag comes from `github.ref`, so `inputs.tag` is always
|
||||
# empty and validating it here would break every real release (#1064).
|
||||
# The downstream "Verify tag matches gitnexus/package.json version" step
|
||||
# handles both event types by falling back to GITHUB_REF.
|
||||
- name: Validate tag input
|
||||
if: github.event_name == 'workflow_call'
|
||||
shell: bash
|
||||
env:
|
||||
TAG_INPUT: ${{ inputs.tag }}
|
||||
run: |
|
||||
if [ -z "${TAG_INPUT}" ]; then
|
||||
echo "::error::No tag provided to docker.yml — refusing to build/push."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# When triggered by workflow_call the caller passes the RC tag as an input;
|
||||
# we check out that tag so the Dockerfile and package.json match the built image.
|
||||
# For tag-push events github.ref is already the tag ref — no override needed.
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
ref: ${{ inputs.tag || github.ref }}
|
||||
|
||||
# ── Lock the docker image version to the npm package version ──────────
|
||||
# Mirrors the check in publish.yml: refuse to build unless the git tag
|
||||
# exactly matches `gitnexus/package.json`'s version. This guarantees
|
||||
# `ghcr.io/<owner>/gitnexus:X.Y.Z` always corresponds to the same
|
||||
# `gitnexus@X.Y.Z` published to npm — no drift, no surprises.
|
||||
- name: Verify tag matches gitnexus/package.json version
|
||||
id: version
|
||||
if: github.event_name != 'workflow_dispatch' && github.event_name != 'pull_request'
|
||||
shell: bash
|
||||
env:
|
||||
# For workflow_call the tag comes from the caller input; for push events
|
||||
# it is derived from GITHUB_REF (set to empty so the else-branch fires).
|
||||
INPUT_TAG: ${{ inputs.tag }}
|
||||
run: |
|
||||
if [ -n "$INPUT_TAG" ]; then
|
||||
TAG_VERSION="${INPUT_TAG#v}"
|
||||
else
|
||||
TAG_VERSION="${GITHUB_REF#refs/tags/v}"
|
||||
fi
|
||||
if ! [[ "$TAG_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$ ]]; then
|
||||
echo "::error::Tag does not follow semver: v$TAG_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
PKG_VERSION=$(node -p "require('./gitnexus/package.json').version")
|
||||
if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
|
||||
echo "::error::Tag version (v$TAG_VERSION) does not match gitnexus/package.json version ($PKG_VERSION)"
|
||||
exit 1
|
||||
fi
|
||||
echo "version=$PKG_VERSION" >> "$GITHUB_OUTPUT"
|
||||
echo "Version verified: $PKG_VERSION"
|
||||
|
||||
# Required for multi-platform (linux/arm64) emulation.
|
||||
- name: Set up QEMU
|
||||
uses: docker/setup-qemu-action@ce360397dd3f832beb865e1373c09c0e9f86d70a # v4.0.0
|
||||
|
||||
- name: Set up Docker Buildx
|
||||
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0
|
||||
|
||||
- name: Install Cosign
|
||||
uses: sigstore/cosign-installer@cad07c2e89fa2edd6e2d7bab4c1aa38e53f76003 # v4.1.1
|
||||
|
||||
- name: Log in to GitHub Container Registry
|
||||
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
|
||||
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
|
||||
with:
|
||||
registry: ghcr.io
|
||||
username: ${{ github.actor }}
|
||||
password: ${{ secrets.GITHUB_TOKEN }}
|
||||
|
||||
# Docker Hub is a mirror of GHCR: same tags, same digests, same Cosign
|
||||
# signatures. GHCR remains authoritative (it is the registry the
|
||||
# ClusterImagePolicy globs against by default), but Docker Hub is the
|
||||
# registry most users reach for first, so we publish there too.
|
||||
# Requires repo secrets DOCKERHUB_USERNAME and DOCKERHUB_TOKEN (a scoped
|
||||
# access token, NOT the account password) with write access to the
|
||||
# `akonlabs/gitnexus` and `akonlabs/gitnexus-web` repos.
|
||||
- name: Log in to Docker Hub
|
||||
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
|
||||
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
|
||||
with:
|
||||
username: ${{ secrets.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
|
||||
# Computes image tags and labels from the verified semver tag:
|
||||
# v1.2.3 → :1.2.3, :1.2, :1, :latest (auto, only for non-prerelease)
|
||||
# v1.2.3-rc.1 → :1.2.3-rc.1 only (prereleases never become :latest)
|
||||
# `:latest` is only emitted for tag pushes thanks to `flavor: latest=auto`,
|
||||
# ensuring it always points at a real npm-published version.
|
||||
#
|
||||
# For workflow_call invocations github.ref is the caller's branch ref, so
|
||||
# the type=semver patterns would not match. In that case we add an explicit
|
||||
# type=raw tag using the version already verified above, so the same
|
||||
# image-naming rules apply regardless of how the workflow was triggered.
|
||||
# NOTE: We check `inputs.tag` rather than `github.event_name` because in a
|
||||
# reusable workflow the github context is inherited from the caller —
|
||||
# `github.event_name` would still be "push", not "workflow_call".
|
||||
- name: Extract Docker metadata
|
||||
id: meta
|
||||
uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
|
||||
with:
|
||||
# Dual-registry publish. metadata-action expands the same tag set
|
||||
# against every image ref listed here, and build-push-action pushes
|
||||
# one build to all of them, so the GHCR and Docker Hub images share
|
||||
# a digest and are byte-identical. The Docker Hub namespace
|
||||
# (`akonlabs`) is hardcoded because it differs from the GitHub org
|
||||
# (`abhigyanpatwari`) — `github.repository_owner` would produce the
|
||||
# wrong ref.
|
||||
images: |
|
||||
ghcr.io/${{ github.repository_owner }}/${{ matrix.image.slug }}
|
||||
docker.io/akonlabs/${{ matrix.image.slug }}
|
||||
flavor: latest=auto
|
||||
tags: |
|
||||
type=semver,pattern={{version}}
|
||||
type=semver,pattern={{major}}.{{minor}}
|
||||
type=semver,pattern={{major}}
|
||||
type=raw,value=${{ steps.version.outputs.version }},enable=${{ inputs.tag != '' }}
|
||||
|
||||
# Transient 502s from GHCR / Docker Hub / GHA cache during multi-platform
|
||||
# exports are retried inside `.github/actions/docker-build-push-retry`
|
||||
# (see docker/build-push-action#1422 — retry policy stays out of the
|
||||
# upstream action). `ignore-error=true` on cache-to avoids cache export
|
||||
# flakes failing an otherwise successful push.
|
||||
- name: Build and push
|
||||
id: build
|
||||
uses: ./.github/actions/docker-build-push-retry
|
||||
with:
|
||||
context: .
|
||||
file: ${{ matrix.image.dockerfile }}
|
||||
platforms: linux/amd64,linux/arm64
|
||||
push: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
|
||||
tags: ${{ steps.meta.outputs.tags }}
|
||||
labels: ${{ steps.meta.outputs.labels }}
|
||||
cache-from: type=gha,scope=${{ matrix.image.slug }}
|
||||
cache-to: type=gha,mode=max,scope=${{ matrix.image.slug }},ignore-error=true
|
||||
|
||||
# Cosign keyless signing. Each pushed tag is signed by the workflow's
|
||||
# OIDC identity, so consumers can verify the image with the strict,
|
||||
# fully-anchored identity regex (kept in sync with README.md and
|
||||
# deploy/kubernetes/cluster-image-policy.yaml — update all three together).
|
||||
# NOTE: `${...}` expression syntax is NOT evaluated inside YAML comments, so
|
||||
# the example below uses literal `<owner>/<repo>` placeholders that consumers
|
||||
# substitute themselves; the canonical, fully-rendered command lives in README.md.
|
||||
# cosign verify ghcr.io/<owner>/<slug>:<tag> \
|
||||
# --certificate-identity-regexp '^https://github\.com/<owner>/<repo>/\.github/workflows/docker\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$' \
|
||||
# --certificate-oidc-issuer https://token.actions.githubusercontent.com
|
||||
# Do NOT relax to `@.*` — that accepts signatures from any ref, including
|
||||
# unprotected branches and PRs, and defeats the supply-chain guarantee.
|
||||
- name: Sign image with Cosign (keyless)
|
||||
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
|
||||
env:
|
||||
# Cosign v2 (installed by sigstore/cosign-installer above) makes
|
||||
# keyless the default. COSIGN_EXPERIMENTAL is a v1-only opt-in flag
|
||||
# that is now deprecated/no-op, so it is intentionally omitted.
|
||||
DIGEST: ${{ steps.build.outputs.digest }}
|
||||
TAGS: ${{ steps.meta.outputs.tags }}
|
||||
run: |
|
||||
# Sign every tag at the same digest so consumers can verify by tag or by digest.
|
||||
# Use `while read` instead of `for $TAGS` to be robust against tags that
|
||||
# could ever contain whitespace (the metadata-action output is newline-
|
||||
# separated, not space-separated).
|
||||
while IFS= read -r tag; do
|
||||
[[ -n "$tag" ]] && cosign sign --yes "${tag}@${DIGEST}"
|
||||
done <<< "$TAGS"
|
||||
|
||||
# Attach the SBOM produced by buildx as a verifiable attestation on the
|
||||
# digest. Attestations are pushed as OCI referrers to the registry named
|
||||
# in `subject-name`, so we call the action once per registry. The digest
|
||||
# is identical across registries (same build, same push), so consumers
|
||||
# pulling from either GHCR or Docker Hub see the same provenance.
|
||||
- name: Generate build provenance attestation (GHCR)
|
||||
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
|
||||
uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
|
||||
with:
|
||||
subject-name: ghcr.io/${{ github.repository_owner }}/${{ matrix.image.slug }}
|
||||
subject-digest: ${{ steps.build.outputs.digest }}
|
||||
push-to-registry: true
|
||||
|
||||
- name: Generate build provenance attestation (Docker Hub)
|
||||
if: ${{ github.event_name != 'pull_request' && !inputs.dry_run }}
|
||||
uses: actions/attest-build-provenance@a2bbfa25375fe432b6a289bc6b6cd05ecd0c4c32 # v4.1.0
|
||||
with:
|
||||
subject-name: docker.io/akonlabs/${{ matrix.image.slug }}
|
||||
subject-digest: ${{ steps.build.outputs.digest }}
|
||||
push-to-registry: true
|
||||
@@ -1,95 +0,0 @@
|
||||
name: PR Description Check
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
types: [opened, edited, reopened]
|
||||
branches: [main]
|
||||
|
||||
permissions:
|
||||
pull-requests: write
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
check-description:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
steps:
|
||||
- name: Check PR description quality
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
const MIN_BODY_LENGTH = 50;
|
||||
const LABEL = 'needs-description';
|
||||
|
||||
const pr = context.payload.pull_request;
|
||||
const body = (pr.body || '').trim();
|
||||
const owner = context.repo.owner;
|
||||
const repo = context.repo.repo;
|
||||
const number = pr.number;
|
||||
|
||||
const hasLabel = pr.labels.some(l => l.name === LABEL);
|
||||
|
||||
if (body.length < MIN_BODY_LENGTH) {
|
||||
// Add label if not already present
|
||||
if (!hasLabel) {
|
||||
await github.rest.issues.addLabels({
|
||||
owner, repo, issue_number: number,
|
||||
labels: [LABEL],
|
||||
});
|
||||
}
|
||||
|
||||
// Post or update a comment
|
||||
const marker = '<!-- pr-desc-check -->';
|
||||
const message = [
|
||||
marker,
|
||||
`### PR description is too short`,
|
||||
'',
|
||||
`This PR's description is **${body.length}** characters, ` +
|
||||
`but the minimum is **${MIN_BODY_LENGTH}**.`,
|
||||
'',
|
||||
'Please update the PR description to explain:',
|
||||
'- **What** this PR changes',
|
||||
'- **Why** the change is needed',
|
||||
'',
|
||||
'Use the PR template as a guide. This check will re-run when you edit the description.',
|
||||
].join('\n');
|
||||
|
||||
// Find existing bot comment to update (avoid spam)
|
||||
const comments = await github.rest.issues.listComments({
|
||||
owner, repo, issue_number: number,
|
||||
});
|
||||
const existing = comments.data.find(c =>
|
||||
c.body && c.body.includes(marker)
|
||||
);
|
||||
|
||||
if (existing) {
|
||||
await github.rest.issues.updateComment({
|
||||
owner, repo, comment_id: existing.id,
|
||||
body: message,
|
||||
});
|
||||
} else {
|
||||
await github.rest.issues.createComment({
|
||||
owner, repo, issue_number: number,
|
||||
body: message,
|
||||
});
|
||||
}
|
||||
|
||||
core.setFailed(
|
||||
`PR description is ${body.length} chars (minimum: ${MIN_BODY_LENGTH})`
|
||||
);
|
||||
} else {
|
||||
// Description is acceptable — remove the label if present
|
||||
if (hasLabel) {
|
||||
await github.rest.issues.removeLabel({
|
||||
owner, repo, issue_number: number,
|
||||
name: LABEL,
|
||||
}).catch(() => {});
|
||||
// .catch: label may have been removed manually
|
||||
}
|
||||
|
||||
core.info(`PR description OK (${body.length} chars)`);
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
name: PR Conventional Labeler
|
||||
|
||||
# Two workflows in one file with different triggers, matched to the minimum
|
||||
# privilege each needs:
|
||||
#
|
||||
# validate-title (on: pull_request)
|
||||
# Fork-safe. Runs with the PR-head's read-only GITHUB_TOKEN. Uses
|
||||
# `amannn/action-semantic-pull-request` to fail the check when the PR
|
||||
# title doesn't follow the conventional-commit format. Because the
|
||||
# action only reads the event payload, no fork-controlled code runs.
|
||||
#
|
||||
# autolabel (on: pull_request_target)
|
||||
# Needs `pull-requests: write` to apply labels, so must be
|
||||
# pull_request_target. Uses `release-drafter/release-drafter` with
|
||||
# `dry-run: true` to only run the autolabeler against the
|
||||
# `.github/release-drafter.yml` config from the BASE ref (release-
|
||||
# drafter reads the config from the repository's default branch, NOT
|
||||
# the PR head — verify with `gh api repos/release-drafter/release-drafter/contents/...`
|
||||
# or a fork-test PR before merging if the repo is high-value).
|
||||
# `sync-labels: true` in the config removes managed autolabels that no
|
||||
# longer match (e.g. when `!` or `BREAKING CHANGE:` is dropped).
|
||||
#
|
||||
# Title format: <type>[(scope)][!]: <subject>
|
||||
# Allowed types: feat, fix, perf, refactor, docs, test, ci, build, chore, revert, deps
|
||||
# Trailing `!` on the type marks a breaking change.
|
||||
# See CONTRIBUTING.md → "Pull request titles".
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
# Title-only changes fire `edited`. `opened` and `reopened` cover creation.
|
||||
# `synchronize` (push to the PR branch) is intentionally excluded — titles
|
||||
# don't change on push, so it only wastes CI minutes and broadens the
|
||||
# privileged-token exposure window on the autolabel job.
|
||||
types: [opened, edited, reopened]
|
||||
pull_request_target:
|
||||
types: [opened, edited, reopened]
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Include `github.event_name` so `pull_request` (validate-title) and
|
||||
# `pull_request_target` (autolabel) runs for the same PR do NOT share a slot
|
||||
# and therefore cannot cancel each other — a cancelled required-check would
|
||||
# permanently block merge until the next title edit.
|
||||
# Within each trigger the latest title edit still supersedes the prior run.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
validate-title:
|
||||
# Fork-safe job — only runs on `pull_request` (not `pull_request_target`).
|
||||
# Token is read-only; writes a commit status that branch protection can
|
||||
# require before merge.
|
||||
name: Validate PR title
|
||||
if: github.event_name == 'pull_request'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
pull-requests: read
|
||||
steps:
|
||||
# Pinned to v6.1.1. Verify SHA via:
|
||||
# gh api repos/amannn/action-semantic-pull-request/git/refs/tags/v6.1.1
|
||||
- uses: amannn/action-semantic-pull-request@48f256284bd46cdaab1048c3721360e808335d50 # v6.1.1
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
with:
|
||||
types: |
|
||||
feat
|
||||
fix
|
||||
perf
|
||||
refactor
|
||||
docs
|
||||
test
|
||||
ci
|
||||
build
|
||||
chore
|
||||
revert
|
||||
deps
|
||||
requireScope: false
|
||||
# Subject must be non-empty. We DO allow capitalized proper nouns
|
||||
# (MCP, GitHub, API, etc.) — the old `^(?![A-Z]).+$` pattern
|
||||
# rejected legitimate titles like `fix: MCP tool schema`.
|
||||
subjectPattern: ^\S.{2,}$
|
||||
subjectPatternError: |
|
||||
The subject "{subject}" in PR title "{title}" is invalid.
|
||||
Subjects must be at least 3 characters and must not start with whitespace.
|
||||
wip: false
|
||||
|
||||
autolabel:
|
||||
# Privileged job — runs only on `pull_request_target` so it can write labels.
|
||||
# Never checks out fork code, never executes fork-controlled input; only
|
||||
# reads the PR metadata (title, body, labels) and calls the GitHub API.
|
||||
name: Apply conventional label
|
||||
if: github.event_name == 'pull_request_target'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
# `contents: read` is required — release-drafter's context.config() reads
|
||||
# `.github/release-drafter.yml` from the repo's default branch via the
|
||||
# repo-contents API. Without it the job silently 403s and no labels are
|
||||
# applied. Job-level permissions nullify all unlisted scopes, so an
|
||||
# explicit grant is necessary here.
|
||||
contents: read
|
||||
pull-requests: write
|
||||
steps:
|
||||
# Pinned to v7.2.0. Verify SHA via:
|
||||
# gh api repos/release-drafter/release-drafter/git/refs/tags/v7.2.0
|
||||
# v7 removed `disable-releaser`; use `dry-run: true` to only autolabel.
|
||||
- uses: release-drafter/release-drafter@563bf132657a13ded0b01fcb723c5a58cdd824e2 # v7.2.1
|
||||
with:
|
||||
config-name: release-drafter.yml
|
||||
dry-run: true
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
@@ -5,55 +5,35 @@ on:
|
||||
tags:
|
||||
- 'v*'
|
||||
|
||||
# No workflow-level permissions — scoped per job below.
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Tag refs are unique per release, so distinct tags run in parallel. Re-pushes of the
|
||||
# same tag serialize. cancel-in-progress: false — never cancel a publish mid-flight.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
permissions:
|
||||
contents: write
|
||||
id-token: write
|
||||
pull-requests: write
|
||||
|
||||
jobs:
|
||||
ci:
|
||||
uses: ./.github/workflows/ci.yml
|
||||
permissions:
|
||||
contents: read
|
||||
actions: read
|
||||
# No pull-requests:write — `ci.yml`'s save-pr-meta job is gated on
|
||||
# `github.event_name == 'pull_request'`, so it never runs during a
|
||||
# tag-triggered publish. Least-privilege for release-critical paths.
|
||||
|
||||
publish:
|
||||
needs: ci
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 15
|
||||
permissions:
|
||||
contents: write
|
||||
id-token: write
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-node@v4
|
||||
with:
|
||||
node-version: 20
|
||||
registry-url: https://registry.npmjs.org
|
||||
cache: npm
|
||||
cache-dependency-path: gitnexus/package-lock.json
|
||||
- name: Build gitnexus-shared
|
||||
run: npm install && npm run build
|
||||
working-directory: gitnexus-shared
|
||||
|
||||
- run: npm ci
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Verify version consistency
|
||||
shell: bash
|
||||
run: |
|
||||
TAG_VERSION="${GITHUB_REF#refs/tags/v}"
|
||||
if ! [[ "$TAG_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$ ]]; then
|
||||
echo "::error::Tag does not follow semver: v$TAG_VERSION"
|
||||
exit 1
|
||||
fi
|
||||
PKG_VERSION=$(node -p "require('./package.json').version")
|
||||
if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
|
||||
echo "::error::Tag version (v$TAG_VERSION) does not match package.json version ($PKG_VERSION)"
|
||||
@@ -76,22 +56,7 @@ jobs:
|
||||
env:
|
||||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||
|
||||
- name: Extract release notes from CHANGELOG
|
||||
id: changelog
|
||||
shell: bash
|
||||
run: |
|
||||
VERSION="${GITHUB_REF#refs/tags/v}"
|
||||
NOTES=$(awk "/^## \\[$VERSION\\]/{found=1; next} /^## \\[/{if(found) exit} found" gitnexus/CHANGELOG.md)
|
||||
if [ -z "$NOTES" ]; then
|
||||
echo "::warning::No CHANGELOG entry found for v$VERSION, falling back to auto-generated notes"
|
||||
echo "fallback=true" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "$NOTES" > /tmp/release-notes.md
|
||||
echo "fallback=false" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
- name: Create GitHub Release
|
||||
uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v2
|
||||
uses: softprops/action-gh-release@v2
|
||||
with:
|
||||
body_path: ${{ steps.changelog.outputs.fallback == 'false' && '/tmp/release-notes.md' || '' }}
|
||||
generate_release_notes: ${{ steps.changelog.outputs.fallback == 'true' }}
|
||||
generate_release_notes: true
|
||||
|
||||
@@ -1,392 +0,0 @@
|
||||
name: Release Candidate
|
||||
|
||||
on:
|
||||
# Publish a release-candidate build whenever a merge/commit lands on main.
|
||||
# Docs/README-only changes are filtered out so prose updates don't
|
||||
# cut a release.
|
||||
push:
|
||||
branches: [main]
|
||||
paths-ignore:
|
||||
- '**.md'
|
||||
- 'docs/**'
|
||||
- 'LICENSE'
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
bump:
|
||||
description: >-
|
||||
Cycle policy. 'auto' (default) continues the active rc cycle on
|
||||
this branch if there is one, otherwise bumps patch from latest.
|
||||
Choose 'patch' / 'minor' / 'major' to explicitly start or reset
|
||||
an rc cycle.
|
||||
required: false
|
||||
default: 'auto'
|
||||
type: choice
|
||||
options:
|
||||
- auto
|
||||
- patch
|
||||
- minor
|
||||
- major
|
||||
force:
|
||||
description: 'Publish even when HEAD already has an rc marker'
|
||||
required: false
|
||||
default: 'false'
|
||||
type: choice
|
||||
options:
|
||||
- 'false'
|
||||
- 'true'
|
||||
|
||||
# No workflow-level permissions — scoped per job below.
|
||||
permissions: {}
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Serialize all runs on the same ref (push + workflow_dispatch) to prevent two publishes
|
||||
# racing on the rc counter. cancel-in-progress: false — the earlier merge publishes first.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
# ── Skip when HEAD already has an rc marker (retry / duplicate dispatch) ──
|
||||
# The marker is a lightweight tag `rc/<HEAD_SHA>` pushed *before* `npm
|
||||
# publish`, so a failed publish leaves the marker in place and the guard
|
||||
# refuses to re-publish. Recovery path after a partial failure:
|
||||
# git push --delete origin rc/<HEAD_SHA> v<RC_VERSION>
|
||||
# then redispatch with force=true.
|
||||
guard:
|
||||
name: Check if release candidate should run
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 5
|
||||
permissions:
|
||||
contents: read
|
||||
outputs:
|
||||
should_run: ${{ steps.decide.outputs.should_run }}
|
||||
head_sha: ${{ steps.decide.outputs.head_sha }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
fetch-depth: 0
|
||||
fetch-tags: true
|
||||
|
||||
- name: Decide
|
||||
id: decide
|
||||
shell: bash
|
||||
env:
|
||||
FORCE: ${{ inputs.force }}
|
||||
BUMP_INPUT: ${{ inputs.bump }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
HEAD_SHA=$(git rev-parse HEAD)
|
||||
echo "head_sha=$HEAD_SHA" >> "$GITHUB_OUTPUT"
|
||||
|
||||
if [ "$FORCE" = "true" ]; then
|
||||
echo "Force flag set — running regardless of marker tag."
|
||||
echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# An explicit cycle reset on dispatch (bump != auto) also bypasses
|
||||
# the dedup guard — the maintainer is deliberately asking for a
|
||||
# new rc from the same commit.
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] \
|
||||
&& [ -n "${BUMP_INPUT:-}" ] \
|
||||
&& [ "${BUMP_INPUT:-auto}" != "auto" ]; then
|
||||
echo "Explicit bump=$BUMP_INPUT — bypassing marker dedup."
|
||||
echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Dedup: is there already an rc/<HEAD_SHA> marker pointing at HEAD?
|
||||
MARKER="rc/${HEAD_SHA}"
|
||||
if git rev-parse "refs/tags/$MARKER" >/dev/null 2>&1; then
|
||||
echo "HEAD already has marker $MARKER — skipping."
|
||||
echo "should_run=false" >> "$GITHUB_OUTPUT"
|
||||
else
|
||||
echo "No marker on HEAD — proceeding."
|
||||
echo "should_run=true" >> "$GITHUB_OUTPUT"
|
||||
fi
|
||||
|
||||
# ── Reuse the stable CI workflow ─────────────────────────────────────
|
||||
ci:
|
||||
needs: guard
|
||||
if: needs.guard.outputs.should_run == 'true'
|
||||
uses: ./.github/workflows/ci.yml
|
||||
permissions:
|
||||
contents: read
|
||||
secrets: inherit
|
||||
|
||||
# ── Publish the rc build to npm + create GitHub prerelease ───────────
|
||||
publish:
|
||||
name: Publish release candidate to npm
|
||||
needs: [guard, ci]
|
||||
if: needs.guard.outputs.should_run == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 20
|
||||
permissions:
|
||||
contents: write # push rc tag + marker
|
||||
id-token: write # npm provenance
|
||||
outputs:
|
||||
vtag: ${{ steps.reltag.outputs.vtag }}
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
with:
|
||||
fetch-depth: 0
|
||||
fetch-tags: true
|
||||
|
||||
- uses: actions/setup-node@48b55a011bda9f5d6aeb4c2d9c7362e8dae4041e # v6.4.0
|
||||
with:
|
||||
node-version: 20
|
||||
registry-url: https://registry.npmjs.org
|
||||
cache: npm
|
||||
cache-dependency-path: gitnexus/package-lock.json
|
||||
|
||||
- name: Build gitnexus-shared
|
||||
run: npm install && npm run build
|
||||
working-directory: gitnexus-shared
|
||||
|
||||
- name: Install gitnexus dependencies
|
||||
run: npm ci
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Resolve rc version
|
||||
id: version
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
env:
|
||||
BUMP_INPUT: ${{ inputs.bump }}
|
||||
EVENT_NAME: ${{ github.event_name }}
|
||||
PKG_NAME: gitnexus
|
||||
run: |
|
||||
set -euo pipefail
|
||||
|
||||
# 1. Current published `latest` — the floor for any new rc base.
|
||||
# Only E404 ("never published") falls back to package.json; any
|
||||
# other error (network, auth, malformed response) fails fast.
|
||||
NPM_STDERR_LATEST="$(mktemp)"
|
||||
if CURRENT_LATEST="$(npm view "$PKG_NAME" version 2>"$NPM_STDERR_LATEST")"; then
|
||||
:
|
||||
else
|
||||
if grep -q 'E404' "$NPM_STDERR_LATEST"; then
|
||||
CURRENT_LATEST="$(node -p "require('./package.json').version")"
|
||||
echo "Package not on registry (E404) — seeding from package.json: $CURRENT_LATEST"
|
||||
else
|
||||
echo "::error::npm registry unreachable for 'view version':" >&2
|
||||
cat "$NPM_STDERR_LATEST" >&2
|
||||
rm -f "$NPM_STDERR_LATEST"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
rm -f "$NPM_STDERR_LATEST"
|
||||
CURRENT_LATEST_CLEAN="${CURRENT_LATEST%%-*}"
|
||||
|
||||
# 2. Full version list — needed for the counter and for active-cycle
|
||||
# inference. Same E404-only fallback.
|
||||
NPM_STDERR_VERSIONS="$(mktemp)"
|
||||
if VERSIONS_JSON="$(npm view "$PKG_NAME" versions --json 2>"$NPM_STDERR_VERSIONS")"; then
|
||||
:
|
||||
else
|
||||
if grep -q 'E404' "$NPM_STDERR_VERSIONS"; then
|
||||
VERSIONS_JSON='[]'
|
||||
echo "No published versions for $PKG_NAME yet (E404)."
|
||||
else
|
||||
echo "::error::npm registry unreachable for 'view versions':" >&2
|
||||
cat "$NPM_STDERR_VERSIONS" >&2
|
||||
rm -f "$NPM_STDERR_VERSIONS"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
rm -f "$NPM_STDERR_VERSIONS"
|
||||
|
||||
# 3. Base selection.
|
||||
# - workflow_dispatch + bump ∈ {patch,minor,major} → explicit cycle
|
||||
# reset from latest.
|
||||
# - Everything else (push, or dispatch with bump=auto) → continue
|
||||
# the highest active rc base > latest if one exists; else
|
||||
# default to patch from latest.
|
||||
if [ "$EVENT_NAME" = "workflow_dispatch" ] \
|
||||
&& [ -n "${BUMP_INPUT:-}" ] \
|
||||
&& [ "${BUMP_INPUT:-auto}" != "auto" ]; then
|
||||
BASE="$(npx --yes -p semver@7 semver -i "$BUMP_INPUT" "$CURRENT_LATEST_CLEAN")"
|
||||
echo "Explicit bump=$BUMP_INPUT → BASE=$BASE"
|
||||
else
|
||||
cat > /tmp/active_base.mjs <<'NODESCRIPT'
|
||||
const latest = process.env.LATEST;
|
||||
let v;
|
||||
try { v = JSON.parse(process.env.VERSIONS_JSON); } catch { v = []; }
|
||||
if (!Array.isArray(v)) v = [v];
|
||||
const parse = s => s.split(".").map(n => parseInt(n, 10));
|
||||
const gt = (a, b) => {
|
||||
const [A, B] = [parse(a), parse(b)];
|
||||
for (let i = 0; i < 3; i++) if (A[i] !== B[i]) return A[i] > B[i];
|
||||
return false;
|
||||
};
|
||||
const bases = new Set();
|
||||
for (const s of v) {
|
||||
const m = /^(\d+\.\d+\.\d+)-rc\.\d+$/.exec(s);
|
||||
if (m && gt(m[1], latest)) bases.add(m[1]);
|
||||
}
|
||||
if (!bases.size) { process.stdout.write(""); process.exit(0); }
|
||||
const sorted = [...bases].sort((a, b) => gt(a, b) ? 1 : -1);
|
||||
process.stdout.write(sorted[sorted.length - 1]);
|
||||
NODESCRIPT
|
||||
ACTIVE_BASE="$(LATEST="$CURRENT_LATEST_CLEAN" VERSIONS_JSON="$VERSIONS_JSON" node /tmp/active_base.mjs)"
|
||||
if [ -n "$ACTIVE_BASE" ]; then
|
||||
BASE="$ACTIVE_BASE"
|
||||
echo "Continuing active rc cycle → BASE=$BASE"
|
||||
else
|
||||
BASE="$(npx --yes -p semver@7 semver -i patch "$CURRENT_LATEST_CLEAN")"
|
||||
echo "No active rc cycle → patch bump from latest → BASE=$BASE"
|
||||
fi
|
||||
fi
|
||||
|
||||
# 4. Counter: 1 + max existing N for `${BASE}-rc.*`, else 1.
|
||||
cat > /tmp/next_rc.mjs <<'NODESCRIPT'
|
||||
const base = process.env.BASE;
|
||||
const prefix = base + "-rc.";
|
||||
let v;
|
||||
try { v = JSON.parse(process.env.VERSIONS_JSON); } catch { v = []; }
|
||||
if (!Array.isArray(v)) v = [v];
|
||||
const ns = v
|
||||
.filter(s => typeof s === "string" && s.startsWith(prefix))
|
||||
.map(s => parseInt(s.slice(prefix.length), 10))
|
||||
.filter(n => Number.isInteger(n) && n >= 0);
|
||||
process.stdout.write(String(ns.length ? Math.max(...ns) + 1 : 1));
|
||||
NODESCRIPT
|
||||
NEXT_N="$(BASE="$BASE" VERSIONS_JSON="$VERSIONS_JSON" node /tmp/next_rc.mjs)"
|
||||
RC_VERSION="${BASE}-rc.${NEXT_N}"
|
||||
echo "Computed rc: $RC_VERSION"
|
||||
|
||||
# 5. Defensive: if the exact version already exists on the registry
|
||||
# (e.g., race with another run), abort before re-publishing.
|
||||
# Same E404-only pattern used above — a transient network
|
||||
# failure must fail loudly, not pretend the version is missing.
|
||||
NPM_STDERR_EXISTS="$(mktemp)"
|
||||
if npm view "$PKG_NAME@$RC_VERSION" version 2>"$NPM_STDERR_EXISTS" >/dev/null; then
|
||||
rm -f "$NPM_STDERR_EXISTS"
|
||||
echo "::error::Version $RC_VERSION already exists on npm — aborting."
|
||||
exit 1
|
||||
else
|
||||
if grep -qiE 'E404|not found' "$NPM_STDERR_EXISTS"; then
|
||||
rm -f "$NPM_STDERR_EXISTS"
|
||||
# Version doesn't exist — safe to proceed.
|
||||
else
|
||||
echo "::error::npm registry unreachable for existence check:" >&2
|
||||
cat "$NPM_STDERR_EXISTS" >&2
|
||||
rm -f "$NPM_STDERR_EXISTS"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
|
||||
echo "base=$BASE" >> "$GITHUB_OUTPUT"
|
||||
echo "rc_n=$NEXT_N" >> "$GITHUB_OUTPUT"
|
||||
echo "rc_version=$RC_VERSION" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Apply rc version in-CI
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
run: |
|
||||
set -euo pipefail
|
||||
npm version "${{ steps.version.outputs.rc_version }}" \
|
||||
--no-git-tag-version --allow-same-version
|
||||
|
||||
- name: Build gitnexus
|
||||
run: npm run build
|
||||
working-directory: gitnexus
|
||||
|
||||
- name: Dry-run publish
|
||||
run: npm publish --dry-run --tag rc
|
||||
working-directory: gitnexus
|
||||
|
||||
# ── Acquire the "rc lock" BEFORE publishing (fixes idempotency) ─────
|
||||
# We create two tags and push them atomically:
|
||||
# v<RC_VERSION> → annotated tag on a detached release commit
|
||||
# whose tree contains the rewritten package.json
|
||||
# (so the tag's source matches the npm tarball)
|
||||
# rc/<HEAD_SHA> → lightweight tag on HEAD; the guard's dedup key
|
||||
# If this push fails, nothing is published — safe.
|
||||
# If this push succeeds but npm publish fails, the marker stays on
|
||||
# the remote and blocks retries until an operator manually cleans up.
|
||||
- name: Create and push rc tags
|
||||
id: reltag
|
||||
shell: bash
|
||||
working-directory: gitnexus
|
||||
env:
|
||||
RC_VERSION: ${{ steps.version.outputs.rc_version }}
|
||||
HEAD_SHA: ${{ needs.guard.outputs.head_sha }}
|
||||
run: |
|
||||
set -euo pipefail
|
||||
VTAG="v${RC_VERSION}"
|
||||
MARKER="rc/${HEAD_SHA}"
|
||||
git config user.name 'github-actions[bot]'
|
||||
git config user.email '41898282+github-actions[bot]@users.noreply.github.com'
|
||||
|
||||
# Detached release commit with the version bump — keeps `main`
|
||||
# pristine but gives the v-tag a tree that matches the published
|
||||
# package contents exactly (fixes release-integrity gap).
|
||||
git add package.json package-lock.json 2>/dev/null || git add package.json
|
||||
git commit -m "release: ${VTAG}" --allow-empty
|
||||
RELEASE_SHA="$(git rev-parse HEAD)"
|
||||
echo "Detached release commit: $RELEASE_SHA"
|
||||
|
||||
# Annotated release tag on the release commit.
|
||||
git tag -a "$VTAG" "$RELEASE_SHA" -m "$VTAG"
|
||||
# Lightweight marker on the user-visible HEAD for the guard.
|
||||
git tag "$MARKER" "$HEAD_SHA"
|
||||
|
||||
# Atomic push of both refs. If either would clobber an existing
|
||||
# remote ref, the push fails and we stop before npm publish.
|
||||
git push --atomic origin "refs/tags/$VTAG" "refs/tags/$MARKER"
|
||||
|
||||
echo "vtag=$VTAG" >> "$GITHUB_OUTPUT"
|
||||
echo "marker=$MARKER" >> "$GITHUB_OUTPUT"
|
||||
echo "release_sha=$RELEASE_SHA" >> "$GITHUB_OUTPUT"
|
||||
|
||||
- name: Publish to npm (rc dist-tag)
|
||||
run: npm publish --provenance --access public --tag rc
|
||||
working-directory: gitnexus
|
||||
env:
|
||||
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
|
||||
|
||||
- name: Create GitHub prerelease
|
||||
uses: softprops/action-gh-release@b4309332981a82ec1c5618f44dd2e27cc8bfbfda # v2
|
||||
with:
|
||||
tag_name: ${{ steps.reltag.outputs.vtag }}
|
||||
name: Release Candidate ${{ steps.reltag.outputs.vtag }}
|
||||
prerelease: true
|
||||
make_latest: 'false'
|
||||
generate_release_notes: true
|
||||
body: |
|
||||
Automated release candidate build from `main`.
|
||||
|
||||
**npm:** `npm install gitnexus@rc`
|
||||
**Version:** `${{ steps.version.outputs.rc_version }}`
|
||||
**Target base:** `${{ steps.version.outputs.base }}` (rc #${{ steps.version.outputs.rc_n }})
|
||||
**Source commit (main):** ${{ needs.guard.outputs.head_sha }}
|
||||
**Release commit (versioned tree):** ${{ steps.reltag.outputs.release_sha }}
|
||||
|
||||
Release candidates are pre-stable builds intended for early testing.
|
||||
Stable releases remain on the `latest` dist-tag.
|
||||
|
||||
# ── Build & push RC Docker images ────────────────────────────────────
|
||||
# Calls docker.yml as a reusable workflow so that the build, signing, and
|
||||
# attestation logic stays in one place. The publish job exposes `vtag`
|
||||
# (e.g. `v1.2.3-rc.1`) as an output so we can pass it as the tag input.
|
||||
# RC images are signed with Cosign keyless signing; the OIDC identity
|
||||
# will be `docker.yml@refs/heads/main` (the caller's ref) rather than a
|
||||
# tag ref — see README.md § Docker for the correct verify command for RCs.
|
||||
docker:
|
||||
name: Build & Push RC Docker images
|
||||
needs: [guard, publish]
|
||||
if: needs.guard.outputs.should_run == 'true' && needs.publish.outputs.vtag != ''
|
||||
uses: ./.github/workflows/docker.yml
|
||||
# Reusable workflows do not receive caller secrets unless inherited; without
|
||||
# this, DOCKERHUB_* / GITHUB_TOKEN are empty in docker.yml → "Username and
|
||||
# password required" on Docker Hub login (see same pattern on `ci:` above).
|
||||
secrets: inherit
|
||||
permissions:
|
||||
contents: read
|
||||
packages: write
|
||||
id-token: write
|
||||
attestations: write
|
||||
with:
|
||||
tag: ${{ needs.publish.outputs.vtag }}
|
||||
@@ -1,185 +0,0 @@
|
||||
name: Tree-sitter Upgrade Readiness
|
||||
|
||||
# Monitors readiness for upgrading tree-sitter to 0.25.x. Tracks:
|
||||
# 1. Peer-dep compatibility — can each grammar install cleanly with
|
||||
# tree-sitter@0.25.0 without --legacy-peer-deps?
|
||||
# 2. Vendored proto drift — has coder3101/tree-sitter-proto moved
|
||||
# ahead of our vendored snapshot?
|
||||
# See .github/scripts/check-tree-sitter-upgrade-readiness.py for the logic.
|
||||
#
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
|
||||
on:
|
||||
schedule:
|
||||
# Daily at 09:00 UTC. Matches Dependabot's daily cadence so drift
|
||||
# and dep PRs surface together.
|
||||
- cron: '0 9 * * *'
|
||||
workflow_dispatch:
|
||||
pull_request:
|
||||
paths:
|
||||
- '.github/scripts/check-tree-sitter-upgrade-readiness.py'
|
||||
- '.github/workflows/tree-sitter-upgrade-readiness.yml'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
readiness:
|
||||
name: Check upgrade readiness
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
permissions:
|
||||
contents: read
|
||||
# Needed to open/update the tracking issue on scheduled runs.
|
||||
issues: write
|
||||
steps:
|
||||
- uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
|
||||
|
||||
- uses: ./.github/actions/setup-gitnexus
|
||||
with:
|
||||
build: 'false'
|
||||
|
||||
- name: Run upgrade readiness check
|
||||
id: readiness
|
||||
shell: bash
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
run: |
|
||||
set +e
|
||||
python3 .github/scripts/check-tree-sitter-upgrade-readiness.py > drift-report.md
|
||||
code=$?
|
||||
set -e
|
||||
echo "exit_code=$code" >> "$GITHUB_OUTPUT"
|
||||
{
|
||||
echo 'report<<DRIFT_EOF'
|
||||
cat drift-report.md
|
||||
echo 'DRIFT_EOF'
|
||||
} >> "$GITHUB_OUTPUT"
|
||||
echo "=== Report ==="
|
||||
cat drift-report.md
|
||||
|
||||
# On PR runs, the script validates that it runs correctly. Blockers
|
||||
# are informational — the scheduled run opens a tracking issue.
|
||||
- name: Annotate PR with readiness status
|
||||
if: github.event_name == 'pull_request' && steps.readiness.outputs.exit_code != '0'
|
||||
run: |
|
||||
echo "::warning::Tree-sitter 0.25 upgrade has blockers. See job output for the full readiness report."
|
||||
|
||||
- name: Upsert tracking issue on scheduled runs
|
||||
if: >
|
||||
github.event_name == 'schedule' &&
|
||||
steps.readiness.outputs.exit_code != '0'
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
env:
|
||||
REPORT: ${{ steps.readiness.outputs.report }}
|
||||
with:
|
||||
script: |
|
||||
const title = 'Tree-sitter 0.25 upgrade readiness';
|
||||
const report = process.env.REPORT;
|
||||
const body = report + '\n\n' +
|
||||
'<sub>Generated daily by `.github/workflows/tree-sitter-upgrade-readiness.yml`. ' +
|
||||
'Closes automatically when all blockers are resolved.</sub>';
|
||||
const { data: open } = await github.rest.issues.listForRepo({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
state: 'open',
|
||||
labels: 'tree-sitter-drift',
|
||||
per_page: 10,
|
||||
});
|
||||
const existing = open.find(i => i.title === title);
|
||||
if (existing) {
|
||||
// Extract ready/total count for the changelog comment.
|
||||
const readyMatch = report.match(/\*\*(\d+)\/(\d+)\*\* grammars ready/);
|
||||
const blockerMatch = report.match(/\*\*(\d+) blocker/);
|
||||
const ready = readyMatch ? readyMatch[1] : '?';
|
||||
const total = readyMatch ? readyMatch[2] : '?';
|
||||
const blockers = blockerMatch ? blockerMatch[1] : '?';
|
||||
|
||||
// Find grammars whose status changed by diffing the old and
|
||||
// new table rows. Each row looks like:
|
||||
// | `tree-sitter-foo` | ... | Ready |
|
||||
// | `tree-sitter-foo` | ... | Blocking |
|
||||
const parseRows = (md) => {
|
||||
const map = {};
|
||||
for (const m of md.matchAll(/\| `(tree-sitter-[^`]+)` \|.*?\| (\S+(?:\s\S+)*?) \|$/gm)) {
|
||||
map[m[1]] = m[2].trim();
|
||||
}
|
||||
return map;
|
||||
};
|
||||
const oldRows = parseRows(existing.body || '');
|
||||
const newRows = parseRows(report);
|
||||
const changes = [];
|
||||
for (const [name, newStatus] of Object.entries(newRows)) {
|
||||
const oldStatus = oldRows[name];
|
||||
if (oldStatus && oldStatus !== newStatus) {
|
||||
changes.push(`\`${name}\`: ${oldStatus} → ${newStatus}`);
|
||||
}
|
||||
}
|
||||
|
||||
const today = new Date().toISOString().slice(0, 10);
|
||||
let comment = `**${today}:** ${ready}/${total} ready. ${blockers} blocker(s) remaining.`;
|
||||
if (changes.length > 0) {
|
||||
comment += '\n\nChanges:\n' + changes.map(c => `- ${c}`).join('\n');
|
||||
} else {
|
||||
comment += ' No changes from previous run.';
|
||||
}
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: existing.number,
|
||||
body: comment,
|
||||
});
|
||||
|
||||
await github.rest.issues.update({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: existing.number,
|
||||
body,
|
||||
});
|
||||
core.info(`Updated existing issue #${existing.number}`);
|
||||
} else {
|
||||
const { data: created } = await github.rest.issues.create({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
title,
|
||||
body,
|
||||
labels: ['tree-sitter-drift', 'dependencies'],
|
||||
});
|
||||
core.info(`Opened issue #${created.number}`);
|
||||
}
|
||||
|
||||
- name: Close tracking issue on clean scheduled runs
|
||||
if: >
|
||||
github.event_name == 'schedule' &&
|
||||
steps.readiness.outputs.exit_code == '0'
|
||||
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
|
||||
with:
|
||||
script: |
|
||||
const title = 'Tree-sitter 0.25 upgrade readiness';
|
||||
const { data: open } = await github.rest.issues.listForRepo({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
state: 'open',
|
||||
labels: 'tree-sitter-drift',
|
||||
per_page: 10,
|
||||
});
|
||||
const existing = open.find(i => i.title === title);
|
||||
if (existing) {
|
||||
await github.rest.issues.createComment({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: existing.number,
|
||||
body: 'All grammars are now compatible with tree-sitter@0.25. Upgrade is ready! Closing automatically.',
|
||||
});
|
||||
await github.rest.issues.update({
|
||||
owner: context.repo.owner,
|
||||
repo: context.repo.repo,
|
||||
issue_number: existing.number,
|
||||
state: 'closed',
|
||||
});
|
||||
core.info(`Closed issue #${existing.number}`);
|
||||
}
|
||||
@@ -1,105 +0,0 @@
|
||||
name: Triage Sweep
|
||||
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
iqr_multiplier:
|
||||
description: >-
|
||||
IQR multiplier for outlier cutoff.
|
||||
cutoff = Q75 + multiplier * IQR.
|
||||
Higher = fewer outliers flagged.
|
||||
type: number
|
||||
default: 3.0
|
||||
max_outlier_pct:
|
||||
description: >-
|
||||
Maximum fraction of items that can be flagged as outliers (0-1).
|
||||
Hard cap to prevent over-flagging.
|
||||
type: number
|
||||
default: 0.05
|
||||
contamination:
|
||||
description: >-
|
||||
Expected fraction of outliers in the data (0-0.5).
|
||||
Controls how aggressively EllipticEnvelope downweights extremes.
|
||||
type: number
|
||||
default: 0.1
|
||||
cosine_threshold:
|
||||
description: >-
|
||||
Cosine similarity threshold for duplicate detection.
|
||||
Pairs with similarity above this are flagged as potential duplicates.
|
||||
Higher = only very similar pairs flagged.
|
||||
type: number
|
||||
default: 0.92
|
||||
max_items:
|
||||
description: >-
|
||||
Maximum number of open issues + PRs to process.
|
||||
Hard cap to prevent runaway costs on very large repos.
|
||||
type: number
|
||||
default: 500
|
||||
dry_run:
|
||||
description: >-
|
||||
Check this to only log results to the workflow summary.
|
||||
Uncheck to create a GitHub issue with the report and apply labels.
|
||||
type: boolean
|
||||
default: false
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
issues: write
|
||||
pull-requests: write
|
||||
|
||||
# Concurrency convention: see CONTRIBUTING.md → "GitHub Actions — Concurrency Convention".
|
||||
# Single global slot — newest manual dispatch supersedes any in-flight run.
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}
|
||||
cancel-in-progress: true
|
||||
|
||||
jobs:
|
||||
sweep:
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
|
||||
with:
|
||||
sparse-checkout: .github/scripts/triage
|
||||
sparse-checkout-cone-mode: false
|
||||
fetch-depth: 1
|
||||
|
||||
- name: Set up Python
|
||||
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
|
||||
with:
|
||||
python-version: '3.12'
|
||||
cache: pip
|
||||
cache-dependency-path: .github/scripts/triage/requirements.txt
|
||||
|
||||
- name: Install dependencies
|
||||
run: pip install -r .github/scripts/triage/requirements.txt
|
||||
|
||||
- name: Cache FastEmbed model weights
|
||||
uses: actions/cache@27d5ce7f107fe9357f9df03efb73ab90386fccae # v5
|
||||
with:
|
||||
path: ${{ github.workspace }}/.fastembed_cache
|
||||
key: fastembed-bge-small-en-v1.5
|
||||
|
||||
- name: Run triage sweep
|
||||
id: sweep
|
||||
env:
|
||||
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
||||
GITHUB_REPOSITORY: ${{ github.repository }}
|
||||
FASTEMBED_CACHE_PATH: ${{ github.workspace }}/.fastembed_cache
|
||||
INPUT_IQR_MULTIPLIER: ${{ inputs.iqr_multiplier }}
|
||||
INPUT_MAX_OUTLIER_PCT: ${{ inputs.max_outlier_pct }}
|
||||
INPUT_CONTAMINATION: ${{ inputs.contamination }}
|
||||
INPUT_COSINE_THRESHOLD: ${{ inputs.cosine_threshold }}
|
||||
INPUT_MAX_ITEMS: ${{ inputs.max_items }}
|
||||
INPUT_DRY_RUN: ${{ inputs.dry_run }}
|
||||
run: python .github/scripts/triage/sweep.py
|
||||
|
||||
- name: Post summary
|
||||
if: always()
|
||||
run: |
|
||||
if [ -f /tmp/triage-report.md ]; then
|
||||
cat /tmp/triage-report.md >> "$GITHUB_STEP_SUMMARY"
|
||||
else
|
||||
echo "No report generated." >> "$GITHUB_STEP_SUMMARY"
|
||||
fi
|
||||
+1
-48
@@ -23,7 +23,6 @@ Thumbs.db
|
||||
.env
|
||||
.env.local
|
||||
.env.*.local
|
||||
docker/.env
|
||||
|
||||
# Logs
|
||||
*.log
|
||||
@@ -34,8 +33,6 @@ coverage/
|
||||
|
||||
# Misc
|
||||
*.local
|
||||
HANDOFF.md
|
||||
HANDOFF*.md
|
||||
|
||||
.vercel
|
||||
|
||||
@@ -51,59 +48,15 @@ HANDOFF*.md
|
||||
# Claude Code worktrees
|
||||
.claude/worktrees/
|
||||
|
||||
# Claude code skills
|
||||
.claude/skills/generated/
|
||||
|
||||
# Assets (screenshots, images)
|
||||
assets/
|
||||
|
||||
# Generated files (should not be indexed)
|
||||
repomix-output*
|
||||
|
||||
# Playwright artifacts
|
||||
gitnexus-web/playwright-report/
|
||||
gitnexus-web/test-results/
|
||||
|
||||
# Python test artifacts
|
||||
eval/.coverage
|
||||
eval/.hypothesis/
|
||||
|
||||
# Design docs (local only)
|
||||
docs/plans/
|
||||
|
||||
gitnexus/test/fixtures/mini-repo/*.md
|
||||
gitnexus/test/fixtures/mini-repo/.claude
|
||||
gitnexus/test/fixtures/mini-repo/.gitignore
|
||||
|
||||
# Ignore csharp generated obj and bin folders
|
||||
gitnexus/test/fixtures/lang-resolution/**/obj
|
||||
gitnexus/test/fixtures/lang-resolution/**/bin
|
||||
GitNexus.sln
|
||||
# Git worktrees
|
||||
.worktrees/
|
||||
|
||||
# Vendored tree-sitter grammar build artifacts (created at install time,
|
||||
# never committed). See docs/plans/2026-04-15-002-fix-tree-sitter-proto-vendor-deps-plan.md
|
||||
gitnexus/vendor/**/build/
|
||||
gitnexus/vendor/**/node_modules/
|
||||
|
||||
/github/scripts/triage/__pycache__/
|
||||
|
||||
.claude-flow/
|
||||
|
||||
.claude/agents/
|
||||
.claude/commands/
|
||||
.claude/helpers
|
||||
.claude/skills/
|
||||
!.claude/skills/gitnexus/
|
||||
|
||||
.history/
|
||||
|
||||
.swarm/
|
||||
|
||||
local_docs/
|
||||
|
||||
# Local agent scratch / review prompts (never commit)
|
||||
.tmp/
|
||||
.agents/
|
||||
.context/
|
||||
gitnexus/test/fixtures/mini-repo/.gitignore
|
||||
@@ -1,33 +0,0 @@
|
||||
import { defineConfig } from 'vitest/config';
|
||||
|
||||
export default defineConfig({
|
||||
test: {
|
||||
globalSetup: ['test/global-setup.ts'],
|
||||
include: ['test/**/*.test.ts'],
|
||||
testTimeout: 30000,
|
||||
hookTimeout: 120000,
|
||||
pool: 'forks',
|
||||
globals: true,
|
||||
setupFiles: ['test/setup.ts'],
|
||||
teardownTimeout: 3000,
|
||||
dangerouslyIgnoreUnhandledErrors: true, // LadybugDB N-API destructor segfaults on fork exit — not a test failure
|
||||
coverage: {
|
||||
provider: 'v8',
|
||||
include: ['src/**/*.ts'],
|
||||
exclude: [
|
||||
'src/cli/index.ts', // CLI entry point (commander wiring)
|
||||
'src/server/**', // HTTP server (requires network)
|
||||
'src/core/wiki/**', // Wiki generation (requires LLM)
|
||||
],
|
||||
// Auto-ratchet: vitest bumps thresholds when coverage exceeds them.
|
||||
// CI will fail if a PR drops below these floors.
|
||||
thresholds: {
|
||||
statements: 26,
|
||||
branches: 23,
|
||||
functions: 28,
|
||||
lines: 27,
|
||||
autoUpdate: true,
|
||||
},
|
||||
},
|
||||
},
|
||||
});
|
||||
@@ -1,26 +0,0 @@
|
||||
#!/usr/bin/env bash
|
||||
# Pre-commit hook: format staged files + typecheck.
|
||||
# Tests run in CI (ci-tests.yml), not here.
|
||||
# Skip with: git commit --no-verify
|
||||
|
||||
ROOT="$(git rev-parse --show-toplevel)"
|
||||
|
||||
# 1. Format staged files with prettier via lint-staged
|
||||
echo "pre-commit: formatting staged files..."
|
||||
"$ROOT/node_modules/.bin/lint-staged" || exit 1
|
||||
|
||||
# 2. Typecheck changed packages
|
||||
WEB_CHANGED=$(git diff --cached --name-only -- 'gitnexus-web/' | head -1)
|
||||
CLI_CHANGED=$(git diff --cached --name-only -- 'gitnexus/' | head -1)
|
||||
|
||||
if [ -n "$WEB_CHANGED" ]; then
|
||||
echo "pre-commit: typechecking gitnexus-web (tsc -b)..."
|
||||
cd "$ROOT/gitnexus-web" && ./node_modules/.bin/tsc -b --noEmit || exit 1
|
||||
fi
|
||||
|
||||
if [ -n "$CLI_CHANGED" ]; then
|
||||
echo "pre-commit: typechecking gitnexus..."
|
||||
cd "$ROOT/gitnexus" && ./node_modules/.bin/tsc --noEmit || exit 1
|
||||
fi
|
||||
|
||||
echo "pre-commit: all checks passed"
|
||||
@@ -1,16 +0,0 @@
|
||||
dist/
|
||||
coverage/
|
||||
gitnexus/vendor/
|
||||
gitnexus/test/fixtures/
|
||||
gitnexus-web/playwright-report/
|
||||
gitnexus-web/test-results/
|
||||
*.d.ts
|
||||
*.snap
|
||||
*.wasm
|
||||
*.md
|
||||
.gitnexus/
|
||||
.vercel/
|
||||
.claude-flow/
|
||||
.swarm/
|
||||
assets/
|
||||
repomix-output*
|
||||
-10
@@ -1,10 +0,0 @@
|
||||
{
|
||||
"semi": true,
|
||||
"singleQuote": true,
|
||||
"trailingComma": "all",
|
||||
"printWidth": 100,
|
||||
"tabWidth": 2,
|
||||
"endOfLine": "lf",
|
||||
"plugins": ["prettier-plugin-tailwindcss"],
|
||||
"tailwindStylesheet": "./gitnexus-web/src/index.css"
|
||||
}
|
||||
@@ -1,130 +1,54 @@
|
||||
<!-- version: 1.7.0 -->
|
||||
<!-- Last updated: 2026-04-23 -->
|
||||
|
||||
Last reviewed: 2026-04-23
|
||||
|
||||
**Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub)
|
||||
|
||||
## Scope
|
||||
|
||||
| Boundary | Rule |
|
||||
|----------|------|
|
||||
| **Reads** | `gitnexus/`, `gitnexus-web/`, `eval/`, plugin packages, `.github/`, `.gitnexus/`, docs. |
|
||||
| **Writes** | Only paths required for the change; keep diffs minimal. Update lockfiles when deps change. |
|
||||
| **Executes** | `npm`, `npx`, `node` under `gitnexus/` and `gitnexus-web/`; `uv run` for Python under `eval/`; documented CI/dev workflows. |
|
||||
| **Off-limits** | Real `.env` / secrets, production credentials, unrelated repos, destructive git ops without confirmation. |
|
||||
|
||||
## Model Configuration
|
||||
|
||||
- **Primary:** Use a named model (e.g. Claude Sonnet 4.x). Avoid `Auto` or unversioned `latest` when reproducibility matters.
|
||||
- **Notes:** The GitNexus CLI indexer does not call an LLM.
|
||||
|
||||
## Execution Sequence (complex tasks)
|
||||
|
||||
For multi-step work, state up front:
|
||||
1. Which rules in this file and **[GUARDRAILS.md](GUARDRAILS.md)** apply (and any relevant Signs).
|
||||
2. Current **Scope** boundaries.
|
||||
3. Which **validation commands** you will run (`cd gitnexus && npm test`, `npx tsc --noEmit`).
|
||||
|
||||
On long threads, *"Remember: apply all AGENTS.md rules"* re-weights these instructions against context dilution.
|
||||
|
||||
## Claude Code hooks
|
||||
|
||||
**PreToolUse** hooks can block tools (e.g. `git_commit`) until checks pass. Adapt to this repo: `cd gitnexus && npm test` before commit.
|
||||
|
||||
## Context budget
|
||||
|
||||
Commands and gotchas live under **Repo reference** below and in **[CONTRIBUTING.md](CONTRIBUTING.md)**. If always-on rules grow, split into **`.cursor/rules/*.mdc`** (globs). **Cursor:** project-wide rules in `.cursor/index.mdc`. **Claude Code:** load `STANDARDS.md` only when needed.
|
||||
|
||||
## Reference docs
|
||||
|
||||
- **[ARCHITECTURE.md](ARCHITECTURE.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**, **[GUARDRAILS.md](GUARDRAILS.md)**
|
||||
- **Call-resolution DAG (legacy path):** See ARCHITECTURE.md § Call-Resolution DAG. Typed 6-stage DAG inside the `parse` phase; language-specific behavior behind `inferImplicitReceiver` / `selectDispatch` hooks on `LanguageProvider`. Shared code in `gitnexus/src/core/ingestion/` must not name languages. Types: `gitnexus/src/core/ingestion/call-types.ts`.
|
||||
- **Scope-resolution pipeline (RFC #909 Ring 3):** See ARCHITECTURE.md § Scope-Resolution Pipeline. Replaces the legacy DAG for languages in `MIGRATED_LANGUAGES` (see `registry-primary-flag.ts`). A language plugs in by implementing `ScopeResolver` (`scope-resolution/contract/scope-resolver.ts`) and registering it in `SCOPE_RESOLVERS`. CI parity gate runs BOTH paths per migrated language on every PR.
|
||||
- **Cursor:** `.cursor/index.mdc` (always-on); `.cursor/rules/*.mdc` (glob-scoped). Legacy `.cursorrules` deprecated.
|
||||
- **GitNexus:** skills in `.claude/skills/gitnexus/`; MCP rules in `gitnexus:start` block below.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Version | Change |
|
||||
|------|---------|--------|
|
||||
| 2026-04-23 | 1.7.0 | TypeScript added to `MIGRATED_LANGUAGES` (registry-primary call resolution by default). |
|
||||
| 2026-04-20 | 1.6.0 | Added scope-resolution pipeline pointer (RFC #909 Ring 3); Python migrated to registry-primary. |
|
||||
| 2026-04-19 | 1.5.0 | Cross-repo impact (#794): `impact`/`query`/`context` accept `repo: "@<group>"` + `service`. Removed `group_query`/`group_contracts`/`group_status` MCP tools; added `gitnexus://group/{name}/contracts` and `gitnexus://group/{name}/status` resources. |
|
||||
| 2026-04-16 | 1.4.0 | Fixed: web UI description, pre-commit behavior, MCP tools (7->16), added gitnexus-shared, removed stale vite-plugin-wasm gotcha. |
|
||||
| 2026-04-13 | 1.3.0 | Updated GitNexus index stats after DAG refactor. |
|
||||
| 2026-03-24 | 1.2.0 | Fixed gitnexus:start block duplication. |
|
||||
| 2026-03-23 | 1.1.0 | Updated agent instructions, references, Cursor layout. |
|
||||
| 2026-03-22 | 1.0.0 | Initial structured header and changelog. |
|
||||
|
||||
---
|
||||
|
||||
<!-- gitnexus:start -->
|
||||
# GitNexus — Code Intelligence
|
||||
|
||||
Indexed as **GitNexus** (4325 symbols, 10556 relationships, 300 execution flows). Use MCP tools to understand code, assess impact, and navigate safely.
|
||||
This project is indexed by GitNexus as **GitNexus** (1650 symbols, 4291 relationships, 125 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
|
||||
|
||||
> If any tool warns the index is stale, run `npx gitnexus analyze` first.
|
||||
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
|
||||
|
||||
## Always Do
|
||||
|
||||
- **MUST run impact analysis before editing any symbol.** `gitnexus_impact({target: "symbolName", direction: "upstream"})` — report blast radius to the user.
|
||||
- **MUST run `gitnexus_detect_changes()` before committing** — verify only expected symbols and flows are affected.
|
||||
- **MUST warn the user** if impact returns HIGH or CRITICAL risk.
|
||||
- Explore unfamiliar code with `gitnexus_query({query: "concept"})` (process-grouped, ranked) instead of grepping.
|
||||
- Full context on a symbol: `gitnexus_context({name: "symbolName"})`.
|
||||
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
|
||||
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
|
||||
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
|
||||
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
|
||||
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
|
||||
|
||||
## When Debugging
|
||||
|
||||
1. `gitnexus_query({query: "<error or symptom>"})` — find related execution flows
|
||||
2. `gitnexus_context({name: "<suspect function>"})` — callers, callees, process participation
|
||||
3. `READ gitnexus://repo/GitNexus/process/{processName}` — trace flow step by step
|
||||
4. Regressions: `gitnexus_detect_changes({scope: "compare", base_ref: "main"})`
|
||||
1. `gitnexus_query({query: "<error or symptom>"})` — find execution flows related to the issue
|
||||
2. `gitnexus_context({name: "<suspect function>"})` — see all callers, callees, and process participation
|
||||
3. `READ gitnexus://repo/GitNexus/process/{processName}` — trace the full execution flow step by step
|
||||
4. For regressions: `gitnexus_detect_changes({scope: "compare", base_ref: "main"})` — see what your branch changed
|
||||
|
||||
## When Refactoring
|
||||
|
||||
- **Rename:** `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` first. Graph edits are safe; text_search edits need manual review.
|
||||
- **Extract/Split:** `gitnexus_context` (incoming/outgoing refs) then `gitnexus_impact` (upstream callers) before moving code.
|
||||
- **After any refactor:** `gitnexus_detect_changes({scope: "all"})` to verify scope.
|
||||
- **Renaming**: MUST use `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` first. Review the preview — graph edits are safe, text_search edits need manual review. Then run with `dry_run: false`.
|
||||
- **Extracting/Splitting**: MUST run `gitnexus_context({name: "target"})` to see all incoming/outgoing refs, then `gitnexus_impact({target: "target", direction: "upstream"})` to find all external callers before moving code.
|
||||
- After any refactor: run `gitnexus_detect_changes({scope: "all"})` to verify only expected files changed.
|
||||
|
||||
## Never Do
|
||||
|
||||
- Edit a symbol without running `gitnexus_impact` first.
|
||||
- Ignore HIGH/CRITICAL risk warnings.
|
||||
- Rename with find-and-replace — use `gitnexus_rename`.
|
||||
- Commit without `gitnexus_detect_changes()`.
|
||||
- Add language-specific behavior to shared ingestion code (`gitnexus/src/core/ingestion/`) — use a `LanguageProvider` hook. Seeing `provider.mroStrategy === 'xxx'` or an import from `languages/xxx.ts` in shared code means stop and add a hook.
|
||||
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
|
||||
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
|
||||
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
|
||||
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
|
||||
|
||||
## Tools Quick Reference
|
||||
|
||||
| Tool | When to use | Example |
|
||||
| Tool | When to use | Command |
|
||||
|------|-------------|---------|
|
||||
| `list_repos` | Discover indexed repos | `gitnexus_list_repos({})` |
|
||||
| `query` | Find code by concept | `gitnexus_query({query: "auth validation"})` |
|
||||
| `context` | 360-degree view of one symbol | `gitnexus_context({name: "validateUser"})` |
|
||||
| `impact` | Blast radius before editing | `gitnexus_impact({target: "X", direction: "upstream"})` |
|
||||
| `detect_changes` | Pre-commit scope check | `gitnexus_detect_changes({scope: "staged"})` |
|
||||
| `rename` | Safe multi-file rename | `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` |
|
||||
| `cypher` | Custom graph queries | `gitnexus_cypher({query: "MATCH ..."})` |
|
||||
| `api_impact` | Pre-change API route impact | `gitnexus_api_impact({route: "/api/users", method: "GET"})` |
|
||||
| `route_map` | Route → handler → consumer map | `gitnexus_route_map({})` |
|
||||
| `tool_map` | MCP/RPC tool definitions | `gitnexus_tool_map({})` |
|
||||
| `shape_check` | Response shape vs consumer access | `gitnexus_shape_check({route: "/api/users"})` |
|
||||
| `group_list` | List repo groups | `gitnexus_group_list({})` |
|
||||
| `group_sync` | Rebuild group Contract Registry | `gitnexus_group_sync({name: "myGroup"})` |
|
||||
| `query` (group mode) | Cross-repo search in a group (RRF-merged) | `gitnexus_query({repo: "@myGroup", query: "auth"})` |
|
||||
| `context` (group mode) | 360° view across all member repos | `gitnexus_context({repo: "@myGroup", name: "validateUser"})` |
|
||||
| `impact` (group mode) | Cross-repo blast radius via Contract Bridge | `gitnexus_impact({repo: "@myGroup", target: "X", direction: "upstream"})` |
|
||||
|
||||
> Group mode: pass `repo: "@<groupName>"` to fan out across all member repos, or `repo: "@<groupName>/<memberPath>"` to target a single member (path keys from `group.yaml`). Optional `service: "<monorepo/path>"` filters by service root. Group-level state (contracts, staleness) lives in the resources table below — there are **no** `group_query` / `group_context` / `group_impact` / `group_contracts` / `group_status` MCP tools.
|
||||
>
|
||||
> For a full walkthrough of setting up a group across multiple repos that communicate over gRPC, see [docs/guides/microservices-grpc.md](docs/guides/microservices-grpc.md).
|
||||
|
||||
## Impact Risk Levels
|
||||
|
||||
| Depth | Meaning | Action |
|
||||
|-------|---------|--------|
|
||||
| d=1 | WILL BREAK — direct callers/importers | MUST update |
|
||||
| d=1 | WILL BREAK — direct callers/importers | MUST update these |
|
||||
| d=2 | LIKELY AFFECTED — indirect deps | Should test |
|
||||
| d=3 | MAY NEED TESTING — transitive | Test if critical path |
|
||||
|
||||
@@ -132,83 +56,23 @@ Indexed as **GitNexus** (4325 symbols, 10556 relationships, 300 execution flows)
|
||||
|
||||
| Resource | Use for |
|
||||
|----------|---------|
|
||||
| `gitnexus://repo/GitNexus/context` | Codebase overview, index freshness |
|
||||
| `gitnexus://repo/GitNexus/context` | Codebase overview, check index freshness |
|
||||
| `gitnexus://repo/GitNexus/clusters` | All functional areas |
|
||||
| `gitnexus://repo/GitNexus/processes` | All execution flows |
|
||||
| `gitnexus://repo/GitNexus/process/{name}` | Step-by-step execution trace |
|
||||
| `gitnexus://group/{name}/contracts` | Group Contract Registry (provider/consumer rows + cross-links) |
|
||||
| `gitnexus://group/{name}/status` | Per-member index + Contract Registry staleness report |
|
||||
|
||||
## Self-Check Before Finishing
|
||||
|
||||
Before completing any code modification task, verify:
|
||||
1. `gitnexus_impact` was run for all modified symbols
|
||||
2. No HIGH/CRITICAL warnings were ignored
|
||||
3. `gitnexus_detect_changes()` confirms expected scope
|
||||
4. All d=1 dependents were updated
|
||||
2. No HIGH/CRITICAL risk warnings were ignored
|
||||
3. `gitnexus_detect_changes()` confirms changes match expected scope
|
||||
4. All d=1 (WILL BREAK) dependents were updated
|
||||
|
||||
## Keeping the Index Fresh
|
||||
## CLI
|
||||
|
||||
```bash
|
||||
npx gitnexus analyze # basic refresh; preserves any existing embeddings
|
||||
npx gitnexus analyze --embeddings # also generate embeddings for new/changed nodes
|
||||
npx gitnexus analyze --drop-embeddings # explicit opt-in to wipe existing embeddings
|
||||
```
|
||||
|
||||
Check `.gitnexus/meta.json` `stats.embeddings` (0 = none). A plain `analyze` no longer drops existing vectors — pass `--drop-embeddings` to wipe.
|
||||
|
||||
> Claude Code: PostToolUse hook detects a stale index after `git commit` and `git merge` and prompts the agent to run `analyze`. The hook does not invoke `analyze` itself.
|
||||
|
||||
## CLI Skills
|
||||
|
||||
| Task | Skill file |
|
||||
|------|-----------|
|
||||
| Architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
|
||||
| Blast radius / "What breaks?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
|
||||
| Debugging / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
|
||||
| Refactoring | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
|
||||
| Tools/resources/schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
|
||||
| CLI commands (index, status, clean, wiki) | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
|
||||
- Re-index: `npx gitnexus analyze`
|
||||
- Check freshness: `npx gitnexus status`
|
||||
- Generate docs: `npx gitnexus wiki`
|
||||
|
||||
<!-- gitnexus:end -->
|
||||
|
||||
## Repo reference
|
||||
|
||||
### Packages
|
||||
|
||||
| Package | Path | Purpose |
|
||||
|---------|------|---------|
|
||||
| **CLI/Core** | `gitnexus/` | TypeScript CLI, indexing pipeline, MCP server. Published to npm. |
|
||||
| **Web UI** | `gitnexus-web/` | React/Vite thin client. All queries via `gitnexus serve` HTTP API. |
|
||||
| **Shared** | `gitnexus-shared/` | Shared TypeScript types and constants. |
|
||||
| Claude Plugin | `gitnexus-claude-plugin/` | Static config for Claude marketplace. |
|
||||
| Cursor Integration | `gitnexus-cursor-integration/` | Static config for Cursor editor. |
|
||||
| Eval | `eval/` | Python evaluation harness (Docker + LLM API keys). |
|
||||
|
||||
### Running services
|
||||
|
||||
```bash
|
||||
cd gitnexus && npm run dev # CLI: tsx watch mode
|
||||
cd gitnexus-web && npm run dev # Web UI: Vite on port 5173
|
||||
npx gitnexus serve # HTTP API on port 4747 (from any indexed repo)
|
||||
```
|
||||
|
||||
### Testing
|
||||
|
||||
**CLI / Core (`gitnexus/`)**
|
||||
- `npm test` — full vitest suite (~2000 tests)
|
||||
- `npm run test:unit` — unit tests only
|
||||
- `npm run test:integration` — integration (~1850 tests). LadybugDB file-locking tests may fail in containers (known env issue).
|
||||
- `npx tsc --noEmit` — typecheck
|
||||
|
||||
**Web UI (`gitnexus-web/`)**
|
||||
- `npm test` — vitest (~200 tests)
|
||||
- `npm run test:e2e` — Playwright (7 spec files; requires `gitnexus serve` + `npm run dev`)
|
||||
- `npx tsc -b --noEmit` — typecheck
|
||||
|
||||
**Pre-commit hook** (`.husky/pre-commit`): formatting (prettier via lint-staged) + typecheck for staged packages. Tests do **not** run in pre-commit — CI only.
|
||||
|
||||
### Gotchas
|
||||
|
||||
- `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (patches tree-sitter-swift, builds tree-sitter-proto). Native bindings need `python3`, `make`, `g++`.
|
||||
- `tree-sitter-kotlin` and `tree-sitter-swift` are optional — install warnings expected.
|
||||
- ESLint configured via `eslint.config.mjs` (TS, React Hooks, unused-imports). No `npm run lint` script; use `npx eslint .`. Prettier runs via lint-staged. CI checks both in `ci-quality.yml`.
|
||||
|
||||
-502
@@ -1,502 +0,0 @@
|
||||
# Architecture — GitNexus
|
||||
|
||||
Monorepo: **CLI/MCP** (`gitnexus/`) + **browser UI** (`gitnexus-web/`).
|
||||
|
||||
## Repository layout
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `gitnexus/` | npm package `gitnexus`: CLI, MCP server (stdio), HTTP API, ingestion pipeline, LadybugDB graph, embeddings. |
|
||||
| `gitnexus-web/` | Vite + React thin client: graph explorer + AI chat. All queries via `gitnexus serve` HTTP API. |
|
||||
| `gitnexus-shared/` | Shared TypeScript types and constants (consumed by CLI and Web). |
|
||||
| `.claude/`, `gitnexus-claude-plugin/`, `gitnexus-cursor-integration/` | Agent skills and plugin metadata. |
|
||||
| `eval/` | Evaluation harnesses for benchmarking tool usage. |
|
||||
| `.github/` | CI workflows + composite actions (`setup-gitnexus/`, `setup-gitnexus-web/`). |
|
||||
|
||||
## End-to-end flow: index → graph → tools
|
||||
|
||||
1. **Ingestion** — `analyze.ts` → `runFullAnalysis` (`run-analyze.ts`) → `runPipelineFromRepo` (`pipeline.ts`). DAG of 12 phases builds a `KnowledgeGraph` in memory, then loads into LadybugDB under `.gitnexus/`. Repo registered in `~/.gitnexus/registry.json` for MCP discovery.
|
||||
|
||||
2. **Persistence** — `repo-manager.ts` (paths, registry, KuzuDB cleanup). `lbug-adapter.ts` (graph load, queries, embedding batches).
|
||||
|
||||
3. **Query layer** — three interfaces to the same backend:
|
||||
- **MCP (stdio):** `mcp.ts` → `LocalBackend` → tools (`tools.ts`) + resources (`resources.ts`)
|
||||
- **HTTP bridge:** `serve.ts` → Express (`api.ts`, `mcp-http.ts`) for web UI
|
||||
- **CLI direct:** `gitnexus query|context|impact|cypher` in `tool.ts`
|
||||
|
||||
4. **Staleness** — `staleness.ts` compares indexed `lastCommit` to `HEAD`, surfaces hints.
|
||||
|
||||
## MCP tools
|
||||
|
||||
| Tool | Purpose |
|
||||
|------|---------|
|
||||
| `list_repos` | Discover indexed repos |
|
||||
| `query` | Hybrid BM25 + vector search over the graph |
|
||||
| `cypher` | Ad hoc Cypher against the schema |
|
||||
| `context` | Callers, callees, processes for one symbol |
|
||||
| `impact` | Blast radius (upstream/downstream) with risk summary |
|
||||
| `detect_changes` | Map git diffs to affected symbols and processes |
|
||||
| `rename` | Graph-assisted multi-file rename with `dry_run` preview |
|
||||
| `api_impact` | Pre-change impact report for an API route handler |
|
||||
| `route_map` | API route → handler → consumer mappings |
|
||||
| `tool_map` | MCP/RPC tool definitions and handlers |
|
||||
| `shape_check` | Response shape vs consumer property access mismatches |
|
||||
| `group_list` | List repo groups or details for one group |
|
||||
| `group_sync` | Rebuild group Contract Registry (`contracts.json`) and bridge graph |
|
||||
|
||||
`query`, `context`, and `impact` are group-aware: pass `repo: "@<groupName>"` (or `"@<groupName>/<memberPath>"` to scope to one member) plus optional `service: "<monorepo/path>"`. Group-mode `query` merges per-repo results via Reciprocal Rank Fusion; group-mode `impact` runs the local walk in the chosen member and fans out across boundaries via the Contract Bridge (`gitnexus/src/core/group/cross-impact.ts`). The previously-planned `group_query`, `group_context`, `group_impact`, `group_contracts`, `group_status` MCP tools are intentionally not introduced — group-level state is exposed via resources instead:
|
||||
|
||||
| Resource URI | Purpose |
|
||||
|--------------|---------|
|
||||
| `gitnexus://group/{name}/contracts` | Contract Registry (provider/consumer rows + cross-links) |
|
||||
| `gitnexus://group/{name}/status` | Per-member index + Contract Registry staleness |
|
||||
|
||||
## Where to change what
|
||||
|
||||
| Concern | Start in |
|
||||
|---------|----------|
|
||||
| CLI commands/flags | `src/cli/` (`index.ts`, per-command modules) |
|
||||
| Parsing/graph construction | `src/core/ingestion/pipeline-phases/` + `pipeline.ts` |
|
||||
| Graph schema/DB | `src/core/lbug/` (`schema.ts`, `lbug-adapter.ts`) |
|
||||
| MCP tools/resources | `src/mcp/server.ts`, `tools.ts`, `resources.ts` |
|
||||
| Cross-repo groups (sync, contracts, `@<group>` routing) | `src/core/group/` (`service.ts`, `cross-impact.ts`, `sync.ts`, `bridge-db.ts`) |
|
||||
| Search ranking | `src/core/search/` (BM25, hybrid fusion) |
|
||||
| Embeddings | `src/core/embeddings/` + `src/core/run-analyze.ts` |
|
||||
| Wiki generation | `src/core/wiki/` |
|
||||
| Language support | `src/core/ingestion/languages/` + `tree-sitter-queries.ts` + `gitnexus-shared/src/languages.ts` |
|
||||
| Import resolution | `src/core/ingestion/import-processor.ts` + `import-resolvers/configs/` + `model/resolution-context.ts` |
|
||||
| Call resolution/MRO | `src/core/ingestion/call-processor.ts` + `model/resolve.ts` |
|
||||
| Type extraction | `src/core/ingestion/type-extractors/` |
|
||||
| Worker pool | `src/core/ingestion/workers/` |
|
||||
| Web UI | `gitnexus-web/src/` |
|
||||
| CI | `.github/workflows/*.yml`, `.github/actions/` |
|
||||
|
||||
> Paths above are relative to `gitnexus/` unless they start with `gitnexus-web/` or `.github/`.
|
||||
|
||||
---
|
||||
|
||||
## Pipeline Phase DAG
|
||||
|
||||
12 phases defined in `gitnexus/src/core/ingestion/pipeline-phases/`, each with explicit `deps` and typed output.
|
||||
|
||||
```
|
||||
scan → structure → [markdown, cobol] → parse → [routes, tools, orm]
|
||||
→ crossFile → mro → communities → processes
|
||||
```
|
||||
|
||||
| Phase | File | Deps | Output |
|
||||
|-------|------|------|--------|
|
||||
| `scan` | `scan.ts` | (root) | File paths + sizes |
|
||||
| `structure` | `structure.ts` | `scan` | File/Folder nodes, CONTAINS edges, `allPathSet` |
|
||||
| `markdown` | `markdown.ts` | `structure` | Section nodes, cross-link edges from .md/.mdx |
|
||||
| `cobol` | `cobol.ts` | `structure` | COBOL program/paragraph/section nodes (regex, no tree-sitter) |
|
||||
| `parse` | `parse.ts` + `parse-impl.ts` | `structure`, `markdown`, `cobol` | Symbol nodes, IMPORTS/CALLS/EXTENDS edges, extracted routes/tools/ORM queries |
|
||||
| `routes` | `routes.ts` | `parse` | Route nodes + HANDLES_ROUTE edges (Next.js, Expo, PHP, decorators) |
|
||||
| `tools` | `tools.ts` | `parse` | Tool nodes + HANDLES_TOOL edges |
|
||||
| `orm` | `orm.ts` | `parse` | QUERIES edges (Prisma, Supabase) |
|
||||
| `crossFile` | `cross-file.ts` + `cross-file-impl.ts` | `parse`, `routes`, `tools`, `orm` | Cross-file type propagation in topological import order |
|
||||
| `mro` | `mro.ts` | `crossFile`, `structure` | METHOD_OVERRIDES + METHOD_IMPLEMENTS edges |
|
||||
| `communities` | `communities.ts` | `mro`, `structure` | Community nodes + MEMBER_OF edges (Leiden algorithm) |
|
||||
| `processes` | `processes.ts` | `communities`, `routes`, `tools`, `structure` | Process nodes + STEP_IN_PROCESS edges |
|
||||
|
||||
**Non-phase files in the same directory:** `parse-impl.ts`, `cross-file-impl.ts` (implementation), `wildcard-synthesis.ts` (whole-module import expansion), `orm-extraction.ts` (sequential ORM fallback), `types.ts`, `runner.ts`, `index.ts`.
|
||||
|
||||
### DAG runner
|
||||
|
||||
`runner.ts` — static phase graph, no plugins, compile-time type safety.
|
||||
|
||||
1. **Validation** — Kahn's topological sort. Rejects on: duplicate names, missing deps, cycles (DFS traces the concrete cycle path, e.g., `A -> B -> C -> A`, plus count of transitively blocked dependents).
|
||||
|
||||
2. **Execution** — sequential in topological order. Each phase receives:
|
||||
- `ctx: PipelineContext` — shared mutable `KnowledgeGraph`, `repoPath`, progress callback, options
|
||||
- `deps: ReadonlyMap<string, PhaseResult>` — **declared deps only** (runner filters the results map to prevent hidden coupling)
|
||||
|
||||
3. **Error handling** — wraps phase errors with the phase name, emits terminal `error` progress event, swallows progress handler errors to preserve the original cause.
|
||||
|
||||
4. **Timing** — per-phase `durationMs` in `PhaseResult`, dev-mode console logging.
|
||||
|
||||
**Design patterns:**
|
||||
- **Single graph accumulator** — all phases mutate the same `KnowledgeGraph` in `ctx`; the graph is the primary output.
|
||||
- **Typed phase access** — `getPhaseOutput<T>(deps, 'name')` for type-safe upstream results.
|
||||
- **Binding accumulator lifecycle** — created in `parse`, disposed by `crossFile` (in `finally`). No other phase should take ownership.
|
||||
- **Skippable phases** — `skipGraphPhases` omits MRO/communities/processes (faster tests). `skipWorkers` forces sequential parsing.
|
||||
|
||||
### How to add a new phase
|
||||
|
||||
1. Create `pipeline-phases/my-phase.ts` with a `PipelinePhase<MyOutput>` (name, deps, execute)
|
||||
2. Export from `pipeline-phases/index.ts`
|
||||
3. Add to `buildPhaseList()` in `pipeline.ts`
|
||||
|
||||
```typescript
|
||||
import type { PipelinePhase, PhaseResult } from './types.js';
|
||||
import { getPhaseOutput } from './types.js';
|
||||
import type { ParseOutput } from './parse.js';
|
||||
|
||||
export interface MyPhaseOutput { /* ... */ }
|
||||
|
||||
export const myPhase: PipelinePhase<MyPhaseOutput> = {
|
||||
name: 'myPhase',
|
||||
deps: ['parse'],
|
||||
async execute(ctx, deps) {
|
||||
const { allPaths } = getPhaseOutput<ParseOutput>(deps, 'parse');
|
||||
// ... write to ctx.graph ...
|
||||
return { /* typed output */ };
|
||||
},
|
||||
};
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Call-Resolution DAG
|
||||
|
||||
Typed 6-stage pipeline in `call-processor.ts` (inside the `parse` phase) that resolves method/function calls and emits CALLS edges. Language behavior plugs in at two `LanguageProvider` hook points (stages 3–4); shared code names no languages. Scope: call resolution only — import resolution, type extraction, heritage, and symbol-table population live in other phases.
|
||||
|
||||
### Stages
|
||||
|
||||
```
|
||||
extract-call ──▶ classify-form ──▶ infer-receiver ──▶ select-dispatch ──▶ resolve-target ──▶ emit-edge
|
||||
(1) (2) (3) [hook] (4) [hook] (5) (6)
|
||||
```
|
||||
|
||||
| Stage | Produces | Location |
|
||||
|-------|----------|----------|
|
||||
| **extract-call** | `ExtractedCallSite` (name, form, receiver, argCount) | `call-extractors/` (per-language); runs in worker |
|
||||
| **classify-form** | callForm (`free`/`member`/`constructor`) + arity | `call-analysis.ts` → `inferCallForm`; shared, runs in worker |
|
||||
| **infer-receiver** | `ReceiverEnriched` (receiver type finalized) | `call-processor.ts`; shared default chain, then `inferImplicitReceiver` hook |
|
||||
| **select-dispatch** | `DispatchDecision` (primary, fallback, ancestryView) | `selectDispatch` hook, falls back to shared default |
|
||||
| **resolve-target** | `TieredCandidates` | `model/resolve.ts` → `lookupMethodByOwnerWithMRO` (MRO walk) |
|
||||
| **emit-edge** | CALLS edge in graph | `call-processor.ts`; writes edge with confidence tier |
|
||||
|
||||
### Provider hooks
|
||||
|
||||
Both hooks are optional on `LanguageProvider`. Ruby is the only current implementer.
|
||||
|
||||
**`inferImplicitReceiver`** — called after shared infer-receiver defaults. Returns `ImplicitReceiverOverride | null`.
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Inputs | `calledName`, `callForm`, `receiverName`, `receiverTypeName`, `callNode` (AST), `filePath` |
|
||||
| Non-null fields | `callForm`, `receiverName`, `receiverTypeName` (required); `receiverSource: 'implicit-self'` (fixed); `hint?` (opaque, passed to `selectDispatch`) |
|
||||
| Null | Keep existing `ReceiverEnriched` state |
|
||||
|
||||
**`selectDispatch`** — called after infer-receiver (including hook). Returns `DispatchDecision | null`; null uses shared default (constructor → `primary:'constructor'`; typed receiver → `primary:'owner-scoped'`; else → `primary:'free'`).
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Inputs | `calledName`, `callForm`, `receiverName`, `receiverTypeName`, `receiverSource`, `hint` |
|
||||
| Non-null fields | `primary: 'owner-scoped' \| 'free' \| 'constructor'`; `fallback?: 'free-arity-narrowed'`; `ancestryView?: 'instance' \| 'singleton'`; `hint?` |
|
||||
|
||||
**`DispatchDecision` field semantics:**
|
||||
- `primary: 'owner-scoped'` — MRO walk from receiver's type; used when receiver type is known.
|
||||
- `fallback: 'free-arity-narrowed'` — after owner-scoped miss, search free-call candidates by arity only (Ruby uses this for implicit-self calls that miss their owner's MRO).
|
||||
- `ancestryView: 'singleton'` — walk singleton/class ancestry instead of instance ancestry (Ruby `def self.foo` bodies, so `extend`-ed methods are found).
|
||||
|
||||
### Adding language behavior
|
||||
|
||||
1. **Implicit receivers** — implement `inferImplicitReceiver`: return null if call already has a receiver; otherwise use `findEnclosingClassInfo` (`ast-helpers.ts`) to find the enclosing context, return `ImplicitReceiverOverride` with `receiverSource: 'implicit-self'`, and optionally set `hint` for `selectDispatch`.
|
||||
2. **Custom dispatch** — implement `selectDispatch`: inspect `receiverSource` and `hint`, return `DispatchDecision` with `primary`, optional `fallback`, optional `ancestryView`; return null to keep shared defaults.
|
||||
3. **MRO strategy** — confirm `mroStrategy` is `'first-wins'`, `'c3'`, `'ruby-mixin'`, or `'none'`; consumed by `lookupMethodByOwnerWithMRO`.
|
||||
|
||||
**Ruby example** (`languages/ruby.ts` + `utils/ruby-self-call.ts`): `inferImplicitReceiver` rewrites bare-identifier calls to `self.method` and sets `hint` to `'instance'`/`'singleton'`; `selectDispatch` uses hint for `ancestryView` and adds `fallback: 'free-arity-narrowed'` for implicit-self calls.
|
||||
|
||||
### Code references
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `core/ingestion/call-types.ts` | DAG types: `ReceiverEnriched`, `DispatchDecision`, `ImplicitReceiverOverride` |
|
||||
| `core/ingestion/language-provider.ts` | Hook signatures: `inferImplicitReceiver`, `selectDispatch` |
|
||||
| `core/ingestion/call-processor.ts` | `processCalls`: stages 3–6 |
|
||||
| `core/ingestion/model/resolve.ts` | `lookupMethodByOwnerWithMRO`: stage 5 MRO walk |
|
||||
| `core/ingestion/languages/ruby.ts` | Both hooks + `mroStrategy: 'ruby-mixin'` |
|
||||
| `core/ingestion/utils/ruby-self-call.ts` | Bare-call rewrite for `inferImplicitReceiver` |
|
||||
|
||||
### Coexistence with the scope-resolution pipeline
|
||||
|
||||
The Call-Resolution DAG is the **legacy path**. RFC #909 Ring 3 introduces a parallel **scope-resolution pipeline** (next section) that replaces stages 1–6 with a scope-indexed registry lookup. Both paths ship side-by-side and are gated per-language via `MIGRATED_LANGUAGES` + the `REGISTRY_PRIMARY_<LANG>` env var.
|
||||
|
||||
- **Unmigrated language** → Call-Resolution DAG runs; scope-resolution phase is a no-op.
|
||||
- **Migrated language** (currently: Python, C#) → scope-resolution owns CALLS/ACCESSES/USES emission; the legacy DAG gates off for that language via `isRegistryPrimary(lang)` checks in `call-processor.ts` and `import-processor.ts`.
|
||||
- `import-processor` still populates `importMap` for migrated languages — heritage's `ctx.resolve` reads it to disambiguate parent classes. Only edge emission is gated.
|
||||
- CI runs BOTH paths for every migrated language on every PR (`.github/workflows/ci-scope-parity.yml`); both must pass.
|
||||
|
||||
#### Same-graph guarantee
|
||||
|
||||
Edges emitted by the scope-resolution pipeline and edges emitted by the legacy DAG are indistinguishable to downstream consumers (MCP tools, HTTP API, embeddings, group bridge):
|
||||
|
||||
- **Node identity** — both paths use `generateId(...)` from `lib/utils.ts`, the same qualified-name keyspace, and the same node labels (`File`, `Folder`, `Class`, `Method`, `Function`, …). Overload disambiguation suffixes `parameterTypes` into the id consistently — see `scope-resolution/graph-bridge/ids.ts` and the legacy emitter in `call-processor.ts`.
|
||||
- **Edge vocabulary** — both paths emit the same reasons: `'import-resolved' | 'global' | 'local-call' | 'same-file' | 'interface-dispatch' | 'read' | 'write'`. Migrating a language must not change which reasons consumers see for previously-resolved edges.
|
||||
- **Confidence tier** — both paths attach a numeric `confidence` to each edge using the same scale.
|
||||
|
||||
The CI parity workflow (`.github/workflows/ci-scope-parity.yml`) runs both paths against every migrated language's fixture corpus and fails on any divergence.
|
||||
|
||||
#### Semantic-model source of truth
|
||||
|
||||
Two independent invariants.
|
||||
|
||||
**ParsedFile = the AST-level truth.** `ParsedFile` (`gitnexus-shared/src/scope-resolution/parsed-file.ts`) is the single per-file artifact both resolution paths consume. Scope-resolution passes MUST NOT build a parallel parse representation. If a per-language hook needs AST-level facts that `ParsedFile` doesn't expose, it should reuse the orchestrator's `treeCache` (`RunScopeResolutionInput.treeCache`) rather than re-invoking `parser.parse(...)` on its own — the C# `populateNamespaceSiblings` hook is the reference implementation of this pattern.
|
||||
|
||||
**SemanticModel = the symbol-level truth.** `SemanticModel` (`gitnexus/src/core/ingestion/model/semantic-model.ts`) is the authoritative store for every symbol-indexed lookup (by `nodeId`, `simpleName`, `qualifiedName`, or `filePath`). Both paths read from here:
|
||||
|
||||
- Legacy Call-Resolution DAG → `call-processor` Tier 1/2/3 via `model.symbols.lookupExactAll`, `model.methods.lookupMethodByName`, `model.types.lookupClassByName`, `lookupMethodByOwnerWithMRO`.
|
||||
- Scope-resolution pipeline → `findOwnedMember`, `pickOverload`, `findExportedDefByName` all consult `model.methods` / `model.fields` / `model.symbols`.
|
||||
|
||||
The scope-resolution pipeline additionally carries `WorkspaceResolutionIndex` for `Scope`-valued lookups (`classScopeByDefId`, `moduleScopeByFile`) that `SemanticModel` structurally cannot hold. No symbol-indexed duplicates exist outside `SemanticModel`.
|
||||
|
||||
**Write / read phase contract.** The model is mutable during three ordered phases and read-only afterward:
|
||||
|
||||
```
|
||||
Phase 1: legacy parse ──► symbolTable.add fans into types/methods/fields
|
||||
Phase 2: scope-resolution ──► reconcileOwnership() registers corrected ownerIds
|
||||
Phase 3: finalize ──► model.attachScopeIndexes(bundle) — one-shot freeze
|
||||
─────────────────────────── phase boundary ───────────────────────────
|
||||
Read phase: all resolution passes + MCP + HTTP + embeddings see
|
||||
SemanticModel (read-only handle); writes are type-errors.
|
||||
```
|
||||
|
||||
`runScopeResolution` narrows `MutableSemanticModel` → `SemanticModel` at the phase boundary so downstream passes physically cannot mutate the model even accidentally.
|
||||
|
||||
**Transitional: reconciliation pass.** `reconcileOwnership` (`scope-resolution/pipeline/reconcile-ownership.ts`) is a shim for languages whose legacy extractor doesn't resolve `enclosingClassId` at parse time (Python class-body methods are the canonical case). It walks `parsed.localDefs[i].ownerId` after `populateOwners` and registers any missed methods/fields into the model. Idempotent — safe to re-run, safe alongside languages whose legacy extractor already carries `ownerId` (C#).
|
||||
|
||||
The architectural end state is for every language's parse-time extractor to emit the correct `ownerId` directly, making reconciliation a no-op (tracked as a follow-up refactor). The dev-mode validator `validateOwnershipParity` surfaces any drift via `onWarn` under `NODE_ENV !== 'production' && VALIDATE_SEMANTIC_MODEL !== '0'`.
|
||||
|
||||
References: `semantic-model.ts` file-head (full write/read contract); `contract/scope-resolver.ts` Contract Invariant I9 (scope-resolution-side rule).
|
||||
|
||||
---
|
||||
|
||||
## Scope-Resolution Pipeline (RFC #909 Ring 3)
|
||||
|
||||
Language-agnostic registry-primary resolver. Replaces the Call-Resolution DAG for migrated languages. Adding a language is one interface implementation (`ScopeResolver`) plus two registrations — no changes to shared code, no new pipeline phase.
|
||||
|
||||
### Pipeline stages
|
||||
|
||||
```
|
||||
ParsedFile[] (extractParsedFile per file)
|
||||
│ finalizeScopeModel (+ provider hooks)
|
||||
▼
|
||||
ScopeResolutionIndexes
|
||||
│ resolveReferenceSites (via MethodRegistry.lookup)
|
||||
▼
|
||||
ReferenceIndex
|
||||
│ emitReceiverBoundCalls ── FIRST
|
||||
│ emitFreeCallFallback ── THEN
|
||||
│ emitReferencesViaLookup ── LAST (uses handledSites)
|
||||
│ emitImportEdges
|
||||
▼
|
||||
KnowledgeGraph (IMPORTS / CALLS / ACCESSES / INHERITS / USES)
|
||||
```
|
||||
|
||||
Orchestrator: `runScopeResolution(input, provider)` in `scope-resolution/pipeline/run.ts`.
|
||||
Pipeline phase: `scopeResolutionPhase` in `scope-resolution/pipeline/phase.ts` — iterates `SCOPE_RESOLVERS ∩ MIGRATED_LANGUAGES`, reads per-file Trees from the parse phase's `scopeTreeCache`, disposes the cache at the end.
|
||||
|
||||
### `ScopeResolver` contract
|
||||
|
||||
Single interface a language implements to plug into the pipeline. Contract fully documented in `scope-resolution/contract/scope-resolver.ts`.
|
||||
|
||||
| Hook | Purpose |
|
||||
|------|---------|
|
||||
| `languageProvider` | Base `LanguageProvider` (tree-sitter query, `emitScopeCaptures`, import/binding interpreters, hooks) |
|
||||
| `populateOwners(parsed)` | Fill deferred `ownerId` fields on method defs (captures can't always know the owning class at parse time) |
|
||||
| `buildMro(graph, parsed, nodeLookup)` | Produce `mroByClassDefId: Map<DefId, DefId[]>` — C3, Ruby-mixin, or first-wins per language |
|
||||
| `resolveImportTarget(target, fromFile, allFiles)` | `(rawImportPath, sourceFile) → targetFilePath` (PEP-328 for Python, etc.) |
|
||||
| `mergeBindings(existing, incoming, scopeId)` | Shadowing / LEGB precedence |
|
||||
| `arityCompatibility` | Provider consumed by registry during `MethodRegistry.lookup` Step 2 |
|
||||
| `importEdgeReason` | Confidence-tier string for IMPORTS edge reason field |
|
||||
| `propagatesReturnTypesAcrossImports?` | Opt out of cross-file return-type propagation (default on) |
|
||||
| `fieldFallbackOnMethodLookup?` | Statically-typed languages turn this OFF — the heuristic over-connects (default on) |
|
||||
| `unwrapCollectionAccessor?` | Property-style collection views (`data.Values` on Dictionary-like receivers) — default off |
|
||||
| `collapseMemberCallsByCallerTarget?` | One CALLS edge per (caller, target) instead of per-site — default off |
|
||||
| `populateNamespaceSiblings?` | Cross-file implicit visibility (compiler-implicit namespace sharing) — default off; ctx carries `treeCache` |
|
||||
| `hoistTypeBindingsToModule?` | Walk up to Module scope when looking up a method's return-type typeBinding — default off; enable only when bindings are stored at module level |
|
||||
|
||||
### Per-language registration
|
||||
|
||||
1. Implement `ScopeResolver` in `languages/<lang>/scope-resolver.ts`.
|
||||
2. Add entry to `SCOPE_RESOLVERS` in `scope-resolution/pipeline/registry.ts`.
|
||||
3. Add the language to `MIGRATED_LANGUAGES` in `registry-primary-flag.ts` when the shadow-harness corpus parity ≥ 99% fixtures / ≥ 98% corpus.
|
||||
|
||||
CI auto-discovers the set via `tsx`. No workflow edit required.
|
||||
|
||||
### Code references
|
||||
|
||||
| Module | Purpose |
|
||||
|--------|---------|
|
||||
| `scope-resolution/contract/scope-resolver.ts` | `ScopeResolver` interface + shared types |
|
||||
| `scope-resolution/pipeline/run.ts` | Generic orchestrator |
|
||||
| `scope-resolution/pipeline/phase.ts` | Pipeline-phase wrapper (deps: `parse`, `structure`) |
|
||||
| `scope-resolution/pipeline/registry.ts` | `SCOPE_RESOLVERS` map |
|
||||
| `scope-resolution/passes/*.ts` | Reference-resolution passes (receiver-bound, free-call fallback, compound-receiver, MRO, cross-file return-type propagation) |
|
||||
| `scope-resolution/graph-bridge/*.ts` | CLI-local translation from resolved references → `KnowledgeGraph` edges |
|
||||
| `scope-resolution/scope/*.ts` | Generic scope-chain walkers + namespace targets |
|
||||
| `scope-resolution/workspace-index.ts` | Build-once O(1) lookup index |
|
||||
| `registry-primary-flag.ts` | `MIGRATED_LANGUAGES` set + `isRegistryPrimary(lang)` |
|
||||
| `languages/python/index.ts` | Python `ScopeResolver` hooks + known-limitation docs |
|
||||
| `languages/python/captures.ts` | `emitPythonScopeCaptures` (honors cross-phase Tree cache) |
|
||||
| `languages/csharp/index.ts` | C# `ScopeResolver` hooks + known-limitation docs |
|
||||
| `languages/csharp/captures.ts` | `emitCsharpScopeCaptures` (honors cross-phase Tree cache) |
|
||||
| `languages/csharp/namespace-siblings.ts` | Cross-file implicit-namespace visibility hook (reads `treeCache`) |
|
||||
|
||||
### Performance notes
|
||||
|
||||
- **Cross-phase Tree cache**: parse phase writes Trees into `scopeTreeCache` (separate from the chunk-local `astCache`) ONLY for languages with `emitScopeCaptures`. Scope-resolution reads from it to skip the second parse. Cleared at end of the phase. Workers leave the cache empty — Trees can't cross MessageChannels; cache miss = fresh parse. `PROF_SCOPE_RESOLUTION=1` emits hit/miss counters and a worker-engaged warning.
|
||||
- **Typed relationship iteration**: heritage + MRO walk only the EXTENDS / IMPLEMENTS / HAS_METHOD edges via `iterRelationshipsByType`, not the full relationship map.
|
||||
- **Workspace-resolution-index**: O(1) `findOwnedMember` / `findExportedDef` / `classScopeByDefId` built once per run.
|
||||
- **SCC-ordered cross-file return-type propagation** (PR #1050): `propagateImportedReturnTypes` walks `indexes.sccs` in reverse-topological order (leaves first), so multi-hop alias chains like `models.User → service.user → app.user` collapse to the terminal class in a single linear pass. Within each importer, the source module's `typeBindings` is chain-followed BEFORE mirroring (so we mirror terminal types, not intermediate refs), and the importer's own `typeBindings` is chain-followed AFTER mirroring (so local `const x = importedFn()` resolves before downstream importers run). Cyclic SCCs reach a partial fixpoint within a single pass without iterating to convergence — see the `ts-circular` cross-file-binding fixture which only asserts pipeline-no-throw. PROF output (`PROF_SCOPE_RESOLUTION=1`) splits `finalize` from `propagate` so quadratic regressions in the chain-follow surface independently.
|
||||
|
||||
---
|
||||
|
||||
## Language-agnostic graph feeding
|
||||
|
||||
16 languages → single unified graph. Four abstraction layers:
|
||||
|
||||
```
|
||||
Unified Graph Schema (44 node types, 21 relationship types)
|
||||
↑
|
||||
Unified Resolution (3-tier name lookup + MRO walk)
|
||||
↑
|
||||
Language Providers (import semantics, type config, export checker, MRO strategy)
|
||||
↑
|
||||
Tree-Sitter Queries (per-language S-expressions, unified capture tags)
|
||||
```
|
||||
|
||||
### Language providers
|
||||
|
||||
Each language implements `LanguageProvider` (`language-provider.ts`). Key fields:
|
||||
|
||||
| Field | Purpose |
|
||||
|-------|---------|
|
||||
| `id`, `extensions` | Language identity and file matching |
|
||||
| `treeSitterQueries` | S-expression queries for AST extraction |
|
||||
| `importSemantics` | `named` / `wildcard-leaf` / `wildcard-transitive` / `namespace` |
|
||||
| `importResolver` | Language-specific path → file resolution |
|
||||
| `exportChecker` | Public/exported symbol detection |
|
||||
| `typeConfig` | Type annotation extraction rules |
|
||||
| `mroStrategy` | `first-wins` / `c3` / `none` |
|
||||
|
||||
16 providers in `languages/index.ts` via `satisfies Record<SupportedLanguages, LanguageProvider>` — missing a language is a compile error.
|
||||
|
||||
### Unified capture tags
|
||||
|
||||
Per-language tree-sitter queries use different AST node names but produce the **same semantic capture tags**: `@definition.class`, `@definition.function`, `@call.name`, `@import.source`, `@heritage.extends`. Downstream extraction needs no language branching. Defined in `tree-sitter-queries.ts`.
|
||||
|
||||
### Import resolution
|
||||
|
||||
Per-language import resolution uses the **configs + factory** pattern (like call/method/class extractors). Each language declares an `ImportResolutionConfig` in `import-resolvers/configs/`, listing an ordered chain of `ImportResolverStrategy` functions. `createImportResolver()` (in `resolver-factory.ts`) composes them: first non-null result wins. Low-level helpers shared across strategies live alongside the configs in `import-resolvers/` (e.g. `go.ts`, `rust.ts`, `python.ts`).
|
||||
|
||||
Unified 3-tier algorithm (`model/resolution-context.ts`), per-language `importSemantics` controls which tier activates:
|
||||
|
||||
| Tier | Confidence | Mechanism |
|
||||
|------|-----------|-----------|
|
||||
| 1 — same-file | 0.95 | Symbol table for caller's file |
|
||||
| 2 — import-scoped | 0.9 | `NamedImportMap` chains (named) or all files in `importMap` (wildcard) |
|
||||
| 3 — global | 0.5 | O(1) index lookups: class, impl, callable. Fallback only |
|
||||
|
||||
| Import strategy | Languages | Behavior |
|
||||
|----------------|-----------|----------|
|
||||
| `named` | TS, JS, Java, C#, Rust, PHP, Kotlin | Only explicitly imported names visible |
|
||||
| `wildcard-leaf` | Go, Ruby, Swift, Dart | Whole-package import, no transitive re-exports |
|
||||
| `wildcard-transitive` | C, C++ | `#include` closure chains through re-exports |
|
||||
| `namespace` | Python | Module aliases resolved at call site |
|
||||
|
||||
### Chunked parse-and-resolve
|
||||
|
||||
`parse` processes files in ~20 MB byte-budget chunks to bound memory. Per chunk:
|
||||
1. Worker pool dispatches files (or sequential fallback via `skipWorkers`)
|
||||
2. Each worker: detect language → load grammar → run queries → return unified `ParseWorkerResult`
|
||||
3. Synthesize wildcard bindings (`wildcard-synthesis.ts`)
|
||||
4. Resolve imports and heritage
|
||||
5. Collect `BindingAccumulator` entries for cross-file propagation
|
||||
|
||||
Workers: `workers/worker-pool.ts`, `workers/parse-worker.ts`.
|
||||
|
||||
### Heritage and MRO
|
||||
|
||||
All languages emit unified `ExtractedHeritage` (child, parent, `EXTENDS`/`IMPLEMENTS`). MRO phase walks the heritage graph using per-language strategy:
|
||||
- **`first-wins`** — Java, C#, C++, TS, Ruby, Go
|
||||
- **`c3`** — Python (C3 linearization)
|
||||
- **`none`** — single-inheritance languages
|
||||
|
||||
Unified walk: `lookupMethodByOwnerWithMRO()` in `model/resolve.ts`.
|
||||
|
||||
---
|
||||
|
||||
## Full analysis flow
|
||||
|
||||
`runFullAnalysis` in `run-analyze.ts` orchestrates everything around the pipeline:
|
||||
|
||||
```
|
||||
CLI (analyze.ts) → runFullAnalysis(repoPath, options, callbacks)
|
||||
1. Early exit if lastCommit == HEAD (unless --force) [0%]
|
||||
2. Cache existing embeddings from prior index [0%]
|
||||
3. runPipelineFromRepo() → KnowledgeGraph [0-60%]
|
||||
4. Clean up legacy KuzuDB files [60%]
|
||||
5. initLbug() → loadGraphToLbug() via CSV streaming [60-85%]
|
||||
6. Create FTS indexes (File, Function, Class, Method...) [85-90%]
|
||||
7. Restore cached embeddings (batch insert) [88%]
|
||||
8. Generate new embeddings if --embeddings [90-98%]
|
||||
9. Save metadata + register repo + update .gitignore [98-100%]
|
||||
10. Generate AI context files (AGENTS.md, CLAUDE.md) [100%]
|
||||
```
|
||||
|
||||
**Options:** `--force` (rebuild regardless), `--embeddings` (opt-in, skipped if >50k nodes), `--skipGit`, `--noStats`.
|
||||
|
||||
## Storage
|
||||
|
||||
```
|
||||
<repo>/.gitnexus/
|
||||
├── lbug # LadybugDB database
|
||||
├── lbug.wal # Write-ahead log
|
||||
├── lbug.lock # Single-writer lock
|
||||
└── meta.json # lastCommit, indexedAt, stats
|
||||
|
||||
~/.gitnexus/
|
||||
└── registry.json # Global repo registry (MCP discovery)
|
||||
```
|
||||
|
||||
Managed by `repo-manager.ts`.
|
||||
|
||||
## LadybugDB schema
|
||||
|
||||
Defined in `lbug/schema.ts`. Separate node tables per type, single `CodeRelation` table.
|
||||
|
||||
**Node tables:** File, Folder, Function, Class, Interface, Method, Constructor, CodeElement, Struct, Enum, Macro, Typedef, Union, Namespace, Trait, Impl, TypeAlias, Const, Static, Property, Record, Delegate, Annotation, Template, Module, Community, Process, Route, Tool, Section, Embedding.
|
||||
|
||||
**Relation types** (`CodeRelation.type`): CONTAINS, DEFINES, CALLS, IMPORTS, EXTENDS, IMPLEMENTS, HAS_METHOD, HAS_PROPERTY, ACCESSES, METHOD_OVERRIDES, METHOD_IMPLEMENTS, MEMBER_OF, STEP_IN_PROCESS, HANDLES_ROUTE, FETCHES, HANDLES_TOOL, ENTRY_POINT_OF.
|
||||
|
||||
## Embeddings and search
|
||||
|
||||
**Embeddings** (`src/core/embeddings/`): Snowflake arctic-embed-xs (384D). Embeddable: File, Function, Class, Method, Interface. Incremental via SHA1 content hash. Separate `Embedding` table.
|
||||
|
||||
**Search** (`src/core/search/`): Hybrid BM25 + semantic vector, merged via Reciprocal Rank Fusion (K=60).
|
||||
|
||||
## Known limitations
|
||||
|
||||
### Overloaded method resolution
|
||||
|
||||
Node IDs use arity suffix (`#<paramCount>`): `Method:file:Class.method#1` vs `#2`.
|
||||
|
||||
**Same-arity disambiguation:** type-hash suffix `~type1,type2` when collision detected and type annotations present. Languages without types (Python, Ruby, JS) use arity-only. TS/JS overload signatures excluded (collapse to implementation body). See #651.
|
||||
|
||||
**C++ const-qualified:** `$const` suffix after type-hash when non-const collision exists: `Method:file:Container.begin#0$const`.
|
||||
|
||||
**Generic/template types:** type-hash uses `rawType` (full AST text including generics): `~vector<int>` vs `~vector<std::string>`.
|
||||
|
||||
**ID stability:** collision-only tags mean IDs change when overloads are added. `save#1` becomes `save#1~int` when `save(String)` is added.
|
||||
|
||||
**Variadic matching:** confidence 0.7 when one side is variadic and the other has fixed count.
|
||||
|
||||
**METHOD_IMPLEMENTS confidence tiering:**
|
||||
|
||||
| Match quality | Confidence |
|
||||
|---|---|
|
||||
| Exact parameter types match | 1.0 |
|
||||
| Arity match, types unavailable | 1.0 |
|
||||
| Variadic vs fixed | 0.7 |
|
||||
| Insufficient info | 0.7 |
|
||||
|
||||
## Related docs
|
||||
|
||||
- [MIGRATION.md](MIGRATION.md) — breaking changes and migration guidance
|
||||
- [RUNBOOK.md](RUNBOOK.md) — operational commands and recovery
|
||||
- [GUARDRAILS.md](GUARDRAILS.md) — safety boundaries for humans and agents
|
||||
- [TESTING.md](TESTING.md) — how to run tests
|
||||
- `AGENTS.md` / `CLAUDE.md` — agent workflows and tool usage
|
||||
@@ -2,63 +2,6 @@
|
||||
|
||||
All notable changes to GitNexus will be documented in this file.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Changed
|
||||
- Migrated from KuzuDB to LadybugDB v0.15 (`@ladybugdb/core`, `@ladybugdb/wasm-core`)
|
||||
- Renamed all internal paths from `kuzu` to `lbug` (storage: `.gitnexus/kuzu` → `.gitnexus/lbug`)
|
||||
- Added automatic cleanup of stale KuzuDB index files
|
||||
- LadybugDB v0.15 requires explicit VECTOR extension loading for semantic search
|
||||
|
||||
## [1.5.3] - 2026-04-01
|
||||
|
||||
### Added
|
||||
|
||||
- **TypeScript/JavaScript MethodExtractor config** — shared extraction config covering abstract methods, visibility modifiers, async/override keywords, decorators, rest/optional/destructured parameters, and return types (#588) — @compound-ai
|
||||
|
||||
### Fixed
|
||||
|
||||
- **Azure OpenAI compatibility** — use `max_completion_tokens` instead of deprecated `max_tokens` (newer models reject `max_tokens`); skip `temperature` for Azure provider (some models reject non-default values) (#618)
|
||||
- **Simplified Azure interactive setup** — 3 prompts (endpoint, deployment, key) instead of 7 (#618)
|
||||
- **Wiki HTML viewer script injection** — escape `</script>` in embedded JSON so LLM-generated markdown no longer breaks the viewer (#618)
|
||||
- Ensure import rewrites survive npm publish lifecycle
|
||||
|
||||
## [1.4.0] - 2026-03-13
|
||||
|
||||
### Added
|
||||
|
||||
- **Language-aware symbol resolution engine** with 3-tier resolver: exact FQN → scope-walk → guarded fuzzy fallback that refuses ambiguous matches (#238) — @magyargergo
|
||||
- **Method Resolution Order (MRO)** with 5 language-specific strategies: C++ leftmost-base, C#/Java class-over-interface, Python C3 linearization, Rust qualified syntax, default BFS (#238) — @magyargergo
|
||||
- **Constructor & struct literal resolution** across all languages — `new Foo()`, `User{...}`, C# primary constructors, target-typed new (#238) — @magyargergo
|
||||
- **Receiver-constrained resolution** using per-file TypeEnv — disambiguates `user.save()` vs `repo.save()` via `ownerId` matching (#238) — @magyargergo
|
||||
- **Heritage & ownership edges** — HAS_METHOD, OVERRIDES, Go struct embedding, Swift extension heritage, method signatures (`parameterCount`, `returnType`) (#238) — @magyargergo
|
||||
- **Language-specific resolver directory** (`resolvers/`) — extracted JVM, Go, C#, PHP, Rust resolvers from monolithic import-processor (#238) — @magyargergo
|
||||
- **Type extractor directory** (`type-extractors/`) — per-language type binding extraction with `Record<SupportedLanguages, Handler>` + `satisfies` dispatch (#238) — @magyargergo
|
||||
- **Export detection dispatch table** — compile-time exhaustive `Record` + `satisfies` pattern replacing switch/if chains (#238) — @magyargergo
|
||||
- **Language config module** (`language-config.ts`) — centralized tsconfig, go.mod, composer.json, .csproj, Swift package config loaders (#238) — @magyargergo
|
||||
- **Optional skill generation** via `npx gitnexus analyze --skills` — generates AI agent skills from KuzuDB knowledge graph (#171) — @zander-raycraft
|
||||
- **First-class C# support** — sibling-based modifier scanning, record/delegate/property/field/event declaration types (#163, #170, #178 via #237) — @Alice523, @benny-yamagata, @jnMetaCode
|
||||
- **C/C++ support fixes** — `.h` → C++ mapping, static-linkage export detection, qualified/parenthesized declarators, 48 entry point patterns (#163, #227 via #237) — @Alice523, @bitgineer
|
||||
- **Rust support fixes** — sibling-based `visibility_modifier` scanning for `pub` detection (#227 via #237) — @bitgineer
|
||||
- **Adaptive tree-sitter buffer sizing** — `Math.min(Math.max(contentLength * 2, 512KB), 32MB)` (#216 via #237) — @JasonOA888
|
||||
- **Call expression matching** in tree-sitter queries (#234 via #237) — @ex-nihilo-jg
|
||||
- **DeepSeek model configurations** (#217) — @JasonOA888
|
||||
- 282+ new unit tests, 178 integration resolver tests across 9 languages, 53 test files, 1146 total tests passing
|
||||
|
||||
### Fixed
|
||||
|
||||
- Skip unavailable native Swift parsers in sequential ingestion (#188) — @Gujiassh
|
||||
- Heritage heuristic language-gated — no longer applies class/interface rules to wrong languages (#238) — @magyargergo
|
||||
- C# `base_list` distinguishes EXTENDS vs IMPLEMENTS via symbol table + `I[A-Z]` heuristic (#238) — @magyargergo
|
||||
- Go `qualified_type` (`models.User`) correctly unwrapped in TypeEnv (#238) — @magyargergo
|
||||
- Global tier no longer blocks resolution when kind/arity filtering can narrow to 1 candidate (#238) — @magyargergo
|
||||
|
||||
### Changed
|
||||
|
||||
- `import-processor.ts` reduced from 1412 → 711 lines (50% reduction) via resolver and config extraction (#238) — @magyargergo
|
||||
- `type-env.ts` reduced from 635 → ~125 lines via type-extractor extraction (#238) — @magyargergo
|
||||
- CI/CD workflows hardened with security fixes and fork PR support (#222, #225) — @magyargergo
|
||||
|
||||
## [1.3.11] - 2026-03-08
|
||||
|
||||
### Security
|
||||
|
||||
@@ -1,54 +1,78 @@
|
||||
<!-- version: 1.3.0 -->
|
||||
<!--
|
||||
Metadata: version, last reviewed, scope, model policy, reference docs, changelog.
|
||||
Last updated: 2026-03-22
|
||||
-->
|
||||
<!-- gitnexus:start -->
|
||||
# GitNexus — Code Intelligence
|
||||
|
||||
Last reviewed: 2026-04-13
|
||||
This project is indexed by GitNexus as **GitNexus** (1650 symbols, 4291 relationships, 125 execution flows). Use the GitNexus MCP tools to understand code, assess impact, and navigate safely.
|
||||
|
||||
**Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub)
|
||||
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
|
||||
|
||||
Follow **AGENTS.md** for the canonical rules; this file adds Claude Code–specific deltas. Cursor-specific notes live only in `AGENTS.md`.
|
||||
## Always Do
|
||||
|
||||
## Scope
|
||||
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
|
||||
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
|
||||
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
|
||||
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
|
||||
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
|
||||
|
||||
See the **Scope** table in [AGENTS.md](AGENTS.md) for read/write/execute/off-limits boundaries. Cursor-specific workflow notes also live only in AGENTS.md.
|
||||
## When Debugging
|
||||
|
||||
## Model Configuration
|
||||
1. `gitnexus_query({query: "<error or symptom>"})` — find execution flows related to the issue
|
||||
2. `gitnexus_context({name: "<suspect function>"})` — see all callers, callees, and process participation
|
||||
3. `READ gitnexus://repo/GitNexus/process/{processName}` — trace the full execution flow step by step
|
||||
4. For regressions: `gitnexus_detect_changes({scope: "compare", base_ref: "main"})` — see what your branch changed
|
||||
|
||||
- **Primary:** Pin per **Claude Code** / Anthropic org policy (explicit model id). Do not rely on an unversioned `latest` alias for governed workflows.
|
||||
- **Fallback:** As configured in Claude Code (organization default or user override).
|
||||
- **Notes:** The GitNexus CLI analyzer does not call an LLM.
|
||||
## When Refactoring
|
||||
|
||||
## Execution Sequence (complex tasks)
|
||||
- **Renaming**: MUST use `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` first. Review the preview — graph edits are safe, text_search edits need manual review. Then run with `dry_run: false`.
|
||||
- **Extracting/Splitting**: MUST run `gitnexus_context({name: "target"})` to see all incoming/outgoing refs, then `gitnexus_impact({target: "target", direction: "upstream"})` to find all external callers before moving code.
|
||||
- After any refactor: run `gitnexus_detect_changes({scope: "all"})` to verify only expected files changed.
|
||||
|
||||
Same discipline as [AGENTS.md](AGENTS.md): before large multi-step work, state which **AGENTS.md** / **GUARDRAILS.md** rules apply, current **Scope**, and planned validation commands (`npm test`, `tsc`, etc.). When pausing, summarize progress in the chat or a **local** scratch file (do not add `HANDOFF.md` to the repo), then `/clear` and resume with that summary.
|
||||
## Never Do
|
||||
|
||||
## Claude Code hooks
|
||||
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
|
||||
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
|
||||
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
|
||||
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
|
||||
|
||||
Prefer **PreToolUse** hooks for hard gates (e.g. tests before `git_commit`). Adapt hook commands to `gitnexus/` npm scripts.
|
||||
## Tools Quick Reference
|
||||
|
||||
## Context budget
|
||||
| Tool | When to use | Command |
|
||||
|------|-------------|---------|
|
||||
| `query` | Find code by concept | `gitnexus_query({query: "auth validation"})` |
|
||||
| `context` | 360-degree view of one symbol | `gitnexus_context({name: "validateUser"})` |
|
||||
| `impact` | Blast radius before editing | `gitnexus_impact({target: "X", direction: "upstream"})` |
|
||||
| `detect_changes` | Pre-commit scope check | `gitnexus_detect_changes({scope: "staged"})` |
|
||||
| `rename` | Safe multi-file rename | `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` |
|
||||
| `cypher` | Custom graph queries | `gitnexus_cypher({query: "MATCH ..."})` |
|
||||
|
||||
If always-on instructions grow, load deep conventions via conditional reads (e.g. *“When writing new code, read STANDARDS.md”*) instead of pasting long blocks here. In Cursor, prefer `.cursor/index.mdc` plus optional `.cursor/rules/*.mdc` globs (see [AGENTS.md](AGENTS.md) § Context budget).
|
||||
## Impact Risk Levels
|
||||
|
||||
## Reference Documentation
|
||||
| Depth | Meaning | Action |
|
||||
|-------|---------|--------|
|
||||
| d=1 | WILL BREAK — direct callers/importers | MUST update these |
|
||||
| d=2 | LIKELY AFFECTED — indirect deps | Should test |
|
||||
| d=3 | MAY NEED TESTING — transitive | Test if critical path |
|
||||
|
||||
- **This repository:** [AGENTS.md](AGENTS.md) (Cursor + monorepo notes), [ARCHITECTURE.md](ARCHITECTURE.md), [CONTRIBUTING.md](CONTRIBUTING.md), [GUARDRAILS.md](GUARDRAILS.md).
|
||||
- **Call-resolution DAG:** See ARCHITECTURE.md § Call-Resolution DAG. Shared pipeline code in `gitnexus/src/core/ingestion/` must not name languages — use `LanguageProvider` hooks instead (see AGENTS.md).
|
||||
- **GitNexus:** `.claude/skills/gitnexus/`; MCP and indexed-repo rules live only in [AGENTS.md](AGENTS.md) (`gitnexus:start` … `gitnexus:end`). See **GitNexus rules** below.
|
||||
## Resources
|
||||
|
||||
## Changelog
|
||||
| Resource | Use for |
|
||||
|----------|---------|
|
||||
| `gitnexus://repo/GitNexus/context` | Codebase overview, check index freshness |
|
||||
| `gitnexus://repo/GitNexus/clusters` | All functional areas |
|
||||
| `gitnexus://repo/GitNexus/processes` | All execution flows |
|
||||
| `gitnexus://repo/GitNexus/process/{name}` | Step-by-step execution trace |
|
||||
|
||||
| Date | Version | Change |
|
||||
|------|---------|--------|
|
||||
| 2026-04-13 | 1.3.0 | Updated GitNexus index stats after DAG refactor. |
|
||||
| 2026-03-24 | 1.2.0 | Removed duplicated gitnexus:start block and scope table; replaced with pointers to AGENTS.md. |
|
||||
| 2026-03-23 | 1.1.0 | Updated agent instructions to match AGENTS.md. |
|
||||
| 2026-03-22 | 1.0.0 | Added structured header and changelog. |
|
||||
## Self-Check Before Finishing
|
||||
|
||||
---
|
||||
Before completing any code modification task, verify:
|
||||
1. `gitnexus_impact` was run for all modified symbols
|
||||
2. No HIGH/CRITICAL risk warnings were ignored
|
||||
3. `gitnexus_detect_changes()` confirms changes match expected scope
|
||||
4. All d=1 (WILL BREAK) dependents were updated
|
||||
|
||||
## GitNexus rules
|
||||
## CLI
|
||||
|
||||
See the `<!-- gitnexus:start --> … <!-- gitnexus:end -->` block in **[AGENTS.md](AGENTS.md)** for the canonical MCP tools, impact analysis rules, and index instructions.
|
||||
- Re-index: `npx gitnexus analyze`
|
||||
- Check freshness: `npx gitnexus status`
|
||||
- Generate docs: `npx gitnexus wiki`
|
||||
|
||||
<!-- gitnexus:end -->
|
||||
|
||||
-174
@@ -1,174 +0,0 @@
|
||||
# Contributing to GitNexus
|
||||
|
||||
How to propose changes, run checks locally, and open pull requests.
|
||||
|
||||
## License
|
||||
|
||||
This project uses the [PolyForm Noncommercial License 1.0.0](https://polyformproject.org/licenses/noncommercial/1.0.0/). By contributing, you agree your contributions are licensed under the same terms unless stated otherwise.
|
||||
|
||||
## Where to discuss
|
||||
|
||||
- **Issues & feature ideas:** use [GitHub Issues](https://github.com/abhigyanpatwari/GitNexus/issues) for the upstream repo, or your fork’s tracker if you work from a fork.
|
||||
- **Community:** see the Discord link in the root [README.md](README.md).
|
||||
|
||||
## Development setup
|
||||
|
||||
1. Clone the repository.
|
||||
2. **CLI / MCP package:** `cd gitnexus && npm install && npm run build`
|
||||
3. **Web UI (if needed):** `cd gitnexus-web && npm install`
|
||||
4. Run tests as described in [TESTING.md](TESTING.md).
|
||||
|
||||
## Branch and pull requests
|
||||
|
||||
- Use short-lived branches off the default branch of the repo you are targeting.
|
||||
- **PR titles MUST follow the conventional-commit format** — `pr-labeler.yml` enforces this on every PR and auto-applies the matching label so release notes group the change correctly.
|
||||
- **PR description:** what changed, why, how to verify (commands), and any risk or rollback notes.
|
||||
|
||||
### Pull request titles
|
||||
|
||||
Format: `<type>[(scope)][!]: <subject>`
|
||||
|
||||
Allowed types and the release-notes section each one lands in (defined in `.github/release.yml`):
|
||||
|
||||
| Type | Label applied | Release-notes section |
|
||||
|------|---------------|-----------------------|
|
||||
| `feat` | `enhancement` | 🚀 Features |
|
||||
| `fix` | `bug` | 🐛 Bug Fixes |
|
||||
| `perf` | `performance` | 🏎️ Performance |
|
||||
| `refactor` | `refactor` | 🔄 Refactoring |
|
||||
| `test` | `test` | 🧪 Tests |
|
||||
| `ci` | `ci` | 👷 CI/CD |
|
||||
| `build` / `deps` | `dependencies` | 📦 Dependencies |
|
||||
| `docs` | `documentation` | (grouped under Other Changes unless a Docs section is added) |
|
||||
| `chore` / `revert` | `chore` | (excluded from release notes) |
|
||||
|
||||
Append `!` to the type (e.g. `feat(api)!: drop /v1 endpoint`) or include `BREAKING CHANGE:` in the PR body to flag a breaking change — the labeler then adds the `breaking` label and the 💥 Breaking Changes section is rendered first.
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
feat(web): add smart chat scroll
|
||||
fix(extractors): resolve silent contract mis-resolution
|
||||
perf: avoid O(n²) traversal in heritage walker
|
||||
chore(deps): bump vitest to 3.0.0
|
||||
ci: standardize workflow concurrency
|
||||
```
|
||||
|
||||
Commits within a PR may use any style — only the **merged PR title** shows up in release notes, so that's the one the convention applies to.
|
||||
|
||||
## Before you open a PR
|
||||
|
||||
- [ ] Tests pass for the packages you touched (`gitnexus` and/or `gitnexus-web`).
|
||||
- [ ] Typecheck passes: `npx tsc --noEmit` in `gitnexus/` and `npx tsc -b --noEmit` in `gitnexus-web/`.
|
||||
- [ ] No secrets, tokens, or machine-specific paths committed.
|
||||
- [ ] Documentation updated if behavior or public CLI/MCP contract changes.
|
||||
- [ ] Pre-commit hook runs clean (`.husky/pre-commit` — formatting via lint-staged + typecheck for staged packages; tests run in CI only).
|
||||
|
||||
## Code review
|
||||
|
||||
Maintainers may request changes for correctness, tests, performance, or consistency with existing patterns. Keeping diffs focused makes review faster.
|
||||
|
||||
## GitHub Actions — Concurrency Convention
|
||||
|
||||
Every workflow under `.github/workflows/` MUST declare a top-level `concurrency:` block using this convention:
|
||||
|
||||
- **Group key** starts with `${{ github.workflow }}` so no two workflows can collide on the same group name. The discriminator that follows is chosen per event shape:
|
||||
- Branch/tag scope: `${{ github.workflow }}-${{ github.ref }}`
|
||||
- Per-PR scope (for `issue_comment`, `pull_request_review*`, `pull_request` meta events): `${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number }}`
|
||||
- `workflow_run` scope (e.g. `ci-report.yml`): `${{ github.workflow }}-${{ github.event.workflow_run.pull_requests[0].number || format('{0}/{1}', github.event.workflow_run.head_repository.full_name, github.event.workflow_run.head_branch) }}` — the fork fallback must be stable across reruns (never `workflow_run.id`, which is per-run-unique and defeats serialization).
|
||||
- Global single-slot (manual dispatch utilities): `${{ github.workflow }}`
|
||||
- **Reusable workflows invoked via `workflow_call`:** do NOT use `${{ github.workflow }}` in the group key — in called-workflow context its evaluation is ambiguous and can resolve to the caller's name, which would deadlock against the caller's own group. Use a hardcoded literal prefix and a `github.event_name`-aware expression that falls through to `github.run_id` for reusable invocations (see `ci.yml` for the canonical form). Approved literal prefixes: `CI-` (`ci.yml`) and `docker-build-push-` (`docker.yml`). The `check-workflow-concurrency.py` validation script must be updated whenever a new approved literal prefix is added.
|
||||
- **Merge queue (`merge_group`)**: when this event is added, use `${{ github.workflow }}-${{ github.event.merge_group.head_ref }}` with `cancel-in-progress: false` (every queue entry is a distinct ref; never cancel).
|
||||
- **`cancel-in-progress` policy:**
|
||||
|
||||
| Event | `cancel-in-progress` | Why |
|
||||
|-------|----------------------|-----|
|
||||
| `pull_request` CI run | `true` | New push supersedes old run |
|
||||
| `push` to `main` | `false` | Every main commit gets validated |
|
||||
| Tag push (`v*` publish) | `false` | Never cancel mid-publish |
|
||||
| `push` to `main` for release-candidate | `false` | Never cancel mid-RC publish |
|
||||
| `workflow_dispatch` (release/publish) | `false` | Manual runs are intentional |
|
||||
| `workflow_run` (sticky-comment reports) | `false` | Serialize, don't race |
|
||||
| Per-PR bot workflows (`@claude`, review) | `false` | Serialize comments per PR |
|
||||
| PR-meta re-checks (pr-description-check) | `true` | Cheap, latest wins |
|
||||
| Single-slot utilities (triage sweep) | `true` | Latest dispatch supersedes |
|
||||
|
||||
- For workflows that serve multiple events at once (e.g. `ci.yml` handles `pull_request`, `push`, and `workflow_call`), make `cancel-in-progress` event-aware:
|
||||
|
||||
```yaml
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: ${{ github.event_name == 'pull_request' }}
|
||||
```
|
||||
|
||||
- When adding a new workflow, copy the concurrency block from an existing workflow of the same event shape.
|
||||
|
||||
## AI-assisted contributions
|
||||
|
||||
If you use coding agents, follow project context files (e.g. `AGENTS.md`, `CLAUDE.md`) and avoid drive-by refactors unrelated to the issue. Prefer incremental, test-backed changes.
|
||||
|
||||
## Releases
|
||||
|
||||
Two publish workflows ship `gitnexus` to npm:
|
||||
|
||||
- **Stable** (`.github/workflows/publish.yml`) — triggered by pushing any `v*`
|
||||
tag. Publishes to the `latest` dist-tag with a changelog-backed GitHub
|
||||
release. Maintainers are expected to tag from `main` as a convention; the
|
||||
workflow itself does not enforce branch reachability.
|
||||
- **Release Candidate** (`.github/workflows/release-candidate.yml`) — runs on
|
||||
every push to `main` (typically a merged PR) plus manual dispatch. Docs-only
|
||||
changes are skipped via `paths-ignore`. Publishes to the `rc` dist-tag with
|
||||
version `X.Y.Z-rc.N` and a GitHub prerelease, where:
|
||||
- `X.Y.Z` is selected automatically. On push (and on dispatch with
|
||||
`bump: auto`, the default) the workflow **continues the active rc cycle**:
|
||||
if the registry already has `X.Y.Z-rc.*` versions with `X.Y.Z` > current
|
||||
`latest`, it reuses the highest such base; otherwise it patch-bumps
|
||||
from `latest`. Dispatching with `bump: patch|minor|major` **resets**
|
||||
the cycle from `latest`.
|
||||
- `N` is auto-incremented against existing `X.Y.Z-rc.*` entries on the
|
||||
registry. First rc for a given base is `rc.1`.
|
||||
- After the npm publish succeeds, the workflow calls `docker.yml` as a
|
||||
reusable workflow to build and push the corresponding RC Docker images
|
||||
(e.g. `ghcr.io/abhigyanpatwari/gitnexus:1.7.0-rc.1`, mirrored to
|
||||
`docker.io/akonlabs/gitnexus:1.7.0-rc.1`). The images are signed
|
||||
with Cosign; the OIDC identity is `docker.yml@refs/heads/main` (the
|
||||
caller's ref — see README.md § Docker for the verify command).
|
||||
|
||||
Idempotency: the workflow pushes an `rc/<HEAD_SHA>` marker tag and a
|
||||
`v<RC>` release tag **atomically, before** calling `npm publish`. The guard
|
||||
refuses to re-run once the marker exists, so a post-publish failure will
|
||||
not mint a duplicate rc for the same commit. The `v<RC>` tag points at a
|
||||
detached release commit whose `package.json` matches the npm tarball
|
||||
exactly (traceable releases). Recovery after a partial failure:
|
||||
|
||||
```bash
|
||||
git push --delete origin rc/<HEAD_SHA> v<RC>
|
||||
# then redispatch the workflow with force: true
|
||||
```
|
||||
|
||||
**Docker-only partial failure:** if `publish` succeeds (npm tarball + tags
|
||||
are live) but the `docker` job subsequently fails (e.g. GHCR flakiness),
|
||||
the npm RC is already published and the `rc/<HEAD_SHA>` marker is in place.
|
||||
Re-running `release-candidate.yml` with `force: true` will abort at the
|
||||
"Version already exists on npm" guard. To recover without cutting a new RC:
|
||||
|
||||
```bash
|
||||
# 1. Manually trigger only the docker workflow, passing the existing RC tag:
|
||||
gh workflow run docker.yml --ref main -f tag=v<RC_VERSION>
|
||||
# (requires a workflow_dispatch trigger on docker.yml — see note below)
|
||||
```
|
||||
|
||||
Because `docker.yml` intentionally has no `workflow_dispatch` (images are
|
||||
tag-driven by design), the practical recovery options are:
|
||||
- Wait for the next commit on `main`, which will cut a new RC that includes
|
||||
the Docker build.
|
||||
- Manually run `docker build` + `docker push` locally and sign with Cosign
|
||||
against the same digest.
|
||||
- Delete `rc/<HEAD_SHA>` and `v<RC>` tags, then redispatch with `force:
|
||||
true` to re-run the full RC pipeline (cuts a new RC number).
|
||||
|
||||
The rc workflow never moves `latest`. To verify after a change, inspect dist-tags:
|
||||
|
||||
```bash
|
||||
npm view gitnexus dist-tags
|
||||
```
|
||||
@@ -1,209 +0,0 @@
|
||||
# Definition of Done — GitNexus
|
||||
|
||||
Last reviewed: 2026-04-23 · Version: 2.0.0
|
||||
|
||||
This document defines the repo-wide completion bar for production-ready changes in GitNexus. It is the stable baseline. Implementation prompts, agent behavior, and review workflows may add task-specific checks, but they must never weaken this bar.
|
||||
|
||||
Use it together with:
|
||||
|
||||
- `AGENTS.md` — agent-facing rules of engagement
|
||||
- `GUARDRAILS.md` — hard safety constraints
|
||||
- `CONTRIBUTING.md` — contributor workflow
|
||||
- `TESTING.md` — test strategy and coverage expectations
|
||||
- `ARCHITECTURE.md` — pipeline boundaries, Call-Resolution DAG, LanguageProvider contract
|
||||
|
||||
## 1. Scope and Intent
|
||||
|
||||
A change is **Done** when it is correct, safely integrated, appropriately tested, operationally sound, and a net improvement to the codebase — not merely "the code compiles and a test passes."
|
||||
|
||||
This DoD applies to:
|
||||
|
||||
- CLI, MCP, and HTTP-bridge behavior in `gitnexus/`
|
||||
- Browser UI in `gitnexus-web/`
|
||||
- Shared contracts in `gitnexus-shared/`
|
||||
- CI workflows, release pipelines, and repo-level docs
|
||||
|
||||
Out of scope: full agent personas, step-by-step implementation prompts, verbose review formatting rules, repo walkthroughs already covered elsewhere, temporary task-specific acceptance criteria. Those belong in prompts, PR templates, or other repo docs.
|
||||
|
||||
## 2. Core Definition of Done
|
||||
|
||||
Every change must satisfy **every relevant item** below. If an item does not apply, say so explicitly in the PR description.
|
||||
|
||||
### 2.1 Correctness and Completeness
|
||||
|
||||
- [ ] The requested behavior is implemented end-to-end in the **real runtime path** for the affected surface — no dead code, partial wiring, test-only shims, or "works in isolation but not in production" seams.
|
||||
- [ ] Edge cases relevant to the changed surface are handled or explicitly documented as out of scope.
|
||||
- [ ] Error handling is proportionate: inputs at system boundaries (user input, external APIs, filesystem, process spawn) are validated; internal, framework-guaranteed paths are trusted.
|
||||
- [ ] The change produces the same result on re-run (idempotent where expected) and does not rely on accidental ordering.
|
||||
|
||||
### 2.2 Architecture and Placement
|
||||
|
||||
- [ ] The change is placed in the correct package and layer:
|
||||
- `gitnexus/` for CLI, MCP, HTTP bridge, ingestion, graph, and runtime logic
|
||||
- `gitnexus-web/` for browser UI (thin client — no WASM workers, all queries via HTTP API)
|
||||
- `gitnexus-shared/` for shared contracts, types, and constants
|
||||
- [ ] Pipeline and architecture boundaries remain explicit. Shared ingestion code in `gitnexus/src/core/ingestion/` must not name languages — use `LanguageProvider` hooks (see `AGENTS.md` and `ARCHITECTURE.md` § Call-Resolution DAG).
|
||||
- [ ] No hidden cross-phase coupling; no leaking of language-specific logic into shared infrastructure without a documented architectural reason.
|
||||
- [ ] Runtime and graph behavior are consistent — the real source of truth is fixed at the source, not symptom-patched in a downstream layer.
|
||||
- [ ] Direct imports from `gitnexus-shared` are used. No barrel re-exports introduced to paper over drift between packages.
|
||||
|
||||
### 2.3 Design and Readability
|
||||
|
||||
- [ ] The implementation is the **smallest correct solution** for the requirement. No speculative abstraction, unnecessary indirection, clever but hard-to-follow control flow, or unrelated cleanup.
|
||||
- [ ] Naming, control flow, ownership, and extension points are clear enough that the next contributor can extend the code without archaeology.
|
||||
- [ ] Comments are minimal and useful — they explain intent, invariants, contracts, or non-obvious constraints. No stale comments, placeholder comments, narrated code, commented-out code, or "what" comments where a good name would do.
|
||||
- [ ] No copy-paste duplication created for convenience; no premature deduplication of three similar lines.
|
||||
|
||||
### 2.4 Contracts and Compatibility
|
||||
|
||||
- [ ] Existing contracts (types in `gitnexus-shared/`, CLI flags, MCP tools/resources, HTTP routes, graph node/edge shapes, persisted IDs) are preserved unless the task explicitly requires a contract change.
|
||||
- [ ] Any contract change is intentional, explicit, and reflected in **every direct consumer** in the same change, with types aligned end-to-end.
|
||||
- [ ] Persisted data changes (graph schema, IDs, embeddings) are backward-compatible or accompanied by a documented migration / reindex path.
|
||||
- [ ] If user-visible behavior, public usage, CLI help, or README examples change, the relevant docs, examples, help text, or migration notes are updated in the same change.
|
||||
|
||||
### 2.5 Security
|
||||
|
||||
- [ ] No new injection surfaces (command, path, SQL/Cypher-style, prompt) introduced on paths that consume untrusted input.
|
||||
- [ ] No secrets, tokens, or credentials committed to the repo, to logs, or to error messages.
|
||||
- [ ] Filesystem access honors the repo-scope and indexed-repo boundaries documented in `AGENTS.md` and `GUARDRAILS.md`.
|
||||
- [ ] Third-party dependencies added or bumped are justified, from reputable sources, and do not regress the supply-chain posture.
|
||||
|
||||
### 2.6 Performance and Resource Use
|
||||
|
||||
- [ ] No repeated avoidable work, unnecessary scans, unnecessary round-trips, unbounded caches, or obvious hot-path regressions.
|
||||
- [ ] Tree-sitter buffer sizing follows the adaptive 512KB–32MB convention (`getTreeSitterBufferSize`) — do not hard-code new buffer sizes.
|
||||
- [ ] Memory and handle lifecycles are explicit: database handles (LadybugDB) close cleanly, no dangling process watchers, no leaked tree-sitter parsers.
|
||||
- [ ] Long-running or large-graph paths remain bounded or are measurably streamed; degradation on large real repos is considered, not assumed benign.
|
||||
|
||||
### 2.7 Tests
|
||||
|
||||
- [ ] Tests cover the **real changed path** — they would fail if behavior, wiring, or contracts were broken, not only if a mock were misconfigured.
|
||||
- [ ] Integration tests hit a real database where the production path does; do not introduce mocks that hide migration or schema drift.
|
||||
- [ ] Assertions are meaningful. Use `toBe` / `toEqual` for exact expectations; avoid `toBeGreaterThanOrEqual` and other bounds-only assertions that mask regressions.
|
||||
- [ ] Fixtures are realistic enough for the risk of the change — a one-file fixture is not sufficient for a pipeline-wide behavior change.
|
||||
- [ ] New tests are deterministic and do not depend on network, clock, or host-specific paths without explicit isolation.
|
||||
|
||||
### 2.8 Observability and Operability
|
||||
|
||||
- [ ] Errors surfaced to users or callers are actionable: they name what failed, what input was involved (without leaking secrets), and how to recover where possible.
|
||||
- [ ] Logging is proportionate — no noisy debug logs left in hot paths, no silent catches that swallow diagnostics.
|
||||
- [ ] CLI exit codes and MCP tool responses are correct for each outcome (success, user error, internal error).
|
||||
- [ ] Progress reporting (`PipelineProgress` and similar shared contracts) remains accurate after the change.
|
||||
|
||||
### 2.9 Reversibility and Risk
|
||||
|
||||
- [ ] The change has a clear rollback story: revert is safe, or migration is accompanied by a documented rollback / reindex procedure.
|
||||
- [ ] Residual risks, compatibility impacts, and operational concerns are either resolved or **clearly stated** in the PR description.
|
||||
- [ ] Destructive or hard-to-reverse operations (graph rebuild, schema change, `git` state manipulation) are opt-in or guarded.
|
||||
|
||||
## 3. Agent-Assisted Workflow Guardrails
|
||||
|
||||
When the change is produced with or reviewed by an AI agent, the following additional gates apply:
|
||||
|
||||
- [ ] **Scope match.** The final diff matches the intended symbols, files, and processes — no speculative refactors, unrelated formatting churn, or collateral edits outside the task scope.
|
||||
- [ ] **Evidence-based edits.** Claims about repo state are verified against the current code, not trusted from memory or stale documentation.
|
||||
- [ ] **Impact analysis.** Where GitNexus graph tooling is available and relevant, impact of non-trivial symbol, contract, or runtime-path changes is checked **before** editing.
|
||||
- [ ] **Embeddings preserved.** If an indexed repo already has embeddings and re-analysis is required, embeddings are preserved — not accidentally dropped by a destructive reindex.
|
||||
- [ ] **No false-done.** "Done" is claimed only after the Validation Baseline below has been run or any gap is explicitly named. Green tests on an unrelated path do not constitute validation.
|
||||
- [ ] **Five-axis self-review** before handing off: correctness, readability, architecture, security, performance.
|
||||
|
||||
## 4. Validation Baseline
|
||||
|
||||
Run the commands relevant to the touched area. If something cannot be run in the current environment, state it explicitly in the handoff.
|
||||
|
||||
### 4.1 Build ordering
|
||||
|
||||
- [ ] `gitnexus-shared/` dist is built before consuming packages are typechecked or tested (CI uses the `setup-gitnexus` action for this — local runs must match).
|
||||
|
||||
### 4.2 If `gitnexus/` changed
|
||||
|
||||
- [ ] `cd gitnexus && npx tsc --noEmit`
|
||||
- [ ] `cd gitnexus && npm test`
|
||||
- [ ] `cd gitnexus && npx prettier --check .` for files in the diff (pre-commit runs the affected-tests subset; do not expand scope)
|
||||
|
||||
### 4.3 If `gitnexus-web/` changed
|
||||
|
||||
- [ ] `cd gitnexus-web && npx tsc -b --noEmit`
|
||||
- [ ] `cd gitnexus-web && npm test`
|
||||
- [ ] `cd gitnexus-web && npm run test:e2e` when browser flows or user-facing UI behavior changed
|
||||
|
||||
### 4.4 If `gitnexus-shared/` changed
|
||||
|
||||
- [ ] Shared package builds cleanly (`npm run build` in `gitnexus-shared/`)
|
||||
- [ ] Dependent packages still typecheck and test after the shared change — verify both CLI and web consumers together
|
||||
|
||||
### 4.5 If CI workflows or release pipelines changed
|
||||
|
||||
- [ ] The workflow passes a dry-run or triggered run before merge; concurrency (`cancel-in-progress`) and the `setup-gitnexus` action remain wired correctly.
|
||||
- [ ] `CHANGELOG.md` is **not** edited here — it is owned by the release process.
|
||||
|
||||
## 5. Review Gates
|
||||
|
||||
A reviewer (human or agent) should be able to answer **yes** to each of the following before approving:
|
||||
|
||||
1. **Correctness** — Does the change do what it claims on the real runtime path?
|
||||
2. **Readability** — Will the next contributor understand this in six months without asking?
|
||||
3. **Architecture** — Is it in the right package, layer, and phase? Are boundaries respected?
|
||||
4. **Security** — No new injection, leak, or trust-boundary violation?
|
||||
5. **Performance** — No obvious regression on realistic inputs?
|
||||
6. **Tests** — Would a regression in the changed behavior fail loudly?
|
||||
7. **Scope** — Does the diff match the intended change, with no unrelated churn?
|
||||
|
||||
## 6. "Not Done" Signals
|
||||
|
||||
A change is **not** Done if any of the following is true, even if CI is green:
|
||||
|
||||
- The runtime path is not actually exercised by the tests.
|
||||
- A contract drifted between `gitnexus/`, `gitnexus-web/`, and `gitnexus-shared/` and only one side was updated.
|
||||
- A language-specific concern leaked into shared ingestion code.
|
||||
- The diff contains unrelated reformatting, refactors, or cleanup beyond the stated task.
|
||||
- Logs, comments, or TODOs were added as placeholders for work not done.
|
||||
- The change depends on a manual step that is not documented.
|
||||
- `CHANGELOG.md` was edited during PR work.
|
||||
- Pre-commit, prettier, or typecheck was bypassed without explicit justification.
|
||||
|
||||
## 7. Task-Specific DoD Template
|
||||
|
||||
Use this in implementation and review prompts. Keep it short and tailor it to the actual change:
|
||||
|
||||
```md
|
||||
# Definition of Done for this implementation
|
||||
|
||||
- [ ] Runtime wiring is complete for the affected path.
|
||||
- [ ] Requested behavior is correct and relevant contracts are preserved or explicitly updated.
|
||||
- [ ] The design stays scoped, readable, and proportionate to the task.
|
||||
- [ ] Tests prove the changed behavior and catch broken wiring.
|
||||
- [ ] Required validation for touched packages has been run, or any gap is explicitly noted.
|
||||
- [ ] Repo boundaries, security, performance, and operational safety are respected.
|
||||
- [ ] The diff contains only the intended change — no unrelated churn.
|
||||
```
|
||||
|
||||
## 8. How to Use This File in Claude Review
|
||||
|
||||
Reference this file as the repo-wide completion bar. Add a task-specific review instruction such as:
|
||||
|
||||
```md
|
||||
Review this change against `DoD.md` and the repo docs (`AGENTS.md`, `GUARDRAILS.md`,
|
||||
`CONTRIBUTING.md`, `TESTING.md`, `ARCHITECTURE.md`). Treat `DoD.md` as the minimum
|
||||
bar for production readiness. Flag anything that is partially wired, contract-unsafe,
|
||||
under-tested, architecturally misplaced, scope-creeping, or harder to maintain than
|
||||
necessary. Apply the five-axis review gate: correctness, readability, architecture,
|
||||
security, performance.
|
||||
```
|
||||
|
||||
## 9. Evolution
|
||||
|
||||
This DoD is living. Revisit it when:
|
||||
|
||||
- A class of incident slips past it (add a gate).
|
||||
- A gate becomes consistently ceremonial without catching issues (remove or merge it).
|
||||
- The architecture evolves in a way that changes what "done" means (update placement, validation, or contracts sections).
|
||||
|
||||
Track material updates in the changelog below. Keep the file tight — if it grows past a single read-in-one-sitting, something has drifted into the wrong place.
|
||||
|
||||
## Changelog
|
||||
|
||||
| Date | Version | Change |
|
||||
| ---------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| 2026-04-23 | 2.0.0 | Restructured into numbered sections; added Security, Observability, Reversibility, Agent-Assisted Guardrails, Review Gates, Not-Done Signals; expanded validation baseline (shared-first build, prettier, CI workflow checks). |
|
||||
| 2026-04-13 | 1.0.0 | Initial repo-wide Definition of Done. |
|
||||
@@ -1,58 +0,0 @@
|
||||
ARG BUILDPLATFORM
|
||||
ARG TARGETPLATFORM
|
||||
|
||||
# ── Builder ────────────────────────────────────────────────────────────
|
||||
# Native modules (tree-sitter-*, onnxruntime-node, node-gyp builds for
|
||||
# tree-sitter-proto / tree-sitter-swift) require python3 + a C/C++ toolchain.
|
||||
FROM node:22-trixie-slim AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Toolchain for node-gyp / native builds.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends python3 make g++ git && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Build gitnexus-shared first — gitnexus depends on it as a workspace.
|
||||
COPY gitnexus-shared/package.json gitnexus-shared/package-lock.json ./gitnexus-shared/
|
||||
RUN npm ci --prefix gitnexus-shared
|
||||
COPY gitnexus-shared ./gitnexus-shared
|
||||
RUN rm -f gitnexus-shared/tsconfig.tsbuildinfo
|
||||
RUN npm run build --prefix gitnexus-shared
|
||||
|
||||
# Copy the full gitnexus package before installing — `npm ci` triggers
|
||||
# `postinstall` (patches tree-sitter-swift, builds the vendored
|
||||
# tree-sitter-proto) and `prepare` (compiles TypeScript via scripts/build.js),
|
||||
# both of which need the source tree.
|
||||
COPY gitnexus ./gitnexus
|
||||
RUN npm ci --prefix gitnexus
|
||||
|
||||
# Drop dev dependencies for a smaller runtime layer.
|
||||
RUN npm prune --omit=dev --prefix gitnexus
|
||||
|
||||
# ── Runtime ────────────────────────────────────────────────────────────
|
||||
FROM node:22-trixie-slim AS runtime
|
||||
|
||||
# curl for the healthcheck; git so `gitnexus` can clone repos at runtime.
|
||||
RUN apt-get update && apt-get install -y --no-install-recommends curl git && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Pre-create the data directory and hand it to the unprivileged `node` user
|
||||
# so the bind-mounted volume is writable without root.
|
||||
RUN mkdir -p /data/gitnexus && chown -R node:node /data
|
||||
|
||||
COPY --from=builder --chown=node:node /app/gitnexus/dist ./gitnexus/dist
|
||||
COPY --from=builder --chown=node:node /app/gitnexus/node_modules ./gitnexus/node_modules
|
||||
COPY --from=builder --chown=node:node /app/gitnexus/package.json ./gitnexus/package.json
|
||||
COPY --from=builder --chown=node:node /app/gitnexus/vendor ./gitnexus/vendor
|
||||
|
||||
USER node
|
||||
|
||||
# The web UI defaults to http://localhost:4747 — keep that contract.
|
||||
ENV GITNEXUS_HOME=/data/gitnexus \
|
||||
NODE_ENV=production \
|
||||
PORT=4747
|
||||
|
||||
EXPOSE 4747
|
||||
|
||||
# Bind to 0.0.0.0 so the server is reachable from the host's mapped port.
|
||||
CMD ["node", "gitnexus/dist/cli/index.js", "serve", "--host", "0.0.0.0", "--port", "4747"]
|
||||
@@ -1,37 +0,0 @@
|
||||
ARG BUILDPLATFORM
|
||||
ARG TARGETPLATFORM
|
||||
|
||||
FROM --platform=$BUILDPLATFORM node:22-alpine AS builder
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY gitnexus-shared/package.json gitnexus-shared/package-lock.json ./gitnexus-shared/
|
||||
RUN npm ci --prefix gitnexus-shared
|
||||
|
||||
COPY gitnexus-shared ./gitnexus-shared
|
||||
RUN npm run build --prefix gitnexus-shared
|
||||
|
||||
COPY gitnexus/package.json ./gitnexus/
|
||||
|
||||
COPY gitnexus-web/package.json gitnexus-web/package-lock.json ./gitnexus-web/
|
||||
RUN npm ci --prefix gitnexus-web
|
||||
|
||||
COPY gitnexus-web ./gitnexus-web
|
||||
RUN npm run build --prefix gitnexus-web
|
||||
|
||||
FROM node:22-alpine AS runtime
|
||||
|
||||
RUN apk add --no-cache curl
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=builder /app/gitnexus-web/dist ./dist
|
||||
COPY docker-server.mjs ./docker-server.mjs
|
||||
|
||||
RUN chown -R node:node /app
|
||||
|
||||
USER node
|
||||
|
||||
EXPOSE 4173
|
||||
|
||||
CMD ["node", "docker-server.mjs"]
|
||||
@@ -1,85 +0,0 @@
|
||||
# Guardrails — GitNexus
|
||||
|
||||
Rules for **human contributors** and **AI agents**. Complements `AGENTS.md` (workflows) and `CONTRIBUTING.md` (PR process).
|
||||
|
||||
## Scope (least privilege)
|
||||
|
||||
- **Read:** Source, tests, docs, public config as needed.
|
||||
- **Write:** Only files required for the fix or feature; no unrelated formatting or refactors.
|
||||
- **Execute:** Tests, typecheck, documented CLI commands. No destructive commands on user data without approval.
|
||||
- **Off-limits:** Other people's machines, production deployments you don't own, credentials you lack permission to use.
|
||||
|
||||
Maintainer may widen scope per task.
|
||||
|
||||
---
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
1. **Never commit secrets** — API keys, tokens, real `.env` values, private URLs, session cookies. Use `.env.example` with placeholders.
|
||||
2. **Never rename with find-and-replace** in GitNexus-indexed projects — use `rename` MCP tool with `dry_run: true` first, review `graph` vs `text_search` edits. No separate `gitnexus rename` CLI exists.
|
||||
3. **Run impact analysis before editing shared symbols** — `impact` (upstream) for functions/classes/methods others call. Do not ignore HIGH/CRITICAL without maintainer sign-off.
|
||||
4. **Run `detect_changes` before commit** — confirm diffs map to expected symbols/processes when the graph is available.
|
||||
5. **Preserve embeddings** — plain `npx gitnexus analyze` now preserves any embeddings recorded in `.gitnexus/meta.json` (the previous behavior wiped them). Use `--embeddings` to also generate vectors for new/changed nodes; use `--drop-embeddings` only when an explicit wipe is intended (e.g., model swap).
|
||||
|
||||
---
|
||||
|
||||
## Signs (recurring failure patterns)
|
||||
|
||||
Format: **Trigger → Instruction → Reason**. Append new Signs when the same mistake repeats.
|
||||
|
||||
### Stale graph after edits
|
||||
|
||||
- **Trigger:** MCP warns index is behind `HEAD`, or search doesn't match latest commit.
|
||||
- **Do:** `npx gitnexus analyze` (plus `--embeddings` if used).
|
||||
- **Why:** Tools query LadybugDB from last analyze; git changes are invisible until re-indexed.
|
||||
|
||||
### Embeddings vanished after analyze
|
||||
|
||||
- **Trigger:** Semantic search quality drops; `stats.embeddings` in `meta.json` is 0 after refresh.
|
||||
- **Do:** Re-run `npx gitnexus analyze --embeddings` to regenerate. Check the analyze log for a `Warning: could not load cached embeddings` line — if present, the cache restore failed (corrupt DB / schema mismatch) and the rebuild had nothing to preserve. If you intentionally passed `--drop-embeddings`, this is expected.
|
||||
- **Why:** Plain `analyze` preserves prior vectors by re-inserting them after the rebuild; the only ways to end up at zero are an explicit `--drop-embeddings`, a cache-load failure (now logged), or a model/dimension change that invalidates the cache.
|
||||
|
||||
### MCP lists no repos
|
||||
|
||||
- **Trigger:** MCP stderr says no indexed repos.
|
||||
- **Do:** `npx gitnexus analyze` in the target repo; verify `npx gitnexus list` shows it.
|
||||
- **Why:** MCP discovers repos via `~/.gitnexus/registry.json`, populated by analyze.
|
||||
|
||||
### Wrong repo in multi-repo setups
|
||||
|
||||
- **Trigger:** Query/impact results belong to another project.
|
||||
- **Do:** Call `list_repos`, then pass `repo` on subsequent tools.
|
||||
- **Why:** Default target is ambiguous when multiple repos are registered.
|
||||
|
||||
### LadybugDB lock / "database busy"
|
||||
|
||||
- **Trigger:** Errors opening `.gitnexus/lbug` while MCP and analyze both run.
|
||||
- **Do:** Stop overlapping processes (one writer at a time). Retry analyze or restart MCP.
|
||||
- **Why:** Embedded DB expects single-process ownership.
|
||||
|
||||
---
|
||||
|
||||
## Publishing & supply chain
|
||||
|
||||
- **npm:** Do not publish from unreviewed automation. Bump version intentionally; tag releases to match `package.json`.
|
||||
- **Dependencies:** Minimal, auditable `package.json` changes; run tests and CI after lockfile updates.
|
||||
- **License:** PolyForm Noncommercial 1.0.0 — do not relicense without maintainer approval.
|
||||
|
||||
---
|
||||
|
||||
## Escalation
|
||||
|
||||
Stop and ask a **human maintainer** when:
|
||||
|
||||
- Impact analysis shows HIGH/CRITICAL risk and the task still requires the change.
|
||||
- You need to alter CI, release, or security-sensitive config.
|
||||
- Requirements conflict (e.g. "speed up analyze" vs "must keep all embeddings on huge repo").
|
||||
- You are unsure whether data loss is acceptable (`clean`, forced migrations, schema changes).
|
||||
|
||||
---
|
||||
|
||||
## Related docs
|
||||
|
||||
- [ARCHITECTURE.md](ARCHITECTURE.md) — components and data flow
|
||||
- [RUNBOOK.md](RUNBOOK.md) — commands for recovery
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) — PR and commit expectations
|
||||
@@ -1,71 +0,0 @@
|
||||
# Migration Guide
|
||||
|
||||
## `impact` tool may now return `{ status: 'ambiguous' }` (PR #888, issue #470)
|
||||
|
||||
Before this change the `impact` MCP tool silently picked the first match
|
||||
when the `target` name hit multiple symbols (Class → Interface → Function
|
||||
→ Method → Constructor priority UNION). This often produced analysis for
|
||||
the wrong symbol with no signal back to the caller.
|
||||
|
||||
After this change, when the resolver finds more than one viable match
|
||||
and the caller supplied none of `target_uid` / `file_path` / `kind`,
|
||||
`impact` returns a disambiguation response shaped like:
|
||||
|
||||
```json
|
||||
{
|
||||
"status": "ambiguous",
|
||||
"message": "Found N symbols matching '<target>'. Use target_uid, file_path, or kind to disambiguate.",
|
||||
"target": { "name": "<target>" },
|
||||
"direction": "upstream",
|
||||
"impactedCount": 0,
|
||||
"risk": "UNKNOWN",
|
||||
"candidates": [
|
||||
{ "uid": "...", "name": "...", "kind": "Function", "filePath": "...", "line": 42, "score": 0.76 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
### Do I need to migrate?
|
||||
|
||||
**Probably not, but check for assumptions.** Callers that unconditionally
|
||||
read `result.byDepth` / `result.summary` / `result.affected_processes`
|
||||
without first checking `result.status` will now see `undefined` in the
|
||||
ambiguous case. The fix is to branch on `result.status === 'ambiguous'`
|
||||
first and follow up with `target_uid` (preferred) or `file_path` / `kind`.
|
||||
|
||||
The `context` tool's ambiguous response is a strict superset of the
|
||||
existing shape — every candidate gains a `score` field, no existing field
|
||||
has changed. No migration required for `context` callers.
|
||||
|
||||
### What happens on re-index?
|
||||
|
||||
Nothing — this is an MCP-surface change only. The graph schema, indexer,
|
||||
and stored data are untouched.
|
||||
|
||||
---
|
||||
|
||||
## OVERRIDES → METHOD_OVERRIDES (PR #642)
|
||||
|
||||
The `OVERRIDES` relationship type has been renamed to `METHOD_OVERRIDES` for
|
||||
consistency with the new `METHOD_IMPLEMENTS` edge type.
|
||||
|
||||
### Do I need to migrate?
|
||||
|
||||
**No.** Backward compatibility is handled automatically at runtime:
|
||||
|
||||
- `local-backend.ts` dual-reads both `OVERRIDES` and `METHOD_OVERRIDES` in all
|
||||
impact-analysis and context queries. Existing stored graphs with `OVERRIDES`
|
||||
edges continue to return correct results without any manual intervention.
|
||||
- The `REL_TYPES` array in `schema-constants.ts` includes both names so Cypher
|
||||
queries that reference either will work.
|
||||
|
||||
### What happens on re-index?
|
||||
|
||||
Running `npx gitnexus analyze` on a repository produces `METHOD_OVERRIDES`
|
||||
edges going forward. The old `OVERRIDES` edges are replaced as part of the
|
||||
normal full re-index.
|
||||
|
||||
### When will the legacy alias be removed?
|
||||
|
||||
The `OVERRIDES` compat alias will remain until a future major version. Removal
|
||||
will be announced in this file and in the changelog before it happens.
|
||||
@@ -1,5 +1,5 @@
|
||||
# GitNexus
|
||||
**⚠️ Important Notice:** GitNexus has NO official cryptocurrency, token, or coin. Any token/coin using the GitNexus name on Pump.fun or any other platform is **not affiliated with, endorsed by, or created by** this project or its maintainers. Do not purchase any cryptocurrency claiming association with GitNexus.
|
||||
⚠️ Important Notice:** GitNexus has NO official cryptocurrency, token, or coin. Any token/coin using the GitNexus name on Pump.fun or any other platform is **not affiliated with, endorsed by, or created by** this project or its maintainers. Do not purchase any cryptocurrency claiming association with GitNexus.
|
||||
|
||||
<div align="center">
|
||||
|
||||
@@ -9,7 +9,7 @@
|
||||
|
||||
<h2>Join the official Discord to discuss ideas, issues etc!</h2>
|
||||
|
||||
<a href="https://discord.gg/MgJrmsqr62">
|
||||
<a href="https://discord.gg/AAsRVT6fGb">
|
||||
<img src="https://img.shields.io/discord/1477255801545429032?color=5865F2&logo=discord&logoColor=white" alt="Discord"/>
|
||||
</a>
|
||||
<a href="https://www.npmjs.com/package/gitnexus">
|
||||
@@ -19,8 +19,6 @@
|
||||
<img src="https://img.shields.io/badge/License-PolyForm%20Noncommercial-blue.svg" alt="License: PolyForm Noncommercial"/>
|
||||
</a>
|
||||
|
||||
<p><strong>Enterprise (SaaS & Self-hosted)</strong> - <a href="https://akonlabs.com">akonlabs.com</a></p>
|
||||
|
||||
</div>
|
||||
|
||||
**Building nervous system for agent context.**
|
||||
@@ -36,7 +34,7 @@ https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
|
||||
|
||||
> *Like DeepWiki, but deeper.* DeepWiki helps you *understand* code. GitNexus lets you *analyze* it — because a knowledge graph tracks every relationship, not just descriptions.
|
||||
|
||||
**TL;DR:** The **Web UI** is a quick way to chat with any repo. The **CLI + MCP** is how you make your AI agent actually reliable — it gives Cursor, Claude Code, Codex, and friends a deep architectural view of your codebase so they stop missing dependencies, breaking call chains, and shipping blind edits. Even smaller models get full architectural clarity, making it compete with Goliath models.
|
||||
**TL;DR:** The **Web UI** is a quick way to chat with any repo. The **CLI + MCP** is how you make your AI agent actually reliable — it gives Cursor, Claude Code, and friends a deep architectural view of your codebase so they stop missing dependencies, breaking call chains, and shipping blind edits. Even smaller models get full architectural clarity, making it compete with goliath models.
|
||||
|
||||
---
|
||||
|
||||
@@ -50,10 +48,10 @@ https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
|
||||
| | **CLI + MCP** | **Web UI** |
|
||||
| ----------------- | -------------------------------------------------------------- | ------------------------------------------------------------ |
|
||||
| **What** | Index repos locally, connect AI agents via MCP | Visual graph explorer + AI chat in browser |
|
||||
| **For** | Daily development with Cursor, Claude Code, Codex, Windsurf, OpenCode | Quick exploration, demos, one-off analysis |
|
||||
| **For** | Daily development with Cursor, Claude Code, Windsurf, OpenCode | Quick exploration, demos, one-off analysis |
|
||||
| **Scale** | Full repos, any size | Limited by browser memory (~5k files), or unlimited via backend mode |
|
||||
| **Install** | `npm install -g gitnexus` | No install — [gitnexus.vercel.app](https://gitnexus.vercel.app) |
|
||||
| **Storage** | LadybugDB native (fast, persistent) | LadybugDB WASM (in-memory, per session) |
|
||||
| **Install** | `npm install -g gitnexus` | No install —[gitnexus.vercel.app](https://gitnexus.vercel.app) |
|
||||
| **Storage** | KuzuDB native (fast, persistent) | KuzuDB WASM (in-memory, per session) |
|
||||
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
|
||||
| **Privacy** | Everything local, no network | Everything in-browser, no server |
|
||||
|
||||
@@ -61,36 +59,6 @@ https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
|
||||
|
||||
---
|
||||
|
||||
## Enterprise
|
||||
|
||||
GitNexus is available as an **enterprise offering** - either as a fully managed **SaaS** or a **self-hosted** deployment. Also available for **commercial use** of the OSS version with proper licensing.
|
||||
|
||||
Enterprise includes:
|
||||
- **PR Review** - automated blast radius analysis on pull requests
|
||||
- **Auto-updating Code Wiki** - always up-to-date documentation (Code Wiki is also available in OSS)
|
||||
- **Auto-reindexing** - knowledge graph stays fresh automatically
|
||||
- **Multi-repo support** - unified graph across repositories
|
||||
- **OCaml support** - additional language coverage
|
||||
- **Priority feature/language support** - request new languages or features
|
||||
|
||||
**Upcoming:**
|
||||
- Auto regression forensics
|
||||
- End-to-end test generation
|
||||
|
||||
👉 Learn more at [akonlabs.com](https://akonlabs.com)
|
||||
|
||||
💬 For commercial licensing or enterprise inquiries, ping us on [Discord](https://discord.gg/AAsRVT6fGb) or drop an email at founders@akonlabs.com
|
||||
|
||||
---
|
||||
|
||||
## Development
|
||||
|
||||
- [ARCHITECTURE.md](ARCHITECTURE.md) — packages, index → graph → MCP flow, where to change code
|
||||
- [RUNBOOK.md](RUNBOOK.md) — analyze, embeddings, stale index, MCP recovery, CI snippets
|
||||
- [GUARDRAILS.md](GUARDRAILS.md) — safety rules and operational “Signs” for contributors and agents
|
||||
- [CONTRIBUTING.md](CONTRIBUTING.md) — license, setup, commits, and pull requests
|
||||
- [TESTING.md](TESTING.md) — test commands for `gitnexus` and `gitnexus-web`
|
||||
|
||||
## CLI + MCP (recommended)
|
||||
|
||||
The CLI indexes your repository and runs an MCP server that gives AI agents deep codebase awareness.
|
||||
@@ -116,39 +84,23 @@ To configure MCP for your editor, run `npx gitnexus setup` once — or set it up
|
||||
| --------------------- | --- | ------ | -------------------- | -------------- |
|
||||
| **Claude Code** | Yes | Yes | Yes (PreToolUse + PostToolUse) | **Full** |
|
||||
| **Cursor** | Yes | Yes | — | MCP + Skills |
|
||||
| **Codex** | Yes | Yes | — | MCP + Skills |
|
||||
| **Windsurf** | Yes | — | — | MCP |
|
||||
| **OpenCode** | Yes | Yes | — | MCP + Skills |
|
||||
|
||||
> **Claude Code** gets the deepest integration: MCP tools + agent skills + PreToolUse hooks that enrich searches with graph context + PostToolUse hooks that detect a stale index after commits and prompt the agent to reindex.
|
||||
> **Claude Code** gets the deepest integration: MCP tools + agent skills + PreToolUse hooks that enrich searches with graph context + PostToolUse hooks that auto-reindex after commits.
|
||||
|
||||
## Community Integrations
|
||||
### Community Integrations
|
||||
|
||||
Built by the community — not officially maintained, but worth checking out.
|
||||
|
||||
| Project | Author | Description |
|
||||
|---------|--------|-------------|
|
||||
| [pi-gitnexus](https://github.com/tintinweb/pi-gitnexus) | [@tintinweb](https://github.com/tintinweb) | GitNexus plugin for [pi](https://pi.dev) — `pi install npm:pi-gitnexus` |
|
||||
| [gitnexus-stable-ops](https://github.com/ShunsukeHayashi/gitnexus-stable-ops) | [@ShunsukeHayashi](https://github.com/ShunsukeHayashi) | Stable ops & deployment workflows (Miyabi ecosystem) |
|
||||
|
||||
> Have a project built on GitNexus? Open a PR to add it here!
|
||||
| Agent | Install | Source |
|
||||
|-------|---------|--------|
|
||||
| [pi](https://pi.dev) | `pi install npm:pi-gitnexus` | [pi-gitnexus](https://github.com/tintinweb/pi-gitnexus) |
|
||||
|
||||
If you prefer manual configuration:
|
||||
|
||||
**Claude Code** (full support — MCP + skills + hooks):
|
||||
|
||||
```bash
|
||||
# macOS / Linux
|
||||
claude mcp add gitnexus -- npx -y gitnexus@latest mcp
|
||||
|
||||
# Windows
|
||||
claude mcp add gitnexus -- cmd /c npx -y gitnexus@latest mcp
|
||||
```
|
||||
|
||||
**Codex** (full support — MCP + skills):
|
||||
|
||||
```bash
|
||||
codex mcp add gitnexus -- npx -y gitnexus@latest mcp
|
||||
```
|
||||
|
||||
**Cursor** (`~/.cursor/mcp.json` — global, works for all projects):
|
||||
@@ -170,34 +122,20 @@ codex mcp add gitnexus -- npx -y gitnexus@latest mcp
|
||||
{
|
||||
"mcp": {
|
||||
"gitnexus": {
|
||||
"type": "local",
|
||||
"command": ["gitnexus", "mcp"]
|
||||
"command": "npx",
|
||||
"args": ["-y", "gitnexus@latest", "mcp"]
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Codex** (`~/.codex/config.toml` for system scope, or `.codex/config.toml` for project scope):
|
||||
|
||||
```toml
|
||||
[mcp_servers.gitnexus]
|
||||
command = "npx"
|
||||
args = ["-y", "gitnexus@latest", "mcp"]
|
||||
```
|
||||
|
||||
### CLI Commands
|
||||
|
||||
```bash
|
||||
gitnexus setup # Configure MCP for your editors (one-time)
|
||||
gitnexus analyze [path] # Index a repository (or update stale index)
|
||||
gitnexus analyze --force # Force full re-index
|
||||
gitnexus analyze --skills # Generate repo-specific skill files from detected communities
|
||||
gitnexus setup # Configure MCP for your editors (one-time)
|
||||
gitnexus analyze [path] # Index a repository (or update stale index)
|
||||
gitnexus analyze --force # Force full re-index
|
||||
gitnexus analyze --skip-embeddings # Skip embedding generation (faster)
|
||||
gitnexus analyze --skip-agents-md # Preserve custom AGENTS.md/CLAUDE.md gitnexus section edits
|
||||
gitnexus analyze --skip-git # Index folders that are not Git repositories
|
||||
gitnexus analyze --embeddings # Enable embedding generation (slower, better search)
|
||||
gitnexus analyze --verbose # Log skipped files when parsers are unavailable
|
||||
gitnexus analyze --worker-timeout 60 # Increase worker idle timeout for slow parses
|
||||
gitnexus mcp # Start MCP server (stdio) — serves all indexed repos
|
||||
gitnexus serve # Start local HTTP server (multi-repo) for web UI connection
|
||||
gitnexus list # List all indexed repositories
|
||||
@@ -207,23 +145,11 @@ gitnexus clean --all --force # Delete all indexes
|
||||
gitnexus wiki [path] # Generate repository wiki from knowledge graph
|
||||
gitnexus wiki --model <model> # Wiki with custom LLM model (default: gpt-4o-mini)
|
||||
gitnexus wiki --base-url <url> # Wiki with custom LLM API base URL
|
||||
|
||||
# Repository groups (multi-repo / monorepo service tracking)
|
||||
gitnexus group create <name> # Create a repository group
|
||||
gitnexus group add <group> <groupPath> <registryName> # Add a repo to a group. <groupPath> is a hierarchy path (e.g. hr/hiring/backend); <registryName> is the repo's name from the registry (see `gitnexus list`)
|
||||
gitnexus group remove <group> <groupPath> # Remove a repo from a group by its hierarchy path
|
||||
gitnexus group list [name] # List groups, or show one group's config
|
||||
gitnexus group sync <name> # Extract contracts and match across repos/services
|
||||
gitnexus group contracts <name> # Inspect extracted contracts and cross-links
|
||||
gitnexus group query <name> <q> # Search execution flows across all repos in a group
|
||||
gitnexus group status <name> # Check staleness of repos in a group
|
||||
```
|
||||
|
||||
If `analyze` reports a worker parse timeout on a large or unusual repository, it keeps running and falls back safely. To give slow worker jobs more time, use `gitnexus analyze --worker-timeout 60` or set `GITNEXUS_WORKER_SUB_BATCH_TIMEOUT_MS=60000`. For very large files, `GITNEXUS_WORKER_SUB_BATCH_MAX_BYTES` controls the worker job byte budget.
|
||||
|
||||
### What Your AI Agent Gets
|
||||
|
||||
**16 tools** exposed via MCP (11 per-repo + 5 group):
|
||||
**7 tools** exposed via MCP:
|
||||
|
||||
| Tool | What It Does | `repo` Param |
|
||||
| ------------------ | ----------------------------------------------------------------- | -------------- |
|
||||
@@ -234,11 +160,6 @@ If `analyze` reports a worker parse timeout on a large or unusual repository, it
|
||||
| `detect_changes` | Git-diff impact — maps changed lines to affected processes | Optional |
|
||||
| `rename` | Multi-file coordinated rename with graph + text search | Optional |
|
||||
| `cypher` | Raw Cypher graph queries | Optional |
|
||||
| `group_list` | List configured repository groups | — |
|
||||
| `group_sync` | Extract contracts and match across repos/services | — |
|
||||
| `group_contracts`| Inspect extracted contracts and cross-links | — |
|
||||
| `group_query` | Search execution flows across all repos in a group | — |
|
||||
| `group_status` | Check staleness of repos in a group | — |
|
||||
|
||||
> When only one repo is indexed, the `repo` parameter is optional. With multiple repos, specify which one: `query({query: "auth", repo: "my-app"})`.
|
||||
|
||||
@@ -268,10 +189,6 @@ If `analyze` reports a worker parse timeout on a large or unusual repository, it
|
||||
- **Impact Analysis** — Analyze blast radius before changes
|
||||
- **Refactoring** — Plan safe refactors using dependency mapping
|
||||
|
||||
**Repo-specific skills** generated with `--skills`:
|
||||
|
||||
When you run `gitnexus analyze --skills`, GitNexus detects the functional areas of your codebase (via Leiden community detection) and generates a `SKILL.md` file for each one under `.claude/skills/generated/`. Each skill describes a module's key files, entry points, execution flows, and cross-area connections — so your AI agent gets targeted context for the exact area of code you're working in. Skills are regenerated on each `--skills` run to stay current with the codebase.
|
||||
|
||||
---
|
||||
|
||||
## Multi-Repo MCP Architecture
|
||||
@@ -300,8 +217,8 @@ flowchart TD
|
||||
Server["server.ts"]
|
||||
Backend["LocalBackend"]
|
||||
Pool["Connection Pool"]
|
||||
ConnA["LadybugDB conn A"]
|
||||
ConnB["LadybugDB conn B"]
|
||||
ConnA["KuzuDB conn A"]
|
||||
ConnB["KuzuDB conn B"]
|
||||
end
|
||||
|
||||
Setup -->|"writes global MCP config"| CursorConfig["~/.cursor/mcp.json"]
|
||||
@@ -318,189 +235,28 @@ flowchart TD
|
||||
ConnB -->|"queries"| RepoB
|
||||
```
|
||||
|
||||
**How it works:** Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo (portable, gitignored) and registers a pointer in `~/.gitnexus/registry.json`. When an AI agent starts, the MCP server reads the registry and can serve any indexed repo. LadybugDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). If only one repo is indexed, the `repo` parameter is optional on all tools — agents don't need to change anything.
|
||||
**How it works:** Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo (portable, gitignored) and registers a pointer in `~/.gitnexus/registry.json`. When an AI agent starts, the MCP server reads the registry and can serve any indexed repo. KuzuDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). If only one repo is indexed, the `repo` parameter is optional on all tools — agents don't need to change anything.
|
||||
|
||||
---
|
||||
|
||||
## Web UI (browser-based)
|
||||
|
||||
A client-side graph explorer and AI chat — your code never leaves your machine.
|
||||
A fully client-side graph explorer and AI chat. No server, no install — your code never leaves the browser.
|
||||
|
||||
**Try it now:** [gitnexus.vercel.app](https://gitnexus.vercel.app) — run `npx gitnexus@latest serve` locally and the page auto-connects to your local backend.
|
||||
**Try it now:** [gitnexus.vercel.app](https://gitnexus.vercel.app) — drag & drop a ZIP and start exploring.
|
||||
|
||||
<img width="2550" height="1343" alt="gitnexus_img" src="https://github.com/user-attachments/assets/cc5d637d-e0e5-48e6-93ff-5bcfdb929285" />
|
||||
|
||||
Or run the frontend locally:
|
||||
Or run locally:
|
||||
|
||||
```bash
|
||||
git clone https://github.com/abhigyanpatwari/gitnexus.git
|
||||
cd gitnexus/gitnexus-shared && npm install && npm run build
|
||||
cd ../gitnexus-web && npm install
|
||||
cd gitnexus/gitnexus-web
|
||||
npm install
|
||||
npm run dev
|
||||
# Then in another terminal, start the backend the frontend connects to:
|
||||
npx gitnexus@latest serve
|
||||
```
|
||||
|
||||
## Docker
|
||||
|
||||
The official Docker setup ships **two signed images** orchestrated by `docker-compose.yaml`. Each image is published to both **GitHub Container Registry** (GHCR) and **Docker Hub** — same build, same digest, same Cosign signature — so pick whichever registry you prefer:
|
||||
|
||||
| Purpose | GHCR (default in `docker-compose.yaml`) | Docker Hub mirror |
|
||||
| ---------------------------------------------------------------------- | --------------------------------------------- | ------------------------------------------- |
|
||||
| CLI / `gitnexus serve` backend (HTTP API on port `4747`, MCP, indexer) | `ghcr.io/abhigyanpatwari/gitnexus:latest` | `akonlabs/gitnexus:latest` |
|
||||
| Static web UI (port `4173`) | `ghcr.io/abhigyanpatwari/gitnexus-web:latest` | `akonlabs/gitnexus-web:latest` |
|
||||
|
||||
> **Heads-up — image rename.** Earlier releases published the web UI under
|
||||
> `ghcr.io/abhigyanpatwari/gitnexus`. Starting with the introduction of the
|
||||
> bundled backend, that slug now hosts the CLI/server image and the UI moved
|
||||
> to `ghcr.io/abhigyanpatwari/gitnexus-web`. The previous tags remain
|
||||
> available for pulling, but new versions are only published under the new
|
||||
> slugs. Update your `docker run` / compose files accordingly (or just adopt
|
||||
> the bundled compose).
|
||||
|
||||
### One-command setup
|
||||
|
||||
```bash
|
||||
docker compose up -d
|
||||
```
|
||||
|
||||
This starts the server on `http://localhost:4747` and the web UI on
|
||||
`http://localhost:4173`. The UI auto-detects the server because the browser
|
||||
runs on the host and reaches the container via the mapped port.
|
||||
|
||||
A named volume (`gitnexus-data`) persists the global registry, indexes, and
|
||||
cloned repos at `/data/gitnexus` inside the server container. To make repos on
|
||||
your host machine indexable, set `WORKSPACE_DIR` before bringing the stack up:
|
||||
|
||||
```bash
|
||||
WORKSPACE_DIR=$HOME/code docker compose up -d
|
||||
# Inside the server container the directory is mounted read-only at /workspace.
|
||||
docker compose exec gitnexus-server gitnexus index /workspace/my-repo
|
||||
```
|
||||
|
||||
### Direct `docker run`
|
||||
|
||||
```bash
|
||||
# Server
|
||||
docker run --rm -d \
|
||||
--name gitnexus-server \
|
||||
-p 4747:4747 \
|
||||
-v gitnexus-data:/data/gitnexus \
|
||||
ghcr.io/abhigyanpatwari/gitnexus:latest
|
||||
|
||||
# Web UI
|
||||
docker run --rm -d \
|
||||
--name gitnexus-web \
|
||||
-p 4173:4173 \
|
||||
ghcr.io/abhigyanpatwari/gitnexus-web:latest
|
||||
```
|
||||
|
||||
Optional env file (override image tags, container names, ports, workspace dir):
|
||||
|
||||
```bash
|
||||
cp .env.example .env
|
||||
docker compose --env-file .env up -d
|
||||
```
|
||||
|
||||
### Versioning & supply-chain protection
|
||||
|
||||
The Docker images are version-locked to the npm package:
|
||||
|
||||
- Stable images are **only published from `vX.Y.Z` git tags** (via `docker.yml`
|
||||
triggered directly by the tag push), and the workflow refuses to build unless
|
||||
the tag exactly matches `gitnexus/package.json`'s version. So
|
||||
`ghcr.io/abhigyanpatwari/gitnexus:1.6.2` (and its Docker Hub mirror
|
||||
`akonlabs/gitnexus:1.6.2`) is byte-for-byte the same release as
|
||||
`npm install gitnexus@1.6.2` — no drift, no floating builds from `main`.
|
||||
Both registries receive the same digest from a single build step, so you can
|
||||
pull from either and the signature verifies identically.
|
||||
- Release-candidate images (e.g. `:1.7.0-rc.1`) are published alongside each
|
||||
RC npm release. They are built by `release-candidate.yml` calling `docker.yml`
|
||||
as a reusable workflow after the RC tag is created and pushed.
|
||||
- `:latest` is auto-promoted only from non-prerelease tags by the Docker
|
||||
metadata action, so it always points at a real, npm-published version.
|
||||
|
||||
Both images are signed with [Cosign keyless signing][cosign-keyless] using the
|
||||
workflow's GitHub OIDC identity, and shipped with build provenance and SBOM
|
||||
attestations. **This is your protection against supply-chain attacks**: even if
|
||||
an attacker republishes a same-named image elsewhere (or somehow pushes to a
|
||||
typo-squatted registry), they cannot forge a Cosign signature tied to
|
||||
`abhigyanpatwari/GitNexus`'s `docker.yml`. Always verify before pulling into
|
||||
sensitive environments:
|
||||
|
||||
**Stable releases** — signed from the `v*` tag ref:
|
||||
|
||||
```bash
|
||||
cosign verify ghcr.io/abhigyanpatwari/gitnexus:1.6.2 \
|
||||
--certificate-identity-regexp '^https://github\.com/abhigyanpatwari/GitNexus/\.github/workflows/docker\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$' \
|
||||
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
||||
|
||||
# Same signature verifies the Docker Hub mirror (identical digest):
|
||||
cosign verify docker.io/akonlabs/gitnexus:1.6.2 \
|
||||
--certificate-identity-regexp '^https://github\.com/abhigyanpatwari/GitNexus/\.github/workflows/docker\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$' \
|
||||
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
||||
```
|
||||
|
||||
The regex pins the certificate identity to this repo's `docker.yml` workflow
|
||||
**run from a `v*` tag** — rejecting unsigned images, images signed by other
|
||||
workflows, and images signed from unprotected refs. It is identical for both
|
||||
registries because both sets of tags were signed at the same digest in one
|
||||
workflow run.
|
||||
|
||||
**Release candidates** — signed from `refs/heads/main` (the caller's ref when
|
||||
`release-candidate.yml` invokes `docker.yml` as a reusable workflow):
|
||||
|
||||
```bash
|
||||
cosign verify ghcr.io/abhigyanpatwari/gitnexus:1.7.0-rc.1 \
|
||||
--certificate-identity 'https://github.com/abhigyanpatwari/GitNexus/.github/workflows/docker.yml@refs/heads/main' \
|
||||
--certificate-oidc-issuer https://token.actions.githubusercontent.com
|
||||
```
|
||||
|
||||
You can also inspect the build provenance and SBOM:
|
||||
|
||||
```bash
|
||||
cosign download attestation ghcr.io/abhigyanpatwari/gitnexus:1.6.2 \
|
||||
--predicate-type https://slsa.dev/provenance/v1
|
||||
```
|
||||
|
||||
#### Kubernetes: enforce signatures at admission
|
||||
|
||||
For Kubernetes deployments, ship the bundled
|
||||
[`ClusterImagePolicy`](deploy/kubernetes/cluster-image-policy.yaml) so the
|
||||
[Sigstore policy-controller][policy-controller] rejects any GitNexus pod whose
|
||||
image is not signed by this repo's `docker.yml` running from a `vX.Y.Z` tag —
|
||||
the same identity the `cosign verify` snippet above pins.
|
||||
|
||||
```bash
|
||||
# 1. Install the controller (one-time, cluster-wide)
|
||||
helm repo add sigstore https://sigstore.github.io/helm-charts && helm repo update
|
||||
helm install policy-controller -n cosign-system --create-namespace \
|
||||
sigstore/policy-controller
|
||||
|
||||
# 2. Opt your namespace in
|
||||
kubectl label namespace <your-ns> policy.sigstore.dev/include=true
|
||||
|
||||
# 3. Apply the policy
|
||||
kubectl apply -f deploy/kubernetes/cluster-image-policy.yaml
|
||||
```
|
||||
|
||||
After this, attempting to deploy an unsigned image — or one signed by anything
|
||||
other than `abhigyanpatwari/GitNexus`'s `docker.yml` at a `v*` tag — fails the
|
||||
admission webhook before a pod is ever created. This turns the verifiable
|
||||
signature into an enforced policy, which is the supply-chain control most
|
||||
clusters actually need.
|
||||
|
||||
[cosign-keyless]: https://docs.sigstore.dev/cosign/signing/overview/
|
||||
[policy-controller]: https://docs.sigstore.dev/policy-controller/overview/
|
||||
|
||||
### Files
|
||||
|
||||
- [Dockerfile.web](Dockerfile.web) — builds `gitnexus-shared` and `gitnexus-web`, then serves the production frontend.
|
||||
- [Dockerfile.cli](Dockerfile.cli) — builds the CLI/server (with its native deps) and runs `gitnexus serve --host 0.0.0.0`.
|
||||
- [docker-compose.yaml](docker-compose.yaml) — starts both signed images side by side.
|
||||
- [.env.example](.env.example) — overrides for image names, container names, ports, and the workspace mount.
|
||||
|
||||
The web UI uses the same indexing pipeline as the CLI but runs entirely in WebAssembly (Tree-sitter WASM, LadybugDB WASM, in-browser embeddings). It's great for quick exploration but limited by browser memory for larger repos.
|
||||
The web UI uses the same indexing pipeline as the CLI but runs entirely in WebAssembly (Tree-sitter WASM, KuzuDB WASM, in-browser embeddings). It's great for quick exploration but limited by browser memory for larger repos.
|
||||
|
||||
**Local Backend Mode:** Run `gitnexus serve` and open the web UI locally — it auto-detects the server and shows all your indexed repos, with full AI chat support. No need to re-upload or re-index. The agent's tools (Cypher queries, search, code navigation) route through the backend HTTP API automatically.
|
||||
|
||||
@@ -508,7 +264,7 @@ The web UI uses the same indexing pipeline as the CLI but runs entirely in WebAs
|
||||
|
||||
## The Problem GitNexus Solves
|
||||
|
||||
Tools like **Cursor**, **Claude Code**, **Codex**, **Cline**, **Roo Code**, and **Windsurf** are powerful — but they don't truly know your codebase structure.
|
||||
Tools like **Cursor**, **Claude Code**, **Cline**, **Roo Code**, and **Windsurf** are powerful — but they don't truly know your codebase structure.
|
||||
|
||||
**What happens:**
|
||||
|
||||
@@ -557,31 +313,14 @@ GitNexus builds a complete knowledge graph of your codebase through a multi-phas
|
||||
|
||||
1. **Structure** — Walks the file tree and maps folder/file relationships
|
||||
2. **Parsing** — Extracts functions, classes, methods, and interfaces using Tree-sitter ASTs
|
||||
3. **Resolution** — Resolves imports, function calls, heritage, constructor inference, and `self`/`this` receiver types across files with language-aware logic
|
||||
3. **Resolution** — Resolves imports and function calls across files with language-aware logic
|
||||
4. **Clustering** — Groups related symbols into functional communities
|
||||
5. **Processes** — Traces execution flows from entry points through call chains
|
||||
6. **Search** — Builds hybrid search indexes for fast retrieval
|
||||
|
||||
### Supported Languages
|
||||
|
||||
| Language | Imports | Named Bindings | Exports | Heritage | Type Annotations | Constructor Inference | Config | Frameworks | Entry Points |
|
||||
|----------|---------|----------------|---------|----------|-----------------|---------------------|--------|------------|-------------|
|
||||
| TypeScript | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| JavaScript | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ |
|
||||
| Python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| Java | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
|
||||
| Kotlin | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
|
||||
| C# | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| Go | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| Rust | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
|
||||
| PHP | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| Ruby | ✓ | — | ✓ | ✓ | — | ✓ | — | ✓ | ✓ |
|
||||
| Swift | — | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
|
||||
| C | — | — | ✓ | — | ✓ | ✓ | — | ✓ | ✓ |
|
||||
| C++ | — | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
|
||||
| Dart | ✓ | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
|
||||
|
||||
**Imports** — cross-file import resolution · **Named Bindings** — `import { X as Y }` / re-export tracking · **Exports** — public/exported symbol detection · **Heritage** — class inheritance, interfaces, mixins · **Type Annotations** — explicit type extraction for receiver resolution · **Constructor Inference** — infer receiver type from constructor calls (`self`/`this` resolution included for all languages) · **Config** — language toolchain config parsing (tsconfig, go.mod, etc.) · **Frameworks** — AST-based framework pattern detection · **Entry Points** — entry point scoring heuristics
|
||||
TypeScript, JavaScript, Python, Java, Kotlin, C, C++, C#, Go, Rust, PHP, Swift
|
||||
|
||||
---
|
||||
|
||||
@@ -720,7 +459,7 @@ The wiki generator reads the indexed graph structure, groups files into modules
|
||||
| ------------------------- | ------------------------------------- | --------------------------------------- |
|
||||
| **Runtime** | Node.js (native) | Browser (WASM) |
|
||||
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
|
||||
| **Database** | LadybugDB native | LadybugDB WASM |
|
||||
| **Database** | KuzuDB native | KuzuDB WASM |
|
||||
| **Embeddings** | HuggingFace transformers.js (GPU/CPU) | transformers.js (WebGPU/WASM) |
|
||||
| **Search** | BM25 + semantic + RRF | BM25 + semantic + RRF |
|
||||
| **Agent Interface** | MCP (stdio) | LangChain ReAct agent |
|
||||
@@ -741,10 +480,9 @@ The wiki generator reads the indexed graph structure, groups files into modules
|
||||
|
||||
### Recently Completed
|
||||
|
||||
- [X] Constructor-Inferred Type Resolution, `self`/`this` Receiver Mapping
|
||||
- [X] Wiki Generation, Multi-File Rename, Git-Diff Impact Analysis
|
||||
- [X] Process-Grouped Search, 360-Degree Context, Claude Code Hooks
|
||||
- [X] Multi-Repo MCP, Zero-Config Setup, 14 Language Support
|
||||
- [X] Multi-Repo MCP, Zero-Config Setup, 11 Language Support
|
||||
- [X] Community Detection, Process Detection, Confidence Scoring
|
||||
- [X] Hybrid Search, Vector Index
|
||||
|
||||
@@ -761,7 +499,7 @@ The wiki generator reads the indexed graph structure, groups files into modules
|
||||
## Acknowledgments
|
||||
|
||||
- [Tree-sitter](https://tree-sitter.github.io/) — AST parsing
|
||||
- [LadybugDB](https://ladybugdb.com/) — Embedded graph database with vector support (formerly KuzuDB)
|
||||
- [KuzuDB](https://kuzudb.com/) — Embedded graph database with vector support
|
||||
- [Sigma.js](https://www.sigmajs.org/) — WebGL graph rendering
|
||||
- [transformers.js](https://huggingface.co/docs/transformers.js) — Browser ML
|
||||
- [Graphology](https://graphology.github.io/) — Graph data structures
|
||||
|
||||
-163
@@ -1,163 +0,0 @@
|
||||
# Runbook — GitNexus
|
||||
|
||||
Short, copy-paste operations for **local development**, **MCP**, and **CI**. Commands assume a Unix shell; on Windows use Git Bash or equivalent paths.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Node.js** ≥ 20 (`gitnexus-web/package.json` `engines`).
|
||||
- **Git** (analyze requires a git repository).
|
||||
- From repo root, install and build the CLI package:
|
||||
|
||||
```bash
|
||||
cd gitnexus
|
||||
npm install
|
||||
npm run build
|
||||
```
|
||||
|
||||
Use `npx gitnexus …` from any path after global/published install, or `node dist/cli/index.js …` when developing from `gitnexus/` with a local build.
|
||||
|
||||
---
|
||||
|
||||
## Index out of date / “stale” tools
|
||||
|
||||
**Symptom:** MCP or resources warn the index is behind `HEAD`, or results don’t reflect recent commits.
|
||||
|
||||
**Fix (from the target repo root):**
|
||||
|
||||
```bash
|
||||
npx gitnexus analyze
|
||||
```
|
||||
|
||||
**Force full rebuild** (same commit but suspect corruption or changed ignore rules):
|
||||
|
||||
```bash
|
||||
npx gitnexus analyze --force
|
||||
```
|
||||
|
||||
**Check status:**
|
||||
|
||||
```bash
|
||||
npx gitnexus status
|
||||
```
|
||||
|
||||
**List what MCP knows about:**
|
||||
|
||||
```bash
|
||||
npx gitnexus list
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Embeddings
|
||||
|
||||
**First time with vectors** (slower, more disk/RAM):
|
||||
|
||||
```bash
|
||||
npx gitnexus analyze --embeddings
|
||||
```
|
||||
|
||||
**Important:** If you already had embeddings, **always** pass `--embeddings` on later analyzes, or they can be dropped. See `stats.embeddings` in `.gitnexus/meta.json` (0 means none).
|
||||
|
||||
**Large repos:** Analyze may skip or limit embedding work when node counts are very high; watch CLI output.
|
||||
|
||||
---
|
||||
|
||||
## MCP: no repos / empty tools
|
||||
|
||||
**Symptom:** `GitNexus: No indexed repos yet` on stderr when starting MCP.
|
||||
|
||||
**Fix:** In each project you want indexed:
|
||||
|
||||
```bash
|
||||
cd /path/to/repo
|
||||
npx gitnexus analyze
|
||||
```
|
||||
|
||||
Restart the editor MCP session if needed. The server **refreshes the registry lazily**; new analyzes are picked up without necessarily reinstalling MCP.
|
||||
|
||||
**Symptom:** Wrong repo when multiple are indexed — pass `repo` on tools or use `list_repos` first.
|
||||
|
||||
---
|
||||
|
||||
## Clean slate (corrupt or huge `.gitnexus`)
|
||||
|
||||
**Current repo only** (prompts for confirmation):
|
||||
|
||||
```bash
|
||||
npx gitnexus clean
|
||||
```
|
||||
|
||||
**Skip confirmation:**
|
||||
|
||||
```bash
|
||||
npx gitnexus clean --force
|
||||
```
|
||||
|
||||
**All registered repos:**
|
||||
|
||||
```bash
|
||||
npx gitnexus clean --all --force
|
||||
```
|
||||
|
||||
Then re-run `npx gitnexus analyze` (and `--embeddings` if you need vectors).
|
||||
|
||||
---
|
||||
|
||||
## Local bridge for the web UI
|
||||
|
||||
```bash
|
||||
cd gitnexus
|
||||
npx gitnexus serve
|
||||
# default http://127.0.0.1:4747 — see serve --help for port/host
|
||||
```
|
||||
|
||||
Use when the browser UI should talk to **local** indexed repos instead of WASM-only mode.
|
||||
|
||||
---
|
||||
|
||||
## CLI equivalents of MCP tools
|
||||
|
||||
Useful for debugging without an editor:
|
||||
|
||||
```bash
|
||||
cd gitnexus
|
||||
npx gitnexus query "authentication flow" --repo MyRepo
|
||||
npx gitnexus context SomeSymbol --repo MyRepo
|
||||
npx gitnexus impact SomeSymbol --direction upstream --repo MyRepo
|
||||
npx gitnexus cypher "MATCH (n) RETURN count(n) LIMIT 1" --repo MyRepo
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## CI failures (contributors)
|
||||
|
||||
Orchestrator: `.github/workflows/ci.yml`.
|
||||
|
||||
| Job | Typical local repro |
|
||||
|-----|---------------------|
|
||||
| **quality** | `cd gitnexus && npx tsc --noEmit` |
|
||||
| **unit-tests** | `cd gitnexus && npx vitest run test/unit` |
|
||||
| **integration** | `cd gitnexus && npx vitest run test/integration` (see workflow matrix for groups) |
|
||||
| **e2e** | Triggered when `gitnexus-web/` changes; `cd gitnexus-web && E2E=1 npx playwright test` (requires `gitnexus serve` + `npm run dev`) |
|
||||
|
||||
**Note:** Pushes that touch only certain markdown paths may be skipped by `paths-ignore` in CI — see workflow file for exact patterns.
|
||||
|
||||
---
|
||||
|
||||
## Memory / analyze crashes
|
||||
|
||||
Analyze re-execs Node with a **large old-space heap** when needed (`analyze.ts`). If you still OOM on huge repos, close other processes, avoid `--embeddings` for a first pass, or analyze a smaller path if supported by your workflow.
|
||||
|
||||
---
|
||||
|
||||
## LadybugDB / lock errors
|
||||
|
||||
Only one process should open a repo’s `.gitnexus/lbug` store at a time. If MCP and a second `analyze` run conflict, stop one process, then retry `analyze` or restart MCP.
|
||||
|
||||
---
|
||||
|
||||
## Where to dig deeper
|
||||
|
||||
- Architecture overview: [ARCHITECTURE.md](ARCHITECTURE.md)
|
||||
- Agent safety rules: [GUARDRAILS.md](GUARDRAILS.md)
|
||||
- Tests: [TESTING.md](TESTING.md)
|
||||
-98
@@ -1,98 +0,0 @@
|
||||
# Testing — GitNexus
|
||||
|
||||
How we structure tests and which commands to run locally and in CI.
|
||||
|
||||
## Packages
|
||||
|
||||
| Package | Path | Runner | Notes |
|
||||
| -------------- | -------------- | -------- | ------------------------------ |
|
||||
| CLI + MCP core | `gitnexus/` | Vitest | Primary test surface in CI |
|
||||
| Web UI | `gitnexus-web/`| Vitest | Unit/component tests |
|
||||
| Web UI E2E | `gitnexus-web/`| Playwright | Run when changing UI flows |
|
||||
|
||||
## Commands (local)
|
||||
|
||||
From repository root, unless noted:
|
||||
|
||||
**`gitnexus` (CLI / library)**
|
||||
|
||||
```bash
|
||||
cd gitnexus
|
||||
npm install
|
||||
npm run build
|
||||
npm test # full suite: vitest run
|
||||
npm run test:unit # unit only: vitest run test/unit
|
||||
npm run test:integration # integration suite
|
||||
npm run test:coverage
|
||||
npx tsc --noEmit # typecheck (matches CI)
|
||||
```
|
||||
|
||||
**`gitnexus-web`**
|
||||
|
||||
```bash
|
||||
cd gitnexus-web
|
||||
npm install
|
||||
npm test # unit tests (vitest)
|
||||
npx tsc -b --noEmit # typecheck (matches CI)
|
||||
npm run test:coverage
|
||||
npm run test:e2e # Playwright (requires gitnexus serve + npm run dev)
|
||||
```
|
||||
|
||||
## Pre-commit hook
|
||||
|
||||
A husky pre-commit hook (`.husky/pre-commit`) runs automatically on every `git commit`:
|
||||
|
||||
1. **Formatting** — `lint-staged` runs prettier on staged files
|
||||
2. **`gitnexus-web/` files staged** → `tsc -b --noEmit`
|
||||
3. **`gitnexus/` files staged** → `tsc --noEmit`
|
||||
|
||||
Tests do **not** run in the pre-commit hook — they run in CI (`ci-tests.yml`) only.
|
||||
|
||||
Skip with `git commit --no-verify` (use sparingly).
|
||||
|
||||
## Test categories
|
||||
|
||||
- **Unit** — Pure logic, parsers, graph/query helpers; fast; no network.
|
||||
- **Integration** — Real combinations (filesystem, MCP wiring, larger pipelines) as already organized under `gitnexus/test/integration`.
|
||||
- **Eval-style / golden sets** — For agent- or classification-style behavior, keep labeled inputs and expected outputs (JSON or table-driven tests) and run them in CI when relevant.
|
||||
- **E2E (web)** — Critical user paths only; prefer `data-testid` attributes for stable selectors. Tests run against real backend (`gitnexus serve`) and Vite dev server.
|
||||
|
||||
## Performance metrics (targets)
|
||||
|
||||
Set targets to match team expectations, then tune to this repo’s CI reality:
|
||||
|
||||
| Metric | Target (initial) | Notes |
|
||||
| ------------------- | ---------------- | ------------------------------------------ |
|
||||
| Unit coverage | Align with CI | CI runs Vitest with coverage in `gitnexus` |
|
||||
| Unit wall time | Fast PR feedback | Use `vitest run test/unit` for tight loop |
|
||||
| Integration duration| < few minutes | Guard heavy tests with env flags if needed |
|
||||
|
||||
## Regression testing
|
||||
|
||||
Re-run the full relevant suite when:
|
||||
|
||||
- Prompt or agent-behavior documentation changes (if tests encode behavior)
|
||||
- Model or embedding-related code paths change
|
||||
- Graph schema, query contracts, or MCP tool shapes change
|
||||
- Dependencies with parsing or runtime impact upgrade
|
||||
|
||||
## CI integration
|
||||
|
||||
GitHub Actions (`.github/workflows/ci.yml`) orchestrate:
|
||||
|
||||
- **`ci-quality.yml`** — prettier format check, eslint lint, `tsc --noEmit` for `gitnexus/`, `tsc -b --noEmit` for `gitnexus-web/`
|
||||
- **`ci-tests.yml`** — `vitest run` with coverage (ubuntu) + cross-platform (macOS, Windows)
|
||||
- **`ci-e2e.yml`** — Playwright E2E tests, gated on `gitnexus-web/**` changes
|
||||
|
||||
Local checks before pushing:
|
||||
|
||||
```bash
|
||||
cd gitnexus && npx tsc --noEmit && npm test
|
||||
cd ../gitnexus-web && npx tsc -b --noEmit && npm test
|
||||
```
|
||||
|
||||
Or rely on the pre-commit hook which runs these automatically for staged files.
|
||||
|
||||
## User acceptance / beta (optional)
|
||||
|
||||
For staged releases or UI betas: deploy to a staging environment, collect structured feedback, watch errors and latency, then iterate before a wider release.
|
||||
@@ -1,57 +0,0 @@
|
||||
---
|
||||
review_agents: [kieran-typescript-reviewer, pattern-recognition-specialist, architecture-strategist, data-integrity-guardian, security-sentinel, performance-oracle, code-simplicity-reviewer]
|
||||
plan_review_agents: [kieran-typescript-reviewer, architecture-strategist, code-simplicity-reviewer]
|
||||
voltagent_agents: [voltagent-lang:typescript-pro, voltagent-qa-sec:security-auditor, voltagent-data-ai:database-optimizer]
|
||||
---
|
||||
|
||||
# Review Context
|
||||
|
||||
## Project Overview
|
||||
GitNexus is a code intelligence tool that builds a knowledge graph from source code using tree-sitter AST parsing across 12 languages and KuzuDB for graph storage. Two packages: `gitnexus/` (CLI/MCP, TypeScript) and `gitnexus-web/` (browser).
|
||||
|
||||
## Cross-Language Pattern Consistency (pattern-recognition-specialist)
|
||||
- 12 language-specific type extractors in `gitnexus/src/core/ingestion/type-extractors/` must follow identical patterns for: async unwrapping, constructor binding, namespace handling, nullable type stripping, for-loop element typing.
|
||||
- Past bugs: C#/Rust missing `await_expression` unwrapping that TypeScript handled correctly; PHP backslash namespace splitting inconsistent with other languages' `::` / `.` splitting.
|
||||
- When reviewing type extractor changes, verify the same pattern exists in ALL applicable language files — asymmetry is the #1 source of bugs.
|
||||
|
||||
## Data Integrity (data-integrity-guardian)
|
||||
- KuzuDB graph operations: schema in `gitnexus/src/core/kuzu/schema.ts`, adapter in `kuzu-adapter.ts`.
|
||||
- The ingestion pipeline writes symbols and relationships to the graph — changes to node/relation schemas or the ingestion pipeline can corrupt the index.
|
||||
- Known issue: KuzuDB `close()` hangs on Linux due to C++ destructor — use `detachKuzu()` pattern.
|
||||
- `lbug-adapter.ts` fallback path needs quote/newline escaping for Cypher injection prevention.
|
||||
|
||||
## Security (security-sentinel)
|
||||
- Cypher query construction in `lbug-adapter.ts` and `kuzu-adapter.ts` — watch for injection via unescaped user-provided symbol names.
|
||||
- CLI accepts `--repo` parameter and file paths — validate against path traversal.
|
||||
- MCP server exposes tools to external AI agents — all tool inputs are untrusted.
|
||||
|
||||
## Performance (performance-oracle)
|
||||
- Tree-sitter buffer size is adaptive (512KB–32MB) via `getTreeSitterBufferSize()` in `constants.ts`.
|
||||
- The ingestion pipeline processes entire repositories — O(n) per file with potential O(n²) in cross-file resolution.
|
||||
- KuzuDB batch inserts vs individual inserts matter for large repos.
|
||||
|
||||
## Architecture (architecture-strategist)
|
||||
- Ingestion pipeline phases: structure → parsing → imports → calls → heritage → processes → type resolution.
|
||||
- Shared modules: `export-detection.ts`, `constants.ts`, `utils.ts` — changes here have wide blast radius.
|
||||
- `gitnexus-web` package drifts behind CLI — flag if a change should be mirrored.
|
||||
|
||||
## Voltagent Supplementary Agents
|
||||
|
||||
Invoke these via the Agent tool alongside `/ce:review` for deeper specialist analysis. These cover gaps that compound-engineering agents don't:
|
||||
|
||||
### voltagent-lang:typescript-pro
|
||||
**When:** Changes touch type-resolution logic, generics, conditional types, or complex type-level programming in `type-env.ts`, `type-extractors/*.ts`, or `types.ts`.
|
||||
**Why:** The type resolution system uses advanced TypeScript patterns (discriminated unions, mapped types, recursive generics) that benefit from deep TS type-system review beyond what kieran-typescript-reviewer covers.
|
||||
|
||||
### voltagent-qa-sec:security-auditor
|
||||
**When:** Changes touch MCP tool handlers, Cypher query construction, CLI argument parsing, or any code that processes external input.
|
||||
**Why:** GitNexus is an MCP server — all tool inputs come from untrusted AI agents. Systematic OWASP-level audit catches injection vectors that spot-checking misses. Past finding: `lbug-adapter.ts` fallback path had unescaped newlines in Cypher queries.
|
||||
|
||||
### voltagent-data-ai:database-optimizer
|
||||
**When:** Changes touch `kuzu-adapter.ts`, `schema.ts`, `lbug-adapter.ts`, or any Cypher query construction/execution.
|
||||
**Why:** No CE agent specializes in graph database optimization. KuzuDB batch insert patterns, index usage, and query planning directly affect analysis speed on large repos.
|
||||
|
||||
## Review Tooling
|
||||
- Use `gitnexus_impact()` before approving changes to any symbol — check d=1 (WILL BREAK) callers.
|
||||
- Use `gitnexus_detect_changes({scope: "compare", base_ref: "main"})` to map PR diffs to affected execution flows.
|
||||
- Use claude-mem to surface past architectural decisions relevant to the code under review.
|
||||
@@ -1,76 +0,0 @@
|
||||
# Sigstore policy-controller ClusterImagePolicy for GitNexus container images.
|
||||
#
|
||||
# This enforces — at admission time — that every Pod pulling a
|
||||
# `ghcr.io/abhigyanpatwari/gitnexus` or `gitnexus-web` image is using a build
|
||||
# that was Cosign-keyless-signed by this repository's `docker.yml` workflow
|
||||
# running from a `vX.Y.Z` git tag. Unsigned images, images signed by other
|
||||
# workflows, and images signed from unprotected refs (e.g. `main`, PR branches)
|
||||
# are rejected.
|
||||
#
|
||||
# Prerequisites
|
||||
# -------------
|
||||
# 1. Install the Sigstore policy-controller in your cluster (Helm):
|
||||
#
|
||||
# helm repo add sigstore https://sigstore.github.io/helm-charts
|
||||
# helm repo update
|
||||
# helm install policy-controller -n cosign-system --create-namespace \
|
||||
# sigstore/policy-controller
|
||||
#
|
||||
# 2. Opt namespaces in to verification:
|
||||
#
|
||||
# kubectl label namespace <your-ns> policy.sigstore.dev/include=true
|
||||
#
|
||||
# 3. Apply this policy:
|
||||
#
|
||||
# kubectl apply -f deploy/kubernetes/cluster-image-policy.yaml
|
||||
#
|
||||
# After this, `kubectl run --image=ghcr.io/abhigyanpatwari/gitnexus:<tag>` in
|
||||
# any opted-in namespace will only succeed if the image carries a valid
|
||||
# Sigstore signature with the pinned identity.
|
||||
#
|
||||
# References
|
||||
# - https://docs.sigstore.dev/policy-controller/overview/
|
||||
# - https://github.com/sigstore/policy-controller
|
||||
apiVersion: policy.sigstore.dev/v1beta1
|
||||
kind: ClusterImagePolicy
|
||||
metadata:
|
||||
name: gitnexus-signed-images
|
||||
spec:
|
||||
# Apply to both published GitNexus images on both registries. Image
|
||||
# references always carry a tag or digest at admission time, so these globs
|
||||
# cover every `gitnexus:<tag>`, `gitnexus@sha256:...`, `gitnexus-web:<tag>`,
|
||||
# and `gitnexus-web@sha256:...` reference on either GHCR or Docker Hub.
|
||||
# The Docker Hub images are byte-for-byte mirrors of the GHCR images (same
|
||||
# build, same digest, same Cosign signature), so the same keyless identity
|
||||
# authority verifies both.
|
||||
images:
|
||||
- glob: 'ghcr.io/abhigyanpatwari/gitnexus*'
|
||||
# Docker Hub references can appear in three forms at admission time
|
||||
# (`docker.io/...`, `index.docker.io/...`, and bare `akonlabs/...` with
|
||||
# the default registry implied). List all three so the policy cannot be
|
||||
# sidestepped by the choice of registry prefix. The Docker Hub namespace
|
||||
# is `akonlabs` rather than `abhigyanpatwari` because the Docker Hub org
|
||||
# differs from the GitHub org.
|
||||
- glob: 'docker.io/akonlabs/gitnexus*'
|
||||
- glob: 'index.docker.io/akonlabs/gitnexus*'
|
||||
- glob: 'akonlabs/gitnexus*'
|
||||
authorities:
|
||||
- name: gitnexus-cosign-keyless
|
||||
keyless:
|
||||
# Public-good Sigstore Fulcio root.
|
||||
url: https://fulcio.sigstore.dev
|
||||
identities:
|
||||
# Pin both the OIDC issuer (GitHub Actions) AND the exact workflow
|
||||
# path running from a `vX.Y.Z` (or `vX.Y.Z-prerelease`) tag. Same
|
||||
# regex the README's `cosign verify` example uses; it rejects:
|
||||
# * unsigned images
|
||||
# * signatures from any other repo / workflow
|
||||
# * signatures from non-tag refs (main, PRs, release branches)
|
||||
# * signatures from arbitrary non-semver tags
|
||||
- issuer: https://token.actions.githubusercontent.com
|
||||
subjectRegExp: ^https://github\.com/abhigyanpatwari/GitNexus/\.github/workflows/docker\.yml@refs/tags/v[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$
|
||||
# Cross-check the signature against the public Rekor transparency log,
|
||||
# so an attacker who briefly compromised Fulcio cannot retroactively
|
||||
# mint a signature without leaving a public, append-only audit record.
|
||||
ctlog:
|
||||
url: https://rekor.sigstore.dev
|
||||
@@ -1,45 +0,0 @@
|
||||
services:
|
||||
gitnexus-server:
|
||||
image: ${SERVER_IMAGE:-ghcr.io/abhigyanpatwari/gitnexus:latest}
|
||||
container_name: ${SERVER_CONTAINER_NAME:-gitnexus-server}
|
||||
# Map the server to the same host port the web UI expects by default
|
||||
# (http://localhost:4747). The browser runs on the host, so the UI's
|
||||
# built-in default works without any reconfiguration.
|
||||
ports:
|
||||
- '${SERVER_HOST_PORT:-4747}:4747'
|
||||
volumes:
|
||||
# Persist the global registry, indexes, and cloned repos across runs.
|
||||
- gitnexus-data:/data/gitnexus
|
||||
# Optional: mount a host workspace so `gitnexus index <path>` can see
|
||||
# repos you already have on disk. The default points at an empty
|
||||
# `./workspace/` sibling that compose will create on first start —
|
||||
# it intentionally does NOT bind-mount the repo root, which would
|
||||
# expose `.git`, `.env`, and CI secrets to the container.
|
||||
# Override with `WORKSPACE_DIR=/abs/path/to/your/repos`.
|
||||
- ${WORKSPACE_DIR:-./workspace}:/workspace:ro
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-fsSI', 'http://localhost:4747/api/heartbeat']
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 15s
|
||||
|
||||
gitnexus-web:
|
||||
image: ${WEB_IMAGE:-ghcr.io/abhigyanpatwari/gitnexus-web:latest}
|
||||
container_name: ${WEB_CONTAINER_NAME:-gitnexus-web}
|
||||
ports:
|
||||
- '${WEB_HOST_PORT:-4173}:4173'
|
||||
depends_on:
|
||||
gitnexus-server:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ['CMD', 'curl', '-f', 'http://localhost:4173/']
|
||||
interval: 30s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
|
||||
volumes:
|
||||
gitnexus-data:
|
||||
@@ -1,81 +0,0 @@
|
||||
import { createReadStream } from 'node:fs';
|
||||
import { stat } from 'node:fs/promises';
|
||||
import { createServer } from 'node:http';
|
||||
import { extname, join, normalize, sep } from 'node:path';
|
||||
|
||||
const host = '0.0.0.0';
|
||||
const port = Number(process.env.PORT || '4173');
|
||||
const root = join(process.cwd(), 'dist');
|
||||
|
||||
const contentTypes = {
|
||||
'.css': 'text/css; charset=utf-8',
|
||||
'.html': 'text/html; charset=utf-8',
|
||||
'.js': 'text/javascript; charset=utf-8',
|
||||
'.json': 'application/json; charset=utf-8',
|
||||
'.map': 'application/json; charset=utf-8',
|
||||
'.png': 'image/png',
|
||||
'.svg': 'image/svg+xml',
|
||||
'.txt': 'text/plain; charset=utf-8',
|
||||
'.woff': 'font/woff',
|
||||
'.woff2': 'font/woff2',
|
||||
};
|
||||
|
||||
function resolvePath(urlPath) {
|
||||
let decoded;
|
||||
try {
|
||||
decoded = decodeURIComponent(urlPath);
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
if (decoded.includes('\0')) return null;
|
||||
const cleanPath = normalize(decoded.replace(/^\/+/, ''));
|
||||
const candidate = join(root, cleanPath);
|
||||
if (candidate !== root && !candidate.startsWith(root + sep)) return null;
|
||||
return candidate;
|
||||
}
|
||||
|
||||
const server = createServer(async (req, res) => {
|
||||
const requestPath = req.url?.split('?')[0] || '/';
|
||||
let filePath = resolvePath(requestPath);
|
||||
|
||||
if (!filePath) {
|
||||
res.writeHead(400);
|
||||
res.end('Bad request');
|
||||
return;
|
||||
}
|
||||
|
||||
try {
|
||||
const fileStat = await stat(filePath).catch(() => null);
|
||||
if (fileStat?.isDirectory()) {
|
||||
filePath = join(filePath, 'index.html');
|
||||
} else if (!fileStat?.isFile()) {
|
||||
filePath = join(root, 'index.html');
|
||||
}
|
||||
|
||||
const finalStat = await stat(filePath).catch(() => null);
|
||||
if (!finalStat?.isFile()) {
|
||||
res.writeHead(404);
|
||||
res.end('Not found');
|
||||
return;
|
||||
}
|
||||
|
||||
res.writeHead(200, {
|
||||
'Cache-Control': filePath.includes('/assets/')
|
||||
? 'public, max-age=31536000, immutable'
|
||||
: 'no-cache',
|
||||
'Content-Type': contentTypes[extname(filePath)] || 'application/octet-stream',
|
||||
'Cross-Origin-Opener-Policy': 'same-origin',
|
||||
'Cross-Origin-Embedder-Policy': 'require-corp',
|
||||
});
|
||||
const stream = createReadStream(filePath);
|
||||
stream.on('error', () => res.destroy());
|
||||
stream.pipe(res);
|
||||
} catch (error) {
|
||||
res.writeHead(500);
|
||||
res.end(error instanceof Error ? error.message : 'Internal server error');
|
||||
}
|
||||
});
|
||||
|
||||
server.listen(port, host, () => {
|
||||
console.log(`gitnexus-web listening on http://${host}:${port}`);
|
||||
});
|
||||
@@ -1,107 +0,0 @@
|
||||
import { mkdir, mkdtemp, rm, unlink, writeFile } from 'node:fs/promises';
|
||||
import http, { createServer } from 'node:http';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { spawn } from 'node:child_process';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
import { after, before, it } from 'node:test';
|
||||
import assert from 'node:assert/strict';
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const serverScript = join(__dirname, 'docker-server.mjs');
|
||||
|
||||
function getFreePort() {
|
||||
return new Promise((resolve) => {
|
||||
const s = createServer();
|
||||
s.listen(0, '127.0.0.1', () => {
|
||||
const { port } = s.address();
|
||||
s.close(() => resolve(port));
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function rawGet(port, path) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = http.request({ host: '127.0.0.1', port, path }, (res) => {
|
||||
let body = '';
|
||||
res.setEncoding('utf8');
|
||||
res.on('data', (chunk) => {
|
||||
body += chunk;
|
||||
});
|
||||
res.on('end', () => resolve({ status: res.statusCode, headers: res.headers, body }));
|
||||
});
|
||||
req.on('error', reject);
|
||||
req.end();
|
||||
});
|
||||
}
|
||||
|
||||
async function waitForServer(port, retries = 30) {
|
||||
for (let i = 0; i < retries; i++) {
|
||||
try {
|
||||
await rawGet(port, '/');
|
||||
return;
|
||||
} catch {
|
||||
await new Promise((r) => setTimeout(r, 100));
|
||||
}
|
||||
}
|
||||
throw new Error('Server did not start in time');
|
||||
}
|
||||
|
||||
let tmpDir, serverPort, child;
|
||||
|
||||
before(async () => {
|
||||
tmpDir = await mkdtemp(join(tmpdir(), 'gitnexus-docker-test-'));
|
||||
const distDir = join(tmpDir, 'dist');
|
||||
const assetsDir = join(distDir, 'assets');
|
||||
await mkdir(assetsDir, { recursive: true });
|
||||
await writeFile(join(distDir, 'index.html'), '<html><body>spa</body></html>');
|
||||
await writeFile(join(assetsDir, 'app.abc123.js'), 'console.log("app")');
|
||||
|
||||
serverPort = await getFreePort();
|
||||
child = spawn(process.execPath, [serverScript], {
|
||||
cwd: tmpDir,
|
||||
env: { ...process.env, PORT: String(serverPort) },
|
||||
stdio: 'pipe',
|
||||
});
|
||||
child.on('error', (err) => {
|
||||
throw err;
|
||||
});
|
||||
|
||||
await waitForServer(serverPort);
|
||||
});
|
||||
|
||||
after(async () => {
|
||||
child?.kill();
|
||||
if (tmpDir) await rm(tmpDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('serves a valid asset with immutable cache header', async () => {
|
||||
const res = await rawGet(serverPort, '/assets/app.abc123.js');
|
||||
assert.equal(res.status, 200);
|
||||
assert.match(res.headers['cache-control'], /immutable/);
|
||||
assert.equal(res.headers['cross-origin-opener-policy'], 'same-origin');
|
||||
assert.equal(res.headers['cross-origin-embedder-policy'], 'require-corp');
|
||||
});
|
||||
|
||||
it('serves SPA fallback for unknown routes', async () => {
|
||||
const res = await rawGet(serverPort, '/some/unknown/route');
|
||||
assert.equal(res.status, 200);
|
||||
assert.match(res.body, /spa/);
|
||||
assert.match(res.headers['cache-control'], /no-cache/);
|
||||
});
|
||||
|
||||
it('rejects path traversal with 400', async () => {
|
||||
const res = await rawGet(serverPort, '/../../../etc/passwd');
|
||||
assert.equal(res.status, 400);
|
||||
});
|
||||
|
||||
it('rejects percent-encoded null bytes with 400', async () => {
|
||||
const res = await rawGet(serverPort, '/foo%00bar');
|
||||
assert.equal(res.status, 400);
|
||||
});
|
||||
|
||||
it('returns 404 when dist/index.html is missing', async () => {
|
||||
await unlink(join(tmpDir, 'dist', 'index.html'));
|
||||
const res = await rawGet(serverPort, '/nonexistent-page');
|
||||
assert.equal(res.status, 404);
|
||||
});
|
||||
@@ -1,100 +0,0 @@
|
||||
# COBOL Code Indexing
|
||||
|
||||
GitNexus indexes COBOL codebases using a **regex-only extraction** strategy, bypassing tree-sitter entirely. This document explains why, how the pipeline works, and links to detailed sub-documents.
|
||||
|
||||
## Why Regex-Only?
|
||||
|
||||
The tree-sitter-cobol grammar (v0.0.1) has three critical limitations that make it unusable for production indexing:
|
||||
|
||||
| Issue | Impact | Severity |
|
||||
|-------|--------|----------|
|
||||
| External scanner hangs on ~5% of files | No timeout mechanism exists for the C scanner; the process blocks indefinitely | **Blocking** |
|
||||
| Only ~15% of paragraph headers detected | Most procedure-division paragraphs are invisible to the grammar | High |
|
||||
| Patch markers in cols 1-6 cause parse errors | Enterprise COBOL uses non-standard sequence area content (e.g., `mzADD`, `estero`, `#FIX`) | High |
|
||||
|
||||
Because the external scanner hang cannot be interrupted (there is no `setTimeoutMicros` equivalent for tree-sitter), using tree-sitter-cobol would hang the indexing pipeline on a non-trivial fraction of real-world files.
|
||||
|
||||
The regex-only approach provides:
|
||||
|
||||
- **Speed**: ~1ms per file average extraction time
|
||||
- **Reliability**: zero hangs, zero crashes across 13,000+ files
|
||||
- **Coverage**: captures all critical symbols -- program name, paragraphs, sections, CALL, PERFORM, COPY, data items (01-77, 88-level), file declarations, FD entries, EXEC SQL/CICS blocks, ENTRY points, and MOVE statements
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[Repository Scan] --> B{File Detection}
|
||||
B -->|Extension match| C[COBOL file]
|
||||
B -->|GITNEXUS_COBOL_DIRS match| C
|
||||
B -->|No match| Z[Skip]
|
||||
|
||||
C --> D{Copybook?}
|
||||
D -->|Yes| E[Add to Copybook Map]
|
||||
D -->|No| F[Source Program]
|
||||
|
||||
E --> G[COPY Expansion Engine]
|
||||
F --> G
|
||||
|
||||
G -->|Inline copybook content| H[Expanded Source]
|
||||
H --> I[Patch Marker Cleanup]
|
||||
I --> J[Regex State Machine]
|
||||
|
||||
J --> K[Extracted Symbols]
|
||||
K --> L[Graph Model Builder]
|
||||
L --> M[Knowledge Graph]
|
||||
|
||||
subgraph "Per-Chunk Processing"
|
||||
G
|
||||
H
|
||||
I
|
||||
J
|
||||
K
|
||||
L
|
||||
end
|
||||
|
||||
subgraph "Post-Processing"
|
||||
M --> N[Community Detection]
|
||||
M --> O[Process Detection]
|
||||
M --> P[Contract Detection]
|
||||
end
|
||||
|
||||
style J fill:#e8f5e9,stroke:#2e7d32
|
||||
style G fill:#e3f2fd,stroke:#1565c0
|
||||
```
|
||||
|
||||
## COBOL vs Tree-Sitter Languages
|
||||
|
||||
| Feature | COBOL (Regex) | Tree-Sitter Languages |
|
||||
|---------|--------------|----------------------|
|
||||
| Parser | Single-pass regex state machine | tree-sitter grammar + queries |
|
||||
| Speed | ~1ms/file | ~5ms/file |
|
||||
| AST available | No | Yes |
|
||||
| COPY expansion | Yes (pre-processing step) | N/A |
|
||||
| Deep indexing | Data items, SQL, CICS, FD, ENTRY | Type annotations, generics, etc. |
|
||||
| Call extraction | PERFORM (intra-file) + CALL (cross-program) | AST-based call site detection |
|
||||
| Import extraction | COPY statements | `import`/`require`/`use`/`#include` |
|
||||
| Coverage | All critical symbols | Language-dependent query coverage |
|
||||
| Failure mode | Never hangs | External scanner can hang (COBOL only) |
|
||||
|
||||
## Sub-Documents
|
||||
|
||||
| Document | Description |
|
||||
|----------|-------------|
|
||||
| [File Detection](./file-detection.md) | Extension mapping, `GITNEXUS_COBOL_DIRS`, copybook classification |
|
||||
| [COPY Expansion](./copy-expansion.md) | Copybook inlining, REPLACING transformations, cycle detection |
|
||||
| [Regex Extraction](./regex-extraction.md) | State machine, regex patterns, line processing |
|
||||
| [Deep Indexing](./deep-indexing.md) | Data items, EXEC SQL/CICS, file declarations, FD, ENTRY, MOVE |
|
||||
| [Graph Model](./graph-model.md) | COBOL-specific node types, edge types, full annotated example |
|
||||
| [Performance](./performance.md) | Benchmarks, worker pool tuning, caps, troubleshooting |
|
||||
|
||||
## Key Source Files
|
||||
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `gitnexus/src/core/ingestion/cobol-preprocessor.ts` | Patch marker cleanup + regex extraction engine |
|
||||
| `gitnexus/src/core/ingestion/cobol-copy-expander.ts` | COPY statement expansion with REPLACING |
|
||||
| `gitnexus/src/core/ingestion/utils.ts` | `getLanguageFromPath`, `getLanguageFromFilename` |
|
||||
| `gitnexus/src/core/ingestion/pipeline.ts` | `isCobolCopybook`, `expandCobolCopies`, `detectCrossProgamContracts` |
|
||||
| `gitnexus/src/core/ingestion/workers/parse-worker.ts` | `processCobolRegexOnly` -- graph model builder |
|
||||
| `gitnexus/src/core/ingestion/workers/worker-pool.ts` | Configurable sub-batch size for COBOL |
|
||||
@@ -1,157 +0,0 @@
|
||||
# COBOL COPY Expansion
|
||||
|
||||
The COPY statement is COBOL's include mechanism -- analogous to `#include` in C or `import` in modern languages. GitNexus expands COPY statements **before** regex extraction so that symbols defined inside copybooks (data items, paragraphs, etc.) are visible in the program's extracted graph.
|
||||
|
||||
## Supported Syntax
|
||||
|
||||
### Basic COPY
|
||||
|
||||
```cobol
|
||||
COPY CPSESP.
|
||||
COPY "WORKGRID.CPY".
|
||||
```
|
||||
|
||||
Inlines the content of the named copybook, replacing the COPY line(s).
|
||||
|
||||
### COPY with REPLACING
|
||||
|
||||
```cobol
|
||||
COPY CPSESP REPLACING "ANAZI-KEY" BY "LK-KEY".
|
||||
COPY CPSESP REPLACING LEADING "ESP-" BY "LK-ESP-"
|
||||
LEADING "KPSESPL" BY "LK-KPSESPL".
|
||||
COPY LINKAGE REPLACING TRAILING "-IN" BY "-OUT".
|
||||
```
|
||||
|
||||
Three REPLACING types are supported:
|
||||
|
||||
| Type | Syntax | Behavior | Example |
|
||||
| ------------ | ------------------------------------ | --------------------------------------- | -------------------------------- |
|
||||
| **EXACT** | `REPLACING "OLD" BY "NEW"` | Replace exact identifier matches | `ANAZI-KEY` becomes `LK-KEY` |
|
||||
| **LEADING** | `REPLACING LEADING "PFX-" BY "NEW-"` | Replace prefix on all COBOL identifiers | `ESP-NAME` becomes `LK-ESP-NAME` |
|
||||
| **TRAILING** | `REPLACING TRAILING "-IN" BY "-OUT"` | Replace suffix on all COBOL identifiers | `DATA-IN` becomes `DATA-OUT` |
|
||||
|
||||
Multiple REPLACING clauses can appear in a single COPY statement. They are applied in order to each COBOL identifier in the copybook content.
|
||||
|
||||
### Multi-Line COPY
|
||||
|
||||
COPY statements can span multiple lines (standard COBOL continuation rules apply):
|
||||
|
||||
```cobol
|
||||
COPY CPSESP REPLACING
|
||||
- LEADING "ESP-" BY "LK-ESP-"
|
||||
- LEADING "KPSESPL" BY "LK-KPSESPL".
|
||||
```
|
||||
|
||||
Continuation lines (indicator `-` in column 7) are merged before COPY statement scanning.
|
||||
|
||||
## Expansion Flow
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant Pipeline
|
||||
participant Expander as COPY Expander
|
||||
participant Resolver
|
||||
participant Reader
|
||||
|
||||
Pipeline->>Pipeline: Identify all COBOL files
|
||||
Pipeline->>Pipeline: Classify copybooks vs programs
|
||||
Pipeline->>Reader: Read all copybook content upfront
|
||||
Reader-->>Pipeline: Copybook content map (name -> content)
|
||||
|
||||
loop For each source file in chunk
|
||||
Pipeline->>Expander: expandCopies(content, filePath, resolveFile, readFile)
|
||||
Expander->>Expander: Merge continuation lines
|
||||
Expander->>Expander: Detect COPY statements via regex
|
||||
|
||||
loop For each COPY statement (reverse order)
|
||||
Expander->>Resolver: resolveFile(copyTarget)
|
||||
Resolver-->>Expander: Copybook key or null
|
||||
|
||||
alt Resolved successfully
|
||||
Expander->>Reader: readFile(resolvedKey)
|
||||
Reader-->>Expander: Copybook content
|
||||
|
||||
Expander->>Expander: Apply REPLACING transformations
|
||||
Expander->>Expander: Recurse for nested COPYs (depth + 1)
|
||||
Expander->>Expander: Splice expanded content into output
|
||||
else Not resolved
|
||||
Expander->>Expander: Keep original COPY line
|
||||
end
|
||||
end
|
||||
|
||||
Expander-->>Pipeline: Expanded content + resolution metadata
|
||||
Pipeline->>Pipeline: Replace file content with expanded content
|
||||
end
|
||||
```
|
||||
|
||||
The return type `CopyExpansionResult` contains `expandedContent` and `copyResolutions`. The `expansionDepth` field has been removed from the return type (it was unused by callers).
|
||||
|
||||
COPY statement line numbers in `CopyResolution` are 1-based (consistent with the preprocessor's line numbering). The splice operation that replaces COPY lines with expanded content adjusts for 0-based array indexing internally.
|
||||
|
||||
## Cycle Detection
|
||||
|
||||
Circular COPY references (e.g., copybook A includes copybook B which includes copybook A) are detected and handled:
|
||||
|
||||
1. Each expansion chain maintains a `visited` set of resolved copybook paths
|
||||
2. If a copybook path is already in the visited set, the expansion is skipped
|
||||
3. A `warnedCircular` set (internal to `expandCopies()`, not a parameter) deduplicates warning messages within a single file expansion
|
||||
|
||||
Known circular copybooks in PROJECT-NAME: `ANAZI`, `ANDIP`, `QDIPE` (self-referential includes).
|
||||
|
||||
## Max Depth
|
||||
|
||||
Nested COPY expansion is limited to **10 levels** (`DEFAULT_MAX_DEPTH`). If a COPY chain exceeds this depth, a warning is logged and the remaining COPY statements are left unexpanded.
|
||||
|
||||
## Max Total Expansions
|
||||
|
||||
A breadth amplification guard caps the total number of COPY expansions across all branches within a single file to **500** (`MAX_TOTAL_EXPANSIONS`). This prevents exponential blowup from diamond-shaped COPY graphs where N copybooks each include N other copybooks. Once the limit is reached, further COPY statements in that file are left unexpanded and a single warning is logged.
|
||||
|
||||
## REPLACING Application Detail
|
||||
|
||||
The REPLACING engine works by scanning all COBOL identifiers (matching `\b[A-Z][A-Z0-9-]*\b`) in the copybook content and applying each replacement rule:
|
||||
|
||||
```
|
||||
Original copybook content:
|
||||
05 ESP-NAME PIC X(30).
|
||||
05 ESP-CODE PIC X(10).
|
||||
05 KPSESPL-FLAG PIC X(01).
|
||||
|
||||
After REPLACING LEADING "ESP-" BY "LK-ESP-" LEADING "KPSESPL" BY "LK-KPSESPL":
|
||||
05 LK-ESP-NAME PIC X(30).
|
||||
05 LK-ESP-CODE PIC X(10).
|
||||
05 LK-KPSESPL-FLAG PIC X(01).
|
||||
```
|
||||
|
||||
For LEADING replacements, the engine checks if each identifier starts with the `from` prefix (case-insensitive) and replaces only the prefix portion, preserving the rest of the identifier.
|
||||
|
||||
For TRAILING replacements, the same logic applies to suffixes.
|
||||
|
||||
For EXACT replacements, only identifiers that match the `from` value exactly (case-insensitive) are replaced.
|
||||
|
||||
## Copybook Resolution
|
||||
|
||||
The resolver tries multiple strategies to match a COPY target name to a copybook file:
|
||||
|
||||
1. **Exact match**: `COPY CPSESP` resolves to copybook named `CPSESP`
|
||||
2. **Strip extension**: `COPY WORKGRID.CPY` strips `.CPY` and resolves to `WORKGRID`
|
||||
3. **Add extension**: `COPY CPSESP` tries `CPSESP.CPY` and `CPSESP.COPY`
|
||||
|
||||
If no match is found, the COPY statement is left in place (unexpanded) and a resolution record with `resolvedPath: null` is created.
|
||||
|
||||
## Pipeline Integration
|
||||
|
||||
The expansion runs **per chunk**, after file content is read but before dispatch to worker threads:
|
||||
|
||||
1. All copybook files are read upfront (they are typically small, collectively under 100MB)
|
||||
2. Per chunk, the copybook map is merged with chunk content (in case a chunk contains copybooks)
|
||||
3. Only programs (not copybooks themselves) undergo expansion
|
||||
4. The expanded content replaces the original content in-place before worker dispatch
|
||||
|
||||
## Inline Comment Handling
|
||||
|
||||
The copy expander's `stripInlineComment()` helper is quote-aware: pipe characters (`|`) inside single- or double-quoted strings are preserved. This matches the same quote-aware logic used by the preprocessor.
|
||||
|
||||
## Source Files
|
||||
|
||||
- `gitnexus/src/core/ingestion/cobol-copy-expander.ts` -- `expandCopies()`, `parseReplacingClause()`, `applyReplacing()`
|
||||
- `gitnexus/src/core/ingestion/pipeline.ts` -- `expandCobolCopies()`, copybook map construction, chunk integration
|
||||
@@ -1,312 +0,0 @@
|
||||
# COBOL Deep Indexing
|
||||
|
||||
Beyond basic symbol extraction (program name, paragraphs, CALL, PERFORM, COPY), GitNexus performs deep indexing of COBOL-specific constructs: data items, EXEC SQL/CICS blocks, file declarations, FD entries, ENTRY points, and MOVE statements.
|
||||
|
||||
## Data Items
|
||||
|
||||
### Level Numbers
|
||||
|
||||
| Level Range | Meaning | Graph Node Type |
|
||||
|-------------|---------|-----------------|
|
||||
| 01 | Record (group item) | `Record` |
|
||||
| 02-49 | Elementary/group items | `Property` |
|
||||
| 66 | RENAMES | `Property` |
|
||||
| 77 | Independent item | `Property` |
|
||||
| 88 | Condition name | `Const` |
|
||||
|
||||
FILLER items are skipped (no useful name for the graph).
|
||||
|
||||
### Clauses Parsed
|
||||
|
||||
The `parseDataItemClauses()` function extracts these clauses from the trailing text of a data item declaration:
|
||||
|
||||
| Clause | Pattern | Example |
|
||||
|--------|---------|---------|
|
||||
| `PIC` / `PICTURE` | `\bPIC(?:TURE)?\s+(?:IS\s+)?(\S+)` | `PIC X(30)`, `PICTURE IS 9(5)V99` |
|
||||
| `USAGE` | `\bUSAGE\s+(?:IS\s+)?(COMP\|BINARY\|...)` | `USAGE IS COMP-3`, `BINARY` |
|
||||
| `REDEFINES` | `\bREDEFINES\s+([A-Z][A-Z0-9-]+)` | `REDEFINES WK-DATE-NUM` |
|
||||
| `OCCURS` | `\bOCCURS\s+(\d+)` | `OCCURS 12 TIMES` |
|
||||
|
||||
Standalone COMP variants (without the `USAGE` keyword) are also detected: `COMP`, `COMP-1` through `COMP-6`, `COMP-X`, `BINARY`, `PACKED-DECIMAL`.
|
||||
|
||||
### Data Hierarchy
|
||||
|
||||
Data items form a hierarchical structure based on level numbers. The extractor uses a **stack algorithm**:
|
||||
|
||||
```
|
||||
Processing order:
|
||||
01 WK-RECORD -> push {01, WK-RECORD} -> parent: Module
|
||||
05 WK-NAME -> push {05, WK-NAME} -> parent: WK-RECORD (01 < 05)
|
||||
10 WK-FIRST -> push {10, WK-FIRST} -> parent: WK-NAME (05 < 10)
|
||||
10 WK-LAST -> pop WK-FIRST, push -> parent: WK-NAME (05 < 10)
|
||||
05 WK-CODE -> pop WK-LAST, WK-NAME -> parent: WK-RECORD (01 < 05)
|
||||
88 WK-ACTIVE -> (88 handled separately) -> parent: WK-CODE
|
||||
```
|
||||
|
||||
The stack maintains items where each entry's level is strictly less than the next. When a new item arrives with a level <= the top of stack, items are popped until the stack top has a smaller level. A `CONTAINS` edge is created from the stack top to the new item.
|
||||
|
||||
For 88-level condition names, the parent is the immediately preceding non-88 data item (found by scanning backwards).
|
||||
|
||||
### Annotated Example
|
||||
|
||||
```cobol
|
||||
01 WK-EMPLOYEE.
|
||||
05 WK-EMP-ID PIC 9(6).
|
||||
05 WK-EMP-NAME PIC X(30).
|
||||
05 WK-EMP-STATUS PIC X(01).
|
||||
88 WK-ACTIVE VALUE "A".
|
||||
88 WK-INACTIVE VALUE "I".
|
||||
05 WK-SALARY PIC 9(7)V99 COMP-3.
|
||||
05 WK-DEPT PIC X(04) OCCURS 3 TIMES.
|
||||
```
|
||||
|
||||
Produces:
|
||||
- `Record` node: `WK-EMPLOYEE` (level 01, section: working-storage)
|
||||
- `Property` nodes: `WK-EMP-ID`, `WK-EMP-NAME`, `WK-EMP-STATUS`, `WK-SALARY`, `WK-DEPT`
|
||||
- `Const` nodes: `WK-ACTIVE` (values: `A`), `WK-INACTIVE` (values: `I`)
|
||||
- `CONTAINS` edges: `WK-EMPLOYEE -> WK-EMP-ID`, `WK-EMPLOYEE -> WK-EMP-NAME`, etc.
|
||||
- `CONTAINS` edges: `WK-EMP-STATUS -> WK-ACTIVE`, `WK-EMP-STATUS -> WK-INACTIVE`
|
||||
|
||||
### Data Item Cap
|
||||
|
||||
A maximum of **500 data items per file** (`MAX_DATA_ITEMS_PER_FILE`) are processed. Some COBOL programs (especially after COPY expansion) can have 10,000+ data items, which would cause graph bloat and push the V8 relationship Map past its 16.7M entry limit across thousands of files.
|
||||
|
||||
The cap applies after extraction: the first 500 items in source order are kept. Since 01-level records appear first, critical top-level structure is preserved.
|
||||
|
||||
## EXEC SQL
|
||||
|
||||
EXEC SQL blocks are accumulated across lines between `EXEC SQL` and `END-EXEC`, then parsed as a unit.
|
||||
|
||||
### Operation Classification
|
||||
|
||||
The first SQL keyword determines the operation:
|
||||
|
||||
| First Keyword | Operation |
|
||||
|---------------|-----------|
|
||||
| `SELECT` | SELECT |
|
||||
| `INSERT` | INSERT |
|
||||
| `UPDATE` | UPDATE |
|
||||
| `DELETE` | DELETE |
|
||||
| `DECLARE` | DECLARE |
|
||||
| `OPEN` | OPEN |
|
||||
| `CLOSE` | CLOSE |
|
||||
| `FETCH` | FETCH |
|
||||
| *(anything else)* | OTHER |
|
||||
|
||||
### Table Extraction
|
||||
|
||||
Tables are extracted from SQL clauses:
|
||||
|
||||
| Clause Pattern | Example |
|
||||
|----------------|---------|
|
||||
| `FROM <table>` | `SELECT * FROM EMPLOYEES` |
|
||||
| `INSERT INTO <table>` | `INSERT INTO EMPLOYEES` |
|
||||
| `UPDATE <table>` | `UPDATE EMPLOYEES SET ...` |
|
||||
| `JOIN <table>` | `LEFT JOIN DEPARTMENTS ON ...` |
|
||||
|
||||
Note: The `INTO` pattern is restricted to `INSERT INTO` to avoid false positives from `FETCH ... INTO :host-var` and `SELECT ... INTO :host-var` statements, where `INTO` introduces host variables rather than table names.
|
||||
|
||||
### Cursor Detection
|
||||
|
||||
```cobol
|
||||
EXEC SQL
|
||||
DECLARE C-EMPLOYEES CURSOR FOR
|
||||
SELECT EMP-ID, EMP-NAME FROM EMPLOYEES
|
||||
WHERE DEPT = :WK-DEPT
|
||||
END-EXEC
|
||||
```
|
||||
|
||||
Extracts: cursor `C-EMPLOYEES`, table `EMPLOYEES`, host variable `WK-DEPT`.
|
||||
|
||||
### Host Variables
|
||||
|
||||
Host variables are COBOL variables referenced in SQL with a `:` prefix. The colon is stripped:
|
||||
|
||||
```sql
|
||||
WHERE EMP-ID = :WK-EMP-ID AND DEPT = :WK-DEPT
|
||||
```
|
||||
|
||||
Extracts: `WK-EMP-ID`, `WK-DEPT`.
|
||||
|
||||
### Graph Output
|
||||
|
||||
- `CodeElement` node per table, with description `sql-table op:{OP}`
|
||||
- `CodeElement` node per cursor, with description `sql-cursor`
|
||||
- `ACCESSES` edge from Module to each CodeElement
|
||||
- Deduplication: if the same table appears in multiple SQL blocks, only one node is created
|
||||
|
||||
## EXEC CICS
|
||||
|
||||
EXEC CICS blocks are accumulated and parsed similarly to SQL blocks.
|
||||
|
||||
### Command Detection
|
||||
|
||||
Two-word commands are detected first (matched against the block start):
|
||||
|
||||
```
|
||||
SEND MAP, RECEIVE MAP, SEND TEXT, SEND CONTROL, READ NEXT, READ PREV
|
||||
```
|
||||
|
||||
If no two-word command matches, the first word is used (e.g., `LINK`, `XCTL`, `RETURN`, `READ`, `WRITE`).
|
||||
|
||||
### Extraction
|
||||
|
||||
| Element | Pattern | Example |
|
||||
|---------|---------|---------|
|
||||
| MAP name | `MAP('name')` or `MAP("name")` | `EXEC CICS SEND MAP('EMPMENU')` |
|
||||
| PROGRAM name | `PROGRAM('name')` or `PROGRAM("name")` | `EXEC CICS LINK PROGRAM('BGTABUP')` |
|
||||
| TRANSID | `TRANSID('name')` or `TRANSID("name")` | `EXEC CICS START TRANSID('EMP1')` |
|
||||
|
||||
### Graph Output
|
||||
|
||||
- MAP: `CodeElement` node with description `cics-map cmd:{CMD}` + `ACCESSES` edge from Module
|
||||
- PROGRAM: `CALLS` edge (cross-program call via CICS LINK/XCTL)
|
||||
- TRANSID: `CodeElement` node with description `cics-transid cmd:{CMD}` + `ACCESSES` edge from Module
|
||||
|
||||
### Annotated Example
|
||||
|
||||
```cobol
|
||||
EXEC CICS
|
||||
SEND MAP('EMPMENU')
|
||||
MAPSET('EMPSET')
|
||||
FROM(WK-MAP-DATA)
|
||||
ERASE
|
||||
END-EXEC
|
||||
```
|
||||
|
||||
Produces:
|
||||
- `CodeElement` node: `EMPMENU` (description: `cics-map cmd:SEND MAP`)
|
||||
- `ACCESSES` edge: Module -> `EMPMENU`
|
||||
|
||||
## File Declarations
|
||||
|
||||
SELECT statements in the INPUT-OUTPUT SECTION are accumulated across multiple lines (until a period terminator) and parsed for:
|
||||
|
||||
| Clause | Pattern | Example |
|
||||
|--------|---------|---------|
|
||||
| SELECT | `SELECT <name>` | `SELECT MASTER-FILE` |
|
||||
| ASSIGN | `ASSIGN TO <file>` | `ASSIGN TO "MASTER.DAT"` |
|
||||
| ORGANIZATION | `ORGANIZATION IS <type>` | `ORGANIZATION IS INDEXED` |
|
||||
| ACCESS | `ACCESS MODE IS <mode>` | `ACCESS MODE IS DYNAMIC` |
|
||||
| RECORD KEY | `RECORD KEY IS <field>` | `RECORD KEY IS WK-EMP-ID` |
|
||||
| FILE STATUS | `FILE STATUS IS <field>` | `FILE STATUS IS WK-FILE-STATUS` |
|
||||
|
||||
### Graph Output
|
||||
|
||||
- `CodeElement` node with description containing all parsed clauses (e.g., `select org:INDEXED access:DYNAMIC key:WK-EMP-ID status:WK-FILE-STATUS assign:MASTER.DAT`)
|
||||
- `RECORD_KEY_OF` edge: from Property node to CodeElement (confidence 0.8)
|
||||
- `FILE_STATUS_OF` edge: from Property node to CodeElement (confidence 0.8)
|
||||
|
||||
## FD Entries
|
||||
|
||||
FD (File Description) entries associate a file name with its record layout:
|
||||
|
||||
```cobol
|
||||
FD MASTER-FILE.
|
||||
01 MASTER-RECORD.
|
||||
05 MR-EMP-ID PIC 9(6).
|
||||
05 MR-EMP-NAME PIC X(30).
|
||||
```
|
||||
|
||||
The extractor tracks `pendingFdName` state: when an `FD` line is seen, the next 01-level data item becomes its record.
|
||||
|
||||
### Graph Output
|
||||
|
||||
- `CodeElement` node with description `fd record:{recordName}`
|
||||
- `CONTAINS` edge: FD CodeElement -> Record node
|
||||
- `CONTAINS` edge: SELECT CodeElement -> FD CodeElement (linking file declaration to file description)
|
||||
|
||||
## ENTRY Points
|
||||
|
||||
The `ENTRY` statement defines additional entry points into a COBOL program (in addition to the main program entry):
|
||||
|
||||
```cobol
|
||||
ENTRY "SUBPROG" USING WK-PARAM-1 WK-PARAM-2.
|
||||
```
|
||||
|
||||
### Graph Output
|
||||
|
||||
- `Constructor` node with description `entry params:{param1},{param2}` (or just `entry` if no parameters)
|
||||
- `CONTAINS` edge: Module -> Constructor
|
||||
- Symbol table entry (so the entry point is discoverable by name)
|
||||
|
||||
## PROCEDURE DIVISION USING
|
||||
|
||||
```cobol
|
||||
PROCEDURE DIVISION USING WK-INPUT-REC WK-OUTPUT-REC.
|
||||
```
|
||||
|
||||
The USING clause identifies parameters received by the program from its caller.
|
||||
|
||||
### Graph Output
|
||||
|
||||
- `RECEIVES` edge: Module -> Property (for each parameter name, confidence 0.8)
|
||||
|
||||
## MOVE Statements
|
||||
|
||||
MOVE statements produce `ACCESSES` edges in the graph:
|
||||
|
||||
```cobol
|
||||
MOVE WK-NAME TO OUT-NAME.
|
||||
MOVE CORRESPONDING WK-INPUT TO WK-OUTPUT.
|
||||
MOVE CORR WK-IN TO WK-OUT.
|
||||
```
|
||||
|
||||
### Extraction Details
|
||||
|
||||
- Source and target identifiers are captured
|
||||
- `CORRESPONDING` and its abbreviation `CORR` are both recognized (bulk field-by-field move)
|
||||
- Figurative constants (SPACES, ZEROS, LOW-VALUES, HIGH-VALUES, QUOTES, ALL) are skipped
|
||||
- The enclosing paragraph (`caller`) is tracked for context
|
||||
|
||||
### MOVE CORRESPONDING / CORR Edge Reasons
|
||||
|
||||
MOVE CORRESPONDING (and CORR) produces distinct edge reasons to differentiate from simple MOVE:
|
||||
|
||||
| Edge | Reason (simple MOVE) | Reason (CORRESPONDING/CORR) |
|
||||
|------|---------------------|-----------------------------|
|
||||
| Read (source) | `cobol-move-read` | `cobol-move-corresponding-read` |
|
||||
| Write (target) | `cobol-move-write` | `cobol-move-corresponding-write` |
|
||||
|
||||
This distinction allows queries to find bulk field-by-field moves separately from simple variable assignments.
|
||||
|
||||
## GO TO DEPENDING ON
|
||||
|
||||
The `GO TO` statement with multiple targets and a `DEPENDING ON` clause is a computed branch:
|
||||
|
||||
```cobol
|
||||
GO TO PARA-1 PARA-2 PARA-3
|
||||
DEPENDING ON WK-SELECTOR.
|
||||
```
|
||||
|
||||
All target paragraph names are extracted and emitted as separate `gotos` entries. Each target produces a `CALLS` edge in the graph (same semantics as PERFORM). The `DEPENDING ON` variable is not currently tracked as a data-flow dependency.
|
||||
|
||||
## SORT INPUT/OUTPUT PROCEDURE
|
||||
|
||||
SORT and MERGE statements can specify procedural entry points instead of file-based I/O:
|
||||
|
||||
```cobol
|
||||
SORT SORT-FILE ON ASCENDING KEY SORT-KEY
|
||||
INPUT PROCEDURE IS PREPARE-INPUT
|
||||
OUTPUT PROCEDURE IS FORMAT-OUTPUT.
|
||||
```
|
||||
|
||||
`INPUT PROCEDURE IS` and `OUTPUT PROCEDURE IS` targets are extracted as control-flow targets (same as PERFORM). They produce `performs` entries and corresponding `CALLS` edges in the graph.
|
||||
|
||||
## Fixed-Format Literal Continuation
|
||||
|
||||
In fixed-format COBOL, string literals can span multiple lines using the continuation indicator (`-` in column 7). When a continuation line starts with a quote character, the extractor joins it with the predecessor by removing the trailing quote from the previous line and the opening quote from the continuation:
|
||||
|
||||
```
|
||||
Line N: MOVE "THIS IS A LONG STRI
|
||||
Line N+1 (cont): - "NG VALUE" TO WK-FIELD.
|
||||
Merged: MOVE "THIS IS A LONG STRING VALUE" TO WK-FIELD.
|
||||
```
|
||||
|
||||
The trailing `"` on line N and the opening `"` on line N+1 are both removed, producing a seamless literal. If no matching quote is found on the predecessor line, the continuation is appended as-is.
|
||||
|
||||
## Source Files
|
||||
|
||||
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- All extraction logic, clause parsers, EXEC block parsers
|
||||
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `processCobolRegexOnly()`, graph node/edge emission
|
||||
- `gitnexus/src/core/ingestion/parsing-processor.ts` -- Sequential fallback with same `MAX_DATA_ITEMS_PER_FILE` cap
|
||||
@@ -1,126 +0,0 @@
|
||||
# COBOL File Detection
|
||||
|
||||
GitNexus detects COBOL files through two mechanisms: extension-based mapping and directory-based override for extensionless files. This document covers both, plus the copybook/program classification logic.
|
||||
|
||||
## Extension Mapping
|
||||
|
||||
### Program Extensions
|
||||
|
||||
| Extension | Type |
|
||||
|-----------|------|
|
||||
| `.cbl` | COBOL program |
|
||||
| `.cob` | COBOL program |
|
||||
| `.cobol` | COBOL program |
|
||||
|
||||
### Copybook Extensions
|
||||
|
||||
| Extension | Type | Notes |
|
||||
|-----------|------|-------|
|
||||
| `.cpy` | Copybook | Standard |
|
||||
| `.copy` | Copybook | Standard |
|
||||
| `.gnm` / `.GNM` | Copybook | Enterprise (GnuCOBOL naming) |
|
||||
| `.fd` / `.FD` | Copybook | File Description fragment |
|
||||
| `.wrk` / `.WRK` | Copybook | Working-Storage fragment |
|
||||
| `.sel` / `.SEL` | Copybook | SELECT clause fragment |
|
||||
| `.open` / `.OPEN` | Copybook | File OPEN fragment |
|
||||
| `.close` / `.CLOSE` | Copybook | File CLOSE fragment |
|
||||
| `.ini` / `.INI` | Copybook | Initialization fragment |
|
||||
| `.def` / `.DEF` | Copybook | Definition fragment |
|
||||
|
||||
All extension matching is case-sensitive in `getLanguageFromFilename` (the extensions above are matched as written, including uppercase variants like `.GNM`).
|
||||
|
||||
## Extensionless File Detection: `GITNEXUS_COBOL_DIRS`
|
||||
|
||||
Many enterprise COBOL repositories use extensionless files -- the filename alone identifies the program (e.g., `s/BGTABFL` is the source for program `BGTABFL`). GitNexus handles this via the `GITNEXUS_COBOL_DIRS` environment variable.
|
||||
|
||||
### Configuration
|
||||
|
||||
Set `GITNEXUS_COBOL_DIRS` to a comma-separated list of directory names:
|
||||
|
||||
```bash
|
||||
# Files in s/, c/, and wfproc/ directories (at any depth) are treated as COBOL
|
||||
export GITNEXUS_COBOL_DIRS=s,c,wfproc
|
||||
```
|
||||
|
||||
The matching is **case-insensitive** and checks all path segments:
|
||||
|
||||
- `/repo/s/BGTABFL` -- matches segment `s` -- COBOL
|
||||
- `/repo/src/c/CPSESP` -- matches segment `c` -- COBOL
|
||||
- `/repo/wfproc/WF001` -- matches segment `wfproc` -- COBOL
|
||||
- `/repo/docs/README` -- no matching segment -- skipped
|
||||
|
||||
### Decision Tree
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
A[getLanguageFromPath] --> B[getLanguageFromFilename]
|
||||
B --> C{Known extension?}
|
||||
C -->|Yes .cbl/.cob/.cobol/.cpy/...| D[Return COBOL]
|
||||
C -->|Yes .ts/.py/.java/...| E[Return other language]
|
||||
C -->|No match| F{Has extension?}
|
||||
|
||||
F -->|"Has dot in basename"| G[Return null]
|
||||
F -->|"No dot = extensionless"| H{GITNEXUS_COBOL_DIRS set?}
|
||||
|
||||
H -->|No| G
|
||||
H -->|Yes| I{Any path segment<br/>matches a configured dir?}
|
||||
|
||||
I -->|Yes| D
|
||||
I -->|No| G
|
||||
|
||||
style D fill:#e8f5e9,stroke:#2e7d32
|
||||
style G fill:#ffebee,stroke:#c62828
|
||||
```
|
||||
|
||||
### Implementation Detail
|
||||
|
||||
The `GITNEXUS_COBOL_DIRS` value is parsed once (on first call) and cached in a `Set<string>`:
|
||||
|
||||
```typescript
|
||||
// From gitnexus/src/core/ingestion/utils.ts
|
||||
const getCobolDirs = (): Set<string> => {
|
||||
if (_cobolDirs) return _cobolDirs;
|
||||
const raw = process.env.GITNEXUS_COBOL_DIRS;
|
||||
_cobolDirs = raw
|
||||
? new Set(raw.split(',').map(d => d.trim().toLowerCase()))
|
||||
: new Set();
|
||||
return _cobolDirs;
|
||||
};
|
||||
```
|
||||
|
||||
The path segment check splits the full path on `/` and tests each segment against the cached set.
|
||||
|
||||
## Copybook vs Program Classification
|
||||
|
||||
After a file is identified as COBOL, it must be classified as either a **program** (to be parsed for symbols) or a **copybook** (to be loaded into the copybook map for COPY expansion).
|
||||
|
||||
### Classification Rules
|
||||
|
||||
A COBOL file is classified as a **copybook** if ANY of these conditions is true:
|
||||
|
||||
1. It has a recognized copybook extension (`.cpy`, `.copy`, `.gnm`, `.fd`, `.wrk`, `.sel`, `.open`, `.close`, `.ini`, `.def`)
|
||||
2. It is an extensionless file whose path contains a directory segment matching one of: `c`, `copy`, `copybooks`, `copylib`, `cpy`
|
||||
|
||||
A file is classified as a **program** if:
|
||||
|
||||
1. It has a program extension (`.cbl`, `.cob`, `.cobol`), OR
|
||||
2. It is extensionless and does NOT match any copybook directory pattern
|
||||
|
||||
### Copybook Name Resolution
|
||||
|
||||
Copybook names are derived from the filename:
|
||||
|
||||
- Strip the extension (if any)
|
||||
- Convert to uppercase
|
||||
|
||||
Examples:
|
||||
- `c/CPSESP` -- name: `CPSESP`
|
||||
- `copy/workgrid.cpy` -- name: `WORKGRID`
|
||||
- `c/ANAZI.GNM` -- name: `ANAZI`
|
||||
|
||||
This name is used to resolve `COPY CPSESP.` statements during expansion.
|
||||
|
||||
## Source Files
|
||||
|
||||
- `gitnexus/src/core/ingestion/utils.ts` -- `getLanguageFromPath()`, `getLanguageFromFilename()`, `getCobolDirs()`
|
||||
- `gitnexus/src/core/ingestion/pipeline.ts` -- `isCobolCopybook()`, `getCopybookName()`, `COPYBOOK_EXTENSIONS`, `COBOL_PROGRAM_EXTENSIONS`
|
||||
@@ -1,193 +0,0 @@
|
||||
# COBOL Graph Model
|
||||
|
||||
This document describes the graph nodes and edges that GitNexus creates for COBOL codebases. The COBOL graph model is richer than most tree-sitter languages because it captures domain-specific constructs: file declarations, FD entries, data hierarchies, SQL tables, CICS maps, and cross-program contracts.
|
||||
|
||||
## Entity-Relationship Diagram
|
||||
|
||||
```mermaid
|
||||
erDiagram
|
||||
File ||--o{ Module : DEFINES
|
||||
File ||--o{ Function : DEFINES
|
||||
File ||--o{ Namespace : DEFINES
|
||||
File ||--o{ Record : DEFINES
|
||||
File ||--o{ Property : DEFINES
|
||||
File ||--o{ Const : DEFINES
|
||||
File ||--o{ CodeElement : DEFINES
|
||||
File ||--o{ Constructor : DEFINES
|
||||
File }o--o{ File : IMPORTS
|
||||
|
||||
Module ||--o{ Record : CONTAINS
|
||||
Module ||--o{ Constructor : CONTAINS
|
||||
Module }o--o{ CodeElement : ACCESSES
|
||||
Module }o--o{ Module : CALLS
|
||||
Module }o--o{ Module : CONTRACTS
|
||||
Module }o--o{ Property : RECEIVES
|
||||
|
||||
Record ||--o{ Property : CONTAINS
|
||||
Record ||--o{ Const : CONTAINS
|
||||
Record }o--o{ Record : REDEFINES
|
||||
|
||||
Property ||--o{ Property : CONTAINS
|
||||
Property ||--o{ Const : CONTAINS
|
||||
Property }o--o{ Property : REDEFINES
|
||||
Property }o--o{ CodeElement : RECORD_KEY_OF
|
||||
Property }o--o{ CodeElement : FILE_STATUS_OF
|
||||
|
||||
CodeElement ||--o{ CodeElement : CONTAINS
|
||||
CodeElement ||--o{ Record : CONTAINS
|
||||
|
||||
Function }o--o{ Function : CALLS
|
||||
```
|
||||
|
||||
## Node Types
|
||||
|
||||
| Node Type | COBOL Concept | Created From | Example |
|
||||
|-----------|--------------|--------------|---------|
|
||||
| `Module` | PROGRAM-ID | `PROGRAM-ID. BGTABFL` | Name: `BGTABFL`, description may include author and date |
|
||||
| `Function` | Paragraph | `PROCESS-RECORD.` at column 8 | Name: `PROCESS-RECORD` |
|
||||
| `Namespace` | Procedure section | `MAIN-LOGIC SECTION.` at column 8 | Name: `MAIN-LOGIC` |
|
||||
| `Record` | 01-level data item | `01 WK-EMPLOYEE.` | Description: `level:01 section:working-storage` |
|
||||
| `Property` | 02-49/66/77 data item | `05 WK-NAME PIC X(30).` | Description: `level:05 pic:X(30) section:working-storage` |
|
||||
| `Const` | 88-level condition | `88 WK-ACTIVE VALUE "A".` | Description: `level:88 values:A` |
|
||||
| `CodeElement` | SELECT, FD, SQL table, CICS map, cursor, transid | Various | Description varies by subtype |
|
||||
| `Constructor` | ENTRY point | `ENTRY "SUBPROG" USING WK-DATA` | Description: `entry params:WK-DATA` |
|
||||
|
||||
### CodeElement Subtypes
|
||||
|
||||
CodeElement is used for multiple COBOL constructs, distinguished by their description prefix:
|
||||
|
||||
| Subtype | ID Pattern | Description Format | Example |
|
||||
|---------|-----------|-------------------|---------|
|
||||
| File SELECT | `CodeElement:{path}:SELECT:{name}` | `select org:INDEXED access:DYNAMIC ...` | `SELECT MASTER-FILE` |
|
||||
| FD entry | `CodeElement:{path}:FD:{name}` | `fd record:{recordName}` | `FD MASTER-FILE` |
|
||||
| SQL table | `CodeElement:{path}:sql-table:{name}` | `sql-table op:SELECT` | Table `EMPLOYEES` |
|
||||
| SQL cursor | `CodeElement:{path}:sql-cursor:{name}` | `sql-cursor` | Cursor `C-EMPLOYEES` |
|
||||
| CICS map | `CodeElement:{path}:cics-map:{name}` | `cics-map cmd:SEND MAP` | Map `EMPMENU` |
|
||||
| CICS transid | `CodeElement:{path}:cics-transid:{name}` | `cics-transid cmd:START` | Transid `EMP1` |
|
||||
|
||||
## Edge Types
|
||||
|
||||
| Edge Type | Source | Target | Created By | Confidence | Example |
|
||||
|-----------|--------|--------|-----------|------------|---------|
|
||||
| `DEFINES` | File | any node | File defines its symbols | 1.0 | File -> Module `BGTABFL` |
|
||||
| `CALLS` | Function | Function | `PERFORM X [THRU Y]` | (via call-processor) | `PROCESS-RECORD` -> `CALC-TAX` |
|
||||
| `CALLS` | Module | Module | `CALL "BGTABUP"` | (via call-processor) | `BGTABFL` -> `BGTABUP` |
|
||||
| `CALLS` | Module | Module | `EXEC CICS LINK PROGRAM('X')` | (via call-processor) | `BGTABFL` -> `BGTABUP` |
|
||||
| `IMPORTS` | File | File | `COPY copybook` | (via import-processor) | Source file -> Copybook file |
|
||||
| `CONTAINS` | Module | Record | Data hierarchy root | 1.0 | `BGTABFL` -> `WK-EMPLOYEE` |
|
||||
| `CONTAINS` | Record | Property | Data hierarchy | 1.0 | `WK-EMPLOYEE` -> `WK-NAME` |
|
||||
| `CONTAINS` | Property | Property | Nested data items | 1.0 | `WK-ADDRESS` -> `WK-CITY` |
|
||||
| `CONTAINS` | Record/Property | Const | 88-level parent | 1.0 | `WK-STATUS` -> `WK-ACTIVE` |
|
||||
| `CONTAINS` | CodeElement (FD) | Record | FD record link | 1.0 | `FD:MASTER-FILE` -> `MASTER-RECORD` |
|
||||
| `CONTAINS` | CodeElement (SELECT) | CodeElement (FD) | SELECT-FD link | 0.9 | `SELECT:MASTER-FILE` -> `FD:MASTER-FILE` |
|
||||
| `CONTAINS` | Module | Constructor | ENTRY in module | 1.0 | `BGTABFL` -> `SUBPROG` |
|
||||
| `REDEFINES` | Record | Record | `01 X REDEFINES Y` | 1.0 | `WK-DATE-NUM` -> `WK-DATE-ALPHA` |
|
||||
| `REDEFINES` | Property | Property | `05 X REDEFINES Y` | 1.0 | `WK-CODE-NUM` -> `WK-CODE-ALPHA` |
|
||||
| `RECORD_KEY_OF` | Property | CodeElement (SELECT) | `RECORD KEY IS field` | 0.8 | `WK-EMP-ID` -> `SELECT:MASTER-FILE` |
|
||||
| `FILE_STATUS_OF` | Property | CodeElement (SELECT) | `FILE STATUS IS field` | 0.8 | `WK-FS` -> `SELECT:MASTER-FILE` |
|
||||
| `ACCESSES` | Module | CodeElement | EXEC SQL/CICS | 0.9 | `BGTABFL` -> `sql-table:EMPLOYEES` |
|
||||
| `RECEIVES` | Module | Property | `PROCEDURE USING` | 0.8 | `BGTABFL` -> `WK-INPUT-REC` |
|
||||
| `CONTRACTS` | Module | Module | Shared copybook detection | 0.9 | `BGTABFL` -> `BGTABUP` (via `CPSESP`) |
|
||||
|
||||
## Full Annotated Example
|
||||
|
||||
Given this COBOL program:
|
||||
|
||||
```cobol
|
||||
IDENTIFICATION DIVISION.
|
||||
PROGRAM-ID. EMPMAINT.
|
||||
AUTHOR. Development Team.
|
||||
|
||||
ENVIRONMENT DIVISION.
|
||||
INPUT-OUTPUT SECTION.
|
||||
FILE-CONTROL.
|
||||
SELECT EMP-FILE
|
||||
ASSIGN TO "EMPLOYEE.DAT"
|
||||
ORGANIZATION IS INDEXED
|
||||
ACCESS MODE IS DYNAMIC
|
||||
RECORD KEY IS EMP-ID
|
||||
FILE STATUS IS WS-FILE-STATUS.
|
||||
|
||||
DATA DIVISION.
|
||||
FILE SECTION.
|
||||
FD EMP-FILE.
|
||||
01 EMP-RECORD.
|
||||
05 EMP-ID PIC 9(6).
|
||||
05 EMP-NAME PIC X(30).
|
||||
|
||||
WORKING-STORAGE SECTION.
|
||||
01 WS-FLAGS.
|
||||
05 WS-FILE-STATUS PIC X(02).
|
||||
05 WS-EOF-FLAG PIC X(01).
|
||||
88 WS-EOF VALUE "Y".
|
||||
|
||||
LINKAGE SECTION.
|
||||
01 LK-SEARCH-KEY PIC 9(6).
|
||||
|
||||
PROCEDURE DIVISION USING LK-SEARCH-KEY.
|
||||
MAIN-LOGIC SECTION.
|
||||
MAIN-START.
|
||||
PERFORM OPEN-FILE
|
||||
PERFORM PROCESS-RECORDS
|
||||
PERFORM CLOSE-FILE
|
||||
STOP RUN.
|
||||
|
||||
OPEN-FILE.
|
||||
OPEN I-O EMP-FILE.
|
||||
|
||||
PROCESS-RECORDS.
|
||||
MOVE LK-SEARCH-KEY TO EMP-ID
|
||||
EXEC SQL
|
||||
SELECT EMP_SALARY INTO :WS-SALARY
|
||||
FROM EMPLOYEES
|
||||
WHERE EMP_ID = :EMP-ID
|
||||
END-EXEC
|
||||
CALL "EMPREPORT".
|
||||
|
||||
CLOSE-FILE.
|
||||
CLOSE EMP-FILE.
|
||||
```
|
||||
|
||||
The graph produced contains:
|
||||
|
||||
**Nodes:**
|
||||
- `Module`: EMPMAINT (description: `author:Development Team`)
|
||||
- `Namespace`: MAIN-LOGIC
|
||||
- `Function`: MAIN-START, OPEN-FILE, PROCESS-RECORDS, CLOSE-FILE
|
||||
- `Record`: EMP-RECORD, WS-FLAGS, LK-SEARCH-KEY
|
||||
- `Property`: EMP-ID, EMP-NAME, WS-FILE-STATUS, WS-EOF-FLAG
|
||||
- `Const`: WS-EOF (values: Y)
|
||||
- `CodeElement`: SELECT:EMP-FILE, FD:EMP-FILE, sql-table:EMPLOYEES
|
||||
- (COPY imports, if any, would produce File IMPORTS edges)
|
||||
|
||||
**Edges:**
|
||||
- `DEFINES`: File -> all nodes
|
||||
- `CONTAINS`: EMPMAINT -> EMP-RECORD, EMPMAINT -> WS-FLAGS, EMPMAINT -> LK-SEARCH-KEY
|
||||
- `CONTAINS`: EMP-RECORD -> EMP-ID, EMP-RECORD -> EMP-NAME
|
||||
- `CONTAINS`: WS-FLAGS -> WS-FILE-STATUS, WS-FLAGS -> WS-EOF-FLAG
|
||||
- `CONTAINS`: WS-EOF-FLAG -> WS-EOF
|
||||
- `CONTAINS`: FD:EMP-FILE -> EMP-RECORD
|
||||
- `CONTAINS`: SELECT:EMP-FILE -> FD:EMP-FILE
|
||||
- `CALLS`: MAIN-START -> OPEN-FILE, MAIN-START -> PROCESS-RECORDS, MAIN-START -> CLOSE-FILE
|
||||
- `CALLS`: EMPMAINT -> EMPREPORT (external CALL)
|
||||
- `ACCESSES`: EMPMAINT -> sql-table:EMPLOYEES
|
||||
- `RECEIVES`: EMPMAINT -> LK-SEARCH-KEY (PROCEDURE USING)
|
||||
- `RECORD_KEY_OF`: EMP-ID -> SELECT:EMP-FILE
|
||||
- `FILE_STATUS_OF`: WS-FILE-STATUS -> SELECT:EMP-FILE
|
||||
|
||||
## How COBOL Differs from Tree-Sitter Languages
|
||||
|
||||
| Aspect | COBOL | Tree-Sitter Languages |
|
||||
|--------|-------|----------------------|
|
||||
| Node variety | 8 types (Module, Function, Namespace, Record, Property, Const, CodeElement, Constructor) | Typically 4-6 (Function, Class, Method, Interface, Module, Const) |
|
||||
| Domain edges | RECORD_KEY_OF, FILE_STATUS_OF, ACCESSES, RECEIVES, CONTRACTS, REDEFINES | Primarily CALLS, IMPORTS, EXTENDS, IMPLEMENTS |
|
||||
| Data hierarchy | Deep CONTAINS chains (01 -> 05 -> 10 -> 88) | Flat class members |
|
||||
| Cross-program calls | CALL "name" + CICS LINK PROGRAM | Import-based resolution |
|
||||
| Contract detection | Shared COPY copybook between caller/callee | Not applicable |
|
||||
| Metadata | AUTHOR, DATE-WRITTEN on Module | JSDoc/docstring (not indexed) |
|
||||
|
||||
## Source Files
|
||||
|
||||
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `processCobolRegexOnly()`, node/edge emission logic
|
||||
- `gitnexus/src/core/ingestion/pipeline.ts` -- `detectCrossProgamContracts()` for CONTRACTS edges
|
||||
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- `CobolRegexResults` interface (all extracted data)
|
||||
@@ -1,261 +0,0 @@
|
||||
# COBOL Performance and Tuning
|
||||
|
||||
This document covers real-world benchmarks, worker pool configuration, memory management, known limitations, and troubleshooting for COBOL indexing.
|
||||
|
||||
## PROJECT-NAME Benchmark
|
||||
|
||||
The PROJECT-NAME project is a large Italian payroll system written in COBOL. It serves as the primary benchmark for COBOL indexing performance.
|
||||
|
||||
### Input
|
||||
|
||||
| Metric | Value |
|
||||
| --------------------------- | ---------------------------------------------------------------------------- |
|
||||
| Paths scanned | 14,217 |
|
||||
| Parseable files | 13,129 |
|
||||
| Total source size | 224 MB |
|
||||
| Chunks | 12 (at 20 MB budget) |
|
||||
| Copybooks loaded | 2,976 |
|
||||
| Copybooks used in expansion | 2,955 |
|
||||
| Key directories | `s/` (7773 programs), `c/` (3036 copybooks), `wfproc/` (1973 workflow files) |
|
||||
|
||||
### Output
|
||||
|
||||
| Metric | Value |
|
||||
| ---------------------- | ------ |
|
||||
| Graph nodes | 2.79M |
|
||||
| Graph edges | 5.67M |
|
||||
| Clusters (communities) | 16,679 |
|
||||
| Execution flows | 300 |
|
||||
|
||||
### Timing
|
||||
|
||||
| Phase | Duration |
|
||||
| ------------------------------- | ----------------- |
|
||||
| Total | ~251s |
|
||||
| KuzuDB write | 132s |
|
||||
| Full-text search indexing | 6.7s |
|
||||
| Regex extraction (avg per file) | ~1ms |
|
||||
| COPY expansion + deep indexing | Remainder (~112s) |
|
||||
|
||||
### Indexing Command
|
||||
|
||||
```bash
|
||||
cd /path/to/PROJECT-NAME
|
||||
GITNEXUS_COBOL_DIRS=s,c,wfproc GITNEXUS_VERBOSE=1 node --max-old-space-size=8192 \
|
||||
/path/to/gitnexus/dist/cli/index.js analyze --force
|
||||
```
|
||||
|
||||
## Open-Source Benchmarks
|
||||
|
||||
### CardDemo (AWS)
|
||||
|
||||
| Metric | Value |
|
||||
| ------ | ----- |
|
||||
| Graph nodes | 12,323 |
|
||||
| Graph edges | 8,893 |
|
||||
| Total time | 7.4s |
|
||||
|
||||
### ACAS
|
||||
|
||||
| Metric | Value |
|
||||
| ------ | ----- |
|
||||
| Graph nodes | 14,016 |
|
||||
| Graph edges | 15,452 |
|
||||
| Total time | 9.3s |
|
||||
|
||||
### Micro-Benchmark (Single-File Extraction)
|
||||
|
||||
| Metric | Value |
|
||||
| ------ | ----- |
|
||||
| Per-iteration | 0.65ms |
|
||||
| Throughput | ~382K lines/sec |
|
||||
|
||||
## Worker Pool Tuning
|
||||
|
||||
### Sub-Batch Size
|
||||
|
||||
The worker pool splits each worker's chunk into sub-batches to bound peak memory per `postMessage` serialization. COBOL repos use a smaller sub-batch size than the default:
|
||||
|
||||
| Parameter | Default | COBOL Mode |
|
||||
| --------------------- | ----------- | ------------------- |
|
||||
| Sub-batch size | 1,500 files | 200 files |
|
||||
| Per sub-batch timeout | 120s | 120s (configurable) |
|
||||
|
||||
**Why 200?** COBOL regex extraction + preprocessing takes ~1ms per file on average, but with COPY expansion and deep indexing the effective time is ~150ms per file. At sub-batch size 1500, that would be ~225s per sub-batch, exceeding the 120s timeout.
|
||||
|
||||
COBOL mode is activated automatically when `GITNEXUS_COBOL_DIRS` is set:
|
||||
|
||||
```typescript
|
||||
// From pipeline.ts
|
||||
const cobolSubBatch = process.env.GITNEXUS_COBOL_DIRS ? 200 : undefined;
|
||||
workerPool = createWorkerPool(workerUrl, undefined, cobolSubBatch);
|
||||
```
|
||||
|
||||
### Worker Count
|
||||
|
||||
Workers default to `min(8, cpus - 1)`. For COBOL repos, this is usually sufficient since regex extraction is CPU-bound but fast. The bottleneck is typically KuzuDB write, not extraction.
|
||||
|
||||
### Timeout Configuration
|
||||
|
||||
| Environment Variable | Default | Purpose |
|
||||
| ------------------------------------ | --------------- | --------------------------------------------------- |
|
||||
| `GITNEXUS_WORKER_TIMEOUT_MS` | 120,000 (2 min) | Per sub-batch processing timeout |
|
||||
| `GITNEXUS_WORKER_STARTUP_TIMEOUT_MS` | 60,000 (1 min) | Worker initialization timeout (tree-sitter loading) |
|
||||
|
||||
For COBOL-only repos, worker startup is faster because tree-sitter native modules are loaded lazily (skipped entirely if only COBOL files are present).
|
||||
|
||||
## Data Item Cap
|
||||
|
||||
### Configuration
|
||||
|
||||
```typescript
|
||||
const MAX_DATA_ITEMS_PER_FILE = 500;
|
||||
```
|
||||
|
||||
This constant appears in both `parse-worker.ts` (worker path) and `parsing-processor.ts` (sequential fallback).
|
||||
|
||||
### Rationale
|
||||
|
||||
Some COBOL programs, especially after COPY expansion, can have 10,000+ data items. At that scale:
|
||||
|
||||
- The in-memory relationship Map (for CONTAINS, REDEFINES, etc.) approaches the V8 16.7M entry limit across thousands of files
|
||||
- KuzuDB write time increases linearly with edge count
|
||||
- Most deep-nested items (level 20+) are rarely queried individually
|
||||
|
||||
### Impact
|
||||
|
||||
The cap truncates data items beyond the 500th in source order. Since 01-level Records appear first in COBOL source, the cap preserves:
|
||||
|
||||
- All 01-level record definitions
|
||||
- The most important 02-49 level items (those closest to the record root)
|
||||
- 88-level conditions associated with early items
|
||||
|
||||
To increase the cap for specific needs, modify the `MAX_DATA_ITEMS_PER_FILE` constant in both files.
|
||||
|
||||
## Memory Management
|
||||
|
||||
### COPY Expansion Breadth Guard
|
||||
|
||||
A per-file `MAX_TOTAL_EXPANSIONS = 500` limit prevents exponential blowup from diamond-shaped COPY graphs (e.g., N copybooks each containing N COPY statements). Once the limit is reached, further COPY statements in that file are left unexpanded. See [copy-expansion.md](copy-expansion.md) for details.
|
||||
|
||||
### COPY Expansion Memory
|
||||
|
||||
All copybook content is loaded upfront into a Map before chunk processing begins. For PROJECT-NAME:
|
||||
|
||||
- 2,976 copybooks, typically under 100MB total
|
||||
- The Map is shared (read-only) across chunk iterations
|
||||
- Per-chunk, the copybook map is merged with chunk file content (in case a chunk contains copybooks not in the pre-loaded set)
|
||||
- After all chunks are processed, the copybook map is freed (`cobolCopybookContents = undefined`)
|
||||
|
||||
### Chunk Budget
|
||||
|
||||
Source files are grouped into chunks of max 20MB (`CHUNK_BYTE_BUDGET`). Each chunk's lifecycle:
|
||||
|
||||
1. Read file content into memory
|
||||
2. Expand COPY statements (mutates content in-place)
|
||||
3. Dispatch to workers for extraction
|
||||
4. Workers return serialized results
|
||||
5. Merge results into graph
|
||||
6. Chunk content goes out of scope (GC reclaims)
|
||||
|
||||
This ensures only ~20MB of source + ~200-400MB of working memory (ASTs, extracted records, serialization) is active at any time.
|
||||
|
||||
### Shared Warning Deduplication
|
||||
|
||||
The `warnedCircular` set (used by the COPY expansion engine) is shared across all files in a chunk. This prevents the same circular copybook warning (e.g., `ANAZI includes itself`) from being logged thousands of times.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
| Limitation | Impact | Workaround |
|
||||
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
|
||||
| tree-sitter-cobol hangs on ~5% of files | Cannot use tree-sitter for COBOL | Regex-only extraction (current approach) |
|
||||
| Data item cap (500/file) | May miss deeply nested items in large programs | Increase `MAX_DATA_ITEMS_PER_FILE` in source |
|
||||
| Circular copybooks (ANAZI, ANDIP, QDIPE) | Self-referential includes cannot be expanded | Detected and skipped with warning |
|
||||
| wfproc/ files may not be pure COBOL | Workflow files may produce extraction noise | Exclude `wfproc` from `GITNEXUS_COBOL_DIRS` if problematic |
|
||||
| No MOVE DATA_FLOW edges yet | Data flow between variables not in graph | Reserved for future release |
|
||||
| Continuation line handling | Some complex multi-line continuations (especially in string literals spanning 3+ lines) may not merge correctly | Known edge case; affects <0.1% of lines |
|
||||
| Single-line EXEC blocks | `EXEC SQL SELECT ... END-EXEC` on one line is handled, but pathological nesting is not | Extremely rare in practice |
|
||||
| Extension case sensitivity | `.GNM` and `.gnm` are matched differently | Use the exact case from the codebase |
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### "COPY expansion failed"
|
||||
|
||||
```
|
||||
[pipeline] COPY expansion failed for s/BGTABFL: Cannot read properties of null
|
||||
```
|
||||
|
||||
**Cause:** A copybook referenced by a COPY statement cannot be found.
|
||||
|
||||
**Fix:**
|
||||
|
||||
1. Verify `GITNEXUS_COBOL_DIRS` includes the directory containing copybooks (typically `c`)
|
||||
2. Check that copybook filenames match the COPY target (case-insensitive, after stripping extensions)
|
||||
3. Ensure copybook files are not in `.gitignore`
|
||||
|
||||
### Worker sub-batch timeout
|
||||
|
||||
```
|
||||
Worker 3 sub-batch timed out after 120s (chunk: 200 items)
|
||||
```
|
||||
|
||||
**Cause:** A sub-batch took longer than the timeout. Typically happens when one file is extremely large (50,000+ lines after COPY expansion).
|
||||
|
||||
**Fix:** Increase the timeout:
|
||||
|
||||
```bash
|
||||
GITNEXUS_WORKER_TIMEOUT_MS=300000 gitnexus analyze
|
||||
```
|
||||
|
||||
### Memory errors (heap out of memory)
|
||||
|
||||
```
|
||||
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
|
||||
```
|
||||
|
||||
**Fix:** Increase Node.js heap size:
|
||||
|
||||
```bash
|
||||
node --max-old-space-size=16384 /path/to/gitnexus/dist/cli/index.js analyze
|
||||
```
|
||||
|
||||
For very large repos (>500MB source), consider `--max-old-space-size=32768`.
|
||||
|
||||
### Concurrent analyze corruption
|
||||
|
||||
**Rule:** Only ONE `gitnexus analyze` process should run at a time per repository. Concurrent writes to KuzuDB corrupt the database.
|
||||
|
||||
If corruption occurs:
|
||||
|
||||
```bash
|
||||
# Remove the KuzuDB directory and re-index
|
||||
rm -rf .gitnexus/kuzu
|
||||
gitnexus analyze --force
|
||||
```
|
||||
|
||||
### Slow KuzuDB write phase
|
||||
|
||||
The KuzuDB write phase (132s for PROJECT-NAME) is the bottleneck for large COBOL repos. This is proportional to the number of nodes and edges being written. Reducing `MAX_DATA_ITEMS_PER_FILE` or excluding non-essential directories from `GITNEXUS_COBOL_DIRS` can help.
|
||||
|
||||
### Verbose output
|
||||
|
||||
Enable verbose logging to see per-phase timing and statistics:
|
||||
|
||||
```bash
|
||||
GITNEXUS_VERBOSE=1 gitnexus analyze
|
||||
```
|
||||
|
||||
This outputs:
|
||||
|
||||
- Scan statistics (paths, parseable files, chunk count)
|
||||
- Worker pool configuration (worker count, sub-batch size)
|
||||
- COPY expansion statistics (copybooks loaded, files expanded)
|
||||
- Community and process detection results
|
||||
- Contract detection results
|
||||
|
||||
## Source Files
|
||||
|
||||
- `gitnexus/src/core/ingestion/workers/worker-pool.ts` -- `DEFAULT_SUB_BATCH_SIZE`, `SUB_BATCH_TIMEOUT_MS`, `WORKER_STARTUP_TIMEOUT_MS`
|
||||
- `gitnexus/src/core/ingestion/pipeline.ts` -- `CHUNK_BYTE_BUDGET`, COBOL sub-batch configuration, chunk lifecycle
|
||||
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `MAX_DATA_ITEMS_PER_FILE`, `processCobolRegexOnly()`
|
||||
- `gitnexus/src/core/ingestion/parsing-processor.ts` -- Sequential fallback `MAX_DATA_ITEMS_PER_FILE`
|
||||
@@ -1,206 +0,0 @@
|
||||
# COBOL Regex Extraction
|
||||
|
||||
The `extractCobolSymbolsWithRegex()` function in `cobol-preprocessor.ts` performs single-pass, state-machine-driven extraction of all COBOL symbols. This document describes the state machine, line processing flow, and every regex pattern used.
|
||||
|
||||
## State Machine: Division Tracking
|
||||
|
||||
The extractor tracks which COBOL division is currently being processed. Division transitions are detected by the `RE_DIVISION` pattern.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> null : Start of file
|
||||
null --> identification : IDENTIFICATION DIVISION
|
||||
identification --> environment : ENVIRONMENT DIVISION
|
||||
environment --> data : DATA DIVISION
|
||||
data --> procedure : PROCEDURE DIVISION
|
||||
|
||||
note right of identification
|
||||
Extracts: PROGRAM-ID, AUTHOR, DATE-WRITTEN
|
||||
end note
|
||||
note right of environment
|
||||
Extracts: SELECT ... ASSIGN ... (file declarations)
|
||||
end note
|
||||
note right of data
|
||||
Extracts: FD entries, data items (01-77, 88), COPY
|
||||
end note
|
||||
note right of procedure
|
||||
Extracts: paragraphs, sections, PERFORM, CALL,
|
||||
ENTRY, MOVE, EXEC SQL/CICS
|
||||
end note
|
||||
```
|
||||
|
||||
## State Machine: Data Section Tracking
|
||||
|
||||
Within the DATA DIVISION, a secondary state machine tracks the current section to tag data items with their origin.
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> unknown : DATA DIVISION entered
|
||||
unknown --> working_storage : WORKING-STORAGE SECTION
|
||||
unknown --> linkage : LINKAGE SECTION
|
||||
unknown --> file : FILE SECTION
|
||||
unknown --> local_storage : LOCAL-STORAGE SECTION
|
||||
working_storage --> linkage : LINKAGE SECTION
|
||||
working_storage --> file : FILE SECTION
|
||||
linkage --> working_storage : WORKING-STORAGE SECTION
|
||||
file --> working_storage : WORKING-STORAGE SECTION
|
||||
file --> linkage : LINKAGE SECTION
|
||||
local_storage --> working_storage : WORKING-STORAGE SECTION
|
||||
```
|
||||
|
||||
Within the ENVIRONMENT DIVISION, the `currentEnvSection` tracks whether we are in `INPUT-OUTPUT` or `CONFIGURATION` section. SELECT statement accumulation only occurs in `INPUT-OUTPUT`.
|
||||
|
||||
## Line Processing Flow
|
||||
|
||||
Each raw source line goes through this pipeline:
|
||||
|
||||
```
|
||||
Raw line
|
||||
|
|
||||
v
|
||||
Length < 7? ---------> Skip (flush pending if any)
|
||||
|
|
||||
v
|
||||
Indicator col 7
|
||||
|
|
||||
+-- '*' or '/' -----> Comment: skip entirely
|
||||
|
|
||||
+-- '-' ------------> Continuation: append to pending line
|
||||
|
|
||||
+-- other ----------> Normal: flush pending, strip inline comments (|),
|
||||
buffer as new pending logical line
|
||||
```
|
||||
|
||||
After all lines are processed, the final pending line is flushed, along with any accumulated SELECT statement, SORT/MERGE accumulator, and any open EXEC block (truncated file without `END-EXEC`).
|
||||
|
||||
### Inline Comment Stripping
|
||||
|
||||
Enterprise COBOL (particularly Italian dialect) uses the pipe character `|` as an inline comment marker. The `stripInlineComment()` helper is **quote-aware**: it tracks whether the scan position is inside a single- or double-quoted string and only treats `|` as a comment marker when outside quotes. Pipe characters inside string literals are preserved.
|
||||
|
||||
Free-format `*>` inline comment stripping uses the same quote-aware approach: the scanner walks character by character, toggling quote state, and only recognizes `*>` as a comment marker when not inside a quoted string.
|
||||
|
||||
### Patch Marker Handling
|
||||
|
||||
The `preprocessCobolSource()` function (run before extraction in the worker) replaces non-standard content in columns 1-6. Standard COBOL expects spaces or digit sequence numbers in this area. If any letter or `#` character is found, the entire sequence area is replaced with 6 spaces:
|
||||
|
||||
```
|
||||
Before: mzADD MOVE WK-AMT TO WK-TOTAL
|
||||
After: MOVE WK-AMT TO WK-TOTAL
|
||||
```
|
||||
|
||||
This preserves exact line count for position mapping.
|
||||
|
||||
## Regex Pattern Reference
|
||||
|
||||
All patterns are compiled once as module-level constants and reused across calls.
|
||||
|
||||
### Division and Section Detection
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_DIVISION` | `\b(IDENTIFICATION\|ENVIRONMENT\|DATA\|PROCEDURE)\s+DIVISION\b` | Division boundary | `PROCEDURE DIVISION` |
|
||||
| `RE_SECTION` | `\b(WORKING-STORAGE\|LINKAGE\|FILE\|LOCAL-STORAGE\|INPUT-OUTPUT\|CONFIGURATION)\s+SECTION\b` | Section boundary | `WORKING-STORAGE SECTION` |
|
||||
|
||||
### IDENTIFICATION DIVISION
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_PROGRAM_ID` | `\bPROGRAM-ID\.\s*([A-Z][A-Z0-9-]*)` | Program name | `PROGRAM-ID. BGTABFL` |
|
||||
| `RE_AUTHOR` | `^\s+AUTHOR\.\s*(.+)` | Author metadata | `AUTHOR. D. Smith` |
|
||||
| `RE_DATE_WRITTEN` | `^\s+DATE-WRITTEN\.\s*(.+)` | Date metadata | `DATE-WRITTEN. 2024-01-15` |
|
||||
|
||||
### ENVIRONMENT DIVISION
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_SELECT_START` | `\bSELECT\s+(?:OPTIONAL\s+)?([A-Z][A-Z0-9-]+)` | File SELECT start (with optional `SELECT OPTIONAL` support) | `SELECT MASTER-FILE`, `SELECT OPTIONAL TRANS-FILE` |
|
||||
|
||||
SELECT statements are accumulated across multiple lines until a period terminator is found, then parsed for ASSIGN, ORGANIZATION, ACCESS, RECORD KEY, and FILE STATUS clauses.
|
||||
|
||||
### DATA DIVISION
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_FD` | `^\s+FD\s+([A-Z][A-Z0-9-]+)` | File description | `FD MASTER-FILE` |
|
||||
| `RE_DATA_ITEM` | `^\s+(\d{1,2})\s+([A-Z][A-Z0-9-]+)\s*(.*)` | Data item (01-77) | `05 WK-NAME PIC X(30)` |
|
||||
| `RE_ANONYMOUS_REDEFINES` | `^\s+(\d{1,2})\s+REDEFINES\s+([A-Z][A-Z0-9-]+)` | Anonymous REDEFINES | `01 REDEFINES WK-REC` |
|
||||
| `RE_88_LEVEL` | `^\s+88\s+([A-Z][A-Z0-9-]+)\s+VALUES?\s+(?:ARE\s+)?(.+)` | Condition name | `88 WK-ACTIVE VALUE "Y"` |
|
||||
|
||||
The trailing clauses of `RE_DATA_ITEM` are parsed by `parseDataItemClauses()` for PIC, USAGE, OCCURS, and REDEFINES.
|
||||
|
||||
### PROCEDURE DIVISION
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_PROC_SECTION` | `^ ([A-Z][A-Z0-9-]+)\s+SECTION\.\s*$` | Procedure section header | ` MAIN-LOGIC SECTION.` |
|
||||
| `RE_PROC_PARAGRAPH` | `^ ([A-Z][A-Z0-9-]+)\.\s*$` | Paragraph header | ` PROCESS-RECORD.` |
|
||||
| `RE_PERFORM` | `\bPERFORM\s+([A-Z][A-Z0-9-]+)(?:\s+THRU\s+([A-Z][A-Z0-9-]+))?` | PERFORM call | `PERFORM CALC-TAX THRU CALC-TAX-EXIT` |
|
||||
| `RE_PROC_USING` | `\bPROCEDURE\s+DIVISION\s+USING\s+([\s\S]*?)(?:\.\|$)` | USING parameters | `PROCEDURE DIVISION USING WK-PARAM` |
|
||||
| `RE_ENTRY` | `\bENTRY\s+"([^"]+)"(?:\s+USING\s+([\s\S]*?))?(?:\.\|$)` | ENTRY point | `ENTRY "SUBPROG" USING WK-DATA` |
|
||||
| `RE_MOVE` | `\bMOVE\s+((?:CORRESPONDING\|CORR)\s+)?([A-Z][A-Z0-9-]+)\s+TO\s+(.+)` | MOVE statement (supports CORR abbreviation and multi-target) | `MOVE WK-NAME TO OUT-NAME`, `MOVE CORR WK-IN TO WK-OUT` |
|
||||
|
||||
The USING parameter list (`RE_PROC_USING`) is split on `\bRETURNING\b` before tokenization -- any RETURNING clause and everything after it is excluded from the parameter list (`.split(/\bRETURNING\b/i)[0]`).
|
||||
|
||||
Note: `RE_PROC_SECTION` and `RE_PROC_PARAGRAPH` require exactly 7 spaces of leading indentation (COBOL area A starting at column 8). This is the standard COBOL paragraph indentation.
|
||||
|
||||
### All-Division Patterns
|
||||
|
||||
These patterns are checked regardless of current division:
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_CALL` | `\bCALL\s+"([^"]+)"` | External program call | `CALL "BGTABUP"` |
|
||||
| `RE_COPY_UNQUOTED` | `\bCOPY\s+([A-Z][A-Z0-9-]+)(?:\s\|\.)` | COPY (unquoted) | `COPY CPSESP.` |
|
||||
| `RE_COPY_QUOTED` | `\bCOPY\s+"([^"]+)"(?:\s\|\.)` | COPY (quoted) | `COPY "WORKGRID.CPY".` |
|
||||
|
||||
### SORT/MERGE Support
|
||||
|
||||
| Constant | Purpose |
|
||||
|----------|---------|
|
||||
| `SORT_CLAUSE_NOISE` | Set of SORT/MERGE clause keywords filtered from USING/GIVING file lists: `ON`, `ASCENDING`, `DESCENDING`, `KEY`, `WITH`, `DUPLICATES`, `IN`, `ORDER`, `COLLATING`, `SEQUENCE`, `IS`, `THROUGH`, `THRU`, `INPUT`, `OUTPUT`, `PROCEDURE` |
|
||||
|
||||
SORT and MERGE statements are accumulated across multiple lines (like SELECT) until a period terminator is found, then parsed for USING/GIVING file lists and INPUT/OUTPUT PROCEDURE targets. The `flushSort()` helper encapsulates the flush-and-parse logic, mirroring the existing `flushSelect()` pattern. Both helpers are called at EOF to handle truncated files.
|
||||
|
||||
### GO TO Multi-Target
|
||||
|
||||
`RE_GOTO` captures all paragraph names in a `GO TO` statement, including the multi-target form `GO TO p1 p2 p3 DEPENDING ON x`. The captured group contains all target names (space-separated), which are split into individual targets. Each target produces a separate `gotos` entry.
|
||||
|
||||
### PROGRAM-ID Detection
|
||||
|
||||
PROGRAM-ID is detected regardless of the current division state. This handles sibling programs that appear after `END PROGRAM` and omit the `IDENTIFICATION DIVISION` header -- the extractor will still capture the PROGRAM-ID and push a new program boundary.
|
||||
|
||||
### EXEC Block Patterns
|
||||
|
||||
| Constant | Pattern | Purpose | Example Match |
|
||||
|----------|---------|---------|---------------|
|
||||
| `RE_EXEC_SQL_START` | `\bEXEC\s+SQL\b` | Start of EXEC SQL block | `EXEC SQL` |
|
||||
| `RE_EXEC_CICS_START` | `\bEXEC\s+CICS\b` | Start of EXEC CICS block | `EXEC CICS` |
|
||||
| `RE_END_EXEC` | `\bEND-EXEC\b` | End of EXEC block | `END-EXEC` |
|
||||
|
||||
EXEC blocks accumulate all lines between `EXEC SQL/CICS` and `END-EXEC`, then delegate to `parseExecSqlBlock()` or `parseExecCicsBlock()` for detailed extraction.
|
||||
|
||||
## Excluded Paragraph Names
|
||||
|
||||
The following names are excluded from paragraph detection to avoid false positives from division/section headers:
|
||||
|
||||
```
|
||||
DECLARATIVES, END, PROCEDURE, IDENTIFICATION,
|
||||
ENVIRONMENT, DATA, WORKING-STORAGE, LINKAGE,
|
||||
FILE, LOCAL-STORAGE, COMMUNICATION, REPORT,
|
||||
SCREEN, INPUT-OUTPUT, CONFIGURATION
|
||||
```
|
||||
|
||||
Additionally, paragraph candidates containing `DIVISION` or `SECTION` as substrings are excluded.
|
||||
|
||||
## MOVE Skip List (Figurative Constants)
|
||||
|
||||
MOVE statements where the source is a figurative constant are skipped:
|
||||
|
||||
```
|
||||
SPACES, ZEROS, ZEROES, LOW-VALUES, LOW-VALUE,
|
||||
HIGH-VALUES, HIGH-VALUE, QUOTES, QUOTE, ALL
|
||||
```
|
||||
|
||||
## Source Files
|
||||
|
||||
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- `preprocessCobolSource()`, `extractCobolSymbolsWithRegex()`, all regex constants
|
||||
@@ -1,300 +0,0 @@
|
||||
# Using GitNexus across gRPC microservices
|
||||
|
||||
## When to use this guide
|
||||
|
||||
This guide is for teams whose product lives in **several separate Git repositories** — one per service — and whose services talk to each other over **gRPC** (possibly alongside HTTP and message topics). GitNexus indexes each repo independently, then a _group_ stitches the per-repo indexes into a single cross-repo view that the `impact`, `query`, and `context` tools can traverse. If your services live in one monorepo, much of this still applies — set each service as a member of a group and use the `service` prefix to scope queries — but the walkthrough assumes the harder multi-repo case.
|
||||
|
||||
## Mental model
|
||||
|
||||
- Each repository has its own `.gitnexus/` index (a LadybugDB graph of symbols, relationships, processes). `gitnexus analyze` in each repo produces that index completely independently.
|
||||
- A **group** is a higher-level construct stored at `~/.gitnexus/groups/<group>/` that references the per-repo indexes by their registry name.
|
||||
- Sync-time extractors walk each member repo and emit **contracts** — provider or consumer records keyed by a canonical `contractId` (`grpc::auth.AuthService/Login`, `http::GET::/orders`, etc.).
|
||||
- The sync step matches providers and consumers that share a `contractId` and writes **cross-links** to `<groupDir>/contracts.json`. Those cross-links are what lets `impact({repo: "@<group>", target: "X"})` hop from one repo into another.
|
||||
- Contracts come from three places: automatic contract extractors (`grpc-extractor`, `http-route-extractor`, `topic-extractor`), a manifest escape hatch (`config.links` in `group.yaml`), and — for same-name symbol matches where no contract is declared — the exact-match matching cascade in [`matching.ts`](../../gitnexus/src/core/group/matching.ts).
|
||||
- Each repo stays editable and re-indexable on its own. Re-run `gitnexus analyze` in a repo when it changes, then `gitnexus group sync <group>` to refresh `contracts.json`. `gitnexus group status` reports which members are stale.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- GitNexus installed and runnable as `gitnexus` or `npx gitnexus` (see the root [README.md](../../README.md)).
|
||||
- Each service repository checked out locally. No requirement that they share a parent directory — the group references them by registry name.
|
||||
- Write access to `~/.gitnexus/` (the default gitnexus home; see `getDefaultGitnexusDir` in [`storage.ts`](../../gitnexus/src/core/group/storage.ts)).
|
||||
|
||||
## Step-by-step walkthrough
|
||||
|
||||
The example uses three services — a TypeScript API gateway, a Go orders service, and a Python inventory service — with gRPC between them. The gateway is an `orders` consumer; the orders service is both an `orders` provider and an `inventory` consumer; the inventory service is an `inventory` provider.
|
||||
|
||||
### 1. Index each repository
|
||||
|
||||
Run `analyze` from inside each service repo (or pass the path). The CLI surface lives in [`gitnexus/src/cli/analyze.ts`](../../gitnexus/src/cli/analyze.ts) and is wired in [`gitnexus/src/cli/index.ts`](../../gitnexus/src/cli/index.ts).
|
||||
|
||||
```bash
|
||||
cd ~/code/gateway && npx gitnexus analyze
|
||||
cd ~/code/orders && npx gitnexus analyze
|
||||
cd ~/code/inventory && npx gitnexus analyze
|
||||
```
|
||||
|
||||
Useful flags:
|
||||
|
||||
- `--force` — reindex even if up to date.
|
||||
- `--embeddings` — generate embedding vectors (needed only if you want semantic search; the exact-match cross-repo cascade does **not** need them).
|
||||
- `--name <alias>` — register the repo under a specific alias when two repos share a basename (e.g. two `api/` folders).
|
||||
- `--skip-git` — index a checkout that isn't a git repo.
|
||||
|
||||
Each run writes a `.gitnexus/` folder in the repo and registers the repo in `~/.gitnexus/registry.json`. Confirm with `npx gitnexus list`.
|
||||
|
||||
### 2. Author `group.yaml`
|
||||
|
||||
Create the group directory and edit the config. Either use the CLI scaffolder or write the file directly — both produce the same shape consumed by [`config-parser.ts`](../../gitnexus/src/core/group/config-parser.ts).
|
||||
|
||||
```bash
|
||||
npx gitnexus group create payments-platform
|
||||
# or manually:
|
||||
mkdir -p ~/.gitnexus/groups/payments-platform
|
||||
$EDITOR ~/.gitnexus/groups/payments-platform/group.yaml
|
||||
```
|
||||
|
||||
Minimal working `group.yaml`:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
name: payments-platform
|
||||
description: Gateway + orders + inventory (gRPC)
|
||||
|
||||
repos:
|
||||
gateway: gateway
|
||||
orders: orders
|
||||
inventory: inventory
|
||||
|
||||
# Only add explicit links when the automatic extractors miss something —
|
||||
# see "When automatic extraction isn't enough" below.
|
||||
links: []
|
||||
|
||||
packages: {}
|
||||
|
||||
detect:
|
||||
http: true
|
||||
grpc: true
|
||||
topics: true
|
||||
shared_libs: true
|
||||
embedding_fallback: false
|
||||
|
||||
matching:
|
||||
bm25_threshold: 0.7
|
||||
embedding_threshold: 0.65
|
||||
max_candidates_per_step: 3
|
||||
# Exclude noisy paths from cross-link matching (contracts are still extracted)
|
||||
exclude_links_paths: [/ping, /health, /healthcheck]
|
||||
exclude_links_param_only_paths: true
|
||||
```
|
||||
|
||||
Field notes (schema in [`types.ts`](../../gitnexus/src/core/group/types.ts)):
|
||||
|
||||
- `version` — must be `1`. The parser rejects anything else.
|
||||
- `name` — required; used for the group directory name and all CLI / MCP calls.
|
||||
- `repos` — a mapping from **group path** (a logical name you choose; can be a hierarchy like `backend/orders`) to **registry name** (the name shown by `npx gitnexus list`). Both sides appear throughout the tooling: contract rows use the group path; `@<group>/<groupPath>` routes tools to a single member.
|
||||
- `links` — optional manifest escape hatch, one entry per explicit cross-repo contract. Validated by the parser: `from` and `to` must be known repo paths, `type` must be one of `http | grpc | topic | lib | custom`, and `role` must be `provider | consumer`.
|
||||
- `detect` — toggles per extractor family. Defaults (set in `config-parser.ts`) turn `http`, `grpc`, `topics`, and `shared_libs` on; disable the ones you don't use to speed up sync.
|
||||
- `matching` — thresholds for the matching cascade. The exact match is always run; other strategies depend on indexer state. Two optional fields reduce false-positive cross-links in large groups:
|
||||
- `exclude_links_paths` — list of HTTP paths to exclude from cross-link matching (default `[]`). Contracts at these paths are still extracted and visible in the registry, but they don't produce cross-repo links. Useful for health-check endpoints (`/ping`, `/health`) that every service exposes. Trailing slashes are normalized.
|
||||
- `exclude_links_param_only_paths` — when `true`, exclude routes where every segment is `{param}` (e.g. `/{param}`, `/{param}/{param}`) from cross-link matching (default `false`). Mixed routes like `/users/{param}` are not affected.
|
||||
|
||||
### 3. Sync the group
|
||||
|
||||
```bash
|
||||
npx gitnexus group sync payments-platform --verbose
|
||||
```
|
||||
|
||||
What this does (see [`sync.ts`](../../gitnexus/src/core/group/sync.ts)):
|
||||
|
||||
1. Opens each member's per-repo LadybugDB.
|
||||
2. Runs the HTTP, gRPC, and topic extractors against the source files.
|
||||
3. Applies manifest `links` through [`manifest-extractor.ts`](../../gitnexus/src/core/group/extractors/manifest-extractor.ts).
|
||||
4. Runs the exact-match cascade, joining providers and consumers that share a normalized `contractId`.
|
||||
5. Writes `contracts.json` in the group directory.
|
||||
|
||||
Flags:
|
||||
|
||||
- `--exact-only` — stop after the exact cascade; skip BM25 and embedding fallback.
|
||||
- `--skip-embeddings` — run exact plus BM25 but not embedding-based matching.
|
||||
- `--allow-stale` — don't warn if a member's index is stale.
|
||||
- `--json` — machine-readable output.
|
||||
|
||||
The same operation is available over MCP as `group_sync({ name: "payments-platform" })` — see [`tools.ts`](../../gitnexus/src/mcp/tools.ts).
|
||||
|
||||
### 4. Inspect the registry
|
||||
|
||||
Use `gitnexus group contracts` for the CLI view or read the `gitnexus://group/<name>/contracts` MCP resource for the same data.
|
||||
|
||||
```bash
|
||||
npx gitnexus group contracts payments-platform --type grpc --json
|
||||
```
|
||||
|
||||
A shortened response:
|
||||
|
||||
```json
|
||||
{
|
||||
"contracts": [
|
||||
{
|
||||
"contractId": "grpc::orders.OrderService/PlaceOrder",
|
||||
"type": "grpc",
|
||||
"role": "provider",
|
||||
"repo": "orders",
|
||||
"symbolRef": { "filePath": "internal/grpc/order_server.go", "name": "RegisterOrderServiceServer" },
|
||||
"confidence": 0.8,
|
||||
"meta": { "service": "OrderService", "method": "PlaceOrder", "source": "go_register" }
|
||||
},
|
||||
{
|
||||
"contractId": "grpc::orders.OrderService/PlaceOrder",
|
||||
"type": "grpc",
|
||||
"role": "consumer",
|
||||
"repo": "gateway",
|
||||
"symbolRef": { "filePath": "src/clients/orders.ts", "name": "OrderServiceClient" },
|
||||
"confidence": 0.75,
|
||||
"meta": { "service": "OrderService", "source": "ts_generated_client" }
|
||||
}
|
||||
],
|
||||
"crossLinks": [
|
||||
{
|
||||
"from": { "repo": "gateway", "symbolUid": "…", "symbolRef": { "filePath": "src/clients/orders.ts", "name": "OrderServiceClient" } },
|
||||
"to": { "repo": "orders", "symbolUid": "…", "symbolRef": { "filePath": "internal/grpc/order_server.go", "name": "RegisterOrderServiceServer" } },
|
||||
"type": "grpc",
|
||||
"contractId": "grpc::orders.OrderService/PlaceOrder",
|
||||
"matchType": "exact",
|
||||
"confidence": 1.0
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Staleness of the underlying indexes shows up in `npx gitnexus group status payments-platform` or the `gitnexus://group/<name>/status` resource.
|
||||
|
||||
### 5. Run cross-repo impact with `@<group>` routing
|
||||
|
||||
From any shell (you do **not** have to `cd` into a member repo), the normal `impact` / `query` / `context` tools accept `repo: "@<group>"` to fan out across all members, or `repo: "@<group>/<memberPath>"` to target one member. Routing is implemented in [`resolve-at-member.ts`](../../gitnexus/src/core/group/resolve-at-member.ts) and described in [`tools.ts`](../../gitnexus/src/mcp/tools.ts).
|
||||
|
||||
Example MCP calls:
|
||||
|
||||
```json
|
||||
{"tool": "impact", "arguments": {
|
||||
"repo": "@payments-platform/orders",
|
||||
"target": "PlaceOrder",
|
||||
"direction": "upstream",
|
||||
"crossDepth": 2
|
||||
}}
|
||||
```
|
||||
|
||||
```json
|
||||
{"tool": "query", "arguments": {
|
||||
"repo": "@payments-platform",
|
||||
"query": "retry logic around PlaceOrder"
|
||||
}}
|
||||
```
|
||||
|
||||
The CLI equivalents still exist for scripting:
|
||||
|
||||
```bash
|
||||
npx gitnexus group impact payments-platform \
|
||||
--repo orders --target PlaceOrder --direction upstream --cross-depth 2
|
||||
```
|
||||
|
||||
Phase 1 walks within the anchor member; Phase 2 hops across the Contract Bridge wherever a cross-link endpoint matches an impacted symbol. See [`cross-impact.ts`](../../gitnexus/src/core/group/cross-impact.ts) for the bridge query.
|
||||
|
||||
## How gRPC extraction works
|
||||
|
||||
`GrpcExtractor` ([`grpc-extractor.ts`](../../gitnexus/src/core/group/extractors/grpc-extractor.ts)) runs two passes per member repo:
|
||||
|
||||
1. **Proto map.** Every `**/*.proto` file is parsed to enumerate `service Foo { rpc Bar(...) }` blocks and (transitively) resolve the package name. Each RPC method becomes a provider contract with `contractId = grpc::<package>.<Service>/<Method>` and `confidence = 0.85`. Parsing uses the vendored `tree-sitter-proto` grammar when available and falls back to a length-preserving manual parser (`extractServiceBlocks`) otherwise, so `.proto` extraction works on platforms where the grammar fails to build.
|
||||
2. **Source scan.** Every source file whose extension matches [`GRPC_SCAN_GLOB`](../../gitnexus/src/core/group/extractors/grpc-patterns/index.ts) is parsed by its language plugin:
|
||||
|
||||
| Language | Provider signal | Consumer signal |
|
||||
|----------|-----------------|-----------------|
|
||||
| Go ([`go.ts`](../../gitnexus/src/core/group/extractors/grpc-patterns/go.ts)) | `pb.RegisterXxxServer(...)`, `pb.UnimplementedXxxServer` embedded in struct | `pb.NewXxxClient(conn)` |
|
||||
| Java ([`java.ts`](../../gitnexus/src/core/group/extractors/grpc-patterns/java.ts)) | `extends XxxServiceGrpc.XxxServiceImplBase` (with or without `@GrpcService`) | `XxxServiceGrpc.newBlockingStub(...)`, `newStub(...)` |
|
||||
| Python ([`python.ts`](../../gitnexus/src/core/group/extractors/grpc-patterns/python.ts)) | `add_XxxServicer_to_server(...)` (bare or `_pb2_grpc.` attribute form) | `XxxStub(channel)` (ignores `Mock`/`Test`/`Fake`/`Stub`) |
|
||||
| Node / TS ([`node.ts`](../../gitnexus/src/core/group/extractors/grpc-patterns/node.ts)) | NestJS `@GrpcMethod('Service','Method')` | `@GrpcClient` field typed `XxxServiceClient`, `client.getService<X>('Service')`, `new XxxServiceClient(...)`, `new foo.bar.XxxService(...)` in files that call `loadPackageDefinition` |
|
||||
|
||||
For each source-scan detection the extractor looks up the short service name in the proto map and picks:
|
||||
|
||||
- `grpc::<package>.<Service>/<Method>` when a method is named and the service resolves against the proto map,
|
||||
- `grpc::<package>.<Service>/*` (wildcard) when only the service is known, or
|
||||
- `grpc::<ServiceName>/*` when no `.proto` is available at all.
|
||||
|
||||
Provider detections land at confidence 0.8 (with proto) or 0.65 (without); consumers at 0.75 or 0.55. NestJS `@GrpcMethod` is fixed at 0.8 because the decorator is self-describing.
|
||||
|
||||
### Matching
|
||||
|
||||
`matching.ts` lowercases the package/service segment before comparing contract ids, so bindings that capitalize names differently (`auth.AuthService` vs `auth.authservice`) still match. Method names are compared case-sensitively because gRPC's wire path is case-sensitive. Service-only wildcards (`grpc::pkg.Svc/*`) match any method on the same service during cross-linking.
|
||||
|
||||
### Known limitations
|
||||
|
||||
- **Ambiguous proto resolution.** If a short service name exists in more than one `.proto` file and the source-scan hit can't be narrowed down by shared directory segments (`resolveProtoConflict` refuses to guess), the extractor skips contract emission and logs a warning.
|
||||
- **Proto packages must be resolvable locally.** Transitive imports that point outside the repo produce an empty package segment, which means the contract id collapses to `grpc::<Service>/<Method>`. Cross-repo matches still work as long as both sides agree on the empty package.
|
||||
- **Rewrite rules are not implemented.** If the provider repo writes `grpc::orders.OrderService/PlaceOrder` and the consumer repo writes `grpc::orderspb.OrderService/PlaceOrder`, they won't cross-link automatically. Use `config.links` to declare the correspondence (see below).
|
||||
- **One sync = one snapshot.** Contracts are extracted against the indexed snapshot of each repo. Re-index first, then re-sync; the `status` command and resource surface staleness.
|
||||
|
||||
## When automatic extraction isn't enough
|
||||
|
||||
The escape hatch is the `links` list in `group.yaml`, handled by [`ManifestExtractor`](../../gitnexus/src/core/group/extractors/manifest-extractor.ts). Each entry is a **one-directional** provider/consumer declaration:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
name: payments-platform
|
||||
repos:
|
||||
gateway: gateway
|
||||
orders: orders
|
||||
inventory: inventory
|
||||
|
||||
links:
|
||||
# Explicit gRPC method: use when naming mismatches stop the
|
||||
# automatic matcher from cross-linking.
|
||||
- from: gateway
|
||||
to: orders
|
||||
type: grpc
|
||||
contract: OrderService/PlaceOrder
|
||||
role: consumer
|
||||
|
||||
# Service-level link when you don't want to enumerate methods.
|
||||
- from: orders
|
||||
to: inventory
|
||||
type: grpc
|
||||
contract: InventoryService
|
||||
role: consumer
|
||||
|
||||
# Works for HTTP too — use `METHOD::/path` form for the exact
|
||||
# handler, or just `/path` for a method-agnostic wildcard.
|
||||
- from: gateway
|
||||
to: orders
|
||||
type: http
|
||||
contract: POST::/orders
|
||||
role: consumer
|
||||
```
|
||||
|
||||
What the manifest extractor does (see [`manifest-extractor.ts`](../../gitnexus/src/core/group/extractors/manifest-extractor.ts)):
|
||||
|
||||
1. Builds a canonical `contractId` with `buildContractId` — the same canonicalization used by the automatic extractors, so manifest links cross-match automatic contracts on the other side.
|
||||
2. Tries to resolve each side to a real graph symbol (the `Route` node for HTTP, a `Function|Method` / `Class|Interface` for gRPC, a `Package|Module` for `lib`).
|
||||
3. If resolution fails, falls back to a deterministic synthetic uid (`manifest::<repo>::<contractId>`) so both sides still line up in cross-impact — name-only links still work when the symbol isn't in the graph.
|
||||
4. Emits both a provider and a consumer `StoredContract` (confidence `1.0`, `source: "manifest"`) and a `CrossLink` with `matchType: "manifest"`.
|
||||
|
||||
Use `links` for exactly the cases the extractor can't infer: different package names across repos (see #701), hand-rolled transports, cases where the provider repo isn't checked out locally but you still want a record, or any contract whose provider and consumer simply don't share a surface the extractors know how to pattern-match.
|
||||
|
||||
History: the manifest extractor used to be silently skipped by the sync pipeline; that was fixed in [#827](https://github.com/abhigyanpatwari/GitNexus/pull/827) (tracking issue #826). If you ever see `config.links` with zero cross-links in `contracts.json`, make sure you're on a build that includes that fix, then re-run `group sync`.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
1. **`contracts.json` is empty after a sync.** Either no member repo contained a recognizable gRPC pattern, or the extractors are disabled in `detect`. Confirm `detect.grpc: true` and re-run with `--verbose`.
|
||||
2. **A known provider/consumer pair doesn't cross-link.** Most common cause: the package segment differs. Check the raw contract ids with `gitnexus group contracts <name> --unmatched` — if you see two same-method contracts with different package prefixes, add a manifest `links:` entry to bridge them (no automatic rewrite rules yet).
|
||||
3. **`matchType: "manifest"` is missing entirely.** The extractor needs `config.links` to be non-empty and the sync pipeline to actually call it — verify you're on a post-#827 build. Empty contract rows for manifest links usually mean `resolveSymbol` couldn't find a graph match; the synthetic uid still lets cross-impact work, it just won't carry a file path.
|
||||
4. **Ambiguous proto warnings.** Look for `[grpc-extractor] Ambiguous proto resolution` in the sync logs; that means a service name exists in multiple `.proto` files under the same repo and the path-distance heuristic couldn't pick a winner. Resolve by renaming the service or declaring the intended pairing in `config.links`.
|
||||
5. **Cross-impact says "stale".** Both sides need a fresh per-repo index _and_ a fresh group sync. Order matters: `gitnexus analyze` in each changed repo, then `gitnexus group sync <name>`. Use `gitnexus group status <name>` to see which side is behind.
|
||||
|
||||
## Related docs and references
|
||||
|
||||
- [AGENTS.md](../../AGENTS.md) — authoritative list of MCP tools and resources, including group-mode routing and the `gitnexus://group/…` resources.
|
||||
- [ARCHITECTURE.md](../../ARCHITECTURE.md) — overall data flow and the call-resolution DAG that the per-repo indexer uses.
|
||||
- [`gitnexus/src/core/group/`](../../gitnexus/src/core/group/) — `service.ts`, `sync.ts`, `config-parser.ts`, `matching.ts`.
|
||||
- [`gitnexus/src/core/group/extractors/grpc-extractor.ts`](../../gitnexus/src/core/group/extractors/grpc-extractor.ts) and [`grpc-patterns/`](../../gitnexus/src/core/group/extractors/grpc-patterns/) — gRPC detection.
|
||||
- [`gitnexus/src/core/group/extractors/manifest-extractor.ts`](../../gitnexus/src/core/group/extractors/manifest-extractor.ts) — the `config.links` escape hatch.
|
||||
- [`gitnexus/src/mcp/tools.ts`](../../gitnexus/src/mcp/tools.ts) — MCP tool schemas (`group_list`, `group_sync`, plus `@<group>` routing on `impact` / `query` / `context`).
|
||||
- [`gitnexus/src/cli/group.ts`](../../gitnexus/src/cli/group.ts) — CLI command definitions and flags.
|
||||
- Upstream issues: [#701](https://github.com/abhigyanpatwari/GitNexus/issues/701), [#826](https://github.com/abhigyanpatwari/GitNexus/issues/826), [#906](https://github.com/abhigyanpatwari/GitNexus/issues/906).
|
||||
@@ -1,326 +0,0 @@
|
||||
---
|
||||
title: "feat: Complete COBOL language feature coverage for maximum knowledge graph value"
|
||||
type: feat
|
||||
status: active
|
||||
date: 2026-03-26
|
||||
origin: Feature audit from v3-integration-architect agent (session 8642401e)
|
||||
---
|
||||
|
||||
## Enhancement Summary
|
||||
|
||||
**Deepened on:** 2026-03-26
|
||||
**Research agents used:** COBOL expert (Phase 1+2), graph value analyst, codebase explorer
|
||||
**Sections enhanced:** Phase 1 (5 features), Phase 2 (4 features), graph value ranking
|
||||
|
||||
### Key Improvements from Research
|
||||
1. **CALL USING** is the #1 highest-value edge type (9.2/10) — fixes ~40% of missing caller references
|
||||
2. **EXEC DLI** requires dual-interface support (EXEC DLI + CBLTDLI CALL) for full IMS coverage
|
||||
3. **DECLARATIVES** is lowest-risk Phase 2 item — existing section/paragraph detection already captures structure
|
||||
4. **SET TO TRUE** accounts for 80-90% of all SET statements — prioritize this form
|
||||
5. **INSPECT** needs multi-line accumulator (like SORT) — can span 5+ continuation lines
|
||||
6. **Graph value ranking**: cobol-call-using (9.2) > cobol-error-handler (9.0) > dli-gu (8.2) > cobol-string (6.2)
|
||||
|
||||
### New Edge Cases Discovered
|
||||
- CALL USING supports mixed modes: `USING BY REFERENCE WS-A BY CONTENT WS-B BY VALUE WS-C`
|
||||
- CALL USING `ADDRESS OF` and `OMITTED` must be filtered from parameter lists
|
||||
- EXEC DLI can have multiple SEGMENT levels in hierarchical retrieval (use matchAll)
|
||||
- DECLARATIVES can have multiple USE sections (one per file + catch-all for INPUT/OUTPUT/I-O/EXTEND)
|
||||
- INSPECT TALLYING can have multiple counters in a single statement
|
||||
- STRING/UNSTRING can span multiple lines (need accumulator pattern)
|
||||
|
||||
---
|
||||
|
||||
# Complete COBOL Language Feature Coverage
|
||||
|
||||
## Overview
|
||||
|
||||
Implement the remaining 25 unhandled COBOL language features and fix 10 partial features to achieve ~95% coverage (up from 71.9%). The goal is to build the richest possible knowledge graph from COBOL codebases, enabling a future `modernize` MCP command (out of scope for this plan) that would use the graph to assist with COBOL-to-modern-language migration.
|
||||
|
||||
## Problem Statement
|
||||
|
||||
The COBOL processor currently handles 54 of 89 applicable language features (71.9%). The 25 unhandled features represent real data loss in the knowledge graph:
|
||||
- **Cross-program data flow** is invisible (CALL ... USING parameters not extracted)
|
||||
- **IMS/DB programs** produce empty graphs (EXEC DLI not recognized)
|
||||
- **String transformation logic** is invisible (STRING/UNSTRING/INSPECT not tracked)
|
||||
- **SQL copybook dependencies** are missing (EXEC SQL INCLUDE not mapped)
|
||||
- **Error handling flows** are lost (DECLARATIVES/USE AFTER not captured)
|
||||
|
||||
## Proposed Solution
|
||||
|
||||
Implement features in 4 phases, ordered by graph value density (edges created per LOC of implementation). Each phase is independently shippable and testable.
|
||||
|
||||
## Technical Approach
|
||||
|
||||
### Phase 1: High-Value Data Flow Edges (~150 LOC, ~8 new edge types)
|
||||
|
||||
The highest-ROI features: they create new ACCESSES and IMPORTS edges that directly improve impact analysis.
|
||||
|
||||
**Critical research finding**: Multi-line statement accumulation is the dominant challenge. CALL USING, STRING/UNSTRING, and multi-line data item clauses all span multiple lines in production COBOL. The free-format path processes each line independently — these features need statement accumulators (like SORT/SELECT) or the free-format path needs multi-line awareness. Estimated LOC increased from 110 to 150 to account for accumulator infrastructure.
|
||||
|
||||
#### 1.1 EXEC SQL INCLUDE -> IMPORTS edges
|
||||
- **File:** `cobol-preprocessor.ts` (parseExecSqlBlock)
|
||||
- **What:** Detect `INCLUDE` as the operation, extract member name, emit as a `copies[]` entry
|
||||
- **Graph:** IMPORTS edge from File to included copybook/SQLCA with reason `sql-include`
|
||||
- **Tests:** Unit test for `EXEC SQL INCLUDE SQLCA END-EXEC` and `EXEC SQL INCLUDE CUSTCOPY END-EXEC`
|
||||
|
||||
**Research insights (EXEC SQL INCLUDE):**
|
||||
- DB2 member names can contain underscores: `EXEC SQL INCLUDE CUST_TBL_DCL END-EXEC` — regex must use `[A-Z][A-Z0-9_-]+`
|
||||
- Quoted literal form: `EXEC SQL INCLUDE 'DBRMLIB.MEMBER' END-EXEC` (z/OS PDS qualified name)
|
||||
- SQLCA/SQLDA are DB2 builtins — won't resolve to repo files. Emit unresolved IMPORTS edge (still valuable)
|
||||
- No REPLACING support on EXEC SQL INCLUDE (unlike COPY)
|
||||
- Add `INCLUDE` to `OP_MAP` in `parseExecSqlBlock`; extract member via `RE_SQL_INCLUDE = /^INCLUDE\s+(?:'([^']+)'|"([^"]+)"|([A-Z][A-Z0-9_-]+))/i`
|
||||
|
||||
#### 1.2 CALL ... USING parameter extraction -> ACCESSES edges (Graph value: 9.2/10)
|
||||
- **File:** `cobol-preprocessor.ts` (processLogicalLine CALL section)
|
||||
- **What:** After capturing CALL target, scan for USING clause. Extract parameter names (reuse USING_KEYWORDS filter). Store as `calls[].parameters: string[]`
|
||||
- **Interface:** Add `parameters?: string[]` to calls array type in CobolRegexResults
|
||||
- **File:** `cobol-processor.ts` (CALL edge block)
|
||||
- **Graph:** For each USING parameter, create ACCESSES edge from caller to data item Property node with reason `cobol-call-using`
|
||||
- **Tests:** `CALL 'AUDITLOG' USING CUST-ID WS-AMOUNT` -> 2 ACCESSES edges
|
||||
|
||||
**Research insights (CALL USING forms):**
|
||||
- Mixed modes: `CALL 'PGM' USING BY REFERENCE WS-A BY CONTENT WS-B BY VALUE WS-C`
|
||||
- Pointer passing: `CALL 'PGM' USING ADDRESS OF WS-A`
|
||||
- Placeholder: `CALL 'PGM' USING OMITTED WS-B`
|
||||
- Filter keywords: add `ADDRESS`, `OMITTED`, `LENGTH` to USING_KEYWORDS (already has BY/VALUE/REFERENCE/CONTENT)
|
||||
- **Impact tool enhancement:** CALL-USING edges enable BFS traversal through parameter data flow — single most impactful edge type for COBOL impact analysis
|
||||
|
||||
#### 1.3 STRING/UNSTRING data flow -> ACCESSES edges
|
||||
- **File:** `cobol-preprocessor.ts` (new section in extractProcedure)
|
||||
- **What:** Accumulate multi-line STRING/UNSTRING until period or END-STRING/END-UNSTRING. Extract sources and INTO targets.
|
||||
- **Interface:** Add `strings: Array<{ sources: string[]; target: string; type: 'string' | 'unstring'; line: number; caller: string | null }>` to CobolRegexResults
|
||||
- **Graph:** read-ACCESSES on sources, write-ACCESSES on INTO target with reason `cobol-string-read` / `cobol-string-write`
|
||||
- **Tests:** 2 unit tests + integration test assertions
|
||||
|
||||
**Research insights (STRING/UNSTRING):**
|
||||
- **Needs statement accumulator** — STRING/UNSTRING always span multiple lines in production
|
||||
- Terminate accumulation at: period, END-STRING/END-UNSTRING, or start of next COBOL verb
|
||||
- STRING sources: identifiers before each `DELIMITED BY`. Filter: STRING, DELIMITED, BY, SIZE, ALL, INTO, WITH, POINTER, ON, OVERFLOW, NOT, END-STRING
|
||||
- UNSTRING: source is first identifier after UNSTRING; INTO targets are identifiers after INTO. Filter: DELIMITER, IN, COUNT, TALLYING, OR
|
||||
- WITH POINTER field is both read AND written (starting position updated)
|
||||
- TALLYING IN / COUNT IN fields are write targets
|
||||
- Literal sources (`'text'`) must be filtered — quote-aware tokenization needed
|
||||
- **Edge case**: STRING terminated by next verb, not period — existing fixture has `STRING ... DISPLAY` without period between them
|
||||
|
||||
#### 1.4 OCCURS DEPENDING ON -> ACCESSES edge
|
||||
- **File:** `cobol-preprocessor.ts` (parseDataItemClauses)
|
||||
- **What:** Extend OCCURS regex to capture DEPENDING ON field, KEY fields, and INDEXED BY names
|
||||
- **Interface:** Add `dependingOn?: string`, `occursMax?: number`, `occursKeys?: Array<{direction: string; fields: string[]}>`, `indexedBy?: string[]` to data items
|
||||
- **Graph:** ACCESSES edge from table item to controlling field with reason `cobol-depends-on`
|
||||
- **Tests:** `05 WS-TABLE OCCURS 100 DEPENDING ON WS-COUNT` -> edge
|
||||
|
||||
**Research insights (OCCURS):**
|
||||
- IBM allows `OCCURS 0 TO n DEPENDING ON` (zero minimum) and `OCCURS UNBOUNDED DEPENDING ON` (V6.4)
|
||||
- Subscripted controlling fields: `DEPENDING ON WS-COUNT(WS-IDX)` — strip subscripts before storing
|
||||
- **Pre-existing gap**: Multi-line data item clauses without continuation indicator are NOT captured. `05 WS-TABLE\n OCCURS 100\n DEPENDING ON WS-COUNT.` — the current RE_DATA_ITEM only gets the first line, `rest` is empty. Fixing properly requires a data item accumulator (like SELECT). **Defer full fix to Phase 3; implement same-line capture now.**
|
||||
- KEY IS fields: `ASCENDING KEY IS WS-KEY-1 WS-KEY-2` — capture for SEARCH ALL resolution
|
||||
- INDEXED BY: `INDEXED BY IDX-1 IDX-2` — capture for SET/SEARCH context
|
||||
|
||||
#### 1.5 VALUE clause for standard data items
|
||||
- **File:** `cobol-preprocessor.ts` (parseDataItemClauses)
|
||||
- **What:** Extract VALUE using a pragmatic function that handles quoted strings, numerics, figurative constants, hex/national literals
|
||||
- **Interface:** Already exists as `values?: string[]` on data items (currently only populated for 88-level)
|
||||
- **Graph:** Stored in Property node description (no new edges)
|
||||
- **Tests:** `01 WS-STATUS PIC X VALUE 'A'` -> values: ['A']
|
||||
|
||||
**Research insights (VALUE forms):**
|
||||
- Hex literals: `VALUE X'F1F2F3F4'`, National: `VALUE N'text'`, DBCS: `VALUE G'text'`
|
||||
- Figurative constants: SPACES, ZEROS, ZEROES, LOW-VALUES, HIGH-VALUES, QUOTES, NULL, NULLS
|
||||
- ALL literal: `VALUE ALL '*'`
|
||||
- Numeric with sign/decimal: `VALUE -123.45`, `VALUE +1`
|
||||
- `VALUE IS` optional — both `VALUE 'A'` and `VALUE IS 'A'` valid
|
||||
- **Decimal vs period ambiguity**: `VALUE 100.` — is `.` decimal or terminator? `parseDataItemClauses` already strips trailing period, so this is handled
|
||||
- IBM V6.4: floating-point `VALUE 1.0E5` — extend numeric regex if needed
|
||||
- Implementation: use a pragmatic `extractValue(rest)` function, not a single complex regex
|
||||
|
||||
### Phase 2: EXEC DLI + DECLARATIVES (~90 LOC, ~4 new edge types)
|
||||
|
||||
IMS/DB support and error handling flows.
|
||||
|
||||
#### 2.1 EXEC DLI (IMS/DB) -> ACCESSES edges (Graph value: 8.2/10)
|
||||
- **File:** `cobol-preprocessor.ts` (processLogicalLine — add RE_EXEC_DLI_START check alongside SQL/CICS)
|
||||
- **What:** Accumulate EXEC DLI blocks like EXEC SQL. Parse DLI verbs (GU, GN, GNP, GHU, GHN, GHNP, ISRT, DLET, REPL, CHKP, SCHD, TERM). Extract segment name, PCB number, INTO/FROM areas, WHERE fields, PSB name.
|
||||
- **Interface:** Add `execDliBlocks: Array<{ line: number; verb: string; pcbNumber?: number; segmentName?: string; intoField?: string; fromField?: string; whereField?: string; psbName?: string }>` to CobolRegexResults
|
||||
- **Graph:** CodeElement node + ACCESSES edge to `<ims>:<segmentName>` Record node with reason `dli-{verb}`; ACCESSES edges to INTO/FROM data areas; PSB ACCESSES for SCHD
|
||||
- **Tests:** `EXEC DLI GU USING PCB(1) SEGMENT(CUSTOMER) INTO(WS-CUST) END-EXEC`
|
||||
|
||||
**Research insights (dual IMS interface):**
|
||||
- **EXEC DLI**: Embedded command interface for CICS-DL/I programs only
|
||||
- **CBLTDLI CALL**: Batch interface via `CALL 'CBLTDLI' USING function-code PCB io-area SSA1..SSA15`
|
||||
- CBLTDLI is already captured as a CALL to 'CBLTDLI' — enrich with USING parameter semantics later
|
||||
- Multiple SEGMENT levels in hierarchical retrieval — use `matchAll` on segment regex
|
||||
- DLI verbs: GU (most common), GN, GNP, GHU, GHN, GHNP, ISRT, REPL, DLET, CHKP, SCHD, TERM, ROLL, ROLB
|
||||
- **Edge case**: DLET/REPL have no SEGMENT clause (operate on current position)
|
||||
- **Recommended order**: Implement AFTER DECLARATIVES and SET (lower risk, higher frequency)
|
||||
|
||||
#### 2.2 DECLARATIVES / USE AFTER STANDARD EXCEPTION (Graph value: 9.0/10)
|
||||
- **File:** `cobol-preprocessor.ts` (processLogicalLine — detect DECLARATIVES keyword, track USE AFTER blocks)
|
||||
- **What:** When `DECLARATIVES.` is encountered, switch to declaratives mode. Extract USE statements binding sections to files/modes.
|
||||
- **Interface:** Add `declaratives: Array<{ sectionName: string; useType: 'error' | 'debug' | 'label' | 'reporting'; target: string; line: number }>` to CobolRegexResults
|
||||
- **Graph:** ACCESSES edge from declarative Namespace to file Record with reason `cobol-declarative-error-handler`
|
||||
- **Tests:** Unit test with DECLARATIVES section, integration test for error flow
|
||||
|
||||
**Research insights (DECLARATIVES syntax):**
|
||||
- `USE AFTER STANDARD {EXCEPTION|ERROR} ON {file-name|INPUT|OUTPUT|I-O|EXTEND}`
|
||||
- EXCEPTION and ERROR are synonymous; STANDARD is optional in IBM dialects
|
||||
- Multiple USE sections allowed (one per file + catch-all for I/O modes)
|
||||
- `END DECLARATIVES.` must NOT reset PROCEDURE DIVISION state
|
||||
- `DECLARATIVES` is already in EXCLUDED_PARA_NAMES — no false paragraph risk
|
||||
- Existing section/paragraph detection already captures structural elements — just need USE binding
|
||||
- **Lowest risk Phase 2 item** — implement first
|
||||
|
||||
#### 2.3 SET statement -> ACCESSES edges
|
||||
- **File:** `cobol-preprocessor.ts` (extractProcedure — new RE_SET regex)
|
||||
- **Interface:** Add `sets: Array<{ targets: string[]; form: 'to-true'|'to-value'|'up-by'|'down-by'|'address-of'|'to-null'|'to-entry'; value?: string; entryTarget?: string; entryIsLiteral?: boolean; line: number; caller: string | null }>` to CobolRegexResults
|
||||
- **Graph:** ACCESSES write edge with reason `cobol-set-condition` (TO TRUE), `cobol-set-index` (TO/UP/DOWN), `cobol-set-address` (ADDRESS OF). SET ENTRY with literal -> CALLS edge.
|
||||
- **Tests:** `SET WS-EOF TO TRUE`, `SET IDX-1 TO 5`, `SET IDX-1 UP BY 1`
|
||||
|
||||
**Research insights (SET forms by frequency):**
|
||||
- `SET condition TO TRUE` — 80-90% of all SET usage. Multiple targets: `SET COND-A COND-B TO TRUE`
|
||||
- `SET index TO/UP BY/DOWN BY` — ~8%. Multiple indices: `SET IDX-1 IDX-2 UP BY 1`
|
||||
- `SET pointer TO ADDRESS OF data-item` / `SET ADDRESS OF data-item TO pointer` — ~2%
|
||||
- `SET proc-ptr TO ENTRY "PROGNAME"` — rare but creates CALLS edge (like dynamic CALL)
|
||||
- Filter OF/IN qualifiers: `SET COND-A OF WS-RECORD TO TRUE` (strip OF WS-RECORD)
|
||||
- **Prioritize**: SET TO TRUE alone covers 80-90% — implement this form first
|
||||
|
||||
#### 2.4 INSPECT -> ACCESSES edges
|
||||
- **File:** `cobol-preprocessor.ts` (extractProcedure — new `inspectAccum` accumulator like SORT)
|
||||
- **What:** Accumulate multi-line INSPECT until period. Extract inspected field + tally counters.
|
||||
- **Interface:** Add `inspects: Array<{ inspectedField: string; counters: string[]; form: 'tallying'|'replacing'|'converting'|'tallying-replacing'; line: number; caller: string | null }>` to CobolRegexResults
|
||||
- **Graph:** ACCESSES read on inspected field always; write if REPLACING/CONVERTING. Write edges for tally counters. Reason: `cobol-inspect-read`/`cobol-inspect-write`/`cobol-inspect-tally`
|
||||
- **Tests:** `INSPECT WS-FIELD TALLYING WS-COUNT FOR ALL 'A'` -> read on WS-FIELD, write on WS-COUNT
|
||||
|
||||
**Research insights (INSPECT forms by frequency):**
|
||||
- REPLACING (~60%): `INSPECT WS-STR REPLACING ALL 'A' BY 'B'`
|
||||
- TALLYING (~25%): `INSPECT WS-STR TALLYING WS-CNT FOR ALL 'A'` — multiple counters possible
|
||||
- CONVERTING (~10%): `INSPECT WS-STR CONVERTING 'abc' TO 'ABC'`
|
||||
- Combined (~5%): TALLYING + REPLACING in single statement
|
||||
- **Needs multi-line accumulator** — INSPECT frequently spans 3-5 lines in production
|
||||
- Extract tally counters with `([A-Z][A-Z0-9-]+)\s+FOR\b` matchAll pattern
|
||||
- Filter figurative constants (SPACES, ZEROS) using existing MOVE_SKIP set
|
||||
|
||||
### Phase 3: Completeness Fixes (~60 LOC)
|
||||
|
||||
Fix the 10 partial features and small gaps.
|
||||
|
||||
#### 3.1 CALL ... RETURNING extraction
|
||||
- Extend RE_CALL processing to capture RETURNING target after the USING clause
|
||||
- Store as `calls[].returning?: string`
|
||||
- Graph: ACCESSES write edge with reason `cobol-call-returning`
|
||||
|
||||
#### 3.2 SELECT OPTIONAL flag preservation
|
||||
- Store `isOptional: boolean` in FileDeclaration interface
|
||||
- Include in Record node description
|
||||
|
||||
#### 3.3 ALTERNATE RECORD KEY extraction
|
||||
- Add regex in parseSelectStatement: `/\bALTERNATE\s+RECORD\s+KEY\s+(?:IS\s+)?([A-Z][A-Z0-9-]+)/i`
|
||||
- Store as `alternateKeys?: string[]`
|
||||
|
||||
#### 3.4 COMMON attribute on nested programs
|
||||
- Extend RE_PROGRAM_ID: `/\bPROGRAM-ID\.\s*([A-Z][A-Z0-9-]+)(?:\s+IS\s+COMMON)?/i`
|
||||
- Store `isCommon: boolean` on Module node
|
||||
- Affects cross-program CALL resolution scope
|
||||
|
||||
#### 3.5 IS EXTERNAL / IS GLOBAL as first-class properties
|
||||
- Change from usage string hack to proper boolean fields on data items
|
||||
- Add `isExternal?: boolean`, `isGlobal?: boolean` to data item interface
|
||||
|
||||
#### 3.6 AUTHOR / DATE-WRITTEN mapped to Module node
|
||||
- Already extracted as programMetadata — map to Module node properties
|
||||
- `graph.addNode({ ..., properties: { ..., author, dateWritten } })`
|
||||
|
||||
#### 3.7 REPLACE statement
|
||||
- Track REPLACE / REPLACE OFF state in preprocessor
|
||||
- Apply text substitutions during preprocessing (before regex extraction)
|
||||
- Complex: requires careful scoping rules
|
||||
|
||||
### Phase 4: Niche Features (~30 LOC)
|
||||
|
||||
Low-priority but nice for completeness.
|
||||
|
||||
#### 4.1 INITIALIZE statement -> write ACCESSES
|
||||
- `/\bINITIALIZE\s+([A-Z][A-Z0-9-]+)/i`
|
||||
- ACCESSES write edge with reason `cobol-initialize`
|
||||
|
||||
#### 4.2 Remaining IDENTIFICATION DIVISION paragraphs
|
||||
- DATE-COMPILED, INSTALLATION, SECURITY, REMARKS
|
||||
- Map to Module node description properties
|
||||
|
||||
#### 4.3 EXEC SQL INCLUDE -> IMPORTS edge (expansion)
|
||||
- For EXEC SQL INCLUDE inside EXEC blocks that reference copybooks containing SQL
|
||||
- Create IMPORTS edge similar to COPY
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
### Functional Requirements
|
||||
|
||||
- [ ] Phase 1: All 5 features implemented with unit + integration tests
|
||||
- [ ] Phase 2: All 4 features implemented with unit + integration tests
|
||||
- [ ] Phase 3: All 7 partial features fixed
|
||||
- [ ] Phase 4: At least 2 of 3 niche features implemented
|
||||
- [ ] All existing 145 tests continue to pass
|
||||
- [ ] TypeScript compiles cleanly
|
||||
|
||||
### Non-Functional Requirements
|
||||
|
||||
- [ ] No performance regression: CardDemo benchmark stays under 8s
|
||||
- [ ] No file exceeds 1500 LOC (preprocessor currently 1326)
|
||||
- [ ] ACAS benchmark shows increased node/edge counts (more data extracted)
|
||||
- [ ] CardDemo benchmark shows increased edge counts (CALL USING, STRING, etc.)
|
||||
|
||||
### Quality Gates
|
||||
|
||||
- [ ] Each phase has its own commit
|
||||
- [ ] Integration test assertions updated with exact counts per phase
|
||||
- [ ] Benchmark run after each phase to track graph growth
|
||||
|
||||
## Dependencies & Risks
|
||||
|
||||
### Dependencies
|
||||
- None. All changes are additive to existing COBOL processor code.
|
||||
- No LanguageProvider changes needed.
|
||||
- No graph schema changes needed (all new constructs map to existing node labels + edge types).
|
||||
|
||||
### Risks
|
||||
- **preprocessor.ts size**: Currently 1326 LOC. Phase 1+2 adds ~200 LOC -> 1526 LOC. May need to extract helpers into a separate `cobol-data-flow.ts` module if it exceeds 1500.
|
||||
- **REPLACE statement** (Phase 3.7) is the most complex feature — requires tracking text substitution state across logical lines. Consider deferring to a separate PR if it takes >100 LOC.
|
||||
- **EXEC DLI** (Phase 2.1) is only testable against IMS codebases. Need fixture data or synthetic test cases.
|
||||
|
||||
## Graph Value Ranking by MCP Tool Impact
|
||||
|
||||
Research agent analyzed all 5 MCP tools (query, context, impact, detect_changes, rename) against planned edge types:
|
||||
|
||||
| Edge Type | QUERY | CONTEXT | IMPACT | DETECT | RENAME | **Overall** |
|
||||
|-----------|-------|---------|--------|--------|--------|-------------|
|
||||
| `cobol-call-using` | 4/5 | 5/5 | 5/5 | 4/5 | 4/5 | **9.2/10** |
|
||||
| `cobol-error-handler` | 5/5 | 4/5 | 5/5 | 5/5 | 2/5 | **9.0/10** |
|
||||
| `dli-*` (IMS verbs) | 4/5 | 4/5 | 5/5 | 4/5 | 2/5 | **8.2/10** |
|
||||
| `cobol-string-*` | 4/5 | 3/5 | 3/5 | 3/5 | 2/5 | **6.2/10** |
|
||||
|
||||
**Key finding**: `cobol-call-using` alone would fix ~40% of missing caller references in COBOL graphs.
|
||||
|
||||
## Future Considerations
|
||||
|
||||
This plan provides the graph data foundation for a future `modernize` MCP command (out of scope) that would:
|
||||
- Use CALL USING edges to map data contracts between programs
|
||||
- Use STRING/UNSTRING edges to identify data transformation logic
|
||||
- Use EXEC SQL/DLI edges to map database access patterns
|
||||
- Use DECLARATIVES to understand error handling architecture
|
||||
- Use the complete knowledge graph to generate migration plans
|
||||
|
||||
**MCP tool enhancements needed** (after this plan ships):
|
||||
- Add `cobol-call-using`, `cobol-error-handler`, `dli-*` to IMPACT tool's default `relationTypes` for COBOL repos
|
||||
- Add confidence floors for new edge types in `IMPACT_RELATION_CONFIDENCE`
|
||||
- Register new edge types in `VALID_RELATION_TYPES` set (`local-backend.ts:52`)
|
||||
|
||||
## Sources & References
|
||||
|
||||
### Internal References
|
||||
- Feature audit: session 8642401e (COBOL expert agent, 123 features audited)
|
||||
- Prior plans: `docs/plans/2026-03-25-feat-cobol-100-percent-feature-coverage-plan.md`
|
||||
- Architecture: `docs/code-indexing/cobol/` (7 documentation files)
|
||||
|
||||
### External References
|
||||
- COBOL features reference: mainframestechhelp.com/tutorials/cobol/features.htm
|
||||
- COBOL-85 standard: ISO/IEC 1989:1985
|
||||
- IBM Enterprise COBOL reference
|
||||
@@ -1,725 +0,0 @@
|
||||
# PR #626 HIGH-Priority Fixes Implementation Plan
|
||||
|
||||
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||
|
||||
**Goal:** Fix 4 HIGH-priority issues from PR #626 code review before merge.
|
||||
|
||||
**Architecture:** Minimal targeted fixes — each task is independent. TDD: tests first, then implementation. No refactoring beyond what's needed.
|
||||
|
||||
**Tech Stack:** TypeScript, Vitest, Node.js fs/path APIs
|
||||
|
||||
**Spec:** `docs/superpowers/specs/2026-04-02-pr626-high-fixes-design.md`
|
||||
|
||||
**Paths:** All file paths are relative to the monorepo root (`GitNexus/`). Git commands run from the root. The `gitnexus/` prefix is a package subdirectory, not a separate repo.
|
||||
|
||||
---
|
||||
|
||||
### Task 1: Path Traversal — Validate Group Name
|
||||
|
||||
**Files:**
|
||||
- Modify: `gitnexus/src/core/group/storage.ts:17-19` (getGroupDir) and `:63-68` (createGroupDir)
|
||||
- Test: `gitnexus/test/unit/group/storage.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for validateGroupName**
|
||||
|
||||
In `gitnexus/test/unit/group/storage.test.ts`, add `createGroupDir` and `validateGroupName` to the existing import from `'../../../src/core/group/storage.js'` (line 6-11). Then add these describe blocks at the end of the outer `describe('Group storage', ...)`:
|
||||
|
||||
```typescript
|
||||
describe('validateGroupName', () => {
|
||||
it('test_validateGroupName_traversal_path_throws', () => {
|
||||
expect(() => validateGroupName('../../evil')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_validateGroupName_slash_in_name_throws', () => {
|
||||
expect(() => validateGroupName('foo/bar')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_validateGroupName_empty_string_throws', () => {
|
||||
expect(() => validateGroupName('')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_validateGroupName_starts_with_dash_throws', () => {
|
||||
expect(() => validateGroupName('-leading-dash')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_validateGroupName_starts_with_underscore_throws', () => {
|
||||
expect(() => validateGroupName('_leading')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_validateGroupName_dots_throws', () => {
|
||||
expect(() => validateGroupName('com.example')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_validateGroupName_valid_alphanumeric_passes', () => {
|
||||
expect(() => validateGroupName('my-group_01')).not.toThrow();
|
||||
});
|
||||
|
||||
it('test_validateGroupName_single_char_passes', () => {
|
||||
expect(() => validateGroupName('A')).not.toThrow();
|
||||
});
|
||||
|
||||
it('test_validateGroupName_all_digits_passes', () => {
|
||||
expect(() => validateGroupName('123')).not.toThrow();
|
||||
});
|
||||
});
|
||||
|
||||
describe('getGroupDir rejects invalid names', () => {
|
||||
it('test_getGroupDir_traversal_throws', () => {
|
||||
expect(() => getGroupDir(tmpDir, '../../etc')).toThrow(/Invalid group name/);
|
||||
});
|
||||
|
||||
it('test_getGroupDir_valid_name_returns_path', () => {
|
||||
const dir = getGroupDir(tmpDir, 'company');
|
||||
expect(dir).toBe(path.join(tmpDir, 'groups', 'company'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('createGroupDir rejects invalid names', () => {
|
||||
it('test_createGroupDir_traversal_throws', async () => {
|
||||
await expect(createGroupDir(tmpDir, '../evil')).rejects.toThrow(/Invalid group name/);
|
||||
});
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify they fail**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/storage.test.ts`
|
||||
Expected: FAIL — `validateGroupName` is not exported, `getGroupDir` does not throw.
|
||||
|
||||
- [ ] **Step 3: Implement validateGroupName and wire into getGroupDir and createGroupDir**
|
||||
|
||||
In `gitnexus/src/core/group/storage.ts`, add the validation function before `getGroupDir` and call it:
|
||||
|
||||
```typescript
|
||||
const GROUP_NAME_RE = /^[a-zA-Z0-9][a-zA-Z0-9_-]*$/;
|
||||
|
||||
export function validateGroupName(name: string): void {
|
||||
if (!GROUP_NAME_RE.test(name)) {
|
||||
throw new Error(
|
||||
`Invalid group name "${name}". Names must start with a letter or digit and contain only [a-zA-Z0-9_-].`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
export function getGroupDir(gitnexusDir: string, groupName: string): string {
|
||||
validateGroupName(groupName);
|
||||
return path.join(gitnexusDir, 'groups', groupName);
|
||||
}
|
||||
```
|
||||
|
||||
`createGroupDir` already calls `getGroupDir` at line 68, so it inherits validation automatically. No change needed in `createGroupDir`.
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/storage.test.ts`
|
||||
Expected: ALL PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd gitnexus && git add src/core/group/storage.ts test/unit/group/storage.test.ts
|
||||
git commit -m "fix(group): validate group name to prevent path traversal
|
||||
|
||||
Add validateGroupName() with regex [a-zA-Z0-9][a-zA-Z0-9_-]*.
|
||||
Called in getGroupDir (defense in depth) which covers all CLI entry
|
||||
points: create, add, remove, status, sync.
|
||||
|
||||
Addresses PR #626 review item 1 (HIGH).
|
||||
|
||||
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 2: Directory Exclusions in Service Boundary Detector
|
||||
|
||||
**Files:**
|
||||
- Modify: `gitnexus/src/core/group/service-boundary-detector.ts:24-51` (add constant), `:78` (walkForBoundaries), `:130` (hasSourceFilesInSubdirs)
|
||||
- Test: `gitnexus/test/unit/group/service-boundary-detector.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for excluded directories**
|
||||
|
||||
Add this describe block inside the existing `detectServiceBoundaries` describe in `gitnexus/test/unit/group/service-boundary-detector.test.ts`:
|
||||
|
||||
```typescript
|
||||
it('test_detect_skips_vendor_directory', async () => {
|
||||
writeFile('services/auth/package.json', '{}');
|
||||
writeFile('services/auth/src/index.ts', '');
|
||||
// vendor should be skipped — its contents should not create a boundary
|
||||
writeFile('vendor/some-dep/package.json', '{}');
|
||||
writeFile('vendor/some-dep/src/lib.go', '');
|
||||
|
||||
const boundaries = await detectServiceBoundaries(tmpDir);
|
||||
|
||||
const paths = boundaries.map((b) => b.servicePath);
|
||||
expect(paths).toContain('services/auth');
|
||||
expect(paths).not.toContain('vendor/some-dep');
|
||||
});
|
||||
|
||||
it('test_detect_skips_target_directory', async () => {
|
||||
writeFile('services/api/go.mod', 'module api');
|
||||
writeFile('services/api/main.go', '');
|
||||
writeFile('target/classes/Main.java', '');
|
||||
writeFile('target/pom.xml', '<project/>');
|
||||
|
||||
const boundaries = await detectServiceBoundaries(tmpDir);
|
||||
|
||||
const paths = boundaries.map((b) => b.servicePath);
|
||||
expect(paths).toContain('services/api');
|
||||
expect(paths).not.toContain('target');
|
||||
});
|
||||
|
||||
it('test_detect_skips_pycache_directory', async () => {
|
||||
writeFile('services/ml/pyproject.toml', '[project]');
|
||||
writeFile('services/ml/model.py', '');
|
||||
// __pycache__ with a marker + source files — would be detected as
|
||||
// a boundary if not excluded, since it has package.json + .py file
|
||||
writeFile('__pycache__/package.json', '{}');
|
||||
writeFile('__pycache__/cached.py', '');
|
||||
|
||||
const boundaries = await detectServiceBoundaries(tmpDir);
|
||||
|
||||
const paths = boundaries.map((b) => b.servicePath);
|
||||
expect(paths).toContain('services/ml');
|
||||
expect(paths.every((p) => !p.includes('__pycache__'))).toBe(true);
|
||||
});
|
||||
|
||||
it('test_detect_skips_dotfile_directories_regression', async () => {
|
||||
writeFile('services/api/package.json', '{}');
|
||||
writeFile('services/api/src/index.ts', '');
|
||||
writeFile('.hidden/package.json', '{}');
|
||||
writeFile('.hidden/src/index.ts', '');
|
||||
|
||||
const boundaries = await detectServiceBoundaries(tmpDir);
|
||||
|
||||
const paths = boundaries.map((b) => b.servicePath);
|
||||
expect(paths).toContain('services/api');
|
||||
expect(paths).not.toContain('.hidden');
|
||||
});
|
||||
|
||||
it('test_detect_does_not_skip_regular_source_directories', async () => {
|
||||
writeFile('services/api/package.json', '{}');
|
||||
writeFile('services/api/src/index.ts', '');
|
||||
|
||||
const boundaries = await detectServiceBoundaries(tmpDir);
|
||||
|
||||
expect(boundaries).toHaveLength(1);
|
||||
expect(boundaries[0].serviceName).toBe('api');
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify `vendor` and `target` tests fail**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/service-boundary-detector.test.ts`
|
||||
Expected: `test_detect_skips_vendor_directory` and `test_detect_skips_target_directory` FAIL (vendor/target not excluded). Other new tests may pass since dotfile exclusion already exists.
|
||||
|
||||
- [ ] **Step 3: Add EXCLUDED_DIRS constant and update both walking functions**
|
||||
|
||||
In `gitnexus/src/core/group/service-boundary-detector.ts`:
|
||||
|
||||
After `SOURCE_EXTENSIONS` (after line 51), add:
|
||||
|
||||
```typescript
|
||||
const EXCLUDED_DIRS = new Set([
|
||||
'node_modules',
|
||||
'vendor',
|
||||
'target',
|
||||
'build',
|
||||
'dist',
|
||||
'__pycache__',
|
||||
'.venv',
|
||||
'venv',
|
||||
'.tox',
|
||||
'.mypy_cache',
|
||||
'.gradle',
|
||||
'.mvn',
|
||||
'out',
|
||||
'bin',
|
||||
]);
|
||||
```
|
||||
|
||||
In `walkForBoundaries`, replace line 78:
|
||||
```typescript
|
||||
if (entry.name.startsWith('.') || entry.name === 'node_modules') continue;
|
||||
```
|
||||
with:
|
||||
```typescript
|
||||
if (entry.name.startsWith('.') || EXCLUDED_DIRS.has(entry.name)) continue;
|
||||
```
|
||||
|
||||
In `hasSourceFilesInSubdirs`, replace line 130:
|
||||
```typescript
|
||||
if (entry.isDirectory() && !entry.name.startsWith('.') && entry.name !== 'node_modules') {
|
||||
```
|
||||
with:
|
||||
```typescript
|
||||
if (entry.isDirectory() && !entry.name.startsWith('.') && !EXCLUDED_DIRS.has(entry.name)) {
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/service-boundary-detector.test.ts`
|
||||
Expected: ALL PASS
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd gitnexus && git add src/core/group/service-boundary-detector.ts test/unit/group/service-boundary-detector.test.ts
|
||||
git commit -m "fix(group): add directory exclusions to service boundary detector
|
||||
|
||||
Add EXCLUDED_DIRS set: vendor, target, build, dist, __pycache__,
|
||||
.venv, venv, .tox, .mypy_cache, .gradle, .mvn, out, bin.
|
||||
Applied in walkForBoundaries and hasSourceFilesInSubdirs.
|
||||
Replaces inline node_modules check.
|
||||
|
||||
Addresses PR #626 review item 3 (HIGH).
|
||||
|
||||
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 3: Remove Double-Close of LadybugDB Pools
|
||||
|
||||
**Files:**
|
||||
- Modify: `gitnexus/src/cli/group.ts:160` (remove import), `:187-189` (remove finally block body)
|
||||
- Test: `gitnexus/test/unit/group/sync.test.ts` (add pool cleanup test)
|
||||
- Test: `gitnexus/test/integration/group/group-cli.test.ts` (verify no blanket close in source)
|
||||
|
||||
- [ ] **Step 1: Write unit tests for per-id pool cleanup in sync.ts**
|
||||
|
||||
Add to `gitnexus/test/unit/group/sync.test.ts`, inside the existing `describe('syncGroup', ...)`:
|
||||
|
||||
```typescript
|
||||
it('test_syncGroup_closes_only_opened_pools', async () => {
|
||||
const config = makeConfig({
|
||||
'app/backend': 'backend-repo',
|
||||
'app/frontend': 'frontend-repo',
|
||||
});
|
||||
|
||||
const closedIds: string[] = [];
|
||||
|
||||
// Mock initLbug/closeLbug via per-repo override that tracks pool lifecycle
|
||||
const { vi } = await import('vitest');
|
||||
const poolAdapter = await import('../../../src/core/lbug/pool-adapter.js');
|
||||
const initSpy = vi.spyOn(poolAdapter, 'initLbug').mockResolvedValue(undefined);
|
||||
const closeSpy = vi.spyOn(poolAdapter, 'closeLbug').mockImplementation(async (id?: string) => {
|
||||
if (id) closedIds.push(id);
|
||||
});
|
||||
|
||||
try {
|
||||
await syncGroup(config, {
|
||||
resolveRepoHandle: async (_name, groupPath) => ({
|
||||
id: groupPath.replace(/\//g, '-'),
|
||||
path: groupPath,
|
||||
repoPath: '/tmp/' + groupPath,
|
||||
storagePath: '/tmp/' + groupPath + '/.gitnexus',
|
||||
}),
|
||||
skipWrite: true,
|
||||
}).catch(() => {});
|
||||
// Regardless of extraction errors, closeLbug should be called per id
|
||||
// closeLbug should only receive specific pool ids, never undefined/empty
|
||||
for (const id of closedIds) {
|
||||
expect(id).toBeTruthy();
|
||||
expect(typeof id).toBe('string');
|
||||
}
|
||||
// No blanket close (no-arg call)
|
||||
const blanketCalls = closeSpy.mock.calls.filter((args) => args.length === 0 || !args[0]);
|
||||
expect(blanketCalls).toHaveLength(0);
|
||||
} finally {
|
||||
initSpy.mockRestore();
|
||||
closeSpy.mockRestore();
|
||||
}
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run sync unit test to verify it passes (sync.ts already does per-id cleanup)**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/sync.test.ts`
|
||||
Expected: PASS — sync.ts already cleans up correctly. This test locks the behavior.
|
||||
|
||||
- [ ] **Step 3: Write test verifying CLI source has no blanket closeLbug()**
|
||||
|
||||
Add to `gitnexus/test/integration/group/group-cli.test.ts`:
|
||||
|
||||
```typescript
|
||||
it('test_sync_command_source_does_not_call_blanket_closeLbug', () => {
|
||||
const cliGroupPath = path.join(repoRoot, 'src', 'cli', 'group.ts');
|
||||
const source = fs.readFileSync(cliGroupPath, 'utf-8');
|
||||
|
||||
// closeLbug() without arguments (blanket close) must not appear.
|
||||
// closeLbug(id) with argument is fine (that's in sync.ts, not here).
|
||||
// Match closeLbug() but not closeLbug(someArg)
|
||||
const blanketClosePattern = /closeLbug\s*\(\s*\)/;
|
||||
expect(source).not.toMatch(blanketClosePattern);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run test to verify it fails**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/integration/group/group-cli.test.ts`
|
||||
Expected: FAIL — `closeLbug()` (no args) exists at line 188.
|
||||
|
||||
- [ ] **Step 5: Remove blanket closeLbug() from cli/group.ts**
|
||||
|
||||
In `gitnexus/src/cli/group.ts`:
|
||||
|
||||
Remove the `closeLbug` import at line 160:
|
||||
```typescript
|
||||
const { closeLbug } = await import('../core/lbug/pool-adapter.js');
|
||||
```
|
||||
|
||||
Replace the try/finally wrapper (lines 162-189):
|
||||
```typescript
|
||||
try {
|
||||
const groupDir = getGroupDir(getDefaultGitnexusDir(), name);
|
||||
const config = await loadGroupConfig(groupDir);
|
||||
|
||||
console.log(`Syncing group "${name}" (${Object.keys(config.repos).length} repos)...\n`);
|
||||
|
||||
const result = await syncGroup(config, {
|
||||
groupDir,
|
||||
allowStale: Boolean(opts.allowStale),
|
||||
verbose: Boolean(opts.verbose),
|
||||
skipEmbeddings: Boolean(opts.skipEmbeddings),
|
||||
exactOnly: Boolean(opts.exactOnly),
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`\nMatching cascade:`);
|
||||
const exactLinks = result.crossLinks.filter((l) => l.matchType === 'exact');
|
||||
console.log(` exact: ${exactLinks.length} cross-links (confidence 1.0)`);
|
||||
console.log(` unmatched: ${result.unmatched.length} contracts`);
|
||||
console.log(
|
||||
`\nWrote contracts.json (${result.contracts.length} contracts, ${result.crossLinks.length} cross-links)`,
|
||||
);
|
||||
}
|
||||
} finally {
|
||||
await closeLbug().catch(() => {});
|
||||
}
|
||||
```
|
||||
|
||||
Becomes (remove try/finally entirely, since sync.ts handles its own cleanup):
|
||||
```typescript
|
||||
const groupDir = getGroupDir(getDefaultGitnexusDir(), name);
|
||||
const config = await loadGroupConfig(groupDir);
|
||||
|
||||
console.log(`Syncing group "${name}" (${Object.keys(config.repos).length} repos)...\n`);
|
||||
|
||||
const result = await syncGroup(config, {
|
||||
groupDir,
|
||||
allowStale: Boolean(opts.allowStale),
|
||||
verbose: Boolean(opts.verbose),
|
||||
skipEmbeddings: Boolean(opts.skipEmbeddings),
|
||||
exactOnly: Boolean(opts.exactOnly),
|
||||
});
|
||||
|
||||
if (opts.json) {
|
||||
console.log(JSON.stringify(result, null, 2));
|
||||
} else {
|
||||
console.log(`\nMatching cascade:`);
|
||||
const exactLinks = result.crossLinks.filter((l) => l.matchType === 'exact');
|
||||
console.log(` exact: ${exactLinks.length} cross-links (confidence 1.0)`);
|
||||
console.log(` unmatched: ${result.unmatched.length} contracts`);
|
||||
console.log(
|
||||
`\nWrote contracts.json (${result.contracts.length} contracts, ${result.crossLinks.length} cross-links)`,
|
||||
);
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 6: Run tests to verify they pass**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/integration/group/group-cli.test.ts test/unit/group/sync.test.ts`
|
||||
Expected: ALL PASS
|
||||
|
||||
- [ ] **Step 7: Commit**
|
||||
|
||||
```bash
|
||||
cd gitnexus && git add src/cli/group.ts test/integration/group/group-cli.test.ts test/unit/group/sync.test.ts
|
||||
git commit -m "fix(group): remove blanket closeLbug() from CLI sync command
|
||||
|
||||
sync.ts already closes pools per-id in its finally block.
|
||||
The blanket closeLbug() in cli/group.ts tears down ALL active pools
|
||||
including unrelated ones in MCP server context.
|
||||
|
||||
Addresses PR #626 review item 4 (HIGH).
|
||||
|
||||
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 4: gRPC Proto Regex — Brace-Depth Counter
|
||||
|
||||
**Files:**
|
||||
- Modify: `gitnexus/src/core/group/extractors/grpc-extractor.ts:101-130` (parseProtoFile)
|
||||
- Test: `gitnexus/test/unit/group/grpc-extractor.test.ts`
|
||||
|
||||
- [ ] **Step 1: Write failing tests for nested braces in proto services**
|
||||
|
||||
Add this describe block inside the existing `proto file parsing` describe in `gitnexus/test/unit/group/grpc-extractor.test.ts`:
|
||||
|
||||
```typescript
|
||||
it('test_extract_proto_with_google_api_http_nested_braces', async () => {
|
||||
writeFile(
|
||||
'api/gateway.proto',
|
||||
`syntax = "proto3";
|
||||
package gateway.v1;
|
||||
|
||||
import "google/api/annotations.proto";
|
||||
|
||||
service GatewayService {
|
||||
rpc GetUser (GetUserRequest) returns (UserResponse) {
|
||||
option (google.api.http) = {
|
||||
get: "/v1/users/{user_id}"
|
||||
};
|
||||
}
|
||||
rpc CreateUser (CreateUserRequest) returns (UserResponse) {
|
||||
option (google.api.http) = {
|
||||
post: "/v1/users"
|
||||
body: "*"
|
||||
};
|
||||
}
|
||||
}`,
|
||||
);
|
||||
|
||||
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
|
||||
const providers = contracts.filter(
|
||||
(c) => c.role === 'provider' && c.symbolRef.filePath === 'api/gateway.proto',
|
||||
);
|
||||
|
||||
expect(providers).toHaveLength(2);
|
||||
const ids = providers.map((c) => c.contractId).sort();
|
||||
expect(ids).toEqual([
|
||||
'grpc::gateway.v1.GatewayService/CreateUser',
|
||||
'grpc::gateway.v1.GatewayService/GetUser',
|
||||
]);
|
||||
});
|
||||
|
||||
it('test_extract_proto_with_multiple_services', async () => {
|
||||
writeFile(
|
||||
'api/multi.proto',
|
||||
`syntax = "proto3";
|
||||
package multi;
|
||||
|
||||
service ServiceA {
|
||||
rpc MethodA (Req) returns (Res);
|
||||
}
|
||||
|
||||
service ServiceB {
|
||||
rpc MethodB1 (Req) returns (Res);
|
||||
rpc MethodB2 (Req) returns (Res);
|
||||
}`,
|
||||
);
|
||||
|
||||
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
|
||||
const providers = contracts.filter(
|
||||
(c) => c.role === 'provider' && c.symbolRef.filePath === 'api/multi.proto',
|
||||
);
|
||||
|
||||
expect(providers).toHaveLength(3);
|
||||
const ids = providers.map((c) => c.contractId).sort();
|
||||
expect(ids).toEqual([
|
||||
'grpc::multi.ServiceA/MethodA',
|
||||
'grpc::multi.ServiceB/MethodB1',
|
||||
'grpc::multi.ServiceB/MethodB2',
|
||||
]);
|
||||
});
|
||||
|
||||
it('test_extract_proto_with_nested_option_blocks_in_rpc', async () => {
|
||||
writeFile(
|
||||
'api/nested.proto',
|
||||
`syntax = "proto3";
|
||||
package nested;
|
||||
|
||||
service DeepService {
|
||||
rpc DeepMethod (Req) returns (Res) {
|
||||
option (google.api.http) = {
|
||||
post: "/v1/deep"
|
||||
body: "*"
|
||||
additional_bindings {
|
||||
get: "/v1/deep/{id}"
|
||||
}
|
||||
};
|
||||
}
|
||||
}`,
|
||||
);
|
||||
|
||||
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
|
||||
const providers = contracts.filter(
|
||||
(c) => c.role === 'provider' && c.symbolRef.filePath === 'api/nested.proto',
|
||||
);
|
||||
|
||||
expect(providers).toHaveLength(1);
|
||||
expect(providers[0].contractId).toBe('grpc::nested.DeepService/DeepMethod');
|
||||
});
|
||||
|
||||
it('test_extract_proto_malformed_unclosed_brace_skips_service', async () => {
|
||||
writeFile(
|
||||
'api/broken.proto',
|
||||
`syntax = "proto3";
|
||||
package broken;
|
||||
|
||||
service IncompleteService {
|
||||
rpc SomeMethod (Req) returns (Res);
|
||||
// Missing closing brace — EOF before depth returns to 0
|
||||
`,
|
||||
);
|
||||
|
||||
// Should not throw; incomplete service is silently skipped
|
||||
const contracts = await extractor.extract(null, tmpDir, makeRepo(tmpDir));
|
||||
const providers = contracts.filter(
|
||||
(c) => c.role === 'provider' && c.symbolRef.filePath === 'api/broken.proto',
|
||||
);
|
||||
|
||||
// The old regex would find partial match; the new parser should skip it
|
||||
expect(providers).toHaveLength(0);
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run tests to verify the nested brace test fails**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/grpc-extractor.test.ts`
|
||||
Expected: `test_extract_proto_with_google_api_http_nested_braces` FAIL — regex stops at first `}` inside the `option` block.
|
||||
|
||||
- [ ] **Step 3: Replace serviceRe regex with extractServiceBlocks function**
|
||||
|
||||
In `gitnexus/src/core/group/extractors/grpc-extractor.ts`, replace the `parseProtoFile` method (lines 101-130):
|
||||
|
||||
```typescript
|
||||
private parseProtoFile(content: string, filePath: string): ExtractedContract[] {
|
||||
const out: ExtractedContract[] = [];
|
||||
|
||||
const pkgMatch = content.match(/^package\s+([\w.]+)\s*;/m);
|
||||
const pkg = pkgMatch ? pkgMatch[1] : '';
|
||||
|
||||
for (const { name: serviceName, body } of extractServiceBlocks(content)) {
|
||||
const rpcRe = /rpc\s+(\w+)\s*\(/g;
|
||||
let rpcMatch: RegExpExecArray | null;
|
||||
while ((rpcMatch = rpcRe.exec(body)) !== null) {
|
||||
const methodName = rpcMatch[1];
|
||||
const cid = contractId(pkg, serviceName, methodName);
|
||||
out.push(
|
||||
makeContract(cid, 'provider', filePath, `${serviceName}.${methodName}`, 0.85, {
|
||||
package: pkg,
|
||||
service: serviceName,
|
||||
method: methodName,
|
||||
source: 'proto',
|
||||
}),
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
```
|
||||
|
||||
Add this function before the class (e.g. after `serviceOnlyContractId`, around line 26):
|
||||
|
||||
```typescript
|
||||
function extractServiceBlocks(content: string): Array<{ name: string; body: string }> {
|
||||
const results: Array<{ name: string; body: string }> = [];
|
||||
const headerRe = /service\s+(\w+)\s*\{/g;
|
||||
let headerMatch: RegExpExecArray | null;
|
||||
|
||||
while ((headerMatch = headerRe.exec(content)) !== null) {
|
||||
const serviceName = headerMatch[1];
|
||||
const bodyStart = headerMatch.index + headerMatch[0].length;
|
||||
let depth = 1;
|
||||
let pos = bodyStart;
|
||||
|
||||
while (pos < content.length && depth > 0) {
|
||||
const ch = content[pos];
|
||||
if (ch === '{') depth++;
|
||||
else if (ch === '}') depth--;
|
||||
pos++;
|
||||
}
|
||||
|
||||
// If EOF before depth returns to 0, skip incomplete service
|
||||
if (depth !== 0) continue;
|
||||
|
||||
// body is between opening { (consumed by regex) and closing } (pos is one past it)
|
||||
const body = content.slice(bodyStart, pos - 1);
|
||||
results.push({ name: serviceName, body });
|
||||
}
|
||||
|
||||
return results;
|
||||
}
|
||||
```
|
||||
|
||||
- [ ] **Step 4: Run tests to verify they pass**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/grpc-extractor.test.ts`
|
||||
Expected: ALL PASS (including existing regression tests)
|
||||
|
||||
- [ ] **Step 5: Commit**
|
||||
|
||||
```bash
|
||||
cd gitnexus && git add src/core/group/extractors/grpc-extractor.ts test/unit/group/grpc-extractor.test.ts
|
||||
git commit -m "fix(group): replace gRPC proto regex with brace-depth counter
|
||||
|
||||
The serviceRe regex used [^}]* which stopped at the first '}'.
|
||||
Proto services with google.api.http annotations contain nested {}
|
||||
blocks, causing methods to be missed.
|
||||
|
||||
New extractServiceBlocks() uses a brace-depth counter (init depth=1
|
||||
after opening {, scan char-by-char). Malformed protos with unclosed
|
||||
braces are silently skipped.
|
||||
|
||||
Addresses PR #626 review item 2 (HIGH).
|
||||
|
||||
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
### Task 5: Run Full Test Suite
|
||||
|
||||
- [ ] **Step 1: Run all group-related tests**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/unit/group/ test/integration/group/`
|
||||
Expected: ALL PASS
|
||||
|
||||
- [ ] **Step 2: Run full test suite to catch regressions**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run`
|
||||
Expected: ALL PASS, 0 failures
|
||||
|
||||
- [ ] **Step 3: Run typecheck**
|
||||
|
||||
Run: `cd gitnexus && npx tsc --noEmit`
|
||||
Expected: No errors
|
||||
|
||||
---
|
||||
|
||||
### Task 6: CLI Integration Smoke Test
|
||||
|
||||
- [ ] **Step 1: Add CLI smoke test for path traversal**
|
||||
|
||||
Add to `gitnexus/test/integration/group/group-cli.test.ts` inside the existing `group CLI` describe:
|
||||
|
||||
```typescript
|
||||
it('test_create_with_invalid_name_fails', () => {
|
||||
const result = runGroup(['create', '../../evil']);
|
||||
expect(result.status).not.toBe(0);
|
||||
expect(result.stderr).toContain('Invalid group name');
|
||||
});
|
||||
```
|
||||
|
||||
- [ ] **Step 2: Run test**
|
||||
|
||||
Run: `cd gitnexus && npx vitest run test/integration/group/group-cli.test.ts`
|
||||
Expected: ALL PASS
|
||||
|
||||
- [ ] **Step 3: Commit**
|
||||
|
||||
```bash
|
||||
cd gitnexus && git add test/integration/group/group-cli.test.ts
|
||||
git commit -m "test(group): add CLI smoke test for path traversal rejection
|
||||
|
||||
Verifies that 'group create ../../evil' fails with Invalid group name.
|
||||
|
||||
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>"
|
||||
```
|
||||
@@ -1,175 +0,0 @@
|
||||
# PR #626 HIGH-Priority Fixes Design
|
||||
|
||||
**Date:** 2026-04-02
|
||||
**PR:** abhigyanpatwari/GitNexus#626 — Intra-repo service communication tracking
|
||||
**Scope:** 4 HIGH-priority issues identified by abhigyanpatwari and xkonjin
|
||||
**Approach:** Minimal targeted fixes (option A) — no refactoring, no scope creep
|
||||
|
||||
---
|
||||
|
||||
## Fix 1: Path Traversal via Group Name
|
||||
|
||||
**File:** `gitnexus/src/core/group/storage.ts`
|
||||
**Risk:** A group name like `../../etc` creates directories outside the intended path.
|
||||
|
||||
### Solution
|
||||
|
||||
Add `validateGroupName(name: string): void` that enforces `/^[a-zA-Z0-9][a-zA-Z0-9_-]*$/`.
|
||||
|
||||
- Call in `createGroupDir` (primary entry point)
|
||||
- Call in `getGroupDir` (defense in depth)
|
||||
- Throw descriptive error on invalid names
|
||||
|
||||
**Legacy:** Groups already on disk with names outside this pattern are not auto-renamed; only new `create` / resolved paths are validated.
|
||||
|
||||
### Why regex over path.resolve + startsWith
|
||||
|
||||
- abhigyanpatwari explicitly requested `[a-zA-Z0-9_-]`
|
||||
- Stricter: disallows spaces, dots, Unicode edge cases
|
||||
- Simpler to reason about
|
||||
|
||||
### Tests
|
||||
|
||||
- `../../evil` throws
|
||||
- `foo/bar` throws
|
||||
- Empty string throws
|
||||
- `my-group_01` passes
|
||||
- `A` (single char) passes
|
||||
- CLI smoke: one integration test that hits `getGroupDir` / `createGroupDir` (e.g. `group create` or `group add`) with an invalid name proves wiring for every subcommand that resolves a group through storage
|
||||
|
||||
### CLI/API entry points accepting groupName
|
||||
|
||||
All paths flow through `getGroupDir` (which validates), so coverage is implicit. For reference:
|
||||
|
||||
| Command | Entry | Calls |
|
||||
|---------|-------|-------|
|
||||
| `group create` | `cli/group.ts` action | `createGroupDir` -> `getGroupDir` |
|
||||
| `group add` | `cli/group.ts` action | `getGroupDir` |
|
||||
| `group remove` | `cli/group.ts` action | `getGroupDir` |
|
||||
| `group list` | `cli/group.ts` action | reads `groups/` dir directly — no traversal risk (reads, not writes) |
|
||||
| `group status` | `cli/group.ts` action | `getGroupDir` |
|
||||
| `group sync` | `cli/group.ts` action | `getGroupDir` |
|
||||
|
||||
**`listGroups`:** Reads directory names from disk without validation. Not a write path, so no traversal risk. May surface manually-created directories with non-conforming names — accepted as-is, not in scope.
|
||||
|
||||
---
|
||||
|
||||
## Fix 2: gRPC Proto Regex -> Brace-Depth Counter
|
||||
|
||||
**File:** `gitnexus/src/core/group/extractors/grpc-extractor.ts`
|
||||
**Risk:** `serviceRe = /service\s+(\w+)\s*\{([^}]*)}/gs` stops at first `}`. Proto services with `google.api.http` annotations inside RPCs contain nested `{ }` blocks.
|
||||
|
||||
### Solution
|
||||
|
||||
Replace `serviceRe` regex with `extractServiceBlocks(content: string): Array<{ name: string; body: string }>`:
|
||||
|
||||
1. Use regex only to find `service <Name> {` start positions (regex consumes the opening `{`)
|
||||
2. Initialise depth to 1 immediately after the opening `{`
|
||||
3. Scan forward char by char: `{` -> depth++, `}` -> depth--; collect into body
|
||||
4. Stop when depth reaches 0 (the matching closing `}`)
|
||||
5. Return name + body pairs
|
||||
|
||||
Inner `rpcRe` regex remains unchanged — it operates on the already-extracted body.
|
||||
|
||||
**Malformed input:** If EOF is reached before `depth` returns to 0, skip the incomplete service (do not add to results). Lock this in the test.
|
||||
|
||||
**Scope limitation (v1):** Brace-depth only — no lexer for string literals or comments containing `{`/`}`. Sufficient for `google.api.http` annotations. Known false positive: braces inside `//` comments or quoted strings within proto options. Accepted for v1; a proper proto lexer is out of scope.
|
||||
|
||||
### Tests
|
||||
|
||||
- Proto with single service, no nesting (regression)
|
||||
- Proto with `google.api.http` nested braces inside RPC options
|
||||
- Proto with multiple services
|
||||
- Proto with nested `option` blocks inside RPC (e.g. `google.api.http`)
|
||||
- Malformed proto with unclosed brace (graceful handling)
|
||||
|
||||
---
|
||||
|
||||
## Fix 3: Directory Exclusions in Service Boundary Detector
|
||||
|
||||
**File:** `gitnexus/src/core/group/service-boundary-detector.ts`
|
||||
**Risk:** Walks entire repo tree, only skipping dotfiles and `node_modules`. Extremely slow on repos with `vendor/`, `target/`, `__pycache__/`, `.venv/`.
|
||||
|
||||
### Solution
|
||||
|
||||
Create `EXCLUDED_DIRS` as a `Set<string>` (alongside existing `SERVICE_MARKERS`, `SOURCE_EXTENSIONS`), for example:
|
||||
|
||||
```text
|
||||
node_modules, vendor, target, build, dist,
|
||||
__pycache__, .venv, venv, .tox, .mypy_cache,
|
||||
.gradle, .mvn, out, bin
|
||||
```
|
||||
|
||||
(Implement as `new Set([...])` — the list above is the membership, not a string literal.)
|
||||
|
||||
Apply in both:
|
||||
- `walkForBoundaries` (line 77-78) — replace current inline `=== 'node_modules'` check with `EXCLUDED_DIRS.has(entry.name)`
|
||||
- `hasSourceFilesInSubdirs` (line 130) — replace `entry.name !== 'node_modules'` with `!EXCLUDED_DIRS.has(entry.name)`
|
||||
|
||||
Note: remove the old `=== 'node_modules'` literal from both locations — it is covered by `EXCLUDED_DIRS`.
|
||||
Dotfile exclusion (`.` prefix) remains as a separate check since it's a pattern, not a name.
|
||||
Exclusions apply only to `isDirectory()` entries — file names are never checked against `EXCLUDED_DIRS`.
|
||||
|
||||
**Tradeoff:** Rare layouts that keep source under names like `out/` or `bin/` will be skipped; accepted for performance on typical monorepos.
|
||||
|
||||
**Case sensitivity:** `Set.has` is case-sensitive (matches current `=== 'node_modules'` behavior). Windows case-insensitive FS not handled — accepted as-is, consistent with existing code.
|
||||
|
||||
### Tests
|
||||
|
||||
- Directory named `vendor/` is skipped
|
||||
- Directory named `target/` is skipped
|
||||
- Directory named `__pycache__/` is skipped
|
||||
- Regular source directories are NOT skipped
|
||||
- Dotfile directories still skipped (regression)
|
||||
|
||||
---
|
||||
|
||||
## Fix 4: Double-Close of LadybugDB Pools
|
||||
|
||||
**Files:**
|
||||
- `gitnexus/src/core/group/sync.ts` (lines 155-157) — per-id cleanup (KEEP)
|
||||
- `gitnexus/src/cli/group.ts` (line 188) — blanket `closeLbug()` (REMOVE)
|
||||
|
||||
**Risk:** In MCP server context, `closeLbug()` without arguments tears down ALL active pools, including ones from unrelated operations.
|
||||
|
||||
### Solution
|
||||
|
||||
Remove the `closeLbug()` call (no arguments) from `cli/group.ts` finally block. The per-id cleanup in `sync.ts` is sufficient:
|
||||
|
||||
```typescript
|
||||
// sync.ts — KEEP: cleans up only pools opened by this sync
|
||||
finally {
|
||||
for (const id of [...new Set(openPoolIds)]) {
|
||||
await closeLbug(id).catch(() => {});
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
```typescript
|
||||
// cli/group.ts — REMOVE: blanket close that kills all pools
|
||||
finally {
|
||||
await closeLbug().catch(() => {}); // DELETE THIS
|
||||
}
|
||||
```
|
||||
|
||||
Remove the `closeLbug` import from `cli/group.ts` — after removing the `finally` call it has no remaining usages.
|
||||
|
||||
### Tests (unit level — mock pool adapter)
|
||||
|
||||
- `syncGroup` closes only the pools it opened (mock `closeLbug`, assert called with specific ids)
|
||||
- Two-pool scenario: sync opens pools A and B, both closed in finally; pool C (opened elsewhere) not touched
|
||||
- CLI `sync` command does not call blanket `closeLbug()` (verify no zero-arg call in source — static check or grep-based test)
|
||||
|
||||
---
|
||||
|
||||
## Out of Scope
|
||||
|
||||
- JSON -> LadybugDB migration (tracked in #606)
|
||||
- MEDIUM/LOW issues (items 5-10 from review summary)
|
||||
- Test gap coverage beyond what's needed for these 4 fixes
|
||||
- Any refactoring or architectural changes
|
||||
|
||||
## Execution Order
|
||||
|
||||
Fixes are independent — can be implemented in parallel or any order.
|
||||
Recommended order for review clarity: 1 -> 3 -> 4 -> 2 (simplest to most complex).
|
||||
@@ -1,83 +0,0 @@
|
||||
import tsPlugin from '@typescript-eslint/eslint-plugin';
|
||||
import tsParser from '@typescript-eslint/parser';
|
||||
import unusedImports from 'eslint-plugin-unused-imports';
|
||||
import reactHooks from 'eslint-plugin-react-hooks';
|
||||
import prettierConfig from 'eslint-config-prettier';
|
||||
|
||||
export default [
|
||||
// Global ignores
|
||||
{
|
||||
ignores: [
|
||||
'**/dist/**',
|
||||
'**/node_modules/**',
|
||||
'**/coverage/**',
|
||||
'gitnexus/vendor/**',
|
||||
'gitnexus-web/src/vendor/**',
|
||||
'gitnexus/test/fixtures/**',
|
||||
'gitnexus-web/playwright-report/**',
|
||||
'gitnexus-web/test-results/**',
|
||||
'**/*.d.ts',
|
||||
'.claude/**',
|
||||
'.history/**',
|
||||
],
|
||||
},
|
||||
|
||||
// Base TypeScript config for all packages
|
||||
{
|
||||
files: ['**/*.{ts,tsx}'],
|
||||
languageOptions: {
|
||||
parser: tsParser,
|
||||
parserOptions: {
|
||||
ecmaVersion: 2022,
|
||||
sourceType: 'module',
|
||||
},
|
||||
},
|
||||
plugins: {
|
||||
'@typescript-eslint': tsPlugin,
|
||||
'unused-imports': unusedImports,
|
||||
},
|
||||
rules: {
|
||||
// Unused imports — auto-fixable
|
||||
'unused-imports/no-unused-imports': 'error',
|
||||
'unused-imports/no-unused-vars': [
|
||||
'warn',
|
||||
{ vars: 'all', varsIgnorePattern: '^_', args: 'after-used', argsIgnorePattern: '^_' },
|
||||
],
|
||||
|
||||
// TypeScript quality
|
||||
'@typescript-eslint/no-unused-vars': 'off', // handled by unused-imports plugin
|
||||
'no-unused-vars': 'off', // handled by unused-imports plugin
|
||||
'@typescript-eslint/no-explicit-any': 'warn',
|
||||
'@typescript-eslint/no-non-null-assertion': 'warn',
|
||||
|
||||
// General quality
|
||||
'no-debugger': 'error',
|
||||
'prefer-const': 'error',
|
||||
'no-var': 'error',
|
||||
eqeqeq: ['error', 'always', { null: 'ignore' }],
|
||||
},
|
||||
},
|
||||
|
||||
// CLI package — allow console.log (it's a CLI tool)
|
||||
{
|
||||
files: ['gitnexus/src/cli/**/*.ts', 'gitnexus/src/server/**/*.ts'],
|
||||
rules: {
|
||||
'no-console': 'off',
|
||||
},
|
||||
},
|
||||
|
||||
// React-specific rules for gitnexus-web
|
||||
{
|
||||
files: ['gitnexus-web/src/**/*.{ts,tsx}'],
|
||||
plugins: {
|
||||
'react-hooks': reactHooks,
|
||||
},
|
||||
rules: {
|
||||
'react-hooks/rules-of-hooks': 'error',
|
||||
'react-hooks/exhaustive-deps': 'warn',
|
||||
},
|
||||
},
|
||||
|
||||
// Disable formatting rules (prettier handles those)
|
||||
prettierConfig,
|
||||
];
|
||||
+2
-6
@@ -50,10 +50,6 @@ All models are routed through **OpenRouter** by default, so a single `OPENROUTER
|
||||
docker pull swebench/sweb.eval.x86_64.django_1776_django-16527:latest
|
||||
```
|
||||
|
||||
### Debug logging
|
||||
|
||||
Set `GITNEXUS_EVAL_DEBUG=1` to include full Python tracebacks in run summaries and logs. By default, errors are sanitized to avoid leaking host paths or stack traces.
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Debug a single instance
|
||||
@@ -152,7 +148,7 @@ Each mode has a `system_{mode}.jinja` + `instance_{mode}.jinja` pair. The agent
|
||||
|
||||
1. Docker container starts with SWE-bench instance (repo at specific commit)
|
||||
2. **GitNexus setup**: Node.js + gitnexus installed, `gitnexus analyze` runs (or restores from cache)
|
||||
3. **Eval-server starts**: `gitnexus eval-server` daemon (persistent HTTP server, keeps LadybugDB warm)
|
||||
3. **Eval-server starts**: `gitnexus eval-server` daemon (persistent HTTP server, keeps KuzuDB warm)
|
||||
4. **Standalone tool scripts installed** in `/usr/local/bin/` — works with `subprocess.run` (no `.bashrc` needed)
|
||||
5. Agent runs with the configured model + system prompt + GitNexus tools
|
||||
6. Agent's patch is extracted as a git diff
|
||||
@@ -171,7 +167,7 @@ Each tool script in `/usr/local/bin/` is standalone — no sourcing, no env inhe
|
||||
### Eval-server
|
||||
|
||||
The eval-server is a lightweight HTTP daemon that:
|
||||
- Keeps LadybugDB warm in memory (no cold start per tool call)
|
||||
- Keeps KuzuDB warm in memory (no cold start per tool call)
|
||||
- Returns LLM-friendly text (not raw JSON — saves tokens)
|
||||
- Includes next-step hints to guide tool chaining (query → context → impact → fix)
|
||||
- Auto-shuts down after idle timeout
|
||||
|
||||
@@ -22,10 +22,8 @@ import time
|
||||
from enum import Enum
|
||||
from pathlib import Path
|
||||
|
||||
from constants import AUGMENT_TIMEOUT_SECONDS
|
||||
from minisweagent import Environment, Model
|
||||
from minisweagent.agents.default import AgentConfig, DefaultAgent
|
||||
from tool_registry import BINARIES_BY_KEY, TOOL_METRIC_KEYS
|
||||
|
||||
logger = logging.getLogger("gitnexus_agent")
|
||||
|
||||
@@ -42,7 +40,7 @@ class GitNexusMode(str, Enum):
|
||||
class GitNexusAgentConfig(AgentConfig):
|
||||
"""Extended config for GitNexus evaluation agent."""
|
||||
gitnexus_mode: GitNexusMode = GitNexusMode.BASELINE
|
||||
augment_timeout: float = AUGMENT_TIMEOUT_SECONDS
|
||||
augment_timeout: float = 5.0
|
||||
augment_min_pattern_length: int = 3
|
||||
track_gitnexus_usage: bool = True
|
||||
|
||||
@@ -154,10 +152,16 @@ class GitNexusAgent(DefaultAgent):
|
||||
"""Track which GitNexus tools the agent uses."""
|
||||
for action in message.get("extra", {}).get("actions", []):
|
||||
command = action.get("command", "")
|
||||
for key, binary in BINARIES_BY_KEY.items():
|
||||
if binary in command and key in self.gitnexus_metrics.tool_calls:
|
||||
self.gitnexus_metrics.tool_calls[key] += 1
|
||||
break
|
||||
if "gitnexus-query" in command:
|
||||
self.gitnexus_metrics.tool_calls["query"] += 1
|
||||
elif "gitnexus-context" in command:
|
||||
self.gitnexus_metrics.tool_calls["context"] += 1
|
||||
elif "gitnexus-impact" in command:
|
||||
self.gitnexus_metrics.tool_calls["impact"] += 1
|
||||
elif "gitnexus-cypher" in command:
|
||||
self.gitnexus_metrics.tool_calls["cypher"] += 1
|
||||
elif "gitnexus-overview" in command:
|
||||
self.gitnexus_metrics.tool_calls["overview"] += 1
|
||||
|
||||
def serialize(self, *extra_dicts) -> dict:
|
||||
"""Serialize with GitNexus-specific metrics."""
|
||||
@@ -176,7 +180,13 @@ class GitNexusMetrics:
|
||||
"""Tracks GitNexus-specific metrics during evaluation."""
|
||||
|
||||
def __init__(self):
|
||||
self.tool_calls: dict[str, int] = {key: 0 for key in TOOL_METRIC_KEYS}
|
||||
self.tool_calls: dict[str, int] = {
|
||||
"query": 0,
|
||||
"context": 0,
|
||||
"impact": 0,
|
||||
"cypher": 0,
|
||||
"overview": 0,
|
||||
}
|
||||
self.augmentation_calls: int = 0
|
||||
self.augmentation_hits: int = 0
|
||||
self.augmentation_errors: int = 0
|
||||
|
||||
@@ -27,8 +27,6 @@ import typer
|
||||
from rich.console import Console
|
||||
from rich.table import Table
|
||||
|
||||
from tool_registry import TOOL_METRIC_KEYS
|
||||
|
||||
logger = logging.getLogger("analyze_results")
|
||||
console = Console()
|
||||
app = typer.Typer(rich_markup_mode="rich", add_completion=False)
|
||||
@@ -79,20 +77,13 @@ def load_run_results(results_dir: Path) -> dict[str, dict]:
|
||||
|
||||
|
||||
def parse_run_id(run_id: str) -> tuple[str, str]:
|
||||
"""Parse 'model_mode' into (model, mode) using known suffixes."""
|
||||
# Match the longest known suffix first to avoid hyphen collisions in model names.
|
||||
known_modes = [
|
||||
"native_augment",
|
||||
"native",
|
||||
"baseline",
|
||||
"mcp",
|
||||
"augment",
|
||||
"full",
|
||||
]
|
||||
for mode in known_modes:
|
||||
suffix = f"_{mode}"
|
||||
if run_id.endswith(suffix):
|
||||
return run_id[: -len(suffix)], mode
|
||||
"""Parse 'model_mode' into (model, mode)."""
|
||||
# Handle multi-word model names like 'minimax-2.5'
|
||||
# Modes are: baseline, mcp, augment, full
|
||||
known_modes = {"baseline", "mcp", "augment", "full"}
|
||||
parts = run_id.rsplit("_", 1)
|
||||
if len(parts) == 2 and parts[1] in known_modes:
|
||||
return parts[0], parts[1]
|
||||
return run_id, "unknown"
|
||||
|
||||
|
||||
@@ -275,17 +266,12 @@ def compare_modes(
|
||||
_, mode = parse_run_id(run_id)
|
||||
metrics[mode] = compute_metrics(run_data)
|
||||
|
||||
mode_order = [
|
||||
mode
|
||||
for mode in ["baseline", "native", "native_augment", "mcp", "augment", "full"]
|
||||
if mode in metrics
|
||||
] or sorted(metrics.keys())
|
||||
|
||||
# Print comparison table
|
||||
table = Table(title=f"Mode Comparison: {model}")
|
||||
table.add_column("Metric", style="bold")
|
||||
for mode in mode_order:
|
||||
table.add_column(mode, justify="right")
|
||||
for mode in ["baseline", "mcp", "augment", "full"]:
|
||||
if mode in metrics:
|
||||
table.add_column(mode, justify="right")
|
||||
|
||||
rows = [
|
||||
("Instances", "n_instances", "d"),
|
||||
@@ -302,7 +288,7 @@ def compare_modes(
|
||||
|
||||
for label, key, fmt in rows:
|
||||
values = []
|
||||
for mode in mode_order:
|
||||
for mode in ["baseline", "mcp", "augment", "full"]:
|
||||
if mode in metrics:
|
||||
v = metrics[mode].get(key, 0)
|
||||
if fmt == ".1%":
|
||||
@@ -321,8 +307,8 @@ def compare_modes(
|
||||
baseline_calls = metrics["baseline"]["avg_api_calls"]
|
||||
|
||||
table.add_section()
|
||||
for mode in mode_order:
|
||||
if mode == "baseline":
|
||||
for mode in ["mcp", "augment", "full"]:
|
||||
if mode not in metrics:
|
||||
continue
|
||||
mode_cost = metrics[mode]["avg_cost"]
|
||||
mode_calls = metrics[mode]["avg_api_calls"]
|
||||
@@ -354,8 +340,10 @@ def gitnexus_usage(
|
||||
|
||||
table = Table(title="Tool Usage by Run")
|
||||
table.add_column("Run", style="bold")
|
||||
for key in TOOL_METRIC_KEYS:
|
||||
table.add_column(key, justify="right")
|
||||
table.add_column("query", justify="right")
|
||||
table.add_column("context", justify="right")
|
||||
table.add_column("impact", justify="right")
|
||||
table.add_column("cypher", justify="right")
|
||||
table.add_column("Total", justify="right")
|
||||
table.add_column("Augment Hits", justify="right")
|
||||
|
||||
@@ -365,7 +353,7 @@ def gitnexus_usage(
|
||||
continue
|
||||
|
||||
# Aggregate tool calls across trajectories
|
||||
tool_totals: dict[str, int] = {key: 0 for key in TOOL_METRIC_KEYS}
|
||||
tool_totals: dict[str, int] = {"query": 0, "context": 0, "impact": 0, "cypher": 0, "overview": 0}
|
||||
augment_hits = 0
|
||||
|
||||
for traj in run_data.get("trajectories", {}).values():
|
||||
@@ -385,7 +373,10 @@ def gitnexus_usage(
|
||||
if total > 0 or augment_hits > 0:
|
||||
table.add_row(
|
||||
run_id,
|
||||
*[str(tool_totals.get(key, 0)) for key in TOOL_METRIC_KEYS],
|
||||
str(tool_totals.get("query", 0)),
|
||||
str(tool_totals.get("context", 0)),
|
||||
str(tool_totals.get("impact", 0)),
|
||||
str(tool_totals.get("cypher", 0)),
|
||||
str(total),
|
||||
str(augment_hits),
|
||||
)
|
||||
|
||||
+52
-95
@@ -2,7 +2,7 @@
|
||||
MCP Bridge for GitNexus
|
||||
|
||||
Starts the GitNexus MCP server as a subprocess and provides a Python interface
|
||||
to call MCP tools. Used by the bash wrapper scripts and the augmentation layer..
|
||||
to call MCP tools. Used by the bash wrapper scripts and the augmentation layer.
|
||||
|
||||
The bridge communicates with the MCP server via stdio using the JSON-RPC protocol.
|
||||
"""
|
||||
@@ -17,14 +17,6 @@ import time
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
|
||||
from constants import (
|
||||
MCP_FIND_GITNEXUS_FALLBACK_TIMEOUT_SECONDS,
|
||||
MCP_FIND_GITNEXUS_TIMEOUT_SECONDS,
|
||||
MCP_READ_TIMEOUT_SECONDS,
|
||||
MCP_STOP_WAIT_SECONDS,
|
||||
)
|
||||
from utils.errors import is_debug_enabled, log_safe_exception
|
||||
|
||||
logger = logging.getLogger("mcp_bridge")
|
||||
|
||||
|
||||
@@ -53,13 +45,13 @@ class MCPBridge:
|
||||
|
||||
try:
|
||||
# Find gitnexus binary
|
||||
gitnexus_cmd = self._find_gitnexus_command()
|
||||
if not gitnexus_cmd:
|
||||
gitnexus_bin = self._find_gitnexus()
|
||||
if not gitnexus_bin:
|
||||
logger.error("GitNexus not found. Install with: npm install -g gitnexus")
|
||||
return False
|
||||
|
||||
self.process = subprocess.Popen(
|
||||
[*gitnexus_cmd, "mcp"],
|
||||
[gitnexus_bin, "mcp"],
|
||||
stdin=subprocess.PIPE,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=subprocess.PIPE,
|
||||
@@ -86,7 +78,7 @@ class MCPBridge:
|
||||
return True
|
||||
|
||||
except Exception as e:
|
||||
log_safe_exception(logger, "Failed to start MCP bridge", e, include_debug=is_debug_enabled())
|
||||
logger.error(f"Failed to start MCP bridge: {e}")
|
||||
self.stop()
|
||||
return False
|
||||
|
||||
@@ -94,14 +86,9 @@ class MCPBridge:
|
||||
"""Stop the MCP server subprocess."""
|
||||
if self.process:
|
||||
try:
|
||||
if self.process.stdin:
|
||||
self.process.stdin.close()
|
||||
if self.process.stdout:
|
||||
self.process.stdout.close()
|
||||
if self.process.stderr:
|
||||
self.process.stderr.close()
|
||||
self.process.stdin.close()
|
||||
self.process.terminate()
|
||||
self.process.wait(timeout=MCP_STOP_WAIT_SECONDS)
|
||||
self.process.wait(timeout=5)
|
||||
except Exception:
|
||||
try:
|
||||
self.process.kill()
|
||||
@@ -152,34 +139,29 @@ class MCPBridge:
|
||||
return contents[0].get("text", "")
|
||||
return None
|
||||
|
||||
def _find_gitnexus_command(self) -> list[str] | None:
|
||||
"""Find the gitnexus CLI command prefix."""
|
||||
def _find_gitnexus(self) -> str | None:
|
||||
"""Find the gitnexus CLI binary."""
|
||||
# Check if npx is available (preferred - uses local install)
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["npx", "gitnexus", "--version"],
|
||||
stdin=subprocess.DEVNULL,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=MCP_FIND_GITNEXUS_TIMEOUT_SECONDS,
|
||||
cwd=self.repo_path,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return ["npx", "gitnexus"]
|
||||
except Exception:
|
||||
pass
|
||||
for cmd in ["npx"]:
|
||||
try:
|
||||
result = subprocess.run(
|
||||
[cmd, "gitnexus", "--version"],
|
||||
capture_output=True, text=True, timeout=15,
|
||||
cwd=self.repo_path,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return cmd # Will use "npx gitnexus mcp"
|
||||
except Exception:
|
||||
continue
|
||||
|
||||
# Check for global install
|
||||
try:
|
||||
result = subprocess.run(
|
||||
["gitnexus", "--version"],
|
||||
stdin=subprocess.DEVNULL,
|
||||
capture_output=True,
|
||||
text=True,
|
||||
timeout=MCP_FIND_GITNEXUS_FALLBACK_TIMEOUT_SECONDS,
|
||||
capture_output=True, text=True, timeout=10,
|
||||
)
|
||||
if result.returncode == 0:
|
||||
return ["gitnexus"]
|
||||
return "gitnexus"
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
@@ -212,7 +194,7 @@ class MCPBridge:
|
||||
self.process.stdin.flush()
|
||||
|
||||
# Read response
|
||||
response = self._read_response(timeout=MCP_READ_TIMEOUT_SECONDS)
|
||||
response = self._read_response(timeout=30)
|
||||
if response and response.get("id") == request_id:
|
||||
if "error" in response:
|
||||
logger.error(f"MCP error: {response['error']}")
|
||||
@@ -221,7 +203,7 @@ class MCPBridge:
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
log_safe_exception(logger, "MCP request failed", e, include_debug=is_debug_enabled())
|
||||
logger.error(f"MCP request failed: {e}")
|
||||
return None
|
||||
|
||||
def _send_notification(self, method: str, params: dict):
|
||||
@@ -242,67 +224,42 @@ class MCPBridge:
|
||||
self.process.stdin.write(message.encode("utf-8"))
|
||||
self.process.stdin.flush()
|
||||
except Exception as e:
|
||||
log_safe_exception(logger, "MCP notification failed", e, include_debug=is_debug_enabled())
|
||||
logger.error(f"MCP notification failed: {e}")
|
||||
|
||||
def _read_content_length(self, deadline: float) -> int | None:
|
||||
"""Read Content-Length header, returning the byte length or None."""
|
||||
if not self.process or not self.process.stdout:
|
||||
return None
|
||||
|
||||
header_line = b""
|
||||
while time.time() < deadline:
|
||||
byte = self.process.stdout.read(1)
|
||||
if not byte:
|
||||
return None
|
||||
header_line += byte
|
||||
if header_line.endswith(b"\r\n\r\n") or header_line.endswith(b"\n\n"):
|
||||
break
|
||||
|
||||
if not header_line:
|
||||
return None
|
||||
|
||||
header_str = header_line.decode("utf-8").strip()
|
||||
for line in header_str.split("\r\n"):
|
||||
if line.lower().startswith("content-length:"):
|
||||
try:
|
||||
return int(line.split(":", 1)[1].strip())
|
||||
except (ValueError, IndexError):
|
||||
return None
|
||||
return None
|
||||
|
||||
def _read_body(self, content_length: int, deadline: float) -> bytes | None:
|
||||
"""Read a response body of the expected length before deadline."""
|
||||
if not self.process or not self.process.stdout:
|
||||
return None
|
||||
|
||||
remaining = content_length
|
||||
chunks: list[bytes] = []
|
||||
|
||||
while remaining > 0 and time.time() < deadline:
|
||||
chunk = self.process.stdout.read(remaining)
|
||||
if not chunk:
|
||||
return None
|
||||
chunks.append(chunk)
|
||||
remaining -= len(chunk)
|
||||
|
||||
if remaining > 0:
|
||||
return None
|
||||
|
||||
return b"".join(chunks)
|
||||
|
||||
def _read_response(self, timeout: float = MCP_READ_TIMEOUT_SECONDS) -> dict | None:
|
||||
def _read_response(self, timeout: float = 30) -> dict | None:
|
||||
"""Read a JSON-RPC response from the MCP server."""
|
||||
if not self.process or not self.process.stdout:
|
||||
return None
|
||||
|
||||
start = time.time()
|
||||
|
||||
try:
|
||||
deadline = time.time() + timeout
|
||||
while time.time() < deadline:
|
||||
content_length = self._read_content_length(deadline)
|
||||
while time.time() - start < timeout:
|
||||
# Read Content-Length header
|
||||
header_line = b""
|
||||
while True:
|
||||
byte = self.process.stdout.read(1)
|
||||
if not byte:
|
||||
return None
|
||||
header_line += byte
|
||||
if header_line.endswith(b"\r\n\r\n"):
|
||||
break
|
||||
if header_line.endswith(b"\n\n"):
|
||||
break
|
||||
|
||||
# Parse content length
|
||||
header_str = header_line.decode("utf-8").strip()
|
||||
content_length = None
|
||||
for line in header_str.split("\r\n"):
|
||||
if line.lower().startswith("content-length:"):
|
||||
content_length = int(line.split(":")[1].strip())
|
||||
break
|
||||
|
||||
if content_length is None:
|
||||
continue
|
||||
|
||||
body = self._read_body(content_length, deadline)
|
||||
# Read body
|
||||
body = self.process.stdout.read(content_length)
|
||||
if not body:
|
||||
return None
|
||||
|
||||
@@ -315,7 +272,7 @@ class MCPBridge:
|
||||
return None
|
||||
|
||||
except Exception as e:
|
||||
log_safe_exception(logger, "Error reading MCP response", e, include_debug=is_debug_enabled())
|
||||
logger.error(f"Error reading MCP response: {e}")
|
||||
return None
|
||||
|
||||
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
# Claude Haiku 4.5 — fast, cheap, good baseline
|
||||
# Via OpenRouter (set OPENROUTER_API_KEY in .env)
|
||||
model:
|
||||
model_name: 'openrouter/anthropic/claude-haiku-4.5'
|
||||
cost_tracking: 'ignore_errors'
|
||||
model_name: "openrouter/anthropic/claude-haiku-4.5"
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 8192
|
||||
temperature: 0
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
# Via OpenRouter (set OPENROUTER_API_KEY in .env)
|
||||
# To use Anthropic directly, change to: anthropic/claude-opus-4-20250514
|
||||
model:
|
||||
model_name: 'openrouter/anthropic/claude-opus-4'
|
||||
cost_tracking: 'ignore_errors'
|
||||
model_name: "openrouter/anthropic/claude-opus-4"
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 16384
|
||||
temperature: 0
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
# Via OpenRouter (set OPENROUTER_API_KEY in .env)
|
||||
# To use Anthropic directly, change to: anthropic/claude-sonnet-4-20250514
|
||||
model:
|
||||
model_name: 'openrouter/anthropic/claude-sonnet-4'
|
||||
cost_tracking: 'ignore_errors'
|
||||
model_name: "openrouter/anthropic/claude-sonnet-4"
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 16384
|
||||
temperature: 0
|
||||
|
||||
@@ -1,13 +0,0 @@
|
||||
model: deepseek-ai/deepseek-chat
|
||||
provider: openrouter
|
||||
cost:
|
||||
input: 0.14 # per 1M tokens
|
||||
output: 0.28 # per 1M tokens
|
||||
|
||||
# Native DeepSeek API (direct)
|
||||
api_key: null
|
||||
base_url: null
|
||||
|
||||
# For OpenRouter, uncomment below and comment out direct config above
|
||||
# api_key: \${OPENROUTER_API_KEY}
|
||||
# base_url: https://openrouter.ai/api/v1
|
||||
@@ -1,15 +0,0 @@
|
||||
model: deepseek-ai/DeepSeek-V3
|
||||
provider: openrouter
|
||||
cost:
|
||||
input: 0.27 # per 1M tokens
|
||||
output: 1.10 # per 1M tokens
|
||||
|
||||
# Native DeepSeek API (direct)
|
||||
# Get your API key at: https://platform.deepseek.com/
|
||||
# Or use OpenRouter with: OPENROUTER_API_KEY
|
||||
api_key: null
|
||||
base_url: null
|
||||
|
||||
# For OpenRouter, uncomment below and comment out direct config above
|
||||
# api_key: \${OPENROUTER_API_KEY}
|
||||
# base_url: https://openrouter.ai/api/v1
|
||||
@@ -1,7 +1,7 @@
|
||||
# GLM 4.7 — via OpenRouter (set OPENROUTER_API_KEY in .env)
|
||||
model:
|
||||
model_name: 'openrouter/zhipuai/glm-4.7'
|
||||
cost_tracking: 'ignore_errors'
|
||||
model_name: "openrouter/zhipuai/glm-4.7"
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 8192
|
||||
temperature: 0
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# GLM 5 — via OpenRouter (set OPENROUTER_API_KEY in .env)
|
||||
model:
|
||||
model_name: 'openrouter/zhipuai/glm-5'
|
||||
cost_tracking: 'ignore_errors'
|
||||
model_name: "openrouter/zhipuai/glm-5"
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 8192
|
||||
temperature: 0
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# MiniMax M1 2.5 — via OpenRouter (set OPENROUTER_API_KEY in .env)
|
||||
model:
|
||||
model_name: 'openrouter/minimax/minimax-m1-2.5'
|
||||
cost_tracking: 'ignore_errors'
|
||||
model_name: "openrouter/minimax/minimax-m1-2.5"
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 8192
|
||||
temperature: 0
|
||||
|
||||
@@ -3,9 +3,9 @@
|
||||
# The action_regex tells mini-swe-agent to parse ```bash blocks from responses.
|
||||
model:
|
||||
model_class: litellm_textbased
|
||||
model_name: 'openrouter/minimax/minimax-m2.5'
|
||||
model_name: "openrouter/minimax/minimax-m2.5"
|
||||
action_regex: "```(?:bash|mswea_bash_command)\\s*\\n(.*?)\\n```"
|
||||
cost_tracking: 'ignore_errors'
|
||||
cost_tracking: "ignore_errors"
|
||||
model_kwargs:
|
||||
max_tokens: 8192
|
||||
temperature: 0
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
# Baseline mode — no GitNexus, pure mini-swe-agent (control group)
|
||||
agent:
|
||||
agent_class: 'eval.agents.gitnexus_agent.GitNexusAgent'
|
||||
gitnexus_mode: 'baseline'
|
||||
agent_class: "eval.agents.gitnexus_agent.GitNexusAgent"
|
||||
gitnexus_mode: "baseline"
|
||||
step_limit: 30
|
||||
cost_limit: 3.0
|
||||
|
||||
environment:
|
||||
environment_class: 'docker'
|
||||
environment_class: "docker"
|
||||
|
||||
@@ -5,14 +5,14 @@
|
||||
#
|
||||
# Use this mode to isolate the value of explicit tools without grep augmentation.
|
||||
agent:
|
||||
agent_class: 'eval.agents.gitnexus_agent.GitNexusAgent'
|
||||
gitnexus_mode: 'native'
|
||||
agent_class: "eval.agents.gitnexus_agent.GitNexusAgent"
|
||||
gitnexus_mode: "native"
|
||||
step_limit: 30
|
||||
cost_limit: 3.0
|
||||
track_gitnexus_usage: true
|
||||
|
||||
environment:
|
||||
environment_class: 'eval.environments.gitnexus_docker.GitNexusDockerEnvironment'
|
||||
environment_class: "eval.environments.gitnexus_docker.GitNexusDockerEnvironment"
|
||||
enable_gitnexus: true
|
||||
skip_embeddings: true
|
||||
gitnexus_timeout: 120
|
||||
|
||||
@@ -8,8 +8,8 @@
|
||||
#
|
||||
# The agent decides when to use explicit tools vs rely on enriched grep results.
|
||||
agent:
|
||||
agent_class: 'eval.agents.gitnexus_agent.GitNexusAgent'
|
||||
gitnexus_mode: 'native_augment'
|
||||
agent_class: "eval.agents.gitnexus_agent.GitNexusAgent"
|
||||
gitnexus_mode: "native_augment"
|
||||
step_limit: 30
|
||||
cost_limit: 3.0
|
||||
augment_timeout: 5.0
|
||||
@@ -17,7 +17,7 @@ agent:
|
||||
track_gitnexus_usage: true
|
||||
|
||||
environment:
|
||||
environment_class: 'eval.environments.gitnexus_docker.GitNexusDockerEnvironment'
|
||||
environment_class: "eval.environments.gitnexus_docker.GitNexusDockerEnvironment"
|
||||
enable_gitnexus: true
|
||||
skip_embeddings: true
|
||||
gitnexus_timeout: 120
|
||||
|
||||
@@ -1,15 +0,0 @@
|
||||
DEBUG_ENV_VAR = "GITNEXUS_EVAL_DEBUG"
|
||||
|
||||
# GitNexus eval-server health checks
|
||||
EVAL_SERVER_HEALTH_RETRIES = 30
|
||||
EVAL_SERVER_HEALTH_INTERVAL_SECONDS = 0.5
|
||||
EVAL_SERVER_HEALTH_TIMEOUT_SECONDS = 3
|
||||
|
||||
# MCP bridge timeouts
|
||||
MCP_FIND_GITNEXUS_TIMEOUT_SECONDS = 15
|
||||
MCP_FIND_GITNEXUS_FALLBACK_TIMEOUT_SECONDS = 10
|
||||
MCP_READ_TIMEOUT_SECONDS = 30
|
||||
MCP_STOP_WAIT_SECONDS = 5
|
||||
|
||||
# Agent defaults
|
||||
AUGMENT_TIMEOUT_SECONDS = 5.0
|
||||
@@ -26,20 +26,72 @@ import shutil
|
||||
import time
|
||||
from pathlib import Path
|
||||
|
||||
from constants import (
|
||||
EVAL_SERVER_HEALTH_INTERVAL_SECONDS,
|
||||
EVAL_SERVER_HEALTH_RETRIES,
|
||||
EVAL_SERVER_HEALTH_TIMEOUT_SECONDS,
|
||||
)
|
||||
from minisweagent.environments.docker import DockerEnvironment
|
||||
from tool_registry import TOOL_SPECS, ToolScriptSpec
|
||||
from utils.errors import is_debug_enabled, log_safe_exception
|
||||
|
||||
logger = logging.getLogger("gitnexus_docker")
|
||||
|
||||
DEFAULT_CACHE_DIR = Path.home() / ".gitnexus-eval-cache"
|
||||
EVAL_SERVER_PORT = 4848
|
||||
|
||||
# Standalone tool scripts installed into /usr/local/bin/ inside the container.
|
||||
# Each script calls the eval-server via curl, with a CLI fallback.
|
||||
# These are standalone — no sourcing, no env inheritance needed.
|
||||
|
||||
TOOL_SCRIPT_QUERY = r'''#!/bin/bash
|
||||
PORT="${GITNEXUS_EVAL_PORT:-__PORT__}"
|
||||
query="$1"; task_ctx="${2:-}"; goal="${3:-}"
|
||||
[ -z "$query" ] && echo "Usage: gitnexus-query <query> [task_context] [goal]" && exit 1
|
||||
args="{\"query\": \"$query\""
|
||||
[ -n "$task_ctx" ] && args="$args, \"task_context\": \"$task_ctx\""
|
||||
[ -n "$goal" ] && args="$args, \"goal\": \"$goal\""
|
||||
args="$args}"
|
||||
result=$(curl -sf -X POST "http://127.0.0.1:${PORT}/tool/query" -H "Content-Type: application/json" -d "$args" 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$result" ]; then echo "$result"; exit 0; fi
|
||||
cd /testbed && npx gitnexus query "$query" 2>&1
|
||||
'''
|
||||
|
||||
TOOL_SCRIPT_CONTEXT = r'''#!/bin/bash
|
||||
PORT="${GITNEXUS_EVAL_PORT:-__PORT__}"
|
||||
name="$1"; file_path="${2:-}"
|
||||
[ -z "$name" ] && echo "Usage: gitnexus-context <symbol_name> [file_path]" && exit 1
|
||||
args="{\"name\": \"$name\""
|
||||
[ -n "$file_path" ] && args="$args, \"file_path\": \"$file_path\""
|
||||
args="$args}"
|
||||
result=$(curl -sf -X POST "http://127.0.0.1:${PORT}/tool/context" -H "Content-Type: application/json" -d "$args" 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$result" ]; then echo "$result"; exit 0; fi
|
||||
cd /testbed && npx gitnexus context "$name" 2>&1
|
||||
'''
|
||||
|
||||
TOOL_SCRIPT_IMPACT = r'''#!/bin/bash
|
||||
PORT="${GITNEXUS_EVAL_PORT:-__PORT__}"
|
||||
target="$1"; direction="${2:-upstream}"
|
||||
[ -z "$target" ] && echo "Usage: gitnexus-impact <symbol_name> [upstream|downstream]" && exit 1
|
||||
result=$(curl -sf -X POST "http://127.0.0.1:${PORT}/tool/impact" -H "Content-Type: application/json" -d "{\"target\": \"$target\", \"direction\": \"$direction\"}" 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$result" ]; then echo "$result"; exit 0; fi
|
||||
cd /testbed && npx gitnexus impact "$target" --direction "$direction" 2>&1
|
||||
'''
|
||||
|
||||
TOOL_SCRIPT_CYPHER = r'''#!/bin/bash
|
||||
PORT="${GITNEXUS_EVAL_PORT:-__PORT__}"
|
||||
query="$1"
|
||||
[ -z "$query" ] && echo "Usage: gitnexus-cypher <cypher_query>" && exit 1
|
||||
result=$(curl -sf -X POST "http://127.0.0.1:${PORT}/tool/cypher" -H "Content-Type: application/json" -d "{\"query\": \"$query\"}" 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$result" ]; then echo "$result"; exit 0; fi
|
||||
cd /testbed && npx gitnexus cypher "$query" 2>&1
|
||||
'''
|
||||
|
||||
TOOL_SCRIPT_AUGMENT = r'''#!/bin/bash
|
||||
cd /testbed && npx gitnexus augment "$1" 2>&1 || true
|
||||
'''
|
||||
|
||||
TOOL_SCRIPT_OVERVIEW = r'''#!/bin/bash
|
||||
PORT="${GITNEXUS_EVAL_PORT:-__PORT__}"
|
||||
echo "=== Code Knowledge Graph Overview ==="
|
||||
result=$(curl -sf -X POST "http://127.0.0.1:${PORT}/tool/list_repos" -H "Content-Type: application/json" -d "{}" 2>/dev/null)
|
||||
if [ $? -eq 0 ] && [ -n "$result" ]; then echo "$result"; exit 0; fi
|
||||
cd /testbed && npx gitnexus list 2>&1
|
||||
'''
|
||||
|
||||
|
||||
class GitNexusDockerEnvironment(DockerEnvironment):
|
||||
"""
|
||||
@@ -81,13 +133,7 @@ class GitNexusDockerEnvironment(DockerEnvironment):
|
||||
try:
|
||||
self._setup_gitnexus()
|
||||
except Exception as e:
|
||||
log_safe_exception(
|
||||
logger,
|
||||
"GitNexus setup failed, continuing without it",
|
||||
e,
|
||||
include_debug=is_debug_enabled(),
|
||||
level="warning",
|
||||
)
|
||||
logger.warning(f"GitNexus setup failed, continuing without it: {e}")
|
||||
self._gitnexus_ready = False
|
||||
|
||||
return result
|
||||
@@ -176,59 +222,27 @@ class GitNexusDockerEnvironment(DockerEnvironment):
|
||||
"timeout": 5,
|
||||
})
|
||||
|
||||
# Wait for the server to be ready (up to ~15s for KuzuDB init)
|
||||
for i in range(EVAL_SERVER_HEALTH_RETRIES):
|
||||
time.sleep(EVAL_SERVER_HEALTH_INTERVAL_SECONDS)
|
||||
# Wait for the server to be ready (up to 15s for KuzuDB init)
|
||||
for i in range(30):
|
||||
time.sleep(0.5)
|
||||
health = self.execute({
|
||||
"command": f"curl -sf http://127.0.0.1:{self.eval_server_port}/health 2>/dev/null || echo 'NOT_READY'",
|
||||
"timeout": EVAL_SERVER_HEALTH_TIMEOUT_SECONDS,
|
||||
"timeout": 3,
|
||||
})
|
||||
output = health.get("output", "").strip()
|
||||
if "NOT_READY" not in output and "ok" in output:
|
||||
logger.info(
|
||||
f"Eval-server ready after {(i + 1) * EVAL_SERVER_HEALTH_INTERVAL_SECONDS:.1f}s"
|
||||
)
|
||||
logger.info(f"Eval-server ready after {(i + 1) * 0.5:.1f}s")
|
||||
return
|
||||
|
||||
log_output = self.execute({
|
||||
"command": "cat /tmp/gitnexus-eval-server.log 2>/dev/null | tail -20",
|
||||
})
|
||||
logger.warning(
|
||||
f"Eval-server didn't become ready in "
|
||||
f"{EVAL_SERVER_HEALTH_RETRIES * EVAL_SERVER_HEALTH_INTERVAL_SECONDS:.1f}s. "
|
||||
f"Eval-server didn't become ready in 15s. "
|
||||
f"Tools will fall back to direct CLI.\n"
|
||||
f"Server log: {log_output.get('output', 'N/A')}"
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _render_tool_script(spec: ToolScriptSpec, port: str) -> str:
|
||||
"""
|
||||
Render a standalone bash script for a GitNexus tool.
|
||||
|
||||
Scripts call the eval-server fast path when an endpoint is present,
|
||||
and fall back to the CLI otherwise.
|
||||
"""
|
||||
lines = ["#!/bin/bash"]
|
||||
|
||||
if spec.endpoint:
|
||||
lines.append(f'PORT="${{GITNEXUS_EVAL_PORT:-{port}}}"')
|
||||
|
||||
if spec.header:
|
||||
lines.append(spec.header.strip())
|
||||
|
||||
if spec.payload_builder:
|
||||
lines.append(spec.payload_builder.strip())
|
||||
|
||||
if spec.endpoint:
|
||||
lines.append(
|
||||
f'result=$(curl -sf -X POST "http://127.0.0.1:${{PORT}}{spec.endpoint}" '
|
||||
'-H "Content-Type: application/json" -d "$payload" 2>/dev/null)'
|
||||
)
|
||||
lines.append('if [ $? -eq 0 ] && [ -n "$result" ]; then echo "$result"; exit 0; fi')
|
||||
|
||||
lines.append(spec.fallback.strip())
|
||||
return "\n".join(lines)
|
||||
|
||||
def _install_tools(self):
|
||||
"""
|
||||
Install standalone GitNexus tool scripts in /usr/local/bin/.
|
||||
@@ -245,20 +259,24 @@ class GitNexusDockerEnvironment(DockerEnvironment):
|
||||
"""
|
||||
port = str(self.eval_server_port)
|
||||
|
||||
for spec in TOOL_SPECS.values():
|
||||
script_content = self._render_tool_script(spec, port).strip()
|
||||
tools = {
|
||||
"gitnexus-query": TOOL_SCRIPT_QUERY,
|
||||
"gitnexus-context": TOOL_SCRIPT_CONTEXT,
|
||||
"gitnexus-impact": TOOL_SCRIPT_IMPACT,
|
||||
"gitnexus-cypher": TOOL_SCRIPT_CYPHER,
|
||||
"gitnexus-augment": TOOL_SCRIPT_AUGMENT,
|
||||
"gitnexus-overview": TOOL_SCRIPT_OVERVIEW,
|
||||
}
|
||||
|
||||
for name, script in tools.items():
|
||||
script_content = script.replace("__PORT__", port).strip()
|
||||
# Use heredoc with quoted delimiter — prevents all variable expansion and quoting issues
|
||||
self.execute({
|
||||
"command": (
|
||||
f"cat << 'GITNEXUS_SCRIPT_EOF' > /usr/local/bin/{spec.bin_name}\n"
|
||||
f"{script_content}\n"
|
||||
"GITNEXUS_SCRIPT_EOF\n"
|
||||
f"chmod +x /usr/local/bin/{spec.bin_name}"
|
||||
),
|
||||
"command": f"cat << 'GITNEXUS_SCRIPT_EOF' > /usr/local/bin/{name}\n{script_content}\nGITNEXUS_SCRIPT_EOF\nchmod +x /usr/local/bin/{name}",
|
||||
"timeout": 5,
|
||||
})
|
||||
|
||||
logger.info(f"Installed {len(TOOL_SPECS)} GitNexus tool scripts in /usr/local/bin/")
|
||||
logger.info(f"Installed {len(tools)} GitNexus tool scripts in /usr/local/bin/")
|
||||
|
||||
def _get_repo_info(self) -> dict:
|
||||
"""Get repository identity info from the container."""
|
||||
@@ -307,13 +325,7 @@ class GitNexusDockerEnvironment(DockerEnvironment):
|
||||
logger.info(f"Cached GitNexus index: {cache_path}")
|
||||
|
||||
except Exception as e:
|
||||
log_safe_exception(
|
||||
logger,
|
||||
"Failed to cache GitNexus index",
|
||||
e,
|
||||
include_debug=is_debug_enabled(),
|
||||
level="warning",
|
||||
)
|
||||
logger.warning(f"Failed to cache GitNexus index: {e}")
|
||||
if cache_path.exists():
|
||||
shutil.rmtree(cache_path, ignore_errors=True)
|
||||
|
||||
@@ -349,13 +361,7 @@ class GitNexusDockerEnvironment(DockerEnvironment):
|
||||
logger.info("GitNexus index restored from cache")
|
||||
|
||||
except Exception as e:
|
||||
log_safe_exception(
|
||||
logger,
|
||||
"Failed to restore cache, re-indexing",
|
||||
e,
|
||||
include_debug=is_debug_enabled(),
|
||||
level="warning",
|
||||
)
|
||||
logger.warning(f"Failed to restore cache, re-indexing: {e}")
|
||||
self._index_repository()
|
||||
|
||||
def stop(self) -> dict:
|
||||
|
||||
+3
-5
@@ -6,7 +6,7 @@ readme = "README.md"
|
||||
requires-python = ">=3.11"
|
||||
dependencies = [
|
||||
"mini-swe-agent>=2.0.0",
|
||||
"litellm>=1.50.0,!=1.82.7,!=1.82.8",
|
||||
"litellm>=1.50.0",
|
||||
"datasets>=3.0.0",
|
||||
"typer>=0.12.0",
|
||||
"rich>=13.0.0",
|
||||
@@ -20,8 +20,6 @@ dependencies = [
|
||||
dev = [
|
||||
"pytest>=8.0.0",
|
||||
"ruff>=0.5.0",
|
||||
"hypothesis>=6.88.0",
|
||||
"coverage>=7.6.0",
|
||||
]
|
||||
|
||||
[project.scripts]
|
||||
@@ -33,8 +31,8 @@ requires = ["hatchling"]
|
||||
build-backend = "hatchling.build"
|
||||
|
||||
[tool.hatch.build.targets.wheel]
|
||||
packages = ["agents", "environments", "analysis", "bridge", "utils"]
|
||||
extra-files = ["run_eval.py", "tool_registry.py", "constants.py"]
|
||||
packages = ["agents", "environments", "analysis", "bridge"]
|
||||
extra-files = ["run_eval.py"]
|
||||
|
||||
[tool.ruff]
|
||||
line-length = 120
|
||||
|
||||
+37
-73
@@ -25,6 +25,7 @@ import logging
|
||||
import os
|
||||
import threading
|
||||
import time
|
||||
import traceback
|
||||
from itertools import product
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
@@ -35,8 +36,6 @@ from rich.console import Console
|
||||
from rich.live import Live
|
||||
from rich.table import Table
|
||||
|
||||
from utils.errors import is_debug_enabled, log_safe_exception
|
||||
|
||||
# Load .env file from eval/ directory
|
||||
_env_file = Path(__file__).parent / ".env"
|
||||
if _env_file.exists():
|
||||
@@ -139,65 +138,6 @@ def get_swebench_docker_image(instance: dict) -> str:
|
||||
return image_name
|
||||
|
||||
|
||||
def _build_model(config: dict):
|
||||
"""Construct the model from config."""
|
||||
from minisweagent.models import get_model
|
||||
|
||||
return get_model(config=config.get("model", {}))
|
||||
|
||||
|
||||
def _build_environment(config: dict, instance: dict):
|
||||
"""Construct the environment for the instance."""
|
||||
env_config = dict(config.get("environment", {}))
|
||||
env_class_name = env_config.pop("environment_class", "docker")
|
||||
|
||||
if env_class_name == "eval.environments.gitnexus_docker.GitNexusDockerEnvironment":
|
||||
from environments.gitnexus_docker import GitNexusDockerEnvironment
|
||||
|
||||
env_config["image"] = get_swebench_docker_image(instance)
|
||||
return GitNexusDockerEnvironment(**env_config)
|
||||
|
||||
from minisweagent.environments.docker import DockerEnvironment
|
||||
|
||||
return DockerEnvironment(image=get_swebench_docker_image(instance), **env_config)
|
||||
|
||||
|
||||
def _build_agent(config: dict, model, env, instance_dir: Path, instance_id: str):
|
||||
"""Construct the GitNexus agent with trajectory output configured."""
|
||||
from agents.gitnexus_agent import GitNexusAgent
|
||||
|
||||
agent_config = dict(config.get("agent", {}))
|
||||
agent_config.pop("agent_class", "eval.agents.gitnexus_agent.GitNexusAgent")
|
||||
traj_path = instance_dir / f"{instance_id}.traj.json"
|
||||
agent_config["output_path"] = traj_path
|
||||
return GitNexusAgent(model, env, **agent_config)
|
||||
|
||||
|
||||
def _extract_submission(env, info: dict, run_id: str) -> str:
|
||||
"""Pull the git diff patch from the container, falling back to the agent submission."""
|
||||
try:
|
||||
patch_output = env.execute({"command": "cd /testbed && git diff"})
|
||||
return patch_output.get("output", "").strip()
|
||||
except Exception as patch_err:
|
||||
logger.warning(f"[{run_id}] Failed to extract patch: {patch_err}")
|
||||
return info.get("submission", "")
|
||||
|
||||
|
||||
def _record_failure(run_id: str, instance_id: str, result: dict, error: Exception):
|
||||
sanitized = log_safe_exception(
|
||||
logger,
|
||||
f"[{run_id}] Error on {instance_id}",
|
||||
error,
|
||||
include_debug=is_debug_enabled(),
|
||||
)
|
||||
result["exit_status"] = sanitized["error_type"]
|
||||
result["error_type"] = sanitized["error_type"]
|
||||
result["error_message"] = sanitized["error_message"]
|
||||
result["error"] = sanitized["error_message"]
|
||||
if "error_detail_debug" in sanitized:
|
||||
result["error_detail_debug"] = sanitized["error_detail_debug"]
|
||||
|
||||
|
||||
def process_instance(
|
||||
instance: dict,
|
||||
config: dict,
|
||||
@@ -209,6 +149,8 @@ def process_instance(
|
||||
Process a single SWE-bench instance with the given config.
|
||||
Returns result dict with instance_id, exit_status, submission, metrics.
|
||||
"""
|
||||
from minisweagent.models import get_model
|
||||
|
||||
instance_id = instance["instance_id"]
|
||||
run_id = f"{model_name}_{mode_name}"
|
||||
instance_dir = output_dir / run_id / instance_id
|
||||
@@ -226,12 +168,31 @@ def process_instance(
|
||||
}
|
||||
|
||||
agent = None
|
||||
env = None
|
||||
|
||||
try:
|
||||
model = _build_model(config)
|
||||
env = _build_environment(config, instance)
|
||||
agent = _build_agent(config, model, env, instance_dir, instance_id)
|
||||
# Build model
|
||||
model = get_model(config=config.get("model", {}))
|
||||
|
||||
# Build environment
|
||||
env_config = dict(config.get("environment", {}))
|
||||
env_class_name = env_config.pop("environment_class", "docker")
|
||||
|
||||
if env_class_name == "eval.environments.gitnexus_docker.GitNexusDockerEnvironment":
|
||||
from environments.gitnexus_docker import GitNexusDockerEnvironment
|
||||
env_config["image"] = get_swebench_docker_image(instance)
|
||||
env = GitNexusDockerEnvironment(**env_config)
|
||||
else:
|
||||
from minisweagent.environments.docker import DockerEnvironment
|
||||
env = DockerEnvironment(image=get_swebench_docker_image(instance), **env_config)
|
||||
|
||||
# Build agent
|
||||
agent_config = dict(config.get("agent", {}))
|
||||
agent_class_name = agent_config.pop("agent_class", "eval.agents.gitnexus_agent.GitNexusAgent")
|
||||
|
||||
from agents.gitnexus_agent import GitNexusAgent
|
||||
traj_path = instance_dir / f"{instance_id}.traj.json"
|
||||
agent_config["output_path"] = traj_path
|
||||
agent = GitNexusAgent(model, env, **agent_config)
|
||||
|
||||
# Run
|
||||
logger.info(f"[{run_id}] Starting {instance_id}")
|
||||
@@ -243,10 +204,18 @@ def process_instance(
|
||||
result["gitnexus_metrics"] = agent.gitnexus_metrics.to_dict()
|
||||
|
||||
# Extract git diff patch from the container (SWE-bench needs the model_patch)
|
||||
result["submission"] = _extract_submission(env, info, run_id)
|
||||
try:
|
||||
patch_output = env.execute({"command": "cd /testbed && git diff"})
|
||||
result["submission"] = patch_output.get("output", "").strip()
|
||||
except Exception as patch_err:
|
||||
logger.warning(f"[{run_id}] Failed to extract patch: {patch_err}")
|
||||
result["submission"] = info.get("submission", "")
|
||||
|
||||
except Exception as e:
|
||||
_record_failure(run_id, instance_id, result, e)
|
||||
logger.error(f"[{run_id}] Error on {instance_id}: {e}")
|
||||
result["exit_status"] = type(e).__name__
|
||||
result["error"] = str(e)
|
||||
result["traceback"] = traceback.format_exc()
|
||||
|
||||
finally:
|
||||
if agent:
|
||||
@@ -318,12 +287,7 @@ def run_configuration(
|
||||
results.append(future.result())
|
||||
except Exception as e:
|
||||
iid = futures[future]
|
||||
log_safe_exception(
|
||||
logger,
|
||||
f"[{run_id}] Uncaught error for {iid}",
|
||||
e,
|
||||
include_debug=is_debug_enabled(),
|
||||
)
|
||||
logger.error(f"[{run_id}] Uncaught error for {iid}: {e}")
|
||||
|
||||
# Save run summary
|
||||
summary = {
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
"""Tests for the GitNexus eval harness."""
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user