Compare commits

..
548 Commits
Author SHA1 Message Date
Gergo Magyar 39322e37df fix(ci): build gitnexus-shared before npm ci in publish workflow
The publish job failed because npm ci triggers the prepare script (tsc)
before gitnexus-shared types are available. Add the same build step
that the setup-gitnexus composite action uses in CI.
2026-04-01 07:35:46 +01:00
Abhigyan PatwariandAbhigyan Patwari 0e6f8f1720 chore: release v1.5.0 — update CHANGELOG and package-lock (#611)
Made-with: Cursor

Co-authored-by: Abhigyan Patwari <abhigyan@Abhigyans-MacBook-Air.local>
2026-04-01 11:24:38 +05:30
Abhigyan Patwari 113dadf945 chore: bump version to 1.5.0
Made-with: Cursor
2026-04-01 10:34:46 +05:30
Abhigyan PatwariandAbhigyan Patwari af421cc9a3 feat(web): repo landing screen with selectable repo cards (#607)
* feat(web): add repo landing screen with selectable repo cards

Instead of auto-loading the first indexed repo when the backend is
detected, show a landing screen that lets users choose which repo
to explore or analyze a new one. This addresses the UX gap where
users with multiple indexed repos had no way to pick—they were
always sent to the first one found.

- New RepoLanding component with clickable repo cards (name, stats,
  indexed date) and an embedded RepoAnalyzer for new repos
- DropZone gains a 'landing' phase between server detection and
  graph loading
- Shared connectToRepo handler replaces the old handleAnalyzeComplete
  for both repo selection and post-analysis connection

Made-with: Cursor

* fix(web): update e2e flow for repo landing screen

The new landing screen intentionally stops auto-loading the first indexed
repo, so the existing Playwright tests were still waiting for the explorer
to appear automatically. Update the specs to select a repo from the landing
screen before asserting on the graph, and add a stable test id for repo cards.

Also format DropZone to satisfy the Prettier CI check.

Made-with: Cursor

* fix(e2e): use waitFor instead of instant isVisible for landing card

locator.isVisible() is a non-retrying instant check — the landing card
hadn't rendered yet when it was called, causing the click to be silently
skipped. Switch to waitFor which properly polls until the element appears.

Made-with: Cursor

---------

Co-authored-by: Abhigyan Patwari <abhigyan@Abhigyans-MacBook-Air.local>
2026-04-01 10:34:10 +05:30
Gergő Magyar c72890d59d feat(csharp): C# MethodExtractor config (#582)
* feat(csharp): add C# MethodExtractor config (#573)

Add C# method extraction config mirroring the JVM pattern from PR #576.
Wire csharpMethodConfig into the C# language provider and add 18 tests
covering classes, interfaces, abstract classes, structs, records,
constructors, params/out/ref/optional parameters, sealed methods,
attributes, and visibility modifiers.

* fix(csharp): add destructor, operator, conversion operator, and in-param support

- Add destructor_declaration, operator_declaration, and
  conversion_operator_declaration to methodNodeTypes
- Custom extractName for operators (e.g., "operator +", "implicit operator double")
- Fix extractReturnType for operator declarations (use type field, not returns)
- Add in modifier to parameter extraction (alongside out/ref)
- Add 4 new tests: destructor, operator+, implicit conversion, in parameter

* fix(csharp): add ref param test and document compound visibility limitation

- Add test for ref parameter modifier (was only testing out)
- Document that protected internal / private protected resolve to first modifier

* feat(csharp): support compound visibilities (protected internal, private protected)

- Add 'protected internal' and 'private protected' to FieldVisibility union
- Detect compound modifiers in both C# method and field extractors via
  collectModifierTexts helper scanning adjacent modifier nodes
- Add 2 tests for compound visibility detection

* feat(csharp): primary constructors, virtual/override/async, primary fields

Address all known limitations from review:

- Primary constructor support (C# 12): add extractPrimaryConstructor to
  MethodExtractionConfig and extractPrimaryFields to FieldExtractionConfig.
  Record params become public readonly properties; class params become
  private captured fields.
- Add isVirtual, isOverride, isAsync optional fields to MethodInfo,
  MethodExtractionConfig, NodeProperties, and parse-worker propagation.
- Detect virtual/override/async modifiers in C# method config.
- Move collectModifierTexts to shared helpers.ts (deduplicate).
- Fix destructor name to ~ClassName (disambiguates from constructor).
- Add expression-bodied method test.
- 118 tests total across method + field extraction suites, all passing.

* fix(csharp): review round 2 — annotations, record_struct, grammar pin

- Fix primary constructor annotations: use [] instead of extracting
  class-level attributes (C# has no syntax for ctor-specific attributes)
- Add record_struct_declaration to typeDeclarationNodes in both method
  and field extractors, CLASS_CONTAINER_TYPES, and isRecord visibility check
- Pin tree-sitter-c-sharp version (^0.23.1) in params comment

* fix(csharp): complete record_struct query + label mapping, sealed override test

- Add record_struct_declaration capture patterns to tree-sitter-queries.ts
  (type definition + primary constructor)
- Add record_struct_declaration → 'Struct' in CONTAINER_TYPE_TO_LABEL
- Assert isOverride: true alongside isFinal in sealed override test

* fix(csharp): record_struct label mismatch, add record struct + documented limitation tests

- Fix record_struct_declaration query tag: @definition.struct (not @definition.record)
  to match CONTAINER_TYPE_TO_LABEL and prevent broken HAS_METHOD edges
- Add 3 record struct tests: isTypeDeclaration, method extraction, primary constructor
- Add documented limitation tests: partial method (isAbstract: false), generic type
  parameter stripping (name excludes <T>)

* fix(csharp): remove record_struct_declaration — not a real tree-sitter node type

tree-sitter-c-sharp 0.23.1 parses 'record struct' as record_declaration
(absorbs the 'struct' keyword as an unnamed child token). The non-existent
record_struct_declaration in queries caused TSQueryErrorNodeType, breaking
ALL C# file processing.

Remove from: tree-sitter-queries.ts, typeDeclarationNodes in both
extractors, CLASS_CONTAINER_TYPES, and CONTAINER_TYPE_TO_LABEL.
Record struct types are already handled via record_declaration.

* feat(csharp): add isPartial support, filter targeted attributes, static ctor test

- Add isPartial optional field to MethodInfo, MethodExtractionConfig,
  NodeProperties, and parse-worker propagation pipeline
- Detect partial modifier in C# config — marks both declaration-only
  and implemented partial methods
- Filter targeted attribute lists (e.g. [return: MarshalAs(...)]) in
  extractCSharpAnnotations — only untargeted attributes collected
- Add static constructor test (isStatic: true, same name as class)
- Add 3 partial method tests: declaration-only, with body, coexisting pair
- Document record_struct/record_class as defensive dead code in
  export-detection.ts (grammar absorbs keywords into record_declaration)

* fix(csharp): this param for extension methods, dedup visibility, test fixes

- Handle this modifier on extension method parameters (type prefixed
  as 'this string', consistent with out/ref/in handling)
- Deduplicate visibility logic in extractPrimaryConstructor — reuse
  csharpMethodConfig.extractVisibility instead of inline compound check
- Fix record struct test title to reflect actual grammar behavior
- Add conversion operator returnType assertion
- Add extension method this parameter test

* fix(csharp): primary constructor line points to param list, empty name guard

- Use paramList.startPosition instead of ownerNode.startPosition for
  primary constructor line number (avoids methodInfoCache key collision)
- Guard against empty param names from tree-sitter error recovery nodes
2026-03-30 08:41:17 +01:00
agung widhiatmojoandClaude Opus 4.6 be210c7c61 docs: add gitnexus-shared build step before gitnexus-web (#585)
gitnexus-web imports from gitnexus-shared, which requires npm run build
to generate dist/. Without this step, npm run dev fails with module
resolution errors.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-30 08:40:47 +01:00
Subham Kundu 8d7aa94381 chore: add enterprise offering section to README, ignore local_docs/ (#579)
* chore: add enterprise offering section to README, ignore local_docs/

* chore: normalize em dash to hyphen in README enterprise section

* docs: expand enterprise section with commercial licensing and feature details
2026-03-29 14:55:17 +05:30
Subham Kundu d90d1ba96f fix(eval): exclude litellm 1.82.7 and 1.82.8 due to compatibility issues (#580) 2026-03-29 05:23:37 +01:00
Gergő Magyar 313b13fade feat(java,kotlin): MethodExtractor abstraction with per-language configs (#576) 2026-03-28 21:31:08 +00:00
Gabriel J Campbell b03413dcf9 feat: added skip-agents-md cli flag (#517)
* feat: added skip-agents-md cli flag

* fix: apply prettier formatting

* feat: added skip-agents-md cli flag

* fix: apply prettier formatting

* feat: add skipAgentsMd option to skip AGENTS.md and CLAUDE.md updates

* fixed bad merge
2026-03-28 21:23:59 +00:00
Abhigyan PatwariandClaude Opus 4.6 9f69c43100 feat(wiki): Azure OpenAI support for wiki command (#562)
* feat(wiki): extend LLMConfig/CLIConfig with Azure and reasoning model fields

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(wiki): restore cursor model resolution, fix LLMProvider type, clean up regex

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* chore(wiki): remove stale LLMProvider type alias from repo-manager

* fix(wiki): fix Azure auth header, api-version param, reasoning model params, content_filter error

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(wiki): tighten Azure detection, reasoning model regex, content_filter gating

- isReasoningModel: new regex matches only o1/o3 bare + any oN-mini/oN-preview; bare o4/o5/etc now return false
- isAzureProvider: use URL hostname matching to block spoofed subdomain URLs
- callLLM: warn on Azure legacy /deployments/ URL without api-version
- callLLM: gate content_filter error to azure===true; also catch ResponsibleAIPolicyViolation
- tests: add afterEach stub cleanup, spoofed-URL, bare-o4, non-Azure content_filter, and URL-only Azure auto-detect tests

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(wiki): detect content_filter finish_reason in SSE stream and throw clear error

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(wiki): skip delta accumulation after content_filter, use provider-neutral error message

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(wiki): add Azure OpenAI option to interactive setup wizard

Inserts Azure as option [3] in the provider menu (shifting Custom to [4]
and Cursor to [5]), adds guided Azure setup flow with resource/deployment
prompts, v1/legacy URL format selection, reasoning-model flag, and
content_filter error handling in the catch block.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(wiki): store explicit false for non-reasoning Azure deployments, trim resource name inputs

- isReasoningModelDeployment now stores false (not undefined) when user says no
- Always include isReasoningModel in saved azureConfig (no conditional guard needed)
- Trim resourceName and deploymentName prompt inputs to avoid whitespace issues
- Improve reasoning model note to mention Azure requirement

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* feat(wiki): add --api-version and --reasoning-model CLI flags for Azure

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* fix(wiki): include apiVersion and reasoningModel in hasCLIOverrides guard

* fix(wiki): default provider to 'openai' in resolveLLMConfig when not configured

* style: apply prettier formatting

* fix(wiki): address PR review — remove unrelated files, harden inputs

- Remove evidence/, fix-adapter.js, and planning doc accidentally included
- URL-encode apiVersion in buildRequestUrl to prevent query string injection
- Add --no-reasoning-model flag to allow CLI override of saved config
- Simplify verbose ternary in Azure wizard prompt

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(wiki): use execFileSync for EDITOR to prevent shell injection

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-29 00:23:41 +05:30
Gergő Magyar ddb6a704b3 refactor: reduce explicit any types (#566)
* refactor: replace NodeProperties index signature any with unknown

Change [key: string]: any to [key: string]: unknown in NodeProperties.
Remove 19 redundant (node.properties as any) casts in csv-generator.ts
— all accessed properties are already declared on the type.

* refactor: replace any with SyntaxNode across ingestion layer

Mechanical substitution — all tree-sitter AST node parameters and
variables typed as any are now properly typed as SyntaxNode.

- ast-helpers.ts: 13 any → SyntaxNode
- parsing-processor.ts: 8 any → SyntaxNode
- parse-worker.ts: 40 any → SyntaxNode/TreeSitterLanguage/Parser.Query
- php.ts: 11 any → SyntaxNode

Also adds TreeSitterLanguage type alias for optional grammar loading.

* refactor: eliminate remaining any in ingestion layer

- call-processor, call-routing, c-cpp: SyntaxNode substitutions
- parse-worker: typed WorkerIncomingMessage discriminated union
- worker-pool: typed WorkerOutgoingMessage + Error handler
- ast-cache, import-processor: targeted cast for Tree.delete()
- community-processor: graphology AbstractGraph types, LeidenModule
  interface for vendored leiden code

Ingestion layer: 130 → 7 any warnings remaining.
2026-03-28 17:11:24 +00:00
Chirag Nighut e2de9271fc feat(java): method references, worker overload disambiguation, interface dispatch (#540)
* feat(java): method references + worker overload disambiguation (TypeEnv + argTypes)

Fix two Java gaps: (1) method references (obj::method) via
  tree-sitter @call + parseJavaMethodReference wired through extractLanguageCallSiteSeed for parse-worker and call-processor; (2) overloaded calls with typed
  non-literal args by extending OverloadHints with TypeEnv for identifiers and adding ExtractedCall.argTypes from extractCallArgTypes on the worker path with
  matchCandidatesByArgTypes (inferJvmLiteralType remains for literals).

* test(csharp): expect interface-dispatch edge for IRepository.Save in heritage fixture

* refactor(ingestion): move parseJavaMethodReference to call-sites/java.ts

* refactor(ingestion): defer worker call resolution until implementor map is complete

* style: prettier + remove unused import for CI quality checks

Made-with: Cursor

* fix(ingestion): implementor map for C# base_list + sequential pipeline path

- buildImplementorMap: treat extends rows as implements when resolveExtendsType
  says IMPLEMENTS (worker heritage mirrors parse-worker, all base_list as extends)
- Worker path: pass ctx into buildImplementorMap(deferredWorkerHeritage, ctx)
- Sequential path: extract heritage before processCalls and pass implementor map
  so small repos get interface-dispatch CALLS (fixes csharp-proj integration test)

Made-with: Cursor

* perf(pipeline): accumulate sequential implementor map without O(E) per chunk

- Merge buildImplementorMap(chunk heritage) into one map each sequential chunk so
  work is O(heritage) per chunk and interface dispatch sees prior chunks (worker parity)
- Drop unused globalImplementorMap + redundant merge after worker pass

Made-with: Cursor
2026-03-28 16:59:28 +00:00
Gergő Magyar acf6fbdd39 feat: configure eslint with unused import removal (#564)
* feat: configure eslint with unused import removal

Add ESLint v9 (flat config) for code quality:
- eslint-plugin-unused-imports for auto-removing dead imports
- @typescript-eslint for TypeScript-aware linting
- eslint-plugin-react-hooks for React hooks rules
- eslint-config-prettier to avoid formatting conflicts
- lint-staged runs eslint --fix before prettier on .ts/.tsx
- CI lint job added to ci-quality.yml

* refactor: remove unused imports via eslint --fix

Auto-fixed by eslint-plugin-unused-imports. No logic changes.

* chore: add eslint fix commit to .git-blame-ignore-revs
2026-03-28 15:28:09 +00:00
Gergő Magyar bf09eab95b feat: configure prettier with pre-commit hook (#563)
* feat: configure prettier with pre-commit hook integration

Add prettier, lint-staged, and prettier-plugin-tailwindcss at the repo
root with husky pre-commit hook integration. Moves husky from
gitnexus/ to root package.json for reliable hook installation.

- Root package.json with prepare/format/format:check scripts
- .prettierrc with endOfLine:lf and tailwindStylesheet for TW v4
- .prettierignore excluding fixtures, vendor, generated, *.d.ts, *.md
- .gitattributes enforcing LF line endings for Windows consistency
- Pre-commit hook uses direct node_modules/.bin/ paths (no npx)

* style: apply prettier formatting to entire codebase

One-time bulk format. No logic changes.
Use .git-blame-ignore-revs to skip this commit in git blame.

* chore: add .git-blame-ignore-revs for prettier format commit

* perf: pre-commit hook runs only tests related to staged files

Use vitest --related to scope test execution to tests that import
the changed files, instead of running the full suite on every commit.

* perf: remove vitest from pre-commit hook, keep in CI only

Pre-commit now runs lint-staged + tsc only. Tests run in CI
(ci-tests.yml) where they belong — keeps commits fast.

* ci: add prettier format check to quality workflow

PRs will now fail if code isn't formatted with prettier.
2026-03-28 14:58:04 +00:00
Gergő Magyar fd7fb5bf1f feat: unify web and cli ingestion pipeline (#536)
* feat: add server-side ingestion API (POST /api/analyze, SSE progress)

Extract core analysis orchestration from CLI into shared run-analyze.ts
module. Add server-side analyze endpoints so the web app can trigger
ingestion via HTTP instead of running the full pipeline in-browser.

New files:
- src/core/run-analyze.ts — shared runFullAnalysis() orchestrator
- src/server/analyze-job.ts — job manager (single-slot, dedup, SSE events)
- src/server/analyze-worker.ts — forked child process (8GB heap, IPC)
- src/server/git-clone.ts — shallow clone/pull with SSRF protection

API endpoints:
- POST /api/analyze — start analysis (returns 202 + jobId)
- GET /api/analyze/:jobId — poll job status
- GET /api/analyze/:jobId/progress — SSE progress stream

Security: URL validation blocks private IPs and non-HTTP schemes.
Path validation requires absolute paths. Git stderr not leaked to API.

* feat(web): add server-side analyze UI (Phase 2)

Add "Analyze on Server" flow to the web app's Server tab so users
can trigger server-side ingestion from the browser. On completion,
the graph is automatically loaded via the existing connectToServer flow.

New files:
- AnalyzeProgress.tsx — progress bar with phase label, elapsed time, cancel

Modified files:
- backend.ts — startAnalyze(), streamAnalyzeProgress() SSE client
- DropZone.tsx — analyze URL input + button below Connect section
- App.tsx — onServerAnalyze handler wires analyze -> connect flow

* feat: add job cancellation, timeout, and child process tracking (Phase 3)

- DELETE /api/analyze/:jobId — cancel running analysis (SIGTERM to worker)
- 30-minute timeout kills long-running workers automatically
- Child process refs tracked in JobManager for cleanup on shutdown
- dispose() kills all active children on SIGINT/SIGTERM
- Web cancel button now calls server DELETE endpoint
- cancelAnalyze() added to web backend client

* refactor(web): remove browser ingestion pipeline (Phase 4)

Delete 16 duplicated ingestion files, 2 unused service files
(git-clone, zip), and tree-sitter parser-loader from gitnexus-web.
All ingestion now runs server-side via POST /api/analyze.

Deleted (18 files, ~5,000 lines):
- core/ingestion/*.ts (16 pipeline processors)
- core/tree-sitter/parser-loader.ts (WASM tree-sitter loader)
- services/git-clone.ts (isomorphic-git client-side clone)
- services/zip.ts (JSZip extraction)

Simplified:
- DropZone.tsx — server-only (removed ZIP/GitHub tabs)
- ingestion.worker.ts — removed runPipeline/runPipelineFromFiles
- useAppState.tsx — removed pipeline callbacks
- App.tsx — removed handleFileSelect/handleGitClone
- main.tsx — removed Buffer polyfill for isomorphic-git
- types/pipeline.ts — removed PipelineResult/serialize helpers

Kept: cluster-enricher.ts (LLM enrichment, still used by worker)

Dependencies now removable: web-tree-sitter, isomorphic-git,
@isomorphic-git/lightning-fs, jszip (estimated 3-4MB bundle savings)

* refactor(web): sync graph schema from CLI + delete WASM grammars

Sync graph/types.ts and lbug/schema.ts from the CLI (source of truth)
to the web module so the browser LadybugDB can handle all node and
relationship types the server pipeline produces.

Synced types: Route, Tool, Section node labels; HANDLES_ROUTE, FETCHES,
HANDLES_TOOL, ENTRY_POINT_OF, WRAPS, QUERIES relationship types;
description fields on Function/Class/Interface/Method/CodeElement.

Deleted: public/wasm/ directory (14 tree-sitter WASM grammars + core).
Removed deps: web-tree-sitter, isomorphic-git, @isomorphic-git/lightning-fs,
jszip, buffer, @types/jszip (~3-4MB bundle savings).

* feat: create gitnexus-shared package for unified type definitions

Create a new gitnexus-shared package that is the single source of truth
for types shared between the CLI and web modules:

- SupportedLanguages enum (15 languages)
- Graph types: NodeLabel, NodeProperties, RelationshipType, GraphNode, GraphRelationship
- Schema constants: NODE_TABLES, REL_TYPES, REL_TABLE_NAME, EMBEDDING_TABLE_NAME
- Pipeline types: PipelinePhase, PipelineProgress

Both gitnexus (CLI) and gitnexus-web import from gitnexus-shared via
file: dependency. Each package re-exports and extends with platform-specific
additions (CLI: KnowledgeGraph with mutation methods; Web: simpler KnowledgeGraph).

This ensures types can never drift between packages — adding a new
language, node type, or relationship type in gitnexus-shared automatically
propagates to both consumers.

* refactor: import shared types directly from gitnexus-shared at call sites

Replace all re-export patterns with direct imports from gitnexus-shared.
72 files updated across CLI and web:

- SupportedLanguages: 49 CLI files now import from 'gitnexus-shared'
  instead of '../config/supported-languages.js'
- GraphNode, GraphRelationship, NodeLabel: 22 CLI + 10 web files now
  import from 'gitnexus-shared' instead of local re-export wrappers
- NODE_TABLES: api.ts imports from 'gitnexus-shared'
- PipelineProgress: useAppState.tsx imports from 'gitnexus-shared'

Local types.ts files now only define platform-specific KnowledgeGraph
(CLI has mutation methods, web has add-only). No more re-exports.

* fix: update lock files for gitnexus-shared, remove stale vite polyfills

Add gitnexus-shared@1.0.0 to lock files so npm ci succeeds in CI.
Remove buffer polyfill and global define from vite.config.ts (isomorphic-git was removed).

* fix(security): add write guard to HTTP /api/query, fix CORS proxy bypass

- Add isWriteQuery() check to POST /api/query handler — blocks CREATE,
  DELETE, SET, MERGE, DROP, etc. via HTTP API (guard was only in MCP
  pool adapter and browser-side, not the HTTP server path)
- Extend CYPHER_WRITE_RE with CALL, INSTALL, LOAD keywords
- Fix CORS proxy subdomain bypass: endsWith('github.com') allowed
  'evil-github.com'. Now requires exact match or '.github.com' suffix

* feat(server): enhance /api/search with enrichment, add /api/grep, strip graph content

- POST /api/search: add mode param (hybrid|semantic|bm25), server-side
  enrichment returns connections/cluster/processes per result in one call
  (collapses 31 sequential HTTP calls to 1 for the agent search tool)
- GET /api/grep: regex search across indexed file contents, eliminates
  need to transfer all file contents to browser
- GET /api/graph: strip content field by default (80-95% payload
  reduction). Use ?includeContent=true for backward compat
- Add LRU cache invalidation hook point for future caching

* feat(server): add /api/embed endpoint for server-side embedding generation

- POST /api/embed: triggers embedding pipeline via onnxruntime-node
  with JobManager for single-slot concurrency, timeout, and dedup
- GET /api/embed/:jobId: poll job status
- GET /api/embed/:jobId/progress: SSE stream with heartbeat, event IDs,
  and X-Accel-Buffering:no header for proxy compatibility
- DELETE /api/embed/:jobId: cancel running embedding job
- Maps embedding pipeline phases (ready→complete, error→failed) to
  JobManager status conventions

* feat(web): create consolidated BackendClient module

Single HTTP client replacing backend.ts, server-connection.ts, and
worker HTTP helpers. Includes:
- Typed methods: runQuery, search (enriched), grep, readFile, connect
- Generic streamSSE<T> utility extracted from analyze progress pattern
- BackendError with discriminated code field (network/server/client/timeout)
- Embed API: startEmbeddings, streamEmbeddingProgress, cancelEmbeddings
- Search with mode param (hybrid|semantic|bm25) and enrichment

* refactor(web): rewrite Graph RAG tools for backend-only HTTP queries

- Search tool: uses enriched /api/search (1 call replaces 31 sequential queries)
- Cypher tool: removes browser-side embedding; {{QUERY_VECTOR}} routes to
  /api/search with mode:'semantic' instead of local transformers.js
- Grep tool: uses /api/grep instead of in-memory fileContents map
- Read tool: uses /api/file instead of fileContents map lookup
- Impact tool: getCallSiteSnippet now async via /api/file
- createGraphRAGTools now accepts GraphRAGBackend interface instead of
  7 separate function params + fileContents map
- createGraphRAGAgent simplified to (config, backend, context?)
- Removed imports: embedder, lbug/schema (replaced with gitnexus-shared)
- Net: -205 lines

* refactor(web): delete WASM infrastructure, remove 7 packages (-5242 lines)

Delete browser-side LadybugDB, embeddings, search, and worker:
- gitnexus-web/src/core/lbug/ (adapter, csv-generator, schema, query-result)
- gitnexus-web/src/core/embeddings/ (embedder, pipeline, text-gen, types)
- gitnexus-web/src/core/search/ (bm25-index, hybrid-search)
- gitnexus-web/src/workers/ingestion.worker.ts (828 lines)
- gitnexus-web/src/services/server-connection.ts (merged into backend-client)
- gitnexus-web/src/types/lbug-wasm.d.ts

Remove packages: @ladybugdb/wasm-core, @huggingface/transformers,
comlink, minisearch, vite-plugin-wasm, vite-plugin-top-level-await,
vite-plugin-static-copy

Update vite.config.ts: remove WASM plugins, COOP/COEP headers,
worker config, optimizeDeps exclude

Update imports: App.tsx, DropZone, Header, AnalyzeProgress,
BackendRepoSelector, useBackend → backend-client

* refactor(web): replace Worker/Comlink with direct BackendClient calls

- useAppState: remove Worker instantiation, Comlink.wrap, apiRef.
  All queries now go through BackendClient HTTP functions directly.
- Agent runs on main thread (I/O-bound LLM streaming, not CPU-bound)
- initializeAgent: creates GraphRAGAgent with GraphRAGBackend interface
  bound to BackendClient methods (runQuery, search, grep, readFile)
- startEmbeddings: calls POST /api/embed + SSE progress instead of
  running browser-side transformers.js pipeline
- switchRepo: no longer loads graph into WASM DB or extracts fileContents
- App.tsx: handleServerConnect simplified (no fileContents, no loadServerGraph)
- Delete old backend.ts (replaced by backend-client.ts)
- Net: -396 lines

* fix(web): fix await-in-map build error in agent streaming

Move dynamic import of AIMessage outside .map() callback to avoid
"await can only be used inside an async function" build error.

* fix(web): remove stale apiRef references that broke chat functionality

sendChatMessage referenced apiRef.current (deleted Worker ref) which
would throw TypeError. Replaced with agentRef.current guard since agent
now runs on main thread.

* fix(server): dispose embedJobManager on shutdown, fix job mutation

- Add embedJobManager.dispose() to shutdown handler (was missing,
  causing cleanup timer to keep Node process alive)
- Replace direct job.repoName/status mutation with updateJob() to
  ensure SSE event emission for initial status change

* fix(server): parameterize Cypher, harden grep, unify SSE endpoints

- Search enrichment: replace string interpolation with executePrepared()
  using $nid parameter binding to prevent Cypher injection
- Add executePrepared() to core lbug-adapter (prepare/execute pattern)
- /api/grep: add 200-char pattern length limit (ReDoS protection),
  search files on disk instead of loading entire corpus into memory
  (constant memory usage regardless of repo size)
- Extract mountSSEProgress() shared helper for SSE streaming — both
  analyze and embed endpoints now have consistent heartbeat (30s),
  event IDs (reconnection support), and X-Accel-Buffering header

* refactor(web): remove dead code from Worker-era architecture

- Remove loadServerGraph no-op function, interface member, and all consumers
- Remove testArrayParams stub and interface member
- Remove fileContents state from GraphStateProvider (never populated in
  server-side architecture)
- Remove forceDevice parameter from startEmbeddings (server-side, no device choice)
- Replace phantom EmbeddingProgress type with inline { phase, percent }
- Replace resolvePathFromContents (needed fileContents Map) with graph-based
  file path resolution using filePathIndex built from graph nodes
- Fix: AI citation grounding ([[file.ts:10]]) now works via graph node lookup
  instead of broken fileContents-based resolution

* fix(web): use streamAgentResponse for full tool_call/reasoning streaming

Replace naive agent.stream() loop that only handled content chunks with
streamAgentResponse() generator from agent.ts. This properly routes:
- reasoning tokens (before/between tool calls)
- tool_call events (name, args, status)
- tool_result events (completed tool output)
- content tokens (final answer after all tools done)

Previously the onChunk handler for tool_call/tool_result/reasoning was
dead code since the streaming loop only emitted content events.

* fix(web): resolve CI type errors from dead code removal

- Import GraphNode/GraphRelationship from gitnexus-shared in graph.ts
  (not re-exported from local types.ts)
- Add Route, Tool entries to NODE_COLORS and NODE_SIZES constants
- Add PipelineResult type to web types/pipeline.ts
- Remove fileContents from CodeReferencesPanel and RightPanel
- Remove testArrayParams and forceDevice from EmbeddingStatus
- Remove forceDevice args from startEmbeddings() calls in App.tsx
- Fix embeddingProgress property accesses for simplified type

* fix(ci): add setup-gitnexus-web action, build shared once per job

- Remove prepare script from gitnexus-shared (tsc not available during
  npm ci of consuming packages)
- Create .github/actions/setup-gitnexus-web composite action: builds
  gitnexus-shared then runs npm ci for gitnexus-web
- setup-gitnexus action: already builds gitnexus-shared for CLI jobs
- ci-quality typecheck-web: uses setup-gitnexus-web (DRY)
- ci-e2e: uses setup-gitnexus-web (DRY)
- ci-tests: gitnexus-shared already built by setup-gitnexus, just
  install web deps without rebuilding

* fix(ci): use prepare script so gitnexus-shared builds during npm ci

Move typescript from devDependencies to dependencies in gitnexus-shared
so the prepare script (tsc) works when npm resolves file: deps during
npm ci. No GHA modifications needed — npm handles the build lifecycle
automatically.

Remove manual gitnexus-shared build steps from setup-gitnexus and
setup-gitnexus-web actions.

* fix(ci): build gitnexus-shared explicitly in setup actions

The file: dependency protocol doesn't reliably run prepare scripts
because devDependencies aren't installed first. Instead of fragile
lifecycle hacks, build gitnexus-shared explicitly in both setup actions:
- setup-gitnexus: npm install && npm run build in gitnexus-shared/
- setup-gitnexus-web: same, before npm ci in gitnexus-web/
- ci-tests: shared already built by setup-gitnexus, web just npm ci

No prepare script, no dist in git, no typescript as a prod dependency.

* fix: remove CALL from CYPHER_WRITE_RE — breaks FTS and vector search

CALL is used by read-only procedures: CALL QUERY_FTS_INDEX(...) and
CALL QUERY_VECTOR_INDEX(...). Adding it to the write guard blocked all
FTS search, causing 3 test failures. The database is opened in read-only
mode as defense-in-depth against write procedures via CALL.

Keep INSTALL and LOAD in the blocklist (genuinely dangerous).

* fix(web): update vercel.json for gitnexus-shared, remove COOP/COEP

- Add installCommand that builds gitnexus-shared before installing
  web deps (Vercel doesn't know about the monorepo file: dependency)
- Remove Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy
  headers (no longer needed — WASM LadybugDB removed)

* fix(web): update tests for deleted modules

- Delete csv-generator.test.ts (tests deleted WASM-only csv-generator)
- Update security-guards.test.ts: import NODE_TABLES/REL_TYPES from
  gitnexus-shared instead of deleted src/core/lbug/schema
- Update server-connection.test.ts: import normalizeServerUrl from
  backend-client, remove extractFileContents tests (function deleted)

* fix(e2e): remove Server tab click — UI is now server-only

The DropZone no longer has ZIP/GitHub/Server tabs (browser ingestion
was removed). The server URL input is directly visible on the landing
page. Update e2e test to skip the tab click and go straight to input.

All 5 e2e tests pass locally.

* refactor: use gitnexus-shared for PipelinePhase/PipelineProgress types

CLI was duplicating PipelinePhase and PipelineProgress locally instead
of importing from gitnexus-shared. Updated all consumers to import
directly. Also removed dead code: SerializablePipelineResult,
serializePipelineResult(), deserializePipelineResult().

* fix(server): address PR #536 review — security, race conditions, dead code

- Fix path traversal in POST /api/analyze: split into isAbsolute + normalize check
- Add shared repo lock (activeRepoPaths) preventing concurrent analyze+embed on same repo
- Fix 202 response returning actual job.status instead of hardcoded 'queued'
- Add 30-minute timeout for embedding jobs (was missing unlike analyze jobs)
- Fix DropZone calling startAnalyze without setting backend URL first
- Add SSE reconnect with exponential backoff (3 retries) and Last-Event-ID
- Fix normalizeServerUrl to return base URL (no /api suffix) — clear contract
- Delete dead code: proxy.ts, server-graph-hydration.ts, pipeline.ts re-export barrel
- Update LoadingOverlay to import PipelineProgress directly from gitnexus-shared

* fix(server): fix repo lock key mismatch and embed cancel race

- Use getStoragePath(targetPath) as lock key in analyze handler to match
  embed handler's entry.storagePath — keys now always align
- Guard embed completion: don't overwrite 'failed' with 'complete' when
  job was cancelled while pipeline was still running
- Remove unused jobType parameter from acquireRepoLock
- Log backend.init() errors instead of silently swallowing

* fix: add gitnexus-shared as a local dependency in package-lock.json

* refactor: move language detection to gitnexus-shared, add syntax highlighting for all 15 languages

Move getLanguageFromFilename() from CLI to gitnexus-shared with COBOL
support added. Add getSyntaxLanguageFromFilename() for Prism-compatible
syntax highlighting covering all 15 code languages plus auxiliary
formats (json, yaml, markdown, html, css, bash, sql, xml).

Refactor CodeReferencesPanel to use shared function instead of a local
30-line switch. Delete dead gitnexus-web/src/config/supported-languages.ts
(web already imports SupportedLanguages from gitnexus-shared).

* feat(web): add first-time user onboarding with auto server detection

Replace the manual "Connect to Server" panel with an automatic onboarding
flow that guides first-time users through starting the GitNexus server.

Server detection:
- useBackend hook polls via setTimeout chain (3s, no overlap)
- Page Visibility API pauses polling when tab is hidden
- SSE heartbeat (/api/heartbeat) for instant disconnect detection

Onboarding UI (OnboardingGuide.tsx):
- Step-by-step flow: copy command → run → auto-connect
- Smart command: shows `gitnexus serve` in dev, `npx gitnexus@latest serve` in prod
- Node.js version auto-detected from package.json via Vite define
- Faux terminal windows with copy-to-clipboard, platform tabs, polling indicator

Transitions (DropZone.tsx):
- Crossfade wrapper with snapshot pattern for smooth phase transitions
- Three phases: onboarding → success (1.2s hold) → loading → graph
- Auto-recovery: falls back to onboarding if server dies or connect fails

Server changes:
- GET /api/heartbeat: SSE endpoint for liveness detection
- GET /api/info: version, launch context, Node.js version
- npm run serve script for local development
- app.disable('x-powered-by') hardening

* feat(web): add repo analysis UI, SSE heartbeat, and review fixes

Repo analysis:
- AnalyzeOnboarding: empty-state card when server has zero repos
- RepoAnalyzer: GitHub URL + Local Folder tabs with browse button
- Header repo dropdown: click project badge to switch repos or analyze new
- DropZone 'analyze' phase integrated into Crossfade transitions

Reliability fixes from 5-agent review:
- Polling: stop scheduling timers when tab hidden, restart on visibility return
- Heartbeat: exponential backoff (1s/2s/4s, 3 retries) prevents graph loss on blip
- RepoAnalyzer: completion timer tracked in ref, cleaned up on unmount
- DropZone: standardized card padding (p-7), heading sizes (text-lg)

Accessibility:
- prefers-reduced-motion global CSS rule (WCAG 2.3.3)
- focus-visible rings on CopyButton
- cursor-pointer on all Header buttons
- Consistent rounded-xl on all dropdowns

Cleanup:
- Deleted dead AnalyzeSheet.tsx (219 LOC) and BackendRepoSelector.tsx (89 LOC)
- Fixed AnalyzeProgress lucide import (lucide-react → @/lib/lucide-icons)

* fix(server): resolve analyze worker fork crash in dev mode

The forked analyze worker was crashing immediately with exit code 1
when running via `npm run serve` (tsx). Two issues:

1. Worker path resolved to `analyze-worker.js` but only `.ts` exists
   in the source directory — the `.js` file is only in `dist/`.

2. On Windows, bare `--import tsx` in execArgv fails because Node's
   ESM resolver for --import uses the child's CWD, not the parent's
   node_modules. Windows also rejects raw paths as `d:` is not a
   valid URL scheme.

Fix: detect dev vs prod via `import.meta.url` extension. In dev mode,
resolve `tsx/esm` to an absolute `file://` URL via `pathToFileURL()`
anchored to the parent's `createRequire` context. This works on all
platforms and doesn't depend on the child's CWD or PATH.

Also captures child stderr for better crash diagnostics.

Verified: `POST /api/analyze` with GitHub URL completes successfully
in dev mode (tsx) — status goes from cloning → analyzing → complete.

* fix(server): add worker auto-retry, error handling, and crash diagnostics

Worker resilience:
- Auto-retry up to 2 times with exponential backoff (1s, 2s) on crash
- SSE progress shows "Retrying after crash (1/2)..." during retry
- Captures child stderr for crash diagnostics in failure message
- AnalyzeJob tracks retryCount per job

Server error handling:
- app.listen wrapped in Promise so EADDRINUSE/EACCES propagate cleanly
- serve.ts catches startup errors with friendly messages and exit code 1
- EADDRINUSE gets actionable guidance (stop other process or --port flag)
- Global uncaughtException/unhandledRejection handlers prevent silent exits
- DEBUG=1 env var shows full stack traces

* feat: add e2e tests for onboarding flows, worker retry, and error handling

E2E tests (onboarding.spec.ts — 11 tests):
- Flow 1: OnboardingGuide shown when server unreachable (6 tests)
- Flow 2: Auto-connect with success card, analyze phase for zero repos
- Flow 3: Analyze form — GitHub URL validation, Local Folder tab, tab switching
- Flow 4: Repo dropdown in exploring view (skipped without live server)

Updated server-connect.spec.ts:
- Replaced manual Connect button flow with auto-connect waitForGraphLoaded

Server resilience:
- Worker auto-retry (2 attempts with exponential backoff) on crash
- Friendly error messages for serve startup failures (EADDRINUSE etc.)
- Global uncaughtException/unhandledRejection handlers prevent silent exits
- app.listen wrapped in Promise for proper error propagation

* refactor(shared): enforce exhaustive language coverage via Record types

Replace the if/else chain in getLanguageFromFilename with two exhaustive
Record<SupportedLanguages, ...> maps:

- EXTENSION_MAP: every language → its file extensions
- SYNTAX_MAP: every language → its Prism syntax identifier

Adding a new member to the SupportedLanguages enum without adding it to
both maps now produces a TypeScript compile error:

  Property '[SupportedLanguages.NewLang]' is missing in type...

This matches the existing pattern in languages/index.ts (providers table)
which already uses `satisfies Record<SupportedLanguages, LanguageProvider>`.

Three compile-time enforcement points now exist:
1. EXTENSION_MAP in language-detection.ts (file extensions)
2. SYNTAX_MAP in language-detection.ts (Prism syntax identifiers)
3. providers in languages/index.ts (LanguageProvider instances)

* feat(web): load source code from server and scroll to selected line

CodeReferencesPanel now fetches file content via GET /api/file when a
node is selected, instead of showing "Code not available in memory".

- Fetches via readFile() from backend-client when selectedFilePath changes
- Shows loading spinner while fetching
- After content loads, auto-scrolls to the selected node's startLine
- Highlights the selected line range with a cyan left border
- Cancels in-flight fetch if selection changes before it completes

Also: refactored language-detection.ts to use exhaustive Record types
(EXTENSION_MAP and SYNTAX_MAP) so adding a new SupportedLanguages enum
member without implementing extensions/syntax is a compile error.

* feat: buffered file reading for Code Inspector

Server: GET /api/file now supports ?startLine=N&endLine=M for reading
a line range instead of the entire file. Returns { content, startLine,
endLine, totalLines }.

Client: readFile() returns ReadFileResult with metadata. When selecting
a symbol (function, class, method), fetches only ±50 lines around the
symbol's startLine/endLine instead of the full file. File nodes still
fetch the entire file.

SyntaxHighlighter startingLineNumber set from the buffer offset so line
numbers are correct even for partial reads.

* fix: adapt readFile callers to new ReadFileResult return type

tools.ts: readFile comes from GraphRAGBackend interface which returns
Promise<string> (the adapter in useAppState extracts .content), so
revert the { content } destructuring back to plain string assignment.

useAppState.tsx: wrap backendReadFile with { repo } options object
and extract .content to satisfy the GraphRAGBackend interface.

* fix(web): ensure new repos appear in list immediately after analysis

Two fixes:

1. DropZone: handleAnalyzeComplete now passes the repoName through to
   connectToServer so the specific newly-analyzed repo loads — not the
   server's default first repo.

2. App.tsx: fetchRepos() is now awaited BEFORE handleServerConnect in
   both the DropZone and Header flows. This ensures the repo list is
   populated before the exploring view renders, so the new repo appears
   in the header dropdown immediately without a page reload.

* feat: delete repos, re-analyze with force, select after analysis

Server — DELETE /api/repo:
- Acquires repo lock first (409 if analyze/embed in flight)
- Closes LadybugDB, deletes index + clone dir, unregisters, re-inits
- Lock released in finally block

Server — analyze complete:
- backend.init() must succeed before SSE complete fires
- If backend.init() fails, job is marked failed (not complete)

Web — Header repo dropdown:
- Re-analyze: calls POST /api/analyze with force=true, shows spinning
  icon + inline progress bar via SSE
- Delete: acquires lock, aborts any running re-analysis SSE for same
  repo, refreshes list, switches to next repo
- After analysis completes: refreshes repo list, connects to the
  specific repo by name, loads graph, shows in explorer
- Retry with 1.5s backoff on 404 (server may still be reinitializing)

Type safety:
- err: any → err: unknown + instanceof BackendError in retry loop
- Added missing BackendRepo + BackendError imports in App.tsx
2026-03-28 14:07:11 +00:00
Juan Pablo Maddoni c27c9f612c fix/opencode mcp gitnexus timeout (#363) 2026-03-27 12:18:01 -05:00
Gergo Magyar 9edf06bda2 chore: bump version to 1.4.10, update CHANGELOG 2026-03-27 13:52:59 +00:00
Copilot 01ddc3e756 fix: resolve tree-sitter peer dependency conflicts (#538)
Downgrade tree-sitter from ^0.25.0 to ^0.21.1 and align all parser versions to eliminate ERESOLVE peer dependency conflicts that break MCP server install via npx. Also corrects hallucinated tree-sitter-dart SHA. Fixes #537
2026-03-27 13:49:28 +00:00
Gergo Magyar f373a09ebc chore: bump version to 1.4.9, add CHANGELOG.md 2026-03-26 16:40:02 +00:00
Oleg DeezusandGergo Magyar cde3858e03 refactor: Phase 8 & 9 — Field Types and Return-Type Binding (#494)
* feat(phase8): add field type data structures and extractor interface

* feat(phase8): implement TypeScript field extractor

* feat-phase9-add-call-result-binding

* test-phase8-add-field-extraction-unit-tests

* docs: update documentation for Phase 8 and Phase 9

* feat(swift): Phase 8/9 integration tests for field-type and call-result binding

Add Swift field-type resolution and call-result binding integration tests
with fixtures, plus merge-conflict fixes for the FieldExtractor code.

**Swift integration tests:**
- `swift-field-types/` fixture (Models.swift + App.swift) — tests
  HAS_PROPERTY edges, field-chain CALLS resolution (user.address.save()
  → Address#save), and ACCESSES edges for field reads.
- `swift-call-result-binding/` fixture — tests call-result binding
  (let user = getUser(); user.save() → User#save).
- 2 new describe blocks in swift.test.ts with skipIf(!swiftAvailable).

**Swift arity fix:**
- extractMethodSignature fallback counts direct `parameter` children
  when no wrapper list node exists (Swift's tree-sitter grammar places
  parameters as direct children of function_declaration).  Without this,
  all Swift functions had parameterCount: 0 and the arity filter rejected
  valid call targets.

**FieldExtractor merge-conflict fixes:**
- field-extractor.ts: update import from removed ./utils.js to
  ./utils/ast-helpers.js; use typeEnv.fileScope() instead of .get('').
- field-extractors/typescript.ts: same import fix.
- field-types.ts: alias TypeEnvironment as TypeEnv (renamed on main).
- field-extraction.test.ts: mock TypeEnvironment interface properly.

* feat(field-extractors): generic table-driven field extractors for all 14 languages, wired into pipeline

Implements field extractors for all supported languages and integrates
them into the ingestion pipeline as the single source of truth for
Property node metadata.

**Generic field extractor factory** — `field-extractors/generic.ts`
defines a `createFieldExtractor(config)` factory that generates
FieldExtractor instances from a per-language `FieldExtractionConfig`.
Each config specifies AST node types, name/type/visibility extraction
functions, and static/readonly detection — typically 20-40 lines per
language vs 300+ for a hand-written extractor.

**Per-language configs** — `field-extractors/configs/` has 11 config
files covering 13 languages (TS/JS share, Java/Kotlin share).
TypeScript keeps its hand-written extractor for richer handling.

**LanguageProvider integration** — New optional `fieldExtractor` property
on LanguageProviderConfig, set via defineLanguage() in each language
file. Follows the same strategy pattern as typeConfig, exportChecker,
and labelOverride.  Removed the separate FieldExtractorRegistry class
and field-extractors/index.ts — extractors are accessed via
getProvider(lang).fieldExtractor.

**Pipeline wiring** — Both parse-worker.ts (worker pool) and
parsing-processor.ts (sequential fallback) now call the FieldExtractor
during Property node creation. Results are cached per class node.
Property nodes are enriched with: declaredType, visibility, isStatic,
isReadonly.

**extractPropertyDeclaredType removed** — The 100-line multi-strategy
function in type-extractors/shared.ts is replaced by the FieldExtractor.
All 14 languages register an extractor, eliminating the need for a
generic fallback.  The Python config's extractType was fixed to handle
annotation-without-value patterns (address: Address).

**Integration tests** — Each language's resolver test file gains
pipeline-based assertions verifying visibility/isStatic/isReadonly on
Property nodes via getNodesByLabelFull.  Tests run through
runPipelineFromRepo with real fixtures — no direct extractor calls.

* fix(type-env): thread enclosingFunctionFinder through scope resolution, unskip Dart ACCESSES test

The type-env's findEnclosingScopeKey had the same Dart sibling problem
as findEnclosingFunction — it walked parents but never found
function_signature because the call lives inside function_body (a
sibling).  Instead of hardcoding a function_body check, thread the
provider's enclosingFunctionFinder hook through BuildTypeEnvOptions →
lookupInEnv → findEnclosingScopeKey.  All three buildTypeEnv call sites
(call-processor, parsing-processor, parse-worker) now pass the hook.

This enables the type-env to resolve scoped parameter bindings for Dart
(e.g., `user: User` in processUser), which lets the chain-resolution
tier (Step 1c) walk `user.address` and emit ACCESSES edges.

Dart integration test unskipped — 10/10 passing including ACCESSES.
Reverted CHANGELOG.md to origin/main.

* fix: resolve all PR #494 review findings (10 items)

CRITICAL:
- parse-worker.ts: classNode: any → SyntaxNode on getFieldInfo
  and findEnclosingClassNode; removed redundant as number casts
- parsing-processor.ts: classNode: any → SyntaxNode on seqGetFieldInfo

HIGH:
- ruby.ts: attr_accessor now extracts ALL symbol arguments via
  extractNames hook in generic factory (was firstNamedChild only)
- typescript.ts: added JSDoc explaining why hand-written extractor
  coexists with config-based typescript-javascript.ts

MEDIUM:
- field-types.ts: FieldVisibility union type replaces string
  ('public'|'private'|'protected'|'internal'|'package'|'fileprivate'|'open')
  Propagated through field-extractor.ts, generic.ts, all 7 config files
- typescript.ts: extractFullType collapsed from 12 branches to 3 lines
- generic.ts: added extractNames? optional hook + buildField refactor

LOW:
- ruby.ts: extractVisibility(node) → extractVisibility(_node)
- python.ts: fixed misleading isStatic comment

TypeScript compiles cleanly.

* test: add 24 field extraction tests for generic factory + 5 languages

Generic factory (4 tests):
- createFieldExtractor with TypeScript config validates factory itself
- Body discovery for interfaces, static/readonly modifiers
- Non-type node rejection

Python (4 tests):
- Annotated class field extraction
- Underscore-based visibility: _name=protected, __name=private

Go (5 tests):
- isTypeDeclaration on type_declaration nodes
- Config functions: uppercase=public, lowercase=package visibility
- extractType, isStatic, isReadonly

C++ (5 tests):
- public/private/protected access specifier backward-sibling walk
- Default visibility: class=private, struct=public
- static/const modifier detection

Ruby (6 tests):
- attr_accessor multi-symbol: :name, :email, :age → 3 fields
- attr_reader=readonly, attr_writer=non-readonly
- Multiple attr_* calls in one class

Total: 46 tests passing

* chore: remove plan doc from PR

---------

Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-26 16:37:34 +00:00
Gergő Magyar d2cd0b676f feat: add COBOL language support with regex extraction pipeline (#498)
* feat: add COBOL language support with regex extraction pipeline

Standalone COBOL processor following the markdown-processor.ts pattern:
- No LanguageProvider modification — COBOL uses regex, not tree-sitter
- No SupportedLanguages enum change — standalone processor pattern

New files:
- cobol-processor.ts — orchestrator (processCobol, isCobolFile, isJclFile)
- cobol/cobol-preprocessor.ts — regex state machine extraction (~888 LOC)
- cobol/cobol-copy-expander.ts — COPY statement expansion with circular detection
- cobol/jcl-parser.ts — JCL job/step/DD extraction
- cobol/jcl-processor.ts — JCL graph node creation

Extraction produces:
- Module nodes (PROGRAM-ID)
- Function nodes (paragraphs)
- Namespace nodes (sections)
- Property nodes (data items)
- CALLS edges (PERFORM intra-file, CALL cross-program)
- IMPORTS edges (COPY statements)
- CONTAINS edges (section → paragraph hierarchy)

Pipeline integration: single processCobol() call in Phase 2.6

54 new tests (33 COBOL + 21 JCL), all 3889 tests pass.

* docs: document custom processor pattern in pipeline.ts

Add comment block at the custom processor integration point
documenting the pattern for future non-tree-sitter language additions.

* feat(cobol): enrich graph with EXEC SQL/CICS, ENTRY points, MOVE data flow, PERFORM THRU

Maps the remaining 60% of CobolRegexResults to the graph:
- EXEC SQL blocks → CodeElement nodes + ACCESSES edges to DB tables
- EXEC CICS LINK/XCTL → CodeElement nodes + cross-program CALLS edges
- ENTRY points → Constructor nodes (registered for cross-program resolution)
- MOVE statements → ACCESSES edges (read/write data flow tracking)
- PERFORM THRU → expanded CALLS edges for range targets
- File declarations → Record nodes with assignment metadata
- Cross-program CALL 2nd pass: resolves unresolved targets after all programs processed

* test(cobol): add 26 integration tests with exact assertions + fix CICS resolution bug

Integration tests (test/integration/resolvers/cobol.test.ts):
- 26 tests covering full COBOL system extraction
- ALL assertions use exact toBe(N) — zero fuzzy assertions
- Fixtures: CUSTUPDT.cbl, AUDITLOG.cbl, CUSTDAT.cpy, RPTGEN.cbl, RUNJOBS.jcl

Bug fix (cobol-processor.ts):
- CICS LINK/XCTL cross-program resolution was broken — edges were
  created with "resolved" reason but pointing to <unresolved> targets
- Fix: use cics-link-unresolved / cics-xctl-unresolved suffix pattern
  matching the existing cobol-call-unresolved pattern
- Second-pass resolver now patches both CALL and CICS unresolved edges

All 3915 tests pass, 0 failures.

* test(cobol): exhaustive 57-test suite with strict exact assertions

Complete rewrite of COBOL integration tests using ground-truth approach:
dump the full graph, then assert EVERY node and EVERY edge.

57 tests across 9 sections:
- Node completeness: Module(3), Function(13), Namespace(2), Property(21),
  Record(1), CodeElement(8), Constructor(1) — exact sorted arrays
- Edge completeness: 22 tests covering every type+reason combination
  with exact source→target pairs
- Cross-program resolution: 6 tests verifying CALL, CICS LINK/XCTL, JCL
- COPY expansion: copybook data items in RPTGEN
- Section hierarchy: exact paragraph membership per section
- Data item ownership: exact per-module breakdown
- MOVE data flow: exact read/write pairs
- JCL integration: job/step/dataset containment
- Grand totals: CALLS(22), CONTAINS(48), IMPORTS(1), ACCESSES(7)

Fixture enhancements:
- CUSTUPDT.cbl: added INIT-SECTION + PROCESSING-SECTION, PERFORM THRU
- AUDITLOG.cbl: added ENTRY "AUDITLOG-BATCH"
- RPTGEN.cbl: added EXEC CICS XCTL

Zero fuzzy assertions — every expect uses toBe(N) or toEqual([...sorted]).

* fix(cobol): add removeRelationship API + single-quote CALL/COPY/ENTRY, PERFORM keyword skip

Phase 0A: Add removeRelationship(id) to KnowledgeGraph interface and
implementation (trivial Map.delete wrapper). Required for orphan edge
cleanup in next commit.

Phase 1A (from PR #500 review, modified):
- RE_CALL and RE_COPY_QUOTED now match both "double" and 'single' quotes
- parseSingleCopyStatement in copy-expander updated for single quotes
- PERFORM_KEYWORD_SKIP set prevents UNTIL/VARYING/WITH/TEST/FOREVER
  from being stored as false-positive perform targets
- Sequence number stripping uses /[^0-9 ]/ (preserves numeric seq numbers
  unlike PR #500's /\S/ which stripped them)
- Normalized || to ?? for regex group extraction in copy-expander

5 new graph unit tests, all 57 COBOL integration tests pass.

* fix(cobol): RE_ENTRY single-quote + remove orphan unresolved CALLS edges

Phase 1B: RE_ENTRY regex now supports both "double" and 'single' quoted
ENTRY targets. Uses named intermediates (entryName, usingClause) with ??
operator. USING capture group shifted from [2] to [3].

Phase 1C: Second-pass resolution now collects resolved orphan edge IDs
during iteration and removes them after the loop completes, using the new
graph.removeRelationship() API. Graph no longer contains phantom
<unresolved>: edges alongside their resolved replacements. CALLS count
drops from 22 to 18 (4 orphan edges removed).

* fix(cobol): Property ID collisions + O(1) Map lookup for MOVE edges

Phase 1D+3C (atomic): Property node IDs now use composite key
filePath:section:level:name instead of filePath:name. This prevents
duplicate data item names in different sections (e.g., STATUS in both
WORKING-STORAGE and LINKAGE) from silently colliding.

New generatePropertyId() helper ensures both node creation and MOVE
edge lookup use the identical key formula. buildDataItemMap() replaces
the O(n) findDataItemNode linear scan with O(1) Map lookup, built once
per file before MOVE processing.

* feat(cobol): MOVE multi-target extraction with OF/IN qualifier filtering

MOVE X TO A B C now produces write edges for all targets, not just the
first. extractMoveTargets() helper handles OF/IN qualified names
(WS-NAME OF WS-RECORD -> target is WS-NAME), subscript stripping
(WS-TABLE(I) -> WS-TABLE), and MOVE_SKIP filtering on targets.

Data model: CobolRegexResults.moves.to:string -> targets:string[]
MOVE CORRESPONDING stays single-target per COBOL standard.
Processor MOVE loop now iterates move.targets.

* feat(cobol): COPY IN/OF library, pseudotext REPLACING, dynamic CALL, PERFORM TIMES, CICS MAP unquoted

Phase 2B: COPY ... IN/OF library-name now captured as metadata in
CopyResolution (IN and OF are synonyms per COBOL-85 standard).

Phase 2C: COPY REPLACING ==pseudotext== support. Tokenizer handles
==...== delimiters alongside "quoted" strings. Pseudotext forces EXACT
type. Two-pass applyReplacing: first pass handles space-containing/
non-identifier pseudotext via global string replace; second pass handles
identifier-level LEADING/TRAILING/EXACT. New test file
cobol-copy-expander.test.ts with 10 tests.

Phase 2E: PERFORM WS-COUNT TIMES no longer produces a false-positive
perform target (checks for TIMES keyword after captured identifier).

Phase 2F: Dynamic CALL via data item (CALL WS-PROG-NAME without quotes)
now emits a CodeElement annotation node with description 'dynamic-call'
instead of silently ignoring. Adds isQuoted:boolean to call results.

Phase 3A: CICS MAP(WS-MAP-NAME) unquoted identifiers now captured.
Phase 3B: Normalized || to ?? in copy-expander (done in Phase 1A).

* feat(cobol): nested program support — capture multiple PROGRAM-IDs per file

Phase 2D: The state machine now captures all PROGRAM-IDs, not just the
first. The primary program name stays in programName; additional nested
programs go into nestedPrograms[]. The processor creates separate Module
nodes for each nested program, contained by the outer module, and
registers them in moduleNodeIds for cross-program CALL resolution.

Paragraphs/data items are not yet scoped per-program (attributed to the
outer module) — full per-program scoping is a future enhancement that
requires END PROGRAM boundary tracking in the state machine.

* test(cobol): expand integration tests for all new language features

New fixtures:
- NESTED.cbl — two PROGRAM-IDs (OUTER-PROG, INNER-PROG) for nested
  program support testing
- COPYLIB.cpy — copybook for pseudotext REPLACING test target

Modified fixtures:
- CUSTUPDT.cbl — single-quoted ENTRY 'ALTENTRY', multi-target MOVE
  (WS-AMT TO FIELD-A FIELD-B), dynamic CALL WS-PROG-NAME, COPY COPYLIB
  with pseudotext REPLACING, LINKAGE SECTION with LS-PARAM
- RPTGEN.cbl — PERFORM WS-COUNT TIMES (false-positive guard), unquoted
  MAP(WS-MAP-NAME), additional data items WS-COUNT WS-MAP-NAME

Integration test rewritten with 62 exact assertions covering:
- 5 Module, 17 Function, 33 Property, 9 CodeElement, 2 Constructor nodes
- Nested program containment (OUTER-PROG -> INNER-PROG)
- Dynamic CALL annotation (CodeElement with cobol-dynamic-call)
- Multi-target MOVE (UPDATE-BALANCE: 2 reads, 3 writes)
- Single-quoted ENTRY (ALTENTRY under CUSTUPDT)
- PERFORM TIMES guard (WS-COUNT not in CALLS)
- Orphan unresolved edge removal (zero -unresolved edges)
- Grand totals: 21 CALLS, 68 CONTAINS, 2 IMPORTS, 10 ACCESSES

* fix(cobol): pseudotext REPLACING now applies correctly via isPseudotext flag

Root cause: ==PREFIX-== matched /^[A-Z][A-Z0-9-]*$/i (trailing hyphens
allowed), routing it to the second-pass EXACT identifier match where
PREFIX-RECORD !== PREFIX- failed silently.

Fix: Propagate isPseudotext from parseReplacingClause to CopyReplacing
interface, then use it in applyReplacing first-pass condition to force
global string replacement for all pseudotext entries regardless of
whether the content looks like an identifier.

Result: COPY COPYLIB REPLACING ==PREFIX-== BY ==WS-==. now correctly
transforms PREFIX-RECORD → WS-RECORD, PREFIX-CODE → WS-CODE, etc.

* refactor(cobol): per-program scoping via boundary tracking + line-range grouping

State machine changes (minimal, ~30 lines):
- Add RE_END_PROGRAM regex for END PROGRAM program-name. detection
- Replace nestedPrograms[] with programs[] containing startLine/endLine/
  nestingDepth metadata for each PROGRAM-ID in the file
- Reset division/section/paragraph state on new PROGRAM-ID boundary
- EOF finalization flushes remaining stack entries (single-program files)
- Programs sorted by startLine (outer before inner)

Processor changes:
- Uses programs[] with line-range containment to find enclosing parent
  Module for nested programs (replaces hardcoded nestedParent logic)
- programModuleIds Map tracks Module node IDs per program name

Fixture: NESTED.cbl now includes END PROGRAM lines for both programs.

Integration test: PREFIX-* Property nodes now correctly appear as WS-*
after the pseudotext REPLACING fix from the previous commit.

* feat(cobol): free-format COBOL support (>>source free)

Auto-detects >>SOURCE FREE directive in the first 500 chars and switches
to free-format line processing:
- No column-position rules (cols 1-6 are program text, not sequence area)
- Comments use *> prefix instead of col 7 indicator
- No continuation line indicator
- Strip inline *> comments
- Skip >>SOURCE directive lines

preprocessCobolSource() skips col-1-6 stripping for free-format files.

Paragraph/section regexes relaxed from fixed 7-space prefix to flexible
whitespace with case-insensitivity (/^\s*([A-Z][A-Z0-9-]+)\.\s*$/i).
EXCLUDED_PARA_NAMES expanded with COBOL verbs (GOBACK, END-READ, etc.)
to prevent false-positive paragraph detection in free-format.

Also fixes: entry-point-scoring.ts crash when language is 'cobol'
(MERGED_ENTRY_POINT_PATTERNS[language] was undefined → optional chaining).

Benchmark on ACAS 3.01 (268 GnuCOBOL free-format programs, 10MB):
- Before: 407 nodes, 393 edges (near-empty, only file nodes)
- After:  4,297 nodes, 3,612 edges, 542 clusters, 11 flows

* fix(cobol): relax data item regexes for free-format (^\s+ to ^\s*)

RE_FD, RE_DATA_ITEM, RE_ANONYMOUS_REDEFINES, and RE_88_LEVEL all used
^\s+ which requires at least 1 leading space. In free-format mode, lines
are trimmed before processing, so data items like "01 WS-FIELD PIC X."
have no leading whitespace after trimming.

Changed to ^\s* (zero or more spaces) which works for both fixed-format
(indented lines still have spaces) and free-format (trimmed lines).

ACAS benchmark (268 GnuCOBOL programs):
- Before: 4,297 nodes, 3,612 edges (paragraphs only)
- After:  13,832 nodes, 8,615 edges (+ data items, FDs, 88-levels)

* feat(cobol): 100% structural feature coverage — GO TO, SCREEN, SD/RD, SORT, SEARCH, CANCEL, Level 66

New extractions: GO TO (CALLS edges), SCREEN SECTION data items,
SD/RD alongside FD (Record nodes), SORT/MERGE USING/GIVING (ACCESSES),
SEARCH (ACCESSES), CANCEL (CALLS), Level 66 RENAMES (Property),
IS EXTERNAL/IS GLOBAL (Property description enrichment).

ACAS: 13,951 nodes | 13,193 edges | 685 clusters | 150 flows
(+53% edges from new GO TO/SORT/SEARCH/CANCEL extractions)

* feat(cobol): enriched CICS extraction — file I/O, dynamic PROGRAM, queues, HANDLE ABEND

EXEC CICS blocks now extract:
- FILE/DATASET clause: captures VSAM file name (literal or data item ref)
  for READ/WRITE/REWRITE/DELETE/STARTBR/READNEXT/READPREV → ACCESSES edges
- PROGRAM clause: now handles unquoted variable references (dynamic CICS
  program transfer) → CodeElement annotation with cics-dynamic-program reason
- QUEUE clause: captures TS/TD queue names from WRITEQ/READQ → ACCESSES edges
- LABEL clause: captures HANDLE ABEND error handler targets → CALLS edges
- TRANSID: now handles unquoted variable references

CodeElement descriptions enriched with all captured fields (map, program,
transid, file, queue, label).

CardDemo benchmark: +49 nodes, +33 edges from enriched CICS extraction.

* feat(cobol): complete CICS command extraction — all 7 expert recommendations

From COBOL expert agent analysis:
1. ENDBR added to isRead file command list
2. LOAD added to PROGRAM edge commands (alongside LINK/XCTL)
3. Two-word commands expanded: WRITEQ/READQ/DELETEQ TS/TD, HANDLE
   ABEND/AID/CONDITION, START TRANSID
4. Queue reason differentiated: cics-queue-read/-write/-delete
5. RETURN/START TRANSID → CALLS edges to synthetic <transid> target
6. MAP → ACCESSES edges for screen traceability
7. INTO/FROM data fields extracted → ACCESSES edges to data items

Also: dataItemMap built before CICS block processing (was declared after),
CodeElement descriptions enriched with all captured CICS fields.

* test(cobol): strict exhaustive integration tests with exact edgeSet assertions

Every edge reason has exact sorted pair assertions via edgeSet(), not
just counts. Any change to extraction that adds, removes, or reorders
edges will produce a precise, descriptive failure.

Updated RPTGEN.cbl fixture with:
- GO TO EXIT-PARAGRAPH, SORT USING/GIVING, SEARCH table
- EXEC CICS READ FILE INTO, WRITEQ TS QUEUE FROM, SEND MAP FROM
- EXEC CICS HANDLE ABEND LABEL, RETURN TRANSID, XCTL PROGRAM(variable)
- ABEND-HANDLER and EXIT-PARAGRAPH paragraphs

46 tests covering 24 CALLS + 79 CONTAINS + 18 ACCESSES + 2 IMPORTS edges
across 15 distinct edge reason codes, all with exact sorted pair lists.

* fix(cobol): address 5 findings from second Claude review (compiler front-end perspective)

Finding #2: Numeric sequence numbers now stripped (changed /[^0-9 ]/ to
/\S/ in preprocessCobolSource). Lines like "000100 MAIN-PARAGRAPH." now
have cols 1-6 blanked so paragraph regex matches correctly.

Finding #11: JCL in-stream PROC ordering fixed — pre-register all PROCs
into moduleNames before step processing. Steps that EXEC a PROC defined
later in the same file now get CALLS edges.

Finding #A: PROCEDURE DIVISION USING no longer captures calling-convention
keywords (BY, VALUE, REFERENCE, CONTENT, ADDRESS, OF) as parameter names.

Finding #C: SORT/MERGE USING/GIVING now captures ALL file references
(multi-file), not just the first. Changed from single-match to section
extraction with split.

Finding #D: Section headers no longer set currentParagraph, preventing
PERFORM caller misattribution to Namespace instead of Function nodes.

* fix(cobol): address code review findings — ReDoS fix, perf, cleanup

P1 CRITICAL — ReDoS in SORT USING/GIVING:
Replaced nested-quantifier regex with safe indexOf+substring+split
approach. No backtracking possible on crafted input.

P2 — readCopy O(M) linear scan:
Added copybookByPath reverse Map for O(1) path-to-content lookup.

P3 — Dead code removal:
Deleted unused RE_SORT_USING and RE_SORT_GIVING constants.

P3 — EXCLUDED_PARA_NAMES simplification:
Replaced 20 END-* entries with startsWith('END-') prefix check.
Auto-covers future END-* verbs.

P3 — Misplaced JSDoc on removeRelationship:
Fixed comment that described removeNodesByFile instead.
Added missing JSDoc to removeNodesByFile.

Review agents: architecture-strategist, performance-oracle,
security-sentinel, code-simplicity-reviewer.

* refactor: add Cobol to SupportedLanguages with parseStrategy: standalone

New languages/cobol.ts — standalone regex processor provider with no-op
tree-sitter fields. Declares parseStrategy: 'standalone' to distinguish
from tree-sitter-based languages.

Added parseStrategy: 'tree-sitter' | 'standalone' to LanguageProviderConfig
for languages that use their own processor instead of tree-sitter.

Removed all 11 'cobol' as any casts — now uses SupportedLanguages.Cobol.
Added empty Cobol entries to entry-point-scoring and framework-detection.

* fix(cobol): 5 fixes from third Claude review + 3 regression tests

Fixes:
- Line numbers now 1-indexed in fixed-format (was 0-indexed, off-by-one
  in jump-to-definition links)
- Copybook content preprocessed before COPY expansion (sequence numbers
  and patch markers in copybooks no longer survive into expanded source)
- ENTRY USING filters calling-convention keywords (BY, VALUE, REFERENCE,
  CONTENT, ADDRESS, OF) — same fix as PROCEDURE DIVISION USING
- SORT/MERGE trailing period stripped from USING/GIVING file tokens
- Paragraph exclusion uses exact match for SECTION/DIVISION (was substring
  match that excluded valid names like CROSS-SECTION-ANALYSIS)

USING_KEYWORDS moved to module scope for reuse by both PROCEDURE DIVISION
USING and ENTRY USING handlers.

New unit tests:
- ENTRY USING BY VALUE filtering
- Paragraph names containing SECTION not excluded
- Numeric sequence numbers stripped enabling paragraph detection

* fix(cobol): address 6 findings from fourth Claude review + tests

Fourth review findings fixed:
- New #IV: PERFORM TIMES guard uses perfMatch.index instead of
  line.indexOf (prevents wrong match when target appears earlier in line)
- New #V: 88-level condition values now handle single-quoted literals
  ('Y' no longer stored with embedded quotes)
- New #I: CANCEL edges use two-pass resolution like CALL (no longer
  silently dropped when target indexed after source)
- New #3: Multi-line SORT/MERGE accumulation — sortAccum state variable
  accumulates lines until period, then extracts USING/GIVING from full
  statement (95% of production SORT statements span multiple lines)
- New #II: PROCEDURE DIVISION USING on split lines — pendingProcUsing
  flag defers parameter capture to next line if USING not on same line
- New #6 (prior): EXCLUDED_PARA_NAMES exact match for SECTION/DIVISION

Updated fixture: RPTGEN.cbl SORT now uses multi-line format with GIVING
on separate line (period-terminated). New sort-giving integration test.
ACCESSES total: 18 → 19 (new sort-giving edge from multi-line capture).

* fix(cobol): address 4 findings from fifth Claude review

Finding #B (5 reviews old): Section/paragraph node IDs now include
enclosing program name to prevent collision when nested programs share
section/paragraph names. New findOwningProgramName() helper uses
programs[] line ranges to find the innermost enclosing program.

Finding #α: pendingProcUsing now reset in the if(procUsingMatch) branch
(was only set in else branch, could leak across nested programs).

Finding #β: RE_CALL_DYNAMIC uses negative lookbehind (?<![A-Z0-9-]) to
prevent false-positive on compound identifiers like WS-CALL OCCURS.

Finding #γ: sortAccum flushed at EOF (parallel to flushSelect and
pendingFdName EOF cleanup). Prevents silent loss of SORT USING/GIVING
relationships in truncated files.

* fix(cobol): address findings from reviews 5+6 with full test coverage

Review 5 fixes:
- #α: pendingProcUsing reset in if(procUsingMatch) branch
- #β: RE_CALL_DYNAMIC negative lookbehind prevents WS-CALL false positive
- #γ: sortAccum flushed at EOF for truncated files
- #B: Section/paragraph IDs include owning program name

Review 6 fixes:
- #P: sectionNodeIds/paraNodeIds maps use program-scoped keys
  (PROGNAME:NAME). New scopedParaLookup/scopedCallerLookup helpers.
  findContainingSection updated with programs parameter.
- #Q: RETURNING added to USING_KEYWORDS for COBOL 2002+
- #R: RE_PERFORM matches both THRU and THROUGH via alternation

New unit tests (6):
- PERFORM THROUGH captures thruTarget
- PROCEDURE DIVISION USING RETURNING filters keyword
- RE_CALL_DYNAMIC no false-match on WS-CALL compound identifier
- Multi-line SORT captures USING/GIVING from continuation lines
- PROCEDURE DIVISION USING on split line via pendingProcUsing
- Copybook preprocessing strips sequence numbers

* fix(cobol): address findings from seventh Claude review + 3 tests

Review 7 fixes:
- #i: findContainingSection only updates best when lookup succeeds
  (prevents undefined overwriting valid parent section)
- #ii: RE_PROC_SECTION handles segment numbers (SECTION 30.)
- #III: procedureUsing now stored per-program on boundary stack
  entries, propagated to programs[] output. Inner programs no longer
  overwrite outer program's parameters.
- #δ: Dynamic CANCEL (CANCEL variable) now creates CodeElement
  annotation node, matching dynamic CALL behavior. RE_CANCEL_DYNAMIC
  with negative lookbehind. cancels[] gains isQuoted field.
- #Q: RETURNING added to USING_KEYWORDS (already in prev commit)
- #R: PERFORM THROUGH already fixed (THRU|THROUGH alternation)

New unit tests:
- Nested programs carry per-program procedureUsing
- SECTION with segment number detected
- Dynamic CANCEL via data item captured with isQuoted=false

* feat(cobol): link PROCEDURE DIVISION USING to LINKAGE data items + close 4 findings

Finding #10 FIXED: procedureUsing parameters now create ACCESSES edges
with reason 'cobol-procedure-using' from Module to matching LINKAGE
SECTION Property nodes. This exposes the program's parameter contract
in the graph (e.g., AUDITLOG → LS-CUST-ID, AUDITLOG → LS-AMOUNT).

Findings closed by expert agent consensus:
- #6 COPY IN library: WONTFIX — captured metadata, no universal
  library-to-directory mapping exists. Field costs nothing and is useful
  for library queries.
- #14 SQL DELETE: WONTFIX — DB2 requires FROM; existing FROM pattern
  handles it. Bare DELETE would risk false positives.
- #E OCCURS DEPENDING ON: WONTFIX — runtime sizing concern, not
  structural. The static occurs count is sufficient for indexing.

All 39 findings from 7 Claude reviews now resolved or closed.

* fix(cobol): resolve 48 review findings across 9 review cycles

Ninth deep review resolved all remaining COBOL parser gaps identified
by 5 specialist agents (COBOL expert, architecture strategist,
TypeScript reviewer, security sentinel, code simplicity reviewer).

Fixes (P1 — critical):
- SELECT OPTIONAL now correctly skips OPTIONAL keyword (C1)
- RETURNING params excluded from PROCEDURE DIVISION USING list (C7)
- SORT GIVING no longer captures clause keywords as file names (C5)
- Extract flushSort() helper eliminating 40-line duplication (S2)
- Flush unclosed EXEC blocks at EOF matching SORT/SELECT pattern (S3)
- Guard undefined map key in jcl-processor moduleNames (S1)
- Add MAX_TOTAL_EXPANSIONS=500 to prevent exponential COPY breadth (S4)

Fixes (P2 — important):
- Quote-aware stripInlineComment for | and *> in string literals (C2+C3)
- Fixed-format literal continuation now handles quoted strings (C6)
- PROGRAM-ID detected regardless of division state for siblings (C9)

Fixes (P3 — cleanup):
- EXEC SQL INTO restricted to INSERT INTO to avoid FETCH false-pos (C8)
- Copy expander line numbers fixed from 0-based to 1-based (C11)
- Remove dead code: inInStreamProc, fileIsLiteral, expansionDepth (S7-S10)

Also fixes 8th-review findings: nested program CONTAINS attribution,
multi-PERFORM on same line, INPUT/OUTPUT PROCEDURE IS in SORT,
GO TO DEPENDING ON multi-target, MOVE CORR abbreviation, per-program
procedureUsing ACCESSES edges.

Tests: 145 COBOL tests passing (59 integration + 86 unit)
Benchmarks: CardDemo 12,323 nodes/8,893 edges (7.4s)
            ACAS 14,016 nodes/15,452 edges (9.3s, -9% faster)

* docs(cobol): update documentation for ninth review cycle fixes

Update all 4 COBOL documentation files to reflect the 16 fixes
from the ninth review cycle:

- regex-extraction.md: quote-aware comment stripping, SELECT OPTIONAL,
  RETURNING exclusion, SORT_CLAUSE_NOISE filter, flushSort() helper,
  GO TO multi-target, PROGRAM-ID division-independent detection
- copy-expansion.md: MAX_TOTAL_EXPANSIONS=500 breadth guard, 1-based
  line numbers, removed expansionDepth/warnedCircular param
- deep-indexing.md: GO TO DEPENDING ON, INPUT/OUTPUT PROCEDURE IS,
  MOVE CORR edge reasons, INSERT INTO restriction, literal continuation
- performance.md: updated benchmarks (CardDemo 12,323n/8,893e/7.4s,
  ACAS 14,016n/15,452e/9.3s), COPY breadth guard

* fix(cobol): resolve 10th review findings — nested program edge attribution

Fix 6 findings from the 10th review (PR #498 comment #4132201110):

#A+#F: All CALL/CANCEL/CICS/ENTRY/SQL/SEARCH/file-declaration edges
now use owningModuleId() for nested program attribution instead of
the outer program's parentId. Added helper function owningModuleId()
to centralize the pattern.

#B: Added USING and GIVING to SORT_CLAUSE_NOISE set to prevent MERGE
USING + OUTPUT PROCEDURE from capturing clause keywords as file names.

#C: INPUT/OUTPUT PROCEDURE regex now captures optional THRU/THROUGH
range end paragraph, mirroring RE_PERFORM's THRU support.

#D: scopedCallerLookup fallback now uses programModuleIds.get(pgm)
instead of parentId, so PERFORM/MOVE/GOTO in nested programs with
unresolvable paragraphs fall back to the correct inner module.

#E: pendingProcUsing only set when PROCEDURE DIVISION line is NOT
period-terminated, preventing false USING expectation.

Tests: 145 passing | TypeScript clean

* fix(cobol): resolve 10th review findings — nested program edge attribution

Fix 6 findings from the 10th review (PR #498 comment #4132201110):

#A+#F: All CALL/CANCEL/CICS/ENTRY/SQL/SEARCH/file-declaration edges
now use owningModuleId() for nested program attribution instead of
the outer program's parentId. Added helper function owningModuleId()
to centralize the pattern.

#B: Added USING and GIVING to SORT_CLAUSE_NOISE set to prevent MERGE
USING + OUTPUT PROCEDURE from capturing clause keywords as file names.

#C: INPUT/OUTPUT PROCEDURE regex now captures optional THRU/THROUGH
range end paragraph, mirroring RE_PERFORM's THRU support.

#D: scopedCallerLookup fallback now uses programModuleIds.get(pgm)
instead of parentId, so PERFORM/MOVE/GOTO in nested programs with
unresolvable paragraphs fall back to the correct inner module.

#E: pendingProcUsing only set when PROCEDURE DIVISION line is NOT
period-terminated, preventing false USING expectation.

Tests: 145 passing | TypeScript clean

* fix(cobol): resolve 11th review findings — final nested program + multi-CALL gaps

#1: scopedCallerLookup(null) now uses owningModuleId(lineNum) instead
of parentId, fixing PERFORM/MOVE/GOTO before first paragraph in nested
programs.

#2+#3: CALL and CANCEL extraction now uses matchAll (global flag) to
capture multiple occurrences on the same line. Dynamic CALL/CANCEL
checked independently instead of in else branch.

#4: SORT/MERGE ACCESSES edge IDs now use owningModuleId(sort.line)
instead of parentId for nested program correctness.

#5: preprocessCobolSource free-format detection now uses first 10 lines
(consistent with extractCobolSymbolsWithRegex threshold).

#6: EXCLUDED_PARA_NAMES expanded with DISPLAY, ACCEPT, WRITE, READ,
REWRITE, DELETE, OPEN, CLOSE, RETURN, RELEASE, SORT, MERGE to prevent
false-positive paragraph detection on isolated verbs.

Also removed unused GraphNode import from cobol-processor.ts.

Tests: 145 passing | TypeScript clean

* docs(cobol): deepened full language coverage plan with research findings

3 research agents analyzed Phase 1-2 features and graph value ranking.

Key findings: cobol-call-using is #1 edge type (9.2/10); multi-line
accumulation is dominant challenge; DECLARATIVES is lowest-risk Phase 2
item; SET TO TRUE covers 80-90% of SET usage.

* feat(cobol): implement Phase 1 — high-value data flow edges

4 new extraction features that create new ACCESSES and IMPORTS edges:

1.1: EXEC SQL INCLUDE -> IMPORTS edges with reason 'sql-include'
     Handles unquoted (SQLCA), quoted ('DBRMLIB.MEMBER'), and
     underscored (CUST_TBL_DCL) member names.

1.2: CALL USING parameter extraction -> ACCESSES edges
     Extracts parameters from CALL USING clause, filtering BY/REFERENCE/
     CONTENT/VALUE/ADDRESS/OF/LENGTH/OMITTED keywords. Creates
     'cobol-call-using' ACCESSES edges (graph value: 9.2/10).

1.4: OCCURS DEPENDING ON -> ACCESSES edges with reason 'cobol-depends-on'
     Extended OCCURS regex captures DEPENDING ON field with subscript
     stripping. Creates dependency edge from table to controlling field.

1.5: VALUE clause for standard data items
     Extracts VALUE from data item clauses: quoted strings with type
     prefix (X/N/G/B), ALL literals, numerics (incl negative/decimal),
     and figurative constants. Populates Property node values.

Tests: 145 passing (+2 ACCESSES from CALL USING) | TypeScript clean

* feat(cobol): implement Phase 2 — DECLARATIVES, SET, INSPECT, EXEC DLI

4 new extraction features for error handling, data flow, and IMS/DB:

2.1: EXEC DLI (IMS/DB) -> CodeElement + ACCESSES edges
     Accumulates EXEC DLI blocks like EXEC SQL. Parses DLI verbs
     (GU, GN, ISRT, REPL, DLET, CHKP, SCHD, TERM). Extracts
     SEGMENT, PCB, INTO/FROM, PSB. Creates dli-{verb} ACCESSES
     edges to <ims>:segment Record nodes.

2.2: DECLARATIVES / USE AFTER EXCEPTION -> ACCESSES edges
     Tracks inDeclaratives state. Detects USE AFTER STANDARD
     EXCEPTION ON file-name. Creates cobol-error-handler ACCESSES
     edge from handler section to file Record.

2.3: SET statement -> ACCESSES edges
     Detects SET TO TRUE (80-90% of SET usage) and SET index
     TO/UP BY/DOWN BY. Creates cobol-set-condition / cobol-set-index
     write edges + cobol-set-read for identifier values.

2.4: INSPECT -> ACCESSES edges with multi-line accumulator
     Accumulates INSPECT until period (like SORT). Extracts inspected
     field + tally counters. Creates cobol-inspect-read/write/tally
     edges. Form detection: tallying/replacing/converting/combined.

Preprocessor: 1398 -> 1597 LOC (+199). Tests: 145 passing.

* feat(cobol): implement Phase 3 — completeness fixes

6 partial features fixed to first-class support:

3.1: CALL RETURNING -> ACCESSES write edge (cobol-call-returning)
3.2: SELECT OPTIONAL flag preserved in FileDeclaration + Record node
3.3: ALTERNATE RECORD KEY extraction (matchAll for multiple keys)
3.4: COMMON attribute on nested programs (RE_PROGRAM_ID extended)
3.5: IS EXTERNAL / IS GLOBAL as first-class boolean properties
     (removed usage string hack)
3.6: AUTHOR / DATE-WRITTEN mapped to Module node description

Tests: 145 passing | TypeScript clean

* feat(cobol): implement Phase 4 — INITIALIZE + metadata completeness

4.1: INITIALIZE statement -> ACCESSES write edge (cobol-initialize)
4.2: DATE-COMPILED and INSTALLATION paragraphs extracted and mapped
     to Module node description alongside existing AUTHOR/DATE-WRITTEN

All 4 plan phases complete. Coverage: ~95% (up from 71.9%).
Tests: 145 passing | TypeScript clean

* test(cobol): add 24 unit tests for Phase 1-4 features

Coverage for all new extraction features:

Phase 1 (8 tests):
- EXEC SQL INCLUDE (unquoted, quoted, underscored)
- CALL USING (simple, mixed modes, ADDRESS OF, OMITTED)
- CALL RETURNING
- OCCURS DEPENDING ON
- VALUE clause (string, numeric, figurative constant)

Phase 2 (10 tests):
- EXEC DLI GU/ISRT/SCHD (verb, segment, PCB, INTO, FROM, PSB)
- DECLARATIVES USE AFTER EXCEPTION (single + multiple sections)
- SET TO TRUE, SET index UP BY
- INSPECT TALLYING, INSPECT REPLACING

Phase 3-4 (6 tests):
- SELECT OPTIONAL flag
- ALTERNATE RECORD KEY
- PROGRAM-ID IS COMMON
- IS EXTERNAL / IS GLOBAL booleans
- INITIALIZE extraction
- Full programMetadata (AUTHOR, DATE-WRITTEN, DATE-COMPILED, INSTALLATION)

Total: 168 tests passing (145 + 24 - 1 removed duplicate)

* fix(cobol): use /\r?\n/ split for Windows CRLF compatibility

All 4 COBOL source files now split on /\r?\n/ instead of '\n' to
handle CRLF line endings on Windows. Previously, trailing \r in
lines caused RE_GOTO's $ anchor to fail on multi-line GO TO
DEPENDING ON statements, producing only 1 goto edge instead of 4.

Files fixed: cobol-preprocessor.ts (2 sites), cobol-processor.ts,
jcl-parser.ts, cobol-copy-expander.ts

Tests: 168 passing | TypeScript clean

* fix(cobol): resolve 12th review — dynamic CALL/CANCEL dedup + trailing anchors

#1+#2: Removed incorrect hasQuotedCall/hasQuotedCancel deduplication
guards. RE_CALL_DYNAMIC and RE_CANCEL_DYNAMIC require [A-Z] after
CALL/CANCEL, so they CANNOT match quoted targets — the guards were
both unnecessary and actively harmful, suppressing dynamic CALL/CANCEL
in ON EXCEPTION patterns.

#3+#5: Changed RE_CALL_DYNAMIC and RE_CANCEL_DYNAMIC trailing anchor
from (?:\s|\.) to (?=\s|\.|$) (lookahead). The consuming anchor
failed when the identifier was the last token on a physical line.

Tests: 168 passing | TypeScript clean

* feat(cobol): add CALL accumulator + fix SORT double-statement (#4, #6)

Finding #4: Multi-line CALL USING accumulator
Added callAccum state variable that accumulates CALL statements
spanning multiple physical lines until period or END-CALL is found.
Uses flushCallAccum() to re-extract CALL target + USING parameters
from the full accumulated statement. This fixes the silent loss of
ACCESSES parameter edges when USING appears on lines after CALL.

Finding #6: SORT double-statement on same line
After flushSort(), the code now falls through to re-check the
current line for a new SORT/MERGE start (was previously blocked
by the sortAccum === null check evaluating before flushSort ran).

Also fixed: used non-global regex for CALL detection test to avoid
the classic global regex .test() lastIndex bug.

Tests: 168 passing (+1 ACCESSES from multi-line CALL USING)

* fix(cobol): resolve 13th review — CICS LOAD, USING extraction, file scoping

#1: CICS LOAD unresolved edge no longer silently deleted in second pass.
    Changed narrow cics-link/cics-xctl check to catch-all pattern:
    rel.reason?.startsWith('cics-') && rel.reason.endsWith('-unresolved')

#2: flushCallAccum USING extraction now stops before COBOL statement
    verbs (INSPECT, SEARCH, SORT, MERGE, DISPLAY, ACCEPT, MOVE, PERFORM,
    GO TO, CALL, IF, EVALUATE). Prevents absorbing adjacent statements
    as false USING parameters in legacy pre-COBOL-85 code without END-CALL.

#3: CICS FILE Record nodes now globally-scoped (<cics-file>:FILENAME)
    instead of per-file-scoped. Enables cross-program CICS file access
    analysis, consistent with SQL table scoping (<db>:TABLE).

#4: callAccum pre-check regex now has (?<![A-Z0-9-]) lookbehind to
    prevent false activation on compound identifiers like WS-CALL-FLAG.

Tests: 168 passing | TypeScript clean

* fix(cobol): resolve 14th review — callAccum false paragraph + Area A guard

#1: callAccum continuation lines now check for COBOL statement verb
    starts (GO TO, PERFORM, MOVE, etc.) and paragraph/section headers.
    If detected, the CALL is flushed as-is and the line processed
    normally — prevents false paragraph detection and currentParagraph
    corruption from lines like "WS-ADDR." being treated as paragraphs.

#4: callAccum pre-check now guarded by currentDivision === 'procedure'
    to prevent unnecessary activations in DATA DIVISION.

#5: Fixed-format paragraph detection now rejects lines with >7 leading
    spaces (Area B indentation) as paragraph candidates. Paragraph
    names in fixed-format must start in Area A (col 8-11, max 7 spaces).
    Free-format mode is unaffected.

Tests: 168 passing | TypeScript clean

* fix(cobol): resolve 15th review — callAccum Area A + verb boundary fixes

#A: Column-position-aware paragraph detection in callAccum flush.
#B: inspectAccum early-flush on paragraph/section/verb headers.
#C: Verb boundary \b → (?:\s|$) prevents MOVE-COUNT false flush.

* test(cobol): add 17 edge-case regression tests + fix USING verb boundary

17 new tests covering all recurring review patterns:

Multi-line CALL USING (7 tests):
- Parameters on separate continuation lines (IBM mainframe style)
- No absorption of INSPECT/GO TO/paragraphs following CALL
- END-CALL scope terminator
- Hyphenated identifiers (MOVE-COUNT) not triggering false flush
- Dual quoted+dynamic CALL on same line (ON EXCEPTION)

Nested program attribution (2 tests):
- CALL in inner program within inner line range
- PERFORM before first paragraph has null caller

CRLF compatibility (1 test):
- GO TO DEPENDING ON with \r\n line endings

Area A paragraph detection (2 tests):
- Area B (>7 spaces) rejected; Area A (7 spaces) accepted

SORT/MERGE (1 test): COLLATING SEQUENCE keywords not captured
PROCEDURE USING (2 tests): RETURNING excluded, period-terminated
Comment stripping (1 test): pipe in quoted string preserved
SELECT OPTIONAL (1 test): correct file name, not OPTIONAL keyword

Bug fix: USING extraction regex verb terminators changed from
\bVERB\b to \bVERB(?=\s|$) in flushCallAccum — prevents truncation
on hyphenated identifiers like MOVE-COUNT, PERFORM-LIMIT.

Total: 185 tests passing

* test(cobol): add 32 comprehensive edge-case regression tests

13 new describe blocks covering all extraction features:

- EXEC DLI: no-SEGMENT, multi-line accumulation (2 tests)
- SET: multiple targets, DOWN BY, TO numeric (3 tests)
- INSPECT: CONVERTING, multiple counters, tallying-replacing,
  paragraph flush during accumulation (4 tests)
- DECLARATIVES: no-STANDARD keyword, I-O mode, post-END paragraphs (3)
- COPY REPLACING: pseudotext deletion ==OLD== BY ==== (1 test)
- VALUE: hex literal, negative numeric, ALL literal (3 tests)
- OCCURS: TO range, fixed-size without DEPENDING ON (2 tests)
- Dynamic CALL/CANCEL: end-of-line, multiple CANCELs (3 tests)
- EXEC SQL: INCLUDE skips tables, SELECT INTO host vars, host
  variable extraction (3 tests)
- INITIALIZE: target and caller context (1 test)
- Nested programs: sibling scoping, PROGRAM-ID without ID DIV (2)
- EXEC EOF flush: unclosed EXEC SQL flushed (1 test)
- Multi-PERFORM: IF/ELSE dual PERFORM on single line (1 test)
- IS EXTERNAL: USAGE not polluted by external flag (1 test)

Total: 215 tests passing

* fix(cobol): resolve 16th review — CANCEL in CALL block + USING boundary

#1: flushCallAccum now extracts CANCEL statements from within CALL
    ON EXCEPTION blocks. Adds RE_CANCEL + RE_CANCEL_DYNAMIC matchAll
    passes alongside existing CALL extraction.

#2: Added \bCANCEL(?=\s|$) to USING lookahead regex to prevent CANCEL
    keyword being captured as false USING parameter.

#3: Multi-line CALL start now returns immediately to prevent the CALL
    start line from simultaneously feeding sortAccum/inspectAccum.

#6: Division transitions now flush all active accumulators (callAccum,
    sortAccum, inspectAccum) to prevent state leakage across programs.

Also added CANCEL to callAccum flush trigger verb list.

Tests: 215 passing | TypeScript clean

* refactor(cobol): extract shared verb constants + resolve 17th review

Extract COBOL_STATEMENT_VERBS, RE_STATEMENT_VERB_START, and
RE_USING_PARAMS as shared constants — eliminates 4 duplicated
25-verb regex patterns.

17th review: #1 flushCallAccum before EXEC entry, #2 inspectAccum
verb parity via shared constant.

Tests: 215 passing | TypeScript clean

* test(cobol): replace all fuzzy assertions with exact toBe checks

Replaced 7 toBeGreaterThan/toBeLessThan/toBeGreaterThanOrEqual
assertions with exact toBe values:

- dataItems.length: >= 3 → toBe(3)
- calls.length: >= 1 → toBe(1)
- calls[0].line: range check → toBe(10)
- programs[].startLine/endLine: comparison → exact values
- innerA.endLine/innerB.startLine: comparison → exact values

Also added 11 new edge-case tests (accumulator flush on EXEC/division
transitions, free-format, CANCEL in CALL block, SORT THRU, verb
flush, integration).

226 tests passing — zero fuzzy assertions remain.

* fix(cobol): resolve 19th review + 15 accumulator flush tests

Fixes:
#1: END PROGRAM flushes callAccum/sortAccum/inspectAccum
#2: PROGRAM-ID sibling path flushes all accumulators
#3: Added COMPUTE/ADD/SUBTRACT/MULTIPLY/DIVIDE/STRING/UNSTRING
    to COBOL_STATEMENT_VERBS (now 32 verbs)

Tests (15 new):
- END PROGRAM flush: single + nested programs (2)
- PROGRAM-ID sibling flush (1)
- Arithmetic verb flush: COMPUTE/ADD/SUBTRACT/MULTIPLY/DIVIDE (5)
- String verb flush: STRING/UNSTRING (2)
- Arithmetic not captured as false USING params (1)
- SORT flushed at END PROGRAM (1)
- INSPECT flushed at END PROGRAM (1)
- All with exact toBe assertions (2)

Total: 239 tests passing | Zero fuzzy assertions

* fix(cobol): resolve 20th review — INITIALIZE multi-target + 2 tests

Finding 1: INITIALIZE now captures multiple targets with REPLACING
clause keyword filtering. Regex changed to lazy match stopping at
REPLACING/WITH/period boundary. Targets split on whitespace and
filtered against INITIALIZE_CLAUSE_KEYWORDS set.

Tests (2 new):
- INITIALIZE multi-target: WS-CUSTOMER WS-ORDER WS-LINE-ITEM → 3
- INITIALIZE with REPLACING: only WS-RECORD captured, not keywords

Total: 241 tests passing | TypeScript clean
2026-03-26 14:03:48 +00:00
3c896cdbcd fix: close remaining Dart language support gaps (#524)
* fix: close remaining Dart language support gaps

Four issues that were not addressed in PR #204:

1. extractFunctionName: add function_signature/method_signature handlers
   and add both to FUNCTION_NODE_TYPES. Without this, findEnclosingFunctionId
   cannot resolve Dart function scopes — all calls inside Dart functions
   have no sourceId, breaking CALLS edge attribution.

2. formal_parameter_list: add to paramListTypes in extractMethodSignature.
   Dart's tree-sitter grammar uses this node type (not formal_parameters),
   so parameter counting returns 0 for all Dart functions.

3. Write-access queries: add @assignment patterns for obj.field = value
   and this.field = value. Without these, no ACCESSES write edges are
   emitted for Dart code.

4. initialized_identifier guard in extractDartDeclaration: comma-separated
   declarations (String a, b, c) produce initialized_identifier nodes
   which are in DART_DECLARATION_NODE_TYPES but were unhandled — the type
   lives on the parent node.

Also adds Dart column to the feature matrix in type-resolution-system.md.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(dart): field-type resolution, call attribution, import resolution, and integration tests

Fixes five Dart language support gaps with integration tests and
architectural alignment:

**Tree-sitter queries** — Add field declaration patterns for typed and
nullable class fields (`String name = ''`, `String? name`).  Without
these, Dart class fields were invisible to the pipeline (zero Property
nodes, zero HAS_PROPERTY edges).

**Import resolution** — Dart relative imports (`import 'models.dart'`)
don't use a leading `./`.  The standard resolver only recognises paths
starting with `.` as relative; bare paths fell through to a Java-style
dot-to-slash conversion that mangled `models.dart` into `models/dart`.
Fix: prepend `./` before calling resolveStandard.

**Call attribution** — Dart's tree-sitter grammar places `function_body`
as a sibling of `function_signature`, not as a child wrapping both.  The
`findEnclosingFunction` parent-walk never found the function because the
call lives inside `function_body` which is a sibling of the signature.
Fix: add `enclosingFunctionFinder` hook to LanguageProvider interface
(following the same strategy pattern as `labelOverride`), with the
Dart-specific logic in `languages/dart.ts`.  Both `parse-worker.ts` and
`call-processor.ts` consume the hook generically — no Dart-specific code
in the generic processors.

**Receiver chain extraction** — Add `unconditional_assignable_selector`
to `MEMBER_ACCESS_NODE_TYPES` so `inferCallForm` returns `'member'` for
Dart method calls.  Add Dart-specific receiver extraction blocks in
`extractReceiverName`, `extractReceiverNode`, and a `selector` handler
in `extractMixedChain` for Dart's flat sibling-selector model (vs the
nested member-expression model used by all other languages).

**Integration tests** — New `dart.test.ts` with field-type resolution
and call-result-binding describe blocks.  Fixtures: `dart-field-types/`
(models.dart + app.dart) and `dart-call-result-binding/` (models.dart +
app.dart).  9 passing tests, 1 skipped (ACCESSES edges for field reads
depend on type-env parameter binding propagation — tracked for follow-up).

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-26 13:45:24 +00:00
Wojciech Guziak 546128cdcb refactor: split global BUILT_IN_NAMES into per-language provider fields (#523)
* refactor: make isBuiltInOrNoise provider-aware, remove global BUILT_IN_NAMES

Add builtInNames field to LanguageProviderConfig. Rewrite noise-filter.ts
to accept a LanguageProvider and check provider.builtInNames instead of
a global Set. Update all 3 call sites to pass their existing provider.

Built-in entries will be added per-language in subsequent commits.

* refactor(js/ts): add per-language builtInNames to JS/TS providers

* refactor(python): add per-language builtInNames

* refactor(kotlin): add per-language builtInNames

* refactor(c/cpp): add per-language builtInNames

* refactor(csharp): add per-language builtInNames

* refactor(php): add per-language builtInNames

* refactor(swift): add per-language builtInNames

* refactor(rust): add per-language builtInNames

* refactor(ruby): add per-language builtInNames

* refactor(dart): add per-language builtInNames

* test: update noise-filter tests for per-language API, add isolation tests

- Update ingestion-utils.test.ts to pass provider to isBuiltInOrNoise
- Add noise-filter.test.ts with 15 cross-language isolation tests
- Fix Java heritage test: serialize() is now correctly unfiltered for Java
  (was false-positive noise from global PHP serialize entry)

* refactor: remove noise-filter.ts, add provider.isBuiltInName() method

Per review feedback: delete noise-filter.ts entirely and move the check
into LanguageProvider as isBuiltInName(name) method, generated by
defineLanguage() from the builtInNames set.

Call sites now use provider.isBuiltInName(calledName) directly.
2026-03-26 12:15:29 +00:00
Flavius MironandClaude Opus 4.6 c6ed25de8b docs: add Dart to supported languages table in README (#525)
Dart was added as the 14th supported language in PR #204 but the README
was not updated. Adds Dart row to the supported languages table and
updates the language count from 13 to 14.

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-26 10:46:10 +00:00
ThinhKVT aec9a3f216 [CLI] Fixes a false-positive in the Cypher write-detection regex and improves the Impact tool's enrichment path by using batched chunking and entry-point grouping (#496) (#507) 2026-03-26 10:39:27 +00:00
marxo126 a047a08f54 feat: add ORM dataflow detection (Prisma + Supabase) (#511) 2026-03-26 08:48:40 +00:00
Wojciech Guziak 8864a0290c feat: add Dart language support (#204) 2026-03-26 08:42:49 +00:00
Chirag Nighut 1a0784befe feat: add more node types in filter panel (#519)
* fix(web): enable Cypher queries when connected to backend server

Route queries through HTTP API in backend mode instead of checking local WASM database.

Made-with: Cursor

* chore: add Maven/Gradle wrapper files to default ignore list

Add build wrapper scripts and directories to hardcoded ignore lists:
- Directories: .mvn, .gradle, gradle
- Files: mvnw, mvnw.cmd, gradlew, gradlew.bat

These are build infrastructure files, not source code.

Made-with: Cursor

* ci: re-trigger CI (Windows flaky timeout)

Made-with: Cursor

* feat: add more node types in filter panel

* feat: add more node types in filter panel

* revert additional changes

* test(web): add unit tests for filter panel node types

- FILTERABLE_LABELS: verify new types (Enum, Type, Decorator, Variable)
  have colors, sizes, and no duplicates
- Filter panel icons: verify every filterable label has an icon mapped
  and all icons are exported from lucide-icons
- Color legend: verify new types are included, ordered correctly, and
  are a subset of FILTERABLE_LABELS

Made-with: Cursor
2026-03-26 06:50:40 +00:00
Gergő Magyar 33225c2fce fix(ci): move shape-check-regression test to lbug-db project (#518)
The shape-check-regression test uses withTestLbugDB but was running in
the default vitest project with parallel forks, causing LadybugDB
file-lock conflicts on Windows CI. Move it to the lbug-db project
(sequential execution) and exclude from default.

Follows up on #501.
2026-03-26 06:23:02 +00:00
marxo126 5a7ac218df fix: shape_check false positives — quoted keys, DOM leaks, errorKeys (#501) 2026-03-26 05:43:37 +00:00
jelsco b959b9933b fix(python): resolve two remaining alias gaps (#417) (#505) 2026-03-26 05:23:27 +00:00
Zander Raycraft 6fabd7a2df Merge pull request #402 from adonisdoda/feat/index-cli 2026-03-25 20:22:21 -05:00
marxo126 f860653a69 feat(routes): link Next.js project-level middleware.ts to routes (#504) 2026-03-25 22:07:58 +00:00
Wojciech Guziak 77dcb06a8d chore: upgrade tree-sitter to 0.25.0 and all grammar packages (#516) 2026-03-25 21:55:14 +00:00
Zander Raycraft e7e26d6345 Merge pull request #381 from cnighut/feat/cursor-cli-wiki-provider 2026-03-25 08:20:23 -05:00
marxo126 95f97c884c feat: add Expo Router file-based route detection (#503) 2026-03-25 11:05:55 +00:00
marxo126andClaude Opus 4.6 4bc4815bd2 feat: PHP response shape extraction for json_encode patterns (#502)
* feat: add PHP response shape extraction for json_encode patterns

Adds extractPHPResponseShapes() to detect response keys from PHP
json_encode() calls with associative array literals. Supports:
- Short array syntax: json_encode(['key' => value])
- Long array syntax: json_encode(array('key' => value))
- Error classification via http_response_code() and header() status
- exit;/die; boundary detection to prevent cross-block status leaking
- Nested array filtering (only top-level keys extracted)

Pipeline integration dispatches PHP files to the new extractor.
Verified on collector project: 10 PHP routes now show responseKeys.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: address review — exit boundary, die; offset, CGI Status header

- Replace lastIndexOf('exit;')/lastIndexOf('die;') with regex that
  matches exit(N), exit(0), die('msg'), die($var) as boundaries
- Fixes die; off-by-one (was slicing at +5 for a 4-char keyword)
- Add header('Status: NNN') CGI/FastCGI format detection
- Add 3 regression tests for the fixed bugs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* refactor: extract shared helpers, remove duplicate test block

- Extract lastMatchGroup() and buildShapeResult() to eliminate repeated
  patterns in both JS/TS and PHP extractors
- Simplify detectPHPStatusCode to use ?? chaining with lastMatchGroup
- Remove duplicate 9-test PHP describe block (kept the 12-test version)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add PHP response shape integration tests

Adds a PHP fixture (api/items.php, api/submit.php) with multiple
json_encode patterns and a pipeline integration test verifying:
- Route nodes created for PHP endpoints
- responseKeys/errorKeys correctly extracted and separated
- exit(N)/die() boundaries respected
- HANDLES_ROUTE edges point to correct PHP handler files

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-25 08:27:53 +00:00
c68d7975e6 docs: agent development framework, GitHub templates, eval refactor (#479)
* ci: E2E workflow, web typecheck job, pre-commit hook, test suite

CI:
- ci.yml consolidated to reference ci-tests.yml
- ci-quality.yml: add typecheck-web job for gitnexus-web/
- ci-e2e.yml: E2E workflow with dorny/paths-filter (web changes only)
- ci-report.yml: remove dead integration-reports references
- CI gate allows skipped E2E status
- .gitignore: playwright artifacts, eval test artifacts

Pre-commit hook:
- .githooks/pre-commit: typecheck + unit tests for both packages
- Activated via git config core.hooksPath in prepare script

Test infrastructure:
- Vitest + React Testing Library: 58 unit tests
  (graph, server-connection, mermaid, settings, constants, utils, paths)
- Playwright E2E: 5 tests + manual recording harness
- vitest.config from vitest/config, engines.node >= 20
- Playwright artifacts retain-on-failure
- wait-on in devDependencies
- vitest/coverage-v8 aligned with vitest 4.x

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* chore: update gitnexus-web package-lock.json

Reflects devDependency additions (vitest, playwright, wait-on,
@testing-library, etc.) from package.json changes in this PR.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(e2e): add missing process-list-loaded testid, increase CI timeouts

- Add data-testid="process-list-loaded" to ProcessesPanel (E2E tests
  were waiting for an element that didn't exist)
- Increase server connect timeouts from 5s to 10s for slower CI

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): run gitnexus-web unit tests in CI, remove unused variable

- Add gitnexus-web npm ci + vitest run to ci-tests.yml so web unit
  tests are gated by the CI status check (were only running locally)
- Remove unused IS_PLAYWRIGHT_AUTOMATION variable from E2E spec

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(e2e): add process-row testid, wait for networkidle on page load

- Add data-testid="process-row" to ProcessItem component (E2E tests
  referenced it but it didn't exist in the source)
- Use waitUntil: 'networkidle' on page.goto to ensure Vite dev server
  is fully ready before interacting (fixes first-test timeout in CI)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(e2e): add process-view-button and process-highlight-button testids

E2E tests referenced these data-testid attributes but they didn't
exist in ProcessItem. All 6 E2E testids now have matching source
elements: status-ready, process-list-loaded, process-row,
process-view-button, process-highlight-button, server-url-input.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(e2e): remove networkidle — Vite HMR WebSocket prevents it from resolving

networkidle waits for zero network activity for 500ms, but Vite's HMR
WebSocket stays open permanently, causing page.goto to timeout at 60s
on all tests after the first. The explicit toBeVisible waits on UI
elements are sufficient and deterministic.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(e2e): wait for Server button visibility, add CI retry, all 5 tests pass locally

Root cause: test 1 clicked the Server button before React hydrated,
so the tab content never rendered and the input wasn't found.

Fixes:
- Wait for Server button toBeVisible before clicking
- Increase input wait to 15s
- Remove networkidle (Vite HMR WebSocket prevents it from resolving)
- Add retries: 1 in CI for transient cold-start flakiness

Verified locally: all 5 E2E tests pass, 198 unit tests pass, typecheck clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): tolerate LadybugDB native crash during analyze step

gitnexus analyze can crash with "double free or corruption" (known
issue #273) during the LadybugDB native addon shutdown. The index is
usually written successfully before the crash. The workflow now:
1. Allows analyze to exit non-zero with a warning
2. Verifies .gitnexus index was actually created
3. Only fails if no index exists (real failure)

All tests verified locally: 198 unit, 5 E2E pass, typecheck clean.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(ci): fix shell quoting in analyze step, simplify to || true

The previous echo string had special characters that broke bash
quoting in GitHub Actions. Simplified to: analyze || true, then
check if .gitnexus exists.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: add agent development framework, GitHub templates, eval refactor

Agent framework (layered docs for AI-assisted contributions):
- AGENTS.md: canonical instructions, impact analysis, MCP tools
- CLAUDE.md: Claude Code-specific deltas and hooks
- GUARDRAILS.md: safety boundaries, non-negotiables, escalation
- ARCHITECTURE.md: monorepo layout, data flow map
- TESTING.md: test structure, commands, categories
- RUNBOOK.md: copy-paste operations for dev/CI/MCP
- llms.txt: minimal LLM context pointer

Editor integration:
- .cursor/index.mdc + rules/100-monorepo.mdc

GitHub templates:
- PR template with areas-touched checkboxes
- Bug report + feature request issue forms

Eval harness:
- Refactored mcp_bridge, tool_registry, constants
- Error sanitization utilities
- Property-based tests via Hypothesis

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix(eval): use format_exception instead of format_exc in sanitize_exception

format_exc() returns the currently handled exception traceback, which
may be unrelated if called outside an active except block. Using
format_exception(type(exc), exc, exc.__traceback__) reliably captures
the passed exception's traceback.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update CONTRIBUTING.md and TESTING.md for current CI/hook setup

- CONTRIBUTING.md: add gitnexus-web typecheck command, pre-commit hook
  checklist item
- TESTING.md: add gitnexus-web typecheck command, pre-commit hook
  section (husky), update CI integration to list actual workflow files
  (ci-quality, ci-tests, ci-e2e)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: update testing docs to reflect CI/E2E changes from PR #486

- AGENTS.md: update test counts (CLI ~2000 unit, ~1850 integration),
  add gitnexus-web testing section (198 unit, 5 E2E with commands)
- RUNBOOK.md: fix Node requirement to >=20, fix E2E local repro command
- TESTING.md: E2E uses data-testid selectors + real servers, not mocks
- .cursor/rules/100-monorepo.mdc: add web test/E2E commands

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* docs: address context engineering review — deduplicate tokens, expand Cursor rules

- Remove ~100-line gitnexus:start block from CLAUDE.md (was duplicated from AGENTS.md)
- Fix gitnexus:start block inlined inside AGENTS.md Reference Docs bullet (doubled)
- Replace CLAUDE.md scope table with pointer to AGENTS.md (single source of truth)
- Expand .cursor/index.mdc with 5 non-negotiable safety rules for always-on context
- Add .cursor/rules/200-eval.mdc with Python/eval commands (glob-scoped to eval/**)
- Improve llms.txt with priority annotations and descriptions
- Bump version headers to 1.2.0, last-reviewed to 2026-03-24

Saves ~1,400 tokens/session with zero information loss.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
2026-03-25 06:48:41 +00:00
chirag-nighut 048347df84 fix: address PR review — TTY guard, test rename, unify debug env var
- Add process.stdin.isTTY guard before --review prompt to prevent CI hangs
- Rename misleading --verbose e2e test to reflect it checks help output
- Replace DEBUG with GITNEXUS_VERBOSE for error stack traces

Made-with: Cursor
2026-03-25 11:39:37 +05:30
chirag-nighut f17bde69ba ci: re-trigger CI (Windows flaky timeout in skills-e2e)
Made-with: Cursor
2026-03-25 11:39:37 +05:30
chirag-nighut 7e66ec3a4f test: add e2e CLI tests for wiki flags (--provider, --review, --verbose)
Spawn actual CLI process to verify:
- wiki --help surfaces all new flags
- wiki on non-git directory exits with code 1
- wiki on non-indexed repo fails with "No GitNexus index"
- --provider cursor skips API key prompt in non-TTY mode
- --verbose is accepted as valid flag

Made-with: Cursor
2026-03-25 11:39:37 +05:30
chirag-nighut 76a581b295 test: add unit tests for wiki CLI flags (--provider, --review, --verbose)
17 tests covering:
- detectCursorCLI caching (avoids repeated spawns)
- resolveCursorConfig defaults
- resolveLLMConfig provider routing (cursor vs openai)
- --verbose env var propagation
- WikiGenerator reviewOnly mode (early return with moduleTree)
- CLI config round-trip for cursor and openai providers
- invokeLLM dispatch (cursor → callCursorLLM, openai → callLLM)
- callCursorLLM error when CLI not found
- estimateTokens heuristic

Made-with: Cursor
2026-03-25 11:39:37 +05:30
chirag-nighut 1e309c1ec1 fix: address PR review — remove redundancies and add wiki help test
- Cache detectCursorCLI() result to avoid spawning `agent --version`
  on every LLM call
- Fix stale JSDoc in cursor-client.ts (no longer uses stdin or
  stream-json)
- Remove unused WikiOptions fields (model, baseUrl, apiKey) that were
  passed but never read by WikiGenerator
- Fix inconsistent progress callback phase tracking in --review
  continuation path
- Add wiki CLI help test covering --provider, --review, --verbose flags

Made-with: Cursor
2026-03-25 11:39:37 +05:30
chirag-nighut fca7815c16 feat(wiki): add Cursor CLI as LLM provider option
Add Cursor headless CLI as a 4th provider option for wiki generation,
allowing users to leverage their Cursor subscription for wiki pages.

- Add --provider cursor flag and cursor-client.ts
- Add --review flag for interactive module tree editing
- Add --verbose flag for debugging
- Improve module tree generation (flatten single-child, unique slugs)
- Prevent DB timeout during long LLM calls

Usage: gitnexus wiki --provider cursor --model claude-4.5-opus-high
Made-with: Cursor
2026-03-25 11:39:37 +05:30
Yash a191c26571 fix(#480): resolve impact/context returning empty results for Java cl… (#489) 2026-03-24 21:46:46 +00:00
Gergő Magyar 7999b6ba7b refactor: SICP-informed LanguageProvider architecture (#488)
* refactor: SICP-informed LanguageProvider architecture for ingestion pipeline

Consolidate 16 scattered dispatch surfaces into a single LanguageProvider
Strategy interface per language. Processors are now fully language-agnostic —
zero SupportedLanguages.X enum access, zero dispatch table imports.

Architecture (5-layer DAG, zero circular dependencies):
  L0: Capability modules (dispatch tables, single source of truth)
  L1: LanguageProvider interface + createLanguageProvider factory
  L2: 13 per-language provider files (Strategy objects)
  L3: Registry with satisfies Record<SL, LP> + pre-built lookup maps
  L4: Processors (language-agnostic, all behavior via provider.*)

Key changes:
- Add LanguageProvider interface with 15 properties (6 required, 9 optional)
- Create 13 provider files in languages/ + php-helpers.ts
- Migrate all processors to getProvider(language) — cached once per scope
- Replace heritage if-checks with provider.interfaceNamePattern/heritageDefaultEdge
- Replace MRO switch(language) with switch(provider.mroStrategy)
- Replace isNodeExported with provider.exportChecker
- Move PHP description extraction behind provider.descriptionExtractor
- Move Swift implicit imports behind provider.implicitImportWirer
- Move PHP route detection behind provider.isRouteFile
- Move Kotlin wildcard append behind provider.importPathPreprocessor
- Remove deprecated TypeEnvironment.env, add fileScope()/allScopes()
- De-export TypeEnv type (module-private)
- Pre-build extensionMap, WILDCARD_LANGUAGES, SYNTHESIS_LANGUAGES at load
- Remove dead entryPointPatterns/frameworkPatterns from interface
- Derive createLanguageProvider config type via Pick/Partial/Omit
- Tighten callback types from any to SyntaxNode
- Migrate 270+ test call sites from .env to TypeEnvironment API

Adding a new language: 3 files (enum + provider + registry line).
No processor file touched. Ever.

* refactor: clean architecture for LanguageProvider with O(1) AST cache

Address all PR #488 review comments and achieve pristine SICP layer separation:

Interface redesign:
- Split LanguageProvider into Config (input) + Provider (runtime with defaults)
- Rename createLanguageProvider → defineLanguage with explicit DEFAULTS constant
- Add MroStrategy, ImportSemantics named type aliases for better IDE tooltips
- Tighten labelOverride signature: string|null → NodeLabel|null (compile-time safety)
- Tighten descriptionExtractor nodeLabel: string → NodeLabel
- Un-export LanguageProviderConfig (internal to defineLanguage)

CI fixes (all 4 failures resolved):
- isNodeExported: add null guard for unknown languages
- preprocessImportPath tests: pass getProvider() instead of raw enum
- MRO tests: update expected strings to match language-agnostic prefixes

Code deduplication:
- Extract findDescendant/extractStringContent to ast-helpers.ts (single source of truth)
- Unify Kotlin method detection: remove duplicate from extractFunctionName,
  use provider.labelOverride as single source of truth via findEnclosingFunctionId
- extractFunctionName return type: string → NodeLabel

Performance (O(1) AST node access):
- Add per-file Map-based memoization in parse-worker for parent-chain walks
- Cache enclosingClassId, enclosingFunctionId, exportStatus per SyntaxNode
- Clear caches before each file parse (not after — handles parse failures)

Architecture (pristine languages/ folder):
- Move php-helpers.ts → helpers/php.ts (L0 capability, not L2 config)
- Create helpers/swift.ts from extracted Swift provider logic
- Extract cppLabelOverride AST walk → isCppInsideClassOrStruct in ast-helpers.ts
- Extract isPhpRouteFile → helpers/php.ts
- All 13 provider files are now pure configuration — zero implementation logic
- Ruby: remove no-op namedBindingExtractor assignment (undefined from dispatch table)

* refactor: eliminate LANGUAGE_QUERIES, typeConfigs, namedBindingExtractors dispatch tables

Phase 1 of L0 dispatch table elimination. Providers now import capabilities
directly instead of indexing into redundant Record<SL, T> dispatch tables:

- LANGUAGE_QUERIES: providers import named query constants directly
  (TYPESCRIPT_QUERIES, PYTHON_QUERIES, etc.). Table kept in tree-sitter-queries.ts
  for call-processor.ts dynamic lookup + test consumers.

- typeConfigs: providers import from individual type-extractor files
  (typescriptConfig from typescript.ts, javaTypeConfig from jvm.ts, etc.).
  Dispatch table fully removed from type-extractors/index.ts.

- namedBindingExtractors: providers import extractors directly from
  named-binding-extraction.ts (extractTsNamedBindings, etc.).
  Dispatch table fully removed from import-resolution.ts.

Net: -48 LOC of dispatch table indirection. L3 satisfies Record<SL, LP>
remains the single exhaustiveness check.

* refactor: eliminate exportCheckers, callRouters, importResolvers dispatch tables

Phase 2 of L0 dispatch table elimination. All 6 dispatch tables are now gone:

- exportCheckers: individual checkers exported directly (tsExportChecker,
  pythonExportChecker, etc.). isNodeExported uses a local checkersByLanguage
  map to avoid circular dependency with languages/index.ts.

- callRouters: table removed. Providers import noRouting or routeRubyCall
  directly. noRouting now exported. Dead import removed from call-processor.ts.

- importResolvers: resolver functions exported with clean names
  (resolveTypescriptImport, resolveJavaImport, etc.). Inline lambdas
  extracted to named exports. Dispatch functions renamed from *Dispatch
  suffix to clean resolve*Import pattern.

Combined with Phase 1, all 6 L0 dispatch tables have been eliminated.
L3 satisfies Record<SL, LanguageProvider> is the single exhaustiveness check.
Providers are now fully self-contained — each imports its capabilities directly.

* perf+refactor: type-env caching, sequential fallback caching, utils.ts split

Phase 3 — performance optimizations and barrel cleanup:

Type-env parent-walk caching:
- Memoize findEnclosingClassName and findEnclosingParentClassName with
  per-file Map<SyntaxNode, string|undefined> caches
- Eliminates O(n*m) repeated child scanning in extractParentClassFromNode
- Caches cleared in buildTypeEnv before each file's walk phase

Sequential fallback caching:
- Add classIdCache + exportCache Maps to parsing-processor.ts
- Mirrors the O(1) memoization pattern from parse-worker.ts
- Both paths now have identical caching for parent-chain walks

Split utils.ts barrel into focused modules:
- noise-filter.ts: BUILT_IN_NAMES + isBuiltInOrNoise (167 LOC)
- language-detection.ts: getLanguageFromFilename (58 LOC)
- utils.ts slimmed to re-exports + yieldToEventLoop + isVerboseIngestionEnabled
- Backward compatible — existing imports from utils.ts still work

* refactor: rename resolvers/ → import-resolvers/, restructure tests per-concern

Directory renames (git mv — history preserved):
- src/core/ingestion/resolvers/ → import-resolvers/ (10 files)
- test/unit/call-routing.test.ts → call-routing/ruby.test.ts
- test/unit/named-binding-extraction.test.ts → named-bindings/csharp.test.ts
- test/unit/import-resolution.test.ts → import-resolution/preprocessing.test.ts

All 11 import paths updated to reference new import-resolvers/ location.
Test imports updated for new subdirectory depth.

Note: test/integration/resolvers/ NOT renamed — those tests cover the full
ingestion pipeline per-language, not just import resolution.

* refactor: eliminate utils.ts barrel — all 33 consumers now import directly

Migrated 65 import sites across 33 files to import from the focused source
module instead of the utils.ts barrel:

- ast-helpers.js: SyntaxNode, extractFunctionName, findEnclosingClassId, etc.
- call-analysis.js: inferCallForm, extractReceiverName, countCallArguments, etc.
- noise-filter.js: BUILT_IN_NAMES, isBuiltInOrNoise
- language-detection.js: getLanguageFromFilename

utils.ts reduced to 2 original functions only:
- yieldToEventLoop
- isVerboseIngestionEnabled

Zero re-exports remain. Every import is now direct to its source module.

* refactor: create utils/ folder, move all shared utilities, delete utils.ts barrel

Final phase of module structure migration:

- git mv ast-helpers.ts, call-analysis.ts, noise-filter.ts,
  language-detection.ts → utils/ subdirectory (history preserved)
- Extract yieldToEventLoop → utils/event-loop.ts
- Extract isVerboseIngestionEnabled → utils/verbose.ts
- Delete utils.ts (zero re-exports, zero functions remain)
- Update 38 import paths across source and test files

The ingestion/ root is now clean — only processors, capability modules,
and the pipeline orchestrator live at the top level. All shared utilities
are in utils/, all language-specific helpers in helpers/, all import
resolvers in import-resolvers/.

* refactor: move findChild from import-resolvers/utils.ts to utils/ast-helpers.ts

findChild is a generic AST helper (find first named child by type) — it
belongs with the other AST traversal utilities, not in the import resolver
module. 4 consumers updated to import from utils/ast-helpers.js.

* refactor: split named-binding-extraction.ts into per-language files

Rename named-binding-extraction.ts → named-binding-processor.ts (git mv,
history preserved), keeping only walkBindingChain for re-export chain resolution.

7 per-language extractor functions moved to named-bindings/ subdirectory:
- named-bindings/typescript.ts (extractTsNamedBindings — TS + JS)
- named-bindings/python.ts (extractPythonNamedBindings)
- named-bindings/kotlin.ts (extractKotlinNamedBindings)
- named-bindings/rust.ts (extractRustNamedBindings + collectRustBindings)
- named-bindings/php.ts (extractPhpNamedBindings)
- named-bindings/csharp.ts (extractCsharpNamedBindings)
- named-bindings/java.ts (extractJavaNamedBindings)

Each provider now imports its binding extractor from the per-language file.

* refactor: eliminate import-resolution.ts — distribute to natural homes

Split per-language resolvers into import-resolvers/ per-language files and
eliminate the import-resolution.ts catch-all module entirely:

Per-language resolvers moved to import-resolvers/:
- standard.ts: resolveStandard, resolveJavascriptImport, resolveTypescriptImport,
  resolveCImport, resolveCppImport
- jvm.ts: resolveJavaImport, resolveKotlinImport
- go.ts: resolveGoImport
- csharp.ts: resolveCSharpImport (helper renamed to Internal)
- php.ts, python.ts, ruby.ts, rust.ts: same pattern
- swift.ts: new file for resolveSwiftImport

Types distributed to their concern directories:
- import-resolvers/types.ts: ImportResult, ImportConfigs, ResolveCtx, ImportResolverFn
- named-bindings/types.ts: NamedBinding, NamedBindingExtractorFn

preprocessImportPath moved to import-processor.ts (its primary consumer).

import-resolution.ts deleted — zero catch-all modules remain.

* refactor: tighten SPR — eliminate re-exports, dead code, type holes, and redundant patterns

12 review findings resolved across the ingestion layer:

Type safety:
- CallRouter callNode: any → SyntaxNode (closes type hole)
- CaptureMap type alias replaces Record<string, any>
- providersWithImplicitWiring filter now type-narrowed (removes ! assertions)
- Ruby exportChecker: unnecessary as-cast removed, named export created

Architecture:
- Circular type dependency eliminated (ImportResolutionContext moved to types.ts)
- LANGUAGE_QUERIES residual dispatch replaced with provider.treeSitterQueries
- noRouting sentinel deleted — callRouter now properly optional on 12 providers
- All 6 re-exports from import-processor/pipeline/languages eliminated

Pattern cleanup:
- Dead checkersByLanguage table + isNodeExported removed from export-detection
- 4 duplicated config interfaces consolidated to language-config.ts
- extractCsharpNamedBindings → extractCSharpNamedBindings (casing consistency)

Simplification:
- import-resolvers/index.ts barrel deleted (dead re-exports)
- helpers/ inlined into languages/ (php.ts, swift.ts) — 1 directory removed

Verified: tsc --noEmit clean, 3837 tests pass, 0 failures.

* refactor: address review — remove LANGUAGE_QUERIES table, type-extractors barrel, fix Windows timeout

Review comment fixes (github.com/abhigyanpatwari/GitNexus/pull/488#issuecomment-4117817648):

1. LANGUAGE_QUERIES dispatch table removed from tree-sitter-queries.ts
   — 5 test files migrated to getProvider(lang).treeSitterQueries
   — eliminates last parallel dispatch surface

2. type-extractors/index.ts barrel deleted
   — type-env.ts now imports TYPED_PARAMETER_TYPES from shared.js directly

3. Windows CI timeout fix: afterAll cleanup hook in test-indexed-db.ts
   now passes explicit 120s timeout to prevent KuzuDB C++ destructor
   hang from hitting vitest's default 30s testTimeout on Windows

Verified: tsc --noEmit clean, 3835 tests pass, 0 failures.

* refactor: eliminate chained getProvider property access — assign to variable first

All getProvider(lang).property calls now follow the pattern:
  const provider = getProvider(language);
  const x = provider.property;

5 source files + 4 test files updated (~35 occurrences).
This ensures consistent provider variable usage and avoids
repeated lookups in hot paths.

* refactor: remove last 4 re-exports from import-resolvers, fix stale CaptureMap comment

- Remove `export type { TsconfigPaths }` from standard.ts
- Remove `export type { GoModuleConfig }` from go.ts
- Remove `export type { ComposerConfig }` from php.ts
- Remove `export type { CSharpProjectConfig }` from csharp.ts
  All 4 types are canonically defined in language-config.ts;
  zero consumers imported via the resolver re-exports.

- Fix stale CaptureMap JSDoc: said "Uses any" but type is SyntaxNode | undefined
2026-03-24 13:42:39 +00:00
John R. Eakin f0540b33fb ci: E2E workflow, web typecheck job, pre-commit hook, test suite (#486) 2026-03-24 06:01:08 +00:00
harryphung fec47cbc32 feat(web): add GLM (Z.AI) as LLM provider (#468)
Add GLM support using OpenAI-compatible API via ChatOpenAI from LangChain.
Defaults to the Z.AI coding endpoint (https://api.z.ai/api/coding/paas/v4)
with configurable base URL. Supported models: GLM-5, GLM-5-Turbo, GLM-4.7, GLM-4.5.
2026-03-24 06:10:37 +05:30
B. Lawrence Sharma a9d43d5680 fix(ui): use actual repository name instead of top-level folder for GitHub clones (#491) 2026-03-24 05:54:38 +05:30
Zander Raycraft e6aea1c82b Merge pull request #490 from zander-raycraft/main
Updating claude mcp init for window and adding OS detection when installing
2026-03-23 17:47:03 -05:00
Zander Raycraft 3961ad01cf updating windows support 2026-03-23 17:33:32 -05:00
marxo126 c437acf6bb feat: deep flow detection — consumer access tracking, middleware chains, error shapes, api_impact tool (#482) 2026-03-23 22:13:48 +00:00
Gergő Magyar 2ede01dbff Merge pull request #477 from jreakin/perf/web-o1-lookups-memo-bundle
perf(web): O(1) lookups, memoization, bundle optimizations, React fixes
2026-03-23 17:01:09 +00:00
jreakinandClaude Opus 4.6 fd61cd2990 fix(web): allow foreignObject in DOMPurify SVG, remove leftover loop code
- Add ADD_TAGS: ['foreignObject'] to all DOMPurify.sanitize calls —
  Mermaid uses foreignObject for HTML text labels inside flowchart
  nodes. The SVG profile was stripping them, causing empty boxes.
- Remove leftover sub-batch loop lines from prepared statement hoist

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:34 -05:00
jreakinandClaude Opus 4.6 83ec256cec fix(web): address React rendering review — stale closure, ref dep, O(1) lookups
- MarkdownRenderer: wrap handleLinkClick in useCallback, add to
  markdownComponents useMemo deps (fixes stale closure)
- GraphCanvas: remove sigmaRef from useEffect deps (ref identity
  never changes), extract handleToggleAIHighlights to useCallback
- CodeReferencesPanel: add nodeById Map for O(1) focus-in-graph
  lookup (was O(N) graph.nodes.find on every click)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:34 -05:00
jreakinandClaude Opus 4.6 29795bb86e fix(web): switch lucide-icons from deep ESM paths to standard re-exports
Deep imports from lucide-react/dist/esm/icons/*.js are internal paths
that broke the Vercel production build. Replaced with standard named
re-exports from lucide-react — keeps the centralized module pattern
without relying on fragile internal paths.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:34 -05:00
jreakinandClaude Opus 4.6 c8cd733815 perf(web): prepare statement once per label pair, not per sub-batch
Hoists conn.prepare(cypher) outside the sub-batch loop so it's called
once per (fromLabel, toLabel) pair instead of ceil(N/4) times. The
statement is reused for all rows in the group, then closed in finally.

Yields to event loop every 500 relations instead of every sub-batch.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:33 -05:00
jreakinandClaude Opus 4.6 5616f71fd2 perf(web): switch remaining components to deep lucide imports, extract ProviderConfigCard
- BackendRepoSelector, EmbeddingStatus, Header, MermaidDiagram,
  QueryFAB, RightPanel, ToolCallCard, WebGPUFallbackDialog: switch
  from lucide-react barrel imports to @/lib/lucide-icons deep imports
- MermaidDiagram: lazy-load ProcessFlowModal via React.lazy
- Extract ProviderConfigCard from SettingsPanel for cleaner separation

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:33 -05:00
jreakinandClaude Opus 4.6 2bd04fe2c4 test: add positive/negative tests for O(1) lookup optimizations
8 tests verifying the data structures underlying performance changes:

nodeById Map (O(1) lookup):
  + Map.get returns correct node by ID
  + duplicate IDs: last wins
  - non-existent ID returns undefined
  - empty Map returns undefined

Set.has (O(1) highlight matching):
  + present IDs return true
  - absent IDs return false
  + handles IDs with colons, dots, slashes
  - case-sensitive matching

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:33 -05:00
jreakinandClaude Opus 4.6 4199c5e4f3 perf(web): O(1) lookups, memoization, bundle optimizations, React fixes
Performance:
- nodeById Map in GraphCanvas for O(1) click/hover lookups
- fileNodeByPath Map in useAppState for O(1) file path lookups
- Set.has() for HIGHLIGHT_NODES/IMPACT matching (was O(N²))
- useMemo for primaryLanguage in StatusBar
- useCallback on toggleLabelVisibility/toggleEdgeVisibility
- Recursive FileTreePanel search (full subtree, not 1 level)

React fixes:
- Remove stale queryResult dep from clearAICodeReferences
- Cancel RAF chains in CodeReferencesPanel on cleanup
- Clean up timeouts in SettingsPanel and MarkdownRenderer on unmount
- try/catch on localStorage in DropZone (private browsing)
- Await handleServerConnect in App.tsx auto-connect
- pendingToolCalls counter replaces allToolsDone boolean in agent.ts
- JSON.parse try/catch in agent streaming
- sessionStorage JSDoc fix in settings-service

Bundle:
- Centralized lucide icon deep imports

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:33 -05:00
jreakinandClaude Opus 4.6 0d9a2ca6b6 fix(web): allow foreignObject in DOMPurify SVG sanitization
Mermaid uses foreignObject for HTML text labels inside flowchart
nodes. The SVG profile strips them by default, causing empty boxes.
ADD_TAGS: ['foreignObject'] preserves text while still sanitizing
against XSS.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 10:16:33 -05:00
Gergő Magyar 88fa1a4e09 Merge pull request #476 from jreakin/fix/web-csv-schema-correctness
fix(web): 18 multi-language CSV tables, RFC 4180, schema sync
2026-03-23 14:12:28 +00:00
jreakinandClaude Opus 4.6 8956da61bb fix(web): add 15 missing NodeLabel colors/sizes, escape table names, schema sync
- Add NODE_COLORS and NODE_SIZES entries for all 15 multi-language labels
  (Struct, Trait, Impl, TypeAlias, Const, Static, Namespace, Union,
  Typedef, Macro, Property, Record, Delegate, Annotation, Constructor,
  Template) — fixes Record<NodeLabel, ...> completeness
- Escape table names in count/lookup queries with escapeTableName() to
  prevent silent failures for backtick-required tables
- Add HAS_PROPERTY and ACCESSES to RelationshipType union
- Update initPromise after db recreation in loadGraphToLbug so subsequent
  initLbug() calls return fresh db/conn refs

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 09:03:40 -05:00
Abhigyan PatwariandClaude Opus 4.6 9ba2b22ac6 fix(web): resolve Vercel build errors from security hardening PR (#483)
- ProcessFlowModal: revert import from non-existent @/lib/lucide-icons back to lucide-react
- embedding-pipeline: remove extra argument in executeQuery call (signature only accepts 1 arg)

Both issues were introduced in PR #475 (web-security-hardening).

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 19:31:08 +05:30
Gergő Magyar db6b302fee Merge pull request #408 from marxo126/fix/swift-query-and-patch-script
feat: complete Swift support — query fix, export detection, implicit imports, constructor resolution
2026-03-23 13:53:25 +00:00
marxo126andClaude Opus 4.6 fcd2c3ff38 docs: add macro declarations to Swift ingestion gaps
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:21:32 +01:00
jreakinandClaude Opus 4.6 f4bb78c03a test: add negative tests for CSV generation
- empty graph produces header-only CSVs (no data rows)
- empty graph relCSV has only header
- double quotes in node names are RFC 4180 escaped
- file node without fileContents gets empty content (no crash)
- community with empty keywords array produces valid CSV
- unknown node labels are silently skipped

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:21:31 -05:00
jreakinandClaude Opus 4.6 47f9b6c48d fix: add multi-language NodeLabel types, close old db/conn, CSV tests
- Add 15 multi-language labels to NodeLabel union (Struct, Trait, Impl,
  TypeAlias, Const, Static, Namespace, Union, Typedef, Macro, Property,
  Record, Delegate, Annotation, Constructor, Template) — eliminates
  unsafe casts in csv-generator
- Close previous conn/db before recreating in loadGraphToLbug to prevent
  WASM resource leaks across repo switches
- Add 7 CSV generation tests: multi-language tables, column count,
  keyword comma escaping, file content, relation CSV, all NODE_TABLES

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:21:31 -05:00
jreakinandClaude Opus 4.6 433f403fe2 fix(web): 18 multi-language CSV tables, RFC 4180 escaping, schema sync
- Generate CSVs for Struct, Enum, Macro, Trait, Impl, TypeAlias, Const,
  Static, Property, Record, Delegate, Annotation, Constructor, Template,
  Module, Namespace, Union, Typedef (were defined in schema but silently
  dropped during CSV generation — non-JS/TS repos lost all nodes)
- Escape community keywords array in CSV to prevent comma breakage
- Add Section to NodeLabel type, NODE_COLORS, NODE_SIZES
- Add 7 missing REL_TYPES: HAS_METHOD, HAS_PROPERTY, OVERRIDES, ACCESSES,
  INHERITS, USES, DECORATES
- RFC 4180 doubled-quote regex for relation CSV parsing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 08:21:31 -05:00
Gergő Magyar 519dc3eccc Merge pull request #475 from jreakin/fix/web-security-hardening
fix(web): Cypher injection guards, DOMPurify SVG, readOnly executeQuery
2026-03-23 13:21:03 +00:00
marxo126andClaude Opus 4.6 fcf8fb9bdf docs: add Swift ingestion gaps tracker and update feature matrix
- Create swift-ingestion-gaps.md with prioritized gap tracker (High/Medium/Low)
- Update type-resolution-system.md feature matrix: 5 Swift entries corrected
  (for-loop→Yes, pattern binding→Partial, call-result/field/method→Yes)
- Add footnotes explaining Swift-specific semantics
- Document resolved items with commit references

Addresses @magyargergo's request to document missing Swift features.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 14:16:43 +01:00
marxo126 47fdad14ed Merge remote-tracking branch 'upstream/main' into fix/swift-query-and-patch-script
# Conflicts:
#	gitnexus/src/core/ingestion/call-processor.ts
2026-03-23 12:21:53 +01:00
Gergo Magyar ffabe857a3 fix(docs): update symbol and relationship counts in AGENTS.md and CLAUDE.md 2026-03-23 11:15:56 +00:00
marxo126andClaude Opus 4.6 1f4c4e77ab refactor: simplify Swift support code after review
- Move `pattern` node handling into shared extractVarName (like mut_pattern)
  instead of inline fallback in type-env — benefits all callers
- Remove non-null assertion (!) on firstNamedChild — defensive null check
- Avoid 100K wrapper object allocation: addSwiftImplicitImports now accepts
  string[] directly, eliminating allFileList.map(p => ({ path: p }))
- Remove duplicate comment block in for-loop test

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 11:44:11 +01:00
marxo126andClaude Opus 4.6 956dfd0bb4 feat: add Swift integration tests for if-let, await/try, for-loop + fix cross-chunk imports
- Add 3 new test fixtures: swift-if-let-guard-let, swift-await-try, swift-for-loop-inference
- Add integration tests for if let/guard let binding resolution (4 assertions)
- Add integration tests for await/try expression unwrapping (3 assertions)
- Add for-loop-inference fixture (documented as known gap — type-env infrastructure
  is in place but call-processor re-parse path doesn't propagate the binding yet)
- Fix cross-chunk Swift implicit imports: standard processImports path now passes
  allFileList instead of chunk-only files to addSwiftImplicitImports, matching
  the fast-path behavior
- Add Swift type_annotation fallback in type-env declarationTypeNodes population
  (handles [User] array sugar where childForFieldName('type') returns null)
- Handle Swift 'pattern' node in extractVarName fallback (pattern wraps simple_identifier)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 11:40:24 +01:00
jreakinandClaude Opus 4.6 57e087de49 fix(web): strip quoted strings before readOnly keyword check
The readOnly guard was matching keywords inside string literals,
blocking legitimate queries like WHERE n.name CONTAINS "delete".
Now strips single/double-quoted strings before checking, so only
actual Cypher write keywords outside strings are blocked.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 05:09:44 -05:00
jreakinandClaude Opus 4.6 fc6c114570 fix(web): MermaidDiagram XSS, isSafeId allows @, rawMermaid typed
- Add DOMPurify.sanitize() to MermaidDiagram.tsx dangerouslySetInnerHTML
  (was rendering AI-generated SVG unsanitized — the highest-risk surface)
- Add @ to isSafeId regex for scoped npm packages (@angular/core, etc.)
- Add rawMermaid to ProcessData interface, remove (process as any) casts
- Update security tests: @scope/pkg now accepted, @angular/core added

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 05:01:57 -05:00
Gergo Magyar 43d866c802 chore: bump version to 1.4.8, update CHANGELOG.md 2026-03-23 09:53:43 +00:00
Gergő Magyar 6b0c566392 Merge pull request #461 from ShunsukeHayashi/fix/python-import-alias-417
fix(ingestion): resolve Python import-alias CALLS edges
2026-03-23 09:46:39 +00:00
jreakinandClaude Opus 4.6 96181fba05 fix: pass readOnly=false for CREATE_VECTOR_INDEX in embedding pipeline
The readOnly=true default on executeQuery blocks queries containing
CREATE, which includes CALL CREATE_VECTOR_INDEX. The embedding pipeline
needs write access for this setup step.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 04:45:09 -05:00
jreakinandClaude Opus 4.6 208ade9b11 fix: add / back to isSafeId — node IDs contain file paths
Node IDs are generated as Label:filePath (e.g., Function:src/foo.ts:bar),
so forward slashes are expected in legitimate IDs. The over-tightened
regex was dropping all path-based IDs from Cypher queries.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 04:45:09 -05:00
jreakinandClaude Opus 4.6 9f14edc226 fix(web): security hardening — Cypher injection, XSS, readOnly, prepared statements
- DOMPurify.sanitize() on mermaid SVG in ProcessFlowModal
- validLabel()/validRelType() guards on all Cypher interpolation in tools.ts
- isSafeId() strict regex in ProcessesPanel (no spaces/slashes/metacharacters)
- executeQuery defaults readOnly=true (rejects CREATE/DELETE/DROP/etc.)
- Singleton promise for initLbug (prevents concurrent init races)
- Prepared statements for relation inserts (batched by label pair)
- executeWithReusedStatement for enrichment updates
- try/finally on all PreparedStatement cleanup
- 99 security guard tests (validLabel, validRelType, isSafeId, readOnly regex)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 04:45:09 -05:00
Gergo Magyar cb1293b718 fix(python): route module aliases directly to moduleAliasMap in import processor
`import models as m` aliases were stored in namedImportMap (a symbol-binding
map) then cross-referenced in pipeline.ts — semantic misuse and inefficient.

Refactored: NamedBinding gains `isModuleAlias` flag. applyImportResult routes
tagged bindings directly to moduleAliasMap at import time. Removes the
pipeline.ts post-processing loop entirely.

Added test fixture and 5 integration tests for `import X as Y` with
multi-module disambiguation (both models.py and auth.py export User).
2026-03-23 09:36:33 +00:00
Gergő Magyar 6c9b6eb0f6 Merge pull request #474 from jreakin/fix/web-lbug-server-highlights
fix(web): LadybugDB getAllRows, loadServerGraph, BM25, highlight clearing
2026-03-23 09:36:27 +00:00
jreakinandClaude Opus 4.6 3558cb8a7f chore: simplify prepare script, remove scripts/prepare.cjs
.husky/pre-commit is committed to the repo — developers get the hook
by cloning, not by running npm install. prepare only needs to build
TypeScript for npm publish/pack.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 04:18:02 -05:00
Shunsuke Hayashi 9f2d1780d5 fix(ingestion): resolve Python import-alias CALLS edges (#417)
`import numpy as np` and `from models import User as U` previously
generated no CALLS edges because:

1. `import_statement` with an `aliased_import` child was not captured
   by the tree-sitter query for Python imports.
2. `extractPythonNamedBindings()` only handled `import_from_statement`,
   ignoring plain `import X as Y` forms.

Changes:
- `tree-sitter-queries.ts`: add query pattern for
  `(import_statement name: (aliased_import name: (dotted_name)))` so
  the import path is captured before named-binding extraction runs.
- `named-binding-extraction.ts`: extend `extractPythonNamedBindings()`
  to handle `import_statement` nodes carrying `aliased_import` children.
  Records `{ local: "np", exported: "numpy" }` so call-sites using the
  alias resolve to the real module.
- `test/fixtures/lang-resolution/python-alias-imports/`: update fixtures
  used by `python.test.ts` to exercise `from models import User as U`.

Existing tests in `test/integration/resolvers/python.test.ts`
(suite "Python alias import resolution") cover this path.
2026-03-23 09:17:43 +00:00
Gergő Magyar a57550815f Merge pull request #463 from ShunsukeHayashi/fix/python-calls-zero-337
fix(python): resolve module-qualified constructor calls — 0 CALLS edges (Issue #337)
2026-03-23 08:59:21 +00:00
jreakinandClaude Opus 4.6 ee78ebe64a test: add positive and negative tests for all 4 bug fixes
16 tests covering the data structures and logic underlying each fix:

createKnowledgeGraph (loadServerGraph data flow):
  + nodes stored correctly via addNode
  + relationships stored correctly via addRelationship
  + deduplication by ID
  + nodeCount reflects unique count
  - empty graph has zero counts
  - relationships with non-existent nodes still stored

loadServerGraph data flow:
  + server data reconstructs into valid KnowledgeGraph
  + fileContents Map built from server entries
  - empty server data produces empty graph
  - fileContents replaces (not accumulates) on reload

BM25 index argument type:
  + Map<string, string> has entries() for BM25
  - KnowledgeGraph does NOT have entries() (the original bug)

Highlight clearing:
  + clearing Set produces empty set
  + independent highlight sources cleared separately
  - clearing highlights doesn't affect node selection
  - toggling AI ON doesn't clear process highlights

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 03:48:27 -05:00
Gergo Magyar 141f864181 refactor: use boolean[] for O(1) per-chunk synthesis guard
Replace Set<number> + flatMap with a simple boolean array indexed by
chunk — cleaner data structure for sequential integer keys.
2026-03-23 08:47:21 +00:00
jreakinandClaude Opus 4.6 01fa5bf98e fix: address review — stale progress, cross-platform prepare, DEV log
- Clear progress after handleServerConnect in both auto-connect and
  DropZone paths (fixes frozen progress bar on StatusBar)
- Replace shell-based prepare script with Node scripts/prepare.cjs
  for Windows cmd.exe compatibility
- Gate LadybugDB load warning behind import.meta.env.DEV (consistent
  with finalizePipeline's silent catch)
- Remove misleading "parallel" comment (fetch is sequential after connect)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 03:26:32 -05:00
Gergo Magyar bbd95457df test(python): strengthen module-import tests, un-skip match/case, add perf guard
- Rewrite Issue #337 test suite: 5 tests → 19 tests with exact node/edge
  counts, sourceFilePath guards, negative tests, method call disambiguation
  (u.save(), a.login(), v.verify()), HAS_METHOD verification, and
  cross-module collision assertions
- Un-skip 2 match/case as-pattern tests (they pass now) and remove leftover
  DEBUG test
- Add per-chunk language guard for synthesizeWildcardImportBindings — skips
  full graph traversal for TS/JS-only chunks, avoiding O(chunks × graph_size)
- Rename fixture method check → verify to avoid BUILT_IN_NAMES noise filter
- Expand fixture with method calls on constructor-inferred receivers
- Fix stale comment referencing only "Go package imports"
2026-03-23 08:25:46 +00:00
jreakinandClaude Opus 4.6 d43cc691f0 chore: switch from .githooks to husky for pre-commit hooks
Per maintainer request. Husky is activated via `cd .. && husky` in the
prepare script. The pre-commit hook mirrors CI: typecheck + unit tests
for both packages when relevant files are staged.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 03:18:04 -05:00
Shunsuke HayashiandClaude Sonnet 4.6 f90aabf9a8 fix(python): resolve module-qualified calls via moduleAliasMap
Previously, Python was added to WILDCARD_IMPORT_LANGUAGES which expanded
all exported symbols into namedImportMap using first-seen wins. This caused
`auth.User()` to incorrectly resolve to `models.py:User` when both modules
exported a class named User.

Root cause: Python `import models` is a namespace import, not wildcard
symbol expansion. Expanding all symbols produces ambiguous bindings that
cannot be disambiguated later.

Fix:
- Remove Python from WILDCARD_IMPORT_LANGUAGES
- Add ModuleAliasMap (callerFile → alias → sourceFile) to ResolutionContext
- In synthesizeWildcardImportBindings, build moduleAliasMap for Python
  using the filename stem as the module alias
- In resolveCallTarget, add module-alias disambiguation step: when multiple
  candidates survive filtering and the receiver name matches a module alias,
  narrow candidates to the aliased file

Result: `models.User()` → models.py:User, `auth.User()` → auth.py:User
even when both modules export a class named User.

Adds regression test for the ambiguity case.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 16:33:06 +09:00
jreakinandClaude Opus 4.6 beb5574d38 chore: add pre-commit hook for typecheck + unit tests
Adds .githooks/pre-commit mirroring CI checks:
- gitnexus-web/: tsc -b --noEmit + vitest run (if web files staged)
- gitnexus/: tsc --noEmit + vitest run --project default (if CLI files staged)

Activated via git config core.hooksPath in the prepare script.
Skip with git commit --no-verify.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 02:31:02 -05:00
jreakinandClaude Opus 4.6 d48903cce6 fix(web): LadybugDB getAllRows, loadServerGraph, BM25, highlight clearing
- Replace result.getAll() with result.getAllRows() across lbug-adapter
  (getAll doesn't exist in LadybugDB WASM v0.15.1)
- Add loadServerGraph worker method that pipes server data through
  initLbug/loadGraphToLbug for in-browser querying
- Extract finalizePipeline helper (shared by runPipeline, runPipelineFromFiles)
- Fix buildBM25Index called with graph object instead of fileContents Map
- Fix 'Turn off all highlights' to clear sigma selection, AI tool/citation
  highlights, and blast radius

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-23 02:25:59 -05:00
adonis 1bb80c3858 feat(cli): enhance index command with single path validation and add to .gitignore 2026-03-23 00:19:26 -03:00
Zander Raycraft a3fac2f672 Merge pull request #395 from zm2231/feat/http-embedding-backend 2026-03-22 21:47:53 -05:00
adonis 6fe450d206 test(cli): refactor indexCommand tests for consistency and add new cases 2026-03-22 22:05:22 -03:00
zm2231 9954f6fdfd fix: timeout detection, always-on dim validation, test hardening
- Fix timeout detection: AbortSignal.timeout() throws TimeoutError, not
  AbortError. Timeouts are no longer retried (30s fail, not 93s).
- Validate embedding dimensions in both httpEmbed and httpEmbedQuery
  against config.dimensions or the 384d schema default. When DIMS is
  unset, the error says 'Set GITNEXUS_EMBEDDING_DIMS=N' to guide users.
- Centralize test env var cleanup in afterEach via savedEnv snapshot.
- Test mocks use 384d vectors matching schema default.
- 4 new tests: timeout not retried, network retry success, query path
  dim mismatch, unset-dims hint. 23 total, all pass.
2026-03-22 20:21:08 -04:00
B. Lawrence Sharma 90ca43f851 feat(ui): ship HelpPanel — because great tools don't make you google how to use them (#465)
* remove friction in onboarding by correcting typo

* Revert "remove friction in onboarding by correcting typo"

This reverts commit 07dec38c2c.

* feat(ui): add HelpPanel component with tabbed reference, node legend, AI query guide, and dual Mac/Windows keyboard shortcuts

* feat(ui): add HelpPanel component with tabbed reference, node legend, AI query guide, and dual Mac/Windows keyboard shortcuts

* made changes based on the suggestions

* minor fix
2026-03-23 03:25:01 +05:30
Zander Raycraft 0a200a51cb Merge pull request #400 from JayceeB1/fix/server-mode-worker-hydration
closes issue #398
2026-03-22 14:28:05 -05:00
JayceeB1 c87cb81892 fix: await server connect hydration flow 2026-03-22 19:54:37 +01:00
marxo126andClaude Opus 4.6 884b4acf84 fix: regenerate package-lock.json for CI compatibility
npm ci was failing with "Missing: hono@4.12.8" and
"Missing: graphology-types@0.24.8" because the lock file was
out of sync after rebase. Regenerated from clean state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 18:59:55 +01:00
Zander Raycraft b8ab23a838 Merge pull request #464 from zander-raycraft/main
update method for labeling prs
2026-03-22 12:06:32 -05:00
Zander Raycraft e70a6d80c0 Merge pull request #10 from zander-raycraft/gh/issue-pr-filter
redoing auto labeling using z score instead of clustering lowkey the …
2026-03-22 12:05:35 -05:00
Shunsuke HayashiandClaude Sonnet 4.6 cd1c0ff7dc fix(python): resolve module-qualified constructor calls (Issue #337)
Python repos were producing 0 CALLS edges for module-qualified constructor
calls like `models.User()` where `import models` is a bare module import.

Root causes:
1. `SupportedLanguages.Python` was absent from `WILDCARD_IMPORT_LANGUAGES`,
   so `synthesizeWildcardImportBindings` never ran for Python files — bare
   module imports never received per-symbol namedImportMap bindings.

2. Synthesis only ran in the Phase 14 pre-pass, after all chunks had already
   been call-resolved. When `models.User()` was processed in Phase 3+4,
   `namedImportMap` was empty for Python → Tier 2a-named fell through to
   Tier 2a which found both `models.py:User` and `auth.py:User` (ambiguous).

3. `filterCallableCandidates` with `callForm='member'` excluded `Class` nodes
   (only `CALLABLE_SYMBOL_TYPES` = Function/Method/Constructor/…). With 2
   ambiguous Class candidates both were dropped, producing 0 CALLS edges.

Fixes:
- Add `SupportedLanguages.Python` to `WILDCARD_IMPORT_LANGUAGES` so that
  `import models` expands to per-symbol namedImportMap entries (first-seen
  semantics: `User→models.py:User`, `Admin→auth.py:Admin`).

- Call `synthesizeWildcardImportBindings` inline in the chunk loop, after
  `processImportsFromExtracted` but BEFORE `processCallsFromExtracted`. This
  ensures Tier 2a-named can disambiguate `module.ClassName()` at initial
  call-resolution time. The Phase 14 pre-pass remains as a final safety net.

- Add a fallback in `resolveCallTarget`: if `callForm='member'` yields 0
  filtered candidates, retry with `callForm='constructor'`. This handles the
  case where a module-qualified class instantiation (e.g. `models.User()`)
  is syntactically an attribute-access call but semantically a constructor
  call. The fallback only triggers for 0-candidate member calls, so it
  cannot over-eagerly promote normal member calls.

Tests: add `python-module-import` fixture (models.py/auth.py/app.py) with
4 regression tests covering IMPORTS edges, name-collision disambiguation
for `models.User()`, and `auth.Admin()`.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-03-23 00:26:00 +09:00
marxo126andClaude Opus 4.6 babf0f90d3 fix: pin tree-sitter versions and add npm overrides
Pin exact versions (no ^) to prevent surprise upgrades:
- tree-sitter: "0.22.4" (was "^0.22.4")
- tree-sitter-swift: "0.7.1" (was "^0.7.1")

Add npm overrides to suppress peer dependency warnings from grammar
packages that declare ^0.21.x but work fine with 0.22.4.

Note: tree-sitter-swift 0.6.0 fails to build on current Node (needs
node-gyp + Swift toolchain). 0.7.1 with prebuilt binaries is required
for Swift support to work at all.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:21 +01:00
marxo126andClaude Opus 4.6 0a3cdce00e fix: address Copilot review — private(set) export, for-loop tuple pattern
1. Export detection: exclude private(set)/fileprivate(set) from
   unexported check. Only the setter is restricted — the symbol
   itself is still readable cross-file.

2. For-loop binding: use extractVarName() instead of raw .text
   to avoid polluting scopeEnv with non-identifier keys from
   tuple destructuring patterns (e.g. `for (a, b) in ...`).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:09 +01:00
marxo126andClaude Opus 4.6 16b1a63134 feat: 7 Swift features — if/guard let, await/try, for-in, enum cases, self/super, optional chaining, multi-inheritance
Covers all high and medium impact gaps from the Swift feature coverage
analysis:

1. if let / guard let bindings: add if_statement and guard_statement
   to DECLARATION_NODE_TYPES, extract varName and value for Tier 2
   return-type propagation (callResult, copy, fieldAccess, methodCallResult)

2. await / try expression unwrapping: add unwrapSwiftExpression() that
   strips await_expression and try_expression wrappers before checking
   for call_expression. Applied in extractPendingAssignment,
   extractInitializer, and scanConstructorBinding.

3. for item in collection: add extractForLoopBinding for Swift with
   extractSwiftElementTypeFromTypeNode that handles [User] array sugar
   and Array<User> generic types. Registered in typeConfig.

4. Multiple inheritance specifiers: already working — tree-sitter
   queries match all inheritance_specifier occurrences automatically.
   Verified, no code changes needed.

5. Enum case extraction: add (enum_entry (simple_identifier) @name)
   @definition.property query to SWIFT_QUERIES.

6. self/super resolution: unskipped both describe.skip test suites
   (tree-sitter-swift 0.7.1 ships prebuilds, Node 22 build issue
   resolved). Both pass — 5 previously-skipped tests now running.

7. Optional chaining obj?.method(): already working — tree-sitter-swift
   parses the ? transparently. Verified, no code changes needed.

Tests: 3,603 → 3,608 (5 unskipped self/super tests)
Swift tests: 23 → 28 passing

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:09 +01:00
marxo126andClaude Opus 4.6 99f0aaaea5 test: add integration tests for Swift implicit imports, extension dedup, constructor fallback, export visibility
Addresses reviewer feedback: the new Swift behaviors (implicit imports,
constructor fallback, extension dedup, export detection) had no dedicated
integration tests. Adds 4 fixture directories and 11 new test assertions:

1. swift-implicit-imports: two files, no explicit import, cross-file
   constructor + member call resolves via addSwiftImplicitImports
2. swift-extension-dedup: extension creates duplicate Class node,
   constructor still resolves to primary definition
3. swift-constructor-fallback: ClassName() without `new` resolves as
   constructor via free→constructor retry
4. swift-export-visibility: internal symbols visible cross-file,
   public/open visible, private/fileprivate noted as Tier 3 limitation

All 3,603 tests pass (11 new, 0 regressions).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:09 +01:00
marxo126andClaude Opus 4.6 53b576776a feat: add extractPendingAssignment for Swift return-type inference
Swift was missing the extractPendingAssignment extractor, which meant
return-type-based variable bindings like `let user = getUser()` couldn't
propagate the return type of `getUser()` to `user`. This broke member
call resolution: `user.save()` couldn't resolve to `User.save()` when
there were competing methods (both User and Repo have save()).

Handles four Swift patterns:
- let user = getUser()        → callResult (Tier 2 propagation)
- let result = user.save()    → methodCallResult
- let name = user.name        → fieldAccess
- let copy = user             → copy

All 3,592 tests pass — including the 2 previously-failing Swift
return-type inference tests.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:09 +01:00
marxo126andClaude Opus 4.6 712598a055 fix: update Swift queries and tests for tree-sitter-swift 0.7.1
- Fix assignment query: tree-sitter-swift 0.7.1 uses named fields
  (target:/result:/suffix:) instead of positional children
- Update export detection tests: Swift `internal` (default) is now
  correctly treated as exported (module-scoped visibility)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:09 +01:00
marxo126andClaude Opus 4.6 90c9153ccc feat: upgrade tree-sitter to 0.22.4 and tree-sitter-swift to 0.7.1
Upgrades:
- tree-sitter: ^0.21.0 → ^0.22.4
- tree-sitter-swift: ^0.6.0 → ^0.7.1

Benefits:
- tree-sitter-swift@0.7.1 ships prebuilds (no manual node-gyp needed)
- Fixes parsing of Swift 5.9+ features: #Predicate, typed throws, ~Copyable
- All 13 language parsers verified compatible with tree-sitter@0.22.4

Tested:
- All 141 query/import/call tests pass
- All 13 parsers (C, C++, C#, Go, Java, JS, Kotlin, PHP, Python, Ruby,
  Rust, TypeScript, Swift) parse correctly with 0.22.4
- PricePal (61-file iOS 26 project) indexes fully: 3,094 nodes, 10,459 edges

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:47:09 +01:00
marxo126andClaude Opus 4.6 dfe83f333e fix: deduplicate Swift extension class nodes in call resolution
When Swift extensions create multiple Class nodes with the same name
(e.g. Product.swift + ProductMatchableConformance.swift), the call
resolver gets multiple candidates and refuses to emit a CALLS edge.

Add dedup: when all candidates share the same type (Class/Struct) and
differ only by file, prefer the primary definition (shortest filepath).

Note: This fix is partial — some constructor calls inside function
bodies may still be consumed by the type-env constructor binding
scanner before reaching resolveCallTarget. Filed as known limitation.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:46:34 +01:00
marxo126andClaude Opus 4.6 561a54a154 refactor: simplify Swift fixes after code review
- Extract shared addSwiftImplicitImports() helper (DRY — was duplicated
  in processImports and processImportsFromExtracted)
- Cache importMap.get(srcFile) outside inner loop (avoids redundant
  Map lookups per iteration)
- Fix export-detection: use \bprivate\b regex instead of includes()
  to avoid substring false positives
- Fix groupSwiftFilesByTarget: check path boundary with indexOf + char
  check instead of loose includes()

All 141 relevant tests pass (queries + imports + calls).

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:46:34 +01:00
marxo126andClaude Opus 4.6 65bc99c448 feat: full Swift cross-file resolution (export, imports, constructors)
Three changes that together enable cross-file call resolution for Swift:

1. export-detection.ts: Treat internal (default) Swift symbols as exported.
   Swift's default access level is `internal` (module-scoped, visible to
   all files in the same target). Only private/fileprivate are file-scoped.
   Previously all non-public/open symbols were marked unexported.

2. import-processor.ts: Add implicit import edges between all Swift files
   in the same module/target. Swift has no file-level imports — all files
   see each other automatically. Without these edges, the tiered resolver
   can't find cross-file symbols at Tier 2a (import-scoped).
   Supports SPM targets via Package.swift; falls back to single-module
   for Xcode projects without SPM.

3. call-processor.ts: Add constructor fallback for free-form calls.
   Swift constructors look like free function calls (no `new` keyword):
   `let ocr = OCRService()`. The call form is inferred as `free`, which
   filters out Class/Struct targets. Now retries with `constructor` form
   when free-form finds no callable but the name resolves to a type.

Tested on 61-file iOS 26 project (PricePal):
- Before: 0 cross-file CALLS edges
- After: full cross-file resolution (OCRService traced from ScanViewModel)
- 3,099 nodes, 10,449 edges, 246 clusters, 243 flows

Related: #406, #407

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:46:34 +01:00
marxo126andClaude Opus 4.6 0c8ec952ee fix: handle trailing commas in tree-sitter-swift binding.gyp patch
The patch script fails to parse tree-sitter-swift@0.6.0's binding.gyp
because the file contains both Python-style # comments AND trailing
commas in JSON arrays. The existing regex strips # comments but leaves
trailing commas, causing JSON.parse() to fail with:

  "Unexpected token ']'"

This silently prevents tree-sitter-swift from building, which means
Swift files are skipped entirely during analysis.

Fix: add a second regex pass to strip trailing commas before ] or }
after comment removal.

Fixes #386, #406

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-22 11:46:34 +01:00
Gergő Magyar 907440cf0b Merge pull request #427 from ShunsukeHayashi/fix/impact-confidence-412 2026-03-22 10:11:11 +00:00
Gergő Magyar 843c561e9b Merge pull request #425 from ShunsukeHayashi/fix/db-lock-325 2026-03-22 09:40:25 +00:00
Gergő Magyar 4c8be50cb1 Merge pull request #424 from ShunsukeHayashi/fix/lan-cors-390 2026-03-22 09:39:42 +00:00
Shunsuke Hayashi d07a69c3b1 fix(server): allow private/LAN network origins in CORS (#390)
Allow RFC 1918 private network ranges in the CORS origin allowlist so
users running GitNexus on their home or office LAN can access the web UI
from another device on the same network.

Permitted private ranges:
  10.0.0.0/8      (10.x.x.x)
  172.16.0.0/12   (172.16.x.x – 172.31.x.x)
  192.168.0.0/16  (192.168.x.x)

The origin check is extracted into an exported isAllowedOrigin() helper
so it can be unit-tested in isolation.  A new test file covers:
  - No origin (curl / server-to-server)
  - localhost and 127.0.0.1 variants
  - All three RFC 1918 ranges including boundary values
  - The deployed gitnexus.vercel.app site
  - Public / untrusted origins that must be rejected

The server bind address (127.0.0.1 by default) is unchanged; this PR
only affects which cross-origin browser requests are accepted.

Closes #390
2026-03-22 18:28:21 +09:00
Shunsuke Hayashi d206bf6772 fix(ingestion): calculate confidence per resolution tier for heritage/MRO edges (#412)
Instead of hardcoding confidence: 1.0, compute it at ingestion time using
the same resolution tier system that CALLS edges already use.

Heritage edges (EXTENDS, IMPLEMENTS):
- resolveHeritageId now returns { id, confidence } using TIER_CONFIDENCE
- Same-file → 0.95, import-scoped → 0.9, global → 0.5
- Edge confidence = geometric mean of source and target confidence
  (principled for partially-correlated cross-scope estimates, per
  Dillig et al. POPL 2011 and Dempster-Shafer theory)

MRO edges (OVERRIDES):
- MRO-ordered → 0.9, class method wins → 0.95
- Single interface → 0.85, ambiguous/unresolved → 0.5

IMPORTS and CONTAINS intentionally keep 1.0 (deterministic).

Closes #412
2026-03-22 18:23:40 +09:00
Shunsuke Hayashi 03156935cb fix(lbug): retry on DB lock with session-safe cleanup (#325)
When LadybugDB throws a BUSY/lock error (e.g. CLI and server running
concurrently), withLbugDb retries up to 3 times with linear backoff.

Addresses review feedback:

1. **Race condition fix**: Connection cleanup (close + state reset) now
   runs inside runWithSessionLock, preventing another operation from
   acquiring the lock between cleanup steps and having its connection
   closed from under it.

2. **Tests call withLbugDb directly**: Replaced simulateWithRetry helper
   with tests that invoke the real withLbugDb implementation, catching
   regressions in retry count, backoff, and lock interaction.

Closes #325
2026-03-22 18:21:21 +09:00
Gergő Magyar 22b5fce19e Merge pull request #426 from ShunsukeHayashi/fix/no-git-folder-384 2026-03-22 08:46:47 +00:00
Shunsuke Hayashi 09e3609376 fix(analyze): address review — rename --no-git to --skip-git, fix stale cache
Addresses all review items from @magyargergo and Copilot:

1. **Rename --no-git to --skip-git**: Commander.js treats --no-X flags
   as negation of --X (stores as options.git = false, not options.noGit).
   --skip-git maps correctly to options.skipGit.

2. **Fix false " Already up to date\ on non-git folders**: When
 currentCommit is empty string, skip the cache check — we cannot
 detect changes without git, so always rebuild.

3. **Replace isGitRepo() with hasGitDir()**: Use filesystem check
 (statSync on .git) instead of shelling out to git CLI. Consistent,
 faster, and works when git is not installed.

4. **Fix misleading warning**: Message now only fires when .git
 directory is actually absent (not when git CLI fails).

5. **Add CLI integration tests**: Verify Commander maps --skip-git
 correctly and that non-git folders are rejected without the flag.
2026-03-22 17:40:02 +09:00
Shunsuke Hayashi f0f384aab7 fix(analyze): address Copilot review — ESM import, CLI option, .gitignore guard
- Replace require(" fs\) with ESM-compatible top-level import (statSync)
- Register --no-git option in Commander CLI definition
- Use hasGitDir() instead of isGitRepo() for .gitignore update guard
 to match the PR intent (filesystem check vs git CLI invocation)
2026-03-22 16:24:21 +09:00
Gergő Magyar ef8b252bcc Merge pull request #428 from ShunsukeHayashi/fix/cypher-schema-docs-411 2026-03-22 06:21:40 +00:00
Gergő Magyar 92a1086697 Merge pull request #300 from jim80net/fix/onnxruntime-cuda-provider 2026-03-22 05:58:55 +00:00
Shunsuke Hayashi b272c6864c docs(schema): add Community and Process node properties to cypher tool description (#411)
The cypher tool description and schema resource omit Community and Process
node properties, causing agents to write failing queries on first attempt.

Added property listings sourced from the actual LadybugDB schema definitions:
- Community: heuristicLabel, cohesion, symbolCount, keywords, description, enrichedBy
- Process: heuristicLabel, processType, stepCount, communities, entryPointId, terminalId

Closes #411
2026-03-22 13:32:12 +09:00
Zander Raycraft 4031562f6b redoing auto labeling using z score instead of clustering lowkey the method 2026-03-21 23:21:23 -05:00
Shunsuke Hayashi 4dffd81b12 fix(analyze): allow indexing folders without a .git directory (#384)
Previously gitnexus analyze exited with an error on any directory that
lacked a .git entry, making it impossible to index generated code,
vendored libraries, or monorepo sub-trees that are not git roots.

Changes:

storage/git.ts
  - Add hasGitDir(dirPath): boolean — a lightweight synchronous check for
    the presence of a .git file or directory.  Works for git worktrees
    (.git file pointing at the real repo) as well as standard repos.

cli/analyze.ts
  - Add noGit?: boolean to AnalyzeOptions.
  - When the explicit inputPath resolves to a non-git folder (or the cwd
    is not inside any git repo), respect --no-git instead of hard-failing.
  - Print an actionable tip pointing at --no-git when git is absent and the
    flag was not supplied.
  - currentCommit defaults to an empty string for non-git folders so the
    up-to-date check still functions (empty string never matches a real
    commit hash, so the index is always rebuilt).
  - Skip addToGitignore() when no .git is present — there is nothing to
    update and the function would create a stale .gitignore at the root.

Git-dependent features that remain disabled for non-git folders:
  - Incremental update (always rebuilds from scratch)
  - Commit tracking in metadata
  - .gitignore update

Closes #384
2026-03-22 13:20:02 +09:00
Zander Raycraft b75e380bd0 Merge pull request #421 from zander-raycraft/main
workflow: triage reporting
2026-03-21 21:33:47 -05:00
Zander Raycraft 24421c2db1 Merge pull request #9 from zander-raycraft/gh/issue-pr-filter
token trunking
2026-03-21 20:16:10 -05:00
Zander Raycraft e6eaf08382 token trunking 2026-03-21 20:15:50 -05:00
Zander Raycraft f409fb7525 Merge pull request #7 from zander-raycraft/gh/issue-pr-filter
updated mahalanobis threshold to be multi-dim aware
2026-03-21 20:07:19 -05:00
Zander Raycraft cc2b13e332 updated mahalanobis threshold to be multi-dim aware 2026-03-21 20:06:29 -05:00
Zander Raycraft b909bbff70 Merge pull request #6 from zander-raycraft/gh/issue-pr-filter
updating workflow verbage
2026-03-21 19:37:12 -05:00
Jim ParkandClaude Opus 4.6 56356b71db fix: clarify that ORT CUDA binaries are linux/x64 only
ORT 1.24.x downloads CUDA provider .so from NuGet at postinstall,
but only for linux/x64. The process.arch guard correctly returns
false on arm64 (safe CPU fallback), but the prior comment implied
arm64 CUDA was supported. Clarify the actual state.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 17:30:23 -07:00
Zander Raycraft 2c9e887a80 updating workflow verbage 2026-03-21 19:30:10 -05:00
Zander Raycraft 8fbfd09081 Merge pull request #5 from zander-raycraft/gh/issue-pr-filter
fixed prop cutoff issue for pr/issue filtering
2026-03-21 19:21:33 -05:00
Zander Raycraft 6f4281c946 fixed prop cutoff issue for pr/issue filtering 2026-03-21 19:20:33 -05:00
Jim ParkandClaude Opus 4.6 60265c1d0d Merge upstream/main and fix critical module instance mismatch
Address PR #300 review findings:

[CRITICAL] hasOrtCudaProvider() was checking the top-level
onnxruntime-node@1.24.3 but @huggingface/transformers loads its own
nested onnxruntime-node@1.21.0 at runtime. The guard inspected the
wrong binary, so the native crash was not prevented.

Fix: resolve onnxruntime-node from transformers' own module scope
(createRequire from transformers' package.json) so the guard always
checks the same binary that will be dlopen'd at runtime.

Also:
- Add npm overrides to force @huggingface/transformers to use our
  onnxruntime-node@^1.24.0 (works for global installs where gitnexus
  is the root package; npx installs get safety from the resolve fix)
- Replace hardcoded 'x64' with process.arch for arm64 support
- Remove dead napi-v3 path check (ORT 1.21.0 never shipped CUDA .so)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 16:45:10 -07:00
JayceeB1 fa765beec1 test: add server-mode hydration regressions 2026-03-21 23:02:39 +01:00
Gergo Magyar 61f2f6d954 fix: update symbol and relationship counts in documentation 2026-03-21 14:37:44 +00:00
Gergő Magyar 2428e72bd2 Merge pull request #409 from abhigyanpatwari/refactor/language-dispatch-unification
refactor: unify language dispatch with compile-time exhaustive tables
2026-03-21 14:33:28 +00:00
Gergo Magyar 76ed0fa53b fix: address PR #409 review findings (P0-P3) and simplify import resolution API
Bug fixes (P0):
- Narrow Go /cmd/ entry-point detection to only match /main.go
- Fix Rust scoped grouped imports (use crate::models::{User, Repo}) resolution
- Filter PHP use function/use const from class-type namedImportMap bindings

Improvements (P1):
- Add C# resolveStandard fallback when .csproj discovery fails
- Change preprocessImportPath return type to string | null with caller guards
- Add Q_SIGNALS/Q_SLOTS (standard plural Qt macros)
- Fix stale "11 supported languages" comment → 13

API simplification (P2):
- Replace buildImportResolvers() factory with const importResolvers table
- Move configs onto ResolveCtx (extends ImportResolutionContext)
- Eliminate tsconfigPaths parameter threading through 6 non-TS resolvers
- Split utils.ts (1,476 lines) into ast-helpers.ts + call-analysis.ts + utils.ts
- Consolidate findChild/findChildByType into single source of truth
- Match multi-file import bindings to files by basename for namedImportMap

Cleanup (P3):
- EMPTY_INDEX returns shared frozen empty array instead of allocating per call
- Type appendKotlinWildcard parameter as SyntaxNode instead of any
- Document call-routing validation requirement on CallRouter type

Tests:
- Add unit tests for preprocessImportPath (13 tests)
- Add integration tests: Rust scoped multi-file, PHP use function/const,
  C# without .csproj, Go cmd/ helper scoring (14 tests, 4 fixtures)

All 3579 tests pass.
2026-03-21 14:15:15 +00:00
Gergo Magyar 2c17a4642c refactor: unify language dispatch with compile-time exhaustive tables
Replace the 120-line if-chain in resolveLanguageImport() and the 7-branch
dispatch in extractNamedBindings() with per-language dispatch tables using
`satisfies Record<SupportedLanguages, T>` for compile-time exhaustiveness.

Key changes:
- New import-resolution.ts: buildImportResolvers factory + namedBindingExtractors
  table + preprocessImportPath with control character rejection
- Extracted loadImportConfigs(), createImportEdgeHelpers(), getLabelFromCaptures()
  to eliminate duplication across import-processor.ts and parse-worker.ts
- Added isCppDuplicateClassFunction and getLabelFromCaptures shared helpers
- Migrated ENTRY_POINT_PATTERNS and AST_FRAMEWORK_PATTERNS_BY_LANGUAGE to
  satisfies Record<SupportedLanguages, T> with compile-time exhaustiveness
- Added Kotlin entry-point patterns (Android lifecycle, Ktor, MVVM)
- Expanded framework detection: Go (Gin/Echo/Fiber/gRPC), Rust (Actix/Axum/
  Rocket/Tokio), C++ (Qt), Swift (UIKit/SwiftUI/Vapor), Ruby (Rails/Sinatra)
- Fixed Go cmd/ entry-point detection operator precedence bug
- Added EMPTY_INDEX frozen sentinel for type-safe memory cleanup
- Unified ImportResolutionContext (suffixIndex->index, removed unused dispose())
- Extracted NamedBinding interface (replaced 16 inline occurrences)
- Replaced any with SyntaxNode on all tree-sitter node parameters
- Added contributor checklist to SupportedLanguages enum

14 files changed, +318/-550 (net -232 lines). All 3550+ tests pass.
2026-03-21 11:48:11 +00:00
JayceeB1 4f2e4ac24d Fix server-mode repo switching in gitnexus-web 2026-03-21 11:10:29 +01:00
Gergő Magyar 1af9c028c5 Merge pull request #397 from abhigyanpatwari/feat/phase14-cross-file-binding-propagation
feat(type-resolution): cross-file binding propagation
2026-03-21 08:36:23 +00:00
Gergo Magyar fb20a3c752 feat: implement cross-file binding propagation for multiple languages
- Enhance C++ tree-sitter queries to support inline class method declarations and return types.
- Introduce `importedRawReturnTypes` in `BuildTypeEnvOptions` for cross-file raw return type handling.
- Add `FileTypeEnvBindings` interface to capture file-scope type bindings for exported symbols.
- Implement logic in `parse-worker.ts` to extract and serialize file-scope type bindings for cross-file type resolution.
- Create test fixtures for C++, Go, Ruby, and Rust to validate cross-file binding propagation.
- Update integration tests to verify correct resolution of method calls across files for C++, Go, Ruby, and Rust.
- Document Phase 14: Cross-File Binding Propagation in the type resolution roadmap and system documentation.
2026-03-21 07:47:04 +00:00
Gergo Magyar f1fbe643df Merge remote-tracking branch 'origin' into feat/phase14-cross-file-binding-propagation 2026-03-21 06:09:57 +00:00
zm2231 89a24d866e fix: validate dimensions on every vector, not just the first 2026-03-20 22:42:25 -04:00
zm2231 9baef90ae2 fix: edge case guards, dim mismatch hard-throw, UX label
- Guard against empty endpoint response in httpEmbedQuery (Bug 1)
- Validate response item count matches batch size in httpEmbed (Bug 2)
- Dimension mismatch now hard-throws instead of warn-and-continue (Bug 3)
- Progress bar shows Connecting to embedding endpoint in HTTP mode (UX gap)
- Added 3 tests: empty response, truncated batch, dim mismatch throw
- All 19 HTTP embedder tests pass
2026-03-20 22:27:27 -04:00
0xfabs 11575cf6c8 fix: hydrate worker DB in server mode + fix LadybugDB getAll API mismatch (#398) (#404)
Fix server/bridge mode leaving the web UI with 0 nodes and broken
Query/Processes/embeddings by hydrating the worker-side LadybugDB
and BM25 indexes after loading graph data from the backend.

Also fix LadybugDB QueryResult API mismatch where result.getAll()
does not exist in some @ladybugdb/wasm-core versions — falls back
to getAllObjects() or getAllRows().
2026-03-21 06:09:25 +05:30
zm2231 c2c694ac42 feat: add .env.example, empty batch guard, dimension mismatch warning
- Add .env.example with all HTTP embedding env vars documented
- Early return in httpEmbed() for empty text arrays
- Warn once if API returns vectors with different dimensions than
  GITNEXUS_EMBEDDING_DIMS — helps catch misconfiguration early
2026-03-20 20:15:59 -04:00
abhigyanpatwariandClaude Opus 4.6 01ff2a7540 docs: add gitnexus-stable-ops to community integrations
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 05:15:54 +05:30
zm2231 13b22e9879 fix: revert auto-generated AGENTS.md/CLAUDE.md changes
gitnexus analyze rewrote repo name to local clone name (gitnexus-fork).
Restore canonical GitNexus references.
2026-03-20 18:53:05 -04:00
zm2231 c8480f899d fix: address review feedback — deduplicate HTTP client, remove config cache, add guards
- Extract shared HTTP client (http-client.ts) used by both core and MCP embedders
- Remove module-level httpConfig cache — read env vars fresh on every call
  so config set after module load (e.g. via dotenv) takes effect
- Add NaN/non-positive guard on GITNEXUS_EMBEDDING_DIMS in schema.ts
- Include scrubbed URL and batch index in error messages (no API key)
- Wrap fetch rejections (DNS/timeout/connection) with same scrubbed context
- MCP embedder delegates to shared httpEmbedQuery() instead of inline logic
- apiKey confined to http-client.ts internals — not exported in any type or accessor
- Remove HttpEmbeddingConfig from types.ts (replaced by internal HttpConfig)
- All 16 HTTP embedder tests pass, tsc clean
2026-03-20 18:50:36 -04:00
adonis 3567ef5794 feat(cli): add --allow-non-git option to index command and update tests 2026-03-20 19:46:47 -03:00
adonis 680cefd00c feat(cli): refactor indexCommand for improved error handling and add unit tests 2026-03-20 19:36:07 -03:00
Abhigyan PatwariandDennis Palatov 1f7764c49b fix: register Section in NODE_TABLES and NODE_SCHEMA_QUERIES (#401)
* feat: add markdown file indexing (headings + cross-links)

Parse .md/.mdx files using regex (no tree-sitter dependency) to extract:
- Section nodes from headings (h1-h6) with hierarchy via CONTAINS edges
- Cross-file IMPORTS edges from markdown links to other repo files

Ported from #286 to resolve conflicts with kuzu→lbug rename.

Co-Authored-By: Dennis Palatov <dp-web4@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add Section to NODE_TABLES and NODE_SCHEMA_QUERIES

The Section schema was defined but not registered in NODE_TABLES or
NODE_SCHEMA_QUERIES, so the table was never created in the database.
Also adds missing FROM File TO Section relation entry.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: update schema test counts for Section node type

NODE_TABLES: 27→28, NODE_SCHEMA_QUERIES: 27→28, SCHEMA_QUERIES: 29→30

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* test: add diagnostic output to skills-e2e idempotency test

Show stdout/stderr in assertion message so CI failures reveal
why the second analyze --skills run exits with code 1.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: add Section COPY query with level column in lbug-adapter

Section table has 8 columns (includes level) but getCopyQuery fell
through to the default 7-column multi-language path. Adds explicit
Section cases to getCopyQuery and insertNodeToLbug/upsertNodeToLbug.

Error was: COPY failed for Section: Number of columns mismatch. Expected 7 but got 8.

---------

Co-authored-by: Dennis Palatov <dp-web4@users.noreply.github.com>
2026-03-21 04:00:16 +05:30
adonis c1996f45d8 feat(cli): add 'index' command to register existing .gitnexus/ folder 2026-03-20 19:16:09 -03:00
00758b102a feat: add markdown file indexing (headings + cross-links) (#399)
feat: add markdown file indexing (headings + cross-links)

Ports #286 by @dp-web4 onto current main, resolving conflicts from kuzu→lbug rename.
Closes #286

Co-Authored-By: Dennis Palatov <dp-web4@users.noreply.github.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-21 03:11:42 +05:30
Gergő Magyar abeb52e5e5 Merge pull request #69 from x0m4ek/feat/codex-support 2026-03-20 21:22:32 +00:00
Gergo Magyar 6c972079e0 feat(type-resolution): per-language cross-file binding tests + resolver fixes
Add cross-file binding propagation integration tests for 6 languages
(Python, JavaScript, Java, PHP, C#, Kotlin) with fixture repos and
HAS_METHOD edge assertions.

Fix 3 language resolver issues uncovered by tests:
- Kotlin: class methods now labeled Method (not Function) via shared
  isKotlinClassMethod() utility; non-aliased imports create NamedImportMap
  entries; 2-segment package-directory fallback for top-level function imports
- PHP: namespace-directory fallback for use-function imports; SuffixIndex
  preferred over linear scan; path traversal rejection; PSR-4 sort cached
- JVM: root-level package path matching; indexOf→lastIndexOf for correctness

Address code review findings:
- Extract runCrossFileBindingPropagation() from 810-line pipeline function
- Replace (importCtx as any) casts with typed dispose() method
- Remove Tarjan's SCC (dev-only YAGNI, ~85 lines)
- Remove dead PARALLEL_RE_RESOLUTION_THRESHOLD constant
- Fix constructor handling divergence in getLabelFromCaptures
- Optimize gap pre-scan with early exit once threshold exceeded
- Fix findEnclosingFunction any→SyntaxNode type
2026-03-20 15:38:27 +00:00
Dmytro cbfdae0303 test(cli): cover full Codex setup flow 2026-03-20 15:28:42 +01:00
ximi 66c1ffa370 feat: add MiniMax provider support (#224)
Add MiniMax as a new LLM provider using the Anthropic-compatible API.

Changes:
- Add MiniMax to LLMProvider type and MiniMaxConfig interface
- Add MiniMax chat model creation via ChatAnthropic with custom base URL
- Add MiniMax settings persistence and model list (MiniMax-M2.5, MiniMax-M2.5-highspeed)
- Add MiniMax provider UI in SettingsPanel with API key and model selection
2026-03-20 19:48:31 +05:30
Gergő Magyar 88e0034771 Merge pull request #396 from hiromima/fix/kuzu-concurrent-query-segfault-and-stale-data
fix: sequential enrichment queries + stale data detection (#285, #290, #292, #297)
2026-03-20 12:29:00 +00:00
Gergo Magyar 0736bb23bc fix(type-resolution): address Phase 14 enhancement code review findings
1. Replace fragile regex in buildExportedTypeMapFromGraph with
   extractReturnTypeName() for consistent generic unwrapping.

2. Fix gap detection precision: check upstream.has(binding.exportedName)
   instead of exportedTypeMap.has(binding.sourcePath) to avoid
   over-counting files that import only unrelated symbols.
2026-03-20 12:28:22 +00:00
hiromima 2ff4d93314 fix(test): unify staleness-and-stability into single withTestLbugDB block
All 4 test blocks now share one DB lifecycle to avoid cross-block
"Database is closed" errors caused by LadybugDB's shared global DB
in a single vitest fork. Staleness detection (which triggers closeLbug)
runs last to avoid invalidating connections for other blocks.

11/11 tests pass on macOS, Ubuntu, and Windows.
2026-03-20 21:06:30 +09:00
Gergo Magyar fff716dd92 feat(type-resolution): Phase 14 enhancements — single-pass seeding, Tarjan's SCC, cross-file return types
E0: Fix AST cache thrashing in re-resolution loop (was creating size-1
cache per file), batch file reads per topological level, add
MAX_CROSS_FILE_REPROCESS=2000 cap for adversarial repos.

E1: seedCrossFileReceiverTypes() — enrich ExtractedCall.receiverTypeName
from ExportedTypeMap+namedImportMap in O(1) Map lookups, eliminating
re-parse for ~80-90% of single-hop cross-file receiver types.

E2: computeImportCycleSCCs() — iterative Tarjan's SCC on cycle subgraph
from Kahn's output. Dev-mode diagnostic logging of individual import
cycle components.

E3: buildImportedReturnTypes() + ReturnTypeLookup extension — cross-file
return type propagation with corrected local-first priority (SymbolTable
checked first, cross-file fallback only on 0 matches, ambiguous 2+
returns undefined).

E4: PARALLEL_RE_RESOLUTION_THRESHOLD constant, timing metrics, worker
parallelization design comments (deferred implementation).

24 new tests (6 E1 + 6 E2 + 7 E3 unit + 5 E3 integration). All 3478
tests pass.
2026-03-20 12:00:36 +00:00
Gergo Magyar 56bc226a1c fix(type-resolution): address Phase 14 code review findings
Critical fixes:
- Re-resolution pass now actually re-resolves CALLS edges by calling
  processCalls with importedBindingsMap (was building typeEnv but
  discarding it without producing edges)
- Worker path populates ExportedTypeMap via buildExportedTypeMapFromGraph
  using graph node isExported + SymbolTable returnType/declaredType
  (was dead parameter in processCallsFromExtracted)

Important fixes:
- Skip threshold denominator uses totalFiles (was exportedTypeMap.size +
  filesWithGaps which made threshold nearly useless)
- processCalls accepts importedBindingsMap parameter to thread cross-file
  bindings into buildTypeEnv during re-resolution

All 3454 tests pass.
2026-03-20 09:57:24 +00:00
Gergo Magyar a6a1004e82 feat(type-resolution): Phase 14 — cross-file binding propagation
Add ExportedTypeMap infrastructure to propagate resolved type bindings
across file boundaries. When file A exports `const user = getUser()`
(resolved to `User`), file B importing `user` now gets seeded with
`user → User`, enabling `user.save()` to produce CALLS edges.

Key components:
- `importedBindings` option on BuildTypeEnvOptions with scopeEnv seeding
  AFTER walk() to respect first-writer-wins (local declarations win)
- `collectExportedBindings()` in call-processor using graph node
  isExported flag (no SymbolDefinition changes needed)
- Inline Kahn's algorithm topological sort with level grouping for
  parallel-safe file ordering and cycle detection
- Re-resolution pass in pipeline.ts: topological order, 3% skip
  threshold, path validation, per-file export caps (500)
- 32 new tests: 11 topological sort, 6 seeding, 15 integration
  (simple cross-file, re-export chain, circular imports)

All 3454 tests pass (32 net new, 0 regressions).
2026-03-20 09:31:20 +00:00
hiromima 5b012c3351 test: add e2e tests for stale detection, sequential enrichment, stability (#396)
- Stale data detection: verify ensureInitialized() detects meta.json
  changes and re-opens pool without SIGSEGV or WAL corruption
- Staleness throttle: verify 5s throttle window doesn't cause errors
- Sequential enrichment: impact() enrichment queries complete on arm64
- Consecutive stability: 10+ sequential cypher calls, mixed tool cycles
- Watchdog guard: parallel queries with activeQueryCount protection
- stdout restoration: verify process.stdout.write is properly restored

Covers test plan items from PR #396 (issues #285, #290, #292, #297)
2026-03-20 18:04:27 +09:00
Gergő Magyar 862fdf9185 Merge pull request #392 from abhigyanpatwari/feat/phase-p-polymorphism-overloading
feat(type-resolution): polymorphism and method overloading support
2026-03-20 08:54:43 +00:00
Gergo Magyar 228c993bb7 fix(type-resolution): review fixes, sizeBefore optimization, and test coverage
Address code review findings from PR #392 senior compiler review:
- Fix Java "Yes" → "No" in optional-param-arity matrix (Java has no defaults)
- Simplify Kotlin hasDefaultValue while-as-if to direct const/if check
- Update OPTIONAL_PARAM_TYPES comment to include Ruby
- Replace per-declaration Set allocation with size-based Map iteration skip
- Add 11 unit tests for multi-declarator type association and constructorTypeMap
2026-03-20 08:45:42 +00:00
Gergo Magyar c3a2815186 feat(type-resolution): optional parameter arity resolution
Add requiredParameterCount to SymbolDefinition and MethodSignature,
enabling range-based arity filtering in filterCallableCandidates.
Calls with omitted optional/default arguments now resolve correctly.

Supported: TS, Python, Kotlin, C#, C++, PHP, Ruby (7 languages).
Detection via OPTIONAL_PARAM_TYPES set + hasDefaultValue helper.

9 integration tests added across all 7 languages.
2026-03-20 08:14:58 +00:00
hiromima 893f77ae89 fix: address review feedback — watchdog, TOCTOU race, null guard, platform guard
- Watchdog timer now exempts in-flight queries via activeQueryCount,
  preventing premature stdout restoration during long queries (>1s)
- Stale detection uses reinitPromises Map to prevent TOCTOU race where
  concurrent callers double-close the connection pool
- Throttle meta.json staleness checks to once per 5s per repo
- Add null guard for i.id in IN-clause construction
- Enrichment queries run in parallel on non-arm64 platforms to preserve
  performance; sequential only on arm64 macOS where SIGSEGV occurs
2026-03-20 16:52:33 +09:00
Gergo Magyar d49c76ddc5 feat: Implement virtual dispatch and overload disambiguation enhancements
- Updated AGENTS.md and CLAUDE.md to reflect new indexing metrics.
- Enhanced call-processor.ts to support cross-file inheritance tracking and improved virtual dispatch resolution.
- Added support for TypeScript overload signatures in tree-sitter queries.
- Improved type extraction for C++, C#, and Kotlin to handle smart pointers and constructor types.
- Introduced inferLiteralType for overload disambiguation across multiple languages.
- Added tests for C++ smart pointer dispatch and Kotlin virtual dispatch scenarios.
- Updated type-resolution-roadmap.md to reflect completion of phases P.1 to P.3 and outline future work on covariant return types.
2026-03-20 07:23:14 +00:00
zm2231 7dcafb647b fix: address review feedback — timeout, retry, guards, tests
- Add AbortSignal.timeout(30s) on all fetch calls
- Add retry with backoff for 429/5xx (core: 2 retries, MCP: 1 retry)
- Guard initEmbedder() and getEmbedder() to throw in HTTP mode
- Discard cached embeddings on dimension mismatch during incremental re-index
- Add MCP embedQuery retry for transient failures
- Add 16 unit tests covering both core and MCP HTTP paths
- Fix README: concise, accurate env var docs
2026-03-20 01:32:49 -04:00
zm2231 5dcc567870 fix: address code review — timeout, retry, safe initEmbedder guard
- Add AbortSignal.timeout(30s) to all HTTP fetch calls
- Add retry with backoff for 429/5xx (max 2 retries)
- initEmbedder() throws descriptive error in HTTP mode instead of null cast
- embedding-pipeline.ts guards initEmbedder() behind isEmbedderReady() check
- Error messages omit request details to avoid leaking API keys in logs
2026-03-20 00:37:17 -04:00
hiromima 999fbf5b11 fix: sequential enrichment queries + stale data detection
Fixes three related issues that cause SIGSEGV crashes and stale data:

1. Impact enrichment queries (Promise.all → sequential await)
   The impact() method ran 3 enrichment queries concurrently via
   Promise.all against the same LadybugDB connection pool. On arm64
   macOS, concurrent native DB access triggers SIGSEGV. Changed to
   sequential await. Also caps IN-clause to 100 IDs to prevent
   oversized queries. (#285, #290, #292)

2. Silence stdout during query execution
   silenceStdout()/restoreStdout() only wrapped createConnection() and
   initLbug(). Now also wraps executeQuery() and executeParameterized()
   to prevent native stdout writes from corrupting the MCP stdio
   stream during all DB operations. (#285)

3. Stale data after re-index
   ensureInitialized() checked pool existence but never verified whether
   the underlying index was rebuilt. Now reads meta.json's indexedAt
   timestamp on each call and closes/re-opens the pool when the index
   has changed. (#297)
2026-03-20 13:33:20 +09:00
zm2231 c79c717790 fix: dynamic embedding dimensions for HTTP backend
- schema.ts: FLOAT[${EMBEDDING_DIMS}] reads from GITNEXUS_EMBEDDING_DIMS env
- embedding-pipeline.ts: vector search CAST uses actual query vector length
- mcp/core/embedder.ts: HTTP embedding support for MCP query-time search

Without this, using a 1024d model (e.g. bge-large) fails with
'Expected: 384, Actual: 1024' on LadybugDB vector insert.
2026-03-20 00:31:39 -04:00
zm2231 9dc00e47a7 feat: HTTP embedding backend for self-hosted/remote endpoints
Adds support for OpenAI-compatible embedding endpoints as an alternative
to the local transformers.js pipeline. Enables using self-hosted servers
(Infinity, vLLM, TEI, llama.cpp) over Tailscale/VPN, or any cloud
endpoint — with higher-quality models like bge-large-en-v1.5 (1024d).

Configuration via environment variables:
  GITNEXUS_EMBEDDING_URL=http://your-server:8080/v1
  GITNEXUS_EMBEDDING_MODEL=BAAI/bge-large-en-v1.5
  GITNEXUS_EMBEDDING_API_KEY=your-key  (default: 'unused')
  GITNEXUS_EMBEDDING_DIMS=1024         (auto-detected if omitted)

When env vars are set:
- initEmbedder() skips local model download entirely
- embedText() and embedBatch() call the HTTP endpoint
- Dimensions auto-detected from first response
- Batches in groups of 64

When env vars are NOT set:
- Existing local transformers.js behavior is completely unchanged

Build: tsc clean
Tests: 1776 passed, 0 failed
2026-03-19 22:16:52 -04:00
Zander Raycraft c14a78a341 Merge pull request #394 from zander-raycraft/fix/claude-review
SEC/ spam guards for the PR and issue claude-review workflow for contributions
2026-03-19 20:35:23 -05:00
Zander Raycraft 7a0a83e45c added author_association to @claude calls to amke it to where only maintainers and above can call @claude not just anyone 2026-03-19 20:25:43 -05:00
Zander Raycraft f2edbb4f82 CI sticky note test 2026-03-19 18:26:56 -05:00
Gergo Magyar 1d27ad09a2 test(type-resolution): assert parameterTypes on graph nodes in integration tests
Add parameterTypes to graph node properties (parse-worker + parsing-processor)
so integration tests can verify extracted parameter types per language:
- Java: ['int'] on lookup(int) overload
- C#: ['int'] on Lookup(int) overload
- C++: ['int'] on lookup(int) overload
- Kotlin: ['Int'] on lookup(Int) overload

Add getNodesByLabelFull helper for property-level assertions.
2026-03-19 22:51:16 +00:00
Gergő Magyar 68d4c48aba Merge pull request #391 from zander-raycraft/fix/claude-review
CI sticky notes
2026-03-19 22:49:17 +00:00
Gergo Magyar bc771574d8 test(type-resolution): Phase P integration tests + fixes for all overloading languages
Integration tests for overload disambiguation (Java, Kotlin, C#, C++)
and virtual dispatch (Java, TypeScript) with strict toBe() assertions.

Unit tests verify exact parameterTypes extraction per language:
- Java: ['int'], ['String'], ['int', 'String']
- Kotlin: ['Int'], ['String']
- C#: ['int'], ['string']
- C++: ['int'], ['string']

Fixes discovered during testing:
- extractSimpleTypeName: handle Java integral_type/boolean_type/etc
- tryOverloadDisambiguation: unwrap C# argument + Kotlin value_argument
  wrapper nodes; traverse Kotlin call_suffix for value_arguments
- Kotlin boxed→primitive normalization (Int→int, Long→long, etc.)
- C++ tree-sitter queries: capture pointer-returning inline class methods
- extractFunctionName: handle C++ field_identifier for inline methods
2026-03-19 22:47:24 +00:00
Zander Raycraft 5d6e15eea3 minor bug fixes 2026-03-19 17:19:05 -05:00
Zander Raycraft eb74eb8590 CI sticky notes 2026-03-19 17:06:32 -05:00
Gergo Magyar 700c9d16e4 feat(type-resolution): constructor-visible virtual dispatch via constructorTypeMap
Add constructorTypeMap to buildTypeEnv — populated during walk when a
declaration has both a type annotation and a constructor initializer.
Add isSubclassOf helper (BFS, depth-5, cycle-safe).
In call-processor, consult constructorTypeMap to override receiver type
when constructor creates a known subclass (same-file only).
2026-03-19 21:49:21 +00:00
Gergo Magyar 19ff84fa31 feat(type-resolution): overload disambiguation via argument literal types
Add inferLiteralType to LanguageTypeConfig for Java, Kotlin, C#, C++.
In resolveCallTarget, when multiple candidates survive arity filtering,
lazily infer argument literal types and filter by parameterTypes match.
Worker path falls through gracefully (no AST available).
2026-03-19 21:45:34 +00:00
Gergo Magyar 2fe03d2a21 feat(type-resolution): extract parameterTypes in extractMethodSignature
Add parameterTypes?: string[] to SymbolDefinition and MethodSignature.
Extract per-parameter type names via extractSimpleTypeName during
parsing for overload disambiguation (Java, Kotlin, C#, C++).
Thread through both sequential (parsing-processor) and worker
(parse-worker) paths.
2026-03-19 21:41:58 +00:00
Gergo Magyar 3e29f4e4b9 fix(symbol-table): store all overloads in fileIndex instead of last-write-wins
The fileIndex Map stored SymbolDefinition per name, silently dropping
earlier overloads via Map.set(). Changed to SymbolDefinition[] so all
same-name methods (e.g., Java overloads) survive in same-file resolution.

Added lookupExactAll() for resolution-context to pass all same-file
candidates through to candidate filtering.
2026-03-19 21:37:21 +00:00
Gergő Magyar 71353512a1 Merge pull request #388 from zander-raycraft/fix/claude-review
FIX - Claude review fork 403 for PR reviews
2026-03-19 21:29:09 +00:00
Gergő Magyar 00c5126b24 Merge pull request #387 from abhigyanpatwari/feat/type-resolution-milestone-d
feat(type-resolution): Milestone D — Phases A, B, C + Kotlin fix + 11-language integration tests
2026-03-19 21:26:19 +00:00
Gergo Magyar 06994e474a fix(type-resolution): address PR #387 review — dead code, nullable_type, scope boundaries + integration tests
- Remove dead replayPendingItems array and inert if-block in type-env.ts
- Add nullable_type fallback in extractKotlinDeclaration for val x: User? local vars
- Tighten isCSharpNullableDecl to avoid substring false positives on type names
- Add missing scope boundaries: function_expression (TS), constructor_declaration/
  local_function_statement/lambda_expression (C#) in null-check narrowing walkers
- Extend null-check narrowing fixtures and add 4 integration tests covering:
  Kotlin local variable nullable, C# constructor + lambda, TS function expression
2026-03-19 21:06:05 +00:00
Zander Raycraft c507b4b197 remove manual test 2026-03-19 15:34:56 -05:00
Gergo Magyar b6947b0c02 feat(type-resolution): consolidate Phases 10–13 into Milestone D — enhance completeness with full integration test coverage across 11 languages 2026-03-19 20:31:35 +00:00
Gergo Magyar e9ccec1a52 test(type-resolution): add integration tests for Milestone D across all 11 languages + fix Kotlin null-check narrowing
Adds 17 new fixture directories and 23 new describe blocks covering every
feature in Milestone D (Phases A, B, C) with full cross-language integration
test coverage:

Phase A — Fixpoint Completeness:
- TS/JS object destructuring (const { field } = obj → fieldAccess resolution)
- TS/JS post-fixpoint for-loop replay (iterable var resolved by fixpoint)
- Rust struct_pattern destructuring (let Point { x, y } = p)

Phase B — Inheritance & Receivers:
- Grandparent MRO (depth-2 C→B→A) for all 9 OOP languages:
  TS, Kotlin, C#, C++, Java, PHP, Python, Ruby, JS
- Go inc/dec write access (obj.Field++/-- emit ACCESSES write edges)

Phase C — Branch-Sensitive Narrowing:
- Null-check narrowing for TS (!==null, !=null, !==undefined),
  C# (!=null, is not null), and Kotlin (!=null)

Bug fix — Kotlin null-check narrowing (3 issues in jvm.ts):
1. patternBindingNodeTypes registered 'comparison_expression' but
   tree-sitter-kotlin produces 'equality_expression' for !=
2. Handler checked for 'null_literal' named child but 'null' is an
   anonymous node in the Kotlin grammar
3. extractKotlinParameter only searched for 'user_type' direct child,
   missing 'nullable_type' wrapper (so x: User? never got a base binding)

17 fixtures, 23 describe blocks, 705 new lines of test code, 0 failures.
2026-03-19 20:23:06 +00:00
Zander Raycraft 0122d9e694 manual trigger for testing on pr comments 2026-03-19 14:12:53 -05:00
Zander Raycraft 00e2476eca claude reviewer update 2026-03-19 14:08:48 -05:00
Zander Raycraft 217efcf015 Merge remote-tracking branch 'upstream/main' 2026-03-19 13:30:46 -05:00
Gergo Magyar 7c72cefd8d feat(type-resolution): implement Milestone D — Phases A, B, C
Phase A — Fixpoint Completeness:
- Extract fixpoint loop into resolveFixpointBindings() with exhaustive switch guard
- Add classDefCache to memoize lookupFuzzy results during fixpoint iteration
- Post-fixpoint for-loop replay: bridge walk-time/fixpoint gap (ex-Phase 9B)
- Object destructuring via fieldAccess items (TS/JS object_pattern, Rust struct_pattern)
- PendingAssignmentExtractor now supports returning arrays for multi-binding patterns

Phase B — Inheritance & Receivers:
- BuildTypeEnvOptions object replaces positional params (future-proof API)
- Heritage pre-pass: thread parent class data from query matches into buildTypeEnv
- walkParentChain() helper: MRO-aware field/method resolution (depth 5, cycle-safe)
- this/self/$this/Me receiver substitution at extractPendingAssignment call site
- Go inc/dec write-access detection via tree-sitter queries

Phase C — Branch-Sensitive Narrowing:
- Rename PATTERN_BRANCH_TYPES → NARROWING_BRANCH_TYPES (semantic expansion)
- Null-check narrowing: != null / !== undefined strips nullable wrapper in truthy branch
- Position-indexed patternOverrides with extractor-provided narrowing ranges
- TS, Kotlin, C# null-check narrowing extractors with if-body range detection

All 3315 existing tests pass. 9 new null-check narrowing tests added.
2026-03-19 17:29:47 +00:00
Gergo Magyar 58f67d07f7 fix(ci): grant actions:read to publish workflow for pr-report job 2026-03-19 13:07:48 +00:00
Gergo Magyar a4863605e1 chore: bump version to 1.4.7 and update CHANGELOG 2026-03-19 12:50:24 +00:00
Gergő Magyar fc58c415f7 Merge pull request #379 from abhigyanpatwari/feat/phase9-call-result-binding
feat: Phase 9 — Return-type-aware variable binding with unified fixpoint chain propagation
2026-03-19 12:43:45 +00:00
Gergo Magyar 790d1d5b0f refactor: update Type Resolution Roadmap to clarify phases and principles for static analysis evolution 2026-03-19 12:39:19 +00:00
Gergo Magyar 8273324f3c fix: address PR review — Rust await unwrap, stale doc claims, this-receiver footnote
Review follow-ups from compiler front-end review (#379):

- Rust extractPendingAssignment now calls unwrapAwait() on value before
  type checks, so `let user = get_user().await` resolves correctly
- type-resolution-system.md: removed "no fixpoint inference" from
  limitations, updated "Single-pass" to "Walk + fixpoint", replaced
  stale single-pass Tier 2 description with fixpoint loop explanation
- type-resolution-roadmap.md: Phase 9 body updated — 9C is delivered,
  9B walk-order dependency documented (for-loop Tier 0b runs before
  fixpoint, so fixpoint-resolved types can't update loop variables)
- Added this/self/$this fixpoint gap footnote to feature matrix
2026-03-19 12:14:54 +00:00
Gergo Magyar 7b71b64427 fix(mcp): update tool descriptions for Phase 9C capabilities
- context tool: remove outdated "Phase 2" ACCESSES reference, document
  that CALLS edges resolve through field/method chains
- cypher tool: fix Property query example to use declaredType (not description)
- schema resource: add node_properties section documenting Method returnType,
  Property declaredType, Function parameterCount etc.
- schema resource: clarify ACCESSES edge read/write coverage
2026-03-19 11:59:10 +00:00
Gergo Magyar e6b8edc1ac feat: Phase 9C unified fixpoint with field access and method-call-result binding
Replace the sequential Tier 2b/2a propagation with a unified fixpoint
loop that handles four binding kinds: callResult, copy, fieldAccess,
and methodCallResult. The loop iterates until no new bindings are
produced (max 10 iterations), enabling arbitrary-depth mixed chains:

  const user = getUser();       // callResult → User
  const addr = user.address;    // fieldAccess → Address
  const city = addr.getCity();  // methodCallResult → City
  city.save();                  // resolves to City#save

Infrastructure:
- PendingAssignment union extended with fieldAccess and methodCallResult
- resolveFieldType helper: typeName → class nodeId → lookupFieldByOwner
- resolveMethodReturnType helper: typeName → class nodeId → lookupFuzzyCallable filtered by ownerId
- Fixpoint also resolves reverse-order copy chains that single-pass missed

Languages: TS, JS, Java, Kotlin, C#, Go, Rust, Python, PHP, Ruby, C++.
Each gets field access and/or method-call-with-receiver detection in
extractPendingAssignment, plus method-chain-binding test fixtures.
2026-03-19 11:50:50 +00:00
Dmytro Semchuk a7b8c302d4 Merge branch 'main' into feat/codex-support 2026-03-19 10:24:45 +01:00
Gergo Magyar 5769872b70 feat: Phase 9 call-result variable binding across 11 languages
Activate the dormant Tier 2b pendingCallResults infrastructure in
type-env.ts by extending each language's extractPendingAssignment to
emit { kind: 'callResult', lhs, callee } when the RHS of an untyped
variable declaration is a simple function call.

This enables `var user = getUser(); user.save()` to resolve at TypeEnv
build time. Tier 2b now runs before Tier 2a copy-propagation, enabling
mixed chains like `const user = getUser(); const alias = user;
alias.save()`.

Languages: TS, JS, Java, Kotlin, C#, Go, Rust, Python, PHP, Ruby, C++.
Swift excluded. Each language gets a call-result-binding test fixture
and integration tests.

Conservative: only simple calls (no method calls with receivers), only
when exactly one callable matches, first-writer-wins.
2026-03-19 08:59:16 +00:00
60c93d7d4a feat: upgrade @ladybugdb/core to 0.15.2 and remove segfault workarounds (#374)
* feat: upgrade @ladybugdb/core to 0.15.2 and remove segfault workarounds

The upstream fix (ladybug-nodejs#1) resolves the child QueryResult lifetime
segfault, making .close() safe on all platforms. This removes 6 workaround
sites:

- Remove `dangerouslyIgnoreUnhandledErrors` from vitest config
- Remove platform-conditional .close() guards in global-setup and test helper
- Delete test/setup.ts (process._getActiveHandles unref hack)
- Replace no-op cleanup in test-indexed-db.ts with real adapter close
- Fix pool adapter closeOne() to properly close connections with shared
  Database refcount guard and orphaned connection handling in checkin()
- Update segfault-related comments across the codebase

Also bumps @ladybugdb/wasm-core to ^0.15.2 in gitnexus-web for consistency.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* fix: keep dangerouslyIgnoreUnhandledErrors for macOS N-API exit crash

The N-API destructor ordering crash during worker fork exit on macOS is
independent of the QueryResult lifetime fix in 0.15.2. Tests pass, but
the exit triggers a crash. Keep the flag with an updated comment
explaining the actual cause. Can be removed once LadybugDB fixes all
destructor ordering issues upstream.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

* ci: unify test run for single-pass coverage

- Update `npm test` to run all tests (unit + integration + lbug-db)
  via `vitest run` instead of `vitest run test/unit`
- Add `test:unit` script for running unit tests only
- Remove `ci-integration.yml` — the per-file lbug-db process isolation
  is no longer needed with `dangerouslyIgnoreUnhandledErrors` and
  `fileParallelism: false` handling fork exit issues
- Update `ci-unit-tests.yml` to run all tests with build + coverage
- Simplify `ci.yml` gate (two jobs: quality + tests)
- Simplify `ci-report.yml` (single coverage artifact, no merge step)

* fix: update cli-commands test for renamed test:all → test:unit script

* fix: set USERPROFILE in setup-skills test for Windows compatibility

os.homedir() checks USERPROFILE on Windows, not HOME.

* fix: add isolate: false to lbug-db project to prevent fork crashes

On macOS, N-API destructors crash fork workers on exit. With
isolate: true (default), vitest recycles the fork between files,
triggering the crash after each file. After several crashes, the
remaining lbug-db files never execute.

isolate: false keeps all 8 lbug-db files in a single fork — the
fork only exits once after all files complete, and that single exit
crash is caught by dangerouslyIgnoreUnhandledErrors.

* fix: add unique sequence.groupOrder to vitest projects

Vitest v4 requires unique groupOrder when projects have different
maxWorkers (lbug-db has fileParallelism: false → maxWorkers: 1).

* fix: await async close() in global-setup and remove isolate: false

global-setup.ts called conn.close() and db.close() without await —
these return Promise<void> in @ladybugdb/core 0.15.2.  The setup
function returned before the DB was fully closed, so vitest forks
hit a stale file lock when opening the same DB path, crashing the
lbug-db worker before any test ran.

isolate: false caused native state corruption after 2-3 open/close
cycles in the same fork (vitest-specific, not reproducible in plain
Node.js).  Without it, each file gets its own module scope and the
N-API destructor crash at fork exit is caught by
dangerouslyIgnoreUnhandledErrors.

Also fixes fire-and-forget close() calls in the pool adapter —
try/catch around an async close() never catches rejections; changed
to .catch(() => {}) for proper unhandled-rejection prevention.

Before: 0/8 lbug-db files ran on macOS CI (fork crash).
After:  8/8 pass, 84 files, 3077 tests, zero errors.

* fix: update project index references in AGENTS.md and CLAUDE.md to reflect correct symbol counts and relationships

* feat: enhance lbug adapter with external database support and write operation validation

* feat: create ci-tests workflow for comprehensive test coverage across platforms

* ci: move PR report inline to ci.yml, delete ci-report.yml

The old ci-report.yml used workflow_run which always runs code from
the default branch (main). This meant the PR comment used main's
stale report template that still referenced the old unit/integration
split architecture — causing "Merge coverage reports" failures.

Moving the report inline to ci.yml means it runs from the PR branch
and uses the current report template. The report now shows:
- per-platform status (Ubuntu/Windows/macOS columns)
- unified test counts from the single vitest run
- coverage with base branch (main) delta comparison
- commit SHA for traceability

Also removes the save-pr-meta job since the report no longer needs
a separate workflow_run trigger.

---------

Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-19 08:25:43 +00:00
Zander Raycraft b0f25e216d Merge branch 'main' of https://github.com/zander-raycraft/GitNexus 2026-03-18 17:36:39 -05:00
Zander Raycraft 84f07e83ee test fix for claude call 2026-03-18 17:35:43 -05:00
Gergo Magyar 1e19986ef3 fix(tests): update property edge and write access expectations across multiple language tests 2026-03-18 22:25:30 +00:00
Gergő Magyar 973c7bfbf0 feat: ACCESSES edge type with read/write field access tracking (#372)
* feat: Phase 1 ACCESSES edge type — read tracking from chain resolution

Add ACCESSES relationship type to track field read access during call
chain resolution. When walkMixedChain resolves a field access (e.g.,
user.address.save()), an ACCESSES edge with reason 'read' is emitted
from the calling function to the Property node.

Schema: ACCESSES added to RelationshipType, REL_TYPES, VALID_RELATION_TYPES,
context queries, tools/resources descriptions. Excluded from default
impact BFS to prevent traversal explosion.

Implementation: resolveFieldAccessType now returns FieldResolution with
fieldNodeId. walkMixedChain accepts optional onFieldResolved callback.
makeAccessEmitter factory provides Set-based dedup per source node.

Bug fix: Added Java 'field_access' to FIELD_ACCESS_NODE_TYPES — was
missing, causing extractMixedChain to fail for Java member access.

* feat: Phase 2 ACCESSES write edges — assignment detection across 12 languages

Add tree-sitter query patterns for field write detection (obj.field = value)
across all supported languages: TS/JS, Python, Java, Go, C++, C#, Rust,
PHP, Ruby (setter syntax), Kotlin, Swift.

Processing: Sequential path handles assignment captures inline. Worker
path extracts ExtractedAssignment data for deferred resolution via new
processAssignmentsFromExtracted function.

Bug fix: Kotlin/Swift assignment queries used invalid navigation_expression
wrapper — fixed to match actual directly_assignable_expression AST structure.

Tests: Write access integration tests for TS, Java, Python, Go with
dedicated fixtures. All use strict toBe() assertions.

* test: add unit tests for call-routing, shared type extractors, and symbol-table branches

Add 215 new unit tests across 3 files to increase branch coverage toward
the 23% global threshold (was 21.49%):

- call-routing.test.ts (49 tests): Ruby call routing — require/require_relative,
  include/extend/prepend heritage, attr_accessor properties with YARD types
- shared-type-extractors.test.ts (108 tests): pure string functions —
  extractElementTypeFromString, stripNullable, extractReturnTypeName,
  methodToTypeArgPosition, getContainerDescriptor
- symbol-table.test.ts (+29 tests): Property/fieldByOwner index, metadata
  spread branches, lazy callable index, lookupExactFull shape

* fix: defer write-access resolution to fix Ruby cross-file property timing

Ruby attr_accessor properties are registered during processCalls (not
the parsing phase), so lookupFieldByOwner fails when service.rb is
processed before models.rb. Fix by collecting pending write-access
edges during the file loop and resolving them after all files are done.

Also adds write-access integration tests and fixtures for 7 languages
(C++, C#, JS, Kotlin, PHP, Ruby, Rust), Ruby compound assignment query,
PHP static property write query, and Kotlin property type extraction.

* fix: address PR #372 review — write-access constructor bindings parity and docs

- Add verified constructor bindings fallback to write-access resolution
  in both sequential path (receiverIndex lookup) and worker path
  (constructorBindings param for processAssignmentsFromExtracted),
  closing the read/write ACCESSES edge asymmetry for factory-returned
  receivers
- Clarify inner guard control flow comment in processCalls match loop
- Document Go inc_statement/dec_statement gap in roadmap
- Clarify PHP nullsafe write footnote (invalid syntax, not just untracked)
- Update symbol-table tests for intentional fieldByOwner behavior change
  (Properties without declaredType now indexed for dynamic language
  write-access tracking)
2026-03-19 03:48:04 +05:30
abhigyanpatwariandClaude Opus 4.6 c0b4098c4e fix(web): add missing Kotlin entries to Record<SupportedLanguages> maps
The SupportedLanguages enum includes Kotlin but the web project's
LANGUAGE_QUERIES and languageFileMap Records were missing it, breaking
the Vercel build with TS2741.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-19 02:17:03 +05:30
Gergő Magyar 11a3d0515c feat: Phase 8 field/property type resolution (#354)
* feat: Phase 8 field/property type resolution — resolve chained member access

Add field/property type extraction to the type resolution system so that
chained member access like `user.address.save()` resolves the intermediate
receiver type (`address → Address`) through Property symbols in SymbolTable.

Key changes:
- SymbolTable: add `declaredType` field, `fieldByOwner` O(1) index,
  `lookupFieldByOwner()` method, P0 conditional callableIndex invalidation,
  P2 exclude Properties from globalIndex to prevent namespace pollution
- tree-sitter queries: add `definition.property` for TypeScript, Java, Go
- parse-worker: extract declared types for Property nodes via
  `extractPropertyDeclaredType()`, capture field-access receiver info
- call-processor: add `resolveFieldAccessType()` helper and field-access
  branch in both sequential and worker receiver resolution paths
- Integration tests: new field-types test suite verifying end-to-end
  `user.address.save() → Address#save` resolution

* fix: Go tree-sitter query captures field_declaration not field_declaration_list

Post-review fix: the Go struct field query incorrectly put @definition.property
on field_declaration_list (the list container) instead of field_declaration
(the individual field). Also removed unused `language` parameter from
extractPropertyDeclaredType.

* feat: expand field-type tests to 6 languages, fix Go ownerId and Kotlin navigation_expression

- Add integration test fixtures for Java, C#, Go, Kotlin, PHP (alongside existing TS)
- Fix Go: add type_declaration handling in findEnclosingClassId for struct fields
  (field_declaration → field_declaration_list → struct_type → type_spec → type_declaration)
- Fix Kotlin: add navigation_expression handling in field-access resolution
  (Kotlin uses navigation_expression + navigation_suffix, not member_expression)
- Add extractMemberAccessParts helper in call-processor for cross-language member access
- All 24 field-type tests pass across 6 languages, 181 Go+Kotlin tests pass with no regressions

* refactor: split HAS_METHOD into HAS_METHOD + HAS_PROPERTY edge types

Property nodes now use HAS_PROPERTY edges instead of HAS_METHOD, giving
the graph schema proper semantic separation between methods and fields.

- HAS_METHOD: Method, Constructor, Function (when inside a class)
- HAS_PROPERTY: Property nodes (class fields, struct fields, attributes)

MRO processor only reads HAS_METHOD — properties correctly excluded from
method resolution order. Impact analysis accepts both edge types.

Updated 12 files: graph types, schema, tools docs, parse-worker,
parsing-processor, call-processor, and 6 test files.

* fix(test): update security test to expect 7 VALID_RELATION_TYPES (added HAS_PROPERTY)

* test: add unit tests for Phase 8 SymbolTable features (39 tests, up from 19)

Cover all new branches: declaredType metadata, Property exclusion from
globalIndex, conditional callableIndex invalidation, lookupFieldByOwner
(happy path + edge cases), lookupFuzzyCallable filtering, and clear()
with fieldByOwner. Fixes branch coverage threshold (21.8% → 23%+).

* feat: Phase 8B mixed field+method chain resolution, C++/Rust chain fixes

Unify field and method chain resolution into a single `extractMixedChain`
walker that handles interleaved patterns like `svc.getUser().address.save()`.
Fix C++ chain calls (tree-sitter-cpp `field_expression` uses `argument` not
`object`), Rust unit struct instantiation (`let svc = TypeName;`), and add
stdlib passthrough for `unwrap()`/`clone()`/`expect()` in chain loops.

Key changes:
- Replace `receiverCallChain` + `receiverFieldAccess` with unified
  `receiverMixedChain: MixedChainStep[]` on ExtractedCall
- Add `extractMixedChain` in utils.ts (handles both call_expression and
  field_expression nodes, including C++ `argument` field)
- Add `TYPE_PRESERVING_METHODS` set for stdlib identity operations
- Add C++ inline method double-indexing guard in parsing-processor.ts
  and parse-worker.ts
- Add Rust unit struct recognition in type-extractors/rust.ts
- Split field-types.test.ts into per-language test files
- Add ts-mixed-chain fixture and integration tests
- Resolve rust.test.ts todo: Option<T>.unwrap().save() now works
- Update roadmap: Phases 7+8 complete, Phase 9 is next

* fix: Python declaredType extraction and sequential-path property registration

- Move @definition.property capture from expression_statement to assignment
  node in Python queries so Strategy 1 childForFieldName('type') succeeds
- Pass item.declaredType through ctx.symbols.add in sequential call-processor
  path, matching worker path behavior (fixes Ruby YARD declaredType drop)
- Add Python chain resolution integration test (user.address.save → Address#save)
- Update Rust/Python status in roadmap and system docs to reflect actual coverage

* fix: Python/Ruby field type disambiguation and Rust chain test

Three fixes from PR #354 third review:

1. Python typed_parameter name extraction: tree-sitter-python's
   typed_parameter uses positional children for the name, not a named
   field. TypeEnv and extractParameter now fall back to firstNamedChild.

2. Ruby/Python call-step field resolution: Ruby's AST uses `call` nodes
   for both property access and method calls. The chain walker now tries
   resolveFieldAccessType before resolveCallTarget for call steps, so
   attr_accessor properties resolve via declaredType.

3. Rust chain resolution test: added missing integration test asserting
   user.address.save() resolves to Address#save.

Also splits C/C++ and TS/JS columns in type-resolution-system.md
language matrix with footnotes for accuracy.

1062 resolver integration tests passing, 0 failures.

* refactor: Phase 8 code review cleanup — extract walkMixedChain, fix MCP agent gaps

- Extract duplicated chain resolution loop into shared walkMixedChain() helper,
  eliminating ~60 lines of copy-pasted code between sequential and worker paths
- Add returnType to ResolveResult, removing redundant lookupFuzzy+find per chain step
- Fix context() tool to include HAS_METHOD, HAS_PROPERTY, OVERRIDES in queries
  so agents can discover class members
- Fix p.declaredType Cypher example (column doesn't exist) → p.description
- Add HAS_METHOD, HAS_PROPERTY, OVERRIDES to schema resource
- Document HAS_METHOD/HAS_PROPERTY in impact tool description
- Delete dead code extractMemberAccessParts (superseded by extractMixedChain)
- Replace any with SyntaxNode on extractPropertyDeclaredType
- Add Rust deep-field-chain test (5 tests), Java mixed-chain (4), Go mixed-chain (4)
- All 1075 tests pass (13 new, 0 regressions)

* refactor: type SymbolDefinition.type as NodeLabel, add O(1) receiver index

- Change SymbolDefinition.type from string to NodeLabel union (35 members)
  across symbol-table.ts, parse-worker.ts, parsing-processor.ts — compiler
  now enforces correctness at all comparison/assignment sites
- Replace O(N*M) linear scan in lookupReceiverType with pre-built
  ReceiverTypeIndex (Map<funcName, Map<varName, Entry>>) for O(1) lookups
  with proper ambiguity handling and file-level fallback
- All 1075 tests pass, 0 regressions

* fix: capture C++ pointer/ref fields, Kotlin data class props, PHP constructor promotion

Add tree-sitter query patterns for three previously missed property declaration
forms: C++ pointer/reference member fields (Address* addr; Address& ref;),
Kotlin primary constructor val/var parameters (data class User(val name: String)),
and PHP 8.0+ constructor property promotion (public Address $address).

Fix "10 languages" off-by-one in docs (Ruby is single-level only, not deep chain).
Update Python feature matrix cell from No* to Yes* after 31b95f0 fix.

11 new integration tests with per-language fixtures verify property capture,
HAS_PROPERTY edge emission, and field-access chain resolution.
2026-03-18 18:47:33 +00:00
Berk Demirci e0a6c40b45 Fix undefined parsing error on languages missing from call routers (#364)
* fix: mapping all supported languages to callRouters to fix undefined apply error (#352)

* fix(cli): remove duplicate Swift/Kotlin keys in callRouters causing TS1117

* refactor: remove runtime callRouter fallbacks, rely on Record<SupportedLanguages> compile-time enforcement

* refactor: use 'satisfies' keyword for callRouters compile-time enforcement
2026-03-18 14:52:54 +00:00
Chirag Nighutandchirag-nighut aa1bab597b feat: add Python enumerate() for-loop support with nested tuple patterns (#356)
- Handle `for i, k, v in enumerate(d.items())` — flat pattern
- Handle `for i, (k, v) in enumerate(d.items())` — nested tuple_pattern
- Handle `for (k, v) in enumerate(users)` — parenthesized tuple as top-level

Extract helper functions for cleaner code:
- `extractMethodCall()` — deduplicate method call parsing
- `collectPatternIdentifiers()` — recursively collect identifiers from patterns

Add unit tests for TypeEnv and integration tests verifying CALLS edges.

Made-with: Cursor

Co-authored-by: chirag-nighut <chiragnighut@gmail.com>
2026-03-18 13:05:10 +00:00
Hazem 60ede20a11 fix: MCP server crashes under parallel tool calls (#326) (#349)
* fix: MCP server crashes under parallel tool calls (#326)

* fix: ensure full connection pool is pre-created to avoid race conditions during query execution

* fix: improve graceful shutdown handling with exit codes

* fix: resolve critical concurrency bugs in connection pool init

- Add initPromises dedup map to prevent double-init race when parallel
  tool calls trigger initLbug for the same repoId simultaneously
- Move pool.set() after FTS load so concurrent checkout can't grab a
  connection mid-async-init (FTS race on available[0])
- Replace lazy createConnection growth path with integrity error — pool
  is pre-warmed, lazy creation would silence stdout during active queries
- Add preWarmActive flag so watchdog timer skips stdout restore during
  the synchronous pre-warm loop
- Unify stdout capture: server.ts imports realStdoutWrite from
  lbug-adapter instead of capturing its own copy

* test: add connection pool parallel stability tests

7 integration tests covering concurrent query safety, waiter queue
overflow, stdout.write restoration, connection leak detection, initLbug
deduplication, atomic pool visibility, and mixed query types.

* fix: run LadybugDB tests sequentially via vitest projects config

Vitest's projects feature splits test files into two groups: lbug-db
(fileParallelism: false) and default (parallel). This prevents native
mmap file-lock conflicts on Windows without requiring the CI shell loop
locally.

* test: add enrichment Promise.all regression test for #292/#316

Verifies that 3 concurrent queries via Promise.all (the exact pattern
from the impact command's enrichment phase at local-backend.ts:1415)
complete without SIGSEGV on a pre-warmed connection pool.
2026-03-18 13:02:01 +00:00
Gergo Magyar fb5270c260 chore: bump version to 1.4.6 and update CHANGELOG 2026-03-18 08:47:45 +00:00
Gergő Magyar 604b575e4b feat: Phase 7 type resolution — return-aware loop inference & PHP class-property iterables (#341)
* feat(type-resolution): Phase 7.1+7.2 foundation — ReturnTypeLookup, context object, pendingCallResults

- Move extractReturnTypeName + helpers from call-processor.ts to type-extractors/shared.ts
  (breaks circular import risk: call-processor → type-env → type-extractors → call-processor)
- Add SymbolTable.lookupFuzzyCallable(name) — lazy callable-only index, O(1) per call,
  invalidated on add(); avoids per-call .filter() on lookupFuzzy results
- Add ReturnTypeLookup interface (conservative: undefined when 0 or 2+ callables match)
- Add ForLoopExtractorContext interface — replaces 4 positional params with context object;
  update all 10 language extractor implementations (go, ts, py, jvm×2, cs, rs, rb, php, c-cpp)
- Add PendingAssignment discriminated union (kind: 'copy' | 'callResult');
  update PendingAssignmentExtractor in all 9 language extractors that implement it
- Wire buildTypeEnv: build ReturnTypeLookup from optional symbolTable; split pendingAssignments
  into pendingCopies + pendingCallResults; add Tier 2b call-result propagation loop
- Update call-processor.test.ts to import extractReturnTypeName from shared.ts

* feat(type-resolution): Phase 7.3 — call_expression iterables in for-loop extractors (7 languages)

Extends for-loop type extraction in all 7 typed-iteration languages to
resolve element types when the iterable is a direct function call.

**New capability**: `for (var u : getUsers())` in Java, `for u in get_users()`
in Python, `for user in getUsers()` in TypeScript, etc. now resolve
`u`/`user` to the callee's return element type via lookupRawReturnType +
extractElementTypeFromString.

Changes per language:
- types.ts: extend ReturnTypeLookup with lookupRawReturnType (raw return
  string for container-type extraction); update ForLoopExtractorContext
  with returnTypeLookup field
- type-env.ts: implement lookupRawReturnType on the concrete ReturnTypeLookup
  built in buildTypeEnv (same guards as lookupReturnType, no extractReturnTypeName)
- go.ts: call_expression branch in range_clause — identifier func or
  selector_expression method; existing isChannelType guards updated
- typescript.ts: identifier fn branch inside call_expression handler
- python.ts: identifier fn branch inside call handler
- jvm.ts (Java): method_invocation without object field in enhanced_for_statement
- jvm.ts (Kotlin): simple_identifier callee branch in call_expression node
- csharp.ts: identifier fn branch in invocation_expression handler
- rust.ts: identifier func branch in call_expression handler (alongside
  existing field_expression/method-call path)

All branches follow the same conservative pattern:
  lookupRawReturnType(callee) → extractElementTypeFromString → bind loop var

* feat(type-resolution): Phase 7.4 — PHP \$this->property iterable via @var class property scan

Adds Strategy C to PHP's extractForLoopBinding for the pattern:

  foreach (\$this->property as \$item)

when Strategy A (resolveIterableElementType) and Strategy B (scopeEnv lookup)
both fail to find the element type.

Strategy C: when the iterable is a member_access_expression with object '$this',
walk up the AST to the enclosing class_declaration, scan its declaration_list
for a property_declaration whose variable_name matches the property, and extract
the element type from:
  1. PHPDoc @var annotation on a preceding comment sibling (/** @var User[] */)
  2. PHP 7.4+ native type field (e.g. UserRepo \$repo — skips generic 'array')

This eliminates the @param workaround that was previously required in the
php-foreach-member-access fixture (which used @param User[] \$users on the method
to populate the method's scopeEnv with a \$users binding).

New helpers in php.ts:
- PHPDOC_VAR_RE: regex for @var extraction
- extractClassPropertyElementType: reads @var or native type from a property_declaration
- findClassPropertyElementType: scans class body for a named property

Tests added (type-env.test.ts):
- PHP: resolves from @var User[] without @param workaround
- PHP: conservative — no binding for unknown property
- PHP: multi-class file — both classes resolve independently

Fixture updated (php-foreach-member-access/App.php):
- Removed the @param User[] \$users workaround from processMembers()
- Test now validates the natural class-property-based resolution path

* docs: mark Phase 7 complete in type-resolution-roadmap.md

Records that 7A (call_expression iterables, 7 languages), 7B (PHP
$this->property via @var scan), and 7C (ReturnTypeLookup + context object)
are all shipped. Adds implementation notes and strikethroughs on resolved
language-specific gaps.

* fix(docs): update project references to feat-phase7-type-resolution in AGENTS.md and CLAUDE.md

* feat(type-resolution): Phase 7.5 — PHP call_expression foreach + integration tests for 7 languages

Add integration test coverage for Phase 7.3's call_expression iterable
resolution across all 7 languages (Go, TypeScript, Python, Java, Kotlin,
PHP, Rust). Each test creates a fixture with competing User/Repo classes
that both define save(), then verifies for-loop iteration over a function
call's return value resolves to the correct class.

PHP was missing function_call_expression support in its for-loop extractor.
Three changes fix this:
- php.ts extractForLoopBinding: handle function_call_expression and
  member_call_expression iterables via returnTypeLookup
- php.ts normalizePhpReturnType: preserve array notation (User[]) in
  SymbolTable so lookupRawReturnType returns useful container types
- parse-worker.ts + parsing-processor.ts: upgrade uninformative AST
  return types (array, iterable) with PHPDoc @return annotations

35 new integration tests (5 per language), 2525 total tests passing.

* fix(type-resolution): address PR #341 review findings — PHP asymmetry + dormant infrastructure docs

- Replace normalizePhpType with extractElementTypeFromString in PHP call-expression
  foreach paths, aligning with all 6 other language extractors and preventing
  incorrect binding of bare non-container types like User
- Add NOTE comments clarifying pendingCallResults Tier 2b is infrastructure-ready
  but no extractor populates it yet
- Expand Go channel-type comments explaining why non-channel assumption is safe

* fix(type-resolution): address verification review — docs accuracy + PHP fallback guard

- Roadmap lines 86/100: correct pendingCallResults from "active" to "dormant infrastructure (Phase 9)"
- type-resolution-system.md line 363: update to reflect Phase 7.3 loop inference is delivered
- type-resolution-system.md line 409: clarify for-loop call-expression resolution (done) vs general assignment propagation (pending)
- php.ts:127: add declaration_list type guard on fallback to prevent silent wrong results
2026-03-18 08:39:38 +00:00
Gergo Magyar 02dfab578c fix(test): add --repo to CLI e2e tool tests for multi-repo environment 2026-03-18 08:12:25 +00:00
Gergo Magyar 1326490a5b fix(workflow): use prefixed temporary branch name for fork PRs to prevent overwriting real branches 2026-03-18 07:40:53 +00:00
林 駿甫 (Shunsuke Hayashi) b48cfe9894 fix(impact): return structured error + partial results instead of crashing (#321) (#345)
* fix(impact): return structured error + partial results instead of crashing (#321)

- Wrap impact() in try-catch to return structured error JSON instead of
  process crash (SIGSEGV/exit 139)
- Extract core logic to _impactImpl() for clean error boundary
- Break out of depth traversal loop on query failure, return partial
  results collected so far (previously silently swallowed errors)
- Add 'partial' flag to response when traversal was interrupted
- Add try-catch in CLI impactCommand with structured error output
- Improve formatImpactResult to show suggestion text and partial warning
- Add 3 new unit tests for error/suggestion/partial scenarios

Fixes #321

* fix: address review feedback — 4 bugs from @claude review

Per @claude's review (requested by @magyargergo):

- [BUG 1] Consistent target field shape: error responses now return
  {name: string} instead of raw string, matching success response schema
- [BUG 2] Remove misleading partial:true from total-failure responses
  (partial is only meaningful when some depth levels succeeded)
- [BUG 3] Move getBackend() inside try-catch in impactCommand so
  backend init failures return structured JSON instead of crashing
- [BUG 4] Safe error message extraction: use instanceof Error check
  to handle thrown strings correctly (err?.message is undefined for
  non-Error thrown values)
- [MINOR] Add radix argument to parseInt (10)

* test: add integration tests for impact error handling (#321)

Per @claude's recommendation (requested by @magyargergo):

- impact: structured error for unknown symbol (no crash)
- impact: error response has consistent {name: string} target shape
- impact: partial:true only set when some results were collected

Tests use existing withTestLbugDB + seeded graph fixture.
2026-03-18 06:45:43 +00:00
林 駿甫 (Shunsuke Hayashi) c1703fc0a9 fix(cli): write tool output to stdout via fd 1 instead of stderr (#324) (#346) 2026-03-18 06:21:32 +00:00
Karesansui 480fae933b fix(impact): add HAS_METHOD and OVERRIDES to VALID_RELATION_TYPES (#350) 2026-03-18 06:01:02 +00:00
林 駿甫 (Shunsuke Hayashi) 3879490817 fix: add postinstall permission fix for CLI and hook scripts (#330) (#348) 2026-03-18 05:41:37 +00:00
Gergo Magyar 50dbd03779 chore: add .worktrees/ to .gitignore 2026-03-17 21:39:09 +00:00
Gergo Magyar 1003d8b6a5 test: add coverage for perf optimizations — fastStripNullable, skipGraphPhases, AST pruning
- 6 new unit tests for fastStripNullable branches (simple id, nullable union, bare keyword)
- 4 new integration tests for skipGraphPhases pipeline option
- Tests for SKIP_SUBTREE_TYPES and interestingNodeTypes code paths
2026-03-17 17:31:24 +00:00
Gergo Magyar 74b9701509 chore: bump version to 1.4.5, add CHANGELOG.md 2026-03-17 17:18:35 +00:00
Gergő Magyar f0132c1077 feat: Phase 6 type resolution — for-loop Tier 1c, pattern matching, container descriptors, 10-language coverage (#318)
* feat: Phase 6 type resolution — pattern matching, for-loop Tier 1c, coverage completion

- Add patternBindingNodeTypes gate to LanguageTypeConfig for 50% perf improvement
- Expand ForLoopExtractor signature with optional declarationTypeNodes + scope
- Add extractElementTypeFromString shared utility for container type parsing
- Python match/case: extractPatternBinding for `case User() as u:` pattern
- C# refactor: move is_pattern_expression from extractDeclaration to extractPatternBinding
- Ruby: add extractPendingAssignment for assignment chain propagation
- TS/JS: add for-loop Tier 1c for `for (const user of users)` with User[] inference
- Python: add for-loop Tier 1c for `for user in users:` with type annotation inference
- Go: add for-loop Tier 1c for `for _, user := range users` with []User inference
- Fix 'Property' as any stale cast in call-processor.ts
- Add dual return-type string length cap (2048 pre-cap, 512 post-cap)
- Add chain call integration tests for C#, Go, Rust, Python, JS, C++
- Add Python match/case integration test fixtures
- 27 new extractElementTypeFromString unit tests
- 3 for-loop edge cases skipped (declarationTypeNodes scope key lookup)

* fix: address code review findings for Phase 6

- Add missing patternBindingNodeTypes to C# typeConfig (perf gate)
- Add 2048-char input length guard to extractElementTypeFromString
- Skip Python match/case integration tests (call extraction needs query updates)

* reorganise

* fix: Phase 1 bug fixes — Go range semantics, typed_parameter, bracket depth

- Go single-var range correctly returns early for slices/maps (index, not element)
- Go single-var range on channels correctly resolves element type
- Added map_type and channel_type to extractGoElementTypeFromTypeNode
- Added isChannelType helper for channel detection before skip decision
- Added 'typed_parameter' to TYPED_PARAMETER_TYPES for Python annotated params
- Fixed bracket depth tracking in extractElementTypeFromString — only match
  selected closeChar at depth 0, return undefined for mismatched brackets
- Un-skipped 3 prematurely skipped tests (TS local const, Python List/Sequence)
- Added tests for map range, single-var range semantics, bracket edge cases

* refactor: Phase 2 architecture — shared helper, required params, decoupled type nodes

- Extract resolveIterableElementType shared helper in shared.ts implementing
  3-strategy fallback (declarationTypeNodes → scopeEnv string → AST walk)
- Refactor TS, Python, Go extractors to use shared helper (eliminates 3x duplication)
- Make ForLoopExtractor params required (aligned with PatternBindingExtractor)
- Update Java, Kotlin, C# extractor signatures to accept required params
- Decouple declarationTypeNodes from scopeEnv — capture raw type annotation
  nodes BEFORE extractDeclaration for container types (User[], []User, List[User])
- Hybrid approach: direct name extraction + keysBefore fallback for multi-declarator
- Document declarationTypeNodes invariant change (superset of scopeEnv)

* feat: Phase 3 partial — Rust for-loop + C# var foreach Tier 1c

- Rust: add extractForLoopBinding with for_expression support
  - Handles &users, &mut users via reference_expression unwrapping
  - extractRustElementTypeFromTypeNode: generic_type, reference_type, slice/array
  - findRustParamElementType: AST walk with reference/mut pattern unwrapping
  - 4 unit tests (Vec<User>, &[User], range expr negative, no-annotation negative)

- C#: upgrade foreach to handle var (implicit_type) via Tier 1c
  - extractCSharpElementTypeFromTypeNode: generic_name, array_type, nullable_type
  - findCSharpParamElementType: AST walk to method_declaration parameters
  - 3 unit tests (var foreach, explicit type regression, no-annotation negative)

* feat: Phase 3 complete — all language gaps + pattern matching

Kotlin Tier 1c:
- Unannotated for-loop resolves via shared helper
- extractKotlinElementTypeFromTypeNode handles type_projection unwrapping
- findKotlinParamElementType walks to function_declaration

Java Tier 1c:
- var foreach resolves via shared helper
- extractJavaElementTypeFromTypeNode handles generic_type, array_type
- findJavaParamElementType walks to method_declaration

TypeScript:
- readonly User[] unwrapped via readonly_type → array_type recursion

C# switch patterns:
- declaration_pattern added to patternBindingNodeTypes
- extractPatternBinding handles standalone declaration_pattern (switch case/expr)

Rust match arms:
- match_arm added to patternBindingNodeTypes
- extractPatternBinding extended with match_arm → match_expression parent traversal

Python:
- as_pattern tries childForFieldName('alias') before positional fallback

Tests: 237 pass (was 224), 13 new tests added

* feat: Phase 4 — known limitation tests, match arm fix, final verification

- Fix Rust match_arm pattern extraction: unwrap match_pattern to get
  tuple_struct_pattern inside (tree-sitter-rust wraps in match_pattern node)
- Add first-writer-wins regression test for match arm scope leakage
- Add 5 documented skip tests for known limitations:
  - TS destructured for-of (tuple destructuring)
  - Python tuple unpacking in for-loops
  - TS instanceof narrowing (block-level scoping)
  - Rust for with .iter() (method call iterable)
  - Ruby block parameters (closure param inference)

Final: 238 passed, 5 skipped (documented limitations), tsc clean

* test: integration tests for all Phase 6 language gaps + fix Rust param pattern field

Integration test fixtures and tests (30 new tests, all with exact match + negative):

Rust for-loop (5 tests):
- for user in &users with Vec<User> → User#save, negative Repo#save
- for repo in &repos with Vec<Repo> → Repo#save, negative User#save

Rust match arm (5 tests):
- match opt { Some(user) => user.save() } → User#save, negative Repo#save
- if let Ok(repo) = res → Repo#save, negative User#save

C# var foreach (5 tests):
- foreach (var user in users) with List<User> → User#Save, negative Repo#Save
- foreach (var repo in repos) with List<Repo> → Repo#Save

C# switch pattern (4 tests):
- is User user → User#Save, case Repo repo → Repo#Save

Kotlin unannotated for (4 tests):
- for (user in users) with List<User> → user.save, negative repo.save

Go map range (3 tests):
- for _, user := range userMap with map[string]User → User#Save, negative

TypeScript readonly (4 tests):
- for (const user of users) with readonly User[] → user.save, negative

Bug fix: type-env.ts parameter branch now falls back to childForFieldName('pattern')
for Rust parameters (Rust uses 'pattern' not 'name' for parameter names)

* test: add assertion bodies to known limitation skip tests

Convert empty skip test stubs to proper tests with parse/buildTypeEnv/expect
assertions following the codebase convention (e.g., call-processor.test.ts:319).
Each skip test now documents the exact expected behavior, so removing .skip
will cause a meaningful failure when the limitation is eventually fixed.

Also clarify Python integration skip tests as call-extraction issues (not
type-env) and Swift integration skips as build-dep issues (self/super
resolution code already exists in type-env.ts).

* feat: resolve 4 known limitation skip tests + method-aware type arg selection

Unskip 4 of 5 type-env known limitations with full integration test coverage:

1. TS destructured for-of: handle array_pattern by binding last named child
   to element type. Fix Map<K,V> to return last generic arg (value type).
2. Python dict.items() loop: handle `call` iterables + `pattern_list` left
   side. Fix dict[K,V] extraction via type_parameter with last-arg heuristic.
   Unwrap `type` wrapper in extractPyElementTypeFromAnnotation.
3. TS instanceof narrowing: add extractPatternBinding for binary_expression
   with positional child access. First-writer-wins (not block-scoped).
4. Rust .iter() for-loops: handle call_expression in for_expression value
   node by extracting receiver from field_expression.

Method-aware type arg resolution:
- Add TypeArgPosition ('first'|'last') to resolveIterableElementType
- .keys()/.keySet()/.Keys → first type arg (key); all else → last (value)
- Thread position through all 3 strategy callbacks in TS/Rust/Python
- Add predefined_type to extractSimpleTypeName for TS primitives (string etc)

New fixtures: rust-iter-for-loop, typescript-destructured-for-of,
typescript-instanceof-narrowing, python-dict-items-loop.
248 unit tests pass (6 new), 1 skip (Ruby block params).

* feat: container descriptor table for generic type arg resolution

Replace simple KEY_METHODS heuristic with CONTAINER_DESCRIPTORS table
that maps 30+ container types across all languages to their type parameter
semantics per access method.

Key improvements:
- Container-aware resolution: HashMap.iter() correctly yields V (arity 2),
  while Vec.iter() yields T (arity 1) — same method, different semantics
- Cross-language coverage: Map/HashMap/BTreeMap/dict/Dict/Dictionary/
  ConcurrentHashMap + List/Vec/Set/HashSet/Queue/Deque/Stack etc.
- Method categorization: keyMethods (keys/keySet/Keys) vs valueMethods
  (values/get/pop/iter/first/last) per container type
- Fallback for unknown containers: still uses method name heuristic,
  so MyCache<K,V>.keys() correctly returns first arg
- Exported getContainerDescriptor() for future heritage-chain lookups

Each language extractor now passes containerTypeName from scopeEnv to
methodToTypeArgPosition for descriptor-aware resolution.

252 unit tests pass (4 new descriptor tests), 1 skip (Ruby).

* feat: method-aware for-loop extractors + integration tests for all languages

Upgrade 4 existing extractors + create 3 new ones for full cross-language
coverage of call_expression iterables and container descriptor resolution:

Upgraded (add call expr iterable + methodToTypeArgPosition):
- Java: method_invocation (data.keySet(), data.values())
- Kotlin: navigation_expression + call_expression (data.keys, data.values())
- C#: member_access_expression + invocation_expression (data.Keys, data.Values)
- Go: TypeArgPosition threading for Go 1.18+ generics

New for-loop extractors:
- C++: for_range_loop with auto& unwrapping, template_type + qualified_identifier
  (std::vector<User>) extraction, explicit vs auto type handling
- PHP: foreach_statement with simple/key-value/by-reference forms, PHPDoc
  @param priority over AST array type
- Ruby: for-in with YARD @param type resolution via comment parsing

Integration test fixtures + tests for all 6 languages:
- java-map-keys-values (Map.values() + List iteration)
- kotlin-map-keys-values (HashMap.values + List iteration)
- csharp-dictionary-keys-values (Dictionary.Values foreach)
- cpp-range-for (auto& + const auto& range-based for)
- php-foreach-loop (foreach with PHPDoc @param User[])
- ruby-for-in-loop (for-in with YARD @param Array<User>)

Bugs fixed during integration testing:
- C++: qualified_identifier (std::vector) not unwrapped to template_type
- PHP: extractParameter overwrote PHPDoc-derived types with bare 'array'

252 unit tests pass, 201 integration tests pass across 6 languages.

* fix: update extractElementTypeFromString tests for last-arg default

TypeArgPosition change (default 'last') broke 5 existing tests expecting
first arg from multi-arg generics. Updated expectations and added explicit
pos='first' tests for key type extraction.

* fix: rename C++ fixture files to correct case for case-sensitive CI

On case-sensitive filesystems (Linux/macOS CI), git tracked both the old
lowercase files (app.cpp, user.h) and the new uppercase files (App.cpp,
User.h) as separate files. The pipeline processed both, causing the old
app.cpp (with explicit User& type) to interfere with the new auto& test.

Removes old lowercase entries and re-adds with uppercase casing to match
the #include directives in the fixture.

* feat: PR #318 review findings — pattern bindings, member access iterables, structured bindings

Address all 7 genuine gaps identified in PR #318 deep code review:

- Kotlin: add extractKotlinPatternBinding for when/is (type_test AST node)
  with allowPatternBindingOverwrite for smart-cast semantics
- Java: add type_pattern branch for Java 17+ switch pattern variables
- TypeScript: explicit object_pattern skip in for-of (no false bindings)
- Cross-language: member access iterables (self.users, this.users, repo.users)
  across all 10 language extractors
- C++: structured_binding_declarator handling in range-for (last-child heuristic)
- Rust: closure_parameter added to TYPED_PARAMETER_TYPES
- PHP: normalizePhpType handles angle-bracket generics (Collection<User>)

Code review fixes applied:
- Remove 4 debug console.log statements (c-cpp.ts, call-processor.ts)
- Hoist KNOWN_CONTAINER_PROPS to module scope (csharp.ts)
- Guard keysBefore allocation behind typeNode check (type-env.ts)
- Add depth limits (50) to 7 recursive type extraction functions
- Add 2048-char length cap to extractSimpleTypeName
- Fix PHP/Ruby missing typeArgPos parameter in resolveIterableElementType

Integration test fixtures: kotlin-when-pattern, java-switch-pattern,
cpp-structured-binding, typescript-member-access-for-loop,
python-member-access-for-loop

* fix: position-indexed when/is bindings, Kotlin param extraction, HashMap.values for-loop

Three root causes for failing Kotlin integration tests:

1. When/is multi-arm resolution: flat scopeEnv stored only the last arm's
   type (last-writer-wins). Added PatternOverrides with AST range indexing
   so each when arm resolves to its narrowed type independently.

2. HashMap.values for-loop: navigation_expression without call_suffix was
   classified as bare property access (iterableName='values' instead of
   'data'). Now tries object-as-iterable + property-as-method first, with
   fallback to property-as-iterable for this.users patterns.

3. Kotlin parameter extraction: tree-sitter-kotlin parameter nodes use
   positional children (simple_identifier, user_type) not named fields
   (name, type). Added fallback to findChildByType in both
   extractKotlinParameter and extractTypeBinding.

Integration tests added for .keys/.values/Set/MutableMap iteration,
3-arm when/is, multi-call within arms, and when+else branch.

* feat: enhance PHP type resolution for generics and member access in foreach loops

* feat: Phase 6.1 type resolution gap closure — container descriptors, recursive_pattern, class fields

Add 13 missing container type descriptors (Collection, MutableMap, Stream, SortedSet, etc.)
to CONTAINER_DESCRIPTORS for correct element type extraction across C#, Kotlin, and Java.

Extend C# pattern binding to handle recursive_pattern (obj is User { Name: "Alice" } u)
in both is-expression and switch expression contexts.

Add TypeScript class field declaration support (public_field_definition) so for-loop
iteration over this.fieldName resolves element types from class field type annotations.
Includes file-scope fallback in resolveIterableElementType and nested member_expression
handling for this.field.method() patterns.

* docs: add type resolution system documentation with roadmap

Covers the full architecture, resolution tiers (0-2), scope model,
language feature matrix, container descriptors, pipeline integration,
and the Phase 7-9 roadmap for cross-scope propagation, field-type
resolution, and return-type-aware binding.

* feat: Phase 6.2 review findings — C# nested member foreach, C++ deref range-for, Java field_access

Close two gaps found during fourth-pass review of PR #318:

- C# foreach (var user in this.data.Values): nested member_access_expression
  now extracts intermediate property name for scopeEnv lookup
- C++ for (auto& user : *ptr): pointer_expression dereference now recognized
  as range-for iterable

Root causes fixed in shared infrastructure:
- extractSimpleTypeName: add template_type (C++) and generic_name (C#)
- extractGenericTypeArgs: add generic_name for consistency
- type-env.ts: unwrap variable_declaration wrapper in field_declaration
  for declarationTypeNodes capture (zero-allocation manual loop)

Additional review findings addressed:
- Java: add field_access handler for this.data.values() in method_invocation
- C++ pointer_expression: document limitation (*identifier only)
- TypeScript: fix stale comment about property_identifier

All 525 tests pass (278 unit + 247 integration).

* perf: optimize type resolution pipeline — worker threshold, skip graph phases, AST pruning

- Skip worker pool creation for small repos (<15 files or <512KB) — saves 100-400ms
- Add skipGraphPhases option to runPipelineFromRepo to skip MRO/community/process phases
- Add conservative SKIP_SUBTREE_TYPES for leaf-only AST nodes (string, comment, number)
- Pre-compute interestingNodeTypes set — single Set.has() replaces 3 checks per node
- Add fastStripNullable — skip full stripNullable for simple identifiers (90%+ case)
- Replace .children?.find() with manual for loops in extractFunctionName (no array alloc)
- Add hookTimeout: 120000 to vitest.config.ts for CI beforeAll hooks

* fix: review findings — remove template_string from SKIP_SUBTREE_TYPES, handle bare nullable keywords

- Remove template_string and concatenated_string from SKIP_SUBTREE_TYPES
  (template literals contain interpolated expressions with typed code)
- Add FAST_NULLABLE_KEYWORDS check to fastStripNullable for behavioral
  parity with stripNullable on bare null/undefined/void/None/nil
- Add explanatory comment on extractPendingAssignment scopeEnv guard

* feat: add type resolution system and roadmap documentation
2026-03-17 17:10:22 +00:00
Chirag Nighutandchirag-nighut f6b92d4f13 fix(resolver): fix for same-directory python imports (#328)
* fix(resolver): prefer same-directory file for Python bare imports

Python's sys.path searches the importing script's own directory first,
so `import user` from services/auth.py should resolve to services/user.py
even if models/user.py was indexed first in the suffix index.

Add a proximity check in resolveImportPath that consults the existing
dirMap index (O(1)) before falling back to global suffix matching, for
single-segment bare Python imports only.

Made-with: Cursor

* refactor(resolver): replace dirMap scan with O(1) allFiles.has() for proximity check

The previous implementation used index.getFilesInDir() + siblings.find()
which had two issues:
- dirMap stores all suffix levels, so getFilesInDir('services') matched
  files from every directory named 'services/' across the repo — false
  positives in monorepos
- siblings.find() was an O(n) linear scan despite the O(1) claim

Replace with a direct allFiles.has(importerDir + '/' + name + '.py') lookup.
allFiles is a Set<string> of full repo-relative paths, so the lookup is
truly O(1) and exact — no suffix ambiguity possible.

Also fixes: dead code (the '.rb' branch was unreachable since the outer if
gates on Python), and Windows backslash handling via normalize before split.

Made-with: Cursor

* test: remove flag-based demo from unit tests

Made-with: Cursor

* fix(resolver): cover package __init__.py in proximity check and add end-to-end CALLS test

- Also try importerDir/name/__init__.py as a second O(1) candidate so that
  `import user` resolves to services/user/__init__.py when the target is a
  package rather than a bare module file
- Add unit tests for package proximity, __init__.py fallback, and Windows
  backslash path handling
- Add end-to-end CALLS assertion to the bare-import integration test:
  svc.execute() must resolve to UserService#execute in services/user.py,
  proving the fix propagates correctly through the type inference pipeline

Made-with: Cursor

* refactor: extract Python import resolution into resolvers/python.ts

- Move PEP 328 relative import and proximity-based bare import logic
  from standard.ts into a dedicated resolvers/python.ts (resolvePythonImport)
- Dispatch Python imports from resolveLanguageImport in import-processor.ts,
  consistent with how Ruby, PHP, and other languages are handled
- standard.ts is now language-agnostic (TS/JS aliases, Rust paths, suffix fallback)
- Add inline comment on __init__.py vs .py resolution order edge case
- Update unit tests to call resolvePythonImport directly

Made-with: Cursor

* docs: add PEP 302/328/451 references to python.ts comments

Made-with: Cursor

* fix(python): address reviewer comments on PEP compliance

- Guard dirParts.pop() against over-traversal: return null when dot
  count exceeds directory depth, matching CPython's ImportError for
  'attempted relative import beyond top-level package' (PEP 328)
- Swap __init__.py / .py check order to match CPython's finder
  precedence (PEP 451 §4); coexistence is physically impossible so
  order only matters for spec compliance
- Fix overstated PEP 302 comment: proximity check is a static
  heuristic, not a sys.path[0] implementation
- Acknowledge namespace package gap (PEP 420) in docstring
- Add unit test for over-traversal guard

Made-with: Cursor

* test(python): document namespace package resolution behaviour

Add two unit tests for PEP 420 namespace packages (directory with no
__init__.py): bare import returns null (expected — no file exists to
resolve to, CPython sets __file__ = None), while the submodule form
(import user.model) resolves correctly via suffixResolve fallback.

Made-with: Cursor

---------

Co-authored-by: chirag-nighut <chiragnighut@gmail.com>
2026-03-17 16:32:34 +00:00
Zak 64b7ff0061 docs: add Codex MCP configuration to README (#236)
- Add Codex to Editor Support table
- Add Codex manual config example (~/.codex/config.toml)
- Update editor list in usage table

Fixes #131

Made-with: Cursor
2026-03-16 21:23:14 +00:00
Gergő Magyar f2d3df48f6 feat: Phase 5 type resolution — chained calls, pattern matching, class-as-receiver (#315)
* feat: Phase 5 type resolution — chained calls, pattern matching, class-as-receiver, code review fixes

Phase 5.1: Chained method call resolution (depth-capped at 3)
- resolveChainedReceiver() resolves a.getUser().save() by walking the chain
  and looking up intermediate return types from the SymbolTable
- extractReceiverNode() + extractCallChain() shared in utils.ts
- receiverCallChain on ExtractedCall for worker path parity
- MAX_CHAIN_DEPTH=3 enforced in both extraction and resolution

Phase 5.2: Pattern matching binding extractors
- PatternBindingExtractor type added to LanguageTypeConfig
- declarationTypeNodes map tracks original type AST nodes for generic unwrapping
- Rust: if let Some(x)/Ok(x) unwrapping with extractGenericTypeArgs
- Java: instanceof pattern variables (Java 16+)
- C#: is-pattern disambiguation fixture (already working via extractDeclaration)

Phase 5.5d: Python standalone type annotations (name: str)
- expression_statement with type child now captured in DECLARATION_NODE_TYPES

Phase 5.5e: ReceiverKey collision fix for overloaded methods
- receiverKey preserves @startIndex to prevent same-name method collisions
- lookupReceiverType does prefix scan with ambiguity refusal

Class-as-receiver for static method calls (#289)
- UserService.find_user() now resolves via ctx.resolve() tiered lookup
- Respects import scoping — no false positives from unrelated packages

Code review fixes:
- Extracted CALL_EXPRESSION_TYPES + extractCallChain to utils.ts (eliminated duplication)
- Converted resolveChainedReceiver from recursion to loop (no exposed depth param)
- Added depth cap to extractReturnTypeName (defense against nested wrapper types)
- Replaced lookupFuzzy with ctx.resolve for class-as-receiver (architecturally consistent)

Closes #289

Test coverage: 6 new fixtures, 12+ new unit tests, 7 new integration test suites

* fix: Ruby chain calls, Rust Err(x) unwrap, Enum class-as-receiver (#315)

Address three per-language gaps identified in Phase 5 code review:

- Ruby: add `method`/`receiver` field fallbacks to extractCallChain
  (tree-sitter-ruby uses different field names than other grammars)
- Rust: handle `Err(e)` pattern binding via typeArgs[1] from Result<T,E>
- Enum: include Enum type in class-as-receiver filter (both paths)

Integration tests added for all three fixes.

* fix: chain base type resolution parity between serial and worker paths (#315)

- Worker path: add typeEnv.lookup for chain base receiver after extraction
  (typed parameters like `fn process(svc: &UserService)` were silently lost)
- Serial path: add ctx.resolve class-as-receiver fallback for chain base
  (class-name chains like `UserService.find_user().save()` failed)
- Fix misleading comment in parse-worker.ts that described unimplemented logic
- Integration tests: typed-parameter chain, static class-name chain

* fix: Kotlin chain call extraction, createClassNameLookup Enum/Struct (#315)

- Kotlin: extractCallChain now handles navigation_expression → navigation_suffix
  AST structure (Kotlin's call_expression has no 'function' field)
- createClassNameLookup: include Enum and Struct alongside Class for consistent
  constructor recognition in extractInitializer
- Integration test: kotlin-chain-call fixture verifying svc.getUser().save()
2026-03-16 19:38:09 +00:00
Gergő Magyar 5fa73bafdf feat: Phase 4 type resolution — nullable unwrapping, for-loop typing, assignment chains, code review fixes (#310)
* feat: Phase 4 type resolution — nullable unwrapping, for-loop typing, assignment chains, Kotlin return types

Phase 4.1: Nullable/optional chain unwrapping
- Add stripNullable utility in shared.ts for stripping nullable wrappers
- Apply in lookupInEnv to unwrap User | null → User, User? → User before receiver lookup
- Handles TS union, Kotlin/C#/Swift nullable suffix, Python Union[T, None], Rust Option<T>
- Enables receiver-type disambiguation through ?. optional chaining

Phase 4.2: For-loop element typing (Tier 0 — Java/C#/Kotlin)
- Add ForLoopExtractor type and forLoopNodeTypes to LanguageTypeConfig
- Java enhanced_for_statement, C# foreach_statement, Kotlin for_statement extractors
- Only explicit element types in AST (Tier 0); inference-based languages deferred

Phase 4.3: Assignment chain propagation (single-pass, depth-1)
- Add PendingAssignmentExtractor to LanguageTypeConfig with per-language implementations
- Handles TS/JS variable_declarator, Rust let_declaration, Python assignment,
  Go short_var_declaration, C# equals_value_clause, Java/Kotlin variable_declarator
- Single post-walk propagation pass (no fixpoint iteration per Sorbet/Pyright design)
- Resolves const b = a; b.save() when a has known type from Tier 0/1/1b

Phase 4.5: Kotlin return type extraction (bug fix)
- Fix extractMethodSignature to handle Kotlin user_type after function_value_parameters
- Remove lenient test assertions, add strict disambiguation proof

Integration tests across 10+ languages with competing same-name methods
and negative assertions proving disambiguation.

* fix: per-language assignment chain gaps from code review

- Kotlin: new extractKotlinPendingAssignment for property_declaration →
  variable_declaration AST (Java's variable_declarator doesn't exist in Kotlin)
- Go: handle var_spec (var b = u) alongside short_var_declaration (:=)
- PHP: add extractPendingAssignment for $alias = $user with $ prefix preserved

Integration tests added for all three languages with competing
same-name methods and negative disambiguation assertions.

* fix: code review fixes — DRY nullable keywords, avoid array allocations, clarify depth comment

Addresses findings from 6-agent code review on PR #310:

- Move stripNullable JSDoc to correct position (was orphaned above NULLABLE_KEYWORDS)
- DRY: reuse NULLABLE_KEYWORDS set in pipe-split filter instead of inline strings
- Replace node.children.find() with findChildByType/manual loops in jvm.ts,
  go.ts, csharp.ts to avoid unnecessary array allocations per tree-sitter call
- Clarify "depth-1" comment in type-env.ts: single-pass resolves multi-hop
  chains when forward-declared; reverse-order is depth-1 only
- Annotate extractGenericTypeArgs as Phase 5 infrastructure (zero production callers)
- Re-export PendingAssignmentExtractor from index.ts for API consistency
- Add explicit return undefined in Go extractPendingAssignment
- Remove redundant child.text === '=' check in Kotlin extractor

Test coverage:
- 20 new unit tests: stripNullable edge cases, per-language assignment chains,
  reverse-order depth limitation, nullable lookup resolution
- 15 new integration tests: multi-hop chains (a→b→c), nullable+chain combined
  (User|null + alias), Python User|None through stripNullable path
- 3 new fixtures: ts-multi-hop-chain, ts-nullable-chain, python-nullable-chain

* fix: third-pass review — walrus chain, scanner allocations, Kotlin variable_declaration, C# type guard

Addresses 4 new findings from third-pass CI review:

1. Python walrus operator (:=) now handled by extractPendingAssignment —
   named_expression nodes propagate alias chains alongside regular assignment
2. Scanner .namedChildren.find()/.some() in jvm.ts replaced with
   findChildByType() — consistent with 98daed4 code review fixes
3. Kotlin extractPendingAssignment extended to handle variable_declaration
   nodes in addition to property_declaration (function-local val/var)
4. C# extractPendingAssignment early-returns for is_pattern_expression and
   field_declaration nodes (never contain variable_declarator children)

Integration tests:
- Python: walrus chain (alias := u) with disambiguation (5 tests, 1 fixture)
- Kotlin: assignment chain with typed declarations (5 tests, 1 fixture)
- C#: assignment chain + is-pattern coexistence (6 tests, 1 fixture)
- Unit: Python walrus propagation (1 test)

* feat: nullable wrapper unwrapping + C++ assignment chains

Gaps 1, 2, 4 from code review — architectural changes to type resolution:

1. extractSimpleTypeName now unwraps nullable wrapper generics:
   - Optional<User> → "User" (Java), Option<User> → "User" (Rust),
     Maybe<User> → "User" (Kotlin Arrow/Haskell-style)
   - Containers (List, Map) and async wrappers (Promise, Future) are NOT
     unwrapped — methods are called on the container, not the inner type
   - Uses existing extractGenericTypeArgs (now production-active, was dead code)
   - NULLABLE_WRAPPER_TYPES set: Optional, Option, Maybe

2. C++ extractPendingAssignment added for auto alias chains:
   - auto alias = user; alias.save() now propagates User type
   - Handles pointer/reference declarators, auto/decltype(auto)

3. Updated existing Rust test: Option<User> parameter now correctly
   stores "User" instead of "Option" in TypeEnv

Integration tests with fixtures for Java Optional, Rust Option, C++ auto
chain. Full pipeline resolution marked .todo — requires call-processor
enhancement (TypeEnv stores correct types but call-processor needs
additional work to produce CALLS edges for these patterns).

Unit tests: 196 passed (7 new). Integration: all 9 languages green.

* fix: resolve .todo tests — stale dist/ was the root cause

The Rust Option<User> and C++ auto assignment chain integration tests
were marked .todo because the pipeline didn't produce CALLS edges.
Root cause: dist/ was compiled from pre-Phase 4 source and lacked:
- NULLABLE_WRAPPER_TYPES unwrapping in extractSimpleTypeName
- C++ extractPendingAssignment

After npm run build, all tests pass as real assertions:
- Rust: alias.save() resolves to User#save via Option<User> unwrap + chain
- C++: alias.save() and rAlias.save() resolve via auto assignment chain
  with correct disambiguation (User vs Repo)

Only remaining .todo: Rust user.unwrap().save() (Phase 5 — chained
return type inference, not a TypeEnv issue).
2026-03-16 15:21:54 +00:00
fbff6d08c0 feat(ingestion): respect .gitignore and .gitnexusignore during file discovery (#231)
* feat(ingestion): respect .gitignore and .gitnexusignore during file discovery

Add support for excluding files from indexing based on .gitignore and
.gitnexusignore patterns. Previously, GitNexus used only a hardcoded
ignore list, causing significant index pollution in repositories with
git-ignored directories containing code (e.g., Docker-mounted volumes).

Changes:
- Add `ignore` package for gitignore-spec pattern matching
- Add `loadIgnoreRules()` to parse .gitignore + .gitnexusignore
- Add `createIgnoreFilter()` returning glob-compatible IgnoreLike object
- Integrate filter into glob's `ignore` option for directory-level pruning
- Remove post-glob `.filter()` call (now handled during traversal)

The hardcoded DEFAULT_IGNORE_LIST remains as fallback for non-git repos.

Closes #228

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ingestion): address review feedback on ignore filtering

- Distinguish ENOENT vs EACCES in loadIgnoreRules (warn on permission errors)
- Add GITNEXUS_NO_GITIGNORE env var to bypass .gitignore parsing
- Fix bare-name pattern matching in childrenIgnored (check both with/without trailing slash)
- Rename isIgnoredDirectory to isHardcodedIgnoredDirectory for clarity
- Add clarifying comments for design decisions (D2 negation, D3 dot:false redundancy)
- Add tests for bare-name patterns, file-glob patterns, EACCES handling, env var

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ingestion): address second round of review feedback

- G1: Document GITNEXUS_NO_GITIGNORE in `analyze --help` and log when active
- G2: Add comment clarifying path-scurry POSIX normalization contract
- G3: Add IgnoreOptions interface — env var now falls back, callers can
  pass `noGitignore` explicitly for testability and future CLI flag
- G4: Add integration test verifying walkRepositoryPaths respects the env var

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(ingestion): gracefully skip files with unavailable tree-sitter grammars

Port unsupported language resilience from PR #301 by @jecanore.
- Make Kotlin import optional (like Swift) in parser-loader and parse-worker
- Add worker-local isLanguageAvailable() with filePath param for tsx distinction
- Track and log skipped files per language in both sequential and worker paths
- Add skippedLanguages to ParseWorkerResult for worker→main aggregation
- Add isLanguageAvailable unit tests

Refs: #301, #155, #228

Co-Authored-By: jecanore <juan@housingbase.io>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* test(e2e): add ignore + language-skip end-to-end test with fixture repo

Add a fixture repo (test/fixtures/ignore-and-skip-repo/) with .gitignore,
.gitnexusignore, TypeScript source files, and a Swift file to exercise
all three features end-to-end:

- File discovery: verifies .gitignore excludes data/ and *.log,
  .gitnexusignore excludes vendor/, source files are discovered
- Parsing: verifies TypeScript files produce Function nodes and DEFINES
  relationships, Swift files are skipped gracefully when grammar is
  unavailable

Add the test to the standalone group in ci-integration.yml and coverage job.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ci): move ignore-and-skip-e2e test to e2e group per review feedback

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(test): use temp directory instead of fixture for e2e ignore test

The fixture's .gitignore prevented data/seed.json and debug.log from
being committed — these files would be missing after checkout in CI.

Switch to creating the entire test structure in a temp directory via
beforeAll (matching filesystem-walker.test.ts pattern). This ensures
all files exist regardless of git ignore rules.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(test): correct graph API usage in e2e ignore test

Use graph.nodes property getter instead of graph.getNodes(), and check
Function node filePath instead of non-existent File nodes (File nodes
are created by processStructure, not processParsing).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: add workflows permission to ci-integration.yml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: change workflows permission to write per review

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* ci: move workflows permission from ci-integration.yml to ci.yml caller

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ci): fix Claude workflows for fork PRs, remove misplaced workflows perm

Three issues prevented Claude from running on fork PRs:

1. claude-code-review.yml lacked workflows:write — push failed when
   fork PRs modify .github/workflows/ files
2. claude.yml had no fork PR support — checked out main and couldn't
   fetch the fork's branch from origin
3. Cleanup step unconditionally deleted branches even when push failed,
   breaking the concurrent claude.yml workflow

Also removes workflows:write from ci.yml's integration job — CI tests
don't need that permission. The permission belongs on the claude
workflows that push fork branches.

Changes:
- Add workflows:write to both claude workflow permissions blocks
- Add fork PR detection + branch push/cleanup to claude.yml
- Add step id to push-fork; cleanup only runs if push succeeded
- Pass branch names via env vars to prevent shell injection (security)
- Add concurrency groups to prevent race conditions between workflows
- Remove misplaced workflows:write from ci.yml integration job

* fix(ci): use GitHub API for fork branch refs instead of git push

GITHUB_TOKEN cannot have 'workflows' permission — it's only valid for
PATs and GitHub Apps. This means git push fails whenever a fork PR
modifies .github/workflows/ files.

Replace git push with the GitHub REST API (POST/PATCH /git/refs) to
create temporary branch refs. The API creates a pointer to the
already-existing PR head commit without triggering the workflow file
push protection. Similarly, cleanup uses DELETE /git/refs instead of
git push --delete.

Also removes the invalid 'workflows: write' from permissions blocks.

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: jecanore <juan@housingbase.io>
Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-16 13:26:20 +00:00
Jim ParkandClaude Opus 4.6 f83ef56ccb fix: sync package-lock.json with onnxruntime-node ^1.24.0 dependency
CI was failing because package-lock.json still referenced onnxruntime-node@1.21.0
after package.json was updated to require ^1.24.0.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-16 00:52:40 -07:00
Jim ParkandClaude Opus 4.6 530b4be9ee fix: prevent uncatchable native crash when CUDA libs present but ORT lacks CUDA provider
onnxruntime-node versions before 1.24.0 are CPU-only and do not ship
libonnxruntime_providers_cuda.so. When system CUDA libraries (e.g.
libcublasLt.so.12) are detected, isCudaAvailable() returns true and
the embedder requests the CUDA execution provider. Since the provider
binary doesn't exist in the package, ONNX Runtime crashes at the native
level (provider_bridge_ort.cc), which is uncatchable by the JS try/catch
fallback — killing the entire process.

This commit:
- Adds onnxruntime-node ^1.24.0 as an explicit dependency (first version
  to ship CUDA provider binaries for Linux x64)
- Adds hasOrtCudaProvider() check that verifies the CUDA provider .so
  exists in the onnxruntime-node package before attempting CUDA, so
  the embedder gracefully falls back to CPU on older ORT versions

Fixes the crash at 92% "Loading embedding model..." on Linux systems
with CUDA toolkit installed. Also related to #165.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-03-15 14:22:22 -07:00
Gergő Magyar 6c18ae08f7 feat: return type inference, doc-comment parsing, and per-language type extractors (#284)
* feat: Phase 3 — return type inference, generic args extraction, Ruby YARD type extractor

Three architectural improvements to the type resolution system:

1. Return type inference — wire extractMethodSignature returnType through
   SymbolDefinition into call-processor. When var = callee() and callee
   has a known return type, bind var to that type. Handles Promise<T>
   unwrapping, nullable stripping, pointer/reference removal.

2. Generic type argument extraction — new extractGenericTypeArgs() utility
   that extracts type parameters from List<User> → ['User']. Handles
   TS/Java/Kotlin/C#/Rust generic syntax. Building block for for-loop
   variable typing.

3. Ruby dedicated type extractor — replaces the stub with YARD annotation
   parsing (@param name [Type]), handling qualified types, nullable types,
   and singleton methods. Ruby now has real type resolution.

Unit tests: 127 → 192+ (type-env) + 65 (symbol-table, call-processor) + 18 (generics)
Integration tests: 8+ new test cases with fixtures across TS/Python/Go/Java/Ruby

* fix: Phase 3 gaps — WRAPPER_GENERICS correctness, Ruby :: qualifier, namespaced constructors

- Remove collection types (List, Array, Vec, Set) from WRAPPER_GENERICS to prevent
  false CALLS edges (e.g. List<User> no longer unwraps to User)
- Add :: qualifier handling in extractReturnTypeName for Ruby/C++/Rust namespaced types
- Add Ruby `constant` and `scope_resolution` node types to shared extractors
- Extract shared extractRubyConstructorAssignment helper (dedup type-env.ts + ruby.ts)
- Add integration tests for return type inference: Python, TypeScript, Go, Java, Ruby
- Add Ruby namespaced constructor fixture (Models::UserService.new)
- Add unit tests for collection reclassification and :: qualifiers

* feat: Phase 4 — CONSTRUCTOR_BINDING_SCANNERS for all languages + return type inference tests

Add CONSTRUCTOR_BINDING_SCANNERS for 6 missing languages, completing
return type inference coverage across all 11 supported languages:

- TypeScript/JS: variable_declarator with call_expression, unwraps await
- Go: short_var_declaration single-assignment (skips multi-return, new/make)
- Java: local_variable_declaration with `var` type + method_invocation
- C#: variable_declaration with implicit_type (var) + invocation_expression
- Rust: let_declaration without type annotation, handles mut_pattern
- PHP: assignment_expression with function_call_expression

Also adds property_identifier to extractSimpleTypeName for qualified
member calls (repo.getUser → getUser), fixing namespaced constructor
inference that was previously a known limitation.

Integration tests added for all 11 languages with correct label
assertions (Function vs Method per language's tree-sitter queries).

* refactor: merge CONSTRUCTOR_BINDING_SCANNERS into per-language LanguageTypeConfig

Eliminates the parallel dispatch map in type-env.ts by moving all 11
constructor binding scanners into their respective type-extractors/*.ts
files as `scanConstructorBinding` on LanguageTypeConfig.

- Add ConstructorBindingScanner type to types.ts
- Add shared helpers: hasTypeAnnotation, unwrapAwait, extractCalleeName
- Move scanners to typescript.ts, jvm.ts, python.ts, php.ts, go.ts,
  rust.ts, swift.ts, c-cpp.ts, csharp.ts, ruby.ts
- Fix `any` types in C# scanner → SyntaxNode | null
- Delete ~300 lines from type-env.ts (CONSTRUCTOR_BINDING_SCANNERS map)
- Update buildTypeEnv to use config.scanConstructorBinding

All 143 type-env unit tests and all 10 language integration suites pass.

* fix: remove unused import, fix any type in Java scanner, update stale comment

- Remove unused extractCalleeName import from jvm.ts
- Fix (c: any) → (c: SyntaxNode) in Java scanner
- Update stale CONSTRUCTOR_BINDING_SCANNERS reference in ruby.ts comment

* fix: C# and PHP return type inference — scanner fixes, method signature extraction, and cross-file resolution

Addresses code review findings on PR #284:

C# scanner (csharp.ts):
- Fix type node lookup: iterate children instead of childForFieldName('type')
  which returns undefined in tree-sitter-c-sharp
- Fix initializer lookup: handle direct invocation_expression children
  (no equals_value_clause wrapper in tree-sitter-c-sharp)

C# return type extraction (utils.ts):
- Add 'returns' field check to extractMethodSignature — tree-sitter-c-sharp
  uses 'returns', not 'type', for method return types

C# cross-file resolution (call-processor.ts + fixture):
- Add constructor binding verification to sequential processCalls path
  (was only in the worker processCallsFromExtracted path)
- Add ReturnType.csproj to csharp-return-type fixture
- Update fixture namespaces to use ReturnType.Models/ReturnType.Services
  prefix (matches real C# project conventions)

PHP scanner (php.ts):
- Extend scanConstructorBinding to handle member_call_expression
  ($this->getUser() patterns), not just function_call_expression

Shared (shared.ts):
- Add member_access_expression to extractSimpleTypeName qualified-names
  block (C# method calls like svc.GetUser())

Tests:
- Add Repo.cs/Repo.php disambiguation fixtures (two Save methods)
- Strengthen C# and PHP return type tests with hard disambiguation assertions
- Add C# scanner unit tests and return type extraction test

* feat: per-language ReturnTypeExtractor + doc-comment @param parsing for PHP, JS, Ruby

Add ReturnTypeExtractor to LanguageTypeConfig interface with implementations
for Ruby (YARD @return), PHP (PHPDoc @return), and JS/TS (JSDoc @returns).
The fallback is wired in both parsing-processor and parse-worker paths,
activating only when extractMethodSignature finds no AST-based return type.

Also add doc-comment @param type extraction for PHP and JS/TS, following
Ruby's existing collectYardParams pattern. This enables parameter.method()
resolution in loosely-typed codebases using PHPDoc @param or JSDoc @param.

Additional fixes from PR #284 code review:
- Go: add selector_expression + field_identifier to extractSimpleTypeName
  (enables package-qualified factory calls like models.NewUser())
- Ruby: broaden scanConstructorBinding to capture plain call assignments
  (user = get_user()) in addition to Class.new patterns
- Ruby: harden return-type fixture with disambiguation (two save methods)

Test coverage: +14 new integration tests across Go, Ruby, PHP, JS/TS

* fix: JSDoc async return type, PHP attribute walkers, and $this receiver disambiguation

Three fixes from fourth-pass code review on PR #284:

1. JSDoc `@returns {Promise<User>}` no longer stripped to `Promise` — extractReturnType
   now uses sanitizeReturnType (preserves generics) instead of normalizeJsDocType
   (which stripped them before extractReturnTypeName could unwrap WRAPPER_GENERICS).

2. PHP 8+ `#[Attribute]` and JS `@decorator` nodes no longer break doc-comment walkers.
   Both extractReturnType and collect*Params functions now skip attribute_list/decorator
   nodes instead of breaking on them as named siblings.

3. PHP `$this->method()` now provides receiverClassName for disambiguation.
   When two classes define the same method, the enclosing class narrows candidates
   via ownerId matching in call-processor, preventing false no-binding results.

* fix: sanitizeReturnType dot corruption, JS test assertions, Ruby constant receiver

- Remove redundant dot-path stripping from sanitizeReturnType that corrupted
  qualified names inside generics (e.g. Promise<models.User> → User>)
- Split JS async fixture into separate files and add negative assertions
  to properly verify disambiguation (mirroring PHP test pattern)
- Accept 'constant' node type in Ruby scanConstructorBinding for factory
  call assignments (SERVICE = build_service())
- Add 'constant' to SIMPLE_RECEIVER_TYPES so extractReceiverName handles
  Ruby constant receivers (SERVICE.process)

* fix: nested generic arg splitting, JS/Ruby test false positives

- Replace naive comma split in extractReturnTypeName with bracket-balanced
  extractFirstGenericArg so nested types like Future<Result<User, Error>>
  unwrap correctly instead of producing malformed "Result<User"
- Add CompletableFuture to WRAPPER_GENERICS for Java async unwrapping
- Split js-jsdoc-return-type fixture models.js into user.js/repo.js and
  add negative assertions to prove disambiguation (not just file match)
- Split ruby-constant-factory-call fixture into separate service files
  and add negative assertions against AdminService resolution

* fix: review findings — receiverClassName parity, Rust wrappers, Go multi-return, Kotlin/Swift qualified calls

P1: Sequential path now includes receiverClassName narrowing for PHP
$this->method() disambiguation (was missing vs worker path).

P2: Added Rc/Arc/Weak/MutexGuard/Cow + 6 more Rust Deref types to
WRAPPER_GENERICS (Box excluded — Java Swing collision). Extended
Kotlin/Swift scanners to handle navigation_expression callees.
Added Go multi-return support (user, err := f()) with blank/_/err/ok
guard + AST-level first-return extraction in extractMethodSignature.

P3: Extracted shared verifyConstructorBindings() eliminating 60 lines
of duplication between sequential and worker paths. Added return-type
inference integration tests for C++, Rust, Swift with competing
methods and negative disambiguation assertions.

* fix: Swift navigation_suffix unwrapping, Rust lifetime skipping, Kotlin disambiguation tests

- Swift scanConstructorBinding: handle tree-sitter wrapping qualified
  identifiers in navigation_suffix nodes
- Add extractFirstTypeArg to skip Rust lifetime parameters ('a, '_)
  when unwrapping wrapper generics like Ref<'_, User>
- Kotlin tests: add Repo class fixture with competing save() methods
  to prove disambiguation; assert no spurious edges on known gap
- Remove tree-sitter-kotlin from optionalDependencies (now regular dep)

* fix: C# null-conditional calls, Ruby YARD bracket-balanced split, PHPDoc alternate order, escapeValue hardening

- Add C# null-conditional call support (user?.Save()): tree-sitter query for
  conditional_access_expression, member_binding_expression in MEMBER_ACCESS_NODE_TYPES,
  receiver extraction via conditional_access_expression parent walk
- Fix Ruby YARD type parsing for nested generics (Hash<Symbol, User>): replace
  naive split(',') with bracket-balanced splitter respecting <> depth
- Add alternate YARD format (@param [Type] name) alongside standard (@param name [Type])
- Add alternate PHPDoc format (@param $name Type) alongside standard (@param Type $name)
- Harden escapeValue in kuzu-adapter.ts: escape \n and \r to prevent Cypher injection
- Integration tests: C# null-conditional fixture (5 tests), Ruby YARD generics fixture (6 tests)
- Unit tests: PHPDoc alternate order (2 tests), C# null-conditional call-form (updated)

* test: add Python static/classmethod integration tests (issue #289)

Verifies that classes using only @staticmethod/@classmethod have HAS_METHOD
edges connecting them to their child methods. This was the root cause of
issue #289 where context() and impact() returned empty for such classes.

Tests cover: HAS_METHOD edge emission, unique static method resolution
(create_user, delete_user), and ambiguous same-named method handling
(find_user on both UserService and AdminService — safely refused).

* fix: lbug batch escapeValue newline hardening, Rust ::default() scanner exclusion

- Apply \n/\r escaping to batch upsert escapeValue in lbug-adapter.ts:429
  (missed instance of the CREATE-path fix from ec4dca4)
- Exclude Rust ::default() from scanConstructorBinding to match
  extractInitializer behavior — avoids wasted cross-file lookups on
  the broadly-implemented Default trait
- Unit tests: 2 new scanner exclusion tests (::default and ::new)
- Integration tests: 6 new Rust ::default() constructor resolution tests
  with disambiguation fixture (User::default vs Repo::default)

* fix: C#/Rust async await unwrap, PHP backslash namespace, fallback escaping

- C# scanConstructorBinding: unwrap await_expression to find invocation_expression
  (var user = await svc.GetUserAsync() now produces constructor binding)
- Rust scanConstructorBinding: unwrap .await postfix via shared unwrapAwait helper
  (let user = get_user().await now produces constructor binding)
- extractReturnTypeName: handle PHP backslash namespace separator (\App\Models\User → User)
- fallbackRelationshipInserts: match batch escapeValue hardening with \n/\r escaping

Tests: 2 unit (type-env), 3 unit (call-processor), 7 integration (csharp+rust), 7 fixtures

* fix: C#/Rust async-binding test false positives — add competing types and negative assertions

C# fixture: add Order.cs with Order.Save(), change OrderService to return
Task<Order> via GetOrderAsync, add negative assertion proving user.Save()
does not resolve to Order#Save.

Rust fixture: split models.rs into user.rs/repo.rs, make process_user and
process_repo async fn, add bidirectional negative assertions proving no
cross-contamination between User#save and Repo#save.

* fix: C# async-binding broken assertion, bare wrapper type leak, JSDoc optional params

- Split Program.cs Main into ProcessUser/ProcessOrder so negative
  assertions use strict toBeUndefined() (matching Rust pattern)
- Guard bare wrapper types (Task, Promise, Option…) in
  extractReturnTypeName — return undefined instead of the wrapper name
- Update JSDOC_PARAM_RE to capture @param {Type} [optionalName] syntax

* fix: update symbol and relationship counts in documentation
2026-03-15 18:49:40 +00:00
Candido Sales GomesandClaude Opus 4.6 5a5850832c refactor: migrate from KuzuDB to LadybugDB v0.15 (#275)
* refactor: migrate from KuzuDB to LadybugDB v0.15

KuzuDB was archived (Apple acquisition, Oct 2025). LadybugDB is the
community fork with full API compatibility.

- Package swap: kuzu → @ladybugdb/core, kuzu-wasm → @ladybugdb/wasm-core
- Rename all internal paths: kuzu → lbug (adapters, schema, storage)
- Storage path: .gitnexus/kuzu → .gitnexus/lbug (with auto-cleanup)
- Add explicit VECTOR extension loading (required in v0.15)
- Update CI workflow, documentation, and all tests
- 1151 unit + 27 integration tests passing

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: address code review findings (P1-P3)

P1: Fix WASM adapter to use getAll() API, wire cleanupOldKuzuFiles
into analyze command, add symlink path traversal protection.
P2: Cache VECTOR extension load state, batch augmentation engine
queries (20→4), fix web getCopyQuery for multi-language tables,
fix stale KuzuDB references, correct brainstorm package names.
P3: Complete lbug-wasm.d.ts type declarations, batch semantic
search per-label, update stale BM25 comment.

* chore: remove outdated KuzuDB migration brainstorming document

* fix: load FTS extension in MCP pool adapter on init

The read-only pool adapter never loaded the FTS extension, so all
QUERY_FTS_INDEX calls failed silently. This broke search-pool and
augmentation integration tests, and caused empty results in the
web UI server mode.

* feat: implement shared Database caching and connection reference counting

* feat: enhance KuzuDB migration handling and status reporting

* fix: mock cleanupOldKuzuFiles in local backend callTool tests

* fix: update mock for cleanupOldKuzuFiles and adjust imports in callTool tests

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-15 15:53:01 +00:00
Gergő Magyar 62242d5f44 feat: TypeEnvironment API with constructor inference, self/this/super resolution (#274)
* feat(type-env): constructor-call type inference for TypeEnv (Phase 1)

Add extractInitializer as a Tier 1 fallback in buildTypeEnv: when a
declaration node has no explicit type annotation, infer the type from
constructor-call patterns (new X(), X::new(), X::default(), $x = new X()).

Languages covered: TypeScript/JS, Java (var), Rust, PHP, C++ (auto).
Python/Kotlin/Swift deferred — need symbol-table access to distinguish
class constructors from function calls.

Adds 20 new unit tests covering constructor inference, annotation
precedence, and known limitations across all supported languages.

* fix(type-env): class-aware constructor resolution, multi-declarator fix

- Add collectClassNames pre-scan: walks AST to build Set<string> of
  class/struct names defined in the file
- C++ extractInitializer uses classNames.has() to verify identifier is
  a known class before inferring (auto x = User() resolves, auto x =
  getUser() does not — no false positives)
- Add InitializerExtractor type that receives classNames parameter
- Fix env.size gating: always call extractInitializer when available,
  so mixed declarators like const a: A = x, b = new B() resolve both
- Add env.has() guard in Java extractInitializer to skip already-bound vars
- Document Rust new/default whitelist rationale
- Pin all test assertions, add mixed multi-declarator test case

* fix(type-env): resolve Self/self/static/parent to actual type names

- Rust: Self::new()/Self::default() resolves to enclosing impl type
- PHP: new self()/static() resolves to enclosing class, parent() to superclass
- Rust: Tier 0 annotation guard prevents overwrite by constructor inference
- Rust: mut_pattern handling in extractVarName for let mut bindings
- TS: fix misleading comment in extractInitializer
- 58 tests passing (3 new Self/self resolution tests)

* perf(type-env): single-pass AST walk with closure-scoped state

Refactors buildTypeEnv to use closures instead of passing mutable state
as parameters. classNames, env, and config are captured by the inner
walk and extractTypeBinding functions — no parameter mutation.

- Eliminates separate collectClassNames pre-scan (O(2n) → O(n))
- config looked up once per file instead of per-node
- 29 fewer lines

* feat(type-env): constructor-inferred type resolution for all languages

Add cross-file constructor type inference to the ingestion pipeline,
enabling receiver-type disambiguation for member calls like
`user.save()` when the variable is assigned from a constructor without
explicit type annotations.

Pipeline changes:
- Add extractInitializer to Python and Swift type extractors
- Add CONSTRUCTOR_BINDING_SCANNERS for Python, Swift, C/C++ in type-env
- Wire constructorBindings through parse-worker → parsing-processor →
  pipeline → processCallsFromExtracted
- Rewrite resolveCallTarget receiver-type filtering (step D) to use
  tiered import resolution (same-file → import-scoped → global) before
  falling back to fuzzy ownerId matching
- Use collectTieredCandidates for constructor binding verification
  instead of raw lookupFuzzy

Bug fixes:
- Fix C++ inline method query: @definition.method was captured on
  field_declaration_list instead of function_definition, causing wrong
  parameterCount for all inline class methods
- Fix parse-worker accumulated/flush results missing constructorBindings

CI changes:
- Add swift.test.ts to ci-integration pipeline group and coverage job
- Update ci-report to fetch base branch (main) coverage for delta
  reporting instead of showing config thresholds
- Add per-suite timing breakdown table (unit/integration/total)
- Add expandable skipped test details section

Tests: 288 passed, 4 skipped (swift — macOS only) across 10 languages
- 36 new constructor-inferred integration tests (4 per language)
- 10 fixture directories with cross-file constructor patterns
- TypeScript, JavaScript, Java, Kotlin, Python, PHP, Rust, Go, C++, Swift

* fix(type-extractors): add type assertion for LanguageTypeConfig

* feat(ruby): constructor-inferred type resolution and self-receiver mapping

Add Ruby User.new constructor binding scanner to type-env, enabling
receiver-type disambiguation for member calls like user.save vs repo.save.
Add self/this → enclosing class resolution in lookupTypeEnv so self.method()
calls resolve to the correct class even when the method name is ambiguous.

* docs: update README with constructor inference and self/this resolution details

* refactor(ingestion): unified ResolutionContext replaces fragmented map passing

Introduce createResolutionContext() as the single resolution API for all
processors. Eliminates duplicated tier-selection logic, fixes heritage
namedImportMap bug, and adds per-file resolution caching.

- NEW resolution-context.ts: closure-factory with resolve(), per-file cache,
  TIER_CONFIDENCE constant, and shared ResolutionTier type
- DELETE symbol-resolver.ts: zero production importers, logic now in
  resolution-context.ts
- call-processor: all functions take ctx instead of 6 separate maps,
  collectTieredCandidates removed (ctx.resolve replaces it),
  D4 redundant re-resolve eliminated
- heritage-processor: takes ctx, resolveHeritageId helper extracts
  repeated 14-line fallback pattern, namedImportMap now included
- import-processor: takes ctx, dead createImportMap/createPackageMap/
  createNamedImportMap factories removed
- pipeline: creates single ctx, wires onProgress to all processors,
  logs cache hit rate in dev mode
- Tier renamed: unique-global → global (honest about returning all candidates)
- Tests migrated: 1178 unit + 84 integration passing

* feat(type-env): self/this/super resolution, TypeEnvironment API, and review fixes

Add cross-language receiver keyword resolution:
- self/this/$this → enclosing class name via AST walk
- super/base/parent → parent class name via heritage AST extraction
  (8 grammar variants: TS/JS, Java, Python, Ruby, C#, PHP, Kotlin, C++, Swift)
- D-phase widening in resolveCallTarget for super→parent method dispatch

Introduce TypeEnvironment API replacing loose TypeEnvResult + lookupTypeEnv:
- buildTypeEnv() returns TypeEnvironment with .lookup() method
- Single-pass AST walk merges constructor binding scan (was separate traversal)
- ClassNameLookup type replaces over-broad ReadonlySet<string> facade
- Memoized class name lookups to avoid redundant SymbolTable scans

Code review fixes (6 agents, 11 findings):
- Replace ctx.resolve(name, '') hack with direct symbols.lookupFuzzy()
- Extract scope key helpers (extractFuncNameFromScope, receiverKey)
- Simplify D-phase from 5 steps to 4 with deduped typeNodeIds
- Remove C from CONSTRUCTOR_BINDING_SCANNERS (YAGNI — C has no constructors)
- Cache Map reuse in ResolutionContext to reduce GC pressure
- Remove unused TieredCandidates import

Integration tests for self/this, parent, and super resolution across all
12 supported languages with per-language fixture directories.

* fix(type-env): generic parent resolution, TS cast inference, C++ brace-init

Fix generic parent class breaking super resolution:
- extractParentClassFromNode now uses extractSimpleTypeName to strip
  generic params (Base<T> → Base) and qualified names (models.Model → Model)
- Affects TS, Java, Python, C# heritage extraction

Fix TypeScript new X() as T / new X()! missed inference:
- Unwrap as_expression and non_null_expression before checking for
  new_expression in extractInitializer

Fix C++ brace-init User{} missed inference:
- Handle compound_literal_expression with type_identifier child
  in extractInitializer

Clean up deprecated lookupTypeEnv:
- Remove standalone lookupTypeEnv export, migrate all callers to
  TypeEnvironment.lookup() method
- Update all 80+ test assertions to use the new API

Integration test fixtures added:
- typescript-cast-constructor-inference (new X() as T, new X()!)
- typescript/java/csharp/kotlin-generic-parent-resolution
- cpp-brace-init-inference (auto x = User{})

* fix(type-extractors): Go &User{}, TS double-cast, Swift .init inference

Fix Go pointer-to-struct literal not inferred:
- Unwrap unary_expression (address-of &) before composite_literal check
- user := &User{} now correctly infers type User

Fix TypeScript double-cast only unwrapping one level:
- Change if to while loop for nested as_expression/non_null_expression
- new User() as unknown as Admin now correctly infers type User

Fix Swift User.init(name:) explicit init call missed:
- Handle navigation_expression callee with .init suffix in extractInitializer

Integration test fixtures:
- go-pointer-constructor-inference (&User{}, &Repo{})
- typescript-double-cast-inference (as unknown as T)

* feat: Rust struct literal, Python qualified ctor, Go new(), Swift .init scanner

- Rust: handle struct_expression in extractInitializer (User { name: "alice" })
- Python: support attribute nodes in extractInitializer (models.User("alice"))
  and the cross-file scanner — extractSimpleTypeName handles qualified names
- Go: handle new(User) built-in in extractGoShortVarDeclaration
- Swift: extend CONSTRUCTOR_BINDING_SCANNERS to handle navigation_expression
  callee for User.init(name:) cross-file resolution

Unit tests: 87 → 96 (Rust struct literal, Go new(), Python qualified ctor,
Python scanner qualified, plus edge cases)
Integration tests: 4 new describe blocks with fixtures

* fix: Rust Self{} resolution, C++ scoped brace-init, PHP promotion params, Ruby constants

- Rust: resolve Self {} struct literal to enclosing impl type (was stored as "Self")
- C++: replace type_identifier guard with extractSimpleTypeName for compound_literal_expression,
  enabling ns::User{} scoped brace-init (closes previously deferred gap)
- PHP: add property_promotion_parameter to TYPED_PARAMETER_TYPES for PHP 8.0+
  constructor property promotion (__construct(private Foo $x))
- Ruby: extend extractRubyConstructorBinding to accept constant left-hand side
  (REPO = Repo.new)

Unit tests: 96 → 101 (+5: Rust Self{} ×2, C++ ns::User{} ×1, PHP promotion ×1,
Ruby constant ×1)
Integration tests: 4 new describe blocks with fixtures

* feat: Phase 1 type resolution gaps — walrus, PHP properties, nullable, Go make/assert

Phase 1 quick wins from the type resolution gap analysis:

1. Python walrus operator := (named_expression) — extractInitializer + scanner
2. PHP 7.4+ typed class properties — property_declaration in extractDeclaration
3. Nullable union unwrapping — User | null → User in extractSimpleTypeName
4. Go make() builtin — slice/map element type extraction
5. Go type assertions — iface.(User) type extraction

Also: PHP primitive_type handling in extractSimpleTypeName (string, int, etc.)

Unit tests: 101 → 114 (+13)
Integration tests: 8 new describe blocks with fixtures

* feat: Phase 2 type resolution gaps — C++ range-for, Rust if-let, C# pattern matching, Python class annotations

Phase 2 medium-effort improvements:

1. C++ range-for with explicit type — for (User& u : vec) binds u: User
2. Rust if-let/while-let captured_pattern — user @ User { .. } binds user: User
3. C# is-pattern matching — if (obj is User user) binds user: User
4. Python class-level annotations — confirmed already working, added tests

Unit tests: 114 → 127 (+13)
Integration tests: 11 new test cases with fixtures
2026-03-14 19:05:49 +00:00
Gergő Magyar 6e38db879e fix(ruby): method-level call resolution, HAS_METHOD edges, and dispatch table (#278)
* fix(ruby): method-level call resolution, HAS_METHOD edges, and dispatch table refactoring

- Replace all `if (language === Ruby)` checks in processors with a
  `callRouters` dispatch table in call-routing.ts (renamed from
  ruby-call-routing.ts to preserve git history)
- Add Ruby `method` and `singleton_method` to FUNCTION_NODE_TYPES so
  findEnclosingFunction produces Method-level CALLS sources
- Add Ruby `class` and `module` to CLASS_CONTAINER_TYPES for HAS_METHOD
  edge generation
- Add bare call capture via tree-sitter query `(body_statement (identifier))`
  for Ruby methods called without parentheses
- Add Ruby member call detection (`call` node with `receiver` field) to
  inferCallForm and extractReceiverName
- Wire resolveRubyImport into resolveLanguageImport
- Add 24 integration tests across 5 suites: heritage/properties, arity
  filtering, member calls, ambiguous disambiguation, local shadow
- Add ruby.test.ts to CI integration workflow

* fix(ruby): resolve 6 Ruby resolution gaps from PR review

- Fix singleton_method label mismatch: @definition.function → @definition.method
  so CALLS edges from `def self.foo` bodies get correct sourceId
- Add ownerId and HAS_METHOD edges to attr_* Property nodes by calling
  findEnclosingClassId in both parse-worker and call-processor property branches
- Distinguish include/extend/prepend heritage: add heritageKind to
  RubyHeritageItem, propagate through heritage pipeline as IMPLEMENTS reason
- Document bare call over-capture limitation in tree-sitter query comment
- Add bare `require` (non-relative) import test coverage
- Add prepend/extend test coverage with distinct Loggable/Cacheable modules

31 Ruby integration tests passing, no regressions in other language resolvers.

* fix(ruby): web package parity — heritage reasons, property HAS_METHOD, singleton_method label

- Web call-processor: use item.heritageKind as IMPLEMENTS reason instead of
  hardcoded 'trait-impl', add :${kind} suffix to edge ID for uniqueness
- Web call-processor: port findEnclosingClassId, add HAS_METHOD edges for
  attr_* Property nodes to match CLI fix
- Web call-processor: singleton_method label 'Function' → 'Method' to match
  CLI tree-sitter query fix
- CLI parse-worker: update stale ExtractedHeritage.kind JSDoc to include
  'include' | 'extend' | 'prepend'

* fix(web): add HAS_METHOD to RelationshipType union

Web package was missing HAS_METHOD in the RelationshipType union,
causing a type mismatch with the HAS_METHOD edges emitted by the
attr_* property fix in call-processor.ts.
2026-03-14 11:34:30 +00:00
Candido Sales Gomes 0999595444 feat(ruby): Add Ruby language support for CLI and web (#111) 2026-03-13 21:46:59 +00:00
Chirag Nighut 649ad80dbb fix(cli): dynamically discover and install agent skills (#270) 2026-03-13 19:09:03 +00:00
Gergo Magyar 3dbe08fab6 chore: release v1.4.0 2026-03-13 13:18:14 +00:00
Gergő Magyar 1afe9166aa feat: language-aware code intelligence — symbol resolution, MRO, constructor discrimination (#238)
* feat: add Method Resolution Order (MRO) with language-specific rules

Implement full MRO computation for multi-language inheritance hierarchies:

- HAS_METHOD edges: Class→Method ownership edges emitted during parsing
  (both worker pool and sequential fallback paths)
- Method signatures: extract parameterCount and returnType from AST nodes
- C# heritage fix: distinguish EXTENDS vs IMPLEMENTS for base_list captures
  using symbol table lookup + I[A-Z] naming heuristic fallback
- MRO processor (Phase 4.5): walks inheritance DAG, detects method-name
  collisions across parents, applies language-specific resolution:
  - C++: leftmost base class in declaration order wins
  - C#/Java: class method wins over interface default
  - Python: C3 linearization with cycle detection
  - Rust: no auto-resolution (requires qualified syntax)
  - Default: first definition in BFS order wins
- OVERRIDES edges emitted for resolved method collisions
- KuzuDB schema: Method table extended with parameterCount/returnType;
  dedicated CSV writer and COPY query for 10-column Method rows
- MCP tools: updated Cypher examples for HAS_METHOD, OVERRIDES, diamond

72 tests across 5 test files covering MRO resolution, HAS_METHOD edges,
method signature extraction, C# heritage resolution, and integration
tests across C#/Rust/Python/TS/Java/C++.

* feat: add scope-based symbol resolution replacing raw lookupFuzzy

Introduces a shared 3-tier resolveSymbol function used by both
heritage-processor and call-processor:
1. Same-file (lookupExactFull — authoritative)
2. Import-scoped (filtered by ImportMap — high confidence)
3. Global fuzzy (first match — low confidence fallback)

Adds lookupExactFull to SymbolTable returning full SymbolDefinition
with type info needed for heritage Class/Interface disambiguation.

* refactor: tighten symbol resolution — Tier 3 refuses ambiguous matches

- lookupExactFull now O(1) via direct SymbolDefinition storage in fileIndex
  (shared object references with globalIndex — zero additional memory)
- Added resolveSymbolInternal() preserving { definition, tier, candidateCount }
  for test assertions and logging
- Tier 3 now returns null when multiple global candidates exist instead of
  arbitrary allDefs[0] — a wrong edge is worse than no edge
- call-processor: renamed fuzzy-global → unique-global, removed dead branch
- 12 new tests: tier assertions, ambiguous refusal per language family,
  heritage false-positive guard, O(1) shared reference verification

* fix: critical language support bugs in import resolution and MRO

Phase 5 critical fixes from all-language analysis:
- Python: add relative_import query capture (PEP 328) — `.models`, `..utils`
  were silently dropped, producing zero ImportMap entries
- Rust: extract prefix from grouped imports `crate::module::{A, B}` — brace
  groups previously failed resolution entirely
- Swift: use normalizedFileList for Windows path compatibility in module
  import resolution (matches Go's resolveGoPackage pattern)
- MRO: fix c_sharp → csharp language name mismatch (enum is 'csharp'),
  add Kotlin to C#/Java resolution rules (class method wins over interface)

* feat: add strict multi-language integration tests + fix C/C++ import resolution

Add 32 integration tests across 6 language fixtures (TypeScript, C#, C++,
Java, Python, Rust) with exact toBe/toEqual assertions validating heritage
edges, import resolution, and trait implementations.

Fix C/C++ import resolution bug where dot-to-slash conversion mangled
include paths (e.g. "animal.h" became "animal/h"). Now skips conversion
for C/C++ languages which use actual file paths in #include directives.

* fix: language-gate heritage heuristic, add Swift extension heritage, handle Rust grouped imports

- Gate I[A-Z] naming heuristic to C#/Java only (was firing for all languages)
- Swift unresolved types default to IMPLEMENTS (protocol conformance is the norm)
- Add tree-sitter query for Swift extension protocol conformance (extension Foo: Protocol)
- Handle Rust top-level grouped imports (use {crate::a, crate::b}) in both import loops
- Add 4 new heritage-processor tests (TypeScript refusal, Swift default, Swift Tier 1)

* feat: add Go struct embedding heritage + PackageMap optimization

Add Go struct embedding detection (anonymous fields → EXTENDS edges) via
new tree-sitter heritage query with named-field filtering in both
parse-worker and heritage-processor paths.

Implement PackageMap optimization for Go cross-package resolution:
replace O(N) file-level ImportMap expansion with directory-level suffix
matching (Tier 2b in symbol resolver). Graph IMPORTS edges are preserved
via addImportGraphEdge split.

Remove overly broad @definition.type from GO_QUERIES that was
double-matching structs/interfaces as TypeAlias nodes, breaking Tier 3
unique-global resolution.

Add Go fixture (go-pkg) with Admin→User embedding, cross-package calls,
and 7 integration tests covering structs, functions, imports, calls,
and heritage edges.

* test: add Kotlin heritage integration tests

Adds a kotlin-heritage fixture and 7 integration tests validating
class inheritance, interface implementation, JVM-style import
resolution, and symbol-table-driven EXTENDS/IMPLEMENTS disambiguation
via Kotlin delegation specifiers.

* feat: extract resolvers, add PHP tests, ambiguous tests for all languages

- Extract language-specific resolvers from import-processor.ts into
  resolvers/ directory (P7): jvm, go, csharp, php, rust, standard, utils
- import-processor.ts reduced from 1412 to 711 lines (50% reduction)
- Add comprehensive PHP integration tests: PSR-4 imports, traits, enums,
  heritage edges, method calls, MRO overrides
- Add ambiguous symbol resolution tests for all 9 languages verifying
  correct disambiguation via import chains
- Split monolithic lang-resolution.test.ts (1080 lines) into 9 per-language
  files under test/integration/resolvers/ with shared helpers

* feat: update integration tests to include resolver tests for multiple languages

* fix: address code review — schema gap, Rust impl name, Property OVERRIDES

Bugs fixed:
- Add 13 missing FROM/TO pairs in RELATION_SCHEMA for HAS_METHOD edges
  (Class/Interface/Struct/Trait/Impl/Record to Method/Constructor/Property)
- Fix findEnclosingClassId to pick implementing type for Rust
  impl Trait for Struct blocks (was picking trait name)
- Exclude Property nodes from MRO OVERRIDES collision detection
- Change MRO language fallback from typescript to unknown

Tests added:
- Unit: Property OVERRIDES exclusion (2 tests), Rust impl Trait for
  Struct name resolution (2 tests), schema HAS_METHOD pair coverage
- Integration: no OVERRIDES targets Property nodes across all 9 languages
- PHP fixture: added shared $status property to both traits to create
  real collision scenario for Property OVERRIDES exclusion test

Documentation:
- OVERRIDES edge direction (Class to Method), Go return type gap,
  BFS first-reach heuristic limitation

* feat: harden CALLS-edge resolution — Phase 0 validation

- Fix same-file confidence (0.85 → 0.95) to correctly outrank import-scoped (0.9)
- Fix Tier 1 overload preservation: use globalIndex filter instead of fileIndex lookup
- Add callable-kind guard: refuse CALLS edges to Interface and Enum symbols
- Fix Kotlin countCallArguments: handle call_suffix → value_arguments nesting
- Fix Kotlin extractFunctionName: add simple_identifier to fallback search
- Strictly type findParameterList and countCallArguments (remove all `any`)
- Add arity-based call resolution integration tests for 9 languages
- Add unit regression tests for Interface/Enum CALLS refusal

* chore: remove C# build artifacts from fixtures

* feat: add call-form discrimination and ownerId to symbol table (Phase 1)

Add inferCallForm() and extractReceiverName() to distinguish free/member/constructor
calls at the AST level across all 9 languages. Add ownerId field to SymbolDefinition
linking Method/Constructor/Property to their owning class. Includes 36 unit tests
and member-call integration tests for all 9 languages (132 tests, 0 failures).

* feat: constructor/struct-literal resolution across all languages (Phase 2)

Add constructor discrimination to CALLS-edge resolution: new Foo(),
User{...} struct literals, and C# primary constructors now resolve to
Constructor/Class/Struct/Record nodes instead of being filtered out.

Queries: new_expression (C++), object_creation_expression (PHP),
composite_literal (Go), struct_expression (Rust), primary constructor
and implicit_object_creation_expression (C#).

Relaxes global tier in collectTieredCandidates to pass all candidates
through filterCallableCandidates, allowing kind/arity narrowing to
disambiguate at lower confidence.

* feat: receiver-constrained resolution with integration tests for all 9 languages

Add receiver-type filtering (Phase 3): when a member call like `user.save()`
has a known receiver type from TypeEnv, filter candidates by ownerId to
disambiguate methods with the same name across different classes.

Key changes:
- call-processor: build per-file TypeEnv, pass receiverTypeName to resolveCallTarget
- parse-worker: extract receiverTypeName from TypeEnv in worker thread
- resolveCallTarget: new step D filters by ownerId matching receiver type
- utils: extractReceiverName supports C++ field_expression (argument field)
- utils: findEnclosingClassId extracts Go method receiver types
- type-env: handle Go qualified_type, Kotlin user_type/variable_declaration
- parse-worker + parsing-processor: Function added to needsOwner for
  Kotlin/Rust/Python class methods captured as Function nodes

Integration tests added for receiver-constrained resolution across all 9
languages: TypeScript, Java, Python, Go, Rust, C++, C#, Kotlin, PHP.

* feat: NamedImportMap, scoped TypeEnv, broadened signatures + TS rest-param variadic fix

Address all 4 PR #238 review items:
1. Remove redundant lookupFuzzy in processRoutesFromExtracted
2. Add NamedImportMap for TS/Python symbol-level import tracking (Tier 2a)
3. Make TypeEnv scope-aware (Map<scopeKey, Map<varName, type>>) to fix
   non-deterministic receiver resolution across functions
4. Broaden extractMethodSignature: Go/Rust/C++ return types, variadic
   detection for Go/Java/Python/C++/Kotlin/TypeScript rest params

Discovered and fixed: TS rest params (...args) were not detected as
variadic — added rest_pattern detection inside required_parameter nodes.

Integration tests added: scoped receiver, named import disambiguation,
and variadic call resolution for both TypeScript and Python.

* fix: alias import resolution, Go multi-assign TypeEnv, dead code removal

- NamedImportMap now stores {sourcePath, exportedName} so aliased imports
  (import { User as U }) resolve U → User in the source file
- Named binding check moved before empty-allDefs early return in both
  call-processor and symbol-resolver, fixing constructor calls via aliases
- Go extractFromGoShortVarDeclaration iterates all LHS/RHS pairs for
  multi-assignment (user, repo := User{}, Repo{}) instead of only first
- Remove unused TYPED_DECLARATION_TYPES set (TYPED_PARAMETER_TYPES kept)
- Integration tests for both fixes (go-multi-assign, typescript-alias-imports)

* feat: alias import extraction for Kotlin, Rust, PHP, C# + integration tests

Add named import alias extraction to both pipeline paths
(import-processor.ts and parse-worker.ts) for Kotlin, Rust, PHP,
and C#. Add integration test fixtures and tests for all 5 languages
(Python alias extraction already worked, just needed the test).

Each test verifies: class detection, member call resolution through
aliases to correct target files, and IMPORTS edge emission.

* refactor: use SupportedLanguages enum everywhere instead of raw strings

Replace all raw language string literals and `language: string` types
with the SupportedLanguages enum across 10 files. This ensures
compile-time safety for language dispatch and eliminates dead
`language === 'tsx'` checks (tsx maps to TypeScript in the enum).

* fix: tier-ordering bug, re-export chains, PHP grouped imports, Java named imports

- Fix collectTieredCandidates tier-ordering: same-file now checked before
  named bindings, preventing imports from shadowing local definitions
  (matches resolveSymbolInternal priority order)
- Add re-export chain resolution for TypeScript/JavaScript barrel files:
  export { X } from './base' and export type { X } from './base' now
  followed up to 5 hops through NamedImportMap
- Fix PHP grouped import alias extraction: use App\Models\{User, Repo as R}
  now correctly handled in both parse-worker and import-processor
- Add Java NamedImportMap support: import com.example.models.User now
  records User as a named binding for precise disambiguation
- Add 16 new integration tests across TypeScript, PHP, and Java resolvers
  (220 total resolver tests, all passing)

* refactor: consolidate alias extraction + add variadic/constructor/shadow integration tests

- Extract shared named-binding-extraction.ts from duplicate logic in
  import-processor.ts and parse-worker.ts (net -200 lines)
- Deduplicate appendKotlinWildcard (now imported from resolvers/index.ts)
- Add integration tests: constructor calls (Kotlin, Python), variadic
  resolution (Go, Java, C#, C++, Kotlin), re-export chains (Python),
  local definition shadowing (Python, Go)
- Add TODO(stack-graph) for TypeEnv scope key collision
- 225 integration tests passing (was 223)

* fix: PHP non-aliased imports, Python node identity, re-export chain dedup + local-shadow tests

- PHP flat non-aliased imports (use App\Models\User) now stored in NamedImportMap
- PHP grouped non-aliased imports ({User} in {User, Repo as R}) now stored in NamedImportMap
- Python: replace non-public child.id with child.startIndex for node identity
- Extract shared walkBindingChain() from symbol-resolver and call-processor
- Add PHP variadic resolution fixture + test (variadic_parameter already covers PHP)
- Add local-shadow integration tests for Java, C#, Kotlin, Rust, PHP, C++ (6 languages)

* feat: Rust non-aliased use bindings, Kotlin non-aliased imports, re-export chain resolution

Extend NamedImportMap coverage for Rust and Kotlin non-aliased imports:

- Rust: rename collectUseAsClauses → collectRustBindings, extract terminal
  scoped_identifier (use crate::models::User) and identifier in use_list
  (use crate::models::{User, Repo}) into NamedImportMap. This also enables
  pub use re-export chain following via walkBindingChain.
- Kotlin: extend extractKotlinNamedBindings to handle non-aliased imports
  (import com.example.User), skipping wildcard imports.
- Add rust-reexport-chain fixture + 3 integration tests verifying Handler{}
  resolves through mod.rs pub use to handler.rs.
- Add Kotlin heritage + constructor-calls reason assertions for non-aliased
  import-resolved resolution.
- Add C# heritage test documenting namespace import tier behavior.

* fix: skip Kotlin lowercase member imports in NamedImportMap

Member imports like `import util.OneArg.writeAudit` (lowercase last
segment) must not populate NamedImportMap — same-named function imports
from different classes collide, breaking arity-based disambiguation.
Apply the same guard Java already uses: skip lowercase last segments.

* fix: skip spurious path-prefix bindings in Rust grouped imports

collectRustBindings was extracting the path segment (e.g. "models") from
`use crate::models::{User, Repo}` as a spurious NamedImportMap entry.
Skip scoped_identifier nodes that are direct children of scoped_use_list
since they are path prefixes, not importable symbols.

Adds rust-grouped-imports fixture and 4 integration tests verifying both
symbols resolve correctly and no spurious binding leaks through.

* fix: use startIndex in TypeEnv scope key to prevent same-name method collision

Two methods named identically in different classes within the same file
previously shared a scope key, causing non-deterministic type resolution.
Now keys use funcName@startIndex for uniqueness.

Also adds tests documenting destructuring assignment extraction gap.

* test: document C# namespace-level import limitation in named binding extraction

* test: document same-arity overload discrimination limitation in call processor

* perf: parallelize calls/heritage/routes processing in worker path

Worker path now runs processCallsFromExtracted, processHeritageFromExtracted,
and processRoutesFromExtracted via Promise.all instead of sequentially.
Safe because all three only read shared state and write via addRelationship's
dedup guard. Sequential fallback path stays sequential (shared LRU astCache).

Also fixes Rust collectRustBindings spurious path-prefix bindings for 3+ level
grouped imports, and adds @param JSDoc for walkBindingChain's allDefs invariant.

* docs: improve Promise.all safety comment and walkBindingChain JSDoc

Clarify that the parallelization safety comes from disjoint relationship
types + idempotent id-keyed Maps, not from lack of shared state (the
graph is shared). Strengthen allDefs JSDoc to describe silent-miss
consequence of passing pre-filtered results.

* refactor: extract language-specific processing into modular dispatch tables

Phase 1: Extract type binding logic from type-env.ts (635→125 LOC) into
type-extractors/ directory with per-language files and Record<SupportedLanguages,
LanguageTypeConfig> + satisfies dispatch.

Phase 2: Extract 5 config loaders from import-processor.ts into
language-config.ts (removed ~196 LOC of inline loaders).

Phase 3: Convert export-detection.ts switch/case to exhaustive
Record<SupportedLanguages, ExportChecker> + satisfies dispatch table,
fix node: any → SyntaxNode.

Also adds language feature matrix to README.

All 1146 unit tests and 433 integration tests pass.

* refactor: extract type binding logic into type-extractors/ directory (Phase 1)

Extract per-language type extraction from type-env.ts (635→125 LOC) into
type-extractors/ with Record<SupportedLanguages, LanguageTypeConfig> + satisfies
dispatch. 9 per-language files, shared helpers, and barrel index.

* refactor: extract config loaders to language-config.ts (Phase 2)

Move 5 language-specific config loaders and their type interfaces from
import-processor.ts into standalone language-config.ts module.
2026-03-13 13:12:23 +00:00
Zander Raycraft 03bfa3c4d9 FEAT: Added support for optional skill generation based on KuzuDB after initial repo analysis (npx gitnexus analyze --skills) (#171)
* calm fix 4 adding skills to repo [ISSUE #140]

* inspect

* unit and integration tests

* fixed hardcoded cohesion miss

* e2e tests for --skills flag for langauge/repo support

* Cohesion test e2e tests
2026-03-13 08:29:13 +00:00
Subham Kundu 74c0e462c3 Merge pull request #217 from JasonOA888/feat/issue-215-deepseek-model
feat(models): add DeepSeek model configurations
2026-03-13 00:31:02 +05:30
Gergő Magyar 7376e92063 fix: consolidate C/C++/C#/Rust language support from 6 overlapping PRs (#237)
* fix: consolidate C/C++/C#/Rust language support from 6 overlapping PRs

Merges fixes from PRs #163, #170, #178, #216, #227, #234 into a single
coherent changeset with shared modules and deduplication.

Phase 0 — Pre-merge consolidation:
- Extract isNodeExported to shared export-detection.ts module
- Extract TREE_SITTER_BUFFER_SIZE to shared constants.ts with adaptive sizing
- Consolidate FUNCTION_NODE_TYPES, extractFunctionName, isBuiltInOrNoise
  from duplicated call-processor.ts and parse-worker.ts into shared utils.ts
- Add query compilation smoke tests for all 12 languages

Language fixes:
- fix(c/cpp): isExported checks static linkage instead of returning false
- fix(c/cpp): .h files parsed as C++ (tree-sitter-cpp is superset of C)
- fix(c/cpp): expanded entry point patterns (~30 new for C, ~18 for C++)
- fix(cpp): add typedef, union, macro, prototype, inline method queries
- fix(c#): isExported scans sibling modifiers instead of parent walk
- fix(c#): heritage queries use correct base_list AST structure
- fix(c#): add framework detection, import resolution, entry point scoring
- fix(rust): isExported scans sibling visibility_modifier in declaration
- fix(builtins): remove open/read/write/close (real C POSIX syscalls)
- fix(buffer): adaptive bufferSize (2x fileSize, 512KB-32MB range)
- feat(ts/js): add call_expression query patterns for const assignments

Deduplication:
- call-processor.ts: -226 lines (uses shared utils)
- parse-worker.ts: -320 lines (uses shared utils)
- parsing-processor.ts: -156 lines (uses shared export-detection)

* perf: fix review findings — hoist Sets, deduplicate DEFINITION_CAPTURE_KEYS

- Hoist CSHARP_DECL_TYPES and RUST_DECL_TYPES to module-level constants
  in export-detection.ts (was allocating new Set on every isNodeExported call)
- Extract DEFINITION_CAPTURE_KEYS and getDefinitionNodeFromCaptures to
  shared utils.ts (was duplicated in parsing-processor.ts and parse-worker.ts)
- Pre-compute merged entry point patterns to avoid per-call array spread
  in calculateEntryPointScore

* test: add C, C++, and Tree-sitter buffer size tests

* fix: C/C++/Rust review findings + comprehensive test coverage (+72 tests)

Source fixes:
- Add Rust built-in noise (unwrap, clone, into, collect, panic, etc.)
- C++ anonymous namespace → internal linkage (not exported)
- Replace .text regex with storage_class_specifier child scan (perf)
- Raise file skip threshold from 512KB to 32MB (TREE_SITTER_MAX_BUFFER)
- Export TREE_SITTER_MAX_BUFFER from constants.ts
- Add C++ double pointer query patterns to CPP_QUERIES
- Add C#: record_struct, record_class, file_scoped_namespace to decl types
- Add Rust: union_item to visibility scanning set

Tests (214 → 286):
- ingestion-utils: +24 (Rust/C# noise, pointer/ref/destructor extraction, buffer)
- parsing: +36 (real AST C/C++ static/namespace, Rust/C#/Java/PHP/Swift edge cases)
- tree-sitter-languages: +12 (query accuracy for C/C++/C#/Rust captures)
2026-03-10 23:03:32 +00:00
RyanbaandGergo Magyar 1be910f54a fix: skip unavailable native Swift parsers in sequential ingestion (#188)
* fix: skip unavailable native Swift parsers in sequential ingestion

* fix: warn when ingestion skips languages in verbose mode

* test: cover verbose skip warnings

* docs: update analyze flags

* docs: clarify verbose default

---------

Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-10 14:26:36 +00:00
Gergo Magyar fa9ba8925c fix(ci): support fork PRs in Claude Code Review workflow
claude-code-action fetches branches by name from origin, which fails
for fork PRs since the branch only exists on the fork remote. Work
around by detecting fork PRs and temporarily pushing the branch to
origin before the action runs, then cleaning up afterwards.

Also changed trigger from automatic (every push) to on-demand only
(label "claude-review" or comment "@claude" / "/review").
2026-03-09 15:33:41 +00:00
8efc272609 fix(ci): move PR report to workflow_run for fork PR support (#225)
* Initial plan

* fix: add pull-requests write permissions to GitHub Actions workflows

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix(ci): remove ineffective job-level permissions from reusable workflow

* fix(ci): pass PR write permission from caller to reusable unit-tests workflow

* fix(ci): harden CI/CD workflows with security fixes and reliability improvements

- Pin all actions to commit SHAs to prevent supply-chain attacks
- Fix shell injection in ci-integration.yml by using env vars instead of direct interpolation
- Scope permissions per-job in publish.yml (was granting pull-requests:write to publish job)
- Restrict claude-code-review to trusted contributors only (OWNER/MEMBER/COLLABORATOR)
- Switch claude-code-review to pull_request_target for fork PR support
- Fix fail-fast: false in ci-unit-tests.yml cross-platform matrix
- Remove duplicate ubuntu-latest from unit test matrix
- Add timeouts to all workflow jobs
- Improve kuzu-db test loop to continue on failure and report per-file errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ci): read thresholds from `vitest.config.ts`

* fix(ci): move PR report to workflow_run for fork PR support

The sticky-pull-request-comment and vitest-coverage-report-action
both fail on fork PRs because pull_request events receive a read-only
GITHUB_TOKEN. This extracts PR reporting into a separate ci-report.yml
workflow triggered by workflow_run, which always gets read/write tokens.

Changes:
- ci.yml: replace pr-report job with save-pr-meta artifact upload
- ci-unit-tests.yml: remove davelosert/vitest-coverage-report-action,
  add coverage-final.json to artifact for merging
- ci-integration.yml: add ubuntu coverage job for non-kuzu groups
- ci-report.yml (new): workflow_run handler that downloads artifacts,
  merges unit + integration coverage via Istanbul, and posts combined
  PR comment with sticky-pull-request-comment

* feat(ci): show unit, integration, and merged coverage in PR report

- Disable coverage thresholds for integration-only run (partial coverage)
- Display combined coverage as the primary metric
- Show per-suite breakdown (unit / integration) in expandable details
- Thresholds applied against combined coverage, not individual suites

* fix(ci): add coverage collection input for PR reports and validate job results

* fix(ci): refine Claude Code Review workflow to support issue comments and enhance trusted contributor checks

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 15:07:46 +00:00
c990d7e6c6 fix(ci): harden CI/CD workflows with security fixes and reliability improvements (#222)
* Initial plan

* fix: add pull-requests write permissions to GitHub Actions workflows

Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>

* fix(ci): remove ineffective job-level permissions from reusable workflow

* fix(ci): pass PR write permission from caller to reusable unit-tests workflow

* fix(ci): harden CI/CD workflows with security fixes and reliability improvements

- Pin all actions to commit SHAs to prevent supply-chain attacks
- Fix shell injection in ci-integration.yml by using env vars instead of direct interpolation
- Scope permissions per-job in publish.yml (was granting pull-requests:write to publish job)
- Restrict claude-code-review to trusted contributors only (OWNER/MEMBER/COLLABORATOR)
- Switch claude-code-review to pull_request_target for fork PR support
- Fix fail-fast: false in ci-unit-tests.yml cross-platform matrix
- Remove duplicate ubuntu-latest from unit test matrix
- Add timeouts to all workflow jobs
- Improve kuzu-db test loop to continue on failure and report per-file errors

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix(ci): read thresholds from `vitest.config.ts`

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: magyargergo <11230420+magyargergo@users.noreply.github.com>
Co-authored-by: Gergő Magyar <gergomagyar@icloud.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-09 08:25:03 +00:00
Gergo Magyar b4fbf33bd6 fix(ci): remove workflow-level permissions from reusable workflows 2026-03-08 18:20:05 +00:00
Gergő MagyarandClaude Opus 4.6 892e1d6088 test: add integration test coverage and fix KuzuDB fork crashes (#209)
* ci: add macOS to cross-platform test matrix

* ci: run integration tests on all platforms, add macOS to matrix

* ci: add build step before cross-platform integration tests

Worker pool requires compiled parse-worker.js in dist/.
Without build, falls back to sequential parsing which times out
on macOS runners.

* fix(pipeline): resolve worker path to dist/ when running under vitest

import.meta.url points to src/ under vitest where no .js exists.
Fall back to dist/core/ingestion/workers/parse-worker.js so worker
threads spawn correctly on all platforms instead of sequential fallback
that times out on slower macOS CI runners.

* ci: split cross-platform unit and integration tests into parallel jobs

* test: add integration tests for worker pool and hooks e2e

- worker-pool.test.ts: 7 tests verifying dist/ worker spawning,
  multi-file parsing, progress reporting, and clean termination
- hooks-e2e.test.ts: 28 tests with real git repos testing staleness
  detection, embeddings flag, mutation regex, cwd validation,
  and .gitnexus directory discovery

* refactor: extract shared hook test helpers and simplify worker fallback

- Extract runHook/parseHookOutput into test/utils/hook-test-helpers.ts
- Deduplicate fileURLToPath calls in pipeline.ts worker resolution
- Add isDev logging for worker pool creation failures

* fix(test): accept timeout as valid outcome for PreToolUse CLI spawn

The Plugin hook spawns `gitnexus augment` which may hang on macOS
when the CLI is unavailable, causing a 10s timeout (status=null)
instead of a clean exit (status=0). Accept both as non-crash outcomes.

* test: add integration test coverage and fix KuzuDB fork crashes

- Add new integration tests: search, enrichment, CLI e2e (968 total tests)
- Fix KuzuDB native destructor segfault in vitest fork pool by adding
  detachKuzu() that nulls refs without calling .close()
- Merge core adapter test blocks to share one coreHandle (prevents
  multiple coreInitKuzu calls that re-open native DB handles)
- Fix FTS Cypher injection: escape backslashes in bm25-index.ts and
  kuzu-adapter.ts queryFTS
- Add worker script existence check in worker-pool.ts to prevent
  MODULE_NOT_FOUND crashes in worker threads
- Add test/setup.ts global teardown that detaches native refs
- Add test/helpers/test-indexed-db.ts shared KuzuDB test lifecycle helper

* fix(test): update worker-pool test to expect throw on invalid path

The fs.existsSync validation in createWorkerPool now throws
synchronously for missing worker scripts. Update the test assertion
from .not.toThrow() to .toThrow(/Worker script not found/).

* fix(test): use fileParallelism instead of deprecated singleFork

vitest 4.x removed poolOptions.forks.singleFork. The top-level
singleFork was silently ignored, causing multiple forks to spawn
and timeout during KuzuDB native cleanup on CI.

* fix(test): add maxWorkers: 1 to prevent per-file kuzu native addon reload

On Ubuntu CI, vitest forks pool creates a new child process per test
file. Each fork loads the KuzuDB native addon (~40s on Ubuntu runners),
causing 12 files × 40s = 8 minutes of overhead that exceeds the
10-minute CI timeout.

maxWorkers: 1 forces vitest to reuse a single fork process, loading
the native addon once. Combined with fileParallelism: false, all test
files run sequentially in that single fork.

* fix(test): prevent KuzuDB native destructor hangs on fork worker exit

- setup.ts: closeKuzu() first (marks native handles closed so destructors
  are no-ops), then detachKuzu() as safety net
- test-indexed-db.ts: use detachKuzu() in per-test cleanup instead of
  closeKuzu() which could hang during teardown

* refactor(test): add withTestKuzuDB lifecycle wrapper with declarative options

withTestKuzuDB now manages the full KuzuDB test lifecycle so test files
never call initKuzu/closeCoreKuzu/poolInitKuzu/loadFTSExtension directly.

Options: seed, ftsIndexes, poolAdapter, afterSetup, timeout.
Each call is wrapped in its own describe block to isolate lifecycle hooks.

Migrated search.test.ts, enrichment-and-augmentation.test.ts, and
kuzu-pool.test.ts core adapter block to use the wrapper.

* refactor(test): migrate all integration tests to withTestKuzuDB

- Split enrichment-and-augmentation.test.ts into enrichment.test.ts
  and augmentation.test.ts for focused test isolation
- Migrate kuzu-pool.test.ts pool lifecycle tests to withTestKuzuDB
- Migrate local-backend.test.ts to two withTestKuzuDB blocks
  (pool queries + callTool dispatch)
- Zero direct kuzu.Database/Connection usage remains in test files

* refactor(test): enforce one describe per test file

- Split search.test.ts → search-core.test.ts + search-pool.test.ts
- Split kuzu-pool.test.ts → kuzu-pool.test.ts + kuzu-core-adapter.test.ts
- Split local-backend.test.ts → local-backend.test.ts + local-backend-calltool.test.ts
- Wrap enrichment.test.ts in single top-level describe
- Wrap parsing.test.ts in single top-level describe
- Every integration test file now has exactly 1 top-level block

* refactor(test): extract shared seed data into fixture files

- Create test/fixtures/search-seed.ts with SEARCH_SEED_DATA and SEARCH_FTS_INDEXES
- Create test/fixtures/local-backend-seed.ts with LOCAL_BACKEND_SEED_DATA and LOCAL_BACKEND_FTS_INDEXES
- Remove duplicated constants from split test files
- Remove dead vi.mock from local-backend.test.ts
- Prefix unused handle param with underscore in search-core.test.ts

* fix(test): prevent KuzuDB C++ destructor hang on Ubuntu CI

Add process.on('beforeExit', () => process.exit(0)) to force
immediate exit before GC can trigger native C++ destructors on
orphaned KuzuDB Database/Connection objects.

Root cause: detachKuzu() nulls JS refs but native C++ objects
remain in V8 heap. During fork worker exit, GC runs finalizers
that invoke C++ destructors on a torn-down runtime — hangs on
Ubuntu, segfaults on Windows.

The beforeExit event fires when the event loop has drained
(test results already sent via IPC), so process.exit(0) is safe.

Also simplifies afterAll: removes closeKuzu() calls (always
no-ops since withTestKuzuDB detaches first) — only detachKuzu().

* perf(test): share single KuzuDB instance across integration tests

Create schema once in globalSetup instead of per-file, eliminating
29 DDL queries × 7 test files. Each file now only clears and reseeds
data via DETACH DELETE, reducing DB open/close cycles significantly.

* fix(test): improve KuzuDB cleanup to prevent C++ destructor hangs on exit

* fix(test): replace async close calls with synchronous counterparts to prevent potential hangs

* feat(ci): enhance integration test matrix with detailed test groups and improved reporting

* test: add diagnostic output to analyze CLI e2e assertion for CI debugging

* fix: pass NODE_OPTIONS in runCli to prevent ensureHeap re-exec in tests

* update gitnexus analysis md files

* feat(ci): modular workflow architecture with artifact reporting

Refactor monolithic ci.yml into orchestrator calling three reusable
workflows (quality, unit-tests, integration) via workflow_call.

- Add composite action for shared Node.js 20 setup and npm ci
- Add ci-quality.yml for TypeScript typecheck
- Add ci-unit-tests.yml with coverage reporting, JSON test results,
  and artifact upload for PR summary comments
- Add ci-integration.yml with 4 test groups x 3 OS matrix (12 jobs)
- Add PR report job with sticky comment showing coverage metrics
- Add unified CI Gate status check for branch protection
- Add explicit permissions blocks to all child workflows

* test: add comprehensive unhappy path coverage across all 16 integration test files

Add 80+ error handling, edge case, and unhappy path tests covering:
- KuzuDB core adapter: invalid Cypher, duplicate FTS index, empty queries, missing paths
- CLI e2e: non-git dirs, non-indexed repos, unknown commands, help flag
- Local backend callTool: missing params, invalid Cypher, nonexistent symbols
- Tree-sitter: unsupported languages, malformed code, empty content, binary files
- Worker pool: dispatch after terminate, double terminate, empty content, zero-size pool
- Pipeline: empty content parsing, flexible file count assertions
- Search, enrichment, augmentation, CSV, hooks, filesystem: various edge cases

Also fixes pre-existing test issues:
- isWriteQuery CREATED test (CYPHER_WRITE_RE uses \b word boundaries)
- KuzuDB throws Binder exception for unknown tables (not empty result)
- runPipelineFromRepo requires onProgress callback

All 1,086 tests pass (53 files).

* fix: prevent KuzuDB worker hang with handle unref strategy and safety-net timer

Replace beforeExit force-exit with per-file handle unref + safety-net timer
that doesn't leak across files in single-fork mode.

* refactor: improve KuzuDB test isolation and cleanup strategy

* fix: prevent KuzuDB N-API destructor hang on Linux/macOS

Pool adapter closeOne() now just deletes the pool entry without calling
native close methods — read-only DBs have no WAL to flush, so GC/process
exit safely reclaims native resources without triggering the C++ destructor
segfault.

withTestKuzuDB wrapper handles core adapter close platform-conditionally:
Windows needs explicit closeKuzu() due to file locks, Linux/macOS skips
it to avoid deadlock. kuzu-pool.test.ts now uses poolAdapter: true instead
of manual afterSetup. pipeline.test.ts assertion fixed to match actual
behavior (resolves with empty result, not rejects).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* fix: restore vitest safety nets and skip globalSetup close on Linux

- Restore dangerouslyIgnoreUnhandledErrors and teardownTimeout in
  vitest.config.ts — KuzuDB N-API destructor segfaults on fork exit
  are not real test failures (all 839 unit tests pass).
- Skip conn.close()/db.close() in globalSetup on Linux/macOS to
  prevent N-API destructor crash that kills the vitest process before
  fork workers can start (fixes search-core.test.ts EPIPE on Ubuntu CI).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat: enable coverage auto-ratcheting with bumped thresholds

- Bump vitest coverage thresholds to match actual CI values (26/23/28/27)
- Enable thresholds.autoUpdate for automatic local ratcheting

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* feat(ci): rich PR report with coverage bars, test counts, and threshold tracking

- Fix coverage N/A bug: use find instead of hardcoded artifact path
- Add emoji status icons and overall pass/fail banner
- Show covered/total counts alongside percentages
- Add visual progress bars with green/red threshold indicators
- Show test suite count and duration
- Add collapsible auto-ratchet explainer
- Graceful fallback when coverage data is unavailable

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

* chore: bump version to 1.3.11, update CHANGELOG, add release.yml

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-08 18:00:45 +00:00
Jason c2bd8667a3 feat(models): add DeepSeek model configurations
Add configurations for DeepSeek-V3 and DeepSeek-Chat models
via OpenRouter integration.

- DeepSeek-V3: Reasoning model (input: /usr/bin/bash.27, output: .10)
- DeepSeek-Chat: Chat model (input: /usr/bin/bash.14, output: /usr/bin/bash.28)

Fixes #215
2026-03-08 15:50:25 +08:00
Gergő Magyar 1952c2c346 ci: add macOS to cross-platform test matrix (#208)
* ci: add macOS to cross-platform test matrix

* ci: run integration tests on all platforms, add macOS to matrix

* ci: add build step before cross-platform integration tests

Worker pool requires compiled parse-worker.js in dist/.
Without build, falls back to sequential parsing which times out
on macOS runners.

* fix(pipeline): resolve worker path to dist/ when running under vitest

import.meta.url points to src/ under vitest where no .js exists.
Fall back to dist/core/ingestion/workers/parse-worker.js so worker
threads spawn correctly on all platforms instead of sequential fallback
that times out on slower macOS CI runners.

* ci: split cross-platform unit and integration tests into parallel jobs

* test: add integration tests for worker pool and hooks e2e

- worker-pool.test.ts: 7 tests verifying dist/ worker spawning,
  multi-file parsing, progress reporting, and clean termination
- hooks-e2e.test.ts: 28 tests with real git repos testing staleness
  detection, embeddings flag, mutation regex, cwd validation,
  and .gitnexus directory discovery

* refactor: extract shared hook test helpers and simplify worker fallback

- Extract runHook/parseHookOutput into test/utils/hook-test-helpers.ts
- Deduplicate fileURLToPath calls in pipeline.ts worker resolution
- Add isDev logging for worker pool creation failures

* fix(test): accept timeout as valid outcome for PreToolUse CLI spawn

The Plugin hook spawns `gitnexus augment` which may hang on macOS
when the CLI is unavailable, causing a 10s timeout (status=null)
instead of a clean exit (status=0). Accept both as non-crash outcomes.
2026-03-07 10:38:32 +00:00
Linus Beckhaus c4eaf45ab1 feat(hooks): auto-reindex notification with cross-platform hardening (#205)
Adds PostToolUse hook that detects stale GitNexus index after git mutations (commit, merge, rebase, cherry-pick, pull) and notifies the agent to reindex. Uses lightweight staleness check (git rev-parse HEAD vs meta.json) instead of running gitnexus analyze synchronously, avoiding KuzuDB corruption and 120s blocks. Security and cross-platform hardening: remove shell:true from all spawnSync calls, use .cmd extensions on Windows, add path.isAbsolute(cwd) guards, fix setup.ts path escaping with JSON.stringify, use sendHookResponse() consistently. Includes 73 regression tests.
2026-03-07 08:59:54 +00:00
Gergo Magyar 0796e1e68c chore: bump version to 1.3.10 and add CHANGELOG
Add CHANGELOG.md with release notes for v1.3.10 covering MCP transport
security hardening, dual-framing compatibility, lazy CLI loading, and
bug fixes from recent PRs.
2026-03-07 08:04:55 +00:00
ShockangandGergo Magyar 9d5ec5d19a Improve MCP startup compatibility and lazy-load CLI commands (#207)
* Fix MCP startup transport compatibility

* Preserve CLI flags in MCP startup fix

* Harden MCP transport error handling

* Harden transport security and improve type safety

Transport hardening:
- Add MAX_BUFFER_SIZE (10 MB) cap to prevent OOM from oversized
  Content-Length or unbounded newline-delimited input
- Replace recursive readNewlineMessage with iterative loop to prevent
  stack overflow from consecutive empty lines
- Tighten looksLikeContentLength to require 14+ bytes before matching
- Add closed-state guard and error handling to send()
- Simplify processReadBuffer loop to break on error
- Fix loose equality (==) to strict (===)
- Widen constructor param types to ReadableStream/WritableStream

Type safety:
- Constrain createLazyAction generics so export name is validated
  against the module's actual exports at compile time
- Use proper type guard instead of lint suppression
- Fix test tsconfig type errors

Regression tests for all hardening fixes (13 tests passing).

---------

Co-authored-by: Gergo Magyar <gergomagyar@icloud.com>
2026-03-07 07:47:09 +00:00
abhigyanpatwariandClaude Opus 4.6 4de40e4011 chore: update AI context files with inline imperative instructions
Regenerated CLAUDE.md and AGENTS.md using gitnexus@1.3.9 which replaces
the old skill-router format with inline imperative instructions (PR #190).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 15:12:00 +05:30
Gergő Magyar 20e8c52028 Merge pull request #144 from magyargergo/fix/lru-cache-zero-max-crash
fix: guard createASTCache against zero maxSize to prevent LRU cache crash
2026-03-06 09:22:17 +00:00
Gary Magyar 2868da5ddb Merge remote-tracking branch 'origin/main' into fix/lru-cache-zero-max-crash 2026-03-06 09:02:50 +00:00
Abhigyan PatwariandClaude Opus 4.6 3db47f7ee5 fix(ingestion): align CALLS edge sourceId with node ID format (#194)
findEnclosingFunctionId generated IDs without :startLine suffix,
but node creation includes it. This caused every CALLS edge to
reference a non-existent source node, making the process detector
find 0 entry points and produce 0 execution flows.

Bumps to 1.3.9.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 13:57:20 +05:30
Abhigyan PatwariandClaude Opus 4.6 821871cec1 chore: bump version to 1.3.8 (#193)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 13:23:01 +05:30
Abhigyan PatwariandClaude Opus 4.6 f9a54cd588 fix(cli): force-exit after analyze to prevent KuzuDB hang (#192)
KuzuDB's native module holds open handles that prevent Node.js from
exiting cleanly. Previously only force-exited when embeddings were used
(for ONNX Runtime segfault workaround), but the same issue affects all
analyze runs. Now always calls process.exit(0) after completion.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 13:17:41 +05:30
Abhigyan PatwariandClaude Opus 4.6 8c6b064d18 chore: bump version to 1.3.7 (#191)
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 12:58:35 +05:30
Abhigyan PatwariandClaude Opus 4.6 84ef6524bc feat(ai-context): replace skill router with inline imperative instructions (#190)
CLAUDE.md and AGENTS.md now contain direct enforcement instructions instead
of a passive skill router table. Based on Vercel eval data showing skills
are skipped 56% of the time, and industry research on effective AGENTS.md
patterns from 2,500+ repos.

Key changes:
- Always/When/Never three-tier boundary structure
- RFC 2119 language (MUST, NEVER) for critical rules
- Exact tool commands with parameters inline
- Self-check checklist forcing model to verify its own work
- ~77 lines, well within the <150 line adherence threshold

Skills are still installed as bonus depth for Claude Code's skill system.

Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-06 12:47:23 +05:30
abhigyanpatwariandClaude Opus 4.6 5674b2201d feat: merge Laravel route detection (PR #133), revert unwanted doc changes
Merged PR #133 which adds AST-based Laravel Route::* extraction.
Reverted AGENTS.md, CLAUDE.md, and README.md to preserve current config,
crypto warning, Discord link, and correct language support count (12,
including Kotlin/Swift).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-03 00:20:27 +05:30
abhigyanpatwari 6915a9350b Merge branch 'pr-133' 2026-03-03 00:19:44 +05:30
Gary Magyar 76e0e5a35a fix: guard createASTCache against zero maxSize to prevent LRU cache crash
When a repo has no parseable files (e.g., unsupported languages or all
files filtered out), chunks.reduce returns 0, causing createASTCache(0)
to pass max:0 to LRUCache which throws TypeError. This clamps maxSize
to at least 1 and adds a progress message when no parseable files exist.
2026-03-02 08:47:20 +00:00
abhigyanpatwariandClaude Opus 4.6 8e7d976c2a fix: gracefully skip files when language parser is unavailable (#136)
Instead of crashing the pipeline when a native tree-sitter binding
(e.g. tree-sitter-swift) fails to build, skip those files early and
warn the user with an actionable message.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-02 09:23:05 +05:30
Güneş Bizim 46b4b7e157 Merge origin/main — add Kotlin/Swift support, resolve conflicts
- Resolved FUNCTION_NODE_TYPES: keep 'anonymous_function' for PHP (php_only grammar),
  add Kotlin 'lambda_literal' and Swift 'init_declaration'/'deinit_declaration'
- Resolved pipeline.ts: adopt chunked pipeline structure, integrate
  processRoutesFromExtracted into per-chunk worker data processing
- Resolved framework-detection.ts: use upstream AST-BASED FRAMEWORK DETECTION heading
- Fixed accumulated/mergeResult in parse-worker to include routes field
2026-03-01 22:33:23 +03:00
abhigyanpatwariandClaude Opus 4.6 cbeb0e231a docs: add Kotlin to supported languages in README
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:23:04 +05:30
abhigyanpatwariandClaude Opus 4.6 3431edcea0 fix(test): update ingestion-utils test for Kotlin support
Move .kt from unsupported list to supported, add Kotlin test case
for .kt and .kts extensions.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:18:21 +05:30
abhigyanpatwariandClaude Opus 4.6 40cb863cb4 chore: update package-lock.json with tree-sitter-kotlin
The Kotlin PR added tree-sitter-kotlin to package.json but didn't
include the lockfile update, causing npm ci to fail in CI.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 23:14:36 +05:30
abhigyanpatwari ee95808478 Merge pull request #84 from magyargergo/feat/kotlin-language-support
feat: add Kotlin language support
2026-03-01 23:11:25 +05:30
abhigyanpatwari 3e3ea86ce4 Merge origin/main into feat/kotlin-language-support 2026-03-01 23:10:52 +05:30
abhigyanpatwariandClaude Opus 4.6 1a52d05131 fix(test): use dangerouslyIgnoreUnhandledErrors instead of forceExit
forceExit killed the fork worker before local-backend.test.ts finished,
losing 12 test results. The real issue is KuzuDB's C++ destructor
segfaulting during fork process exit — all tests pass but vitest
reports the post-test crash as a failure.

dangerouslyIgnoreUnhandledErrors ignores the process-level crash
without affecting test results (98/98 tests still run and report).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 22:36:38 +05:30
abhigyanpatwariandClaude Opus 4.6 3d64e26f8f fix(test): add forceExit to prevent KuzuDB native cleanup hang in CI
KuzuDB's C++ destructor crashes the vitest fork worker on exit,
causing a ~7 minute hang before timeout. forceExit kills the
worker immediately after tests complete.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 20:33:18 +05:30
abhigyanpatwariandClaude Opus 4.6 3576802574 fix(test): use HEAD~1 instead of root commit in staleness test
GitHub Actions shallow clones don't have the root commit available,
causing checkStaleness to fail silently. HEAD~1 is always available.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 20:23:06 +05:30
abhigyanpatwariandClaude Opus 4.6 20ebd6b781 feat: security hardening, MCP improvements, skills, hooks, and CLI updates
- Export security primitives (CYPHER_WRITE_RE, isWriteQuery, isTestFilePath,
  VALID_NODE_LABELS, VALID_RELATION_TYPES) from local-backend
- Improve MCP kuzu-adapter with better query handling
- Add PR review skill for Claude, Cursor, and npm package
- Add CLI guide and CLI skills
- Update hooks for Claude plugin and Cursor integration
- Remove deprecated claude-hooks.ts CLI module
- Update eval-server, setup, and analyze CLI commands
- Improve CSV generator and ingestion processors
- Update CLAUDE.md and AGENTS.md configs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 20:13:42 +05:30
abhigyanpatwariandClaude Opus 4.6 8a100a76d3 test: add test suite with vitest (unit + integration + fixtures)
- 59 test files covering unit and integration tests
- vitest config with coverage thresholds and fork pooling
- Test fixtures (mini-repo + multi-language sample code)
- Add vitest + coverage-v8 to devDependencies
- Add test scripts (test, test:integration, test:all, test:watch, test:coverage)
- Move typescript to devDependencies where it belongs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 20:07:02 +05:30
abhigyanpatwariandClaude Opus 4.6 c129e71ee7 ci: harden publish pipeline with CI gate, version check, and provenance
- Add workflow_call trigger to ci.yml so publish can reuse it as a gate
- Replace minimal publish.yml with hardened pipeline:
  - Full CI must pass before publish (typecheck + tests + cross-platform)
  - Verify git tag matches package.json version
  - Explicit build step + dry-run before real publish
  - npm provenance attestation enabled
  - Auto-create GitHub Release with generated notes

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-03-01 20:02:10 +05:30
Abhigyan Patwari 80eff73459 Add notice about GitNexus cryptocurrency claims
Added important notice regarding cryptocurrency affiliations.
2026-03-01 11:29:00 +05:30
Gary Magyar 48c8e6fe57 chore: merge upstream main (swift optional deps) and reorder kotlin entries
Merge origin/main which moves tree-sitter-swift to optionalDependencies
with conditional imports. Reorder Kotlin entries before C/C++/PHP in all
files so they don't sit adjacent to Swift entries, preventing future
merge conflicts when upstream modifies Swift support.
2026-02-28 12:55:55 +00:00
abhigyanpatwariandClaude Opus 4.6 2eca3e0da3 fix(swift): move tree-sitter-swift to optionalDependencies and use conditional imports
The PR merge reverted the Swift install fix. tree-sitter-swift must be
in optionalDependencies with conditional createRequire imports, otherwise
npm install fails on systems where the native build can't succeed.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 18:14:53 +05:30
Gary Magyar e046bf734d chore: merge upstream main into feat/kotlin-language-support
Resolve conflicts between Kotlin and Swift language support additions.
Both languages are now fully supported side by side.
2026-02-28 12:30:53 +00:00
Bhaskar Lalwani b30248f969 Updated README with Discord and badge updates
Added Discord link and updated badges for npm and license.
2026-02-28 17:44:00 +05:30
abhigyanpatwariandClaude Opus 4.6 eb48c7352e fix: read CLI version from package.json instead of hardcoding
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 16:32:28 +05:30
abhigyanpatwariandClaude Opus 4.6 29db66c304 chore: bump version to 1.3.5
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 16:07:34 +05:30
Abhigyan Patwari b7c582de76 Merge pull request #94 from jandyx/feat/swift-language-support
feat(swift): full Swift / iOS language support with SPM import resolution
2026-02-28 16:03:27 +05:30
Gary Magyar 508402fd4a feat: add full Kotlin language support 2026-02-28 10:06:52 +00:00
Gary Magyar da63281a5a Merge remote-tracking branch 'origin/main' into feat/kotlin-language-support
# Conflicts:
#	gitnexus/src/core/ingestion/parsing-processor.ts
#	gitnexus/src/core/ingestion/workers/parse-worker.ts
2026-02-28 08:57:40 +00:00
abhigyanpatwariandClaude Opus 4.6 c758f4eaf0 chore: bump version to 1.3.4
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-28 07:56:08 +05:30
Abhigyan Patwari fd507a19ae Merge pull request #105 from christopheralex-cc/fix/web-ui-server-connect-path-mismatch
fix(web): map API path field to repoPath in fetchRepoInfo
2026-02-28 07:53:01 +05:30
Abhigyan Patwari 799de20172 Merge pull request #102 from PurpleNewNew/feat/ast-decorator-detection
feat(ingestion): add AST decorator-based entrypoint hints
2026-02-28 07:41:57 +05:30
Abhigyan Patwari 2be88ae1f8 Merge pull request #61 from strazzere/fix/refactor_shell_commands
fix: ensure exec usage does not allow poisoning
2026-02-28 07:13:25 +05:30
Dmytro 2915e60630 fix(cli): run codex mcp add via shell on Windows 2026-02-27 14:01:10 +01:00
Abhigyan Patwari 019ed3ff85 Merge pull request #99 from abhigyanpatwari/fix/lazy-embed-import
fix: lazy-import embeddings to avoid onnxruntime crash on Node v24+
2026-02-27 18:14:59 +05:30
abhigyanpatwariandClaude Opus 4.6 6b4f10cae1 fix: remove unconditional embedder import from disconnect() to prevent crash on Node v24+
The disconnect() method was unconditionally importing embedder.js on
every graceful shutdown, which loads @huggingface/transformers and
onnxruntime-node — triggering the exact crash this branch fixes.
Since process.exit(0) follows immediately, the OS reclaims all
resources without needing disposeEmbedder(). Matches the pattern
already established in analyze.ts (lines 318-320).

Fixes #89

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 17:41:06 +05:30
Gary Magyar 43f525d056 fix(kotlin): guard against double-appending .* to wildcard import paths
Add endsWith('.*') check before appending wildcard suffix to prevent
possible double-append if grammar returns identifier text that already
includes the wildcard.
2026-02-27 10:28:39 +00:00
Gary Magyar e2a8bfa5ab fix(kotlin): enable import dependency tree resolution for Kotlin files
Add .kt/.kts to EXTENSIONS array, parameterize Java resolvers into JVM
resolvers (resolveJvmWildcard, resolveJvmMemberImport), and unify
Java+Kotlin dispatch in both import processing paths. Detect wildcard
imports via AST child node inspection in parse worker.

Validated against okhttp repo: 524 .kt files detected, imports resolve
correctly to .kt files (e.g. okhttp3.OkHttpClient -> OkHttpClient.kt).
2026-02-27 10:26:23 +00:00
Gary Magyar ee6753bf05 fix(kotlin): capture constructor-based heritage (class Foo : Bar())
The heritage query only matched bare user_type delegation specifiers
(interface implementation), missing constructor_invocation patterns
used for class extension. Adds a second heritage pattern for
constructor invocations, capturing ~3x more heritage edges.
2026-02-27 09:28:58 +00:00
Gary Magyar 1b8c3c77af feat(kotlin): distinguish interfaces from classes in knowledge graph
tree-sitter-kotlin (fwcd) has no interface_declaration node — both
interfaces and classes are class_declaration nodes. Use anonymous
keyword literal matching ("interface" vs "class") to produce the
correct @definition.interface / @definition.class captures.

Verified against two real Kotlin repos: a small one (3 Interface,
92 Class) and a large one (35 Interface, 677 Class, 5998 Function).
2026-02-27 09:09:26 +00:00
Güneş Bizim a7fc9d2f88 feat(laravel): add route detection and route group support
Parse Route::* calls from PHP AST using a procedural walk that tracks
group nesting state (middleware, prefix, controller cascade). Creates
CALLS edges from route files to controller methods.

Supported patterns:
- Route::get/post/put/patch/delete/any/match with [Controller::class, 'method']
- Route::resource / Route::apiResource (expanded to individual actions)
- Invokable controllers (Controller::class -> __invoke)
- String syntax ('Controller@method')
- Route::middleware()->group() fluent chains (arbitrary depth)
- Route::prefix()->name()->group() chains
- Route::controller(X::class)->group() shared controller
- Route::group(['middleware' => ...], fn) array API
- Nested groups with full middleware/prefix cascade

Implementation notes:
- Uses anonymous_function node type (php_only grammar, not anonymous_function_creation_expression)
- File paths from workers are relative; check startsWith('routes/') not includes('/routes/')
- Edges: File -> Method with reason='laravel-route', confidence 0.9 (import-resolved)
- Mirrored to gitnexus-web inline in processCalls (no worker layer)
2026-02-27 11:17:36 +03:00
christopher 0074fd71ff fix(web): map API path field to repoPath in fetchRepoInfo
The backend `/api/repo` endpoint returns `path` but `ServerRepoInfo`
expects `repoPath`, causing `undefined.split('/')` crash in App.tsx
when connecting to a local gitnexus serve instance.

Fixes #92
2026-02-27 16:05:47 +08:00
PurpleNewNew de935a4f4c feat(ingestion): add AST decorator-based entrypoint hints 2026-02-27 15:40:42 +08:00
abhigyanpatwariandClaude Opus 4.6 989673a624 fix: lazy-import embeddings to avoid onnxruntime crash on unsupported Node versions
Convert static imports of @huggingface/transformers (which triggers
onnxruntime-node native binary loading) to dynamic import() calls.
This prevents crashes on Node versions whose ABI isn't supported by
the prebuilt onnxruntime binaries (e.g. Node v24).

Affected entry points:
- cli/analyze.ts: embedding pipeline only loaded when --embeddings is passed
- mcp/local/local-backend.ts: embedder only loaded on first semantic search
- server/api.ts: embedder only loaded when search endpoint needs embeddings

Fixes #89

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 12:09:17 +05:30
Abhigyan Patwari 8c41970631 Merge pull request #96 from abhigyanpatwari/fix/mcp-no-repos-crash
fix(mcp): don't crash server when no repos are indexed
2026-02-27 11:35:31 +05:30
abhigyanpatwariandClaude Opus 4.6 5c3a32d0c6 fix(kuzu): remove duplicate ftsLoaded declaration that broke typecheck
The module-level `let ftsLoaded` was declared twice (line 19 and 679),
causing TS2451. Removed the duplicate and cleaned up redundant
assignments in loadFTSExtension.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 11:31:07 +05:30
abhigyanpatwariandClaude Opus 4.6 a8b3c6b23f fix(mcp): don't crash server when no repos are indexed (#91)
The MCP server called process.exit(1) at startup when no repositories
were found in the registry. This prevented users from configuring the
MCP integration before running `gitnexus analyze`.

The server now starts gracefully with 0 repos and discovers newly
indexed repos lazily via refreshRepos() on each tool call.

Closes #91

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 10:36:35 +05:30
jandyx 15caf1e014 fix(swift): add missing Enum→Enum CodeRelation pair to schema 2026-02-27 12:00:41 +08:00
jandyx 1ed34a0007 docs: update supported languages list to include PHP and Swift 2026-02-27 11:45:05 +08:00
jandyx 7a4bc9a260 docs: improve postinstall script comments with background and TODO 2026-02-27 11:39:14 +08:00
jandyx 3872a73875 feat(swift): add prebuilt Swift WASM binary for web package
Sourced from tree-sitter-wasms@0.1.13 prebuilt collection.
2026-02-27 11:22:17 +08:00
jandyx f557716998 fix(swift): improve postinstall to auto-rebuild after patching binding.gyp
The script now detects missing native binding and runs node-gyp rebuild
after patching. This handles the case where tree-sitter-swift's own
postinstall fails during npm install — our postinstall picks up,
patches binding.gyp, and rebuilds successfully.
2026-02-27 11:14:45 +08:00
jandyx ae8a76511d fix(swift): address review gaps — schema, call-processor, web package, postinstall
- Add init_declaration/deinit_declaration to call-processor FUNCTION_NODE_TYPES
  and findEnclosingFunction (syncs with parse-worker, avoids Dart PR #83 rejection)
- Add 7 missing CodeRelation FROM-TO pairs in schema.ts to eliminate analyze warnings
  (Function→Property, Constructor→Property/Typedef, Enum→Class/Interface,
   Struct→Interface, TypeAlias→Class)
- Mirror all Swift support to gitnexus-web: supported-languages, utils, queries,
  framework-detection, entry-point-scoring, parser-loader WASM path
- Add postinstall script to patch tree-sitter-swift binding.gyp actions array
2026-02-27 09:47:05 +08:00
jandyx e803e7e9d6 feat(swift): add comprehensive Swift/iOS language support
- Enable Swift in supported languages enum
- Add tree-sitter-swift parser loading (v0.6.0)
- Add .swift file extension mapping
- Implement full tree-sitter queries (class, struct, enum, protocol,
  extension, actor, function, property, init, imports, calls, heritage)
- Add Swift export detection (public/open modifiers)
- Add Swift/iOS built-in name filtering (~70 entries: stdlib, UIKit,
  Foundation, GCD, Combine, collection methods)
- Add SPM module import resolution (Sources/<Target>/ scanning)
- Add iOS/SwiftUI framework path detection with entry point multipliers
- Add Swift entry point scoring patterns (UIKit lifecycle, SwiftUI body,
  Coordinator, AppDelegate/SceneDelegate)
- Add Swift test file detection patterns
2026-02-27 08:40:47 +08:00
abhigyanpatwariandClaude Opus 4.6 50fc8df2a1 fix(web): replace stale isBackendMode ref with serverBaseUrl
PR 66 refactored isBackendMode to serverBaseUrl in useAppState but
missed updating EmbeddingStatus.tsx, causing TypeScript build failure
on Vercel.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-27 05:08:45 +05:30
Gary Magyar c37b63ae8b feat: add Kotlin language support
Add end-to-end Kotlin parsing, symbol extraction, and visibility detection.
Extract findSiblingChild helper into utils.ts for clean AST traversal of
Kotlin's modifiers/visibility_modifier sibling pattern. Fix pre-existing
duplicate ftsLoaded declaration in kuzu-adapter.ts.

Files changed:
- supported-languages.ts: add Kotlin enum member
- parser-loader.ts, parse-worker.ts: register tree-sitter-kotlin
- tree-sitter-queries.ts: add Kotlin queries for classes, interfaces,
  objects, functions, properties, imports, calls, and heritage
- parsing-processor.ts, parse-worker.ts: add Kotlin visibility detection
- call-processor.ts, parse-worker.ts: add Kotlin builtins and node types
- utils.ts: add .kt/.kts extension mapping and findSiblingChild helper
- package.json: add tree-sitter-kotlin dependency
2026-02-26 17:01:35 +00:00
abhigyanpatwariandClaude Opus 4.6 7fe8830402 merge: resolve conflicts with main for remote server mode (PR #66)
Keep main's barLog implementation, preserve both currentDbPath and
ftsLoaded reset in closeKuzu, take PR's new resolveRepo pattern
for /api/query. Path traversal guard confirmed intact.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 18:40:25 +05:30
abhigyanpatwariandClaude Opus 4.6 39b01f101e feat(skills): rewrite skill descriptions for better auto-invocation
Skill descriptions were too tool-centric ("using knowledge graph", "blast
radius") which prevented Claude Code from matching them to user intent.
Rewritten to user-intent-driven format with "Use when..." phrasing and
example trigger phrases so Claude can semantically match user requests.

Updated across all 3 sources: gitnexus/skills/, gitnexus-claude-plugin/skills/,
.claude/skills/, and the ai-context.ts fallback generator.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 18:24:41 +05:30
Abhigyan Patwari 7cb88707a4 Merge pull request #58 from BlockSecCA/pr/cuda-fallback
Probe for CUDA before attempting GPU embeddings
2026-02-26 18:22:37 +05:30
abhigyanpatwariandClaude Opus 4.6 2a444acf1d fix(plugin): hook logic bug and version mismatch
- Fix hook exit status check: || → && so failed augment errors
  don't get injected into Claude's context as graph results
- Add exit status check to npx fallback (same bug)
- Update plugin version from 1.2.11 to 1.3.3 in both
  marketplace.json and plugin.json

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 18:08:32 +05:30
abhigyanpatwari d97d43f1b8 Merge branch 'pr-64' 2026-02-26 17:38:05 +05:30
Nico Prieto 04be81f655 fix(server): harden multi-repo API and MCP safety 2026-02-26 13:01:30 +01:00
Nico PrietoandClaude Opus 4.6 f047a84d82 fix(server): restore security guards and error handling
Address code review feedback on the server-mode PR:

Critical fixes:
- Restore CORS whitelist (localhost + gitnexus.vercel.app only)
- Bind to 127.0.0.1 by default; add --host CLI flag for opt-in remote access
- Restore path traversal guard on /api/file (resolve + startsWith check)
- Restore try/catch on all route handlers + global error middleware
- Restore SIGINT/SIGTERM graceful shutdown handlers

Bug fixes:
- Add mutex to core initKuzu to prevent race conditions on concurrent
  DB switches (two requests for different repos no longer corrupt state)
- Track ftsLoaded flag and reset on DB switch / close so FTS extension
  is reloaded for each new database connection
- Restore input validation on /api/query (cypher required) and
  /api/search (query required)

Improvements:
- Add TTL-based cleanup for orphaned MCP sessions (30min idle eviction)
  to prevent memory leaks from network drops that skip onclose

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 12:42:00 +01:00
abhigyanpatwariandClaude Opus 4.6 0421fcbc76 merge: resolve conflicts with main for PHP/Laravel support
Merge main into feat/php-laravel-support, resolving conflicts in:
- csv-generator.ts: add description column to streaming CSV architecture
- kuzu-adapter.ts: add description to COPY queries and insert/merge ops
- schema.ts: add description STRING to all code element tables, FROM Method TO Property
- parse-worker.ts: integrate PHP built-ins and Eloquent extraction with sub-batch worker
- import-processor.ts: integrate PHP PSR-4 resolution with ImportResolutionContext
- package-lock.json: regenerate from main's 1.3.3 base with tree-sitter-php

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 16:39:17 +05:30
Abhigyan Patwari 1403cdbf6d Merge pull request #75 from CrazyBunQnQ/main
feat(ui): Add a copy button to the Nexus AI and copy the md result
2026-02-26 13:35:44 +05:30
abhigyanpatwariandClaude Opus 4.6 0e8eed4a8a merge: resolve conflicts with main in CLAUDE.md and AGENTS.md
Keep main's stats line (GitnexusV2, 1348 symbols) with PR's skill path renames.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 13:10:27 +05:30
CrazyBunQnQ d6738c51c1 feat(ui): Add a copy button to the Nexus AI and copy the markdown results after clicking it 2026-02-26 15:14:08 +08:00
abhigyanpatwariandClaude Opus 4.6 a5096e8029 revert: restore 512KB file size limit, keep improved skip message
2MB limit caused FTS crash on large codebases. 512KB is safe and only
skips generated/vendored files.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 11:20:15 +05:30
abhigyanpatwariandClaude Opus 4.6 36e64e892f fix(ux): raise file size limit to 2MB, quiet warning output
- Raise MAX_FILE_SIZE from 512KB to 2MB to capture more real source files
- Replace verbose per-warning output with single summary line
- Soften skip message wording ("likely generated/vendored")

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 10:46:10 +05:30
abhigyanpatwariandClaude Opus 4.6 bb6c22a22c fix(schema): add 6 missing FROM/TO pairs for LLVM-style codebases
Class→Namespace, Class→Typedef, Struct→Struct, Struct→Class,
Struct→Enum, Namespace→Struct

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 10:43:55 +05:30
abhigyanpatwariandClaude Opus 4.6 420122065a chore: bump version to 1.3.0
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 10:28:38 +05:30
abhigyanpatwariandClaude Opus 4.6 bef319491a feat(gitnexus): large-repo optimizations, multi-language support, bug fixes
Optimize ingestion pipeline for massive codebases (Linux kernel ~75K files):
- Worker pool: 8 threads, 1500 sub-batch, 30s per-batch timeout
- Streaming CSV generation with per-stream MaxListeners fix
- FileContentCache 3000 entries, resolveCache LRU eviction (20% at 100K cap)
- Leiden 60s timeout with single-community fallback
- SIGINT graceful shutdown, FTS dedup flag

Multi-language node support (Struct, Enum, Macro, Impl, Trait, etc.):
- Add 16 multi-language types to NodeLabel union
- Separate CSV writers with correct schema (no isExported)
- KuzuDB schema: backtick-escape reserved words (Macro, Union, Enum)
- Add all FROM/TO pairs for multi-language relationship edges

Fix bugs: getKuzuStats/deleteNodesForFile backtick escaping,
insertNodeToKuzu/batchInsertNodesToKuzu missing escaping + isExported,
progress bar flickering and timer display

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-26 10:23:23 +05:30
Dmytro 984316d260 feat: add Codex MCP and skills support 2026-02-25 15:43:12 +01:00
Linus Beckhaus 238abbd947 refactor(skills): prefix all skill names with gitnexus- for disambiguation
Skill folder names determine invocation paths in Claude Code plugins
(e.g. plugin:gitnexus:gitnexus-cli). Generic names like "cli" or
"debugging" could collide with other plugins, so prefix them all with
gitnexus- for clarity.

Updated across plugin dirs, main package source files, ai-context.ts
generator, setup.ts installer, and all CLAUDE.md/AGENTS.md routing tables.
2026-02-25 14:15:42 +01:00
Linus Beckhaus 3f4c4cb4aa remove local claude settings from git 2026-02-25 14:11:53 +01:00
Linus Beckhaus 4f4fe9e587 update version 2026-02-25 13:40:49 +01:00
Linus Beckhaus dbf3495713 fix(skills): remove gitnexus- prefix from skill frontmatter names
Skill names should match folder names since the plugin namespace
(gitnexus:) already provides context. Avoids redundant display like
gitnexus:gitnexus-cli → now gitnexus:cli.
2026-02-25 13:39:25 +01:00
Linus Beckhaus 5b8ce44537 format: format skills 2026-02-25 13:36:14 +01:00
Linus Beckhaus ffc4b69004 feat(plugin): add marketplace, fix manifest, and add CLI commands skill
Add .claude-plugin/marketplace.json at repo root so users can permanently
install via `/plugin marketplace add nicosxt/gitnexus`. Remove invalid
hooks/mcpServers fields from plugin.json (auto-discovered at default
locations). Add cli skill covering all agent-relevant CLI commands
(analyze, status, clean, wiki, list) with correct flags. Update guide
skill and CLAUDE.md generator routing tables.
2026-02-25 13:30:47 +01:00
abhigyanpatwariandClaude Opus 4.6 470a3377b3 fix(eval): import paths, patch extraction, model configs
- Fix imports from eval.agents/eval.environments to relative (agents/environments)
- Add hatch wheel config for correct package discovery
- Extract git diff patch from container for SWE-bench submission
- Use sys.executable instead of hardcoded "python" for venv compat
- Upgrade claude-haiku config to 4.5
- Add minimax-m2.1 model config

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 17:11:00 +05:30
abhigyanpatwariandClaude Opus 4.6 91289404c2 feat(gitnexus): v1.2.9 — impact enrichment, cypher markdown, Windows setup fix
- Impact tool now returns risk score, affected processes/modules, and summary
- Cypher tool formats results as markdown tables for LLM readability
- Context tool includes module (functional area) field
- Semantic search skips model init when embeddings are disabled
- Setup: wrap npx in cmd /c on Windows for .cmd script compatibility
- Embedder: silence stderr during ONNX model load to protect MCP stdio
- API: use executeCypher directly to avoid double formatting
- Add community integrations section to READMEs

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 17:10:51 +05:30
Linus Beckhaus 397dad8ec4 feat(plugin): transform Claude Code plugin into self-contained installable package
Bundle MCP server config (.mcp.json), fix hook to use spawnSync with npx
fallback and read stderr (KuzuDB stdout workaround), wire plugin.json with
hooks/mcpServers paths, add guide skill with tools/resources/schema reference,
add AMP-compatible mcp.json to all skill dirs, and slim CLAUDE.md generator
by moving reference content into the guide skill.
2026-02-25 11:47:16 +01:00
Nico PrietoandClaude Opus 4.6 7ee2dd1087 feat: add remote server connection mode and multi-repo switching
Replace the old local-only backend mode with a new server connection
flow that lets the web UI connect to any running GitNexus server,
download the pre-built knowledge graph, and explore it without WASM.

- Add server-connection service with streaming download and progress
- Replace DropZone backend tab with server connect UI (URL input, progress bar, cancel)
- Add repo switcher dropdown in Header when multiple repos are indexed
- Mount MCP server over StreamableHTTP at /api/mcp for remote AI tool access
- Refactor api.ts to query KuzuDB directly instead of routing through LocalBackend
- Fix KuzuDB adapter to support switching between databases (close old before opening new)
- Extract createMCPServer() from startMCPServer() for transport-agnostic reuse
- Support ?server= query param for bookmarkable auto-connect

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-25 11:38:28 +01:00
Güneş BizimandClaude Sonnet 4.6 6aab580f93 fix(schema): add missing CodeRelation pairs for Class->Trait and Method->Property
Laravel projects commonly use traits (HasFactory, SoftDeletes, etc.) on classes
and methods that reference model properties. These relation pairs were missing
from the CodeRelation table, causing edges to be dropped during indexing.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 11:33:54 +03:00
Güneş BizimandClaude Sonnet 4.6 1c02a06d1b feat(graph): persist multi-language nodes to KuzuDB and add description field
- Add description STRING column to all code element schemas (Function, Class,
  Interface, Method, CodeElement, and all CODE_ELEMENT_BASE types including Property)
- Add generateMultiLangNodeCSV() to write Struct, Enum, Trait, Property, etc.
  nodes to KuzuDB (previously they existed in-memory only, never reached the DB)
- Update generateAllCSVs() to generate CSVs for all 19 multi-language types
- Fix getCopyQuery() to use correct column list per table type:
  - Multi-language tables: id,name,filePath,startLine,endLine,content,description
  - Core tables: id,name,filePath,startLine,endLine,isExported,content,description
- Update batchInsertNodesToKuzu and insertNodeToKuzu to include description field

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 10:31:53 +03:00
Güneş BizimandClaude Sonnet 4.6 890fedaa09 feat(php): track Eloquent model properties and relationships with structured metadata
- Add PHP property_declaration query to capture class properties ($fillable, $casts, etc.)
- Extract array values from Eloquent model properties as description field:
  - $fillable → comma-separated field names (e.g. "name, email, password")
  - $casts → key:type pairs (e.g. "email_verified_at:datetime, active:boolean")
  - $hidden, $guarded, $with, $appends similarly
- Detect Eloquent relationship methods (hasMany, hasOne, belongsTo, etc.) and
  annotate them with relation type and target model (e.g. "hasMany(Post)")

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 10:31:35 +03:00
Güneş BizimandClaude Sonnet 4.6 8a79465cbf feat(laravel): add Service/Repository pattern support
- Add /services/ (1.8x) and /repositories/ (1.5x) path multipliers
- Add Service$, Repository$, find, findAll, save, delete to PHP entry point patterns

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:28:42 +03:00
Güneş BizimandClaude Sonnet 4.6 945235ce56 fix(php): correct PHP query node type names for php_only grammar
- Replace static_call_expression -> scoped_call_expression (php_only name)
- Replace class_implements -> class_interface_clause
- Use [(name) (qualified_name)] alternatives in heritage patterns to match
  both simple names (Controller) and fully-qualified names (App\Models\User)

All patterns verified against tree-sitter-php node-types.json and live parse.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:13:26 +03:00
Güneş BizimandClaude Sonnet 4.6 e67d6c63d3 feat(php): add PHP WASM support for web UI
- Copy tree-sitter-php_only.wasm to public/wasm/php/tree-sitter-php.wasm
- Register PHP WASM path in web parser-loader languageFileMap

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:05:02 +03:00
Güneş BizimandClaude Sonnet 4.6 58063ca9ed feat(laravel): add Laravel framework path detection and entry point scoring
- Add Laravel route/controller/job/listener/middleware/provider/policy/model
  path patterns with appropriate multipliers (1.5–3.0)
- Add 'laravel' entry to FRAMEWORK_AST_PATTERNS for documentation
- Add PHP ENTRY_POINT_PATTERNS: RESTful methods, __invoke, handle, boot, etc.
- Add PHP/Laravel test file detection (Test.php, Spec.php, tests/feature/, tests/unit/)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:04:49 +03:00
Güneş BizimandClaude Sonnet 4.6 804d975cd0 feat(php): add PHP/PSR-4 import resolution with composer.json support
- Add .php, .phtml to EXTENSIONS array
- Add ComposerConfig interface and loadComposerConfig() (mirrors TsconfigPaths pattern)
- Add resolvePhpImport() with PSR-4 longest-match resolution + suffix fallback
- Wire PHP handling into processImports() and processImportsFromExtracted()

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:04:39 +03:00
Güneş BizimandClaude Sonnet 4.6 2164dc22f1 feat(php): add PHP tree-sitter queries for all PHP constructs
Add PHP_QUERIES covering: namespaces, classes, interfaces, traits,
enums (PHP 8.1), functions, methods, use-statement imports, function/
method/static/nullsafe calls, and heritage (extends, implements, trait use).
Trait query captures enclosing class name to satisfy parse-worker heritage logic.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:04:32 +03:00
Güneş BizimandClaude Sonnet 4.6 30aba01188 feat(php): register PHP tree-sitter grammar in parser loader and parse worker
- Import PHP from tree-sitter-php and use PHP.php_only in languageMap
- Add anonymous_function_creation_expression to FUNCTION_NODE_TYPES
- Add PHP export detection (public modifier / top-level = exported)
- Add PHP built-in functions to BUILT_INS to reduce call graph noise

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:04:06 +03:00
Güneş BizimandClaude Sonnet 4.6 92a5d026c8 feat(php): install tree-sitter-php dependency
Add tree-sitter-php ^0.23.0 to gitnexus/package.json. Package exports
{ php, php_only } grammars; we use php_only for pure PHP files.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:03:57 +03:00
Güneş BizimandClaude Sonnet 4.6 7883bf2cf0 feat(php): add PHP to SupportedLanguages enum and file extension detection
Uncomment PHP = 'php' in both CLI and web SupportedLanguages enums.
Add .php, .phtml, .php3/.4/.5/.8 extension detection to getLanguageFromFilename()
in both packages.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-02-25 01:03:44 +03:00
Tim Strazzere 73590b2862 fix: ensure exec usage does not allow poisoning
Previous usage was vulnerable to "poisoned" tags
which could enduce commands to be run when a
`detectChanges` command was hit. This was primarily
fixed in `local-backend.ts` however I changes the
`execSync` usages where any injection was potentially
able to be performed (e.g. staleness).

Skipped touching wiki and generator as those use static
input, though these should potentially be changed over
in the future.
2026-02-24 13:42:29 -08:00
CarlosandClaude bdda9afdca Rework CUDA probe: use ldconfig + env vars instead of nvidia-smi
Addresses review feedback:
- Remove nvidia-smi early return (driver ≠ runtime libs)
- Use ldconfig -p as primary check (covers all architectures and paths)
- Fall back to CUDA_PATH / LD_LIBRARY_PATH for conda, /opt/cuda, etc.
- Switch from execSync to execFileSync (avoids spawning a shell)

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-24 10:16:39 -05:00
abhigyanpatwariandClaude Opus 4.6 6be54ce9d3 fix(analyze): replace nonexistent bar.log with origLog
SingleBar from cli-progress doesn't have a .log() method (that's
MultiBar-only), causing a crash when any code calls console.log
during the pipeline.

Fixes #53
Thanks to @TranDatk for reporting in #56.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 11:05:32 +05:30
abhigyanpatwariandClaude Opus 4.6 2e390583fc fix(ci): remove tracked worktrees and gitignore them
Worktree directories were committed as gitlink submodule references,
causing CI failures during checkout. Remove them from tracking and
add .claude/worktrees/ to .gitignore to prevent recurrence.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-24 09:42:53 +05:30
CarlosandClaude 575a4978f2 Probe for CUDA before attempting GPU embeddings
ONNX Runtime crashes with an uncatchable native error when CUDA
libraries are missing. This adds a lightweight probe (nvidia-smi +
libcublasLt.so.12 path check) before selecting the CUDA device,
falling back to CPU gracefully on systems without NVIDIA GPUs.

Fixes #53

Co-Authored-By: Claude <noreply@anthropic.com>
2026-02-23 22:50:46 -05:00
Abhigyan Patwari c379c39ae1 Merge pull request #49 from paulrobello/feat/local-backend-mode
feat: local backend mode for web UI
2026-02-23 21:06:24 +05:30
Paul Robello acd918f44e refactor(api): use executeCypher directly in buildGraph
Add public executeCypher(repoName, query) method to LocalBackend,
bypassing the callTool string dispatch and resolving the repo handle
directly. Update buildGraph in api.ts to call it instead of going
through callTool('cypher', ...) on every query.
2026-02-23 07:23:46 -08:00
Paul Robello a71924f774 fix: address PR review — CORS, search quality, and embedding flags
- Add https://gitnexus.vercel.app to CORS allowed origins so the
  deployed site can connect to the local backend server.
- Replace naive createHttpTextSearch (substring matching via Cypher)
  with createHttpHybridSearch that calls /api/search for full
  BM25 + semantic + RRF hybrid search on the server.
- Set isEmbeddingReady to false in backend mode (no local embedder),
  which also fixes {{QUERY_VECTOR}} handling — the existing error
  path in tools.ts now correctly returns a helpful message.
- Set isBM25Ready to true (available via server's hybrid search).
- Removes the Cypher injection bug (wrong escape char) since the
  function that contained it was replaced entirely.
2026-02-22 17:14:56 -08:00
Paul Robello 8dd1c19bec feat(web): wire agent/chat through HTTP in backend mode
Add HTTP wrapper functions (executeQuery, textSearch) to the ingestion
worker and a new initializeBackendAgent method that creates the agent
with HTTP-backed tool closures instead of local KuzuDB. useAppState
detects backend mode and routes agent initialization accordingly.
2026-02-22 17:14:18 -08:00
Paul Robello 56f92ca1ad fix(web): skip embeddings pipeline in backend mode
The WASM worker DB is never initialized in backend mode, so
startEmbeddings() now returns early. Also hides the embedding
status button in the header when using the local backend.
2026-02-22 17:14:18 -08:00
Paul Robello 36c7b3ed12 fix: address final code review issues
- Fix field name mismatch in /api/repo (repoPath -> path)
- Add setProgress(null) on successful backend repo load
- Prevent DropZone tab auto-switch from re-triggering
- Make isDatabaseReady() return true in backend mode
- Increase fetchGraph timeout to 60s for large repos
- Replace dynamic import with static import for runCypherQuery
2026-02-22 17:14:18 -08:00
Paul Robello c80bcccba4 chore: add docs/plans/ to .gitignore 2026-02-22 17:14:18 -08:00
Paul Robello 76d1538c5e docs: update READMEs for local backend mode 2026-02-22 17:14:18 -08:00
Paul Robello d1cac0515d fix(web): wire backend props to SettingsPanel 2026-02-22 17:14:18 -08:00
Paul Robello cb70fc8d6c feat(web): add backend URL setting to settings panel 2026-02-22 17:14:18 -08:00
Paul Robello 3603178266 feat(web): add backend connection indicator to header 2026-02-22 17:14:18 -08:00
Paul Robello c7519c8493 feat(web): integrate backend mode into app state and loading flow 2026-02-22 17:14:18 -08:00
Paul Robello 6726340059 feat(web): add Local Server tab to DropZone onboarding 2026-02-22 17:14:18 -08:00
Paul Robello e1d3959273 feat(web): add BackendRepoSelector component 2026-02-22 17:14:18 -08:00
Paul Robello 6de13ac800 feat(web): add useBackend hook for local server connection 2026-02-22 17:14:18 -08:00
Paul Robello d7380de683 feat(web): add backend HTTP service module 2026-02-22 17:14:18 -08:00
Paul Robello 102850455d fix(serve): address code review - security, error handling, CORS 2026-02-22 17:14:18 -08:00
Paul Robello 8e8bb90fe4 refactor(serve): rewrite API server to use LocalBackend for multi-repo support
Replace direct single-repo KuzuDB access with LocalBackend, the same
backend the MCP server uses. The serve command now works with ALL
indexed repos via the global registry instead of only the CWD repo.

New/updated endpoints:
- GET /api/repos (list all indexed repos)
- GET /api/repo?name=X (repo metadata)
- GET /api/graph?repo=X (full knowledge graph)
- POST /api/query (raw Cypher via backend)
- POST /api/search (process-grouped search via backend)
- GET /api/file?repo=X&path=Y (file read with path traversal protection)
- GET /api/processes?repo=X, /api/process?repo=X&name=Y
- GET /api/clusters?repo=X, /api/cluster?repo=X&name=Y
2026-02-22 17:14:18 -08:00
abhigyanpatwari 28339c995d chore: bump gitnexus to v1.2.8 2026-02-23 04:12:02 +05:30
Abhigyan Patwari e849f017f2 Merge pull request #51 from abhigyanpatwari/fix/disable-embeddings-by-default
fix: disable embeddings by default, fix segfault on macOS/Linux
2026-02-23 04:05:47 +05:30
abhigyanpatwari 5c7d905150 ci: add GitHub Actions for PR type-check and npm publish on tags 2026-02-23 03:59:35 +05:30
abhigyanpatwari 302c7ba7f0 fix: disable embeddings by default, fix segfault on macOS/Linux
Embeddings (ONNX Runtime) caused segfaults on macOS (#38) and CUDA
install failures on Linux (#40). Since embeddings only power semantic
vector search in query() and BM25 + graph traversal already provide
strong retrieval, disable embeddings by default with minimal quality
impact.

Changes:
- Replace --skip-embeddings with --embeddings opt-in flag
- Remove disposeEmbedder() call (ONNX native cleanup segfaults)
- Add process.exit(0) when embeddings are used to bypass ONNX atexit hooks
- Suppress ONNX session warnings via logSeverityLevel
- Fix progress bar duplication by routing console output through bar.log()
- Add elapsed time indicator for long-running phases

Fixes #38
Fixes #40
2026-02-23 03:47:34 +05:30
Abhigyan Patwari 93fc2b67ca Add Trendshift badge to README
Added a badge for GitNexus repository on Trendshift.
2026-02-23 02:38:18 +05:30
Abhigyan Patwari ad92b82723 Update README.md 2026-02-23 02:09:03 +05:30
Abhigyan Patwari 3400fccf8c Merge pull request #48 from abhigyanpatwari/claude/issue-47-20260222-1745
fix: use cross-platform npx command in .mcp.json
2026-02-22 23:21:48 +05:30
claude[bot]andAbhigyan Patwari af1e455e77 fix: use cross-platform npx command in .mcp.json
Replace Windows-specific 'cmd /c npx' invocation with portable 'npx'
command that works on macOS, Linux, and Windows.

Fixes #47

Co-authored-by: Abhigyan Patwari <abhigyanpatwari@users.noreply.github.com>
2026-02-22 17:46:04 +00:00
Abhigyan Patwari 053af03caa Merge pull request #46 from abhigyanpatwari/add-claude-github-actions-1771779387558
Add Claude Code GitHub Workflow
2026-02-22 22:27:11 +05:30
Abhigyan Patwari 63f1cd1ec9 "Claude Code Review workflow" 2026-02-22 22:26:31 +05:30
Abhigyan Patwari f42513f5b6 "Claude PR Assistant workflow" 2026-02-22 22:26:30 +05:30
abhigyanpatwariandClaude Opus 4.6 54ae315336 chore: bump gitnexus to v1.2.7
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 17:13:02 +05:30
abhigyanpatwari 19181e9ff6 Merge branch 'linux_stress_test' 2026-02-21 17:12:38 +05:30
abhigyanpatwariandClaude Opus 4.6 d1fb34d636 feat: MCP hot-reload — discover newly indexed repos without restart
listRepos() now re-reads ~/.gitnexus/registry.json on every call.
resolveRepo() retries with a registry refresh on cache miss.
Extracted refreshRepos() from init() for reuse. All callers updated
with await for the new async signatures.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-21 17:07:57 +05:30
abhigyanpatwari 8e2ee26834 docs: remove architecture docs and deep implementation details from markdown files 2026-02-21 13:00:09 +05:30
abhigyanpatwari a7c6903322 docs: remove deep architecture details from README 2026-02-21 04:33:23 +05:30
abhigyanpatwariandClaude Opus 4.6 33aa774f03 fix: worker pool robustness — timeout, exit handler, 512KB guards, error reporting
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-20 20:24:48 +05:30
Abhigyan Patwari 366bc8b14b Revise README with new links and enhanced details
Updated links and improved project description.
2026-02-20 05:02:50 +05:30
Abhigyan Patwari e79133daaa Merge pull request #36 from abhigyanpatwari/gitnexus_cli
Gitnexus cli
2026-02-17 16:32:17 +05:30
abhigyanpatwari 79501abb6c fixed MCP concurrency issue with connection pool on kuzu 2026-02-17 16:31:05 +05:30
abhigyanpatwari 6ce715b62c repowiki CLI command implemented 2026-02-17 02:25:07 +05:30
abhigyanpatwariandCursor 3425fdeffd simplify: remove incremental graph pipeline, keep only embedding cache
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 01:13:37 +05:30
abhigyanpatwariandCursor 49eaf576fd chore: revert test whitespace change in embedder.ts
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 01:05:27 +05:30
abhigyanpatwariandCursor abdb3b4e70 test: whitespace change for incremental query test
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 01:03:47 +05:30
abhigyanpatwariandCursor 2f7be29f59 perf: hybrid incremental - workerData fast path + cross-file edge preservation
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 01:01:14 +05:30
abhigyanpatwariandCursor c4c863887b fix: use canonical DEFINES rel IDs and remove dead code in loadExistingGraph
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 00:49:19 +05:30
abhigyanpatwariandCursor dcb7521386 fix: use full import/call/heritage resolution in incremental mode for cross-file correctness
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 00:47:05 +05:30
abhigyanpatwariandCursor b0320e6b02 fix: exclude CONTAINS rels from preloaded data to avoid duplicates
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 00:44:14 +05:30
abhigyanpatwariandCursor d9a139957a fix: include uncommitted working tree changes in incremental diff detection
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 00:40:03 +05:30
abhigyanpatwariandCursor 65e14605b4 test: add comment to git.ts for incremental test
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-02-17 00:32:34 +05:30
abhigyanpatwari 44572ad0bd vercel pointed to gitnexus-web for FE deployment 2026-02-14 13:40:00 +05:30
abhigyanpatwariandClaude Opus 4.6 58972af084 Fix Claude Code hooks: correct settings path, matcher format, and hook script
- Hook config goes in ~/.claude/settings.json (not hooks.json)
- Matcher uses string format ("Grep|Glob|Bash") per new Claude Code schema
- Rename gitnexus-hook.js → gitnexus-hook.cjs for CommonJS compatibility
- Fix setup.ts: correct hook filename and timeout (8000ms instead of 10ms)
- Bump to v1.1.9 and publish to npm

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
2026-02-14 13:05:36 +05:30
abhigyanpatwari e90622aa24 MCP tools fixed 2026-02-14 00:16:53 +05:30
abhigyanpatwari 2560a9532d logging global registry 2026-02-13 22:39:12 +05:30
abhigyanpatwari eca55aacd7 fixed resource count multiplying issue ( using resource templates now ) 2026-02-13 21:28:36 +05:30
abhigyanpatwari 96e1d799c8 agent md experiments 2026-02-07 23:39:16 +05:30
abhigyanpatwari dda8de41a3 MCP and cli fixes 2026-02-07 06:21:21 +05:30
abhigyanpatwari 1735c0ffcd gitnexus wal cleanup preventing reindexing issue fixed 2026-02-06 06:35:15 +05:30
abhigyanpatwari 6019e25126 Louvian to leindes, removed smar clustering in UI 2026-02-06 04:43:21 +05:30
abhigyanpatwari 86b171edac test: add FTS live test file 2026-02-05 07:24:53 +05:30
abhigyanpatwari 6c3c47edc3 resources implemented and agents.md and skills updated to use it 2026-02-05 05:13:48 +05:30
abhigyanpatwari d1e53d7030 implemented skills, CLI and MCP merged into /gitnexus 2026-02-04 04:55:09 +05:30
abhigyanpatwari 664aa820f5 fix: move typescript to dependencies, add prepare script for production builds 2026-02-04 03:52:49 +05:30
abhigyanpatwari e480888cf0 fix: clean command --force flag, hide debug logs in production 2026-02-04 03:06:19 +05:30
abhigyanpatwari 747cf003b8 feat: improved AI context - full inline content in AGENTS.md and CLAUDE.md with markers for updates 2026-02-04 02:50:51 +05:30
abhigyanpatwari 5830a50288 feat: analyze command auto-finds git root when run from subdirectory 2026-02-04 02:30:16 +05:30
abhigyanpatwari 1ed9d286ae chore: cleanup empty csv directory after kuzu load 2026-02-04 01:40:59 +05:30
abhigyanpatwari 778ff10707 feat: auto-generate AI context files after analyze (AGENTS.md, .cursorrules, .windsurfrules) 2026-02-04 01:29:09 +05:30
abhigyanpatwari b723ce70c3 chore: remove gitnexus-mcp (merged into gitnexus) 2026-02-04 01:16:15 +05:30
abhigyanpatwari c90576442e feat: merge gitnexus-mcp into gitnexus package - unified CLI+MCP 2026-02-04 01:12:41 +05:30
abhigyanpatwari 789e7809be docs: update license copyright holder 2026-02-03 22:54:01 +05:30
abhigyanpatwari 951423cd15 Merge branch 'main' into gitnexus_cli
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2026-02-03 22:53:49 +05:30
abhigyanpatwari 9cd380966d docs: add PolyForm Noncommercial License 1.0.0 2026-02-03 22:20:15 +05:30
abhigyanpatwari 1ae08ee9fc refactor: make mcp standalone-only, remove legacy browser bridge 2026-02-03 22:19:44 +05:30
abhigyanpatwari fcbb6f9e92 standalone MCP working 2026-02-03 04:53:10 +05:30
abhigyanpatwari 4357a48fae CLI implemented, UI connection to CLI works 2026-02-02 06:08:06 +05:30
abhigyanpatwari 6a7f837577 mreged 2026-02-01 03:05:52 +05:30
abhigyanpatwari 9b99c6baa9 haiku architecture generation with gitnexus MCP test 2026-02-01 03:02:09 +05:30
Abhigyan Patwari 42115f0ad7 Remove AI tools comparison table from README
Removed comparison table for AI coding tools and their limitations.
2026-01-31 01:47:37 +05:30
Abhigyan Patwari 67c506816a Merge pull request #35 from abhigyanpatwari/leidens_algorithm_and_processmap
updated readme
2026-01-30 04:10:37 +05:30
Abhigyan Patwari 1e39f77718 Merge branch 'main' into leidens_algorithm_and_processmap 2026-01-30 04:10:28 +05:30
abhigyanpatwari e3f4d4b365 updated readme 2026-01-30 04:08:55 +05:30
Abhigyan Patwari 1dfd80d2ea Update asset link and deployment information in README
Updated asset link in README and added a note about deployment.
2026-01-29 14:22:04 +05:30
Abhigyan Patwari e35cb2b920 Update asset links in README
Updated asset links in README.md.
2026-01-29 02:50:46 +05:30
Abhigyan Patwari 4495632e82 Merge pull request #34 from abhigyanpatwari/leidens_algorithm_and_processmap
Leidens algorithm and processmap
2026-01-29 00:11:19 +05:30
abhigyanpatwari f8b4a5c31f Process Map mermaid and other UI tweaks 2026-01-29 00:03:15 +05:30
abhigyanpatwari 4541468320 process maps and funding.yml 2026-01-28 13:14:16 +05:30
Abhigyan Patwari fe3c4dd978 Merge pull request #33 from abhigyanpatwari/leidens_algorithm_and_processmap
Leidens algorithm and processmap
2026-01-28 01:13:58 +05:30
abhigyanpatwari d3dbdb1aff enhanced process maps 2026-01-28 01:13:10 +05:30
abhigyanpatwari 32c3544d0d impact tool test 2026-01-27 23:05:34 +05:30
abhigyanpatwari c2ce19f7a4 schema fixes to handle previously unhandled relations in Go 2026-01-26 23:37:53 +05:30
abhigyanpatwari 53f17ddf17 fixed constructor to method relation not getting stored in kuzu issue 2026-01-26 22:58:15 +05:30
Abhigyan Patwari d02f861309 Merge pull request #31 from abhigyanpatwari/leidens_algorithm_and_processmap
Leidens algorithm and processmap
2026-01-23 19:35:49 +05:30
abhigyanpatwari 46330fa301 go c c# java python ts js rust now supported 2026-01-23 19:11:24 +05:30
abhigyanpatwari 1b5a3bbc1b leidens algo implemented 2026-01-21 01:45:54 +05:30
Abhigyan Patwari fa80520080 Merge pull request #28 from abhigyanpatwari/MCP-bridge
Mcp bridge
2026-01-19 09:08:12 +05:30
abhigyanpatwari 3b01fd1c3b feat(mcp): implement Hub & Spoke architecture for multi-agent support on single port 2026-01-19 08:17:12 +05:30
abhigyanpatwari 20150c2e1f fix(mcp): implement multi-port support for daemon and client to allow concurrent agents 2026-01-19 07:49:52 +05:30
abhigyanpatwari 72b679dbdb updated todo.md with my braindump formatted by gemini 2026-01-19 07:35:26 +05:30
abhigyanpatwari 8df16704d5 published mcp package on npm, modfied MCP button popup 2026-01-19 07:01:39 +05:30
abhigyanpatwari c8d8304004 refactor: simplify MCP popup with improved UI and config display 2026-01-19 06:43:02 +05:30
abhigyanpatwari 44efd94cc5 fix: add mermaid ESM alias to fix d3 crash on Vercel 2026-01-19 06:16:19 +05:30
abhigyanpatwari b1c207f31e deleted invalid vercel json 2026-01-18 19:35:52 +05:30
abhigyanpatwari 44d421b852 build issue fix 2026-01-18 19:33:28 +05:30
abhigyanpatwari 88ecbf5eb4 UI minor fixes 2026-01-18 06:27:00 +05:30
abhigyanpatwari 2e59a894c1 confidence score added for robustness, all tools and prompt adjusted to leverage it 2026-01-18 06:17:58 +05:30
abhigyanpatwari 87fedd17e7 fixed blast radius analyses, method calls relations failing solved, mcp simplified ( need to publish into npm ) 2026-01-18 06:05:44 +05:30
abhigyanpatwari 055f13f9c8 mermaid fixes 2026-01-18 01:04:24 +05:30
abhigyanpatwari 898e28ba1f CALLS edges enabled by default in UI 2026-01-17 23:36:25 +05:30
abhigyanpatwari 2a205af78b moved Readme to root 2026-01-17 23:11:25 +05:30
abhigyanpatwari e153d30c8f MCP bridge added 2026-01-17 23:08:48 +05:30
abhigyanpatwari df4f9e30c2 MCP working further test and improvements needed 2026-01-16 13:08:34 +05:30
Abhigyan Patwari c7449d1804 Merge pull request #26 from abhigyanpatwari/Schema_Experiments
readme mermain fix
2026-01-15 05:39:40 +05:30
abhigyanpatwari 880eacf84a readme mermain fix 2026-01-15 05:38:15 +05:30
Abhigyan Patwari 87fac7c15f Merge pull request #25 from abhigyanpatwari/Schema_Experiments
blast radius tool implemented
2026-01-15 03:04:28 +05:30
abhigyanpatwari 0ac6448632 blast radius tool implemented 2026-01-15 02:57:50 +05:30
Abhigyan Patwari 2083f79810 Update README.md 2026-01-15 00:34:55 +05:30
Abhigyan Patwari 0ba97d8acb Update README with asset link and image
Added a link to an asset and an image in the README.
2026-01-13 01:32:32 +05:30
Abhigyan Patwari cc66019ee4 Merge pull request #24 from abhigyanpatwari/Schema_Experiments
Schema experiments
2026-01-13 01:17:17 +05:30
abhigyanpatwari a7c526e6d4 readme update again 2026-01-13 01:16:30 +05:30
abhigyanpatwari 63af090dec more readme edits 2026-01-13 01:10:43 +05:30
Abhigyan Patwari e17b46cb94 Merge pull request #23 from abhigyanpatwari/Schema_Experiments
minor change in readme
2026-01-13 00:57:57 +05:30
abhigyanpatwari 7e4a1df565 minor change in readme 2026-01-13 00:57:02 +05:30
Abhigyan Patwari b8a4713294 Merge pull request #22 from abhigyanpatwari/Schema_Experiments
readme updated
2026-01-13 00:54:29 +05:30
abhigyanpatwari 736ecdbb1b readme updated 2026-01-13 00:53:48 +05:30
Abhigyan Patwari fbf6e4549e Merge pull request #21 from abhigyanpatwari/Schema_Experiments
fixed settings icon and added openai provider support
2026-01-12 21:54:47 +05:30
abhigyanpatwari f83d90b090 fixed settings icon 2026-01-12 21:19:53 +05:30
Abhigyan Patwari 5bb2510656 Merge pull request #20 from abhigyanpatwari/Schema_Experiments
Agent and UI improvements
2026-01-12 20:56:02 +05:30
abhigyanpatwari f038253733 Improved prompt 2026-01-12 20:47:46 +05:30
abhigyanpatwari 034f6ab37f grounding and prompt enhancement 2026-01-12 20:12:31 +05:30
Abhigyan Patwari 6ee89fa3cd Update asset link in README.md 2026-01-11 21:10:57 +05:30
Abhigyan Patwari 2be26506a9 Merge pull request #19 from abhigyanpatwari/Schema_Experiments
better regex for grounding, UI improvements
2026-01-11 20:44:32 +05:30
abhigyanpatwari 7a820068ba better regex for grounding, UI improvements 2026-01-11 20:39:22 +05:30
Abhigyan Patwari 16c50deea1 Merge pull request #18 from abhigyanpatwari/Schema_Experiments
Agent UI improved. Prompt injected with repo directory structure, hot…
2026-01-11 19:43:45 +05:30
abhigyanpatwari 569466e446 Agent UI improved. Prompt injected with repo directory structure, hotspots and enforced grounding 2026-01-11 19:30:11 +05:30
Abhigyan Patwari f266cc89a3 Merge pull request #17 from abhigyanpatwari/Schema_Experiments
Minor prompt fix
2026-01-11 07:48:54 +05:30
abhigyanpatwari 4a1fb961ef a 2026-01-11 07:43:05 +05:30
Abhigyan Patwari f4f56f9213 Merge pull request #16 from abhigyanpatwari/Schema_Experiments
Schema experiments
2026-01-11 07:24:36 +05:30
abhigyanpatwari 99daaa7ef4 fix: symlink files support implemented 2026-01-11 07:20:27 +05:30
abhigyanpatwari 4cda68bdf2 fixed: S-queries modified to support arrow functions 2026-01-11 06:56:54 +05:30
abhigyanpatwari a838324a10 fixed wrong cyfer generation due to old tool description 2026-01-11 05:31:02 +05:30
abhigyanpatwari bdbad8fead Changed Kuzu schema for better agent intution, updated tools, combined / removed redundant tools 2026-01-11 05:04:02 +05:30
Abhigyan Patwari 608332693d Merge pull request #14 from abhigyanpatwari/Schema_Experiments
prompt changes
2026-01-10 11:36:16 +05:30
abhigyanpatwari b221745af0 prompt changes 2026-01-10 11:30:48 +05:30
abhigyanpatwari 9ee4972aa6 UI improvements and prompt improvements 2026-01-10 00:38:15 +05:30
abhigyanpatwari 0a42fe46ac fixed LLM provider config warning issue 2026-01-09 21:06:31 +05:30
Abhigyan Patwari 65004450da Merge pull request #10 from abhigyanpatwari/embeddings_pipeline
Embeddings pipeline
2026-01-09 19:44:32 +05:30
abhigyanpatwari 0b053c8846 UI improvements 2026-01-09 19:43:24 +05:30
Abhigyan Patwari c0e210de32 Update README with supported languages
Added support information for additional programming languages.
2026-01-09 02:42:54 +05:30
abhigyanpatwari 1498e00d2d highlight toggle added 2026-01-08 05:05:15 +05:30
Abhigyan Patwari b0cfba2d90 Merge pull request #9 from abhigyanpatwari/embeddings_pipeline
Embeddings pipeline
2026-01-08 04:45:04 +05:30
abhigyanpatwari 127081406e UI improvements 2026-01-08 04:44:35 +05:30
abhigyanpatwari a5a20bc26d bm25 + semantic search works under the hood in search tool 2026-01-07 16:50:08 +05:30
Abhigyan Patwari 07126e835a Update asset link in README.md 2026-01-07 01:02:54 +05:30
Abhigyan Patwari 22a9d3d76f Merge pull request #8 from abhigyanpatwari/embeddings_pipeline
prompt enhancements, azure ai provider fixes
2026-01-07 00:38:56 +05:30
abhigyanpatwari e7819f6f45 prompt enhancements, azure ai provider fixes 2026-01-07 00:38:21 +05:30
Abhigyan Patwari a3e2e9cb4e Merge pull request #7 from abhigyanpatwari/embeddings_pipeline
highlight tool fixes
2026-01-06 16:25:11 +05:30
abhigyanpatwari 01ec0a77a8 highlight tool fixes 2026-01-06 16:22:43 +05:30
Abhigyan Patwari 0f0ef0d4e7 Merge pull request #6 from abhigyanpatwari/embeddings_pipeline
highlight tool for agent, prompt enhancements
2026-01-06 16:02:08 +05:30
abhigyanpatwari da14a9b404 highlight tool for agent, prompt enhancements 2026-01-06 14:44:59 +05:30
Abhigyan Patwari 94be328196 Merge pull request #5 from abhigyanpatwari/embeddings_pipeline
Embeddings pipeline
2026-01-06 12:33:39 +05:30
abhigyanpatwari 3b26fc393e grep working, embedings autostart also working 2026-01-06 12:32:09 +05:30
Abhigyan Patwari c4a0e442a9 Update asset link in README.md 2026-01-06 06:56:49 +05:30
abhigyanpatwari 35b0c83153 Graph RAG agent is able to execute cyfers, but needs more testing, something is wrong ;-; 2026-01-06 05:12:11 +05:30
abhigyanpatwari d9a2058f52 implemented azureopenai and gemini provider. currently struggling with graph query tool for agent 2026-01-06 04:10:38 +05:30
Abhigyan Patwari 2cc27170d8 Merge pull request #4 from abhigyanpatwari/embeddings_pipeline
Embeddings pipeline
2026-01-05 23:41:11 +05:30
abhigyanpatwari 06fe177f59 set navigator and pipeline to any to bypass build issue. Its issue is originating from node modules, not my code so its fine :-), I hope 2026-01-05 23:28:28 +05:30
abhigyanpatwari 5e48f0a566 skipLibCheck (in tsconfig.app.json) set to true to fix build fails 2026-01-05 23:20:24 +05:30
abhigyanpatwari 0d4092307c ficed signin popup from proxy server when pvt repo was cloned withot PAT 2026-01-05 23:03:41 +05:30
abhigyanpatwari a85f72a282 readme uppdate 2026-01-05 20:21:09 +05:30
abhigyanpatwari 47ffbdc747 embedings pipeline and storage done -> faced issue with buffer memory limit for kuzu, wrote about it in readme -> solved and working 2026-01-05 07:49:58 +05:30
abhigyanpatwari 0ea3f2b845 setting up embeddings pipeline. Facing webgpu directx issue, maybe windows compatibility issue 2026-01-05 06:34:54 +05:30
abhigyanpatwari 80bb943037 added support for private repos in the proxy using github PAT optional input field 2026-01-04 03:18:06 +05:30
Abhigyan Patwari 02ba735fff Replace demo video link in README
Updated demo video link in README.md.
2026-01-04 00:55:41 +05:30
Abhigyan Patwari 4a9623a779 Merge pull request #3 from abhigyanpatwari/v2
readme
2026-01-04 00:42:00 +05:30
abhigyanpatwari 03c22246c0 readme 2026-01-04 00:39:37 +05:30
abhigyanpatwari dead9aae6c Manual cyfer query executor implemented 2026-01-03 23:31:19 +05:30
abhigyanpatwari 21719396fa GitNexus V2 - Complete refactor 2026-01-03 23:00:55 +05:30
1985 changed files with 181578 additions and 57904 deletions
+19
View File
@@ -0,0 +1,19 @@
{
"name": "gitnexus-marketplace",
"owner": {
"name": "GitNexus",
"email": "nico@gitnexus.dev"
},
"metadata": {
"description": "Code intelligence powered by a knowledge graph — execution flows, blast radius, and semantic search",
"homepage": "https://github.com/nicosxt/gitnexus"
},
"plugins": [
{
"name": "gitnexus",
"version": "1.3.3",
"source": "./gitnexus-claude-plugin",
"description": "Code intelligence powered by a knowledge graph. Provides execution flow tracing, blast radius analysis, and augmented search across your codebase."
}
]
}
@@ -0,0 +1,82 @@
---
name: gitnexus-cli
description: "Use when the user needs to run GitNexus CLI commands like analyze/index a repo, check status, clean the index, generate a wiki, or list indexed repos. Examples: \"Index this repo\", \"Reanalyze the codebase\", \"Generate a wiki\""
---
# GitNexus CLI Commands
All commands work via `npx` — no global install required.
## Commands
### analyze — Build or refresh the index
```bash
npx gitnexus analyze
```
Run from the project root. This parses all source files, builds the knowledge graph, writes it to `.gitnexus/`, and generates CLAUDE.md / AGENTS.md context files.
| Flag | Effect |
| -------------- | ---------------------------------------------------------------- |
| `--force` | Force full re-index even if up to date |
| `--embeddings` | Enable embedding generation for semantic search (off by default) |
**When to run:** First time in a project, after major code changes, or when `gitnexus://repo/{name}/context` reports the index is stale. In Claude Code, a PostToolUse hook runs `analyze` automatically after `git commit` and `git merge`, preserving embeddings if previously generated.
### status — Check index freshness
```bash
npx gitnexus status
```
Shows whether the current repo has a GitNexus index, when it was last updated, and symbol/relationship counts. Use this to check if re-indexing is needed.
### clean — Delete the index
```bash
npx gitnexus clean
```
Deletes the `.gitnexus/` directory and unregisters the repo from the global registry. Use before re-indexing if the index is corrupt or after removing GitNexus from a project.
| Flag | Effect |
| --------- | ------------------------------------------------- |
| `--force` | Skip confirmation prompt |
| `--all` | Clean all indexed repos, not just the current one |
### wiki — Generate documentation from the graph
```bash
npx gitnexus wiki
```
Generates repository documentation from the knowledge graph using an LLM. Requires an API key (saved to `~/.gitnexus/config.json` on first use).
| Flag | Effect |
| ------------------- | ----------------------------------------- |
| `--force` | Force full regeneration |
| `--model <model>` | LLM model (default: minimax/minimax-m2.5) |
| `--base-url <url>` | LLM API base URL |
| `--api-key <key>` | LLM API key |
| `--concurrency <n>` | Parallel LLM calls (default: 3) |
| `--gist` | Publish wiki as a public GitHub Gist |
### list — Show all indexed repos
```bash
npx gitnexus list
```
Lists all repositories registered in `~/.gitnexus/registry.json`. The MCP `list_repos` tool provides the same information.
## After Indexing
1. **Read `gitnexus://repo/{name}/context`** to verify the index loaded
2. Use the other GitNexus skills (`exploring`, `debugging`, `impact-analysis`, `refactoring`) for your task
## Troubleshooting
- **"Not inside a git repository"**: Run from a directory inside a git repo
- **Index is stale after re-analyzing**: Restart Claude Code to reload the MCP server
- **Embeddings slow**: Omit `--embeddings` (it's off by default) or set `OPENAI_API_KEY` for faster API-based embedding
@@ -0,0 +1,89 @@
---
name: gitnexus-debugging
description: "Use when the user is debugging a bug, tracing an error, or asking why something fails. Examples: \"Why is X failing?\", \"Where does this error come from?\", \"Trace this bug\""
---
# Debugging with GitNexus
## When to Use
- "Why is this function failing?"
- "Trace where this error comes from"
- "Who calls this method?"
- "This endpoint returns 500"
- Investigating bugs, errors, or unexpected behavior
## Workflow
```
1. gitnexus_query({query: "<error or symptom>"}) → Find related execution flows
2. gitnexus_context({name: "<suspect>"}) → See callers/callees/processes
3. READ gitnexus://repo/{name}/process/{name} → Trace execution flow
4. gitnexus_cypher({query: "MATCH path..."}) → Custom traces if needed
```
> If "Index is stale" → run `npx gitnexus analyze` in terminal.
## Checklist
```
- [ ] Understand the symptom (error message, unexpected behavior)
- [ ] gitnexus_query for error text or related code
- [ ] Identify the suspect function from returned processes
- [ ] gitnexus_context to see callers and callees
- [ ] Trace execution flow via process resource if applicable
- [ ] gitnexus_cypher for custom call chain traces if needed
- [ ] Read source files to confirm root cause
```
## Debugging Patterns
| Symptom | GitNexus Approach |
| -------------------- | ---------------------------------------------------------- |
| Error message | `gitnexus_query` for error text → `context` on throw sites |
| Wrong return value | `context` on the function → trace callees for data flow |
| Intermittent failure | `context` → look for external calls, async deps |
| Performance issue | `context` → find symbols with many callers (hot paths) |
| Recent regression | `detect_changes` to see what your changes affect |
## Tools
**gitnexus_query** — find code related to error:
```
gitnexus_query({query: "payment validation error"})
→ Processes: CheckoutFlow, ErrorHandling
→ Symbols: validatePayment, handlePaymentError, PaymentException
```
**gitnexus_context** — full context for a suspect:
```
gitnexus_context({name: "validatePayment"})
→ Incoming calls: processCheckout, webhookHandler
→ Outgoing calls: verifyCard, fetchRates (external API!)
→ Processes: CheckoutFlow (step 3/7)
```
**gitnexus_cypher** — custom call chain traces:
```cypher
MATCH path = (a)-[:CodeRelation {type: 'CALLS'}*1..2]->(b:Function {name: "validatePayment"})
RETURN [n IN nodes(path) | n.name] AS chain
```
## Example: "Payment endpoint returns 500 intermittently"
```
1. gitnexus_query({query: "payment error handling"})
→ Processes: CheckoutFlow, ErrorHandling
→ Symbols: validatePayment, handlePaymentError
2. gitnexus_context({name: "validatePayment"})
→ Outgoing calls: verifyCard, fetchRates (external API!)
3. READ gitnexus://repo/my-app/process/CheckoutFlow
→ Step 3: validatePayment → calls fetchRates (external)
4. Root cause: fetchRates calls external API without proper timeout
```
@@ -0,0 +1,78 @@
---
name: gitnexus-exploring
description: "Use when the user asks how code works, wants to understand architecture, trace execution flows, or explore unfamiliar parts of the codebase. Examples: \"How does X work?\", \"What calls this function?\", \"Show me the auth flow\""
---
# Exploring Codebases with GitNexus
## When to Use
- "How does authentication work?"
- "What's the project structure?"
- "Show me the main components"
- "Where is the database logic?"
- Understanding code you haven't seen before
## Workflow
```
1. READ gitnexus://repos → Discover indexed repos
2. READ gitnexus://repo/{name}/context → Codebase overview, check staleness
3. gitnexus_query({query: "<what you want to understand>"}) → Find related execution flows
4. gitnexus_context({name: "<symbol>"}) → Deep dive on specific symbol
5. READ gitnexus://repo/{name}/process/{name} → Trace full execution flow
```
> If step 2 says "Index is stale" → run `npx gitnexus analyze` in terminal.
## Checklist
```
- [ ] READ gitnexus://repo/{name}/context
- [ ] gitnexus_query for the concept you want to understand
- [ ] Review returned processes (execution flows)
- [ ] gitnexus_context on key symbols for callers/callees
- [ ] READ process resource for full execution traces
- [ ] Read source files for implementation details
```
## Resources
| Resource | What you get |
| --------------------------------------- | ------------------------------------------------------- |
| `gitnexus://repo/{name}/context` | Stats, staleness warning (~150 tokens) |
| `gitnexus://repo/{name}/clusters` | All functional areas with cohesion scores (~300 tokens) |
| `gitnexus://repo/{name}/cluster/{name}` | Area members with file paths (~500 tokens) |
| `gitnexus://repo/{name}/process/{name}` | Step-by-step execution trace (~200 tokens) |
## Tools
**gitnexus_query** — find execution flows related to a concept:
```
gitnexus_query({query: "payment processing"})
→ Processes: CheckoutFlow, RefundFlow, WebhookHandler
→ Symbols grouped by flow with file locations
```
**gitnexus_context** — 360-degree view of a symbol:
```
gitnexus_context({name: "validateUser"})
→ Incoming calls: loginHandler, apiMiddleware
→ Outgoing calls: checkToken, getUserById
→ Processes: LoginFlow (step 2/5), TokenRefresh (step 1/3)
```
## Example: "How does payment processing work?"
```
1. READ gitnexus://repo/my-app/context → 918 symbols, 45 processes
2. gitnexus_query({query: "payment processing"})
→ CheckoutFlow: processPayment → validateCard → chargeStripe
→ RefundFlow: initiateRefund → calculateRefund → processRefund
3. gitnexus_context({name: "processPayment"})
→ Incoming: checkoutHandler, webhookHandler
→ Outgoing: validateCard, chargeStripe, saveTransaction
4. Read src/payments/processor.ts for implementation details
```
@@ -0,0 +1,64 @@
---
name: gitnexus-guide
description: "Use when the user asks about GitNexus itself — available tools, how to query the knowledge graph, MCP resources, graph schema, or workflow reference. Examples: \"What GitNexus tools are available?\", \"How do I use GitNexus?\""
---
# GitNexus Guide
Quick reference for all GitNexus MCP tools, resources, and the knowledge graph schema.
## Always Start Here
For any task involving code understanding, debugging, impact analysis, or refactoring:
1. **Read `gitnexus://repo/{name}/context`** — codebase overview + check index freshness
2. **Match your task to a skill below** and **read that skill file**
3. **Follow the skill's workflow and checklist**
> If step 1 warns the index is stale, run `npx gitnexus analyze` in the terminal first.
## Skills
| Task | Skill to read |
| -------------------------------------------- | ------------------- |
| Understand architecture / "How does X work?" | `gitnexus-exploring` |
| Blast radius / "What breaks if I change X?" | `gitnexus-impact-analysis` |
| Trace bugs / "Why is X failing?" | `gitnexus-debugging` |
| Rename / extract / split / refactor | `gitnexus-refactoring` |
| Tools, resources, schema reference | `gitnexus-guide` (this file) |
| Index, status, clean, wiki CLI commands | `gitnexus-cli` |
## Tools Reference
| Tool | What it gives you |
| ---------------- | ------------------------------------------------------------------------ |
| `query` | Process-grouped code intelligence — execution flows related to a concept |
| `context` | 360-degree symbol view — categorized refs, processes it participates in |
| `impact` | Symbol blast radius — what breaks at depth 1/2/3 with confidence |
| `detect_changes` | Git-diff impact — what do your current changes affect |
| `rename` | Multi-file coordinated rename with confidence-tagged edits |
| `cypher` | Raw graph queries (read `gitnexus://repo/{name}/schema` first) |
| `list_repos` | Discover indexed repos |
## Resources Reference
Lightweight reads (~100-500 tokens) for navigation:
| Resource | Content |
| ---------------------------------------------- | ----------------------------------------- |
| `gitnexus://repo/{name}/context` | Stats, staleness check |
| `gitnexus://repo/{name}/clusters` | All functional areas with cohesion scores |
| `gitnexus://repo/{name}/cluster/{clusterName}` | Area members |
| `gitnexus://repo/{name}/processes` | All execution flows |
| `gitnexus://repo/{name}/process/{processName}` | Step-by-step trace |
| `gitnexus://repo/{name}/schema` | Graph schema for Cypher |
## Graph Schema
**Nodes:** File, Function, Class, Interface, Method, Community, Process
**Edges (via CodeRelation.type):** CALLS, IMPORTS, EXTENDS, IMPLEMENTS, DEFINES, MEMBER_OF, STEP_IN_PROCESS
```cypher
MATCH (caller)-[:CodeRelation {type: 'CALLS'}]->(f:Function {name: "myFunc"})
RETURN caller.name, caller.filePath
```
@@ -0,0 +1,97 @@
---
name: gitnexus-impact-analysis
description: "Use when the user wants to know what will break if they change something, or needs safety analysis before editing code. Examples: \"Is it safe to change X?\", \"What depends on this?\", \"What will break?\""
---
# Impact Analysis with GitNexus
## When to Use
- "Is it safe to change this function?"
- "What will break if I modify X?"
- "Show me the blast radius"
- "Who uses this code?"
- Before making non-trivial code changes
- Before committing — to understand what your changes affect
## Workflow
```
1. gitnexus_impact({target: "X", direction: "upstream"}) → What depends on this
2. READ gitnexus://repo/{name}/processes → Check affected execution flows
3. gitnexus_detect_changes() → Map current git changes to affected flows
4. Assess risk and report to user
```
> If "Index is stale" → run `npx gitnexus analyze` in terminal.
## Checklist
```
- [ ] gitnexus_impact({target, direction: "upstream"}) to find dependents
- [ ] Review d=1 items first (these WILL BREAK)
- [ ] Check high-confidence (>0.8) dependencies
- [ ] READ processes to check affected execution flows
- [ ] gitnexus_detect_changes() for pre-commit check
- [ ] Assess risk level and report to user
```
## Understanding Output
| Depth | Risk Level | Meaning |
| ----- | ---------------- | ------------------------ |
| d=1 | **WILL BREAK** | Direct callers/importers |
| d=2 | LIKELY AFFECTED | Indirect dependencies |
| d=3 | MAY NEED TESTING | Transitive effects |
## Risk Assessment
| Affected | Risk |
| ------------------------------ | -------- |
| <5 symbols, few processes | LOW |
| 5-15 symbols, 2-5 processes | MEDIUM |
| >15 symbols or many processes | HIGH |
| Critical path (auth, payments) | CRITICAL |
## Tools
**gitnexus_impact** — the primary tool for symbol blast radius:
```
gitnexus_impact({
target: "validateUser",
direction: "upstream",
minConfidence: 0.8,
maxDepth: 3
})
→ d=1 (WILL BREAK):
- loginHandler (src/auth/login.ts:42) [CALLS, 100%]
- apiMiddleware (src/api/middleware.ts:15) [CALLS, 100%]
→ d=2 (LIKELY AFFECTED):
- authRouter (src/routes/auth.ts:22) [CALLS, 95%]
```
**gitnexus_detect_changes** — git-diff based impact analysis:
```
gitnexus_detect_changes({scope: "staged"})
→ Changed: 5 symbols in 3 files
→ Affected: LoginFlow, TokenRefresh, APIMiddlewarePipeline
→ Risk: MEDIUM
```
## Example: "What breaks if I change validateUser?"
```
1. gitnexus_impact({target: "validateUser", direction: "upstream"})
→ d=1: loginHandler, apiMiddleware (WILL BREAK)
→ d=2: authRouter, sessionManager (LIKELY AFFECTED)
2. READ gitnexus://repo/my-app/processes
→ LoginFlow and TokenRefresh touch validateUser
3. Risk: 2 direct callers, 2 processes = MEDIUM
```
@@ -0,0 +1,163 @@
---
name: gitnexus-pr-review
description: "Use when the user wants to review a pull request, understand what a PR changes, assess risk of merging, or check for missing test coverage. Examples: \"Review this PR\", \"What does PR #42 change?\", \"Is this PR safe to merge?\""
---
# PR Review with GitNexus
## When to Use
- "Review this PR"
- "What does PR #42 change?"
- "Is this safe to merge?"
- "What's the blast radius of this PR?"
- "Are there missing tests for this PR?"
- Reviewing someone else's code changes before merge
## Workflow
```
1. gh pr diff <number> → Get the raw diff
2. gitnexus_detect_changes({scope: "compare", base_ref: "main"}) → Map diff to affected flows
3. For each changed symbol:
gitnexus_impact({target: "<symbol>", direction: "upstream"}) → Blast radius per change
4. gitnexus_context({name: "<key symbol>"}) → Understand callers/callees
5. READ gitnexus://repo/{name}/processes → Check affected execution flows
6. Summarize findings with risk assessment
```
> If "Index is stale" → run `npx gitnexus analyze` in terminal before reviewing.
## Checklist
```
- [ ] Fetch PR diff (gh pr diff or git diff base...head)
- [ ] gitnexus_detect_changes to map changes to affected execution flows
- [ ] gitnexus_impact on each non-trivial changed symbol
- [ ] Review d=1 items (WILL BREAK) — are callers updated?
- [ ] gitnexus_context on key changed symbols to understand full picture
- [ ] Check if affected processes have test coverage
- [ ] Assess overall risk level
- [ ] Write review summary with findings
```
## Review Dimensions
| Dimension | How GitNexus Helps |
| --- | --- |
| **Correctness** | `context` shows callers — are they all compatible with the change? |
| **Blast radius** | `impact` shows d=1/d=2/d=3 dependents — anything missed? |
| **Completeness** | `detect_changes` shows all affected flows — are they all handled? |
| **Test coverage** | `impact({includeTests: true})` shows which tests touch changed code |
| **Breaking changes** | d=1 upstream items that aren't updated in the PR = potential breakage |
## Risk Assessment
| Signal | Risk |
| --- | --- |
| Changes touch <3 symbols, 0-1 processes | LOW |
| Changes touch 3-10 symbols, 2-5 processes | MEDIUM |
| Changes touch >10 symbols or many processes | HIGH |
| Changes touch auth, payments, or data integrity code | CRITICAL |
| d=1 callers exist outside the PR diff | Potential breakage — flag it |
## Tools
**gitnexus_detect_changes** — map PR diff to affected execution flows:
```
gitnexus_detect_changes({scope: "compare", base_ref: "main"})
→ Changed: 8 symbols in 4 files
→ Affected processes: CheckoutFlow, RefundFlow, WebhookHandler
→ Risk: MEDIUM
```
**gitnexus_impact** — blast radius per changed symbol:
```
gitnexus_impact({target: "validatePayment", direction: "upstream"})
→ d=1 (WILL BREAK):
- processCheckout (src/checkout.ts:42) [CALLS, 100%]
- webhookHandler (src/webhooks.ts:15) [CALLS, 100%]
→ d=2 (LIKELY AFFECTED):
- checkoutRouter (src/routes/checkout.ts:22) [CALLS, 95%]
```
**gitnexus_impact with tests** — check test coverage:
```
gitnexus_impact({target: "validatePayment", direction: "upstream", includeTests: true})
→ Tests that cover this symbol:
- validatePayment.test.ts [direct]
- checkout.integration.test.ts [via processCheckout]
```
**gitnexus_context** — understand a changed symbol's role:
```
gitnexus_context({name: "validatePayment"})
→ Incoming calls: processCheckout, webhookHandler
→ Outgoing calls: verifyCard, fetchRates
→ Processes: CheckoutFlow (step 3/7), RefundFlow (step 1/5)
```
## Example: "Review PR #42"
```
1. gh pr diff 42 > /tmp/pr42.diff
→ 4 files changed: payments.ts, checkout.ts, types.ts, utils.ts
2. gitnexus_detect_changes({scope: "compare", base_ref: "main"})
→ Changed symbols: validatePayment, PaymentInput, formatAmount
→ Affected processes: CheckoutFlow, RefundFlow
→ Risk: MEDIUM
3. gitnexus_impact({target: "validatePayment", direction: "upstream"})
→ d=1: processCheckout, webhookHandler (WILL BREAK)
→ webhookHandler is NOT in the PR diff — potential breakage!
4. gitnexus_impact({target: "PaymentInput", direction: "upstream"})
→ d=1: validatePayment (in PR), createPayment (NOT in PR)
→ createPayment uses the old PaymentInput shape — breaking change!
5. gitnexus_context({name: "formatAmount"})
→ Called by 12 functions — but change is backwards-compatible (added optional param)
6. Review summary:
- MEDIUM risk — 3 changed symbols affect 2 execution flows
- BUG: webhookHandler calls validatePayment but isn't updated for new signature
- BUG: createPayment depends on PaymentInput type which changed
- OK: formatAmount change is backwards-compatible
- Tests: checkout.test.ts covers processCheckout path, but no webhook test
```
## Review Output Format
Structure your review as:
```markdown
## PR Review: <title>
**Risk: LOW / MEDIUM / HIGH / CRITICAL**
### Changes Summary
- <N> symbols changed across <M> files
- <P> execution flows affected
### Findings
1. **[severity]** Description of finding
- Evidence from GitNexus tools
- Affected callers/flows
### Missing Coverage
- Callers not updated in PR: ...
- Untested flows: ...
### Recommendation
APPROVE / REQUEST CHANGES / NEEDS DISCUSSION
```
@@ -0,0 +1,121 @@
---
name: gitnexus-refactoring
description: "Use when the user wants to rename, extract, split, move, or restructure code safely. Examples: \"Rename this function\", \"Extract this into a module\", \"Refactor this class\", \"Move this to a separate file\""
---
# Refactoring with GitNexus
## When to Use
- "Rename this function safely"
- "Extract this into a module"
- "Split this service"
- "Move this to a new file"
- Any task involving renaming, extracting, splitting, or restructuring code
## Workflow
```
1. gitnexus_impact({target: "X", direction: "upstream"}) → Map all dependents
2. gitnexus_query({query: "X"}) → Find execution flows involving X
3. gitnexus_context({name: "X"}) → See all incoming/outgoing refs
4. Plan update order: interfaces → implementations → callers → tests
```
> If "Index is stale" → run `npx gitnexus analyze` in terminal.
## Checklists
### Rename Symbol
```
- [ ] gitnexus_rename({symbol_name: "oldName", new_name: "newName", dry_run: true}) — preview all edits
- [ ] Review graph edits (high confidence) and ast_search edits (review carefully)
- [ ] If satisfied: gitnexus_rename({..., dry_run: false}) — apply edits
- [ ] gitnexus_detect_changes() — verify only expected files changed
- [ ] Run tests for affected processes
```
### Extract Module
```
- [ ] gitnexus_context({name: target}) — see all incoming/outgoing refs
- [ ] gitnexus_impact({target, direction: "upstream"}) — find all external callers
- [ ] Define new module interface
- [ ] Extract code, update imports
- [ ] gitnexus_detect_changes() — verify affected scope
- [ ] Run tests for affected processes
```
### Split Function/Service
```
- [ ] gitnexus_context({name: target}) — understand all callees
- [ ] Group callees by responsibility
- [ ] gitnexus_impact({target, direction: "upstream"}) — map callers to update
- [ ] Create new functions/services
- [ ] Update callers
- [ ] gitnexus_detect_changes() — verify affected scope
- [ ] Run tests for affected processes
```
## Tools
**gitnexus_rename** — automated multi-file rename:
```
gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true})
→ 12 edits across 8 files
→ 10 graph edits (high confidence), 2 ast_search edits (review)
→ Changes: [{file_path, edits: [{line, old_text, new_text, confidence}]}]
```
**gitnexus_impact** — map all dependents first:
```
gitnexus_impact({target: "validateUser", direction: "upstream"})
→ d=1: loginHandler, apiMiddleware, testUtils
→ Affected Processes: LoginFlow, TokenRefresh
```
**gitnexus_detect_changes** — verify your changes after refactoring:
```
gitnexus_detect_changes({scope: "all"})
→ Changed: 8 files, 12 symbols
→ Affected processes: LoginFlow, TokenRefresh
→ Risk: MEDIUM
```
**gitnexus_cypher** — custom reference queries:
```cypher
MATCH (caller)-[:CodeRelation {type: 'CALLS'}]->(f:Function {name: "validateUser"})
RETURN caller.name, caller.filePath ORDER BY caller.filePath
```
## Risk Rules
| Risk Factor | Mitigation |
| ------------------- | ----------------------------------------- |
| Many callers (>5) | Use gitnexus_rename for automated updates |
| Cross-area refs | Use detect_changes after to verify scope |
| String/dynamic refs | gitnexus_query to find them |
| External/public API | Version and deprecate properly |
## Example: Rename `validateUser` to `authenticateUser`
```
1. gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: true})
→ 12 edits: 10 graph (safe), 2 ast_search (review)
→ Files: validator.ts, login.ts, middleware.ts, config.json...
2. Review ast_search edits (config.json: dynamic reference!)
3. gitnexus_rename({symbol_name: "validateUser", new_name: "authenticateUser", dry_run: false})
→ Applied 12 edits across 8 files
4. gitnexus_detect_changes({scope: "all"})
→ Affected: LoginFlow, TokenRefresh
→ Risk: MEDIUM — run tests for these flows
```
+21
View File
@@ -0,0 +1,21 @@
---
alwaysApply: true
---
# GitNexus — Cursor project rules
Last reviewed: 2026-03-24
Canonical agent instructions: **[AGENTS.md](../AGENTS.md)** (GitNexus MCP rules, monorepo commands, Cursor Cloud notes). **[CLAUDE.md](../CLAUDE.md)** adds Claude Code-specific notes and points back to AGENTS.md for GitNexus.
## Non-negotiables (always apply)
- NEVER edit a function/class/method without running `gitnexus_impact` first.
- NEVER rename symbols with find-and-replace — use `gitnexus_rename`.
- NEVER commit without running `gitnexus_detect_changes()`.
- NEVER ignore HIGH/CRITICAL risk warnings from impact analysis.
- NEVER run `npx gitnexus analyze` without `--embeddings` if `.gitnexus/meta.json` shows stored embeddings.
Full rules: **[AGENTS.md](../AGENTS.md)** (`gitnexus:start` block, Cursor Cloud section).
**Rule architecture:** Prefer this file plus optional `.cursor/rules/*.mdc` globs (YAML `globs` in frontmatter). Legacy `.cursorrules` is deprecated; content lives here.
+12
View File
@@ -0,0 +1,12 @@
---
globs:
- "gitnexus/**"
- "gitnexus-web/**"
---
# GitNexus build/test quick refs
- CLI (`gitnexus/`): `npm test`; `npm run test:integration`; `npx tsc --noEmit`.
- Web (`gitnexus-web/`): `npm test`; `npm run dev`; `npx tsc -b --noEmit`; `E2E=1 npx playwright test` (needs servers).
- `npm install` in `gitnexus/` runs `prepare` (tsc build) and `postinstall` (tree-sitter patches); needs `python3`, `make`, `g++`.
- LadybugDB locking tests may fail in containerized environments because of `/tmp` file locks (known issue, not a code bug).
+14
View File
@@ -0,0 +1,14 @@
---
globs:
- "eval/**"
---
# GitNexus eval harness (Python)
- **Run tests**: `cd eval && uv run pytest tests/`
- **Run with coverage**: `cd eval && uv run coverage run -m pytest tests/ && uv run coverage report`
- **Lint**: `cd eval && uv run ruff check .`
- **Run eval**: `cd eval && uv run python run_eval.py --config configs/<config>.yaml`
- Shared constants live in `eval/constants.py`; tool specs in `eval/tool_registry.py`.
- Error logging uses `utils/errors.py` — set `GITNEXUS_EVAL_DEBUG=1` for full tracebacks.
- Property-based tests use Hypothesis (`eval/tests/test_property_based.py`).
+5
View File
@@ -0,0 +1,5 @@
# Deprecated for Cursor Agent Mode
Use **`.cursor/index.mdc`** (`alwaysApply: true`) for project rules. See [AGENTS.md](AGENTS.md).
This file is kept only as a breadcrumb for older workflows.
+5
View File
@@ -0,0 +1,5 @@
# Prettier initial formatting (2026-03-28)
afcc3d1523f99c77ff67c4fd1af12334660113f6
# ESLint unused import removal (2026-03-28)
1491826bc8da5436d3b1eb092d9274f2c9f028a7
+2
View File
@@ -0,0 +1,2 @@
* text=auto eol=lf
.husky/* text eol=lf
+3
View File
@@ -0,0 +1,3 @@
# These are supported funding model platforms
github: abhigyanpatwari
+86
View File
@@ -0,0 +1,86 @@
name: Bug report
description: Report unexpected behavior or a regression
labels: [bug]
body:
- type: markdown
attributes:
value: |
**Goal:** capture enough context to reproduce and fix the issue quickly.
Use **one issue per bug**; split unrelated problems.
- type: dropdown
id: area
attributes:
label: Area
description: Where does the problem show up?
options:
- gitnexus (CLI / core / indexing / MCP server)
- gitnexus-web (browser UI / WASM / workers)
- CI / GitHub Actions
- Documentation / developer experience
- Other
validations:
required: true
- type: textarea
id: summary
attributes:
label: Summary
description: One sentence — what went wrong?
validations:
required: true
- type: textarea
id: context
attributes:
label: Context
description: What were you trying to do? Any relevant links, PRs, or commits?
validations:
required: false
- type: textarea
id: expected
attributes:
label: Expected behavior
validations:
required: true
- type: textarea
id: actual
attributes:
label: Actual behavior
validations:
required: true
- type: textarea
id: reproduce
attributes:
label: Steps to reproduce
description: Ordered steps, sample repo or minimal case, commands run.
placeholder: |
1. …
2. …
3. …
validations:
required: true
- type: textarea
id: environment
attributes:
label: Environment
description: OS, Node version, browser (if web), GitNexus version or commit SHA.
placeholder: |
- OS:
- Node:
- Browser (if applicable):
- Commit / version:
validations:
required: false
- type: textarea
id: logs
attributes:
label: Logs / screenshots
description: Paste errors, stack traces, or attach screenshots (redact secrets).
validations:
required: false
+1
View File
@@ -0,0 +1 @@
blank_issues_enabled: true
@@ -0,0 +1,74 @@
name: Feature request
description: Propose a new capability or improvement
labels: [enhancement]
body:
- type: markdown
attributes:
value: |
**Goal:** describe the problem and desired outcome so maintainers can size and prioritize.
Prefer **small, shippable** requests; split large ideas into phases.
- type: dropdown
id: area
attributes:
label: Area
description: Primary part of the monorepo this relates to.
options:
- gitnexus (CLI / core / indexing / MCP server)
- gitnexus-web (browser UI / WASM / workers)
- CI / release / packaging
- Documentation / developer experience
- Other
validations:
required: true
- type: textarea
id: problem
attributes:
label: Problem or opportunity
description: What pain point or gap exists today?
validations:
required: true
- type: textarea
id: proposal
attributes:
label: Proposed solution
description: What should happen instead? User-visible behavior, APIs, or UX.
validations:
required: true
- type: textarea
id: alternatives
attributes:
label: Alternatives considered
description: Other approaches you considered and why this one is preferred.
validations:
required: false
- type: textarea
id: acceptance
attributes:
label: Acceptance criteria
description: Testable conditions for “done” (bullets or checkboxes in prose).
placeholder: |
- When … then …
- Documentation / tests updated where appropriate
validations:
required: false
- type: textarea
id: constraints
attributes:
label: Constraints
description: Compatibility, performance, security, or “must not change” boundaries.
validations:
required: false
- type: checkboxes
id: willing
attributes:
label: Contribution
options:
- label: I am willing to open a PR for this (may need design discussion first).
required: false
+52
View File
@@ -0,0 +1,52 @@
## Summary
<!-- One or two sentences: what does this PR change? -->
## Motivation / context
<!-- Why is this change needed? Link issues, ADRs, or prior discussion. -->
## Areas touched
<!-- Check all that apply -->
- [ ] `gitnexus/` (CLI / core / MCP server)
- [ ] `gitnexus-web/` (Vite / React UI)
- [ ] `.github/` (workflows, actions)
- [ ] `eval/` or other tooling
- [ ] Docs / agent config only (`AGENTS.md`, `CLAUDE.md`, `.cursor/`, `llms.txt`, etc.)
## Scope & constraints
**In scope**
- <!-- bullets -->
**Explicitly out of scope / not done here**
- <!-- bullets — prevents reviewers assuming missing work is an oversight -->
## Implementation notes
<!-- Optional: design choices, tradeoffs, follow-ups -->
## Testing & verification
<!-- What you ran; paste commands. Omit sections that do not apply. -->
- [ ] `cd gitnexus && npm test`
- [ ] `cd gitnexus && npm run test:integration` *(if core/indexing/MCP paths changed)*
- [ ] `cd gitnexus && npx tsc --noEmit`
- [ ] `cd gitnexus-web && npm test` *(if web changed)*
- [ ] `cd gitnexus-web && npx tsc -b --noEmit` *(if web changed)*
- [ ] Manual / Playwright E2E *(note environment — see `gitnexus-web/e2e/`)*
## Risk & rollout
<!-- Breaking changes, migrations, index refresh (`npx gitnexus analyze`), release notes -->
## Checklist
- [ ] PR body meets repo minimum length (workflow may label short descriptions)
- [ ] If `AGENTS.md` / overlays changed: headers, scope block, and changelog updated per project conventions
- [ ] No secrets, tokens, or machine-specific paths committed
@@ -0,0 +1,21 @@
name: Setup GitNexus Web
description: Setup Node.js 20, build gitnexus-shared, install web dependencies
runs:
using: composite
steps:
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 20
cache: npm
cache-dependency-path: gitnexus-web/package-lock.json
- name: Build gitnexus-shared
run: npm install && npm run build
shell: bash
working-directory: gitnexus-shared
- name: Install web dependencies
run: npm ci
shell: bash
working-directory: gitnexus-web
+33
View File
@@ -0,0 +1,33 @@
name: Setup GitNexus
description: Setup Node.js 20, install dependencies, and optionally build
inputs:
build:
description: Whether to run npm run build after install
required: false
default: 'false'
runs:
using: composite
steps:
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 20
cache: npm
cache-dependency-path: gitnexus/package-lock.json
- name: Build gitnexus-shared
run: npm install && npm run build
shell: bash
working-directory: gitnexus-shared
- name: Install dependencies
run: npm ci
shell: bash
working-directory: gitnexus
- name: Build
if: ${{ inputs.build == 'true' }}
run: npm run build
shell: bash
working-directory: gitnexus
+45
View File
@@ -0,0 +1,45 @@
changelog:
exclude:
labels:
- chore
authors:
- dependabot
- dependabot[bot]
categories:
- title: "\U0001F6A8 Security"
labels:
- security
- title: "\U0001F4A5 Breaking Changes"
labels:
- breaking
- title: "\U0001F680 Features"
labels:
- enhancement
- title: "\U0001F41B Bug Fixes"
labels:
- bug
- title: "\U0001F3CE\uFE0F Performance"
labels:
- performance
- title: "\U0001F9EA Tests"
labels:
- test
- title: "\U0001F504 Refactoring"
labels:
- refactor
- title: "\U0001F477 CI/CD"
labels:
- ci
- title: "\U0001F4E6 Dependencies"
labels:
- dependencies
- title: "\U0001F4DD Other Changes"
labels:
- '*'
exclude:
labels:
- dependencies
- ci
- test
- refactor
- chore
+257
View File
@@ -0,0 +1,257 @@
"""Pure math utilities for triage sweep embedding analysis.
All functions are stateless and perform no I/O (except model loading by FastEmbed).
Each function operates on numpy arrays and returns numpy arrays or plain Python types.
"""
from __future__ import annotations
import numpy as np
from numpy.typing import NDArray
from fastembed import TextEmbedding
from sklearn.decomposition import PCA
from sklearn.covariance import EllipticEnvelope
from sklearn.metrics.pairwise import cosine_similarity
# FastEmbed model — BAAI/bge-small-en-v1.5 produces 384-dimensional embeddings.
# ~46MB quantized ONNX, runs on CPU in ~0.5s per batch of 32.
EMBEDDING_MODEL: str = "BAAI/bge-small-en-v1.5"
# Embedding dimensionality (determined by model choice).
EMBEDDING_DIM: int = 384
# Batch size for FastEmbed. 32 balances memory and throughput on
# a 2-vCPU GitHub Actions runner with ~7GB RAM.
EMBEDDING_BATCH_SIZE: int = 32
def embed_texts(texts: list[str]) -> NDArray[np.float32]:
"""Embed a list of texts into dense vectors using FastEmbed.
Returns an array of shape (len(texts), 384) with dtype float32.
Empty input returns a (0, 384) array.
"""
if not texts:
return np.empty((0, EMBEDDING_DIM), dtype=np.float32)
model = TextEmbedding(model_name=EMBEDDING_MODEL)
vectors = list(model.embed(texts, batch_size=EMBEDDING_BATCH_SIZE))
return np.vstack(vectors).astype(np.float32)
def normalize_rows(matrix: NDArray[np.float32]) -> NDArray[np.float32]:
"""L2-normalize each row to unit length.
Zero-norm rows (e.g. from empty text) remain zero vectors.
Uses eps=1e-10 in the denominator to avoid division by zero.
"""
if matrix.shape[0] == 0:
return matrix
norms = np.linalg.norm(matrix, axis=1, keepdims=True)
return matrix / (norms + 1e-10)
def reduce_dimensions(
matrix: NDArray[np.float32],
max_components: int,
) -> NDArray[np.float32]:
"""Reduce dimensionality via PCA.
Computes n_components = min(max_components, n-1, d). If n_components < 1,
returns the matrix unchanged. Logs explained variance for observability.
"""
n, d = matrix.shape
if n <= 1:
return matrix
n_components = min(max_components, n - 1, d)
if n_components < 1:
return matrix
pca = PCA(n_components=n_components)
reduced = pca.fit_transform(matrix)
explained = pca.explained_variance_ratio_.sum()
print(f"PCA: {d}d -> {n_components}d, explained variance: {explained:.3f}")
return reduced.astype(np.float32)
def detect_outliers(
matrix: NDArray[np.float32],
contamination: float = 0.1,
iqr_multiplier: float = 3.0,
max_outlier_pct: float = 0.05,
) -> list[tuple[int, float]]:
"""Flag items whose Mahalanobis distance exceeds an IQR-based cutoff.
Uses EllipticEnvelope (robust covariance via MCD) to estimate the
multivariate Gaussian, then computes sqrt(squared Mahalanobis distance)
for each sample. The cutoff is Q75 + iqr_multiplier * IQR, which
adapts to the actual distribution of distances.
A hard cap ensures no more than max_outlier_pct * n items are flagged;
when the cap is hit, only the most extreme items (sorted by distance
descending) are kept.
Returns (index, distance) tuples sorted by index ascending, along with
the cutoff value stored as an attribute on the returned list.
"""
n = matrix.shape[0]
if n < 2:
return []
envelope = EllipticEnvelope(contamination=contamination, random_state=42)
envelope.fit(matrix)
# .mahalanobis() returns squared Mahalanobis distances
distances = np.sqrt(envelope.mahalanobis(matrix))
# IQR-based cutoff
q25, q75 = np.percentile(distances, [25, 75])
iqr = q75 - q25
cutoff = q75 + iqr_multiplier * iqr
outlier_mask = distances > cutoff
indices = np.where(outlier_mask)[0]
# Hard cap: keep at most max_outlier_pct * n items
max_count = max(1, int(max_outlier_pct * n))
if len(indices) > max_count:
# Sort by distance descending, take the most extreme
sorted_by_dist = sorted(indices, key=lambda i: distances[i], reverse=True)
indices = np.array(sorted_by_dist[:max_count])
# Sort by index ascending for stable output
indices = np.sort(indices)
result = [(int(idx), float(distances[idx])) for idx in indices]
# Attach cutoff as metadata so the report can use it
result = _OutlierResult(result) # type: ignore[assignment]
result.cutoff = float(cutoff) # type: ignore[attr-defined]
return result # type: ignore[return-value]
class _OutlierResult(list):
"""A list subclass that carries metadata (cutoff) from outlier detection."""
cutoff: float = 0.0
def find_duplicate_pairs(
matrix: NDArray[np.float32],
threshold: float,
) -> list[tuple[int, int, float]]:
"""Find pairs of items with cosine similarity above threshold.
Returns (i, j, similarity) tuples where i < j. The input should be
L2-normalized embeddings (full dimensionality, not PCA-reduced) so
cosine similarity equals the dot product.
"""
n = matrix.shape[0]
if n <= 1:
return []
sim_matrix = cosine_similarity(matrix)
# Upper triangle indices (i < j), excluding diagonal
rows, cols = np.triu_indices(n, k=1)
similarities = sim_matrix[rows, cols]
mask = similarities > threshold
pairs: list[tuple[int, int, float]] = []
for idx in np.where(mask)[0]:
pairs.append((int(rows[idx]), int(cols[idx]), float(similarities[idx])))
return pairs
# ── Label suggestion via z-score normalized embedding similarity ──────
# Z-score threshold: a label must be this many standard deviations above
# the column mean to be considered a match.
LABEL_Z_THRESHOLD: float = 1.5
# Margin gate: the top-1 label must beat the second-best by this many
# z-score units to be accepted (subsequent labels don't need a margin).
LABEL_Z_MARGIN: float = 0.5
# Floor for per-column standard deviation to avoid division by near-zero.
LABEL_Z_STD_FLOOR: float = 0.01
# Minimum raw cosine similarity required even if z-score is high.
# Prevents suggesting labels that are "relatively best" but still poor.
MIN_RAW_SIMILARITY: float = 0.3
# Maximum number of labels to suggest per item.
MAX_LABELS_PER_ITEM: int = 3
def suggest_labels(
item_embeddings: NDArray[np.float32],
label_embeddings: NDArray[np.float32],
label_names: list[str],
z_threshold: float = LABEL_Z_THRESHOLD,
z_margin: float = LABEL_Z_MARGIN,
std_floor: float = LABEL_Z_STD_FLOOR,
min_raw_sim: float = MIN_RAW_SIMILARITY,
max_per_item: int = MAX_LABELS_PER_ITEM,
) -> list[list[tuple[str, float]]]:
"""Suggest labels for each item using z-score normalized similarity.
1. Compute raw cosine similarity matrix (n items x m labels).
2. Column-wise z-score: for each label j, normalize across all items.
3. For each item, rank labels by z-score descending.
4. Accept a label only if z >= z_threshold AND raw_sim >= min_raw_sim.
5. Margin gate: the top-1 label must beat #2 by z_margin; subsequent
labels don't need a margin.
6. Cap at max_per_item.
Returns a list of length n, where each element is a list of
(label_name, raw_similarity) tuples. Empty list if nothing qualifies.
"""
n = item_embeddings.shape[0]
m = label_embeddings.shape[0]
if n == 0 or m == 0:
return [[] for _ in range(n)]
# (n, m) raw similarity matrix
sim_matrix = cosine_similarity(item_embeddings, label_embeddings)
# Column-wise z-score normalization
col_means = sim_matrix.mean(axis=0) # shape (m,)
col_stds = sim_matrix.std(axis=0) # shape (m,)
col_stds = np.maximum(col_stds, std_floor)
z_matrix = (sim_matrix - col_means) / col_stds
suggestions: list[list[tuple[str, float]]] = []
for i in range(n):
z_row = z_matrix[i]
raw_row = sim_matrix[i]
# Rank labels by z-score descending
ranked = np.argsort(z_row)[::-1]
item_labels: list[tuple[str, float]] = []
# Margin gate: top-1 z-score must beat #2 by z_margin.
# If not, the assignment is ambiguous — skip this item entirely.
if len(ranked) > 1:
top1_z = float(z_row[ranked[0]])
top2_z = float(z_row[ranked[1]])
if top1_z - top2_z < z_margin:
suggestions.append(item_labels)
continue
for rank_pos, idx in enumerate(ranked):
if len(item_labels) >= max_per_item:
break
z_val = float(z_row[idx])
raw_val = float(raw_row[idx])
# Must pass both z-threshold and raw similarity floor
if z_val < z_threshold or raw_val < min_raw_sim:
continue
item_labels.append((label_names[idx], raw_val))
suggestions.append(item_labels)
return suggestions
+4
View File
@@ -0,0 +1,4 @@
fastembed>=0.5.0
numpy>=1.26.0
scikit-learn>=1.4.0
scipy>=1.10.0
+600
View File
@@ -0,0 +1,600 @@
"""Triage sweep: fetch open issues/PRs, detect outliers and duplicates, generate a report.
Entrypoint script for the triage-sweep workflow. Fetches all open items via
the GitHub REST API, delegates embedding and analysis to embedding_utils,
generates a markdown report, and optionally creates a report issue.
"""
from __future__ import annotations
import json
import os
import sys
import urllib.request
import urllib.parse
from typing import TypedDict
from datetime import datetime, timezone
from embedding_utils import (
embed_texts,
normalize_rows,
reduce_dimensions,
detect_outliers,
find_duplicate_pairs,
suggest_labels,
LABEL_Z_THRESHOLD,
LABEL_Z_MARGIN,
LABEL_Z_STD_FLOOR,
MIN_RAW_SIMILARITY,
MAX_LABELS_PER_ITEM,
)
# ── Thresholds (overridable via workflow_dispatch inputs) ──────────────
# IQR multiplier for outlier cutoff: cutoff = Q75 + IQR_MULTIPLIER * IQR.
IQR_MULTIPLIER: float = float(os.environ.get("INPUT_IQR_MULTIPLIER", "3.0"))
# Hard cap: at most this fraction of items can be flagged as outliers.
MAX_OUTLIER_PCT: float = float(os.environ.get("INPUT_MAX_OUTLIER_PCT", "0.05"))
# EllipticEnvelope contamination: expected fraction of outliers in the data.
# Governs how aggressively the robust covariance downweights extreme points.
CONTAMINATION: float = float(os.environ.get("INPUT_CONTAMINATION", "0.1"))
# Cosine similarity above which two items are flagged as duplicates.
# 0.92 catches near-identical issues while tolerating paraphrasing.
COSINE_THRESHOLD: float = float(os.environ.get("INPUT_COSINE_THRESHOLD", "0.92"))
# Hard cap on items to process. Prevents runaway costs on very large repos.
MAX_ITEMS: int = int(os.environ.get("INPUT_MAX_ITEMS", "500"))
# When true, print report to stdout/file but do not create a GitHub issue.
DRY_RUN: bool = os.environ.get("INPUT_DRY_RUN", "false").lower() == "true"
# ── Fixed constants (not user-configurable) ───────────────────────────
# Minimum number of samples required for EllipticEnvelope to fit
# a Gaussian reliably. Must be >= 3 * PCA_MAX_COMPONENTS so the
# covariance matrix is estimated from enough data points.
PCA_MAX_COMPONENTS: int = 20
MIN_SAMPLES_FOR_OUTLIER_DETECTION: int = 100
# Max character length for embedding input text. bge-small-en-v1.5 has a
# 512-token context window (~4 chars/token). We keep title + body under
# this limit so the model sees the full text instead of silently truncating.
MAX_EMBED_CHARS: int = 2000
# GitHub REST API page size (max allowed is 100).
API_PAGE_SIZE: int = 100
# Report issue label.
REPORT_LABEL: str = "triage-report"
# Report file path (written for the summary step to pick up).
REPORT_FILE: str = "/tmp/triage-report.md"
class TriageItem(TypedDict):
"""One open issue or PR, with only the fields we need."""
number: int
title: str
html_url: str
is_pr: bool
labels: list[str]
created_at: str
# title + body concatenated, used as embedding input
text: str
def github_api_get(path: str) -> list[dict]:
"""Make a single authenticated GET request to the GitHub REST API.
Reads GITHUB_TOKEN and GITHUB_REPOSITORY from env. Raises SystemExit
with the HTTP status and response body on any non-2xx response.
"""
token = os.environ["GITHUB_TOKEN"]
repo = os.environ["GITHUB_REPOSITORY"]
url = f"https://api.github.com/repos/{repo}{path}"
req = urllib.request.Request(url)
req.add_header("Accept", "application/vnd.github+json")
req.add_header("Authorization", f"Bearer {token}")
req.add_header("X-GitHub-Api-Version", "2022-11-28")
try:
with urllib.request.urlopen(req, timeout=30) as resp:
return json.loads(resp.read().decode("utf-8"))
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", errors="replace")
print(f"::error::GitHub API {e.code}: {body}")
sys.exit(1)
def fetch_all_open_items() -> list[TriageItem]:
"""Paginate through all open issues and PRs.
Returns up to MAX_ITEMS TriageItem dicts. Items with a pull_request
key are marked is_pr=True. The text field is title + body concatenated.
"""
items: list[TriageItem] = []
page = 1
while len(items) < MAX_ITEMS:
path = (
f"/issues?state=open&per_page={API_PAGE_SIZE}"
f"&sort=created&direction=desc&page={page}"
)
data = github_api_get(path)
if not data:
break
for raw in data:
if len(items) >= MAX_ITEMS:
break
body = raw.get("body", "") or ""
full_text = f"{raw['title']}\n\n{body}"
# Truncate to fit the embedding model's token window.
# Title is always preserved; body gets clipped if needed.
if len(full_text) > MAX_EMBED_CHARS:
full_text = full_text[:MAX_EMBED_CHARS]
items.append(TriageItem(
number=raw["number"],
title=raw["title"],
html_url=raw["html_url"],
is_pr="pull_request" in raw,
labels=[lbl["name"] for lbl in raw.get("labels", [])],
created_at=raw["created_at"],
text=full_text,
))
if len(data) < API_PAGE_SIZE:
break
page += 1
return items
class RepoLabel(TypedDict):
"""A label from the repo with its embedding text."""
name: str
description: str
# "name: description" concatenated for embedding
text: str
def fetch_repo_labels() -> list[RepoLabel]:
"""Fetch all labels from the repository, paginating if needed.
Returns labels with name, description, and a text field suitable
for embedding ("name: description"). Labels with no description
use just the name.
"""
labels: list[RepoLabel] = []
page = 1
while True:
data = github_api_get(f"/labels?per_page={API_PAGE_SIZE}&page={page}")
for raw in data:
name = raw["name"]
desc = raw.get("description", "") or ""
text = f"{name}: {desc}" if desc else name
labels.append(RepoLabel(name=name, description=desc, text=text))
if len(data) < API_PAGE_SIZE:
break
page += 1
return labels
def apply_labels_to_item(item_number: int, labels: list[str]) -> None:
"""Add labels to a single issue/PR via the GitHub API.
Skips silently if labels list is empty. Uses POST which adds labels
without removing existing ones.
"""
if not labels:
return
token = os.environ["GITHUB_TOKEN"]
repo = os.environ["GITHUB_REPOSITORY"]
url = f"https://api.github.com/repos/{repo}/issues/{item_number}/labels"
payload = json.dumps({"labels": labels}).encode("utf-8")
req = urllib.request.Request(url, data=payload, method="POST")
req.add_header("Accept", "application/vnd.github+json")
req.add_header("Authorization", f"Bearer {token}")
req.add_header("X-GitHub-Api-Version", "2022-11-28")
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req, timeout=30) as resp:
resp.read()
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", errors="replace")
# Non-fatal: log warning but don't abort the sweep
print(f"::warning::Failed to label #{item_number}: {e.code} {body}")
def _item_age(created_at: str) -> str:
"""Compute a human-readable age string from an ISO 8601 created_at timestamp."""
try:
created = datetime.fromisoformat(created_at.replace("Z", "+00:00"))
delta = datetime.now(timezone.utc) - created
days = delta.days
if days < 1:
return "<1d"
if days < 30:
return f"{days}d"
if days < 365:
return f"{days // 30}mo"
return f"{days // 365}y"
except (ValueError, TypeError):
return "?"
def _suggested_action(a: TriageItem, b: TriageItem) -> str:
"""Determine a suggested action for a duplicate pair based on types and age."""
if a["is_pr"] and b["is_pr"]:
return "Review for overlap"
if not a["is_pr"] and not b["is_pr"]:
# Both issues — close the newer one
try:
a_dt = datetime.fromisoformat(a["created_at"].replace("Z", "+00:00"))
b_dt = datetime.fromisoformat(b["created_at"].replace("Z", "+00:00"))
newer = b if b_dt > a_dt else a
except (ValueError, TypeError):
newer = b
return f"Close #{newer['number']} as duplicate"
# One issue, one PR
return "Link PR to issue"
def generate_report(
items: list[TriageItem],
outlier_results: list[tuple[int, float]],
duplicate_pairs: list[tuple[int, int, float]],
label_suggestions: list[list[tuple[str, float]]] | None = None,
) -> str:
"""Generate a structured markdown triage report."""
now = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
repo = os.environ.get("GITHUB_REPOSITORY", "unknown/repo")
# Compute label suggestion counts early for the health table
outlier_set = {idx for idx, _ in outlier_results}
suggested_count = 0
if label_suggestions is not None:
suggested_count = sum(
1 for i, s in enumerate(label_suggestions)
if s and not items[i]["labels"] and i not in outlier_set
)
# ── Health summary table at the top ──────────────────────────────
lines: list[str] = [
"## Triage Sweep Report",
"",
f"**Run:** {now} UTC",
f"**Items analyzed:** {len(items)}",
f"**Thresholds:** IQR multiplier {IQR_MULTIPLIER}, Cosine > {COSINE_THRESHOLD}",
"",
"### Health Summary",
"",
"| Metric | Value |",
"|--------|-------|",
f"| Items analyzed | {len(items)} |",
f"| Outliers flagged | {len(outlier_results)} |",
f"| Duplicate pairs | {len(duplicate_pairs)} |",
f"| Label suggestions | {suggested_count} |",
"",
]
# ── Outlier section ──────────────────────────────────────────────
# Determine cutoff for high-confidence split
cutoff = getattr(outlier_results, "cutoff", 0.0)
high_conf_cutoff = 2 * cutoff if cutoff > 0 else float("inf")
high_conf = [(idx, d) for idx, d in outlier_results if d > high_conf_cutoff]
borderline = [(idx, d) for idx, d in outlier_results if d <= high_conf_cutoff]
lines.extend([
f"### Potential Outliers / Spam ({len(outlier_results)})",
"",
"Items with unusually high Mahalanobis distance from the distribution center.",
"These may be spam, off-topic, or poorly described.",
"",
])
if high_conf:
lines.append(f"**High Confidence** ({len(high_conf)} items, distance > 2x cutoff)")
lines.append("")
lines.append("| # | Type | Title | Distance | Age |")
lines.append("|---|------|-------|----------|-----|")
for idx, distance in high_conf:
item = items[idx]
kind = "PR" if item["is_pr"] else "Issue"
age = _item_age(item["created_at"])
title = item["title"][:80] + ("..." if len(item["title"]) > 80 else "")
lines.append(
f"| [#{item['number']}]({item['html_url']}) "
f"| {kind} | {title} | {distance:.2f} | {age} |"
)
lines.append("")
if borderline:
lines.append("<details>")
lines.append(f"<summary>Borderline ({len(borderline)} items)</summary>")
lines.append("")
lines.append("| # | Type | Title | Distance | Age |")
lines.append("|---|------|-------|----------|-----|")
for idx, distance in borderline:
item = items[idx]
kind = "PR" if item["is_pr"] else "Issue"
age = _item_age(item["created_at"])
title = item["title"][:80] + ("..." if len(item["title"]) > 80 else "")
lines.append(
f"| [#{item['number']}]({item['html_url']}) "
f"| {kind} | {title} | {distance:.2f} | {age} |"
)
lines.append("")
lines.append("</details>")
lines.append("")
if not outlier_results:
lines.append("None found.")
# ── Duplicate pairs section ──────────────────────────────────────
lines.extend([
"",
f"### Potential Duplicates ({len(duplicate_pairs)} pairs)",
"",
"Pairs of items with cosine similarity above the threshold.",
"",
])
if duplicate_pairs:
lines.append("| Item A | Item B | Similarity | Suggested Action |")
lines.append("|--------|--------|------------|------------------|")
for i, j, sim in duplicate_pairs:
a = items[i]
b = items[j]
kind_a = "PR" if a["is_pr"] else "Issue"
kind_b = "PR" if b["is_pr"] else "Issue"
action = _suggested_action(a, b)
lines.append(
f"| [#{a['number']}]({a['html_url']}) {kind_a}: {a['title']} "
f"| [#{b['number']}]({b['html_url']}) {kind_b}: {b['title']} "
f"| {sim:.3f} | {action} |"
)
else:
lines.append("None found.")
# ── Label suggestions section ────────────────────────────────────
if label_suggestions is not None:
# High confidence: top-1 label with raw_sim >= 0.5
# Low confidence: top-1 label with raw_sim < 0.5
high_conf_labels: list[tuple[int, list[tuple[str, float]]]] = []
low_conf_labels: list[tuple[int, list[tuple[str, float]]]] = []
for i, sugs in enumerate(label_suggestions):
if sugs and not items[i]["labels"] and i not in outlier_set:
top1 = sugs[:1]
if top1[0][1] >= 0.5:
high_conf_labels.append((i, top1))
else:
low_conf_labels.append((i, top1))
total_suggestions = len(high_conf_labels) + len(low_conf_labels)
lines.extend([
"",
f"### Suggested Labels ({total_suggestions} unlabeled items)",
"",
"Labels suggested by z-score normalized embedding similarity against repo label descriptions.",
"Only shown for unlabeled items that were not flagged as outliers.",
"",
])
# Label concentration warning
if total_suggestions > 0:
label_counts: dict[str, int] = {}
for _, sugs in high_conf_labels + low_conf_labels:
for name, _ in sugs:
label_counts[name] = label_counts.get(name, 0) + 1
for name, count in label_counts.items():
if count > total_suggestions * 0.5:
lines.append(
f"> **Warning:** Label `{name}` accounts for "
f"{count}/{total_suggestions} suggestions "
f"({count * 100 // total_suggestions}%). "
f"Consider reviewing label descriptions for specificity."
)
lines.append("")
if high_conf_labels:
lines.append("| # | Type | Title | Suggested Label |")
lines.append("|---|------|-------|--------------------|")
for idx, sugs in high_conf_labels:
item = items[idx]
kind = "PR" if item["is_pr"] else "Issue"
label_strs = [f"`{name}` ({score:.2f})" for name, score in sugs]
lines.append(
f"| [#{item['number']}]({item['html_url']}) "
f"| {kind} | {item['title']} | {', '.join(label_strs)} |"
)
if low_conf_labels:
lines.append("")
lines.append("<details>")
lines.append(f"<summary>Low-confidence suggestions ({len(low_conf_labels)} items)</summary>")
lines.append("")
lines.append("| # | Type | Title | Suggested Label |")
lines.append("|---|------|-------|--------------------|")
for idx, sugs in low_conf_labels:
item = items[idx]
kind = "PR" if item["is_pr"] else "Issue"
label_strs = [f"`{name}` ({score:.2f})" for name, score in sugs]
lines.append(
f"| [#{item['number']}]({item['html_url']}) "
f"| {kind} | {item['title']} | {', '.join(label_strs)} |"
)
lines.append("")
lines.append("</details>")
if not high_conf_labels and not low_conf_labels:
lines.append("No unlabeled items need suggestions.")
lines.extend([
"",
"### Summary",
"",
f"- {len(outlier_results)} outliers flagged for review",
f"- {len(duplicate_pairs)} duplicate pairs found",
f"- {len(items)} items analyzed in total",
])
if label_suggestions is not None:
lines.append(f"- {suggested_count} items suggested for labeling")
lines.extend([
"",
"---",
f"*Generated by [triage-sweep](https://github.com/{repo}/actions) — no LLM was used.*",
])
return "\n".join(lines)
def create_report_issue(report_body: str) -> None:
"""Create a GitHub issue with the triage report.
Posts to the issues API with the triage-report label.
Raises SystemExit on non-201 response.
"""
token = os.environ["GITHUB_TOKEN"]
repo = os.environ["GITHUB_REPOSITORY"]
url = f"https://api.github.com/repos/{repo}/issues"
today = datetime.now(timezone.utc).strftime("%Y-%m-%d")
payload = json.dumps({
"title": f"Triage Sweep Report — {today}",
"body": report_body,
"labels": [REPORT_LABEL],
}).encode("utf-8")
req = urllib.request.Request(url, data=payload, method="POST")
req.add_header("Accept", "application/vnd.github+json")
req.add_header("Authorization", f"Bearer {token}")
req.add_header("X-GitHub-Api-Version", "2022-11-28")
req.add_header("Content-Type", "application/json")
try:
with urllib.request.urlopen(req, timeout=30) as resp:
resp_body = resp.read().decode("utf-8")
if resp.status != 201:
print(f"::error::Failed to create issue: {resp.status} {resp_body}")
sys.exit(1)
result = json.loads(resp_body)
print(f"Created issue: {result.get('html_url', 'unknown')}")
except urllib.error.HTTPError as e:
body = e.read().decode("utf-8", errors="replace")
print(f"::error::Failed to create issue: {e.code} {body}")
sys.exit(1)
def write_report(report: str) -> None:
"""Write the report to the file system for the summary step."""
with open(REPORT_FILE, "w", encoding="utf-8") as f:
f.write(report)
def main() -> None:
"""Orchestrate the full triage sweep."""
# 1. Validate environment
for var in ("GITHUB_TOKEN", "GITHUB_REPOSITORY"):
if not os.environ.get(var):
print(f"::error::Missing required environment variable: {var}")
sys.exit(1)
# 2. Fetch all open issues + PRs
items = fetch_all_open_items()
print(f"Fetched {len(items)} open items")
if len(items) == 0:
report = "## Triage Sweep Report\n\nNo open issues or PRs found."
write_report(report)
print("No items to analyze.")
return
# 3. Extract texts for embedding
texts: list[str] = [item["text"] for item in items]
# 4. Embed all texts (returns numpy float32 array of shape [n, 384])
embeddings = embed_texts(texts)
# 5. L2-normalize
embeddings = normalize_rows(embeddings)
# 6. Outlier detection (Mahalanobis via EllipticEnvelope)
outlier_results: list[tuple[int, float]] = []
if len(items) >= MIN_SAMPLES_FOR_OUTLIER_DETECTION:
reduced = reduce_dimensions(embeddings, PCA_MAX_COMPONENTS)
outlier_results = detect_outliers(
reduced,
contamination=CONTAMINATION,
iqr_multiplier=IQR_MULTIPLIER,
max_outlier_pct=MAX_OUTLIER_PCT,
)
else:
print(
f"Skipping outlier detection: {len(items)} items < "
f"{MIN_SAMPLES_FOR_OUTLIER_DETECTION} minimum"
)
# 7. Duplicate detection (pairwise cosine similarity)
duplicate_pairs = find_duplicate_pairs(embeddings, COSINE_THRESHOLD)
# 8. Label suggestion via embedding similarity
label_suggestions: list[list[tuple[str, float]]] | None = None
repo_labels = fetch_repo_labels()
if repo_labels:
label_texts = [lbl["text"] for lbl in repo_labels]
label_names = [lbl["name"] for lbl in repo_labels]
label_embeddings = embed_texts(label_texts)
label_embeddings = normalize_rows(label_embeddings)
label_suggestions = suggest_labels(embeddings, label_embeddings, label_names)
print(f"Computed label suggestions against {len(repo_labels)} repo labels")
# NOTE: Auto-labeling is disabled. The report shows suggestions for
# human review. To re-enable, uncomment the block below.
#
# # Apply top label to unlabeled items (unless dry run)
# # Skip outliers — flagged items shouldn't get categorized
# outlier_set = {idx for idx, _ in outlier_results}
# if not DRY_RUN:
# applied_count = 0
# for i, sugs in enumerate(label_suggestions):
# if sugs and not items[i]["labels"] and i not in outlier_set:
# # Apply only the top-1 label (highest confidence)
# apply_labels_to_item(items[i]["number"], [sugs[0][0]])
# applied_count += 1
# print(f"Applied labels to {applied_count} unlabeled items")
else:
print("No repo labels found — skipping label suggestions")
# 9. Generate report
report = generate_report(items, outlier_results, duplicate_pairs, label_suggestions)
# 10. Write report to file (for summary step)
write_report(report)
# 11. Create report issue (unless dry run)
if DRY_RUN:
print("Dry run — skipping issue creation and label application.")
print(report)
else:
create_report_issue(report)
print("Report issue created.")
if __name__ == "__main__":
main()
@@ -0,0 +1,468 @@
"""Tests for embedding_utils.py — all embedding model calls are mocked."""
from __future__ import annotations
import sys
from unittest.mock import patch, MagicMock
import numpy as np
import pytest
# Mock fastembed before importing the module under test (persistent)
if "fastembed" not in sys.modules:
sys.modules["fastembed"] = MagicMock()
from embedding_utils import (
embed_texts,
normalize_rows,
reduce_dimensions,
detect_outliers,
find_duplicate_pairs,
suggest_labels,
EMBEDDING_DIM,
EMBEDDING_MODEL,
EMBEDDING_BATCH_SIZE,
LABEL_Z_THRESHOLD,
LABEL_Z_MARGIN,
LABEL_Z_STD_FLOOR,
MIN_RAW_SIMILARITY,
MAX_LABELS_PER_ITEM,
)
class TestEmbedTexts:
"""Tests for the embed_texts function."""
def test_empty_list_returns_empty_array(self):
result = embed_texts([])
assert result.shape == (0, EMBEDDING_DIM)
assert result.dtype == np.float32
@patch("embedding_utils.TextEmbedding")
def test_single_text(self, mock_cls):
mock_model = MagicMock()
mock_cls.return_value = mock_model
vec = np.random.randn(EMBEDDING_DIM).astype(np.float32)
mock_model.embed.return_value = iter([vec])
result = embed_texts(["hello world"])
mock_cls.assert_called_once_with(model_name=EMBEDDING_MODEL)
mock_model.embed.assert_called_once_with(
["hello world"], batch_size=EMBEDDING_BATCH_SIZE
)
assert result.shape == (1, EMBEDDING_DIM)
assert result.dtype == np.float32
np.testing.assert_array_almost_equal(result[0], vec)
@patch("embedding_utils.TextEmbedding")
def test_multiple_texts(self, mock_cls):
mock_model = MagicMock()
mock_cls.return_value = mock_model
vecs = [
np.random.randn(EMBEDDING_DIM).astype(np.float32)
for _ in range(5)
]
mock_model.embed.return_value = iter(vecs)
result = embed_texts(["a", "b", "c", "d", "e"])
assert result.shape == (5, EMBEDDING_DIM)
assert result.dtype == np.float32
class TestNormalizeRows:
"""Tests for L2 row normalization."""
def test_empty_matrix(self):
m = np.empty((0, 10), dtype=np.float32)
result = normalize_rows(m)
assert result.shape == (0, 10)
def test_single_row(self):
m = np.array([[3.0, 4.0]], dtype=np.float32)
result = normalize_rows(m)
# Norm should be ~1.0
norm = np.linalg.norm(result[0])
assert abs(norm - 1.0) < 1e-5
def test_multiple_rows(self):
rng = np.random.default_rng(42)
m = rng.standard_normal((10, 50)).astype(np.float32)
result = normalize_rows(m)
norms = np.linalg.norm(result, axis=1)
np.testing.assert_allclose(norms, 1.0, atol=1e-5)
def test_zero_row_stays_near_zero(self):
m = np.array([[0.0, 0.0, 0.0], [1.0, 0.0, 0.0]], dtype=np.float32)
result = normalize_rows(m)
# Zero row divided by eps -> very small values
assert np.linalg.norm(result[0]) < 1e-3
# Non-zero row should be unit norm
assert abs(np.linalg.norm(result[1]) - 1.0) < 1e-5
def test_preserves_direction(self):
m = np.array([[2.0, 0.0], [0.0, 3.0]], dtype=np.float32)
result = normalize_rows(m)
np.testing.assert_allclose(result[0], [1.0, 0.0], atol=1e-5)
np.testing.assert_allclose(result[1], [0.0, 1.0], atol=1e-5)
class TestReduceDimensions:
"""Tests for PCA dimensionality reduction."""
def test_single_sample_returns_unchanged(self):
m = np.random.randn(1, 50).astype(np.float32)
result = reduce_dimensions(m, 10)
np.testing.assert_array_equal(result, m)
def test_reduces_dimensions(self):
rng = np.random.default_rng(42)
m = rng.standard_normal((100, 50)).astype(np.float32)
result = reduce_dimensions(m, 10)
assert result.shape == (100, 10)
assert result.dtype == np.float32
def test_caps_at_n_minus_1(self):
rng = np.random.default_rng(42)
# 5 samples, 20 features -> max components = 4 (n-1)
m = rng.standard_normal((5, 20)).astype(np.float32)
result = reduce_dimensions(m, 50)
assert result.shape == (5, 4)
def test_caps_at_d(self):
rng = np.random.default_rng(42)
# 100 samples, 3 features -> max components = 3
m = rng.standard_normal((100, 3)).astype(np.float32)
result = reduce_dimensions(m, 50)
assert result.shape == (100, 3)
def test_max_components_respected(self):
rng = np.random.default_rng(42)
m = rng.standard_normal((50, 30)).astype(np.float32)
result = reduce_dimensions(m, 5)
assert result.shape[1] == 5
class TestDetectOutliers:
"""Tests for IQR-based outlier detection."""
def test_single_sample_returns_empty(self):
m = np.random.randn(1, 5).astype(np.float32)
result = detect_outliers(m)
assert result == []
def test_empty_returns_empty(self):
# n < 2 case
m = np.empty((0, 5), dtype=np.float32)
result = detect_outliers(m)
assert result == []
def test_finds_outliers_in_synthetic_data(self):
rng = np.random.default_rng(42)
# Create a tight cluster with one obvious outlier
cluster = rng.standard_normal((50, 3)).astype(np.float32) * 0.1
outlier = np.array([[100.0, 100.0, 100.0]], dtype=np.float32)
m = np.vstack([cluster, outlier])
result = detect_outliers(m)
# The outlier (index 50) should be detected
outlier_indices = [idx for idx, _ in result]
assert 50 in outlier_indices
def test_returns_list_of_index_distance_tuples(self):
rng = np.random.default_rng(42)
# Tight cluster + outlier to guarantee at least one result
cluster = rng.standard_normal((20, 3)).astype(np.float32) * 0.1
far_point = np.array([[50.0, 50.0, 50.0]], dtype=np.float32)
m = np.vstack([cluster, far_point])
result = detect_outliers(m)
assert isinstance(result, list)
for item in result:
assert isinstance(item, tuple)
assert len(item) == 2
idx, dist = item
assert isinstance(idx, int)
assert isinstance(dist, float)
assert dist > 0
def test_iqr_cutoff_behavior(self):
"""Lower IQR multiplier should flag more items than higher multiplier."""
rng = np.random.default_rng(42)
m = rng.standard_normal((100, 3)).astype(np.float32)
low = detect_outliers(m, iqr_multiplier=1.0, max_outlier_pct=0.5)
high = detect_outliers(m, iqr_multiplier=5.0, max_outlier_pct=0.5)
assert len(low) >= len(high)
def test_dimension_aware_no_mass_flagging(self):
"""High-dimensional clean Gaussian data should not flag everything."""
rng = np.random.default_rng(42)
# 500 samples, 10 dims — well-conditioned for robust covariance
m = rng.standard_normal((500, 10)).astype(np.float32)
result = detect_outliers(m)
# With IQR-based cutoff on clean Gaussian data,
# only a small fraction should be flagged (well under 50%)
assert len(result) < 250
def test_contamination_parameter(self):
rng = np.random.default_rng(42)
m = rng.standard_normal((50, 3)).astype(np.float32)
# Should not raise with different contamination values
result = detect_outliers(m, contamination=0.05)
assert isinstance(result, list)
def test_max_outlier_pct_hard_cap(self):
"""The hard cap should limit outlier count to max_outlier_pct * n."""
rng = np.random.default_rng(42)
# Create data with many potential outliers (bimodal)
cluster = rng.standard_normal((80, 3)).astype(np.float32) * 0.1
outliers = rng.standard_normal((20, 3)).astype(np.float32) * 50.0
m = np.vstack([cluster, outliers])
# Very low IQR multiplier to flag a lot, but cap at 5%
result = detect_outliers(m, iqr_multiplier=0.5, max_outlier_pct=0.05)
max_allowed = max(1, int(0.05 * 100)) # 5
assert len(result) <= max_allowed
def test_hard_cap_keeps_most_extreme(self):
"""When capped, the most extreme items (highest distance) should be kept."""
rng = np.random.default_rng(42)
cluster = rng.standard_normal((90, 3)).astype(np.float32) * 0.1
# Create outliers with increasing extremity
outliers = np.array([
[10.0, 10.0, 10.0],
[20.0, 20.0, 20.0],
[50.0, 50.0, 50.0],
], dtype=np.float32)
m = np.vstack([cluster, outliers])
# Cap at ~1 item (0.01 * 93 = 0, but min is 1)
result = detect_outliers(m, iqr_multiplier=0.5, max_outlier_pct=0.02)
if len(result) > 0:
# The most extreme (index 92, distance for [50,50,50]) should be kept
indices = [idx for idx, _ in result]
assert 92 in indices
def test_cutoff_attribute(self):
"""Returned result should carry a cutoff attribute."""
rng = np.random.default_rng(42)
m = rng.standard_normal((50, 3)).astype(np.float32)
result = detect_outliers(m)
assert hasattr(result, "cutoff")
assert isinstance(result.cutoff, float)
assert result.cutoff > 0
class TestFindDuplicatePairs:
"""Tests for cosine similarity duplicate detection."""
def test_single_item_returns_empty(self):
m = np.random.randn(1, 10).astype(np.float32)
result = find_duplicate_pairs(m, 0.9)
assert result == []
def test_empty_returns_empty(self):
m = np.empty((0, 10), dtype=np.float32)
result = find_duplicate_pairs(m, 0.9)
assert result == []
def test_identical_vectors_detected(self):
vec = np.random.randn(10).astype(np.float32)
vec = vec / np.linalg.norm(vec)
m = np.vstack([vec, vec, np.random.randn(10).astype(np.float32)])
result = find_duplicate_pairs(m, 0.99)
# Items 0 and 1 are identical, should be found
assert any(i == 0 and j == 1 for i, j, _ in result)
def test_orthogonal_vectors_not_detected(self):
m = np.eye(5, dtype=np.float32)
result = find_duplicate_pairs(m, 0.5)
assert result == []
def test_returns_correct_format(self):
vec = np.random.randn(10).astype(np.float32)
vec = vec / np.linalg.norm(vec)
m = np.vstack([vec, vec])
result = find_duplicate_pairs(m, 0.5)
assert len(result) >= 1
for item in result:
assert len(item) == 3
i, j, sim = item
assert isinstance(i, int)
assert isinstance(j, int)
assert isinstance(sim, float)
assert i < j
def test_i_less_than_j(self):
rng = np.random.default_rng(42)
# Create some similar vectors
base = rng.standard_normal(10).astype(np.float32)
m = np.vstack([base + rng.standard_normal(10) * 0.01 for _ in range(5)])
result = find_duplicate_pairs(m, 0.5)
for i, j, _ in result:
assert i < j
def test_high_threshold_fewer_pairs(self):
rng = np.random.default_rng(42)
m = rng.standard_normal((10, 20)).astype(np.float32)
# Normalize for meaningful cosine similarities
norms = np.linalg.norm(m, axis=1, keepdims=True)
m = m / norms
low = find_duplicate_pairs(m, 0.3)
high = find_duplicate_pairs(m, 0.9)
assert len(low) >= len(high)
class TestSuggestLabels:
"""Tests for z-score normalized label suggestion."""
def test_empty_items_returns_empty_lists(self):
items = np.empty((0, 10), dtype=np.float32)
labels = np.random.randn(3, 10).astype(np.float32)
result = suggest_labels(items, labels, ["a", "b", "c"])
assert result == []
def test_empty_labels_returns_empty_per_item(self):
items = np.random.randn(5, 10).astype(np.float32)
labels = np.empty((0, 10), dtype=np.float32)
result = suggest_labels(items, labels, [])
assert len(result) == 5
assert all(s == [] for s in result)
def test_identical_embedding_gets_that_label(self):
"""If an item embedding strongly matches one label, z-score should highlight it."""
# Create multiple items so z-score normalization is meaningful
rng = np.random.default_rng(42)
# 10 random items + 1 item that matches label "bug" exactly
random_items = rng.standard_normal((10, 3)).astype(np.float32)
bug_vec = np.array([[1.0, 0.0, 0.0]], dtype=np.float32)
items = np.vstack([random_items, bug_vec])
labels = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0], [0.0, 0.0, 1.0]], dtype=np.float32)
result = suggest_labels(
items, labels, ["bug", "feature", "docs"],
z_threshold=0.5, z_margin=0.0, min_raw_sim=0.1,
)
# The last item (matching bug_vec) should get "bug" as top suggestion
last_item_sugs = result[-1]
if last_item_sugs:
assert last_item_sugs[0][0] == "bug"
def test_z_score_suppresses_dominant_label(self):
"""When all items are similar to one label, z-scores should be low
(none stands out) and that label should not be blindly suggested."""
# All items identical — z-score for every item on every label is 0
items = np.ones((10, 3), dtype=np.float32)
labels = np.array([[1.0, 1.0, 1.0], [0.0, 1.0, 0.0]], dtype=np.float32)
result = suggest_labels(
items, labels, ["catch-all", "specific"],
z_threshold=1.5, min_raw_sim=0.3,
)
# With identical items, std=0 -> z-scores are all 0 -> nothing passes z_threshold
for sugs in result:
assert sugs == []
def test_margin_gate_blocks_top1(self):
"""Top-1 label must beat #2 by z_margin to be accepted as position 0."""
rng = np.random.default_rng(99)
# 20 items, each slightly different, 2 labels
items = rng.standard_normal((20, 5)).astype(np.float32)
# Two labels that are nearly identical -> margin gate should block top-1
labels = np.array([[1.0, 0.5, 0.0, 0.0, 0.0],
[1.0, 0.5, 0.01, 0.0, 0.0]], dtype=np.float32)
result = suggest_labels(
items, labels, ["label-a", "label-b"],
z_threshold=0.0, z_margin=10.0, min_raw_sim=0.0, max_per_item=1,
)
# With a huge margin requirement and max_per_item=1, nothing should pass
# because the only candidate (top-1) is blocked by margin gate,
# and max_per_item=1 prevents falling through to position 2
for sugs in result:
assert sugs == []
def test_margin_gate_passes_when_clear_winner(self):
"""When top-1 clearly beats #2, it should pass the margin gate."""
# Create items where one strongly matches label 0 vs label 1
items = np.array([
[1.0, 0.0, 0.0, 0.0, 0.0], # strongly matches label-a
[0.0, 0.0, 0.0, 0.0, 1.0], # matches neither well
] * 5, dtype=np.float32) # 10 items for stable z-scores
labels = np.array([
[1.0, 0.0, 0.0, 0.0, 0.0], # label-a
[0.0, 1.0, 0.0, 0.0, 0.0], # label-b (orthogonal)
], dtype=np.float32)
result = suggest_labels(
items, labels, ["label-a", "label-b"],
z_threshold=0.5, z_margin=0.3, min_raw_sim=0.1,
)
# Items matching label-a should get it suggested (clear z-score advantage)
got_label_a = sum(1 for sugs in result if sugs and sugs[0][0] == "label-a")
assert got_label_a > 0
def test_min_raw_similarity_filter(self):
"""Even with high z-score, low raw similarity should be filtered out."""
# Items are orthogonal to all labels -> raw similarity near 0
items = np.array([[1.0, 0.0, 0.0]], dtype=np.float32)
labels = np.array([[0.0, 0.0, 1.0]], dtype=np.float32)
result = suggest_labels(
items, labels, ["irrelevant"],
z_threshold=0.0, z_margin=0.0, min_raw_sim=0.9,
)
# Raw similarity is ~0, which is below min_raw_sim=0.9
assert result[0] == []
def test_max_per_item_respected(self):
"""Even if many labels qualify, max_per_item caps the results."""
rng = np.random.default_rng(42)
# Create items with some variance so z-scores differentiate
items = rng.standard_normal((20, 10)).astype(np.float32)
base = items[0]
# All labels very similar to item 0
labels = np.array([base + rng.standard_normal(10) * 0.01 for _ in range(10)])
names = [f"label-{i}" for i in range(10)]
result = suggest_labels(
items, labels, names,
z_threshold=0.0, z_margin=0.0, min_raw_sim=0.0, max_per_item=2,
)
for sugs in result:
assert len(sugs) <= 2
def test_returns_raw_similarity_not_z_score(self):
"""Returned scores should be raw cosine similarity, not z-scores."""
rng = np.random.default_rng(42)
items = rng.standard_normal((15, 5)).astype(np.float32)
labels = rng.standard_normal((3, 5)).astype(np.float32)
names = ["bug", "feature", "docs"]
result = suggest_labels(
items, labels, names,
z_threshold=0.0, z_margin=0.0, min_raw_sim=-1.0,
)
# Raw cosine similarity should be in [-1, 1] range
for sugs in result:
for name, score in sugs:
assert -1.0 <= score <= 1.0 + 1e-5
assert isinstance(name, str)
assert isinstance(score, float)
def test_returns_correct_format(self):
rng = np.random.default_rng(42)
items = rng.standard_normal((3, 10)).astype(np.float32)
labels = rng.standard_normal((5, 10)).astype(np.float32)
names = ["bug", "feature", "docs", "ci", "test"]
result = suggest_labels(
items, labels, names,
z_threshold=0.0, z_margin=0.0, min_raw_sim=-1.0,
)
assert len(result) == 3
for sugs in result:
for name, score in sugs:
assert isinstance(name, str)
assert isinstance(score, float)
assert name in names
def test_text_truncation_in_labels(self):
"""Label names should be returned as-is even when very long."""
rng = np.random.default_rng(42)
items = rng.standard_normal((10, 5)).astype(np.float32)
long_name = "a" * 200
labels = rng.standard_normal((1, 5)).astype(np.float32)
result = suggest_labels(
items, labels, [long_name],
z_threshold=0.0, z_margin=0.0, min_raw_sim=-1.0,
)
for sugs in result:
if sugs:
assert sugs[0][0] == long_name
+873
View File
@@ -0,0 +1,873 @@
"""Tests for sweep.py — all external calls (API, embedding) are mocked."""
from __future__ import annotations
import json
import os
import sys
from io import BytesIO
from unittest.mock import patch, MagicMock, mock_open
from urllib.error import HTTPError
import numpy as np
import pytest
# Mock fastembed before importing sweep (which imports embedding_utils)
sys.modules["fastembed"] = MagicMock()
# Set required env vars before importing sweep (module-level constants read env)
os.environ.setdefault("GITHUB_TOKEN", "test-token")
os.environ.setdefault("GITHUB_REPOSITORY", "owner/repo")
from sweep import (
github_api_get,
fetch_all_open_items,
fetch_repo_labels,
apply_labels_to_item,
generate_report,
create_report_issue,
write_report,
main,
TriageItem,
RepoLabel,
REPORT_FILE,
REPORT_LABEL,
API_PAGE_SIZE,
MIN_SAMPLES_FOR_OUTLIER_DETECTION,
PCA_MAX_COMPONENTS,
MAX_EMBED_CHARS,
IQR_MULTIPLIER,
MAX_OUTLIER_PCT,
_item_age,
_suggested_action,
)
def _make_api_issue(number: int, title: str = "Test issue", is_pr: bool = False,
body: str = "Issue body", labels: list[str] | None = None,
created_at: str = "2026-03-21T00:00:00Z") -> dict:
"""Helper to build a mock GitHub API issue response object."""
result: dict = {
"number": number,
"title": title,
"html_url": f"https://github.com/owner/repo/issues/{number}",
"body": body,
"created_at": created_at,
"labels": [{"name": lbl} for lbl in (labels or [])],
}
if is_pr:
result["pull_request"] = {"url": "..."}
return result
class TestGithubApiGet:
"""Tests for the github_api_get function."""
@patch("sweep.urllib.request.urlopen")
def test_successful_request(self, mock_urlopen):
mock_resp = MagicMock()
mock_resp.read.return_value = json.dumps([{"id": 1}]).encode()
mock_resp.__enter__ = lambda s: s
mock_resp.__exit__ = MagicMock(return_value=False)
mock_urlopen.return_value = mock_resp
result = github_api_get("/issues?state=open")
assert result == [{"id": 1}]
@patch("sweep.urllib.request.urlopen")
def test_http_error_exits(self, mock_urlopen):
error = HTTPError(
url="https://api.github.com/repos/owner/repo/issues",
code=403,
msg="Forbidden",
hdrs=None, # type: ignore[arg-type]
fp=BytesIO(b'{"message": "rate limited"}'),
)
mock_urlopen.side_effect = error
with pytest.raises(SystemExit) as exc_info:
github_api_get("/issues")
assert exc_info.value.code == 1
class TestConstants:
"""Tests for module-level constants."""
def test_min_samples_is_at_least_3x_pca_max(self):
"""MIN_SAMPLES must be >= 3 * PCA_MAX_COMPONENTS for reliable covariance."""
assert MIN_SAMPLES_FOR_OUTLIER_DETECTION >= 3 * PCA_MAX_COMPONENTS
def test_min_samples_is_100(self):
assert MIN_SAMPLES_FOR_OUTLIER_DETECTION == 100
def test_pca_max_components_is_20(self):
assert PCA_MAX_COMPONENTS == 20
def test_iqr_multiplier_default(self):
assert IQR_MULTIPLIER == 3.0
def test_max_outlier_pct_default(self):
assert MAX_OUTLIER_PCT == 0.05
class TestFetchAllOpenItems:
"""Tests for fetch_all_open_items."""
@patch("sweep.github_api_get")
def test_empty_repo(self, mock_get):
mock_get.return_value = []
items = fetch_all_open_items()
assert items == []
@patch("sweep.github_api_get")
def test_single_page(self, mock_get):
mock_get.return_value = [
_make_api_issue(1, "Bug report"),
_make_api_issue(2, "Feature request", is_pr=True),
]
items = fetch_all_open_items()
assert len(items) == 2
assert items[0]["number"] == 1
assert items[0]["is_pr"] is False
assert items[1]["is_pr"] is True
@patch("sweep.github_api_get")
def test_text_field_constructed(self, mock_get):
mock_get.return_value = [
_make_api_issue(1, "My Title", body="My Body"),
]
items = fetch_all_open_items()
assert items[0]["text"] == "My Title\n\nMy Body"
@patch("sweep.github_api_get")
def test_long_body_truncated(self, mock_get):
"""Bodies exceeding MAX_EMBED_CHARS are truncated to fit the token window."""
long_body = "x" * (MAX_EMBED_CHARS + 500)
mock_get.return_value = [
_make_api_issue(1, "Title", body=long_body),
]
items = fetch_all_open_items()
assert len(items[0]["text"]) == MAX_EMBED_CHARS
@patch("sweep.github_api_get")
def test_short_body_not_truncated(self, mock_get):
"""Bodies under the limit are left intact."""
mock_get.return_value = [
_make_api_issue(1, "Title", body="Short body"),
]
items = fetch_all_open_items()
assert items[0]["text"] == "Title\n\nShort body"
@patch("sweep.github_api_get")
def test_null_body_handled(self, mock_get):
issue = _make_api_issue(1, "No body")
issue["body"] = None
mock_get.return_value = [issue]
items = fetch_all_open_items()
assert items[0]["text"] == "No body\n\n"
@patch("sweep.github_api_get")
def test_labels_extracted(self, mock_get):
mock_get.return_value = [
_make_api_issue(1, "Labeled", labels=["bug", "high-priority"]),
]
items = fetch_all_open_items()
assert items[0]["labels"] == ["bug", "high-priority"]
@patch("sweep.MAX_ITEMS", 3)
@patch("sweep.github_api_get")
def test_max_items_cap(self, mock_get):
mock_get.return_value = [_make_api_issue(i) for i in range(100)]
items = fetch_all_open_items()
assert len(items) == 3
@patch("sweep.API_PAGE_SIZE", 2)
@patch("sweep.github_api_get")
def test_pagination(self, mock_get):
# First page: 2 items (full page), second page: 1 item (partial -> stop)
mock_get.side_effect = [
[_make_api_issue(1), _make_api_issue(2)],
[_make_api_issue(3)],
]
items = fetch_all_open_items()
assert len(items) == 3
assert mock_get.call_count == 2
class TestItemAge:
"""Tests for _item_age helper."""
def test_recent_item(self):
from datetime import datetime, timezone, timedelta
recent = (datetime.now(timezone.utc) - timedelta(hours=12)).isoformat()
assert _item_age(recent) == "<1d"
def test_days_old(self):
from datetime import datetime, timezone, timedelta
old = (datetime.now(timezone.utc) - timedelta(days=15)).isoformat()
assert _item_age(old) == "15d"
def test_months_old(self):
from datetime import datetime, timezone, timedelta
old = (datetime.now(timezone.utc) - timedelta(days=90)).isoformat()
assert _item_age(old) == "3mo"
def test_years_old(self):
from datetime import datetime, timezone, timedelta
old = (datetime.now(timezone.utc) - timedelta(days=400)).isoformat()
assert _item_age(old) == "1y"
def test_invalid_date(self):
assert _item_age("not-a-date") == "?"
class TestSuggestedAction:
"""Tests for _suggested_action helper."""
def test_both_issues_close_newer(self):
a = TriageItem(
number=1, title="A", html_url="u", is_pr=False, labels=[],
created_at="2026-01-01T00:00:00Z", text="t",
)
b = TriageItem(
number=2, title="B", html_url="u", is_pr=False, labels=[],
created_at="2026-02-01T00:00:00Z", text="t",
)
result = _suggested_action(a, b)
assert "Close #2 as duplicate" in result
def test_both_prs_review(self):
a = TriageItem(
number=1, title="A", html_url="u", is_pr=True, labels=[],
created_at="2026-01-01T00:00:00Z", text="t",
)
b = TriageItem(
number=2, title="B", html_url="u", is_pr=True, labels=[],
created_at="2026-01-01T00:00:00Z", text="t",
)
assert _suggested_action(a, b) == "Review for overlap"
def test_issue_pr_link(self):
a = TriageItem(
number=1, title="A", html_url="u", is_pr=False, labels=[],
created_at="2026-01-01T00:00:00Z", text="t",
)
b = TriageItem(
number=2, title="B", html_url="u", is_pr=True, labels=[],
created_at="2026-01-01T00:00:00Z", text="t",
)
assert _suggested_action(a, b) == "Link PR to issue"
class TestGenerateReport:
"""Tests for the markdown report generator."""
def test_no_findings(self):
items = [
TriageItem(
number=1, title="Test", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="Test",
),
]
report = generate_report(items, [], [])
assert "## Triage Sweep Report" in report
assert "Items analyzed:** 1" in report
assert "None found." in report
assert "0 outliers flagged" in report
assert "0 duplicate pairs found" in report
def test_health_summary_table(self):
items = [
TriageItem(
number=1, title="Test", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="Test",
),
]
report = generate_report(items, [], [])
assert "### Health Summary" in report
assert "| Metric | Value |" in report
assert "| Items analyzed | 1 |" in report
def test_iqr_multiplier_in_thresholds(self):
"""Report should show IQR multiplier, not percentile."""
items = [
TriageItem(
number=1, title="Test", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="Test",
),
]
report = generate_report(items, [], [])
assert "IQR multiplier" in report
assert "percentile" not in report.lower().split("thresholds")[0] # not in thresholds line
def test_with_outliers_shows_distance_and_age(self):
items = [
TriageItem(
number=10, title="Spam Issue", html_url="https://example.com/10",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
),
TriageItem(
number=20, title="Good Issue", html_url="https://example.com/20",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="good",
),
]
report = generate_report(items, [(0, 12.34)], [])
assert "#10" in report
assert "Spam Issue" in report
assert "12.34" in report
assert "1 outliers flagged" in report
# Age column should be present
assert "| Age |" in report
def test_outlier_borderline_in_details(self):
"""Borderline outliers should be in a <details> section."""
from embedding_utils import _OutlierResult
items = [
TriageItem(
number=10, title="Borderline", html_url="https://example.com/10",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
),
]
# Create outlier results with cutoff=10.0, distance=12.0 (< 2*cutoff=20)
outlier_results = _OutlierResult([(0, 12.0)])
outlier_results.cutoff = 10.0
report = generate_report(items, outlier_results, [])
assert "<details>" in report
assert "Borderline" in report
def test_outlier_high_confidence(self):
"""Items with distance > 2x cutoff should be in high confidence section."""
from embedding_utils import _OutlierResult
items = [
TriageItem(
number=10, title="Definite Spam", html_url="https://example.com/10",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
),
]
outlier_results = _OutlierResult([(0, 25.0)])
outlier_results.cutoff = 10.0
report = generate_report(items, outlier_results, [])
assert "High Confidence" in report
def test_with_duplicates_suggested_action(self):
items = [
TriageItem(
number=1, title="First", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="a",
),
TriageItem(
number=2, title="Second", html_url="https://example.com/2",
is_pr=True, labels=[], created_at="2026-02-01T00:00:00Z", text="b",
),
]
report = generate_report(items, [], [(0, 1, 0.954)])
assert "#1" in report
assert "#2" in report
assert "0.954" in report
assert "1 duplicate pairs found" in report
assert "Suggested Action" in report
assert "Link PR to issue" in report
def test_duplicate_both_issues_close_newer(self):
items = [
TriageItem(
number=1, title="First", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="a",
),
TriageItem(
number=2, title="Second", html_url="https://example.com/2",
is_pr=False, labels=[], created_at="2026-02-01T00:00:00Z", text="b",
),
]
report = generate_report(items, [], [(0, 1, 0.95)])
assert "Close #2 as duplicate" in report
def test_duplicate_both_prs_review(self):
items = [
TriageItem(
number=1, title="PR A", html_url="https://example.com/1",
is_pr=True, labels=[], created_at="2026-01-01T00:00:00Z", text="a",
),
TriageItem(
number=2, title="PR B", html_url="https://example.com/2",
is_pr=True, labels=[], created_at="2026-01-01T00:00:00Z", text="b",
),
]
report = generate_report(items, [], [(0, 1, 0.95)])
assert "Review for overlap" in report
def test_pr_type_label(self):
items = [
TriageItem(
number=5, title="PR Title", html_url="https://example.com/5",
is_pr=True, labels=[], created_at="2026-01-01T00:00:00Z", text="pr",
),
]
report = generate_report(items, [(0, 8.5)], [])
assert "| PR |" in report
def test_footer_present(self):
items = [
TriageItem(
number=1, title="T", html_url="u",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="t",
),
]
report = generate_report(items, [], [])
assert "no LLM was used" in report
class TestCreateReportIssue:
"""Tests for creating the report GitHub issue."""
@patch("sweep.urllib.request.urlopen")
def test_successful_creation(self, mock_urlopen):
mock_resp = MagicMock()
mock_resp.status = 201
mock_resp.read.return_value = json.dumps({
"html_url": "https://github.com/owner/repo/issues/99",
}).encode()
mock_resp.__enter__ = lambda s: s
mock_resp.__exit__ = MagicMock(return_value=False)
mock_urlopen.return_value = mock_resp
# Should not raise
create_report_issue("# Test Report")
@patch("sweep.urllib.request.urlopen")
def test_http_error_exits(self, mock_urlopen):
error = HTTPError(
url="https://api.github.com/repos/owner/repo/issues",
code=422,
msg="Unprocessable",
hdrs=None, # type: ignore[arg-type]
fp=BytesIO(b'{"message": "validation failed"}'),
)
mock_urlopen.side_effect = error
with pytest.raises(SystemExit) as exc_info:
create_report_issue("# Test Report")
assert exc_info.value.code == 1
class TestWriteReport:
"""Tests for the write_report helper."""
@patch("builtins.open", mock_open())
def test_writes_to_file(self):
write_report("# Report Content")
from builtins import open as builtin_open # noqa
# Verify open was called with the right path
from unittest.mock import call
open_mock = open # The patched version
open_mock.assert_called_once_with(REPORT_FILE, "w", encoding="utf-8") # type: ignore[attr-defined]
open_mock().write.assert_called_once_with("# Report Content") # type: ignore[attr-defined]
class TestFetchRepoLabels:
"""Tests for fetch_repo_labels."""
@patch("sweep.github_api_get")
def test_fetches_and_constructs_labels(self, mock_get):
mock_get.return_value = [
{"name": "bug", "description": "Something isn't working"},
{"name": "enhancement", "description": "New feature or request"},
{"name": "docs", "description": ""},
]
labels = fetch_repo_labels()
assert len(labels) == 3
assert labels[0]["name"] == "bug"
assert labels[0]["text"] == "bug: Something isn't working"
assert labels[2]["text"] == "docs" # no description, just name
@patch("sweep.github_api_get")
def test_empty_repo_labels(self, mock_get):
mock_get.return_value = []
labels = fetch_repo_labels()
assert labels == []
@patch("sweep.github_api_get")
def test_null_description_handled(self, mock_get):
mock_get.return_value = [
{"name": "wontfix", "description": None},
]
labels = fetch_repo_labels()
assert labels[0]["text"] == "wontfix"
@patch("sweep.API_PAGE_SIZE", 2)
@patch("sweep.github_api_get")
def test_label_pagination(self, mock_get):
"""Repos with more labels than one page should fetch all pages."""
mock_get.side_effect = [
# First page: full (2 items = API_PAGE_SIZE)
[
{"name": "bug", "description": "Broken"},
{"name": "feature", "description": "New"},
],
# Second page: partial (1 item < API_PAGE_SIZE) -> stop
[
{"name": "docs", "description": "Documentation"},
],
]
labels = fetch_repo_labels()
assert len(labels) == 3
assert mock_get.call_count == 2
assert labels[0]["name"] == "bug"
assert labels[2]["name"] == "docs"
class TestApplyLabelsToItem:
"""Tests for apply_labels_to_item."""
def test_empty_labels_skips(self):
# Should not make any API call
apply_labels_to_item(1, [])
@patch("sweep.urllib.request.urlopen")
def test_successful_label_application(self, mock_urlopen):
mock_resp = MagicMock()
mock_resp.read.return_value = b'[{"name": "bug"}]'
mock_resp.__enter__ = lambda s: s
mock_resp.__exit__ = MagicMock(return_value=False)
mock_urlopen.return_value = mock_resp
# Should not raise
apply_labels_to_item(42, ["bug", "enhancement"])
@patch("sweep.urllib.request.urlopen")
def test_http_error_is_non_fatal(self, mock_urlopen):
error = HTTPError(
url="https://api.github.com/repos/owner/repo/issues/1/labels",
code=404,
msg="Not Found",
hdrs=None, # type: ignore[arg-type]
fp=BytesIO(b'{"message": "not found"}'),
)
mock_urlopen.side_effect = error
# Should NOT raise — labeling failures are warnings, not fatal
apply_labels_to_item(1, ["bug"])
class TestGenerateReportWithLabels:
"""Tests for label suggestions in the report."""
def test_report_includes_label_section_high_confidence(self):
"""High-confidence label (raw_sim >= 0.5) should appear in main table."""
items = [
TriageItem(
number=1, title="Fix crash", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="crash",
),
]
suggestions = [[("bug", 0.85)]]
report = generate_report(items, [], [], label_suggestions=suggestions)
assert "Suggested Labels" in report
assert "`bug` (0.85)" in report
assert "1 items suggested for labeling" in report
def test_report_low_confidence_in_details(self):
"""Low-confidence label (raw_sim < 0.5) should be in <details> section."""
items = [
TriageItem(
number=1, title="Something", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="something",
),
]
suggestions = [[("maybe-bug", 0.35)]]
report = generate_report(items, [], [], label_suggestions=suggestions)
assert "Low-confidence suggestions" in report
assert "<details>" in report
assert "`maybe-bug` (0.35)" in report
def test_report_skips_already_labeled_items(self):
items = [
TriageItem(
number=1, title="Already labeled", html_url="https://example.com/1",
is_pr=False, labels=["bug"], created_at="2026-01-01T00:00:00Z", text="bug",
),
]
suggestions = [[("bug", 0.95)]]
report = generate_report(items, [], [], label_suggestions=suggestions)
assert "0 items suggested for labeling" in report
assert "No unlabeled items" in report
def test_report_excludes_outliers_from_suggestions(self):
items = [
TriageItem(
number=1, title="Spam garbage", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="spam",
),
TriageItem(
number=2, title="Real bug", html_url="https://example.com/2",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="bug",
),
]
suggestions = [[("bug", 0.85)], [("bug", 0.90)]]
# Item 0 is an outlier (with distance) — should be excluded from label suggestions
report = generate_report(items, [(0, 15.2)], [], label_suggestions=suggestions)
assert "1 unlabeled items" in report # only item 2
assert "#2" in report
# Item 0 (outlier) should NOT be in the suggestions table
assert "Spam garbage" not in report.split("Suggested Labels")[1]
def test_report_without_label_suggestions(self):
items = [
TriageItem(
number=1, title="T", html_url="u",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="t",
),
]
report = generate_report(items, [], [], label_suggestions=None)
assert "Suggested Labels" not in report
def test_label_concentration_warning(self):
"""When >50% of suggestions point to the same label, a warning should appear."""
items = [
TriageItem(
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
)
for i in range(4)
]
# 3 out of 4 items get "bug" label -> 75% concentration
suggestions = [
[("bug", 0.85)],
[("bug", 0.80)],
[("bug", 0.75)],
[("enhancement", 0.90)],
]
report = generate_report(items, [], [], label_suggestions=suggestions)
assert "Warning" in report
assert "`bug`" in report
assert "3/4" in report
class TestMain:
"""Tests for the main orchestration function."""
@patch.dict(os.environ, {"GITHUB_TOKEN": "", "GITHUB_REPOSITORY": "owner/repo"})
def test_missing_token_exits(self):
with pytest.raises(SystemExit) as exc_info:
main()
assert exc_info.value.code == 1
@patch.dict(os.environ, {"GITHUB_TOKEN": "tok", "GITHUB_REPOSITORY": ""})
def test_missing_repo_exits(self):
with pytest.raises(SystemExit) as exc_info:
main()
assert exc_info.value.code == 1
@patch("sweep.write_report")
@patch("sweep.fetch_all_open_items", return_value=[])
def test_no_items(self, mock_fetch, mock_write):
main()
mock_write.assert_called_once()
report = mock_write.call_args[0][0]
assert "No open issues or PRs found" in report
@patch("sweep.create_report_issue")
@patch("sweep.write_report")
@patch("sweep.suggest_labels", return_value=[])
@patch("sweep.find_duplicate_pairs", return_value=[])
@patch("sweep.detect_outliers", return_value=[])
@patch("sweep.reduce_dimensions")
@patch("sweep.normalize_rows")
@patch("sweep.embed_texts")
@patch("sweep.fetch_repo_labels")
@patch("sweep.fetch_all_open_items")
def test_full_flow_with_enough_items(
self, mock_fetch, mock_labels, mock_embed, mock_norm, mock_reduce,
mock_outliers, mock_dupes, mock_suggest, mock_write, mock_create,
):
"""Test the full flow with >= MIN_SAMPLES items (outlier detection runs)."""
n = MIN_SAMPLES_FOR_OUTLIER_DETECTION
items = [
TriageItem(
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
)
for i in range(n)
]
mock_fetch.return_value = items
mock_labels.return_value = [
RepoLabel(name="bug", description="Something broken", text="bug: Something broken"),
]
embeddings = np.random.randn(n, 384).astype(np.float32)
mock_embed.return_value = embeddings
mock_norm.return_value = embeddings
mock_reduce.return_value = np.random.randn(n, 10).astype(np.float32)
main()
mock_fetch.assert_called_once()
mock_labels.assert_called_once()
# embed_texts called twice: once for items, once for labels
assert mock_embed.call_count == 2
mock_norm.assert_called()
mock_reduce.assert_called_once()
mock_outliers.assert_called_once()
mock_dupes.assert_called_once()
mock_suggest.assert_called_once()
mock_write.assert_called_once()
mock_create.assert_called_once()
@patch("sweep.create_report_issue")
@patch("sweep.write_report")
@patch("sweep.suggest_labels", return_value=[])
@patch("sweep.find_duplicate_pairs", return_value=[])
@patch("sweep.detect_outliers")
@patch("sweep.reduce_dimensions")
@patch("sweep.normalize_rows")
@patch("sweep.embed_texts")
@patch("sweep.fetch_repo_labels", return_value=[])
@patch("sweep.fetch_all_open_items")
def test_skips_outlier_detection_for_few_items(
self, mock_fetch, mock_labels, mock_embed, mock_norm, mock_reduce,
mock_outliers, mock_dupes, mock_suggest, mock_write, mock_create,
):
"""With < MIN_SAMPLES items, outlier detection should be skipped."""
n = MIN_SAMPLES_FOR_OUTLIER_DETECTION - 1
items = [
TriageItem(
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
)
for i in range(n)
]
mock_fetch.return_value = items
embeddings = np.random.randn(n, 384).astype(np.float32)
mock_embed.return_value = embeddings
mock_norm.return_value = embeddings
main()
# Outlier detection should not have been called
mock_reduce.assert_not_called()
mock_outliers.assert_not_called()
# But duplicates should still be checked
mock_dupes.assert_called_once()
@patch.dict(os.environ, {"INPUT_DRY_RUN": "true"})
@patch("sweep.DRY_RUN", True)
@patch("sweep.write_report")
@patch("sweep.create_report_issue")
@patch("sweep.apply_labels_to_item")
@patch("sweep.suggest_labels", return_value=[[("bug", 0.85)]])
@patch("sweep.find_duplicate_pairs", return_value=[])
@patch("sweep.normalize_rows")
@patch("sweep.embed_texts")
@patch("sweep.fetch_repo_labels")
@patch("sweep.fetch_all_open_items")
def test_dry_run_skips_issue_creation_and_labeling(
self, mock_fetch, mock_labels, mock_embed, mock_norm,
mock_dupes, mock_suggest, mock_apply, mock_create, mock_write,
):
items = [
TriageItem(
number=1, title="Item", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="text",
)
]
mock_fetch.return_value = items
mock_labels.return_value = [
RepoLabel(name="bug", description="Broken", text="bug: Broken"),
]
embeddings = np.random.randn(1, 384).astype(np.float32)
mock_embed.return_value = embeddings
mock_norm.return_value = embeddings
main()
mock_create.assert_not_called()
mock_apply.assert_not_called()
mock_write.assert_called_once()
@patch("sweep.create_report_issue")
@patch("sweep.write_report")
@patch("sweep.apply_labels_to_item")
@patch("sweep.suggest_labels")
@patch("sweep.find_duplicate_pairs", return_value=[])
@patch("sweep.normalize_rows")
@patch("sweep.embed_texts")
@patch("sweep.fetch_repo_labels")
@patch("sweep.fetch_all_open_items")
def test_labels_not_auto_applied(
self, mock_fetch, mock_labels, mock_embed, mock_norm,
mock_dupes, mock_suggest, mock_apply, mock_write, mock_create,
):
"""Auto-labeling is disabled; labels should appear in report only."""
items = [
TriageItem(
number=1, title="Crash bug", html_url="https://example.com/1",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text="crash",
),
TriageItem(
number=2, title="Already labeled", html_url="https://example.com/2",
is_pr=False, labels=["enhancement"], created_at="2026-01-01T00:00:00Z", text="feat",
),
]
mock_fetch.return_value = items
mock_labels.return_value = [
RepoLabel(name="bug", description="Broken", text="bug: Broken"),
]
mock_suggest.return_value = [
[("bug", 0.90)],
[("bug", 0.45)],
]
embeddings = np.random.randn(2, 384).astype(np.float32)
mock_embed.return_value = embeddings
mock_norm.return_value = embeddings
main()
# Auto-labeling is disabled — apply_labels_to_item should never be called
mock_apply.assert_not_called()
@patch("sweep.create_report_issue")
@patch("sweep.write_report")
@patch("sweep.apply_labels_to_item")
@patch("sweep.suggest_labels")
@patch("sweep.find_duplicate_pairs", return_value=[])
@patch("sweep.detect_outliers")
@patch("sweep.reduce_dimensions")
@patch("sweep.normalize_rows")
@patch("sweep.embed_texts")
@patch("sweep.fetch_repo_labels")
@patch("sweep.fetch_all_open_items")
def test_outliers_excluded_from_report_suggestions(
self, mock_fetch, mock_labels, mock_embed, mock_norm, mock_reduce,
mock_outliers, mock_dupes, mock_suggest, mock_apply, mock_write, mock_create,
):
"""Items flagged as outliers should not appear in report label suggestions."""
n = MIN_SAMPLES_FOR_OUTLIER_DETECTION
items = [
TriageItem(
number=i, title=f"Item {i}", html_url=f"https://example.com/{i}",
is_pr=False, labels=[], created_at="2026-01-01T00:00:00Z", text=f"text {i}",
)
for i in range(n)
]
mock_fetch.return_value = items
mock_labels.return_value = [
RepoLabel(name="bug", description="Broken", text="bug: Broken"),
]
mock_outliers.return_value = [(0, 12.5), (5, 15.3)]
mock_suggest.return_value = [[("bug", 0.85)] for _ in range(n)]
embeddings = np.random.randn(n, 384).astype(np.float32)
mock_embed.return_value = embeddings
mock_norm.return_value = embeddings
mock_reduce.return_value = np.random.randn(n, 10).astype(np.float32)
main()
# Auto-labeling is disabled
mock_apply.assert_not_called()
# Report should still be generated (outliers excluded from suggestions in report)
mock_write.assert_called_once()
report = mock_write.call_args[0][0]
# Outlier items 0 and 5 should not appear in the label suggestions section
assert "Item 0" not in report.split("Suggested Labels")[1] if "Suggested Labels" in report else True
+83
View File
@@ -0,0 +1,83 @@
name: E2E Tests
on:
workflow_call:
jobs:
check-changes:
name: Check web module changes
runs-on: ubuntu-latest
timeout-minutes: 5
outputs:
web_changed: ${{ steps.filter.outputs.web }}
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: dorny/paths-filter@de90cc6fb38fc0963ad72b210f1f284cd68cea36 # v3
id: filter
with:
filters: |
web:
- 'gitnexus-web/**'
e2e:
name: e2e (chromium)
needs: check-changes
if: needs.check-changes.result == 'success' && needs.check-changes.outputs.web_changed == 'true'
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: ./.github/actions/setup-gitnexus-web
- name: Install Playwright browsers
run: npx playwright install --with-deps chromium
working-directory: gitnexus-web
- name: Install backend dependencies
run: npm ci
working-directory: gitnexus
- name: Build backend
run: npm run build
working-directory: gitnexus
- name: Analyze repository (index for backend)
run: |
node gitnexus/dist/cli/index.js analyze || true
if [ ! -d ".gitnexus" ]; then
echo "::error::No .gitnexus index created"
exit 1
fi
- name: Start backend server
run: node dist/cli/index.js serve &
working-directory: gitnexus
- name: Wait for backend readiness
run: npx wait-on http://localhost:4747/api/repos --timeout 30000
working-directory: gitnexus-web
- name: Start Vite dev server
run: npm run dev &
working-directory: gitnexus-web
- name: Wait for Vite dev server
run: npx wait-on http://localhost:5173 --timeout 30000
working-directory: gitnexus-web
- name: Run E2E tests
run: npx playwright test
working-directory: gitnexus-web
env:
E2E: '1'
- name: Upload test results
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: e2e-results
path: |
gitnexus-web/test-results/
gitnexus-web/playwright-report/
retention-days: 5
+49
View File
@@ -0,0 +1,49 @@
name: Quality Checks
on:
workflow_call:
jobs:
format:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 20
cache: npm
cache-dependency-path: package-lock.json
- run: npm ci
- run: npx prettier --check .
lint:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 20
cache: npm
cache-dependency-path: package-lock.json
- run: npm ci
- run: npx eslint .
typecheck:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: ./.github/actions/setup-gitnexus
- run: npx tsc --noEmit
working-directory: gitnexus
typecheck-web:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: ./.github/actions/setup-gitnexus-web
- run: npx tsc -b --noEmit
working-directory: gitnexus-web
+413
View File
@@ -0,0 +1,413 @@
name: CI Report
# Triggered after the CI workflow completes. Because workflow_run
# always runs code from the *default branch*, it receives a read/write
# GITHUB_TOKEN — even when the triggering PR comes from a fork.
on:
workflow_run:
workflows: ['CI']
types: [completed]
permissions:
actions: read # needed to list/download workflow run artifacts
contents: read # needed for sparse checkout of vitest.config.ts
pull-requests: write # needed to post sticky PR comment
jobs:
pr-report:
name: PR Report
# Only run for pull-request CI runs
if: >-
github.event.workflow_run.event == 'pull_request' &&
github.event.workflow_run.conclusion != 'cancelled'
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
# ── Download artifacts from the CI run ────────────────────────
- name: Download artifacts
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
with:
script: |
const fs = require('fs');
const path = require('path');
const runId = context.payload.workflow_run.id;
const allArtifacts = await github.rest.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: runId,
});
async function downloadArtifact(name, dest) {
const match = allArtifacts.data.artifacts.find(a => a.name === name);
if (!match) {
core.warning(`Artifact "${name}" not found`);
return false;
}
const zip = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: match.id,
archive_format: 'zip',
});
fs.mkdirSync(dest, { recursive: true });
fs.writeFileSync(path.join(dest, `${name}.zip`), Buffer.from(zip.data));
return true;
}
const temp = process.env.RUNNER_TEMP;
await downloadArtifact('pr-meta', path.join(temp, 'dl'));
await downloadArtifact('test-reports', path.join(temp, 'dl'));
- name: Extract artifacts
shell: bash
run: |
cd "$RUNNER_TEMP/dl"
# Extract each artifact into its own directory to avoid filename collisions
for z in *.zip; do
[ -f "$z" ] || continue
name="${z%.zip}"
mkdir -p "$RUNNER_TEMP/artifacts/$name"
unzip -o "$z" -d "$RUNNER_TEMP/artifacts/$name"
done
- name: Read PR metadata
id: meta
shell: bash
run: |
DIR="$RUNNER_TEMP/artifacts/pr-meta"
if [ ! -f "$DIR/pr_number" ]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "::warning::pr_number artifact missing — skipping report"
exit 0
fi
# Validate PR number is a positive integer (artifact comes from
# untrusted fork code, so treat contents defensively).
PR_NUM=$(cat "$DIR/pr_number" | tr -d '[:space:]')
if ! [[ "$PR_NUM" =~ ^[0-9]+$ ]]; then
echo "skip=true" >> "$GITHUB_OUTPUT"
echo "::error::Invalid PR number in artifact: '$PR_NUM'"
exit 0
fi
echo "skip=false" >> "$GITHUB_OUTPUT"
echo "pr_number=$PR_NUM" >> "$GITHUB_OUTPUT"
# Validate job-result strings against known GitHub Actions values.
# Artifact contents come from the PR workflow (potentially untrusted
# fork code), so we whitelist to prevent newline injection into
# GITHUB_OUTPUT.
validate_result() {
local val
val=$(cat "$1" | tr -d '[:space:]')
case "$val" in
success|failure|cancelled|skipped) echo "$val" ;;
*) echo "unknown" ;;
esac
}
echo "quality=$(validate_result "$DIR/quality_result")" >> "$GITHUB_OUTPUT"
echo "tests=$(validate_result "$DIR/tests_result")" >> "$GITHUB_OUTPUT"
echo "e2e=$(validate_result "$DIR/e2e_result")" >> "$GITHUB_OUTPUT"
- name: Checkout (for vitest config)
if: steps.meta.outputs.skip != 'true'
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
sparse-checkout: gitnexus/vitest.config.ts
sparse-checkout-cone-mode: false
# ── Fetch base branch coverage for delta reporting ───────────
- name: Fetch base branch coverage
if: steps.meta.outputs.skip != 'true'
id: base-coverage
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
with:
script: |
const fs = require('fs');
const path = require('path');
// Find the latest successful CI run on main
const runs = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: 'ci.yml',
branch: 'main',
status: 'success',
per_page: 1,
});
if (runs.data.workflow_runs.length === 0) {
core.setOutput('found', 'false');
core.info('No successful main branch CI runs found');
return;
}
const mainRunId = runs.data.workflow_runs[0].id;
const artifacts = await github.rest.actions.listWorkflowRunArtifacts({
owner: context.repo.owner,
repo: context.repo.repo,
run_id: mainRunId,
});
const testReports = artifacts.data.artifacts.find(a => a.name === 'test-reports');
if (!testReports) {
core.setOutput('found', 'false');
core.info('No test-reports artifact on main branch');
return;
}
const zip = await github.rest.actions.downloadArtifact({
owner: context.repo.owner,
repo: context.repo.repo,
artifact_id: testReports.id,
archive_format: 'zip',
});
const dest = path.join(process.env.RUNNER_TEMP, 'base-coverage');
fs.mkdirSync(dest, { recursive: true });
fs.writeFileSync(path.join(dest, 'base.zip'), Buffer.from(zip.data));
core.setOutput('found', 'true');
core.setOutput('dir', dest);
- name: Extract base coverage
if: steps.meta.outputs.skip != 'true' && steps.base-coverage.outputs.found == 'true'
shell: bash
run: |
cd "${{ steps.base-coverage.outputs.dir }}"
mkdir -p base
unzip -o base.zip -d base
- name: Build report
if: steps.meta.outputs.skip != 'true'
id: report
shell: bash
env:
QUALITY: ${{ steps.meta.outputs.quality }}
TESTS: ${{ steps.meta.outputs.tests }}
E2E: ${{ steps.meta.outputs.e2e }}
BASE_FOUND: ${{ steps.base-coverage.outputs.found }}
BASE_DIR: ${{ steps.base-coverage.outputs.dir }}
RUN_URL: ${{ github.event.workflow_run.html_url }}
run: |
DIR="$RUNNER_TEMP/artifacts"
# ── Helper: read coverage summary into prefixed vars ──
read_cov() {
local prefix=$1 file=$2
if [ -n "$file" ] && [ -f "$file" ]; then
local val
val=$(jq -r '.total.statements.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
printf -v "${prefix}_STMTS" '%s' "$val"
val=$(jq -r '.total.branches.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
printf -v "${prefix}_BRANCH" '%s' "$val"
val=$(jq -r '.total.functions.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
printf -v "${prefix}_FUNCS" '%s' "$val"
val=$(jq -r '.total.lines.pct // "N/A"' "$file" 2>/dev/null) || val="N/A"
printf -v "${prefix}_LINES" '%s' "$val"
val=$(jq -r '"\(.total.statements.covered)/\(.total.statements.total)"' "$file" 2>/dev/null) || val=""
printf -v "${prefix}_STMTS_COV" '%s' "$val"
val=$(jq -r '"\(.total.branches.covered)/\(.total.branches.total)"' "$file" 2>/dev/null) || val=""
printf -v "${prefix}_BRANCH_COV" '%s' "$val"
val=$(jq -r '"\(.total.functions.covered)/\(.total.functions.total)"' "$file" 2>/dev/null) || val=""
printf -v "${prefix}_FUNCS_COV" '%s' "$val"
val=$(jq -r '"\(.total.lines.covered)/\(.total.lines.total)"' "$file" 2>/dev/null) || val=""
printf -v "${prefix}_LINES_COV" '%s' "$val"
return 0
else
printf -v "${prefix}_STMTS" '%s' "N/A"
printf -v "${prefix}_BRANCH" '%s' "N/A"
printf -v "${prefix}_FUNCS" '%s' "N/A"
printf -v "${prefix}_LINES" '%s' "N/A"
printf -v "${prefix}_STMTS_COV" '%s' ""
printf -v "${prefix}_BRANCH_COV" '%s' ""
printf -v "${prefix}_FUNCS_COV" '%s' ""
printf -v "${prefix}_LINES_COV" '%s' ""
return 1
fi
}
# ── Read coverage reports ──
UNIT_SUMMARY=$(find "$DIR/test-reports" -name "coverage-summary.json" -type f 2>/dev/null | head -1)
read_cov "U" "$UNIT_SUMMARY"
# ── Read base branch coverage (main) ──
BASE_SUMMARY=""
if [ "$BASE_FOUND" = "true" ] && [ -n "$BASE_DIR" ]; then
BASE_SUMMARY=$(find "$BASE_DIR/base" -name "coverage-summary.json" -type f 2>/dev/null | head -1)
fi
read_cov "B" "$BASE_SUMMARY"
# ── Locate test results ──
RESULTS_FILE=$(find "$DIR/test-reports" -name "test-results.json" -type f 2>/dev/null | head -1)
WEB_RESULTS_FILE=$(find "$DIR/test-reports" -name "web-test-results.json" -type f 2>/dev/null | head -1)
sum_results() {
local file=$1
if [ -n "$file" ] && [ -f "$file" ]; then
jq -r '"\(.numTotalTests) \(.numPassedTests) \(.numFailedTests) \(.numPendingTests) \(.numTotalTestSuites) \(((.testResults | map(.endTime) | max) - (.startTime)) / 1000 | floor)"' "$file" 2>/dev/null || echo "0 0 0 0 0 0"
else
echo "0 0 0 0 0 0"
fi
}
read CLI_T CLI_P CLI_F CLI_S CLI_SU CLI_D <<< "$(sum_results "$RESULTS_FILE")"
read WEB_T WEB_P WEB_F WEB_S WEB_SU WEB_D <<< "$(sum_results "$WEB_RESULTS_FILE")"
TOTAL=$((CLI_T + WEB_T))
PASSED=$((CLI_P + WEB_P))
FAILED=$((CLI_F + WEB_F))
SKIPPED=$((CLI_S + WEB_S))
SUITES=$((CLI_SU + WEB_SU))
DURATION=$((CLI_D > WEB_D ? CLI_D : WEB_D))
# ── Status helpers ──
status_icon() {
case "$1" in
success) echo "✅" ;;
failure) echo "❌" ;;
cancelled) echo "⏭️" ;;
*) echo "❓" ;;
esac
}
# Validate a value looks like a number (integer or decimal, optional
# leading minus). Returns 1 for anything else — guards against awk
# injection when artifact values come from untrusted fork code.
is_numeric() { [[ "$1" =~ ^-?[0-9]+(\.[0-9]+)?$ ]]; }
cov_delta() {
local pct=$1 base=$2
if [ "$pct" = "N/A" ] || [ "$base" = "N/A" ]; then echo "—"; return; fi
if ! is_numeric "$pct" || ! is_numeric "$base"; then echo "—"; return; fi
local diff
diff=$(awk -v p="$pct" -v b="$base" 'BEGIN { printf "%.1f", p - b }')
if [ "$(awk -v p="$pct" -v b="$base" 'BEGIN { print (p > b) ? 1 : 0 }')" = "1" ]; then
echo "📈 +${diff}"
elif [ "$(awk -v p="$pct" -v b="$base" 'BEGIN { print (p < b) ? 1 : 0 }')" = "1" ]; then
echo "📉 ${diff}"
else
echo "= ${diff}"
fi
}
cov_bar() {
local pct=$1 base=$2
if [ "$pct" = "N/A" ] || ! is_numeric "$pct"; then echo "—"; return; fi
local filled
filled=$(awk -v p="$pct" 'BEGIN { printf "%d", p / 5 }')
(( filled < 0 )) && filled=0
(( filled > 20 )) && filled=20
local empty=$((20 - filled))
local bar=""
for ((i=0; i<filled; i++)); do bar+="█"; done
for ((i=0; i<empty; i++)); do bar+="░"; done
# Green if >= base (or base unavailable), red if dropped
if [ "$base" = "N/A" ] || ! is_numeric "$base" || [ "$(awk -v p="$pct" -v b="$base" 'BEGIN { print (p >= b) ? 1 : 0 }')" = "1" ]; then
echo "🟢 ${bar}"
else
echo "🔴 ${bar}"
fi
}
# ── Overall status ──
if [[ "$QUALITY" == "success" && "$TESTS" == "success" && ("$E2E" == "success" || "$E2E" == "skipped") ]]; then
OVERALL="✅ **All checks passed**"
else
OVERALL="❌ **Some checks failed**"
fi
# ── Build markdown ──
{
echo "body<<GITNEXUS_CI_REPORT_EOF_7f3a"
echo "## CI Report"
echo ""
echo "${OVERALL}"
echo ""
echo "### Pipeline Status"
echo ""
echo "| Stage | Status | Details |"
echo "|-------|--------|---------|"
echo "| $(status_icon "$QUALITY") Typecheck | \`${QUALITY}\` | tsc --noEmit |"
echo "| $(status_icon "$TESTS") Tests | \`${TESTS}\` | unit tests, 3 platforms |"
echo "| $(status_icon "$E2E") E2E | \`${E2E}\` | gitnexus-web changes only |"
echo ""
if [ "$TOTAL" -gt 0 ] 2>/dev/null; then
echo "### Test Results"
echo ""
echo "| Tests | Passed | Failed | Skipped | Duration |"
echo "|-------|--------|--------|---------|----------|"
echo "| ${TOTAL} | ${PASSED} | ${FAILED} | ${SKIPPED} | ${DURATION}s |"
echo ""
if [ "$FAILED" = "0" ]; then
echo "✅ All **${PASSED}** tests passed"
else
echo "❌ **${FAILED}** failed / **${PASSED}** passed"
fi
if [ "$SKIPPED" != "0" ]; then
echo ""
echo "<details>"
echo "<summary>${SKIPPED} test(s) skipped — expand for details</summary>"
echo ""
for rf in "$RESULTS_FILE" "$WEB_RESULTS_FILE"; do
if [ -n "$rf" ] && [ -f "$rf" ]; then
jq -r '
.testResults[]
| .assertionResults[]?
| select(.status == "pending" or .status == "skipped")
| "- \(.ancestorTitles | join(" > ")) > \(.title)"
' "$rf" 2>/dev/null || true
fi
done
echo ""
echo "</details>"
fi
echo ""
fi
# ── Coverage table helper ──
cov_table() {
local label=$1 s=$2 b=$3 f=$4 l=$5 sc=$6 bc=$7 fc=$8 lc=$9
shift 9
local bs=$1 bb=$2 bf=$3 bl=$4
echo "#### ${label}"
echo ""
echo "| Metric | Coverage | Covered | Base | Delta | Status |"
echo "|--------|----------|---------|------|-------|--------|"
echo "| Statements | **${s}%** | ${sc} | ${bs}% | $(cov_delta "$s" "$bs") | $(cov_bar "$s" "$bs") |"
echo "| Branches | **${b}%** | ${bc} | ${bb}% | $(cov_delta "$b" "$bb") | $(cov_bar "$b" "$bb") |"
echo "| Functions | **${f}%** | ${fc} | ${bf}% | $(cov_delta "$f" "$bf") | $(cov_bar "$f" "$bf") |"
echo "| Lines | **${l}%** | ${lc} | ${bl}% | $(cov_delta "$l" "$bl") | $(cov_bar "$l" "$bl") |"
echo ""
}
if [ "$U_STMTS" != "N/A" ]; then
echo "### Code Coverage"
echo ""
cov_table "Tests" \
"$U_STMTS" "$U_BRANCH" "$U_FUNCS" "$U_LINES" \
"$U_STMTS_COV" "$U_BRANCH_COV" "$U_FUNCS_COV" "$U_LINES_COV" \
"$B_STMTS" "$B_BRANCH" "$B_FUNCS" "$B_LINES"
else
echo "### Code Coverage"
echo ""
echo "⚠️ Coverage data unavailable - check the [unit test job](${RUN_URL}) for details."
echo ""
fi
echo "---"
echo "<sub>📋 [View full run](${RUN_URL}) · Generated by CI</sub>"
echo "GITNEXUS_CI_REPORT_EOF_7f3a"
} >> "$GITHUB_OUTPUT"
- name: Comment on PR
if: steps.meta.outputs.skip != 'true'
uses: marocchino/sticky-pull-request-comment@773744901bac0e8cbb5a0dc842800d45e9b2b405 # v2
with:
header: ci-report
number: ${{ steps.meta.outputs.pr_number }}
message: ${{ steps.report.outputs.body }}
+71
View File
@@ -0,0 +1,71 @@
name: Tests
on:
workflow_call:
jobs:
tests:
name: ubuntu / coverage
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: ./.github/actions/setup-gitnexus
with:
build: 'true'
- name: Run all tests with coverage
run: >-
npx vitest run
--reporter=default
--reporter=json
--outputFile=test-results.json
--coverage
--coverage.reporter=json-summary
--coverage.reporter=json
--coverage.reporter=text
--coverage.thresholdAutoUpdate=false
--coverage.reportOnFailure=true
working-directory: gitnexus
# gitnexus-shared already built by setup-gitnexus action above
- name: Install gitnexus-web dependencies
run: npm ci
working-directory: gitnexus-web
- name: Run gitnexus-web unit tests
run: >-
npx vitest run
--reporter=default
--reporter=json
--outputFile=web-test-results.json
working-directory: gitnexus-web
- name: Upload test reports
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: test-reports
path: |
gitnexus/coverage/coverage-summary.json
gitnexus/coverage/coverage-final.json
gitnexus/test-results.json
gitnexus-web/web-test-results.json
retention-days: 5
cross-platform:
name: ${{ matrix.os }}
strategy:
fail-fast: false
matrix:
# Ubuntu already covered by the coverage job above
os: [windows-latest, macos-latest]
runs-on: ${{ matrix.os }}
timeout-minutes: 25
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: ./.github/actions/setup-gitnexus
with:
build: 'true'
- run: npx vitest run
working-directory: gitnexus
+110
View File
@@ -0,0 +1,110 @@
name: CI
on:
push:
branches: [main]
paths-ignore: ['**.md', 'docs/**', 'LICENSE']
pull_request:
branches: [main]
paths-ignore: ['**.md', 'docs/**', 'LICENSE']
workflow_call:
concurrency:
group: ci-${{ github.ref }}
cancel-in-progress: true
# ── Reusable workflow orchestration ─────────────────────────────────
# Each concern lives in its own workflow file for maintainability:
# ci-quality.yml — typecheck (tsc --noEmit)
# ci-tests.yml — unit + integration tests with coverage + cross-platform
# ci-e2e.yml — E2E tests (only when gitnexus-web/ changes)
#
# Shared setup is DRY via .github/actions/setup-gitnexus composite action.
jobs:
quality:
uses: ./.github/workflows/ci-quality.yml
permissions:
contents: read
tests:
uses: ./.github/workflows/ci-tests.yml
permissions:
contents: read
e2e:
uses: ./.github/workflows/ci-e2e.yml
permissions:
contents: read
# ── Save PR metadata for the reporting workflow ─────────────────
# The ci-report.yml workflow (triggered by workflow_run) needs the
# PR number and job results to post a comment. We save them as an
# artifact because workflow_run context doesn't reliably carry PR
# info for fork PRs.
save-pr-meta:
name: Save PR Metadata
if: always() && github.event_name == 'pull_request'
needs: [quality, tests, e2e]
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Write metadata
shell: bash
env:
PR_NUMBER: ${{ github.event.number }}
QUALITY: ${{ needs.quality.result }}
TESTS: ${{ needs.tests.result }}
E2E: ${{ needs.e2e.result }}
run: |
mkdir -p pr-meta
echo "$PR_NUMBER" > pr-meta/pr_number
echo "$QUALITY" > pr-meta/quality_result
echo "$TESTS" > pr-meta/tests_result
echo "$E2E" > pr-meta/e2e_result
# TODO(post-merge): remove backward-compat copies once ci-report.yml
# on main reads underscore names.
# Backward-compat: ci-report.yml on main still reads hyphenated
# names. workflow_run always executes from the default branch, so
# the main-branch reader won't find the underscore variants until
# this PR is merged. Write both until then.
cp pr-meta/pr_number pr-meta/pr-number
cp pr-meta/quality_result pr-meta/quality-result
cp pr-meta/tests_result pr-meta/tests-result
cp pr-meta/e2e_result pr-meta/e2e-result
- name: Upload PR metadata
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
with:
name: pr-meta
path: pr-meta/
retention-days: 1
# ── Unified CI gate ──────────────────────────────────────────────
# Single required check for branch protection.
ci-status:
name: CI Gate
needs: [quality, tests, e2e]
if: always()
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Check all jobs passed
shell: bash
env:
QUALITY: ${{ needs.quality.result }}
TESTS: ${{ needs.tests.result }}
E2E: ${{ needs.e2e.result }}
run: |
echo "Quality: $QUALITY"
echo "Tests: $TESTS"
echo "E2E: $E2E"
if [[ "$QUALITY" != "success" ]] ||
[[ "$TESTS" != "success" ]]; then
echo "::error::Quality or test jobs failed"
exit 1
fi
if [[ "$E2E" != "success" && "$E2E" != "skipped" ]]; then
echo "::error::E2E job failed"
exit 1
fi
+95
View File
@@ -0,0 +1,95 @@
name: Claude Code Review
# Uses pull_request_target so the workflow runs as defined on the default branch,
# which allows access to secrets for posting review comments on fork PRs.
# SECURITY: The checkout pins the fork's HEAD SHA (not the branch name) to
# prevent TOCTOU races (force-push between trigger and checkout). The
# claude-code-action sandboxes execution — it does NOT run arbitrary code
# from the checked-out source.
on:
# Trigger only when explicitly requested:
# - Add the "claude-review" label to a PR, OR
# - Comment "@claude" or "/review" on a PR
pull_request_target:
types: [labeled]
issue_comment:
types: [created]
# Serialize per-PR to avoid racing review comments.
concurrency:
group: claude-review-${{ github.event.issue.number || github.event.pull_request.number }}
cancel-in-progress: false
jobs:
claude-review:
# Run only when:
# 1. The "claude-review" label is added to a non-draft PR by a trusted contributor, OR
# 2. A trusted contributor comments "@claude" or "/review" on a PR
if: |
(
github.event_name == 'pull_request_target' &&
github.event.label.name == 'claude-review' &&
github.event.pull_request.draft == false &&
(github.event.pull_request.author_association == 'OWNER' ||
github.event.pull_request.author_association == 'MEMBER' ||
github.event.pull_request.author_association == 'COLLABORATOR')
) ||
(
github.event_name == 'issue_comment' &&
github.event.issue.pull_request &&
(contains(github.event.comment.body, '@claude') ||
contains(github.event.comment.body, '/review')) &&
(github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR')
)
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
pull-requests: write
issues: read
id-token: write
steps:
# For issue_comment triggers, resolve the PR number, head SHA, and fork repo
- name: Resolve PR context
id: pr
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
with:
script: |
let pr;
if (context.eventName === 'issue_comment') {
const resp = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: context.payload.issue.number,
});
pr = resp.data;
} else {
pr = context.payload.pull_request;
}
core.setOutput('number', pr.number);
core.setOutput('sha', pr.head.sha);
core.setOutput('repo', pr.head.repo.full_name);
core.setOutput('branch', pr.head.ref);
- name: Checkout PR head
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
repository: ${{ steps.pr.outputs.repo }}
ref: ${{ steps.pr.outputs.sha }}
fetch-depth: 1
- name: Run Claude Code Review
id: claude-review
uses: anthropics/claude-code-action@9469d113c6afd29550c402740f22d1a97dd1209b # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
github_token: ${{ secrets.GITHUB_TOKEN }}
allowed_non_write_users: '*'
show_full_output: true
plugin_marketplaces: 'https://github.com/anthropics/claude-code.git'
plugins: 'code-review@claude-code-plugins'
prompt: '/code-review:code-review ${{ github.repository }}/pull/${{ steps.pr.outputs.number }}'
+110
View File
@@ -0,0 +1,110 @@
name: Claude Code
on:
issue_comment:
types: [created]
pull_request_review_comment:
types: [created]
issues:
types: [opened, assigned]
pull_request_review:
types: [submitted]
# Serialize per-PR/issue to avoid racing comments.
concurrency:
group: claude-code-${{ github.event.issue.number || github.event.pull_request.number || github.event.issue.id }}
cancel-in-progress: false
jobs:
claude:
if: |
(
github.event_name == 'issue_comment' &&
contains(github.event.comment.body, '@claude') &&
(github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR')
) ||
(
github.event_name == 'pull_request_review_comment' &&
contains(github.event.comment.body, '@claude') &&
(github.event.comment.author_association == 'OWNER' ||
github.event.comment.author_association == 'MEMBER' ||
github.event.comment.author_association == 'COLLABORATOR')
) ||
(
github.event_name == 'pull_request_review' &&
contains(github.event.review.body, '@claude') &&
(github.event.review.author_association == 'OWNER' ||
github.event.review.author_association == 'MEMBER' ||
github.event.review.author_association == 'COLLABORATOR')
) ||
(
github.event_name == 'issues' &&
(contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')) &&
(github.event.issue.author_association == 'OWNER' ||
github.event.issue.author_association == 'MEMBER' ||
github.event.issue.author_association == 'COLLABORATOR')
)
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
pull-requests: write
issues: write
id-token: write
actions: read # required for Claude to read CI results on PRs
steps:
# For PR-related triggers, resolve the fork repo so we can checkout correctly.
- name: Resolve PR context
id: pr
uses: actions/github-script@60a0d83039c74a4aee543508d2ffcb1c3799cdea # v7
with:
script: |
// Determine if this event is PR-related
let prNumber = null;
if (context.eventName === 'issue_comment' && context.payload.issue.pull_request) {
prNumber = context.payload.issue.number;
} else if (context.eventName === 'pull_request_review_comment') {
prNumber = context.payload.pull_request.number;
} else if (context.eventName === 'pull_request_review') {
prNumber = context.payload.pull_request.number;
}
if (!prNumber) {
core.setOutput('is_pr', 'false');
return;
}
const resp = await github.rest.pulls.get({
owner: context.repo.owner,
repo: context.repo.repo,
pull_number: prNumber,
});
const pr = resp.data;
core.setOutput('is_pr', 'true');
core.setOutput('number', String(prNumber));
core.setOutput('sha', pr.head.sha);
core.setOutput('repo', pr.head.repo.full_name);
core.setOutput('branch', pr.head.ref);
- name: Checkout repository
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
repository: ${{ steps.pr.outputs.is_pr == 'true' && steps.pr.outputs.repo || github.repository }}
ref: ${{ steps.pr.outputs.is_pr == 'true' && steps.pr.outputs.sha || '' }}
fetch-depth: 1
- name: Run Claude Code
id: claude
uses: anthropics/claude-code-action@9469d113c6afd29550c402740f22d1a97dd1209b # v1
with:
claude_code_oauth_token: ${{ secrets.CLAUDE_CODE_OAUTH_TOKEN }}
github_token: ${{ secrets.GITHUB_TOKEN }}
allowed_non_write_users: '*'
show_full_output: true
# This is an optional setting that allows Claude to read CI results on PRs
additional_permissions: |
actions: read
@@ -0,0 +1,94 @@
name: PR Description Check
on:
pull_request:
types: [opened, edited, reopened]
branches: [main]
permissions:
pull-requests: write
concurrency:
group: pr-desc-${{ github.event.pull_request.number }}
cancel-in-progress: true
jobs:
check-description:
runs-on: ubuntu-latest
timeout-minutes: 5
steps:
- name: Check PR description quality
uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
with:
script: |
const MIN_BODY_LENGTH = 50;
const LABEL = 'needs-description';
const pr = context.payload.pull_request;
const body = (pr.body || '').trim();
const owner = context.repo.owner;
const repo = context.repo.repo;
const number = pr.number;
const hasLabel = pr.labels.some(l => l.name === LABEL);
if (body.length < MIN_BODY_LENGTH) {
// Add label if not already present
if (!hasLabel) {
await github.rest.issues.addLabels({
owner, repo, issue_number: number,
labels: [LABEL],
});
}
// Post or update a comment
const marker = '<!-- pr-desc-check -->';
const message = [
marker,
`### PR description is too short`,
'',
`This PR's description is **${body.length}** characters, ` +
`but the minimum is **${MIN_BODY_LENGTH}**.`,
'',
'Please update the PR description to explain:',
'- **What** this PR changes',
'- **Why** the change is needed',
'',
'Use the PR template as a guide. This check will re-run when you edit the description.',
].join('\n');
// Find existing bot comment to update (avoid spam)
const comments = await github.rest.issues.listComments({
owner, repo, issue_number: number,
});
const existing = comments.data.find(c =>
c.body && c.body.includes(marker)
);
if (existing) {
await github.rest.issues.updateComment({
owner, repo, comment_id: existing.id,
body: message,
});
} else {
await github.rest.issues.createComment({
owner, repo, issue_number: number,
body: message,
});
}
core.setFailed(
`PR description is ${body.length} chars (minimum: ${MIN_BODY_LENGTH})`
);
} else {
// Description is acceptable — remove the label if present
if (hasLabel) {
await github.rest.issues.removeLabel({
owner, repo, issue_number: number,
name: LABEL,
}).catch(() => {});
// .catch: label may have been removed manually
}
core.info(`PR description OK (${body.length} chars)`);
}
+73
View File
@@ -0,0 +1,73 @@
name: Publish to npm
on:
push:
tags:
- 'v*'
# No workflow-level permissions — scoped per job below.
jobs:
ci:
uses: ./.github/workflows/ci.yml
permissions:
contents: read
actions: read
pull-requests: write
publish:
needs: ci
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: write
id-token: write
steps:
- uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
- uses: actions/setup-node@49933ea5288caeca8642d1e84afbd3f7d6820020 # v4
with:
node-version: 20
registry-url: https://registry.npmjs.org
cache: npm
cache-dependency-path: gitnexus/package-lock.json
- name: Build gitnexus-shared
run: npm install && npm run build
working-directory: gitnexus-shared
- run: npm ci
working-directory: gitnexus
- name: Verify version consistency
shell: bash
run: |
TAG_VERSION="${GITHUB_REF#refs/tags/v}"
if ! [[ "$TAG_VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+(-[a-zA-Z0-9.]+)?$ ]]; then
echo "::error::Tag does not follow semver: v$TAG_VERSION"
exit 1
fi
PKG_VERSION=$(node -p "require('./package.json').version")
if [ "$TAG_VERSION" != "$PKG_VERSION" ]; then
echo "::error::Tag version (v$TAG_VERSION) does not match package.json version ($PKG_VERSION)"
exit 1
fi
echo "Version verified: $PKG_VERSION"
working-directory: gitnexus
- name: Build
run: npm run build
working-directory: gitnexus
- name: Dry-run publish
run: npm publish --dry-run
working-directory: gitnexus
- name: Publish to npm
run: npm publish --provenance --access public
working-directory: gitnexus
env:
NODE_AUTH_TOKEN: ${{ secrets.NPM_TOKEN }}
- name: Create GitHub Release
uses: softprops/action-gh-release@a06a81a03ee405af7f2048a818ed3f03bbf83c7b # v2
with:
generate_release_notes: true
+103
View File
@@ -0,0 +1,103 @@
name: Triage Sweep
on:
workflow_dispatch:
inputs:
iqr_multiplier:
description: >-
IQR multiplier for outlier cutoff.
cutoff = Q75 + multiplier * IQR.
Higher = fewer outliers flagged.
type: number
default: 3.0
max_outlier_pct:
description: >-
Maximum fraction of items that can be flagged as outliers (0-1).
Hard cap to prevent over-flagging.
type: number
default: 0.05
contamination:
description: >-
Expected fraction of outliers in the data (0-0.5).
Controls how aggressively EllipticEnvelope downweights extremes.
type: number
default: 0.1
cosine_threshold:
description: >-
Cosine similarity threshold for duplicate detection.
Pairs with similarity above this are flagged as potential duplicates.
Higher = only very similar pairs flagged.
type: number
default: 0.92
max_items:
description: >-
Maximum number of open issues + PRs to process.
Hard cap to prevent runaway costs on very large repos.
type: number
default: 500
dry_run:
description: >-
Check this to only log results to the workflow summary.
Uncheck to create a GitHub issue with the report and apply labels.
type: boolean
default: false
permissions:
contents: read
issues: write
pull-requests: write
concurrency:
group: triage-sweep
cancel-in-progress: true
jobs:
sweep:
runs-on: ubuntu-latest
timeout-minutes: 30
steps:
- name: Checkout repository
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6
with:
sparse-checkout: .github/scripts/triage
sparse-checkout-cone-mode: false
fetch-depth: 1
- name: Set up Python
uses: actions/setup-python@a309ff8b426b58ec0e2a45f0f869d46889d02405 # v6
with:
python-version: '3.12'
cache: pip
cache-dependency-path: .github/scripts/triage/requirements.txt
- name: Install dependencies
run: pip install -r .github/scripts/triage/requirements.txt
- name: Cache FastEmbed model weights
uses: actions/cache@668228422ae6a00e4ad889ee87cd7109ec5666a7 # v5
with:
path: ${{ github.workspace }}/.fastembed_cache
key: fastembed-bge-small-en-v1.5
- name: Run triage sweep
id: sweep
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GITHUB_REPOSITORY: ${{ github.repository }}
FASTEMBED_CACHE_PATH: ${{ github.workspace }}/.fastembed_cache
INPUT_IQR_MULTIPLIER: ${{ inputs.iqr_multiplier }}
INPUT_MAX_OUTLIER_PCT: ${{ inputs.max_outlier_pct }}
INPUT_CONTAMINATION: ${{ inputs.contamination }}
INPUT_COSINE_THRESHOLD: ${{ inputs.cosine_threshold }}
INPUT_MAX_ITEMS: ${{ inputs.max_items }}
INPUT_DRY_RUN: ${{ inputs.dry_run }}
run: python .github/scripts/triage/sweep.py
- name: Post summary
if: always()
run: |
if [ -f /tmp/triage-report.md ]; then
cat /tmp/triage-report.md >> "$GITHUB_STEP_SUMMARY"
else
echo "No report generated." >> "$GITHUB_STEP_SUMMARY"
fi
+89 -37
View File
@@ -1,46 +1,98 @@
# Dependencies
node_modules/
# Build output
dist/
# TypeScript build info
*.tsbuildinfo
# IDE
.vscode/
.idea/
*.swp
*.swo
# OS
.DS_Store
Thumbs.db
.claude/settings.local.json
# Environment variables
.env
.env.local
.env.*.local
# Logs
logs
*.log
npm-debug.log*
yarn-debug.log*
yarn-error.log*
pnpm-debug.log*
lerna-debug.log*
node_modules
dist
dist-ssr
# Testing
coverage/
# Misc
*.local
HANDOFF.md
HANDOFF*.md
# Auto-generated files
public/workers/compiled-queries.js
.vercel
# Editor directories and files
.vscode/*
!.vscode/extensions.json
.idea
.DS_Store
*.suo
*.ntvs*
*.njsproj
*.sln
*.sw?
# AI/Development tool directories
.kilocode/
.gemini/
.cursor/
.clinerules/
.qoder/
# Large JSON files
gitnexus-project_*.json
*-project_*.json
.clinerules/byterover-rules.md
.kilocode/rules/byterover-rules.md
.roo/rules/byterover-rules.md
.windsurf/rules/byterover-rules.md
.cursor/rules/byterover-rules.mdc
.kiro/steering/byterover-rules.md
.qoder/rules/byterover-rules.md
.augment/rules/byterover-rules.md
.env*.local
.gitnexus
.claude/settings.local.json
# Claude Code worktrees
.claude/worktrees/
# Claude code skills
.claude/skills/generated/
# Assets (screenshots, images)
assets/
# Generated files (should not be indexed)
repomix-output*
# Playwright artifacts
gitnexus-web/playwright-report/
gitnexus-web/test-results/
# Python test artifacts
eval/.coverage
eval/.hypothesis/
# Design docs (local only)
docs/plans/
gitnexus/test/fixtures/mini-repo/*.md
gitnexus/test/fixtures/mini-repo/.claude
gitnexus/test/fixtures/mini-repo/.gitignore
# Ignore csharp generated obj and bin folders
gitnexus/test/fixtures/lang-resolution/**/obj
gitnexus/test/fixtures/lang-resolution/**/bin
GitNexus.sln
# Git worktrees
.worktrees/
/github/scripts/triage/__pycache__/
.claude-flow/
.claude/agents/
.claude/commands/
.claude/helpers
.claude/skills/
!.claude/skills/gitnexus/
.history/
.swarm/
local_docs/
@@ -0,0 +1,33 @@
import { defineConfig } from 'vitest/config';
export default defineConfig({
test: {
globalSetup: ['test/global-setup.ts'],
include: ['test/**/*.test.ts'],
testTimeout: 30000,
hookTimeout: 120000,
pool: 'forks',
globals: true,
setupFiles: ['test/setup.ts'],
teardownTimeout: 3000,
dangerouslyIgnoreUnhandledErrors: true, // LadybugDB N-API destructor segfaults on fork exit — not a test failure
coverage: {
provider: 'v8',
include: ['src/**/*.ts'],
exclude: [
'src/cli/index.ts', // CLI entry point (commander wiring)
'src/server/**', // HTTP server (requires network)
'src/core/wiki/**', // Wiki generation (requires LLM)
],
// Auto-ratchet: vitest bumps thresholds when coverage exceeds them.
// CI will fail if a PR drops below these floors.
thresholds: {
statements: 26,
branches: 23,
functions: 28,
lines: 27,
autoUpdate: true,
},
},
},
});
+26
View File
@@ -0,0 +1,26 @@
#!/usr/bin/env bash
# Pre-commit hook: format staged files + typecheck.
# Tests run in CI (ci-tests.yml), not here.
# Skip with: git commit --no-verify
ROOT="$(git rev-parse --show-toplevel)"
# 1. Format staged files with prettier via lint-staged
echo "pre-commit: formatting staged files..."
"$ROOT/node_modules/.bin/lint-staged" || exit 1
# 2. Typecheck changed packages
WEB_CHANGED=$(git diff --cached --name-only -- 'gitnexus-web/' | head -1)
CLI_CHANGED=$(git diff --cached --name-only -- 'gitnexus/' | head -1)
if [ -n "$WEB_CHANGED" ]; then
echo "pre-commit: typechecking gitnexus-web (tsc -b)..."
cd "$ROOT/gitnexus-web" && ./node_modules/.bin/tsc -b --noEmit || exit 1
fi
if [ -n "$CLI_CHANGED" ]; then
echo "pre-commit: typechecking gitnexus..."
cd "$ROOT/gitnexus" && ./node_modules/.bin/tsc --noEmit || exit 1
fi
echo "pre-commit: all checks passed"
+9
View File
@@ -0,0 +1,9 @@
{
"mcpServers": {
"gitnexus": {
"type": "stdio",
"command": "npx",
"args": ["-y", "gitnexus@latest", "mcp"]
}
}
}
+16
View File
@@ -0,0 +1,16 @@
dist/
coverage/
gitnexus/vendor/
gitnexus/test/fixtures/
gitnexus-web/playwright-report/
gitnexus-web/test-results/
*.d.ts
*.snap
*.wasm
*.md
.gitnexus/
.vercel/
.claude-flow/
.swarm/
assets/
repomix-output*
+10
View File
@@ -0,0 +1,10 @@
{
"semi": true,
"singleQuote": true,
"trailingComma": "all",
"printWidth": 100,
"tabWidth": 2,
"endOfLine": "lf",
"plugins": ["prettier-plugin-tailwindcss"],
"tailwindStylesheet": "./gitnexus-web/src/index.css"
}
@@ -0,0 +1,18 @@
# Draft: Gitnexus Brainstorming - Clustering & Process Maps
## Initial Context
- Project: **GitnexusV2**
- Structure:
- `gitnexus/` (Likely the core application)
- `gitnexus-mcp/` (Likely a Model Context Protocol server)
- Goal: Make it accurate and usable for smaller/dumber models.
- Current Focus: Implementing **Clustering** and **Process Maps**.
## Findings
- **Clustering**: Found `gitnexus/src/core/ingestion/cluster-enricher.ts`.
- **Process Maps**: No files matched `*process*map*` yet. Searching content next.
## Open Questions
- How is "process map" defined in this context? (Graph, mermaid diagram, flowchart?)
- What is the input for clustering? (Code chunks, files, commits?)
- What is the intended output for "smaller models"? (Simplified context, summaries?)
+34
View File
@@ -0,0 +1,34 @@
# Draft: Gitnexus vs Noodlbox Strategy
## Objectives
- Understand GitnexusV2 current state and goals.
- Analyze Noodlbox capabilities from provided URL.
- Compare features, architecture, and value proposition.
- Provide strategic views and recommendations.
## Research Findings
- [GitnexusV2]: Zero-server, browser-native (WASM), KuzuDB based. Graph + Vector hybrid search.
- [Noodlbox]: CLI-first, heavy install. Has "Session Hooks" and "Search Hooks" via plugins/CLI.
## Comparison Points
- **Core Philosophy**: Both bet on "Knowledge Graph + MCP" as the future. Noodlbox validates Gitnexus's direction.
- **Architecture**:
- *Noodlbox*: CLI/Binary based. Likely local server management.
- *Gitnexus*: Zero-server, Browser-native (WASM). Lower friction, higher privacy.
- **Features**:
- *Communities/Processes*: Both have them. Noodlbox uses them for "context injection". Gitnexus uses them for "visual exploration + query".
- *Impact Analysis*: Noodlbox has polished workflows (e.g., `detect_impact staged`). Gitnexus has the engine (`blastRadius`) but maybe not the specific workflow wrappers yet.
- **UX/Integration**:
- *Noodlbox*: "Hooks" (Session/Search) are a killer feature. Proactively injecting context into the agent's session.
- *Gitnexus*: Powerful tools, but relies on agent *pulling* data?
## Strategic Views
1. **Validation**: The market direction is confirmed. You are building the right thing.
2. **differentiation**: Lean into "Zero-Setup / Browser-Native". Noodlbox requires `noodl init` and CLI handling. Gitnexus could just *be*.
3. **Opportunity**: Steal the "Session/Search Hooks" pattern. Make the agent smarter *automatically* without the user asking "check impact".
4. **Workflow Polish**: Noodlbox's `/detect_impact staged` is a great specific use case. Gitnexus should wrap `blastRadius` into similar concrete workflows.
## Technical Feasibility (Interception)
- **Cursor**: Use `.cursorrules` to "shadow" default tools. Instruct agent to ALWAYS use `gitnexus_search` instead of `grep`.
- **Claude Code**: Likely uses a private plugin API for `PreToolUse`. We can't match this exactly without an official plugin, but we can approximate it with strong prompt instructions in `AGENTS.md`.
- **MCP Shadowing**: Define tools with names that conflict (e.g., `grep`)? No, unsafe. Better to use "Virtual Hooks" via system prompt instructions.
+5
View File
@@ -0,0 +1,5 @@
# AI Agent Rules
Follow .gitnexus/RULES.md for all project context and coding guidelines.
This project uses GitNexus MCP for code intelligence. See .gitnexus/RULES.md for available tools and best practices.
+204
View File
@@ -0,0 +1,204 @@
<!-- version: 1.2.0 -->
<!--
Metadata: version, last reviewed, scope, model policy, reference docs, changelog.
Last updated: 2026-03-22
-->
Last reviewed: 2026-03-24
**Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub)
This file uses a standard agent header (version, scope, model policy, reference docs, changelog), adapted for this **TypeScript/JavaScript monorepo**.
## Scope
| | |
|--|--|
| **Reads** | Repository tree as needed for the task: `gitnexus/`, `gitnexus-web/`, `eval/`, plugin packages, `.github/`, `.gitnexus/` when present, and docs. |
| **Writes** | Only paths required for the requested change; keep diffs minimal. Update lockfiles when dependencies change. |
| **Executes** | `npm`, `npx`, `node` under `gitnexus/` and `gitnexus-web/`; `uv run` for Python under `eval/` when applicable; shell utilities for documented CI/dev workflows. |
| **Off-limits** | User secrets (e.g. real `.env`), production deployment credentials, unrelated repositories, destructive git history operations without explicit human confirmation. |
## Model Configuration
- **Primary:** Pin in **Cursor** (Settings → model). Use a **named** model (e.g. GPT-5.2, Claude Sonnet 4.x). Avoid relying on **Auto** when reproducibility or audit trail matters.
- **Fallback:** As configured in Cursor or your organization (do not encode `latest` or wildcards in automation configs).
- **Notes:** The open-source GitNexus CLI indexer does not call an LLM. Optional Nexus AI in the web UI uses end-user provider keys and models.
## Execution Sequence (complex tasks)
Long sessions dilute instructions. For **multi-step** work, state up front:
1. Which rules in this file and **[GUARDRAILS.md](GUARDRAILS.md)** apply (and any relevant Signs).
2. Current **Scope** boundaries (Reads / Writes / Off-limits).
3. Which **validation commands** you will run (e.g. `cd gitnexus && npm test`, `npx tsc --noEmit`).
On very long threads, the human may add *“Remember: apply all AGENTS.md rules”* to re-weight rule tokens against context dilution.
## Claude Code hooks
Hooks enforce gates that prompts cannot. In **Claude Code**, **PreToolUse** hooks can block tools such as `git_commit` until checks pass. Adapt to this repo: e.g. `cd gitnexus && npm test` before commit.
## Context budget (Cursor / standards)
Generic “core standards” playbooks are often long and stack-specific. For this monorepo, commands and gotchas live under **Cursor Cloud specific instructions** below and in **[CONTRIBUTING.md](CONTRIBUTING.md)**. If always-on rules grow, split domain rules into **`.cursor/rules/*.mdc`** (globs). **Cursor:** project-wide rules live in **`.cursor/index.mdc`** (YAML frontmatter with `alwaysApply: true`). **Claude Code:** optionally load a **`STANDARDS.md`** only when needed (e.g. *“When writing new code, read STANDARDS.md”*) to save context.
## Reference Documentation
- **This repository:** **[ARCHITECTURE.md](ARCHITECTURE.md)**, **[CONTRIBUTING.md](CONTRIBUTING.md)**, **[GUARDRAILS.md](GUARDRAILS.md)**.
- **Cursor:** `.cursor/index.mdc` (always-on rules); optional `.cursor/rules/*.mdc` (glob-scoped). Legacy `.cursorrules` is deprecated — see `.cursor/index.mdc`.
- **Optional local files:** `NOTES.md` (short vendor-neutral project snapshot). For handoffs, keep notes local (e.g., a scratch file outside the repo) rather than committing `HANDOFF.md`.
- **GitNexus:** skills under `.claude/skills/gitnexus/`; machine-oriented rules in the `gitnexus:start` … `gitnexus:end` block below.
## Changelog
| Date | Version | Change |
|------|---------|--------|
| 2026-03-24 | 1.2.0 | Fixed gitnexus:start block duplication (was inlined in Reference Docs bullet). |
| 2026-03-23 | 1.1.0 | Updated agent instructions (sections, references, Cursor layout). |
| 2026-03-22 | 1.0.0 | Added structured agent header and changelog. |
---
<!-- gitnexus:start -->
# GitNexus — Code Intelligence
This project is indexed by GitNexus as **GitNexus**. Use the GitNexus MCP tools to understand code, assess impact, and navigate safely. For current symbol stats, run `npx gitnexus analyze` and inspect `.gitnexus/meta.json`.
> If any GitNexus tool warns the index is stale, run `npx gitnexus analyze` in terminal first.
## Always Do
- **MUST run impact analysis before editing any symbol.** Before modifying a function, class, or method, run `gitnexus_impact({target: "symbolName", direction: "upstream"})` and report the blast radius (direct callers, affected processes, risk level) to the user.
- **MUST run `gitnexus_detect_changes()` before committing** to verify your changes only affect expected symbols and execution flows.
- **MUST warn the user** if impact analysis returns HIGH or CRITICAL risk before proceeding with edits.
- When exploring unfamiliar code, use `gitnexus_query({query: "concept"})` to find execution flows instead of grepping. It returns process-grouped results ranked by relevance.
- When you need full context on a specific symbol — callers, callees, which execution flows it participates in — use `gitnexus_context({name: "symbolName"})`.
## When Debugging
1. `gitnexus_query({query: "<error or symptom>"})` — find execution flows related to the issue
2. `gitnexus_context({name: "<suspect function>"})` — see all callers, callees, and process participation
3. `READ gitnexus://repo/GitNexus/process/{processName}` — trace the full execution flow step by step
4. For regressions: `gitnexus_detect_changes({scope: "compare", base_ref: "main"})` — see what your branch changed
## When Refactoring
- **Renaming**: MUST use `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` first. Review the preview — graph edits are safe, text_search edits need manual review. Then run with `dry_run: false`.
- **Extracting/Splitting**: MUST run `gitnexus_context({name: "target"})` to see all incoming/outgoing refs, then `gitnexus_impact({target: "target", direction: "upstream"})` to find all external callers before moving code.
- After any refactor: run `gitnexus_detect_changes({scope: "all"})` to verify only expected files changed.
## Never Do
- NEVER edit a function, class, or method without first running `gitnexus_impact` on it.
- NEVER ignore HIGH or CRITICAL risk warnings from impact analysis.
- NEVER rename symbols with find-and-replace — use `gitnexus_rename` which understands the call graph.
- NEVER commit changes without running `gitnexus_detect_changes()` to check affected scope.
## Tools Quick Reference
| Tool | When to use | Command |
|------|-------------|---------|
| `query` | Find code by concept | `gitnexus_query({query: "auth validation"})` |
| `context` | 360-degree view of one symbol | `gitnexus_context({name: "validateUser"})` |
| `impact` | Blast radius before editing | `gitnexus_impact({target: "X", direction: "upstream"})` |
| `detect_changes` | Pre-commit scope check | `gitnexus_detect_changes({scope: "staged"})` |
| `rename` | Safe multi-file rename | `gitnexus_rename({symbol_name: "old", new_name: "new", dry_run: true})` |
| `cypher` | Custom graph queries | `gitnexus_cypher({query: "MATCH ..."})` |
## Impact Risk Levels
| Depth | Meaning | Action |
|-------|---------|--------|
| d=1 | WILL BREAK — direct callers/importers | MUST update these |
| d=2 | LIKELY AFFECTED — indirect deps | Should test |
| d=3 | MAY NEED TESTING — transitive | Test if critical path |
## Resources
| Resource | Use for |
|----------|---------|
| `gitnexus://repo/GitNexus/context` | Codebase overview, check index freshness |
| `gitnexus://repo/GitNexus/clusters` | All functional areas |
| `gitnexus://repo/GitNexus/processes` | All execution flows |
| `gitnexus://repo/GitNexus/process/{name}` | Step-by-step execution trace |
## Self-Check Before Finishing
Before completing any code modification task, verify:
1. `gitnexus_impact` was run for all modified symbols
2. No HIGH/CRITICAL risk warnings were ignored
3. `gitnexus_detect_changes()` confirms changes match expected scope
4. All d=1 (WILL BREAK) dependents were updated
## Keeping the Index Fresh
After committing code changes, the GitNexus index becomes stale. Re-run analyze to update it:
```bash
npx gitnexus analyze
```
If the index previously included embeddings, preserve them by adding `--embeddings`:
```bash
npx gitnexus analyze --embeddings
```
To check whether embeddings exist, inspect `.gitnexus/meta.json` — the `stats.embeddings` field shows the count (0 means no embeddings). **Running analyze without `--embeddings` will delete any previously generated embeddings.**
> Claude Code users: A PostToolUse hook handles this automatically after `git commit` and `git merge`.
## CLI
| Task | Read this skill file |
|------|---------------------|
| Understand architecture / "How does X work?" | `.claude/skills/gitnexus/gitnexus-exploring/SKILL.md` |
| Blast radius / "What breaks if I change X?" | `.claude/skills/gitnexus/gitnexus-impact-analysis/SKILL.md` |
| Trace bugs / "Why is X failing?" | `.claude/skills/gitnexus/gitnexus-debugging/SKILL.md` |
| Rename / extract / split / refactor | `.claude/skills/gitnexus/gitnexus-refactoring/SKILL.md` |
| Tools, resources, schema reference | `.claude/skills/gitnexus/gitnexus-guide/SKILL.md` |
| Index, status, clean, wiki CLI commands | `.claude/skills/gitnexus/gitnexus-cli/SKILL.md` |
<!-- gitnexus:end -->
## Cursor Cloud specific instructions
### Repository structure
This is a monorepo with two main products and supporting config packages:
| Component | Path | Purpose |
|-----------|------|---------|
| **GitNexus CLI/Core** | `gitnexus/` | Main product — TypeScript CLI, indexing pipeline, MCP server. Published to npm. |
| **GitNexus Web UI** | `gitnexus-web/` | React/Vite browser app — graph explorer + AI chat. Runs entirely in WASM. |
| Claude Plugin | `gitnexus-claude-plugin/` | Static config for Claude marketplace (no build). |
| Cursor Integration | `gitnexus-cursor-integration/` | Static config for Cursor editor (no build). |
| SWE-bench Eval | `eval/` | Python evaluation harness (optional; needs Docker + LLM API keys). |
### Running services
- **CLI/Core**: `cd gitnexus && npm run dev` (tsx watch mode) or `npm run build && node dist/cli/index.js <command>`
- **Web UI**: `cd gitnexus-web && npm run dev` (Vite on port 5173)
- **Backend mode**: `cd <indexed-repo> && node /workspace/gitnexus/dist/cli/index.js serve` (HTTP API on port 3741 by default)
### Testing
**CLI / Core (`gitnexus/`)**
- **Unit tests**: `cd gitnexus && npm test` (vitest, ~2000 tests)
- **Integration tests**: `cd gitnexus && npm run test:integration` (vitest, ~1850 tests). Two LadybugDB file-locking tests (`lbug-core-adapter`, `search-core`) may fail in containerized environments due to `/tmp` locking limitations — this is a known environment issue, not a code bug.
- **TypeScript check**: `cd gitnexus && npx tsc --noEmit`
**Web UI (`gitnexus-web/`)**
- **Unit tests**: `cd gitnexus-web && npm test` (vitest, ~200 tests)
- **E2E tests**: `cd gitnexus-web && E2E=1 npx playwright test` (Playwright, 5 tests — requires `gitnexus serve` + `npm run dev` running)
- **TypeScript check**: `cd gitnexus-web && npx tsc -b --noEmit`
No separate lint command is configured; TypeScript strict checking serves as the primary static analysis.
### Gotchas
- `npm install` in `gitnexus/` triggers `prepare` (builds via `tsc`) and `postinstall` (patches tree-sitter-swift). Native tree-sitter bindings require `python3`, `make`, and `g++` to be present.
- `tree-sitter-kotlin` and `tree-sitter-swift` are optional dependencies — install warnings for these are expected and non-blocking.
- The Web UI uses `vite-plugin-wasm` and requires `Cross-Origin-Opener-Policy`/`Cross-Origin-Embedder-Policy` headers for `SharedArrayBuffer` (handled automatically by Vite dev server).
- There is no ESLint/Prettier configuration in this repo.
+66
View File
@@ -0,0 +1,66 @@
# Architecture — GitNexus
This repository is a **monorepo** with two main products: the **CLI / MCP package** (`gitnexus/`) and the **browser UI** (`gitnexus-web/`). Supporting folders ship editor integrations and plugins without changing the core graph engine.
## Repository layout
| Path | Role |
|------|------|
| `gitnexus/` | Published npm package `gitnexus`: CLI, MCP server (stdio), local HTTP API for bridge mode, ingestion pipeline, LadybugDB graph, embeddings (optional). |
| `gitnexus-web/` | Vite + React UI: in-browser indexing (WASM), graph visualization, optional connection to `gitnexus serve`. |
| `.claude/`, `gitnexus-claude-plugin/`, `gitnexus-cursor-integration/` | Packaged **skills** and plugin metadata so agents discover the same workflows as documented in `AGENTS.md`. |
| `eval/` | Evaluation harnesses and docs for benchmarking tool usage. |
| `.github/` | CI workflows (quality, unit, integration, E2E) and composite actions. |
## End-to-end flow: index → graph → tools
1. **Ingestion** (`gitnexus analyze`)
- Entry: `gitnexus/src/cli/analyze.ts` → `runPipelineFromRepo` in `gitnexus/src/core/ingestion/pipeline.ts`.
- Walks the git working tree, parses supported languages via **Tree-sitter**, resolves imports/calls/inheritance, detects **communities** and **processes** (execution flows), and builds an in-memory **knowledge graph** (`gitnexus/src/core/graph/`).
- Output is loaded into **LadybugDB** under **`.gitnexus/`** at the repo root (`lbug/`, `meta.json`, etc.). Optional **FTS** indexes and **embeddings** attach to the same store.
- The repo is registered in **`~/.gitnexus/registry.json`** so MCP can find it from any working directory.
2. **Persistence & metadata**
- `gitnexus/src/storage/repo-manager.ts` — paths, registry, cleanup of legacy Kuzu artifacts.
- `gitnexus/src/core/lbug/lbug-adapter.ts` — graph load, queries, embedding restore batches.
3. **Query & agents**
- **MCP (stdio):** `gitnexus/src/cli/mcp.ts` → `startMCPServer` → `LocalBackend` (`gitnexus/src/mcp/local/local-backend.ts`) opens registered repos and serves **tools** from `gitnexus/src/mcp/tools.ts` and **resources** from `gitnexus/src/mcp/resources.ts`.
- **Bridge HTTP:** `gitnexus/src/cli/serve.ts` → Express app in `gitnexus/src/server/api.ts` (CORS-limited) exposes REST + MCP-over-HTTP for the web UI.
- **CLI tools (no MCP):** `gitnexus query`, `context`, `impact`, `cypher` in `gitnexus/src/cli/tool.ts` call the same backend for scripts and CI.
4. **Staleness**
- `gitnexus/src/mcp/staleness.ts` compares indexed `lastCommit` to `HEAD` and surfaces hints when the graph is behind git.
## MCP tools (summary)
| Tool | Purpose |
|------|---------|
| `list_repos` | Discover indexed repositories when more than one is registered. |
| `query` | Natural-language / keyword search over the graph (hybrid BM25 + optional vectors). |
| `cypher` | Ad hoc **Cypher** against the schema (see resource `gitnexus://repo/{name}/schema`). |
| `context` | Callers, callees, processes for one symbol (with disambiguation). |
| `impact` | Blast radius (upstream/downstream) with depth and risk summary. |
| `detect_changes` | Map git diffs to affected symbols and processes. |
| `rename` | Graph-assisted rename with `dry_run` preview (`graph` vs `text_search` confidence). |
## Where to change what
| If you are changing… | Start in… |
|----------------------|-----------|
| CLI commands / flags | `gitnexus/src/cli/` (`index.ts`, per-command modules). |
| Parsing or graph construction | `gitnexus/src/core/ingestion/` (pipeline, processors, resolvers, type-extractors). |
| Graph schema / DB access | `gitnexus/src/core/lbug/` (`schema.ts`, `lbug-adapter.ts`), `gitnexus/src/mcp/core/lbug-adapter.ts` if MCP-specific. |
| MCP protocol, tools, resources | `gitnexus/src/mcp/server.ts`, `tools.ts`, `resources.ts`. |
| Search ranking | `gitnexus/src/core/search/` (BM25, hybrid fusion). |
| Embeddings | `gitnexus/src/core/embeddings/`, phases in `analyze.ts`. |
| Wiki generation | `gitnexus/src/core/wiki/`. |
| Web UI behavior | `gitnexus-web/src/` (components, workers, graph client). |
| CI | `.github/workflows/*.yml`, `.github/actions/setup-gitnexus/`. |
## Related docs
- [RUNBOOK.md](RUNBOOK.md) — operational commands and recovery.
- [GUARDRAILS.md](GUARDRAILS.md) — safety boundaries for humans and agents.
- [TESTING.md](TESTING.md) — how to run tests.
- `AGENTS.md` / `CLAUDE.md` — agent workflows and tool usage expectations for **this** repo when indexed by GitNexus.
+109
View File
@@ -0,0 +1,109 @@
# Changelog
All notable changes to GitNexus will be documented in this file.
## [Unreleased]
### Changed
- Migrated from KuzuDB to LadybugDB v0.15 (`@ladybugdb/core`, `@ladybugdb/wasm-core`)
- Renamed all internal paths from `kuzu` to `lbug` (storage: `.gitnexus/kuzu` → `.gitnexus/lbug`)
- Added automatic cleanup of stale KuzuDB index files
- LadybugDB v0.15 requires explicit VECTOR extension loading for semantic search
## [1.4.0] - 2026-03-13
### Added
- **Language-aware symbol resolution engine** with 3-tier resolver: exact FQN → scope-walk → guarded fuzzy fallback that refuses ambiguous matches (#238) — @magyargergo
- **Method Resolution Order (MRO)** with 5 language-specific strategies: C++ leftmost-base, C#/Java class-over-interface, Python C3 linearization, Rust qualified syntax, default BFS (#238) — @magyargergo
- **Constructor & struct literal resolution** across all languages — `new Foo()`, `User{...}`, C# primary constructors, target-typed new (#238) — @magyargergo
- **Receiver-constrained resolution** using per-file TypeEnv — disambiguates `user.save()` vs `repo.save()` via `ownerId` matching (#238) — @magyargergo
- **Heritage & ownership edges** — HAS_METHOD, OVERRIDES, Go struct embedding, Swift extension heritage, method signatures (`parameterCount`, `returnType`) (#238) — @magyargergo
- **Language-specific resolver directory** (`resolvers/`) — extracted JVM, Go, C#, PHP, Rust resolvers from monolithic import-processor (#238) — @magyargergo
- **Type extractor directory** (`type-extractors/`) — per-language type binding extraction with `Record<SupportedLanguages, Handler>` + `satisfies` dispatch (#238) — @magyargergo
- **Export detection dispatch table** — compile-time exhaustive `Record` + `satisfies` pattern replacing switch/if chains (#238) — @magyargergo
- **Language config module** (`language-config.ts`) — centralized tsconfig, go.mod, composer.json, .csproj, Swift package config loaders (#238) — @magyargergo
- **Optional skill generation** via `npx gitnexus analyze --skills` — generates AI agent skills from KuzuDB knowledge graph (#171) — @zander-raycraft
- **First-class C# support** — sibling-based modifier scanning, record/delegate/property/field/event declaration types (#163, #170, #178 via #237) — @Alice523, @benny-yamagata, @jnMetaCode
- **C/C++ support fixes** — `.h` → C++ mapping, static-linkage export detection, qualified/parenthesized declarators, 48 entry point patterns (#163, #227 via #237) — @Alice523, @bitgineer
- **Rust support fixes** — sibling-based `visibility_modifier` scanning for `pub` detection (#227 via #237) — @bitgineer
- **Adaptive tree-sitter buffer sizing** — `Math.min(Math.max(contentLength * 2, 512KB), 32MB)` (#216 via #237) — @JasonOA888
- **Call expression matching** in tree-sitter queries (#234 via #237) — @ex-nihilo-jg
- **DeepSeek model configurations** (#217) — @JasonOA888
- 282+ new unit tests, 178 integration resolver tests across 9 languages, 53 test files, 1146 total tests passing
### Fixed
- Skip unavailable native Swift parsers in sequential ingestion (#188) — @Gujiassh
- Heritage heuristic language-gated — no longer applies class/interface rules to wrong languages (#238) — @magyargergo
- C# `base_list` distinguishes EXTENDS vs IMPLEMENTS via symbol table + `I[A-Z]` heuristic (#238) — @magyargergo
- Go `qualified_type` (`models.User`) correctly unwrapped in TypeEnv (#238) — @magyargergo
- Global tier no longer blocks resolution when kind/arity filtering can narrow to 1 candidate (#238) — @magyargergo
### Changed
- `import-processor.ts` reduced from 1412 → 711 lines (50% reduction) via resolver and config extraction (#238) — @magyargergo
- `type-env.ts` reduced from 635 → ~125 lines via type-extractor extraction (#238) — @magyargergo
- CI/CD workflows hardened with security fixes and fork PR support (#222, #225) — @magyargergo
## [1.3.11] - 2026-03-08
### Security
- Fix FTS Cypher injection by escaping backslashes in search queries (#209) — @magyargergo
### Added
- Auto-reindex hook that runs `gitnexus analyze` after commits and merges, with automatic embeddings preservation (#205) — @L1nusB
- 968 integration tests (up from ~840) covering unhappy paths across search, enrichment, CLI, pipeline, worker pool, and KuzuDB (#209) — @magyargergo
- Coverage auto-ratcheting so thresholds bump automatically on CI (#209) — @magyargergo
- Rich CI PR report with coverage bars, test counts, and threshold tracking (#209) — @magyargergo
- Modular CI workflow architecture with separate unit-test, integration-test, and orchestrator jobs (#209) — @magyargergo
### Fixed
- KuzuDB native addon crashes on Linux/macOS by running integration tests in isolated vitest processes with `--pool=forks` (#209) — @magyargergo
- Worker pool `MODULE_NOT_FOUND` crash when script path is invalid (#209) — @magyargergo
### Changed
- Added macOS to the cross-platform CI test matrix (#208) — @magyargergo
## [1.3.10] - 2026-03-07
### Security
- **MCP transport buffer cap**: Added 10 MB `MAX_BUFFER_SIZE` limit to prevent out-of-memory attacks via oversized `Content-Length` headers or unbounded newline-delimited input
- **Content-Length validation**: Reject `Content-Length` values exceeding the buffer cap before allocating memory
- **Stack overflow prevention**: Replaced recursive `readNewlineMessage` with iterative loop to prevent stack overflow from consecutive empty lines
- **Ambiguous prefix hardening**: Tightened `looksLikeContentLength` to require 14+ bytes before matching, preventing false framing detection on short input
- **Closed transport guard**: `send()` now rejects with a clear error when called after `close()`, with proper write-error propagation
### Added
- **Dual-framing MCP transport** (`CompatibleStdioServerTransport`): Auto-detects Content-Length (Codex/OpenCode) and newline-delimited JSON (Cursor/Claude Code) framing on the first message, responds in the same format (#207)
- **Lazy CLI module loading**: All CLI subcommands now use `createLazyAction()` to defer heavy imports (tree-sitter, ONNX, KuzuDB) until invocation, significantly improving `gitnexus mcp` startup time (#207)
- **Type-safe lazy actions**: `createLazyAction` uses constrained generics to validate export names against module types at compile time
- **Regression test suite**: 13 unit tests covering transport framing, security hardening, buffer limits, and lazy action loading
### Fixed
- **CALLS edge sourceId alignment**: `findEnclosingFunctionId` now generates IDs with `:startLine` suffix matching node creation format, fixing process detector finding 0 entry points (#194)
- **LRU cache zero maxSize crash**: Guard `createASTCache` against `maxSize=0` when repos have no parseable files (#144)
### Changed
- Transport constructor accepts `NodeJS.ReadableStream` / `NodeJS.WritableStream` (widened from concrete `ReadStream`/`WriteStream`)
- `processReadBuffer` simplified to break on first error instead of stale-buffer retry loop
## [1.3.9] - 2026-03-06
### Fixed
- Aligned CALLS edge sourceId with node ID format in parse worker (#194)
## [1.3.8] - 2026-03-05
### Fixed
- Force-exit after analyze to prevent KuzuDB native cleanup hang (#192)
+52
View File
@@ -0,0 +1,52 @@
<!-- version: 1.2.0 -->
<!--
Metadata: version, last reviewed, scope, model policy, reference docs, changelog.
Last updated: 2026-03-22
-->
Last reviewed: 2026-03-24
**Project:** GitNexus · **Environment:** dev · **Maintainer:** repository maintainers (see GitHub)
Follow **AGENTS.md** for the canonical rules; this file adds Claude Code–specific deltas. Cursor-specific notes live only in `AGENTS.md`.
## Scope
See the **Scope** table in [AGENTS.md](AGENTS.md) for read/write/execute/off-limits boundaries. Cursor-specific workflow notes also live only in AGENTS.md.
## Model Configuration
- **Primary:** Pin per **Claude Code** / Anthropic org policy (explicit model id). Do not rely on an unversioned `latest` alias for governed workflows.
- **Fallback:** As configured in Claude Code (organization default or user override).
- **Notes:** The GitNexus CLI analyzer does not call an LLM.
## Execution Sequence (complex tasks)
Same discipline as [AGENTS.md](AGENTS.md): before large multi-step work, state which **AGENTS.md** / **GUARDRAILS.md** rules apply, current **Scope**, and planned validation commands (`npm test`, `tsc`, etc.). When pausing, summarize progress in the chat or a **local** scratch file (do not add `HANDOFF.md` to the repo), then `/clear` and resume with that summary.
## Claude Code hooks
Prefer **PreToolUse** hooks for hard gates (e.g. tests before `git_commit`). Adapt hook commands to `gitnexus/` npm scripts.
## Context budget
If always-on instructions grow, load deep conventions via conditional reads (e.g. *“When writing new code, read STANDARDS.md”*) instead of pasting long blocks here. In Cursor, prefer `.cursor/index.mdc` plus optional `.cursor/rules/*.mdc` globs (see [AGENTS.md](AGENTS.md) § Context budget).
## Reference Documentation
- **This repository:** [AGENTS.md](AGENTS.md) (Cursor + monorepo notes), [ARCHITECTURE.md](ARCHITECTURE.md), [CONTRIBUTING.md](CONTRIBUTING.md), [GUARDRAILS.md](GUARDRAILS.md).
- **GitNexus:** `.claude/skills/gitnexus/`; MCP and indexed-repo rules live only in [AGENTS.md](AGENTS.md) (`gitnexus:start` … `gitnexus:end`). See **GitNexus rules** below.
## Changelog
| Date | Version | Change |
|------|---------|--------|
| 2026-03-24 | 1.2.0 | Removed duplicated gitnexus:start block and scope table; replaced with pointers to AGENTS.md. |
| 2026-03-23 | 1.1.0 | Updated agent instructions to match AGENTS.md. |
| 2026-03-22 | 1.0.0 | Added structured header and changelog. |
---
## GitNexus rules
GitNexus MCP rules are in the `<!-- gitnexus:start -->` … `<!-- gitnexus:end -->` block in **[AGENTS.md](AGENTS.md)** — load that section when working with MCP tools or the graph index.
-207
View File
@@ -1,207 +0,0 @@
# 🔍 Comprehensive End-to-End Verification Report
## Executive Summary
✅ **ALL SYSTEMS VERIFIED** - The parallel processing implementation is now fully optimized with proper LRU caching, memory management, and cleanup mechanisms.
## 🚨 Critical Issue Found & Fixed
### **THE PROBLEM: LRU Cache Bypass in Parallel Mode**
The parallel processing was **completely bypassing the LRU cache** for the main parsing logic, causing:
- ❌ Every file parsed from scratch (no cache benefits)
- ❌ Memory bloat and performance degradation
- ❌ Sluggish behavior due to redundant processing
### **THE FIX: Cache-First Processing**
✅ Implemented **LRU cache-first processing** in `ParallelParsingProcessor.processFilesInParallel()`:
```typescript
// NEW: Check LRU cache before sending to workers
for (const filePath of filePaths) {
const cacheKey = this.lruCache.generateFileCacheKey(filePath, contentHash);
const cachedResult = this.lruCache.getParsedFile(cacheKey);
if (cachedResult) {
// Use cached result ✅
cachedResults.push({...});
} else {
// Send to workers ✅
uncachedFiles.push(filePath);
}
}
```
---
## 📋 Detailed Verification Results
### 1. ✅ LRU Cache Integration
**Status: VERIFIED & OPTIMIZED**
#### Single-threaded Mode (`ParsingProcessor`):
- ✅ File caching: `lruCache.getParsedFile()` / `setParsedFile()`
- ✅ Query caching: `lruCache.getQueryResult()` / `setQueryResult()`
- ✅ Parser caching: `lruCache.getParser()` / `setParser()`
- ✅ Cache key generation: `generateFileCacheKey()` / `generateQueryCacheKey()`
#### Parallel Mode (`ParallelParsingProcessor`):
- ✅ **FIXED**: Now checks cache before worker processing
- ✅ File caching: Same as single-threaded
- ✅ Worker result caching: Results cached after processing
- ✅ Cache statistics: `lruCache.getStats()` / `getCacheHitRate()`
#### Expected Console Output:
```
ParallelParsingProcessor: Cache hits: X, Files to process: Y
ParallelParsingProcessor: Cache hit for /path/to/file.ts
ParallelParsingProcessor: Total results: Z (X cached, Y processed)
```
### 2. ✅ Memory Management & Cleanup
**Status: VERIFIED & ROBUST**
#### Memory Monitoring:
- ✅ **30-second interval monitoring** in parallel mode
- ✅ **Memory threshold triggers** (500MB → cleanup, 800MB → aggressive cleanup)
- ✅ **AST map size limits** (1000 entries max with cleanup)
- ✅ **LRU cache statistics** logging
#### Cleanup Mechanisms:
- ✅ **Memory Manager**: `memoryManager.clearCache()`
- ✅ **LRU Cache**: `lruCache.clearAll()` / `clearFileCache()` / `clearQueryCache()`
- ✅ **AST Map**: `cleanupASTMap()` with LRU-based eviction
- ✅ **Duplicate Detector**: `duplicateDetector.clear()`
#### Expected Console Output:
```
ParallelParsingProcessor Memory Stats:
- Memory Manager: XXXmb used, YYY files cached
- LRU File Cache: A/200 entries, B.XMB
- LRU Query Cache: C/100 entries, D.XMB
- Cache Hit Rates: File XX.X%, Query YY.Y%
- AST Map Size: ZZZ entries
```
### 3. ✅ Worker Pool Lifecycle & Cleanup
**Status: VERIFIED & SECURE**
#### Worker Pool Management:
- ✅ **Proper initialization** with CPU-optimized settings
- ✅ **Event listener cleanup** on task completion/error
- ✅ **Worker termination** with Promise.all for parallel shutdown
- ✅ **Singleton cleanup** via `FileProcessingPool.shutdownInstance()`
#### Global Cleanup Handlers:
- ✅ **Page unload**: `beforeunload` event → `cleanupAllPools()`
- ✅ **Page hidden**: `visibilitychange` event → cleanup when hidden
- ✅ **Memory pressure**: Automatic cleanup at 80% memory usage
- ✅ **Manual cleanup**: `WebWorkerPoolUtils.cleanupAllPools()`
#### Shutdown Sequence:
```typescript
// ParallelParsingProcessor.shutdown()
1. Clear memory monitor interval
2. Clear AST map & processed files
3. Shutdown worker pool (terminate all workers)
4. Clear LRU caches
5. Clear language parsers
```
### 4. ✅ Cache Consistency Between Modes
**Status: VERIFIED & IDENTICAL**
#### Consistent Cache Key Generation:
- ✅ **Same hash algorithm**: Both use identical `generateContentHash()`
- ✅ **Same cache keys**: `lruCache.generateFileCacheKey(filePath, contentHash)`
- ✅ **Same cache structure**: Identical cache data format
- ✅ **Same LRU service**: Both use `LRUCacheService.getInstance()`
#### Cache Data Format (Both Modes):
```typescript
{
ast: Parser.Tree,
definitions: ParsedDefinition[],
language: string,
lastModified: number,
fileSize: number
}
```
### 5. ✅ Performance Monitoring & Logging
**Status: VERIFIED & COMPREHENSIVE**
#### Parallel Mode Logging:
- ✅ **Cache hit rates**: Shows cached vs processed files
- ✅ **Worker pool stats**: Active workers, completed tasks, errors
- ✅ **Memory statistics**: Real-time memory usage monitoring
- ✅ **Processing times**: Per-file and total processing duration
- ✅ **Progress tracking**: Real-time progress updates
#### Single-threaded Mode Logging:
- ✅ **Memory statistics**: Memory manager stats
- ✅ **Cache statistics**: LRU cache hit rates
- ✅ **Processing stats**: File counts and definitions extracted
---
## 🎯 Performance Improvements Expected
### First Run (Cold Cache):
- **Single-threaded**: Baseline performance
- **Parallel**: Faster due to worker parallelization + caching setup
### Subsequent Runs (Warm Cache):
- **Both modes**: **Dramatically faster** due to cache hits
- **Cache hit ratio**: Should be 80-95% for unchanged files
- **Memory usage**: Stable and controlled via cleanup mechanisms
### Memory Behavior:
- **Before fix**: Unlimited growth → sluggishness
- **After fix**: Controlled growth with automatic cleanup
---
## 🔧 Verification Commands
To verify the fixes are working, look for these console outputs:
### Cache Verification:
```bash
# Should see cache hits on subsequent runs
ParallelParsingProcessor: Cache hits: 150, Files to process: 50
```
### Memory Monitoring:
```bash
# Should see regular memory stats
ParallelParsingProcessor Memory Stats:
- Memory Manager: 245MB used, 1250 files cached
- Cache Hit Rates: File 87.5%, Query 92.3%
```
### Worker Pool Stats:
```bash
# Should see worker efficiency
ParallelParsingProcessor: Worker pool stats: {
activeWorkers: 4,
completedTasks: 200,
failedTasks: 0
}
```
---
## ✅ Conclusion
**ALL CRITICAL ISSUES RESOLVED:**
1. **🚨 LRU Cache Bypass** → ✅ **Cache-first processing implemented**
2. **🚨 Memory Leaks** → ✅ **Comprehensive cleanup mechanisms**
3. **🚨 Worker Pool Leaks** → ✅ **Proper lifecycle management**
4. **🚨 Inconsistent Caching** → ✅ **Identical cache behavior**
5. **🚨 Poor Monitoring** → ✅ **Comprehensive performance logging**
**The sluggishness should be significantly reduced** because:
- ✅ **First run**: Files get cached after processing
- ✅ **Subsequent runs**: Most files served from cache (near-instant)
- ✅ **Memory management**: Automatic cleanup prevents bloat
- ✅ **Worker efficiency**: Only uncached files sent to workers
**🚀 Ready for production use with optimal performance!**
-75
View File
@@ -1,75 +0,0 @@
# GitNexus Configuration
## Feature Flags
GitNexus uses feature flags to control the visibility of experimental and advanced features. By default, the UI is kept clean and simple for end users.
### Available Feature Flags
- `showEngineSelector` - Show engine selection dropdown
- `showEnginePerformanceInfo` - Display performance comparison data
- `showEngineCapabilities` - Show engine capabilities grid
- `enableNextGenEngine` - Enable next-generation processing engine
- `enableEngineComparison` - Run performance comparisons between engines
- `enableDebugMode` - Show debugging information
- `showProcessingDetails` - Display detailed processing information
### Enabling Features for Development
To enable advanced features during development, modify `/src/config/feature-flags.ts`:
```typescript
export const getFeatureFlags = (): FeatureFlags => {
if (import.meta.env.DEV) {
return {
...defaultFeatureFlags,
// Enable engine features for development
showEngineSelector: true,
showEnginePerformanceInfo: true,
showEngineCapabilities: true,
enableDebugMode: true,
showProcessingDetails: true,
};
}
return defaultFeatureFlags;
};
```
### Production Configuration
For production builds, keep feature flags disabled to maintain a clean user interface:
```typescript
export const defaultFeatureFlags: FeatureFlags = {
showEngineSelector: false, // Hidden from end users
showEnginePerformanceInfo: false, // Hidden from end users
showEngineCapabilities: false, // Hidden from end users
enableNextGenEngine: true, // Enabled behind the scenes
enableEngineComparison: false, // Disabled to save resources
enableDebugMode: false,
showProcessingDetails: false,
};
```
### Environment Variables
You can also control features via environment variables:
```bash
# Enable engine selector in development
VITE_SHOW_ENGINE_SELECTOR=true npm run dev
# Enable all debug features
VITE_DEBUG_MODE=true npm run dev
```
## Layout Configuration
The new layout prioritizes the graph visualization:
- **Left Sidebar (400px)**: Processing status, chat interface, and actions
- **Right Side (flex)**: Full graph visualization
- **Mobile**: Responsive single-column layout
This provides maximum space for the graph while keeping the chat interface easily accessible.
+50
View File
@@ -0,0 +1,50 @@
# Contributing to GitNexus
How to propose changes, run checks locally, and open pull requests.
## License
This project uses the [PolyForm Noncommercial License 1.0.0](https://polyformproject.org/licenses/noncommercial/1.0.0/). By contributing, you agree your contributions are licensed under the same terms unless stated otherwise.
## Where to discuss
- **Issues & feature ideas:** use [GitHub Issues](https://github.com/abhigyanpatwari/GitNexus/issues) for the upstream repo, or your fork’s tracker if you work from a fork.
- **Community:** see the Discord link in the root [README.md](README.md).
## Development setup
1. Clone the repository.
2. **CLI / MCP package:** `cd gitnexus && npm install && npm run build`
3. **Web UI (if needed):** `cd gitnexus-web && npm install`
4. Run tests as described in [TESTING.md](TESTING.md).
## Branch and pull requests
- Use short-lived branches off the default branch of the repo you are targeting.
- Prefer **conventional commits** (short prefix + description), for example:
```text
feat: add graph export option
fix: correct MCP tool schema for query
test: cover cluster merge edge case
docs: clarify analyze flags
```
- **PR title:** `[area] Short description` (e.g. `[cli] Fix index refresh race`).
- **PR description:** what changed, why, how to verify (commands), and any risk or rollback notes.
## Before you open a PR
- [ ] Tests pass for the packages you touched (`gitnexus` and/or `gitnexus-web`).
- [ ] Typecheck passes: `npx tsc --noEmit` in `gitnexus/` and `npx tsc -b --noEmit` in `gitnexus-web/`.
- [ ] No secrets, tokens, or machine-specific paths committed.
- [ ] Documentation updated if behavior or public CLI/MCP contract changes.
- [ ] Pre-commit hook runs clean (`.husky/pre-commit` — typecheck + unit tests for staged packages).
## Code review
Maintainers may request changes for correctness, tests, performance, or consistency with existing patterns. Keeping diffs focused makes review faster.
## AI-assisted contributions
If you use coding agents, follow project context files (e.g. `AGENTS.md`, `CLAUDE.md`) and avoid drive-by refactors unrelated to the issue. Prefer incremental, test-backed changes.
-148
View File
@@ -1,148 +0,0 @@
# 🔍 FINAL COMPREHENSIVE VERIFICATION REPORT
## ✅ **VERIFICATION COMPLETE - ALL CRITICAL ISSUES RESOLVED**
After an extremely thorough examination, both single-threaded and parallel processing modes are now **fully synchronized** and **memory-optimized**.
---
## 🚨 **CRITICAL ISSUES FOUND AND FIXED**
### **Issue #1: Wrong Pipeline Selection in Worker** 🔥 **CRITICAL**
**Problem**: Worker was always using `GraphPipeline` instead of checking the feature flag.
**Impact**: "Parallel processing" was actually running single-threaded code.
**Fix**: ✅ Worker now correctly selects `ParallelGraphPipeline` when parallel mode is enabled.
### **Issue #2: LRU Cache Completely Disabled in Single-Threaded Mode** 🔥 **CRITICAL**
**Problem**: Single-threaded processor had all LRU caching commented out as "TEMPORARILY DISABLED FOR DEBUGGING".
**Impact**: Single-threaded mode had no caching, parallel mode did - major performance inconsistency.
**Fix**: ✅ Re-enabled all LRU cache operations in single-threaded processor.
### **Issue #3: Incorrect Duplicate Detection** 🔥 **CRITICAL**
**Problem**:
- Single-threaded: `checkAndMark()` (correct)
- Parallel: `isDuplicate()` (wrong - doesn't mark as processed)
**Fix**: ✅ Changed parallel processor to use `checkAndMark()`.
### **Issue #4: Property Format Inconsistency** 🔶 **MEDIUM**
**Problem**: Parallel processor stored arrays as comma-separated strings.
**Fix**: ✅ Made both processors store identical array formats.
### **Issue #5: Memory Leak Prevention** 🔶 **MEDIUM**
**Problem**: Worker pools, event listeners, and AST maps could accumulate without cleanup.
**Fix**: ✅ Comprehensive memory management implemented.
---
## 📊 **FINAL VERIFICATION CHECKLIST**
### **✅ Pipeline Architecture**
- [x] Worker correctly selects `GraphPipeline` vs `ParallelGraphPipeline` based on feature flag
- [x] Both pipelines use identical 4-pass structure (Structure → Parsing → Import → Call)
- [x] Both pipelines use same processors (except parsing processor)
- [x] Progress callbacks properly integrated
### **✅ LRU Cache Consistency**
- [x] Both modes use `LRUCacheService.getInstance()`
- [x] File caching enabled in both modes (200 max, 1 hour TTL)
- [x] Query caching enabled in both modes (1000 max, 15 min TTL)
- [x] Parser caching enabled in both modes (10 max, 24 hours TTL)
- [x] Cache hit rate tracking in both modes
### **✅ Data Processing Consistency**
- [x] Identical duplicate detection logic (`checkAndMark()`)
- [x] Identical node ID generation
- [x] Identical node label mapping
- [x] Identical property formats (arrays as arrays, not strings)
- [x] Identical relationship creation
### **✅ Memory Management**
- [x] Worker pools properly terminated with event listener cleanup
- [x] AST maps size-limited (1000 entries) with automatic cleanup
- [x] LRU caches automatically manage memory (100MB total limit)
- [x] Singleton instances properly cleaned up
- [x] Memory monitoring with automatic triggers (500MB threshold)
- [x] Global cleanup handlers for page unload/visibility changes
### **✅ Error Handling**
- [x] Graceful worker termination on errors
- [x] Proper resource cleanup in finally blocks
- [x] Error recovery without resource leaks
---
## 🎯 **EXPECTED BEHAVIOR AFTER FIXES**
### **Single-Threaded Mode (`isParallelParsingEnabled() = false`)**:
- Uses `GraphPipeline` with `ParsingProcessor`
- Sequential file processing on main thread
- Full LRU caching enabled
- Direct Tree-sitter AST parsing
### **Parallel Mode (`isParallelParsingEnabled() = true`)**:
- Uses `ParallelGraphPipeline` with `ParallelParsingProcessor`
- Worker pool parallel processing (2-8 workers based on CPU cores)
- Full LRU caching enabled
- Worker-based parsing + main thread AST recreation
### **Identical Output Guaranteed**:
Both modes will now produce:
- ✅ **Same node counts** by type (Function, Class, Variable, etc.)
- ✅ **Same relationship counts** by type (CONTAINS, DEFINES, IMPORTS, CALLS)
- ✅ **Same import relationships** - no more "missing import relationships"
- ✅ **Same function call relationships** - no more "missing call relationships"
- ✅ **Same graph connectivity** - proper relationships between files and definitions
---
## 🚀 **PERFORMANCE IMPROVEMENTS**
### **Memory Usage**:
- **LRU Cache**: Automatic memory management with 100MB limit
- **AST Maps**: Size-limited to 1000 entries with cleanup
- **Worker Pools**: Proper termination prevents accumulation
- **Global Monitoring**: Memory usage tracked every 30 seconds
### **Processing Speed**:
- **Single-threaded**: Now benefits from LRU caching (was disabled)
- **Parallel**: 2-8x speedup on large codebases + LRU caching benefits
- **Cache Hit Rates**: Both modes show file/query cache performance
### **Resource Management**:
- **No Memory Leaks**: All resources properly cleaned up
- **Automatic Cleanup**: Triggers at 80% memory usage
- **Graceful Shutdown**: Proper cleanup on page close/tab switch
---
## 🧪 **TESTING VERIFICATION**
To verify the fixes work:
1. **Process the same codebase** with both modes
2. **Compare console output** - should show identical relationship counts
3. **Check for these success indicators**:
```
✅ Relationships by type: {CONTAINS: X, DEFINES: Y, IMPORTS: Z, CALLS: W}
✅ No warnings about "missing import relationships"
✅ No warnings about "missing function call relationships"
✅ Memory usage stays stable during processing
✅ LRU cache hit rates displayed in both modes
```
4. **Performance comparison**:
- Single-threaded: Should be faster than before (LRU cache now enabled)
- Parallel: Should be significantly faster on large codebases
---
## 🎉 **CONCLUSION**
The parallel processing implementation now produces **100% identical output** to single-threaded processing while maintaining all performance benefits:
- **✅ Data Consistency**: Identical graph structure and relationships
- **✅ Memory Efficiency**: Comprehensive leak prevention and monitoring
- **✅ Performance**: LRU caching enabled in both modes + parallel speedup
- **✅ Reliability**: Proper error handling and resource cleanup
**The system is now production-ready with both processing modes fully synchronized!** 🚀
+88
View File
@@ -0,0 +1,88 @@
# Guardrails — GitNexus (repo + agents)
Rules for **human contributors** and **AI agents** working on this codebase or publishing artifacts. These complement `AGENTS.md` / `CLAUDE.md` (which focus on GitNexus-in-GitNexus workflows).
## Scope (typical agent session)
When automating changes in this repository, treat scope as **least privilege**:
- **Read:** Source, tests, docs, public config as needed for the task.
- **Write:** Only files required for the requested fix or feature; avoid unrelated formatting or refactors.
- **Execute:** Tests, typecheck, and documented CLI commands; do not run destructive commands on user data outside the repo without explicit approval.
- **Off-limits:** Other people’s machines, production deployments you don’t own, and credentials you didn’t receive permission to use.
Adjust explicitly if the maintainer defines a different scope for a task.
---
## Non-negotiables
1. **Never commit secrets** — API keys, tokens, `.env` with real values, private URLs, or session cookies. Use `.env.example` with placeholders only.
2. **Never rename symbols with blind find-and-replace** when working in a GitNexus-indexed project — use the **`rename` MCP tool** with **`dry_run: true` first**, then review `graph` vs `text_search` edits. (There is no separate `gitnexus rename` CLI; renaming goes through MCP or editor integration.)
3. **Run impact analysis before editing shared symbols** — use **`impact`** (upstream) for functions/classes/methods others call; do not ignore **HIGH** / **CRITICAL** risk without maintainer sign-off.
4. **Prefer `detect_changes` before commit** — confirm diffs map to expected symbols/processes when the graph is available.
5. **Preserve embeddings** — if `.gitnexus/meta.json` shows embeddings, run `npx gitnexus analyze --embeddings` when refreshing the index; plain `analyze` can drop them.
---
## Signs (recurring failure patterns)
Use this format: **Trigger → Instruction → Reason**.
Append new Signs here when the same mistake repeats (e.g. CI broken twice the same way).
### Sign: Stale graph after edits
- **Trigger:** MCP or resources warn the index is behind `HEAD`, or code search doesn’t match latest commit.
- **Instruction:** Run `npx gitnexus analyze` from the repo root (plus `--embeddings` if the project used them).
- **Reason:** Tools query LadybugDB built at last analyze; git changes are invisible until re-indexed.
### Sign: Embeddings vanished after analyze
- **Trigger:** Semantic search quality drops; `stats.embeddings` in `.gitnexus/meta.json` is 0 after a refresh.
- **Instruction:** Re-run `npx gitnexus analyze --embeddings` and confirm `meta.json` reflects stored embeddings.
- **Reason:** Embedding generation is opt-in; analyze without the flag does not preserve prior vectors.
### Sign: MCP lists no repos
- **Trigger:** MCP stderr says no indexed repos.
- **Instruction:** Run `npx gitnexus analyze` in the target repository; verify `npx gitnexus list` shows it.
- **Reason:** The MCP server discovers repos via `~/.gitnexus/registry.json`, populated by analyze.
### Sign: Wrong repo in multi-repo setups
- **Trigger:** Query/impact results clearly belong to another project.
- **Instruction:** Call `list_repos`, then pass **`repo`** on subsequent tools (or use per-workspace MCP config).
- **Reason:** Default target may be ambiguous when multiple repos are registered.
### Sign: LadybugDB lock / “database busy”
- **Trigger:** Errors opening `.gitnexus/lbug` while MCP and analyze both run.
- **Instruction:** Stop overlapping processes; one writer at a time. Retry analyze or restart MCP.
- **Reason:** Embedded DB expects single-process ownership of the store.
---
## Publishing & supply chain
- **npm:** Do not publish from unreviewed automation; follow maintainer release process. Bump version intentionally; tag releases to match `package.json`.
- **Dependencies:** Prefer minimal, auditable changes to `package.json`; run tests and CI after lockfile updates.
- **License:** This project ships under **PolyForm Noncommercial 1.0.0** — do not relicense or imply a different license in docs or metadata without maintainer approval.
---
## Escalation
Stop and ask a **human maintainer** when:
- Impact analysis shows **HIGH** / **CRITICAL** risk and the task still requires the change.
- You need to alter **CI**, **release**, or **security-sensitive** config.
- Requirements conflict (e.g. “speed up analyze” vs “must keep all embeddings on huge repo”).
- You are unsure whether data loss is acceptable (`clean`, forced migrations, schema changes).
---
## Related docs
- [ARCHITECTURE.md](ARCHITECTURE.md) — components and data flow.
- [RUNBOOK.md](RUNBOOK.md) — commands for recovery.
- [CONTRIBUTING.md](CONTRIBUTING.md) — PR and commit expectations.
-153
View File
@@ -1,153 +0,0 @@
# How to Enable KuzuDB COPY Feature
## Quick Start
To enable the new COPY-based bulk loading feature in GitNexus:
### 1. Enable Feature Flag
Edit your `gitnexus.config.ts` file and set:
```typescript
export default {
// ... other config
features: {
// ... other features
enableKuzuCopy: true, // Enable COPY-based bulk loading
// ... other features
}
}
```
### 2. Verify Configuration
The feature flag can also be enabled via environment variables or runtime configuration. Check your current config with:
```javascript
// In browser console
import { isKuzuCopyEnabled } from './src/config/features.ts';
console.log('COPY enabled:', isKuzuCopyEnabled());
```
### 3. Monitor Performance
Once enabled, you'll see different log messages in the browser console:
**COPY Success:**
```
🚀 COPY: Starting COPY-based commit of 150 Function nodes
📝 Written 12543 bytes to /temp_Function_nodes_1703123456789.csv
✅ COPY: Successfully loaded 150 Function nodes via COPY
```
**COPY Fallback:**
```
⚠️ COPY failed for Function, falling back to MERGE: FS API not available
🔄 BATCH: Committing 150 Function nodes in single query
✅ BATCH: Successfully committed all nodes in batches
```
## Performance Expectations
### Small Repositories (< 100 files)
- **Improvement**: 2-3x faster
- **COPY vs MERGE**: Minimal difference due to overhead
### Medium Repositories (100-500 files)
- **Improvement**: 5-7x faster
- **COPY vs MERGE**: Significant improvement in batch operations
### Large Repositories (1000+ files)
- **Improvement**: 10-15x faster
- **COPY vs MERGE**: Dramatic improvement, especially for complex codebases
## Troubleshooting
### COPY Not Working
1. **Check Feature Flag**: Ensure `enableKuzuCopy: true` in config
2. **Browser Compatibility**: COPY requires Web Workers support
3. **KuzuDB Version**: Ensure kuzu-wasm@0.11.1 or later
4. **FS API**: Check browser console for FS API availability
### Fallback to MERGE
The system automatically falls back to MERGE if:
- FS API is not available
- COPY statement execution fails
- CSV generation encounters errors
- KuzuDB schema issues
This ensures **zero downtime** and **no data loss**.
### Common Error Messages
| Error | Cause | Solution |
|-------|-------|----------|
| `FS API not available` | Browser/environment limitation | Normal fallback, no action needed |
| `COPY failed: Table X does not exist` | Schema not initialized | Check KuzuDB schema initialization |
| `CSV generation failed` | Data format issue | Check node/relationship properties |
## Monitoring & Metrics
### Success Indicators
- ✅ COPY success messages in console
- 📊 Faster ingestion times
- 💾 Lower memory usage during batch operations
### Performance Comparison
```javascript
// Before (MERGE): ~30 seconds for 1000 nodes
🔄 BATCH: Committing 1000 Function nodes in single query
✅ BATCH: Successfully committed all nodes in batches (29.8s)
// After (COPY): ~3 seconds for 1000 nodes
🚀 COPY: Starting COPY-based commit of 1000 Function nodes
✅ COPY: Successfully loaded 1000 Function nodes via COPY (2.9s)
```
## Rollback Plan
To disable COPY and revert to MERGE:
```typescript
export default {
features: {
enableKuzuCopy: false, // Disable COPY feature
}
}
```
Changes take effect immediately on next repository ingestion.
## Advanced Configuration
### Batch Size Optimization
The system automatically calculates optimal batch sizes, but you can tune performance:
```typescript
// In KuzuKnowledgeGraph initialization
const kuzuGraph = new KuzuKnowledgeGraph(queryEngine, {
batchSize: 200, // Increase for better COPY performance
autoCommit: true, // Keep enabled for COPY
enableCache: true // Recommended for performance
});
```
### Memory Management
For very large repositories, the system uses chunked processing:
- **< 1000 items**: Single CSV generation
- **1000-5000 items**: 1000-item chunks
- **> 5000 items**: 1500-item chunks
## Next Steps
1. **Enable Feature**: Set `enableKuzuCopy: true`
2. **Test Small Repository**: Verify functionality with a small codebase
3. **Monitor Performance**: Check console logs for COPY success
4. **Scale Up**: Test with larger repositories
5. **Report Issues**: Document any fallback scenarios or performance issues
The COPY feature is designed to be **safe**, **fast**, and **transparent** - it should work seamlessly with your existing GitNexus workflow while providing significant performance improvements.
-392
View File
@@ -1,392 +0,0 @@
# KuzuDB COPY Implementation Guide for GitNexus
## Executive Summary
**Status**: ✅ **PROVEN WORKING** - COPY approach successfully tested and verified
**Performance**: 5-10x faster than current MERGE batch operations
**Recommendation**: Implement with fallback to current MERGE approach
## Test Results Summary
| Component | Status | Details |
| ----------------- | ---------- | ----------------------------------------------- |
| KuzuDB WASM Init | ✅ Working | Initializes successfully in browser environment |
| FS.writeFile | ✅ Working | Successfully writes CSV to WASM filesystem |
| FS.readFile | ✅ Working | Reads back data (fixed data type handling) |
| COPY Statements | ✅ Working | Bulk loads data from CSV files |
| Data Verification | ✅ Working | All data queryable after COPY operations |
## Technical Implementation Details
### 1. Environment Requirements
**Working Environment**: Browser with Web Workers support
**Failed Environment**: Node.js (Worker2 constructor not available)
**KuzuDB Version**: kuzu-wasm@0.11.1
```javascript
// Confirmed working initialization
await kuzu.init();
const db = new kuzu.Database(''); // In-memory database
const conn = new kuzu.Connection(db);
```
### 2. FS API Implementation
**Key Finding**: FS API is available and functional in browser environment
```javascript
// Verified working pattern
await kuzu.FS.writeFile('/path/file.csv', csvData);
const readData = await kuzu.FS.readFile('/path/file.csv');
```
**Critical Issue Solved**: FS.readFile data type handling
- **Problem**: `readData.substring is not a function`
- **Cause**: FS.readFile returns Buffer/Uint8Array, not string
- **Solution**: Proper data type conversion
```javascript
// Fixed data handling
let dataStr;
if (typeof readData === 'string') {
dataStr = readData;
} else if (readData instanceof Uint8Array || readData instanceof ArrayBuffer) {
dataStr = new TextDecoder().decode(readData);
} else if (readData && readData.toString) {
dataStr = readData.toString();
} else {
dataStr = String(readData);
}
```
### 3. COPY Statement Implementation
**Verified Working Pattern**:
```javascript
// 1. Write CSV to WASM filesystem
await kuzu.FS.writeFile('/users.csv', csvData);
// 2. Execute COPY statement
const result = await conn.query("COPY User FROM '/users.csv'");
await result.close();
// 3. Data is immediately available for queries
const verifyResult = await conn.query('MATCH (u:User) RETURN count(u)');
```
**CSV Format Requirements**:
- Standard CSV format (comma-separated)
- Header row with column names matching schema
- Proper escaping for special characters
- No additional formatting needed
### 4. Schema Management
**Critical Issue Solved**: Table existence conflicts
- **Problem**: `Binder exception: User already exists in catalog`
- **Cause**: Multiple test runs without cleanup
- **Solution**: Drop tables before creation
```javascript
// Required cleanup pattern
try {
await conn.query('DROP TABLE User IF EXISTS');
await conn.query('DROP TABLE City IF EXISTS');
} catch (cleanupError) {
// Tables might not exist, ignore errors
}
// Then create fresh schema
await conn.query('CREATE NODE TABLE User(name STRING, age INT64, PRIMARY KEY (name))');
```
## GitNexus Integration Strategy
### 1. CSV Generator Service
**Location**: `src/core/kuzu/csv-generator.ts`
```typescript
export class GitNexusCSVGenerator {
static generateNodeCSV(nodes: GraphNode[], label: string): string {
const filteredNodes = nodes.filter(node => node.label === label);
if (filteredNodes.length === 0) return '';
// Get all unique properties for schema
const allProps = new Set(['id']);
filteredNodes.forEach(node => {
Object.keys(node.properties).forEach(key => allProps.add(key));
});
const columns = Array.from(allProps);
const header = columns.join(',');
const rows = filteredNodes.map(node => {
return columns.map(col => {
if (col === 'id') return this.escapeCSV(node.id);
const value = node.properties[col];
return value !== undefined ? this.escapeCSV(value) : '';
}).join(',');
});
return [header, ...rows].join('\n');
}
static generateRelationshipCSV(relationships: GraphRelationship[], type: string): string {
// Similar implementation for relationships
// Include source, target, and properties
}
static escapeCSV(value: any): string {
if (value === null || value === undefined) return '';
const str = String(value);
if (str.includes(',') || str.includes('"') || str.includes('\n')) {
return '"' + str.replace(/"/g, '""') + '"';
}
return str;
}
}
```
### 2. Enhanced KuzuKnowledgeGraph
**Location**: `src/core/graph/kuzu-knowledge-graph.ts`
**Replace current batch methods**:
```typescript
// Current method (keep as fallback)
private async commitNodesBatchWithMERGE(label: string, nodes: GraphNode[]): Promise<void> {
// Existing MERGE implementation
}
// New COPY method
private async commitNodesBatchWithCOPY(label: string, nodes: GraphNode[]): Promise<void> {
try {
// Generate CSV
const csvData = GitNexusCSVGenerator.generateNodeCSV(nodes, label);
// Write to WASM filesystem
const csvPath = `/temp_${label}_${Date.now()}.csv`;
await this.kuzuModule.FS.writeFile(csvPath, csvData);
// Execute COPY statement
const result = await this.queryEngine.executeQuery(`COPY ${label} FROM '${csvPath}'`);
await result.close();
console.log(`✅ COPY: Successfully loaded ${nodes.length} ${label} nodes`);
} catch (error) {
console.error(`❌ COPY failed for ${label}:`, error);
throw error;
}
}
// Main batch method with fallback
private async commitNodesBatch(label: string, nodes: GraphNode[]): Promise<void> {
try {
await this.commitNodesBatchWithCOPY(label, nodes);
} catch (copyError) {
console.warn(`⚠️ COPY failed, falling back to MERGE for ${label}:`, copyError.message);
await this.commitNodesBatchWithMERGE(label, nodes);
}
}
```
### 3. KuzuDB Module Access
**Location**: `src/core/kuzu/kuzu-npm-integration.ts`
**Add FS API access**:
```typescript
// Expose FS API in KuzuInstance interface
export interface KuzuInstance {
// ... existing methods
getFS(): any; // Access to FS API
}
// In createKuzuInstance()
return {
// ... existing methods
getFS(): any {
return kuzuModule.default.FS;
}
};
```
### 4. Integration Points
**Files to modify**:
1. `src/core/graph/kuzu-knowledge-graph.ts` - Add COPY batch methods
2. `src/core/kuzu/kuzu-npm-integration.ts` - Expose FS API
3. `src/core/kuzu/csv-generator.ts` - New CSV generation service
4. `src/config/features.ts` - Add COPY feature flag
**Feature Flag**:
```typescript
export function isKuzuCopyEnabled(): boolean {
return cachedConfig?.features.enableKuzuCopy ?? false;
}
```
## Performance Characteristics
### Current MERGE Approach
- **Operations**: N individual MERGE statements per batch
- **Memory**: String concatenation for large queries
- **Database Load**: N query parsing operations
- **Scalability**: Linear degradation with batch size
### COPY Approach
- **Operations**: 1 FS write + 1 COPY statement per batch
- **Memory**: Streaming CSV generation
- **Database Load**: 1 optimized bulk operation
- **Scalability**: Constant time regardless of batch size
### Expected Performance Gains
- **Small batches (10-50 items)**: 2-3x improvement
- **Medium batches (100-500 items)**: 5-7x improvement
- **Large batches (1000+ items)**: 10-15x improvement
## Error Handling Strategy
### 1. Environment Detection
```typescript
function isCopySupported(): boolean {
return !!(kuzu.FS && kuzu.FS.writeFile && typeof kuzu.FS.writeFile === 'function');
}
```
### 2. Graceful Degradation
```typescript
if (isCopySupported() && isKuzuCopyEnabled()) {
try {
await commitNodesBatchWithCOPY(label, nodes);
} catch (copyError) {
await commitNodesBatchWithMERGE(label, nodes);
}
} else {
await commitNodesBatchWithMERGE(label, nodes);
}
```
### 3. Error Categories
- **FS Errors**: File system operations (writeFile/readFile)
- **COPY Errors**: SQL execution errors (syntax, schema mismatch)
- **Data Errors**: CSV format or encoding issues
## Testing Strategy
### 1. Unit Tests
- CSV generation with various data types
- Error handling for malformed data
- Schema compatibility validation
### 2. Integration Tests
- End-to-end COPY workflow
- Fallback mechanism verification
- Performance benchmarking
### 3. Browser Compatibility
- Test across different browsers
- Verify Web Worker support
- Memory usage monitoring
## Deployment Considerations
### 1. Feature Flag Rollout
- **Phase 1**: Internal testing with feature flag disabled
- **Phase 2**: Gradual rollout to subset of users
- **Phase 3**: Full deployment with monitoring
### 2. Monitoring Metrics
- COPY success/failure rates
- Performance improvement measurements
- Memory usage comparison
- Error frequency and types
### 3. Rollback Strategy
- Feature flag can instantly disable COPY approach
- Automatic fallback ensures no service disruption
- Existing MERGE approach remains fully functional
## Known Limitations
### 1. Environment Constraints
- **Browser Only**: COPY approach requires browser environment
- **Web Workers**: Depends on Web Worker support
- **Memory**: WASM filesystem is in-memory only
### 2. Data Constraints
- **CSV Format**: Data must be CSV-compatible
- **File Paths**: Limited to WASM filesystem paths
- **Encoding**: UTF-8 encoding required
### 3. Schema Constraints
- **Table Existence**: Tables must exist before COPY
- **Column Matching**: CSV columns must match schema
- **Data Types**: Proper type conversion required
## Future Enhancements
### 1. Streaming CSV Generation
- Process large datasets without loading into memory
- Incremental file writing for very large batches
### 2. Parallel COPY Operations
- Multiple concurrent COPY statements
- Batch processing optimization
### 3. Advanced Error Recovery
- Partial batch recovery on COPY failures
- Detailed error reporting and diagnostics
## Implementation Checklist
- [ ] Create CSV generator service
- [ ] Implement COPY-based bulk loader
- [ ] Add FS API access to KuzuInstance
- [ ] Implement fallback strategy
- [ ] Add feature flag support
- [ ] Create comprehensive tests
- [ ] Performance benchmarking
- [ ] Documentation updates
- [ ] Gradual rollout plan
- [ ] Monitoring and alerting setup
## Conclusion
The COPY approach has been **proven to work** through comprehensive testing. Implementation should proceed with:
1. **Immediate**: CSV generator and COPY bulk loader
2. **Short-term**: Feature flag and fallback mechanism
3. **Long-term**: Performance optimization and monitoring
This implementation will provide significant performance improvements for GitNexus, especially for large repository ingestion scenarios.
File diff suppressed because it is too large Load Diff
-213
View File
@@ -1,213 +0,0 @@
# KuzuDB Integration Status Report
## 🎉 **IMPLEMENTATION COMPLETE!**
The full KuzuDB integration has been successfully implemented according to the implementation plan. The system now supports **dual-write functionality** where data is written to both JSON (primary) and KuzuDB (secondary) storage systems simultaneously.
---
## ✅ **What's Been Implemented**
### **Phase 1: Foundation Setup - COMPLETE**
- ✅ **KuzuDB WASM Loader** (`src/core/kuzu/kuzu-loader.ts`) - Full implementation
- ✅ **KuzuDB Query Engine** (`src/core/graph/kuzu-query-engine.ts`) - Complete with caching, transactions, performance monitoring
- ✅ **KuzuDB Knowledge Graph** (`src/core/graph/kuzu-knowledge-graph.ts`) - Full implementation with batching and caching
- ✅ **KuzuDB Schema Manager** (`src/core/kuzu/kuzu-schema.ts`) - Complete schema definitions for all node and relationship types
- ✅ **Feature Flag Integration** - Full KuzuDB feature flag support
### **Phase 2: Parallel Storage Implementation - COMPLETE**
- ✅ **KuzuProcessorBase** - Abstract base class with dual-write pattern, transaction management, and statistics
- ✅ **Enhanced StructureProcessor** - Dual-write support for Project, Folder, File nodes and CONTAINS relationships
- ✅ **Enhanced ParsingProcessor** - Dual-write support for all definition nodes and relationships
- ✅ **Enhanced ImportProcessor** - Dual-write support for IMPORTS relationships
- ✅ **Enhanced CallProcessor** - Dual-write support for CALLS relationships
### **Core Features Implemented**
- ✅ **Dual-Write Pattern** - Data written to both JSON and KuzuDB simultaneously
- ✅ **Transaction Management** - Begin, commit, rollback support
- ✅ **Error Handling** - Graceful degradation to JSON-only mode
- ✅ **Performance Monitoring** - Comprehensive statistics and timing metrics
- ✅ **Batch Processing** - Optimized batch operations for better performance
- ✅ **Caching System** - LRU cache for improved query performance
- ✅ **Schema Validation** - Complete schema definitions for all node and relationship types
---
## 🏗️ **Architecture Overview**
```
┌─────────────────────────────────────────────────────────────────┐
│ GitNexus KuzuDB Integration │
├─────────────────────────────────────────────────────────────────┤
│ │
│ ┌─────────────────┐ ┌─────────────────┐ ┌──────────────┐ │
│ │ JSON Storage │ │ KuzuDB Storage │ │ Feature Flags│ │
│ │ (Primary) │ │ (Secondary) │ │ (Control) │ │
│ └─────────────────┘ └─────────────────┘ └──────────────┘ │
│ │ │ │ │
│ └───────────────────────┼──────────────────────┘ │
│ │ │
│ ┌─────────────────────────────────────────────────────────────┐ │
│ │ KuzuProcessorBase │ │
│ │ • Dual-write pattern │ │
│ │ • Transaction management │ │
│ │ • Error handling & graceful degradation │ │
│ │ • Performance monitoring & statistics │ │
│ └─────────────────────────────────────────────────────────────┘ │
│ │ │ │ │ │
│ ┌──────────────┐ ┌──────────────┐ ┌──────────────┐ ┌──────────┐ │
│ │ Structure │ │ Parsing │ │ Import │ │ Call │ │
│ │ Processor │ │ Processor │ │ Processor │ │Processor │ │
│ └──────────────┘ └──────────────┘ └──────────────┘ └──────────┘ │
│ │
└─────────────────────────────────────────────────────────────────┘
```
---
## 🚀 **How to Use KuzuDB Integration**
### **1. Enable KuzuDB (Currently Disabled by Default)**
```typescript
import { featureFlags } from './src/config/feature-flags';
// Enable KuzuDB integration
featureFlags.enableKuzuDB();
// Check status
console.log('KuzuDB enabled:', featureFlags.getFlag('enableKuzuDB'));
```
### **2. Current Storage Behavior**
**With KuzuDB Disabled (Default):**
- ✅ Data stored in JSON format (existing functionality)
- ✅ All processors work as before
- ✅ No performance impact
**With KuzuDB Enabled:**
- ✅ Data written to **both** JSON and KuzuDB simultaneously
- ✅ JSON remains primary storage (no breaking changes)
- ✅ KuzuDB failures gracefully degrade to JSON-only mode
- ✅ Enhanced logging and statistics available
### **3. Enhanced Console Output**
When KuzuDB is enabled, you'll see enhanced logging:
```
📁 Processing structure for MyProject with 150 paths...
🚀 Initializing KuzuDB integration...
✅ KuzuDB integration initialized successfully.
✅ Structure processing completed. Hidden 45 items from display.
📊 StructureProcessor Statistics:
Total Nodes Processed: 105
Total Relationships Processed: 104
KuzuDB Nodes Written: 105
KuzuDB Relationships Written: 104
KuzuDB Errors: 0
Processing Time: 1,234.56ms
```
---
## 📊 **Current Status**
| Component | Status | Notes |
|-----------|--------|-------|
| **KuzuDB WASM Loader** | ✅ Complete | Ready for WASM binary integration |
| **Query Engine** | ✅ Complete | Full Cypher query support, caching, transactions |
| **Knowledge Graph** | ✅ Complete | Drop-in replacement for SimpleKnowledgeGraph |
| **Schema Manager** | ✅ Complete | All node and relationship types defined |
| **Dual-Write Pattern** | ✅ Complete | All 4 processors support dual-write |
| **Feature Flags** | ✅ Complete | Full control over KuzuDB integration |
| **Error Handling** | ✅ Complete | Graceful degradation to JSON-only mode |
| **Performance Monitoring** | ✅ Complete | Comprehensive statistics and timing |
| **Transaction Management** | ✅ Complete | ACID compliance with rollback support |
---
## 🔧 **What's Missing (Optional Enhancements)**
1. **KuzuDB WASM Binary**: Need to add the actual KuzuDB WASM file to `public/kuzu/`
2. **Query Migration**: Phase 3 implementation (read operations from KuzuDB)
3. **UI Integration**: Update UI components to use KuzuDB queries
4. **Advanced Analytics**: Graph algorithms and complex queries
---
## 🎯 **Key Benefits Achieved**
### **1. Zero Breaking Changes**
- All existing functionality preserved
- JSON storage remains primary
- Backward compatibility maintained
### **2. Production-Ready Error Handling**
- KuzuDB failures don't break the system
- Graceful degradation to JSON-only mode
- Comprehensive error logging
### **3. Performance & Monitoring**
- Detailed statistics for all operations
- Performance timing and success rates
- Transaction management with rollback
### **4. Scalable Architecture**
- Dual-write pattern supports gradual migration
- Feature flags enable controlled rollout
- Extensible base classes for future enhancements
---
## 🧪 **Testing the Integration**
### **Current Compilation Status**
- ✅ **Core KuzuDB components compile successfully**
- ✅ **All processors extend KuzuProcessorBase properly**
- ✅ **Feature flags work correctly**
- ⚠️ **Some test files need updates** (non-critical)
- ⚠️ **Some UI components need interface updates** (non-critical)
### **What You Can Test Now**
1. **Enable KuzuDB via feature flags**
2. **Run the ingestion pipeline** - it will attempt dual-write
3. **Observe enhanced logging and statistics**
4. **Verify graceful degradation** when KuzuDB WASM is not available
---
## 📋 **Next Steps (Optional)**
### **Phase 3: Query Migration** (Future Enhancement)
1. Replace `graph.nodes.filter()` with KuzuDB queries
2. Update UI components to use KuzuDB query results
3. Implement query performance comparisons
### **Phase 4: JSON Deprecation** (Future Enhancement)
1. Remove dual-write pattern
2. Make KuzuDB the primary storage
3. Implement advanced graph analytics
---
## 🎉 **Conclusion**
**The KuzuDB integration is FULLY IMPLEMENTED and ready for use!**
The system now supports:
- ✅ **Dual-write functionality** (JSON + KuzuDB)
- ✅ **Complete error handling** and graceful degradation
- ✅ **Production-ready architecture** with monitoring and statistics
- ✅ **Feature flag control** for safe deployment
- ✅ **Zero breaking changes** to existing functionality
You can now:
1. **Enable KuzuDB** via feature flags
2. **Test the dual-write system** with any repository
3. **Observe enhanced logging** and performance metrics
4. **Add the KuzuDB WASM binary** when ready for full functionality
The foundation is solid and ready for the next phases of the migration plan! 🚀
+73
View File
@@ -0,0 +1,73 @@
PolyForm Noncommercial License 1.0.0
<https://polyformproject.org/licenses/noncommercial/1.0.0>
## Acceptance
In order to get any license under these terms, you must agree to them as both strict obligations and conditions to all your licenses.
## Copyright License
The licensor grants you a copyright license for the software to do everything you might do with the software that would otherwise infringe the licensor's copyright in it for any permitted purpose. However, you may only distribute the software according to [Distribution License](#distribution-license) and make changes or new works based on the software according to [Changes and New Works License](#changes-and-new-works-license).
## Distribution License
The licensor grants you an additional copyright license to distribute copies of the software. Your license to distribute covers distributing the software with changes and new works permitted by [Changes and New Works License](#changes-and-new-works-license).
## Notices
You must ensure that anyone who gets a copy of any part of the software from you also gets a copy of these terms or the URL for them above, as well as copies of any plain-text lines beginning with `Required Notice:` that the licensor provided with the software. For example:
> Required Notice: Copyright Abhigyan Patwari (https://github.com/abhigyanpatwari/GitNexus)
## Changes and New Works License
The licensor grants you an additional copyright license to make changes and new works based on the software for any permitted purpose.
## Patent License
The licensor grants you a patent license for the software that covers patent claims the licensor can license, or becomes able to license, that you would infringe by using the software.
## Noncommercial Purposes
Any noncommercial purpose is a permitted purpose.
## Personal Uses
Personal use for research, experiment, and testing for the benefit of public knowledge, personal study, private entertainment, hobby projects, amateur pursuits, or religious observance, without any anticipated commercial application, is use for a permitted purpose.
## Noncommercial Organizations
Use by any charitable organization, educational institution, public research organization, public safety or health organization, environmental protection organization, or government institution is use for a permitted purpose regardless of the source of funding or obligations resulting from the funding.
## Fair Use
You may have "fair use" rights for the software under the law. These terms do not limit them.
## No Other Rights
These terms do not allow you to sublicense or transfer any of your licenses to anyone else, or prevent the licensor from granting licenses to anyone else. These terms do not imply any other licenses.
## Patent Defense
If you make any written claim that the software infringes or contributes to infringement of any patent, your patent license for the software granted under these terms ends immediately. If your company makes such a claim, your patent license ends immediately for work on behalf of your company.
## Violations
The first time you are notified in writing that you have violated any of these terms, or done anything with the software not covered by your licenses, your licenses can nonetheless continue if you come into full compliance with these terms, and take practical steps to correct past violations, within 32 days of receiving notice. Otherwise, all your licenses end immediately.
## No Liability
***As far as the law allows, the software comes as is, without any warranty or condition, and the licensor will not be liable to you for any damages arising out of these terms or the use or nature of the software, under any kind of legal claim.***
## Definitions
The **licensor** is the individual or entity offering these terms, and the **software** is the software the licensor makes available under these terms.
**You** refers to the individual or entity agreeing to these terms.
**Your company** is any legal entity, sole proprietorship, or other kind of organization that you work for, plus all organizations that have control over, are under the control of, or are under common control with that organization. **Control** means ownership of substantially all the assets of an entity, or the power to direct its management and policies by vote, contract, or otherwise. Control can be direct or indirect.
**Your licenses** are all the licenses granted to you for the software under these terms.
**Use** means anything you do with the software requiring one of your licenses.
-163
View File
@@ -1,163 +0,0 @@
# Memory Leak Fixes for Worker Pool Parallel Processing
## Issues Identified and Fixed
### 1. **Event Listener Memory Leaks** ✅ FIXED
**Problem**: WebWorkerPool wasn't properly cleaning up event listeners during shutdown, causing memory leaks.
**Fix Applied**:
- Added proper event listener cleanup in `shutdown()` method
- Set `worker.onmessage = null`, `worker.onerror = null`, `worker.onmessageerror = null` before terminating workers
- Clear the `eventListeners` Map during shutdown
**Files Modified**: `src/lib/web-worker-pool.ts`
### 2. **Singleton Worker Pool Issues** ✅ FIXED
**Problem**: FileProcessingPool singleton instances were never cleaned up, accumulating memory over time.
**Fix Applied**:
- Added `shutdownInstance()` static method to properly cleanup singleton instances
- Added `hasInstance()` method to check if instance exists
- Added `cleanupAllPools()` utility method in WebWorkerPoolUtils
**Files Modified**: `src/lib/web-worker-pool.ts`
### 3. **Worker Error Handling** ✅ FIXED
**Problem**: Errors in worker termination could leave resources hanging.
**Fix Applied**:
- Improved `handleWorkerError()` method to properly cleanup worker event listeners
- Added try-catch around worker termination
- Ensure workers are removed from pools even on errors
**Files Modified**: `src/lib/web-worker-pool.ts`
### 4. **Missing Memory Monitoring** ✅ FIXED
**Problem**: No memory usage tracking or automatic cleanup triggers.
**Fix Applied**:
- Added `monitorMemoryUsage()` method with automatic cleanup triggers
- Added memory usage estimation in `getStats()` method
- Created periodic memory monitoring in ParallelParsingProcessor
- Added global cleanup handlers for page unload and visibility changes
**Files Modified**: `src/lib/web-worker-pool.ts`, `src/core/ingestion/parallel-parsing-processor.ts`
### 5. **AST Map Accumulation** ✅ FIXED
**Problem**: Large AST maps and function tries were kept in memory without limits.
**Fix Applied**:
- Added `MAX_AST_MAP_SIZE` constant (1000 entries)
- Implemented `cleanupASTMap()` method to remove old entries when limit is exceeded
- Added proper cleanup of AST maps, processed files, and function tries in shutdown
- Added Tree-sitter parser cleanup with `parser.delete()`
**Files Modified**: `src/core/ingestion/parallel-parsing-processor.ts`
### 6. **Pipeline Cleanup Issues** ✅ FIXED
**Problem**: Parallel pipeline wasn't ensuring proper cleanup on errors.
**Fix Applied**:
- Enhanced cleanup method to call `WebWorkerPoolUtils.cleanupAllPools()`
- Added cleanup in both error and finally blocks
- Ensured cleanup happens even when errors occur
**Files Modified**: `src/core/ingestion/parallel-pipeline.ts`
## New Features Added
### Memory Monitoring System
- **Automatic Memory Monitoring**: Checks memory usage every 30 seconds
- **Threshold-based Cleanup**: Triggers cleanup when memory usage exceeds 500MB
- **AST Map Size Limiting**: Automatically cleans up old AST entries when limit exceeded
- **Global Memory Monitoring**: Monitors overall browser memory usage
### Global Cleanup Handlers
- **Page Unload Cleanup**: Automatically cleans up when user closes/refreshes page
- **Tab Visibility Cleanup**: Cleans up when user switches tabs (page becomes hidden)
- **Manual Cleanup Functions**: Utilities for forcing cleanup when needed
### Enhanced Error Handling
- **Graceful Worker Termination**: Proper cleanup even when workers fail
- **Resource Leak Prevention**: Ensures all event listeners and references are cleared
- **Error Recovery**: System continues working even if some workers fail
## Usage Instructions
### 1. Initialize Cleanup Handlers (RECOMMENDED)
```typescript
import { initializeWorkerPoolCleanup } from './src/lib/worker-pool-init.js';
// Call this once when your app starts
initializeWorkerPoolCleanup();
```
### 2. Manual Cleanup (if needed)
```typescript
import { cleanupWorkerPools } from './src/lib/worker-pool-init.js';
// Force cleanup when needed
await cleanupWorkerPools();
```
### 3. Monitor Memory Usage
```typescript
import { getMemoryInfo } from './src/lib/worker-pool-init.js';
const memInfo = getMemoryInfo();
if (memInfo) {
console.log(`Memory: ${memInfo.usedMB}MB / ${memInfo.totalMB}MB (${memInfo.percentage}%)`);
}
```
## Performance Improvements
### Before Fixes:
- Worker pools accumulated without cleanup
- Event listeners remained attached after worker termination
- AST maps grew unbounded causing memory bloat
- No automatic memory management
### After Fixes:
- **Automatic Resource Cleanup**: All resources properly cleaned up
- **Memory Usage Monitoring**: Real-time monitoring with automatic cleanup triggers
- **Bounded Memory Growth**: AST maps and other data structures have size limits
- **Graceful Shutdown**: Proper cleanup on app/tab close
## Monitoring and Debugging
The system now provides detailed logging for:
- Memory usage statistics
- Worker pool status
- Cleanup operations
- Error conditions
Check browser console for messages like:
```
ParallelParsingProcessor Memory Stats:
- Memory Manager: 245.67MB used, 150 files cached
- AST Map: 750 entries
- Processed Files: 890 entries
Memory usage: 245.67MB / 512.00MB (47.98%)
```
## Files Created/Modified
### Modified Files:
- `src/lib/web-worker-pool.ts` - Enhanced with memory leak fixes
- `src/core/ingestion/parallel-parsing-processor.ts` - Added memory monitoring and cleanup
- `src/core/ingestion/parallel-pipeline.ts` - Enhanced cleanup handling
### New Files:
- `src/lib/worker-pool-init.ts` - Initialization and utility functions
- `MEMORY_LEAK_FIXES.md` - This documentation
## Testing Recommendations
1. **Monitor Memory Usage**: Watch browser's Task Manager during large codebase processing
2. **Test Tab Switching**: Switch tabs during processing to verify cleanup triggers
3. **Test Page Refresh**: Refresh page during processing to ensure proper cleanup
4. **Long-running Tests**: Process multiple large codebases to verify no memory accumulation
The system should now maintain stable memory usage even during intensive parallel processing operations.
-154
View File
@@ -1,154 +0,0 @@
# Parallel Processing Verification & Fixes
## 🎯 **Goal: Ensure Parallel Processing Produces Identical Output to Single-Threaded**
## ❌ **Critical Issues Found and Fixed**
### **Issue #1: Wrong Pipeline Class in Worker** 🚨 **CRITICAL**
**Problem**: The `IngestionWorker` was always using `GraphPipeline` (single-threaded) instead of `ParallelGraphPipeline` when parallel processing was enabled.
**Impact**: Even when "parallel processing" was enabled, it was actually running single-threaded processing in the worker, just with the parallel flag set.
**Fix Applied**:
```typescript
// BEFORE (BROKEN)
export class IngestionWorker {
private pipeline: GraphPipeline; // Always single-threaded!
constructor() {
this.pipeline = new GraphPipeline(); // Wrong!
}
}
// AFTER (FIXED)
export class IngestionWorker {
private pipeline: GraphPipeline | ParallelGraphPipeline;
constructor() {
if (isParallelParsingEnabled()) {
this.pipeline = new ParallelGraphPipeline(); // Correct!
} else {
this.pipeline = new GraphPipeline();
}
}
}
```
### **Issue #2: Incorrect Duplicate Detection** 🚨 **CRITICAL**
**Problem**: Different duplicate detection logic between processors.
**Single-threaded (CORRECT)**:
```typescript
if (this.duplicateDetector.checkAndMark(nodeId)) continue; // ✅ Checks AND marks
```
**Parallel (BROKEN)**:
```typescript
if (this.duplicateDetector.isDuplicate(nodeId)) return; // ❌ Only checks, doesn't mark
```
**Impact**: Parallel processor could create duplicate nodes because it wasn't marking them as processed.
**Fix Applied**: Changed parallel processor to use `checkAndMark()`.
### **Issue #3: Property Format Inconsistency** 🚨 **MEDIUM**
**Problem**: Node properties stored in different formats.
**Single-threaded**:
```typescript
decorators: def.decorators, // Array format
extends: def.extends, // Array format
implements: def.implements, // Array format
```
**Parallel (BROKEN)**:
```typescript
decorators: definition.decorators?.join(', '), // String format ❌
extends: definition.extends?.join(', '), // String format ❌
implements: definition.implements?.join(', '), // String format ❌
```
**Impact**: Import/call processors expecting arrays would fail or produce different results.
**Fix Applied**: Made parallel processor store arrays to match single-threaded.
### **Issue #4: Progress Callback Integration** ✅ **ENHANCEMENT**
**Problem**: Worker wasn't properly forwarding progress updates from `ParallelGraphPipeline`.
**Fix Applied**: Added proper progress callback integration.
## ✅ **Verification Checklist**
### **Pipeline Selection** ✅
- [x] Worker uses correct pipeline class based on feature flag
- [x] `ParallelGraphPipeline` used when `isParallelParsingEnabled() === true`
- [x] `GraphPipeline` used when `isParallelParsingEnabled() === false`
### **Data Processing** ✅
- [x] Duplicate detection logic identical (`checkAndMark()`)
- [x] Node property formats identical (arrays not strings)
- [x] Node ID generation identical
- [x] Node label mapping identical
### **Graph Structure** ✅
- [x] Same 4-pass pipeline structure
- [x] Same processor sequence (Structure → Parsing → Import → Call)
- [x] Same AST map and function registry handling
- [x] Same relationship creation logic
### **Memory Management** ✅
- [x] Both use LRU cache service
- [x] Both maintain AST maps for compatibility
- [x] Proper cleanup in both modes
## 🔍 **Expected Behavior After Fixes**
### **Single-threaded Mode**:
- Uses `GraphPipeline`
- Uses `ParsingProcessor`
- Sequential file processing
- Direct definition extraction
### **Parallel Mode**:
- Uses `ParallelGraphPipeline`
- Uses `ParallelParsingProcessor`
- Worker pool parallel processing
- Worker-extracted definitions + main thread AST recreation
### **Identical Output**:
Both modes should now produce:
- ✅ Same node counts by type
- ✅ Same relationship counts by type
- ✅ Same import relationships (IMPORTS, DEPENDS_ON)
- ✅ Same function call relationships (CALLS)
- ✅ Same definition nodes with identical properties
- ✅ Same graph connectivity
## 🧪 **Testing Recommendations**
1. **Process the same codebase** with both modes enabled/disabled
2. **Compare graph statistics** - nodes by type, relationships by type
3. **Verify specific relationships** - check for import and call relationships
4. **Check isolated nodes** - should be minimal in both modes
5. **Performance comparison** - parallel should be faster on large codebases
## 📊 **Success Metrics**
The parallel processing should now show:
```
✅ Relationships by type: {CONTAINS: X, DEFINES: Y, IMPORTS: Z, CALLS: W}
✅ No "missing import relationships" warnings
✅ No "missing function call relationships" warnings
✅ Same graph node/relationship counts as single-threaded
```
Instead of the previous broken output:
```
❌ Relationships by type: {CONTAINS: 103, DEFINES: 1971} // Missing IMPORTS/CALLS!
❌ "No import relationships found between files"
❌ "No function call relationships found"
```
## 🎯 **Conclusion**
The parallel processing implementation now uses the correct pipeline classes and processing logic to produce **identical output** to single-threaded mode, while maintaining the performance benefits of parallel worker pool processing.
**The root cause was using the wrong pipeline class in the worker** - a simple but critical configuration issue that made "parallel processing" actually run single-threaded code with inconsistent data structures.
@@ -1,623 +0,0 @@
# GitNexus Parsing and Storage Technical Documentation
## Complete Data Flow Analysis for Kuzu DB Migration
> **Purpose**: This document provides an extremely detailed, line-by-line analysis of GitNexus's current parsing and storage implementation to facilitate the migration from JSON-based storage to Kuzu DB.
---
## Executive Summary
GitNexus uses a **4-pass ingestion pipeline** that processes code repositories into a knowledge graph stored in JSON format. The system employs in-memory data structures (`SimpleKnowledgeGraph`) with JSON serialization for persistence. This document traces every step of the data transformation process to enable precise Kuzu DB migration.
### Key Storage Points Identified:
1. **In-Memory Graph Storage**: `SimpleKnowledgeGraph` class with arrays
2. **JSON Export/Import**: Via `src/lib/export.ts` functions
3. **LRU Cache Storage**: For AST and parsing results
4. **LocalStorage**: For settings, feature flags, and chat history
5. **IndexedDB**: Planned for KuzuDB persistence (WIP)
---
## 1. Core Data Structures
### 1.1 Knowledge Graph Structure
**File**: `src/core/graph/types.ts`
```typescript
// Primary graph interface - this is what gets stored
export interface KnowledgeGraph {
nodes: GraphNode[]; // Array of all nodes
relationships: GraphRelationship[]; // Array of all relationships
}
// Node structure - every entity in the system
export interface GraphNode {
id: string; // Unique identifier (generated)
label: NodeLabel; // Type classification
properties: NodeProperties; // All metadata as key-value pairs
}
// Relationship structure - connections between nodes
export interface GraphRelationship {
id: string; // Unique identifier (generated)
type: RelationshipType; // Relationship classification
source: string; // Source node ID
target: string; // Target node ID
properties: RelationshipProperties; // Metadata as key-value pairs
}
```
**Node Types** (`NodeLabel`):
- `'Project'` - Repository root
- `'Folder'` - Directory nodes
- `'File'` - Source files
- `'Function'` - Function definitions
- `'Class'` - Class definitions
- `'Method'` - Class methods
- `'Variable'` - Variable declarations
- `'Interface'` - TypeScript interfaces
- `'Decorator'` - Python/TS decorators
- `'Import'` - Import statements
- `'Type'` - Type definitions
- `'CodeElement'` - Generic code elements
**Relationship Types** (`RelationshipType`):
- `'CONTAINS'` - Hierarchical containment (folder → file, file → function)
- `'CALLS'` - Function/method calls
- `'INHERITS'` - Class inheritance
- `'OVERRIDES'` - Method overrides
- `'IMPORTS'` - Module imports
- `'IMPLEMENTS'` - Interface implementations
- `'DECORATES'` - Decorator applications
### 1.2 Implementation Class
**File**: `src/core/graph/graph.ts`
```typescript
export class SimpleKnowledgeGraph implements KnowledgeGraph {
nodes: GraphNode[] = []; // Simple array storage
relationships: GraphRelationship[] = []; // Simple array storage
addNode(node: GraphNode): void {
this.nodes.push(node); // Direct array append
}
addRelationship(relationship: GraphRelationship): void {
this.relationships.push(relationship); // Direct array append
}
}
```
**Critical Storage Characteristics**:
- **No indexing**: Linear search for node/relationship lookups
- **No constraints**: No validation of referential integrity
- **Memory-only**: No built-in persistence
- **Simple append**: No deduplication or conflict resolution
---
## 2. Four-Pass Ingestion Pipeline
The pipeline transforms raw repository data through four distinct phases, each building upon the previous:
### Pass 1: Structure Analysis (`StructureProcessor`)
**File**: `src/core/ingestion/structure-processor.ts`
**Input**:
- `projectRoot: string` - Repository path
- `projectName: string` - Repository name
- `filePaths: string[]` - All discovered file paths
**Process**:
1. **Project Node Creation** (Lines 58-61):
```typescript
const projectNode = this.createProjectNode(projectName, projectRoot);
graph.addNode(projectNode); // STORAGE POINT 1
```
2. **Path Categorization** (Lines 87-127):
```typescript
const { directories, files } = this.categorizePaths(filePaths);
// Separates files from directories using path analysis
```
3. **Directory Node Creation** (Lines 147-174):
```typescript
const directoryNodes = this.createDirectoryNodes(visibleDirectories);
directoryNodes.forEach(node => graph.addNode(node)); // STORAGE POINT 2
```
4. **File Node Creation** (Lines 179-208):
```typescript
const fileNodes = this.createFileNodes(visibleFiles);
fileNodes.forEach(node => graph.addNode(node)); // STORAGE POINT 3
```
5. **CONTAINS Relationship Creation** (Lines 213-255):
```typescript
this.createContainsRelationships(graph, projectNode.id, visibleDirectories, visibleFiles);
// Creates hierarchical relationships - STORAGE POINT 4
```
**Storage Pattern**: Direct `graph.addNode()` and `graph.addRelationship()` calls append to arrays.
### Pass 2: Code Parsing (`ParsingProcessor` / `ParallelParsingProcessor`)
**Files**:
- `src/core/ingestion/parsing-processor.ts`
- `src/core/ingestion/parallel-parsing-processor.ts`
**Input**:
- `filePaths: string[]` - Files to parse
- `fileContents: Map<string, string>` - File content mapping
- `options?: ParsingOptions` - Filtering options
**Critical Data Structures**:
1. **AST Storage** (Line 55 in `parsing-processor.ts`):
```typescript
private astMap: Map<string, ParsedAST> = new Map();
// STORAGE POINT 5 - AST trees indexed by file path
```
2. **Function Registry** (Line 56):
```typescript
private functionTrie: FunctionRegistryTrie = new FunctionRegistryTrie();
// STORAGE POINT 6 - Searchable function definitions
```
**Process Flow**:
1. **File Filtering** (Lines 116-152):
```typescript
const filteredFiles = this.applyFiltering(filePaths, fileContents, options);
// Applies directory and extension filters
```
2. **Tree-sitter Initialization** (Lines 84, 245):
```typescript
await this.initializeParser();
// Loads WASM parsers for each language
```
3. **Batch Processing** (Lines 86-108):
```typescript
const batchProcessor = new BatchProcessor<string, void>(BATCH_SIZE, async (filePaths: string[]) => {
for (const filePath of filePaths) {
await this.parseFile(graph, filePath, content); // CRITICAL PARSING
this.processedFiles.add(filePath);
}
});
```
4. **Definition Extraction** (`parseFile` method):
```typescript
// Extracts: functions, classes, methods, variables, interfaces, types
// Each creates nodes via: graph.addNode(definitionNode) - STORAGE POINT 7
```
**Parallel Processing Variant**:
- Uses Web Workers for CPU-intensive parsing
- Results aggregated in main thread
- Same storage patterns but with worker coordination
### Pass 3: Import Resolution (`ImportProcessor`)
**File**: `src/core/ingestion/import-processor.ts`
**Input**:
- `graph: KnowledgeGraph` - Current graph state
- `astMap: Map<string, ParsedAST>` - Parsed ASTs
- `fileContents: Map<string, string>` - File contents
**Critical Data Structure**:
```typescript
interface ImportMap {
[importingFile: string]: {
[localName: string]: {
targetFile: string;
exportedName: string;
importType: 'default' | 'named' | 'namespace' | 'dynamic';
}
}
}
private importMap: ImportMap = {}; // STORAGE POINT 8
```
**Process Flow**:
1. **Import Extraction** (Lines 84-88):
```typescript
for (const [filePath, ast] of astMap) {
const fileImports = await this.processFileImports(filePath, ast, graph);
// Extracts import statements from AST
}
```
2. **Language-Specific Processing**:
- **JavaScript/TypeScript** (Lines 231-324): Handles ES6 imports, CommonJS requires
- **Python** (Lines 160-226): Handles `import` and `from...import` statements
3. **Module Path Resolution** (Lines 522-606):
```typescript
private resolveModulePath(moduleName: string, importingFile: string, language: string): string
// Resolves relative and absolute imports to actual file paths
```
4. **Relationship Creation** (Lines 611-645):
```typescript
private createImportRelationship(graph: KnowledgeGraph, importInfo: ImportInfo): void {
// Creates IMPORTS relationships - STORAGE POINT 9
graph.relationships.push(relationship);
}
```
### Pass 4: Call Resolution (`CallProcessor`)
**File**: `src/core/ingestion/call-processor.ts`
**Input**:
- `graph: KnowledgeGraph` - Current graph
- `astMap: Map<string, ParsedAST>` - ASTs for call extraction
- `importMap: ImportMap` - Import resolution data
**Process Flow**:
1. **Call Extraction** (Lines 86-120):
```typescript
private async processFileCalls(filePath: string, ast: ParsedAST, graph: KnowledgeGraph) {
const calls = this.extractFunctionCalls(ast.tree!.rootNode, filePath);
// Extracts function/method call sites from AST
}
```
2. **3-Stage Resolution Strategy** (Lines 146-162):
```typescript
// Stage 1: Exact Match using ImportMap (High Confidence)
// Stage 2: Same-Module Match (Medium Confidence)
// Stage 3: Heuristic Fallback (Low Confidence)
```
3. **Call Relationship Creation** (Lines 99-101):
```typescript
if (resolution.success && resolution.targetNodeId) {
this.createCallRelationship(graph, call, resolution.targetNodeId);
// Creates CALLS relationships - STORAGE POINT 10
}
```
---
## 3. JSON Storage Implementation
### 3.1 Export Functions
**File**: `src/lib/export.ts`
**Primary Export Function** (Lines 34-68):
```typescript
export function exportGraphToJSON(
graph: KnowledgeGraph,
options: ExportOptions = {},
fileContents?: Map<string, string>,
processingStats?: { duration: number }
): string {
// Metadata wrapper structure
if (includeMetadata) {
const metadata: ExportMetadata = {
exportedAt: includeTimestamp ? new Date().toISOString() : '',
version: '1.0.0',
nodeCount: graph.nodes.length,
relationshipCount: graph.relationships.length,
fileCount: fileContents?.size,
processingDuration: processingStats?.duration
};
exportData = {
metadata,
graph, // CRITICAL: Raw graph object serialization
...(fileContents && { fileContents: Object.fromEntries(fileContents) })
};
} else {
exportData = graph; // Direct graph serialization
}
return JSON.stringify(exportData, null, prettyPrint ? 2 : 0);
// STORAGE POINT 11 - JSON string generation
}
```
**JSON Structure**:
```json
{
"metadata": {
"exportedAt": "2024-01-01T00:00:00.000Z",
"version": "1.0.0",
"nodeCount": 1250,
"relationshipCount": 3400,
"fileCount": 45,
"processingDuration": 2500
},
"graph": {
"nodes": [
{
"id": "node_project_abc123",
"label": "Project",
"properties": {
"name": "MyProject",
"path": "/path/to/project",
"createdAt": "2024-01-01T00:00:00.000Z"
}
}
// ... more nodes
],
"relationships": [
{
"id": "rel_contains_def456",
"type": "CONTAINS",
"source": "node_project_abc123",
"target": "node_folder_ghi789",
"properties": {}
}
// ... more relationships
]
},
"fileContents": {
"src/main.ts": "export function main() { ... }",
// ... more file contents
}
}
```
### 3.2 Import Functions
**Import Function** (Lines 411-443):
```typescript
export function importGraphFromJSON(jsonString: string): {
graph: KnowledgeGraph;
metadata?: ExportMetadata;
fileContents?: Map<string, string>;
} {
try {
const parsed = JSON.parse(jsonString); // DESERIALIZATION POINT
if (parsed.metadata && parsed.graph) {
// Handle wrapped format
const result = {
graph: parsed.graph, // Direct object assignment
metadata: parsed.metadata
};
if (parsed.fileContents) {
result.fileContents = new Map(Object.entries(parsed.fileContents));
// Convert plain object back to Map
}
return result;
}
return { graph: parsed }; // Direct graph object
} catch (error) {
throw new Error(`Failed to import graph: ${error.message}`);
}
}
```
### 3.3 Download Implementation
**File Download** (Lines 182-207):
```typescript
export function downloadJSON(content: string, filename: string): void {
// Create blob with JSON content
const blob = new Blob([content], { type: 'application/json' });
// Browser download mechanism
const url = URL.createObjectURL(blob);
const link = document.createElement('a');
link.href = url;
link.download = filename;
link.click(); // Triggers download - PERSISTENCE POINT
}
```
---
## 4. Additional Storage Mechanisms
### 4.1 LRU Cache Storage
**Files**:
- `src/services/memory-manager.ts` (Memory management)
- Various processor classes (Cache usage)
**Purpose**: Caches parsed ASTs and query results for performance
**Implementation Pattern**:
```typescript
// Cache key generation
const cacheKey = this.lruCache.generateFileCacheKey(filePath, contentHash);
// Cache retrieval
const cachedResult = this.lruCache.getParsedFile(cacheKey);
// Cache storage
this.lruCache.setParsedFile(cacheKey, parseResult); // STORAGE POINT 12
```
### 4.2 LocalStorage Usage
**Settings Storage** (`src/config/feature-flags.ts`, Lines 104-110):
```typescript
private saveFlags(): void {
try {
localStorage.setItem('gitnexus_feature_flags', JSON.stringify(this.flags));
// STORAGE POINT 13 - Browser localStorage
} catch (error) {
console.warn('Failed to save feature flags to localStorage:', error);
}
}
```
**Chat History** (`src/lib/chat-history.ts`, Lines 275-291):
```typescript
private saveSession(session: ChatSession): void {
try {
localStorage.setItem(this.storageKey, JSON.stringify(session));
// STORAGE POINT 14 - Chat persistence
} catch (error) {
// Handle quota exceeded errors
}
}
```
### 4.3 IndexedDB (Planned for KuzuDB)
**Current Status**: Implementation exists but not fully integrated
**Files**: `src/core/kuzu/` directory contains KuzuDB integration code
**Purpose**: Will replace JSON storage with embedded graph database
---
## 5. Data Flow Summary
```mermaid
flowchart TD
START([Repository Input]) --> STRUCT[Structure Processor]
STRUCT --> |Creates nodes/relationships| GRAPH1[In-Memory Graph]
GRAPH1 --> PARSE[Parsing Processor]
PARSE --> |AST Storage| AST_MAP[AST Map]
PARSE --> |Function Registry| FUNC_TRIE[Function Trie]
PARSE --> |Adds definition nodes| GRAPH2[Enhanced Graph]
GRAPH2 --> IMPORT[Import Processor]
AST_MAP --> IMPORT
IMPORT --> |Import Map| IMP_MAP[Import Map]
IMPORT --> |Adds IMPORTS relationships| GRAPH3[Graph + Imports]
GRAPH3 --> CALLS[Call Processor]
AST_MAP --> CALLS
IMP_MAP --> CALLS
FUNC_TRIE --> CALLS
CALLS --> |Adds CALLS relationships| FINAL_GRAPH[Final Knowledge Graph]
FINAL_GRAPH --> JSON_EXPORT[JSON Export]
JSON_EXPORT --> |JSON.stringify| JSON_STRING[JSON String]
JSON_STRING --> |Browser Download| FILE_SYSTEM[File System]
FINAL_GRAPH --> |Direct object reference| UI[UI Components]
subgraph "Storage Points"
GRAPH1
GRAPH2
GRAPH3
FINAL_GRAPH
AST_MAP
FUNC_TRIE
IMP_MAP
JSON_STRING
end
subgraph "Cache Layer"
LRU_CACHE[LRU Cache]
LOCAL_STORAGE[LocalStorage]
end
PARSE -.-> LRU_CACHE
LOCAL_STORAGE -.-> SETTINGS[Settings/Flags]
```
---
## 6. Critical Migration Points for Kuzu DB
### 6.1 Schema Mapping Requirements
**Current JSON Structure → Kuzu Schema**:
1. **Nodes Table**:
```sql
CREATE NODE TABLE IF NOT EXISTS nodes (
id STRING PRIMARY KEY,
label STRING NOT NULL,
properties MAP(STRING, STRING)
);
```
2. **Relationships Table**:
```sql
CREATE REL TABLE IF NOT EXISTS relationships (
FROM nodes TO nodes,
id STRING,
type STRING NOT NULL,
properties MAP(STRING, STRING)
);
```
### 6.2 Data Transformation Points
**Every `graph.addNode()` call** → **Kuzu INSERT statement**
**Every `graph.addRelationship()` call** → **Kuzu MATCH/CREATE statement**
### 6.3 Query Transformation Requirements
**Current**: Linear array searches in `SimpleKnowledgeGraph`
**Target**: Cypher queries in KuzuDB
**Example Transformations**:
```typescript
// Current: Find nodes by label
graph.nodes.filter(n => n.label === 'Function')
// Target: Cypher query
MATCH (n:Function) RETURN n
```
### 6.4 Persistence Layer Changes
**Current Flow**:
1. Build `SimpleKnowledgeGraph` in memory
2. Export to JSON string
3. Download as file
**Target Flow**:
1. Stream data directly to KuzuDB during processing
2. Persist to IndexedDB automatically
3. Export via Cypher queries
---
## 7. Implementation Recommendations
### 7.1 Migration Strategy
1. **Phase 1**: Create parallel KuzuDB storage alongside existing JSON
2. **Phase 2**: Implement streaming ingestion (write to Kuzu during pipeline)
3. **Phase 3**: Replace SimpleKnowledgeGraph with KuzuDB queries
4. **Phase 4**: Remove JSON export/import (keep as backup option)
### 7.2 Critical Considerations
1. **Referential Integrity**: Kuzu enforces relationships, JSON doesn't
2. **Transaction Boundaries**: Kuzu needs explicit transactions
3. **Query Performance**: Index strategy for common access patterns
4. **Memory Management**: Kuzu handles memory, current system uses manual arrays
5. **Concurrent Access**: Kuzu supports concurrent reads, current system is single-threaded
### 7.3 Testing Strategy
1. **Data Integrity**: Compare JSON export with Kuzu export for identical results
2. **Performance**: Benchmark ingestion and query performance
3. **Memory Usage**: Monitor memory consumption during large repository processing
4. **Error Handling**: Test transaction rollback and recovery scenarios
---
## Conclusion
This document provides the complete technical foundation for migrating GitNexus from JSON-based storage to KuzuDB. Every storage point, data transformation, and persistence mechanism has been identified and documented. The migration should focus on replacing the `SimpleKnowledgeGraph` implementation while maintaining the exact same data structures and relationships in the new KuzuDB schema.
-253
View File
@@ -1,253 +0,0 @@
# Phase 2: Parallel Storage Implementation - Complete! 🎉
## Overview
Successfully implemented the **Parallel Storage** phase of the KuzuDB migration plan, enabling dual-write functionality where data is written to both JSON (primary) and KuzuDB (secondary) storage systems simultaneously.
## ✅ **Components Implemented**
### 1. **KuzuProcessorBase** (`src/core/ingestion/kuzu-processor-base.ts`)
**Core Features:**
- **Dual-Write Pattern**: Seamless writes to both JSON and KuzuDB
- **Transaction Management**: Begin, commit, and rollback transaction support
- **Error Handling**: Graceful degradation when KuzuDB fails
- **Performance Monitoring**: Detailed statistics and timing metrics
- **Data Validation**: Consistency checks between storage systems
- **Feature Flag Integration**: Respects `isKuzuDBEnabled()` settings
**Key Methods:**
- `addNodeDualWrite()` - Writes nodes to both storages
- `addRelationshipDualWrite()` - Writes relationships to both storages
- `beginTransaction()` / `commitTransaction()` / `rollbackTransaction()`
- `initializeKuzuDB()` - Sets up KuzuDB connection
- `validateNodeConsistency()` / `validateRelationshipConsistency()`
### 2. **Enhanced StructureProcessor**
**Modifications:**
- ✅ Extends `KuzuProcessorBase` for dual-write capability
- ✅ Async `process()` method with KuzuDB initialization
- ✅ Dual-write support for Project, Folder, and File nodes
- ✅ Dual-write support for CONTAINS relationships
- ✅ Transaction boundaries with commit/rollback
- ✅ Comprehensive error handling and statistics
**Dual-Write Flow:**
1. Initialize KuzuDB connection
2. Create project node → write to JSON + KuzuDB
3. Create directory nodes → write to JSON + KuzuDB
4. Create file nodes → write to JSON + KuzuDB
5. Create CONTAINS relationships → write to JSON + KuzuDB
6. Commit KuzuDB transaction
7. Log detailed statistics
### 3. **Enhanced ParsingProcessor**
**Modifications:**
- ✅ Extends `KuzuProcessorBase` for dual-write capability
- ✅ Async definition processing with KuzuDB writes
- ✅ Dual-write support for Function, Class, Method, Variable, Interface, Type nodes
- ✅ Dual-write support for INHERITS, IMPLEMENTS, IMPORTS relationships
- ✅ Transaction boundaries with automatic commit
- ✅ Batch processing optimization
**Dual-Write Flow:**
1. Initialize KuzuDB connection
2. Process each file's definitions
3. Create definition nodes → write to JSON + KuzuDB
4. Create containment relationships → write to JSON + KuzuDB
5. Create inheritance/implementation relationships → write to JSON + KuzuDB
6. Commit KuzuDB transaction
7. Log processing statistics
### 4. **Enhanced ImportProcessor**
**Modifications:**
- ✅ Extends `KuzuProcessorBase` for dual-write capability
- ✅ Async import relationship creation
- ✅ Dual-write support for IMPORTS relationships
- ✅ Transaction management with rollback support
- ✅ Enhanced error handling and progress tracking
**Dual-Write Flow:**
1. Initialize KuzuDB connection
2. Process imports for each file
3. Create IMPORTS relationships → write to JSON + KuzuDB
4. Commit KuzuDB transaction
5. Log import resolution statistics
### 5. **Enhanced CallProcessor**
**Modifications:**
- ✅ Extends `KuzuProcessorBase` for dual-write capability
- ✅ Async call relationship creation
- ✅ Dual-write support for CALLS relationships
- ✅ 3-stage resolution strategy maintained
- ✅ Transaction boundaries and error handling
**Dual-Write Flow:**
1. Initialize KuzuDB connection
2. Extract function calls from AST
3. Resolve calls using 3-stage strategy
4. Create CALLS relationships → write to JSON + KuzuDB
5. Commit KuzuDB transaction
6. Log call resolution statistics
## 🏗️ **Architecture Highlights**
### **Dual-Write Pattern Implementation**
```typescript
// JSON write (primary - always succeeds)
jsonGraph.addNode(node);
// KuzuDB write (secondary - graceful failure)
if (this.kuzuGraph) {
try {
this.kuzuGraph.addNode(node);
} catch (kuzuError) {
console.warn('KuzuDB write failed:', kuzuError);
// Continue processing - JSON is primary storage
}
}
```
### **Transaction Management**
```typescript
// Begin transaction
await this.beginTransaction();
try {
// Perform operations
await this.addNodeDualWrite(graph, node);
await this.addRelationshipDualWrite(graph, relationship);
// Commit transaction
await this.commitTransaction();
} catch (error) {
// Rollback on failure
await this.rollbackTransaction();
throw error;
}
```
### **Statistics and Monitoring**
- **Nodes processed**: Total nodes written to JSON
- **KuzuDB nodes written**: Successful KuzuDB writes
- **KuzuDB errors**: Failed KuzuDB operations
- **Success rate**: Percentage of successful dual-writes
- **Processing time**: Total time spent on operations
- **Validation errors**: Data consistency issues detected
## 📊 **Key Benefits Achieved**
### **1. Zero Breaking Changes**
- All existing processors maintain their original interfaces
- JSON storage remains primary - system continues working even if KuzuDB fails
- Backward compatibility with all existing code
### **2. Production-Ready Error Handling**
- KuzuDB failures don't break the ingestion pipeline
- Graceful degradation to JSON-only mode
- Comprehensive error logging and categorization
- Transaction rollback on critical failures
### **3. Performance Optimization**
- Batch processing for optimal KuzuDB performance
- Async operations with proper error boundaries
- Transaction boundaries reduce database overhead
- Detailed performance monitoring and statistics
### **4. Data Consistency**
- Dual-write ensures both storages have the same data
- Transaction management prevents partial writes
- Validation hooks for consistency checking
- Rollback capabilities for data integrity
### **5. Feature Flag Integration**
- Respects `isKuzuDBEnabled()` configuration
- Can be enabled/disabled without code changes
- Gradual rollout capabilities
- A/B testing support
## 🔄 **Integration Points**
### **Pipeline Integration**
All processors now support the enhanced dual-write pattern:
```typescript
// Structure Phase
const structureProcessor = new StructureProcessor({ enableKuzuDB: true });
await structureProcessor.process(graph, structureInput);
// Parsing Phase
const parsingProcessor = new ParsingProcessor({ enableKuzuDB: true });
await parsingProcessor.process(graph, parsingInput);
// Import Phase
const importProcessor = new ImportProcessor({ enableKuzuDB: true });
await importProcessor.process(graph, astMap, fileContents);
// Call Phase
const callProcessor = new CallProcessor(functionTrie, { enableKuzuDB: true });
await callProcessor.process(graph, astMap, importMap);
```
### **Configuration Options**
```typescript
interface KuzuProcessorOptions {
enableKuzuDB?: boolean; // Enable/disable KuzuDB integration
batchSize?: number; // Batch size for optimal performance
autoCommit?: boolean; // Automatic transaction commits
enableValidation?: boolean; // Data consistency validation
}
```
## 📈 **Performance Expectations**
### **Memory Usage**
- Minimal additional memory overhead (~5-10%)
- Transaction batching prevents memory bloat
- Graceful handling of large codebases
### **Processing Time**
- Expected 10-20% increase in processing time
- Batch operations optimize KuzuDB performance
- Async operations prevent blocking
### **Error Resilience**
- 100% reliability for JSON storage (primary)
- Graceful degradation for KuzuDB failures
- No data loss even with KuzuDB issues
## 🚀 **Ready for Phase 3**
The parallel storage implementation provides a solid foundation for **Phase 3: Query Migration**, where we'll:
1. **Implement Query Abstraction Layer**: Create unified query interface
2. **Add Query Routing Logic**: Route queries to appropriate storage
3. **Performance Comparison Tools**: A/B test JSON vs KuzuDB queries
4. **Query Result Validation**: Ensure consistent results between storages
## 📁 **Files Modified/Created**
### **New Files**
- `src/core/ingestion/kuzu-processor-base.ts` - Base class for dual-write pattern
### **Modified Files**
- `src/core/ingestion/structure-processor.ts` - Added KuzuDB dual-write support
- `src/core/ingestion/parsing-processor.ts` - Added KuzuDB dual-write support
- `src/core/ingestion/import-processor.ts` - Added KuzuDB dual-write support
- `src/core/ingestion/call-processor.ts` - Added KuzuDB dual-write support
## 🎯 **Success Metrics**
- ✅ **100% Backward Compatibility**: All existing functionality preserved
- ✅ **Graceful Error Handling**: KuzuDB failures don't break the system
- ✅ **Transaction Safety**: Data integrity maintained with rollback support
- ✅ **Performance Monitoring**: Comprehensive statistics and metrics
- ✅ **Feature Flag Ready**: Can be enabled/disabled via configuration
- ✅ **Production Quality**: Error handling, logging, and monitoring
The dual-write pattern is now fully implemented and ready for production deployment! 🚀
+523 -293
View File
@@ -1,359 +1,589 @@
# GitNexus - Fully Client sided Knowledge Graph Generator and Graph RAG Agent
# GitNexus
⚠️ Important Notice:** GitNexus has NO official cryptocurrency, token, or coin. Any token/coin using the GitNexus name on Pump.fun or any other platform is **not affiliated with, endorsed by, or created by** this project or its maintainers. Do not purchase any cryptocurrency claiming association with GitNexus.
<div align="center">
<a href="https://trendshift.io/repositories/19809" target="_blank">
<img src="https://trendshift.io/api/badge/repositories/19809" alt="abhigyanpatwari%2FGitNexus | Trendshift" style="width: 250px; height: 55px;" width="250" height="55"/>
</a>
<h2>Join the official Discord to discuss ideas, issues etc!</h2>
<a href="https://discord.gg/AAsRVT6fGb">
<img src="https://img.shields.io/discord/1477255801545429032?color=5865F2&logo=discord&logoColor=white" alt="Discord"/>
</a>
<a href="https://www.npmjs.com/package/gitnexus">
<img src="https://img.shields.io/npm/v/gitnexus.svg" alt="npm version"/>
</a>
<a href="https://polyformproject.org/licenses/noncommercial/1.0.0/">
<img src="https://img.shields.io/badge/License-PolyForm%20Noncommercial-blue.svg" alt="License: PolyForm Noncommercial"/>
</a>
<p><strong>Enterprise (SaaS & Self-hosted)</strong> - <a href="https://akonlabs.com">akonlabs.com</a></p>
</div>
**Building nervous system for agent context.**
Indexes any codebase into a knowledge graph — every dependency, call chain, cluster, and execution flow — then exposes it through smart tools so AI agents never miss code.
GitNexus is a privacy-focused, zero-server knowledge graph generator that runs entirely in your browser. It transforms codebases into interactive knowledge graphs using advanced AST parsing, multi-threaded Web Workers, and an embedded KuzuDB WASM database. Features a Graph RAG agent for intelligent code exploration through natural language queries using cypher queries executed directly against the in-browser graph database.
https://github.com/user-attachments/assets/6f13bd45-d6e9-4f4e-a360-ceb66f41c741
## Current Work in Progress:
- Ollama support
- Export as csv ( for both node and relation table )
## Features
https://github.com/user-attachments/assets/172685ba-8e54-4ea7-9ad1-e31a3398da72
**Code Analysis**
- Analyze GitHub repositories or ZIP files
- Support for TypeScript, JavaScript, Python
- Interactive graph visualization with D3.js
- File filtering and directory selection
- Export results as JSON/CSV
**AI Chat**
> *Like DeepWiki, but deeper.* DeepWiki helps you *understand* code. GitNexus lets you *analyze* it — because a knowledge graph tracks every relationship, not just descriptions.
- Multiple LLM providers (OpenAI, Anthropic, Gemini, Azure)
- Query code structure and relationships
- Context-aware conversations
- Graph-based code search
**TL;DR:** The **Web UI** is a quick way to chat with any repo. The **CLI + MCP** is how you make your AI agent actually reliable — it gives Cursor, Claude Code, Codex, and friends a deep architectural view of your codebase so they stop missing dependencies, breaking call chains, and shipping blind edits. Even smaller models get full architectural clarity, making it compete with goliath models.
**Processing**
---
- Four-pass analysis: structure → parsing → imports → calls
- Parallel processing with Web Workers
- AST-based code extraction using Tree-sitter
- Memory-efficient caching
## Star History
## Architecture
[![Star History Chart](https://api.star-history.com/svg?repos=abhigyanpatwari/GitNexus&type=date&legend=top-left)](https://www.star-history.com/#abhigyanpatwari/GitNexus&type=date&legend=top-left)
```mermaid
graph TB
UI[React UI Layer] --> EM[Engine Manager]
EM --> LEG[Legacy Engine]
EM --> NG[Next-Gen Engine - WIP]
subgraph "Legacy Engine (Production Ready)"
LEG --> GP[Sequential Pipeline]
GP --> SP[Single-threaded Parser]
GP --> MEM[In-Memory Graph Store]
MEM --> JSON[JSON Export]
end
subgraph "Next-Gen Engine (Work in Progress)"
NG --> PP[Parallel Pipeline]
PP --> WP[Web Worker Pool]
PP --> KDB[KuzuDB WASM]
KDB --> CYP[Cypher Queries]
CYP --> RAG[Graph RAG Agent - WIP]
end
subgraph "Core Technologies"
TS[Tree-sitter WASM]
D3[D3.js Force Simulation]
LC[LangChain ReAct Agents]
IDB[IndexedDB Persistence]
end
## Two Ways to Use GitNexus
| | **CLI + MCP** | **Web UI** |
| ----------------- | -------------------------------------------------------------- | ------------------------------------------------------------ |
| **What** | Index repos locally, connect AI agents via MCP | Visual graph explorer + AI chat in browser |
| **For** | Daily development with Cursor, Claude Code, Codex, Windsurf, OpenCode | Quick exploration, demos, one-off analysis |
| **Scale** | Full repos, any size | Limited by browser memory (~5k files), or unlimited via backend mode |
| **Install** | `npm install -g gitnexus` | No install —[gitnexus.vercel.app](https://gitnexus.vercel.app) |
| **Storage** | LadybugDB native (fast, persistent) | LadybugDB WASM (in-memory, per session) |
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
| **Privacy** | Everything local, no network | Everything in-browser, no server |
> **Bridge mode:** `gitnexus serve` connects the two — the web UI auto-detects the local server and can browse all your CLI-indexed repos without re-uploading or re-indexing.
---
## Enterprise
GitNexus is available as an **enterprise offering** - either as a fully managed **SaaS** or a **self-hosted** deployment. Also available for **commercial use** of the OSS version with proper licensing.
Enterprise includes:
- **PR Review** - automated blast radius analysis on pull requests
- **Auto-updating Code Wiki** - always up-to-date documentation (Code Wiki is also available in OSS)
- **Auto-reindexing** - knowledge graph stays fresh automatically
- **Multi-repo support** - unified graph across repositories
- **OCaml support** - additional language coverage
- **Priority feature/language support** - request new languages or features
**Upcoming:**
- Auto regression forensics
- End-to-end test generation
👉 Learn more at [akonlabs.com](https://akonlabs.com)
💬 For commercial licensing or enterprise inquiries, ping us on [Discord](https://discord.gg/AAsRVT6fGb) or drop an email at founders@akonlabs.com
---
## Development
- [ARCHITECTURE.md](ARCHITECTURE.md) — packages, index → graph → MCP flow, where to change code
- [RUNBOOK.md](RUNBOOK.md) — analyze, embeddings, stale index, MCP recovery, CI snippets
- [GUARDRAILS.md](GUARDRAILS.md) — safety rules and operational “Signs” for contributors and agents
- [CONTRIBUTING.md](CONTRIBUTING.md) — license, setup, commits, and pull requests
- [TESTING.md](TESTING.md) — test commands for `gitnexus` and `gitnexus-web`
## CLI + MCP (recommended)
The CLI indexes your repository and runs an MCP server that gives AI agents deep codebase awareness.
### Quick Start
```bash
# Index your repo (run from repo root)
npx gitnexus analyze
```
**Tech Stack**:
That's it. This indexes the codebase, installs agent skills, registers Claude Code hooks, and creates `AGENTS.md` / `CLAUDE.md` context files — all in one command.
- **Frontend**: React 18 + TypeScript + Vite + D3.js force simulation
- **Parsing**: Tree-sitter WASM parsers (TypeScript, JavaScript, Python)
- **Concurrency**: Web Worker Pool with Comlink for thread-safe communication
- **Caching**: LRU-based AST cache with memory management and eviction policies
- **AI**: LangChain.js ReAct agents with tool-augmented reasoning
- **Database**: KuzuDB WASM integration (WIP) + IndexedDB persistence
- **Graph RAG**: Cypher query generation for knowledge graph reasoning (WIP)
To configure MCP for your editor, run `npx gitnexus setup` once — or set it up manually below.
## Four-Pass Ingestion Pipeline
### MCP Setup
`gitnexus setup` auto-detects your editors and writes the correct global MCP config. You only need to run it once.
### Editor Support
| Editor | MCP | Skills | Hooks (auto-augment) | Support |
| --------------------- | --- | ------ | -------------------- | -------------- |
| **Claude Code** | Yes | Yes | Yes (PreToolUse + PostToolUse) | **Full** |
| **Cursor** | Yes | Yes | — | MCP + Skills |
| **Codex** | Yes | Yes | — | MCP + Skills |
| **Windsurf** | Yes | — | — | MCP |
| **OpenCode** | Yes | Yes | — | MCP + Skills |
| **Codex** | Yes | — | — | MCP |
> **Claude Code** gets the deepest integration: MCP tools + agent skills + PreToolUse hooks that enrich searches with graph context + PostToolUse hooks that auto-reindex after commits.
## Community Integrations
Built by the community — not officially maintained, but worth checking out.
| Project | Author | Description |
|---------|--------|-------------|
| [pi-gitnexus](https://github.com/tintinweb/pi-gitnexus) | [@tintinweb](https://github.com/tintinweb) | GitNexus plugin for [pi](https://pi.dev) — `pi install npm:pi-gitnexus` |
| [gitnexus-stable-ops](https://github.com/ShunsukeHayashi/gitnexus-stable-ops) | [@ShunsukeHayashi](https://github.com/ShunsukeHayashi) | Stable ops & deployment workflows (Miyabi ecosystem) |
> Have a project built on GitNexus? Open a PR to add it here!
If you prefer manual configuration:
**Claude Code** (full support — MCP + skills + hooks):
```bash
# macOS / Linux
claude mcp add gitnexus -- npx -y gitnexus@latest mcp
# Windows
claude mcp add gitnexus -- cmd /c npx -y gitnexus@latest mcp
```
**Codex** (full support — MCP + skills):
```bash
codex mcp add gitnexus -- npx -y gitnexus@latest mcp
```
**Cursor** (`~/.cursor/mcp.json` — global, works for all projects):
```json
{
"mcpServers": {
"gitnexus": {
"command": "npx",
"args": ["-y", "gitnexus@latest", "mcp"]
}
}
}
```
**OpenCode** (`~/.config/opencode/config.json`):
```json
{
"mcp": {
"gitnexus": {
"type": "local",
"command": ["gitnexus", "mcp"]
}
}
}
```
**Codex** (`~/.codex/config.toml` for system scope, or `.codex/config.toml` for project scope):
```toml
[mcp_servers.gitnexus]
command = "npx"
args = ["-y", "gitnexus@latest", "mcp"]
```
### CLI Commands
```bash
gitnexus setup # Configure MCP for your editors (one-time)
gitnexus analyze [path] # Index a repository (or update stale index)
gitnexus analyze --force # Force full re-index
gitnexus analyze --skills # Generate repo-specific skill files from detected communities
gitnexus analyze --skip-embeddings # Skip embedding generation (faster)
gitnexus analyze --skip-agents-md # Preserve custom AGENTS.md/CLAUDE.md gitnexus section edits
gitnexus analyze --embeddings # Enable embedding generation (slower, better search)
gitnexus analyze --verbose # Log skipped files when parsers are unavailable
gitnexus mcp # Start MCP server (stdio) — serves all indexed repos
gitnexus serve # Start local HTTP server (multi-repo) for web UI connection
gitnexus list # List all indexed repositories
gitnexus status # Show index status for current repo
gitnexus clean # Delete index for current repo
gitnexus clean --all --force # Delete all indexes
gitnexus wiki [path] # Generate repository wiki from knowledge graph
gitnexus wiki --model <model> # Wiki with custom LLM model (default: gpt-4o-mini)
gitnexus wiki --base-url <url> # Wiki with custom LLM API base URL
```
### What Your AI Agent Gets
**7 tools** exposed via MCP:
| Tool | What It Does | `repo` Param |
| ------------------ | ----------------------------------------------------------------- | -------------- |
| `list_repos` | Discover all indexed repositories | — |
| `query` | Process-grouped hybrid search (BM25 + semantic + RRF) | Optional |
| `context` | 360-degree symbol view — categorized refs, process participation | Optional |
| `impact` | Blast radius analysis with depth grouping and confidence | Optional |
| `detect_changes` | Git-diff impact — maps changed lines to affected processes | Optional |
| `rename` | Multi-file coordinated rename with graph + text search | Optional |
| `cypher` | Raw Cypher graph queries | Optional |
> When only one repo is indexed, the `repo` parameter is optional. With multiple repos, specify which one: `query({query: "auth", repo: "my-app"})`.
**Resources** for instant context:
| Resource | Purpose |
| ----------------------------------------- | ---------------------------------------------------- |
| `gitnexus://repos` | List all indexed repositories (read this first) |
| `gitnexus://repo/{name}/context` | Codebase stats, staleness check, and available tools |
| `gitnexus://repo/{name}/clusters` | All functional clusters with cohesion scores |
| `gitnexus://repo/{name}/cluster/{name}` | Cluster members and details |
| `gitnexus://repo/{name}/processes` | All execution flows |
| `gitnexus://repo/{name}/process/{name}` | Full process trace with steps |
| `gitnexus://repo/{name}/schema` | Graph schema for Cypher queries |
**2 MCP prompts** for guided workflows:
| Prompt | What It Does |
| ----------------- | ------------------------------------------------------------------------- |
| `detect_impact` | Pre-commit change analysis — scope, affected processes, risk level |
| `generate_map` | Architecture documentation from the knowledge graph with mermaid diagrams |
**4 agent skills** installed to `.claude/skills/` automatically:
- **Exploring** — Navigate unfamiliar code using the knowledge graph
- **Debugging** — Trace bugs through call chains
- **Impact Analysis** — Analyze blast radius before changes
- **Refactoring** — Plan safe refactors using dependency mapping
**Repo-specific skills** generated with `--skills`:
When you run `gitnexus analyze --skills`, GitNexus detects the functional areas of your codebase (via Leiden community detection) and generates a `SKILL.md` file for each one under `.claude/skills/generated/`. Each skill describes a module's key files, entry points, execution flows, and cross-area connections — so your AI agent gets targeted context for the exact area of code you're working in. Skills are regenerated on each `--skills` run to stay current with the codebase.
---
## Multi-Repo MCP Architecture
GitNexus uses a **global registry** so one MCP server can serve multiple indexed repos. No per-project MCP config needed — set it up once and it works everywhere.
```mermaid
flowchart TD
START([Repository Input]) --> PASS1
subgraph PASS1 ["Pass 1: Structure Analysis"]
P1A[Recursive Directory Traversal] --> P1B[File Type Classification]
P1B --> P1C[Project/Folder/File Nodes]
P1C --> P1D[CONTAINS Relationships]
subgraph CLI [CLI Commands]
Setup["gitnexus setup"]
Analyze["gitnexus analyze"]
Clean["gitnexus clean"]
List["gitnexus list"]
end
subgraph PASS2 ["Pass 2: Code Parsing & AST"]
P2A[Tree-sitter WASM Init] --> P2B[Grammar Loading]
P2B --> P2C[AST Generation]
P2C --> P2D[Symbol Extraction]
P2D --> P2E[LRU Cache Storage]
subgraph Registry ["~/.gitnexus/"]
RegFile["registry.json"]
end
subgraph PASS3 ["Pass 3: Import Resolution"]
P3A[Import Statement Extraction] --> P3B[Module Path Resolution]
P3B --> P3C[Cross-Reference Tables]
P3C --> P3D[IMPORTS Relationships]
subgraph Repos [Project Repos]
RepoA[".gitnexus/ in repo A"]
RepoB[".gitnexus/ in repo B"]
end
subgraph PASS4 ["Pass 4: Call Graph Analysis"]
P4A[Function Call Pattern Matching] --> P4B[Exact Match via Import Map]
P4B --> P4C[Fuzzy Match + Levenshtein]
P4C --> P4D[CALLS Relationships]
subgraph MCP [MCP Server]
Server["server.ts"]
Backend["LocalBackend"]
Pool["Connection Pool"]
ConnA["LadybugDB conn A"]
ConnB["LadybugDB conn B"]
end
PASS1 --> PASS2
PASS2 --> PASS3
PASS3 --> PASS4
PASS4 --> END([Knowledge Graph])
classDef passBox fill:#e1f5fe,stroke:#01579b,stroke-width:2px,color:#000
classDef startEnd fill:#c8e6c9,stroke:#2e7d32,stroke-width:3px,color:#000
classDef step fill:#fff3e0,stroke:#ef6c00,stroke-width:1px,color:#000
class PASS1,PASS2,PASS3,PASS4 passBox
class START,END startEnd
class P1A,P1B,P1C,P1D,P2A,P2B,P2C,P2D,P2E,P3A,P3B,P3C,P3D,P4A,P4B,P4C,P4D step
Setup -->|"writes global MCP config"| CursorConfig["~/.cursor/mcp.json"]
Analyze -->|"registers repo"| RegFile
Analyze -->|"stores index"| RepoA
Clean -->|"unregisters repo"| RegFile
List -->|"reads"| RegFile
Server -->|"reads registry"| RegFile
Server --> Backend
Backend --> Pool
Pool -->|"lazy open"| ConnA
Pool -->|"lazy open"| ConnB
ConnA -->|"queries"| RepoA
ConnB -->|"queries"| RepoB
```
### Data Flow & Storage Architecture
**How it works:** Each `gitnexus analyze` stores the index in `.gitnexus/` inside the repo (portable, gitignored) and registers a pointer in `~/.gitnexus/registry.json`. When an AI agent starts, the MCP server reads the registry and can serve any indexed repo. LadybugDB connections are opened lazily on first query and evicted after 5 minutes of inactivity (max 5 concurrent). If only one repo is indexed, the `repo` parameter is optional on all tools — agents don't need to change anything.
```mermaid
flowchart TD
START([Repository Input]) --> STRUCT[Structure Processor]
STRUCT --> |Creates nodes/relationships| GRAPH1[In-Memory Graph]
GRAPH1 --> PARSE[Parsing Processor]
PARSE --> |AST Storage| AST_MAP[AST Map]
PARSE --> |Function Registry| FUNC_TRIE[Function Trie]
PARSE --> |Adds definition nodes| GRAPH2[Enhanced Graph]
GRAPH2 --> IMPORT[Import Processor]
AST_MAP --> IMPORT
IMPORT --> |Import Map| IMP_MAP[Import Map]
IMPORT --> |Adds IMPORTS relationships| GRAPH3[Graph + Imports]
GRAPH3 --> CALLS[Call Processor]
AST_MAP --> CALLS
IMP_MAP --> CALLS
FUNC_TRIE --> CALLS
CALLS --> |Adds CALLS relationships| FINAL_GRAPH[Final Knowledge Graph]
FINAL_GRAPH --> JSON_EXPORT[JSON Export]
JSON_EXPORT --> |JSON.stringify| JSON_STRING[JSON String]
JSON_STRING --> |Browser Download| FILE_SYSTEM[File System]
FINAL_GRAPH --> |Direct object reference| UI[UI Components]
subgraph "Storage Points"
GRAPH1
GRAPH2
GRAPH3
FINAL_GRAPH
AST_MAP
FUNC_TRIE
IMP_MAP
JSON_STRING
end
subgraph "Cache Layer"
LRU_CACHE[LRU Cache]
LOCAL_STORAGE[LocalStorage]
end
PARSE -.-> LRU_CACHE
LOCAL_STORAGE -.-> SETTINGS[Settings/Flags]
```
---
### Technical Implementation Details
## Web UI (browser-based)
**Pass 1: Structure Analysis**
A fully client-side graph explorer and AI chat. No server, no install — your code never leaves the browser.
- Implements recursive directory traversal with configurable depth limits
- File type detection using MIME types and extension mapping
- Creates hierarchical node structure with parent-child relationships
- Establishes CONTAINS relationships for project organization
**Try it now:** [gitnexus.vercel.app](https://gitnexus.vercel.app) — drag & drop a ZIP and start exploring.
**Pass 2: Code Parsing & AST Extraction**
<img width="2550" height="1343" alt="gitnexus_img" src="https://github.com/user-attachments/assets/cc5d637d-e0e5-48e6-93ff-5bcfdb929285" />
- Initializes Tree-sitter WASM parsers with language-specific grammars
- Generates Abstract Syntax Trees for each source file
- Implements AST traversal algorithms to extract code symbols
- **LRU Cache System**: Memory-efficient AST storage with configurable eviction policies
- **Parallel Processing**: Web Worker Pool distributes parsing across multiple threads
- **Memory Management**: Automatic cleanup and garbage collection for large codebases
**Pass 3: Import Resolution**
- Extracts import/require statements using AST pattern matching
- Implements module resolution algorithms (Node.js, ES6, Python)
- Builds cross-reference tables for dependency mapping
- Handles relative/absolute path resolution with fallback strategies
**Pass 4: Call Graph Analysis**
- **Stage 1**: Exact function call matching using import resolution data
- **Stage 2**: Fuzzy matching with Levenshtein distance for unresolved calls
- **Stage 3**: Heuristic-based matching for dynamic calls and method chaining
- Creates CALLS relationships with confidence scoring
## Getting Started
**Prerequisites**: Node.js 18+, API keys for AI features
Or run locally:
```bash
git clone <repository-url>
cd gitnexus
npm install
npm run dev
git clone https://github.com/abhigyanpatwari/gitnexus.git
cd gitnexus/gitnexus-shared && npm install && npm run build
cd ../gitnexus-web && npm install
npm run dev
```
Open http://localhost:5173
The web UI uses the same indexing pipeline as the CLI but runs entirely in WebAssembly (Tree-sitter WASM, LadybugDB WASM, in-browser embeddings). It's great for quick exploration but limited by browser memory for larger repos.
**Configuration**
**Local Backend Mode:** Run `gitnexus serve` and open the web UI locally — it auto-detects the server and shows all your indexed repos, with full AI chat support. No need to re-upload or re-index. The agent's tools (Cypher queries, search, code navigation) route through the backend HTTP API automatically.
- GitHub token (optional): Increases rate limit to 5,000/hour
- AI API keys: OpenAI, Anthropic, Gemini, or Azure OpenAI
- Performance: Set file limits and directory filters
---
## Usage
## The Problem GitNexus Solves
**Analyze Repository**
Tools like **Cursor**, **Claude Code**, **Codex**, **Cline**, **Roo Code**, and **Windsurf** are powerful — but they don't truly know your codebase structure.
1. Enter GitHub URL or upload ZIP file
2. Set filters (optional): directories, file patterns, size limits
3. Click "Analyze" and wait for processing
4. Explore the interactive graph
**What happens:**
**AI Chat**
1. AI edits `UserService.validate()`
2. Doesn't know 47 functions depend on its return type
3. **Breaking changes ship**
1. Configure API key in settings
2. Ask questions about the codebase:
- "What functions are in main.py?"
- "Show classes that inherit from BaseClass"
- "How does authentication work?"
### Traditional Graph RAG vs GitNexus
**Export Data**
- Click Export button to download graph as JSON/CSV
## Advanced Features & Work in Progress
### Web Worker Pool Architecture
Traditional approaches give the LLM raw graph edges and hope it explores enough. GitNexus **precomputes structure at index time** — clustering, tracing, scoring — so tools return complete context in one call:
```mermaid
graph LR
MT[Main Thread] --> WM[Worker Manager]
WM --> W1[Worker 1<br/>Tree-sitter Parser]
WM --> W2[Worker 2<br/>Tree-sitter Parser]
WM --> W3[Worker N<br/>Tree-sitter Parser]
W1 --> AST1[AST Cache]
W2 --> AST2[AST Cache]
W3 --> AST3[AST Cache]
AST1 --> LRU[LRU Eviction Policy]
AST2 --> LRU
AST3 --> LRU
```
flowchart TB
subgraph Traditional["Traditional Graph RAG"]
direction TB
U1["User: What depends on UserService?"]
U1 --> LLM1["LLM receives raw graph"]
LLM1 --> Q1["Query 1: Find callers"]
Q1 --> Q2["Query 2: What files?"]
Q2 --> Q3["Query 3: Filter tests?"]
Q3 --> Q4["Query 4: High-risk?"]
Q4 --> OUT1["Answer after 4+ queries"]
end
### LRU Cache Implementation
- **Memory-bounded AST storage** with configurable size limits (default: 1000 entries)
- **Automatic eviction policies** based on access patterns and memory pressure
- **Thread-safe operations** across Web Worker boundaries using Comlink
- **Cache hit optimization** for repeated file analysis and import resolution
- **Garbage collection integration** with browser memory management APIs
### KuzuDB Integration Status (Work in Progress)
```mermaid
graph TD
APP[Application Layer] --> RAG[Graph RAG Agent]
RAG --> CYP[Cypher Query Generator]
CYP --> KDB[KuzuDB WASM Engine]
KDB --> IDB[IndexedDB Persistence]
subgraph STATUS ["Current Status"]
IMPL[KuzuDB WASM Integration - Complete]
PERS[IndexedDB Persistence - Complete]
SCHEMA[Graph Schema Definition - Complete]
QUERY[Cypher Query Execution - WIP]
AGENT[Graph RAG Agent - WIP]
subgraph GN["GitNexus Smart Tools"]
direction TB
U2["User: What depends on UserService?"]
U2 --> TOOL["impact UserService upstream"]
TOOL --> PRECOMP["Pre-structured response:
8 callers, 3 clusters, all 90%+ confidence"]
PRECOMP --> OUT2["Complete answer, 1 query"]
end
classDef complete fill:#c8e6c9,stroke:#2e7d32,stroke-width:2px
classDef wip fill:#fff3e0,stroke:#f57c00,stroke-width:2px
classDef main fill:#e3f2fd,stroke:#1976d2,stroke-width:2px
class IMPL,PERS,SCHEMA complete
class QUERY,AGENT wip
class APP,RAG,CYP,KDB,IDB main
```
**Implementation Status**:
**Core innovation: Precomputed Relational Intelligence**
- ✅ **KuzuDB WASM Engine**: Fully integrated embedded graph database
- ✅ **Graph Schema**: Node and relationship type definitions implemented
- ✅ **Data Ingestion**: Knowledge graph storage in KuzuDB format
- 🚧 **Cypher Query Engine**: Query execution layer under development
- 🚧 **Graph RAG Agent**: AI agent with graph querying capabilities (blocked by Cypher integration)
- **Reliability** — LLM can't miss context, it's already in the tool response
- **Token efficiency** — No 10-query chains to understand one function
- **Model democratization** — Smaller LLMs work because tools do the heavy lifting
**Current Limitation**: The Graph RAG agent cannot execute sophisticated graph queries because the Cypher query execution layer is still being implemented. Basic AI chat works with in-memory graph traversal, but advanced graph reasoning requires the KuzuDB Cypher integration to be completed.
---
### Dual-Engine Architecture
## How It Works
- **Legacy Engine**: Production-ready single-threaded processing with JSON storage
- **Next-Gen Engine**: Parallel processing with KuzuDB persistence (4-8x performance improvement)
- **Automatic Fallback**: System gracefully degrades to legacy engine if next-gen fails
- **Runtime Switching**: Users can toggle between engines without data loss
GitNexus builds a complete knowledge graph of your codebase through a multi-phase indexing pipeline:
## Deployment
1. **Structure** — Walks the file tree and maps folder/file relationships
2. **Parsing** — Extracts functions, classes, methods, and interfaces using Tree-sitter ASTs
3. **Resolution** — Resolves imports, function calls, heritage, constructor inference, and `self`/`this` receiver types across files with language-aware logic
4. **Clustering** — Groups related symbols into functional communities
5. **Processes** — Traces execution flows from entry points through call chains
6. **Search** — Builds hybrid search indexes for fast retrieval
### Supported Languages
| Language | Imports | Named Bindings | Exports | Heritage | Type Annotations | Constructor Inference | Config | Frameworks | Entry Points |
|----------|---------|----------------|---------|----------|-----------------|---------------------|--------|------------|-------------|
| TypeScript | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| JavaScript | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ |
| Python | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Java | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Kotlin | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| C# | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Go | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| Rust | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| PHP | ✓ | ✓ | ✓ | — | ✓ | ✓ | ✓ | ✓ | ✓ |
| Ruby | ✓ | — | ✓ | ✓ | — | ✓ | — | ✓ | ✓ |
| Swift | — | — | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ |
| C | — | — | ✓ | — | ✓ | ✓ | — | ✓ | ✓ |
| C++ | — | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
| Dart | ✓ | — | ✓ | ✓ | ✓ | ✓ | — | ✓ | ✓ |
**Imports** — cross-file import resolution · **Named Bindings** — `import { X as Y }` / re-export tracking · **Exports** — public/exported symbol detection · **Heritage** — class inheritance, interfaces, mixins · **Type Annotations** — explicit type extraction for receiver resolution · **Constructor Inference** — infer receiver type from constructor calls (`self`/`this` resolution included for all languages) · **Config** — language toolchain config parsing (tsconfig, go.mod, etc.) · **Frameworks** — AST-based framework pattern detection · **Entry Points** — entry point scoring heuristics
---
## Tool Examples
### Impact Analysis
```
impact({target: "UserService", direction: "upstream", minConfidence: 0.8})
TARGET: Class UserService (src/services/user.ts)
UPSTREAM (what depends on this):
Depth 1 (WILL BREAK):
handleLogin [CALLS 90%] -> src/api/auth.ts:45
handleRegister [CALLS 90%] -> src/api/auth.ts:78
UserController [CALLS 85%] -> src/controllers/user.ts:12
Depth 2 (LIKELY AFFECTED):
authRouter [IMPORTS] -> src/routes/auth.ts
```
Options: `maxDepth`, `minConfidence`, `relationTypes` (`CALLS`, `IMPORTS`, `EXTENDS`, `IMPLEMENTS`), `includeTests`
### Process-Grouped Search
```
query({query: "authentication middleware"})
processes:
- summary: "LoginFlow"
priority: 0.042
symbol_count: 4
process_type: cross_community
step_count: 7
process_symbols:
- name: validateUser
type: Function
filePath: src/auth/validate.ts
process_id: proc_login
step_index: 2
definitions:
- name: AuthConfig
type: Interface
filePath: src/types/auth.ts
```
### Context (360-degree Symbol View)
```
context({name: "validateUser"})
symbol:
uid: "Function:validateUser"
kind: Function
filePath: src/auth/validate.ts
startLine: 15
incoming:
calls: [handleLogin, handleRegister, UserController]
imports: [authRouter]
outgoing:
calls: [checkPassword, createSession]
processes:
- name: LoginFlow (step 2/7)
- name: RegistrationFlow (step 3/5)
```
### Detect Changes (Pre-Commit)
```
detect_changes({scope: "all"})
summary:
changed_count: 12
affected_count: 3
changed_files: 4
risk_level: medium
changed_symbols: [validateUser, AuthService, ...]
affected_processes: [LoginFlow, RegistrationFlow, ...]
```
### Rename (Multi-File)
```
rename({symbol_name: "validateUser", new_name: "verifyUser", dry_run: true})
status: success
files_affected: 5
total_edits: 8
graph_edits: 6 (high confidence)
text_search_edits: 2 (review carefully)
changes: [...]
```
### Cypher Queries
```cypher
-- Find what calls auth functions with high confidence
MATCH (c:Community {heuristicLabel: 'Authentication'})<-[:CodeRelation {type: 'MEMBER_OF'}]-(fn)
MATCH (caller)-[r:CodeRelation {type: 'CALLS'}]->(fn)
WHERE r.confidence > 0.8
RETURN caller.name, fn.name, r.confidence
ORDER BY r.confidence DESC
```
---
## Wiki Generation
Generate LLM-powered documentation from your knowledge graph:
```bash
npm run build
npm run preview
# Requires an LLM API key (OPENAI_API_KEY, etc.)
gitnexus wiki
# Use a custom model or provider
gitnexus wiki --model gpt-4o
gitnexus wiki --base-url https://api.anthropic.com/v1
# Force full regeneration
gitnexus wiki --force
```
**Environment Variables**
The wiki generator reads the indexed graph structure, groups files into modules via LLM, generates per-module documentation pages, and creates an overview page — all with cross-references to the knowledge graph.
```env
VITE_OPENAI_API_KEY=sk-...
VITE_DEFAULT_MAX_FILES=500
VITE_ENABLE_DEBUG_LOGGING=false
```
---
## Tech Stack
| Layer | CLI | Web |
| ------------------------- | ------------------------------------- | --------------------------------------- |
| **Runtime** | Node.js (native) | Browser (WASM) |
| **Parsing** | Tree-sitter native bindings | Tree-sitter WASM |
| **Database** | LadybugDB native | LadybugDB WASM |
| **Embeddings** | HuggingFace transformers.js (GPU/CPU) | transformers.js (WebGPU/WASM) |
| **Search** | BM25 + semantic + RRF | BM25 + semantic + RRF |
| **Agent Interface** | MCP (stdio) | LangChain ReAct agent |
| **Visualization** | — | Sigma.js + Graphology (WebGL) |
| **Frontend** | — | React 18, TypeScript, Vite, Tailwind v4 |
| **Clustering** | Graphology | Graphology |
| **Concurrency** | Worker threads + async | Web Workers + Comlink |
---
## Roadmap
### Actively Building
- [ ] **LLM Cluster Enrichment** — Semantic cluster names via LLM API
- [ ] **AST Decorator Detection** — Parse @Controller, @Get, etc.
- [ ] **Incremental Indexing** — Only re-index changed files
### Recently Completed
- [X] Constructor-Inferred Type Resolution, `self`/`this` Receiver Mapping
- [X] Wiki Generation, Multi-File Rename, Git-Diff Impact Analysis
- [X] Process-Grouped Search, 360-Degree Context, Claude Code Hooks
- [X] Multi-Repo MCP, Zero-Config Setup, 14 Language Support
- [X] Community Detection, Process Detection, Confidence Scoring
- [X] Hybrid Search, Vector Index
---
## Security & Privacy
- All processing happens in your browser
- API keys stored locally, never transmitted
- No code or results stored remotely
- Uses GitHub public API only
- **CLI**: Everything runs locally on your machine. No network calls. Index stored in `.gitnexus/` (gitignored). Global registry at `~/.gitnexus/` stores only paths and metadata.
- **Web**: Everything runs in your browser. No code uploaded to any server. API keys stored in localStorage only.
- Open source — audit the code yourself.
## Contributing
1. Fork the repository
2. Create feature branch: `git checkout -b feature/name`
3. Make changes and test
4. Commit: `git commit -m 'Add feature'`
5. Push and open Pull Request
**Code Style**: TypeScript strict mode, ESLint rules, minimal comments
## License
MIT License - see [LICENSE](LICENSE) file
---
## Acknowledgments
- Tree-sitter for syntax parsing
- LangChain.js for AI agents
- D3.js for graph visualization
- KuzuDB for embedded database
- [code-graph-rag](https://github.com/vitali87/code-graph-rag) for reference implementation
- [Tree-sitter](https://tree-sitter.github.io/) — AST parsing
- [LadybugDB](https://ladybugdb.com/) — Embedded graph database with vector support (formerly KuzuDB)
- [Sigma.js](https://www.sigmajs.org/) — WebGL graph rendering
- [transformers.js](https://huggingface.co/docs/transformers.js) — Browser ML
- [Graphology](https://graphology.github.io/) — Graph data structures
- [MCP](https://modelcontextprotocol.io/) — Model Context Protocol
+163
View File
@@ -0,0 +1,163 @@
# Runbook — GitNexus
Short, copy-paste operations for **local development**, **MCP**, and **CI**. Commands assume a Unix shell; on Windows use Git Bash or equivalent paths.
## Prerequisites
- **Node.js** ≥ 20 (`gitnexus-web/package.json` `engines`).
- **Git** (analyze requires a git repository).
- From repo root, install and build the CLI package:
```bash
cd gitnexus
npm install
npm run build
```
Use `npx gitnexus …` from any path after global/published install, or `node dist/cli/index.js …` when developing from `gitnexus/` with a local build.
---
## Index out of date / “stale” tools
**Symptom:** MCP or resources warn the index is behind `HEAD`, or results don’t reflect recent commits.
**Fix (from the target repo root):**
```bash
npx gitnexus analyze
```
**Force full rebuild** (same commit but suspect corruption or changed ignore rules):
```bash
npx gitnexus analyze --force
```
**Check status:**
```bash
npx gitnexus status
```
**List what MCP knows about:**
```bash
npx gitnexus list
```
---
## Embeddings
**First time with vectors** (slower, more disk/RAM):
```bash
npx gitnexus analyze --embeddings
```
**Important:** If you already had embeddings, **always** pass `--embeddings` on later analyzes, or they can be dropped. See `stats.embeddings` in `.gitnexus/meta.json` (0 means none).
**Large repos:** Analyze may skip or limit embedding work when node counts are very high; watch CLI output.
---
## MCP: no repos / empty tools
**Symptom:** `GitNexus: No indexed repos yet` on stderr when starting MCP.
**Fix:** In each project you want indexed:
```bash
cd /path/to/repo
npx gitnexus analyze
```
Restart the editor MCP session if needed. The server **refreshes the registry lazily**; new analyzes are picked up without necessarily reinstalling MCP.
**Symptom:** Wrong repo when multiple are indexed — pass `repo` on tools or use `list_repos` first.
---
## Clean slate (corrupt or huge `.gitnexus`)
**Current repo only** (prompts for confirmation):
```bash
npx gitnexus clean
```
**Skip confirmation:**
```bash
npx gitnexus clean --force
```
**All registered repos:**
```bash
npx gitnexus clean --all --force
```
Then re-run `npx gitnexus analyze` (and `--embeddings` if you need vectors).
---
## Local bridge for the web UI
```bash
cd gitnexus
npx gitnexus serve
# default http://127.0.0.1:4747 — see serve --help for port/host
```
Use when the browser UI should talk to **local** indexed repos instead of WASM-only mode.
---
## CLI equivalents of MCP tools
Useful for debugging without an editor:
```bash
cd gitnexus
npx gitnexus query "authentication flow" --repo MyRepo
npx gitnexus context SomeSymbol --repo MyRepo
npx gitnexus impact SomeSymbol --direction upstream --repo MyRepo
npx gitnexus cypher "MATCH (n) RETURN count(n) LIMIT 1" --repo MyRepo
```
---
## CI failures (contributors)
Orchestrator: `.github/workflows/ci.yml`.
| Job | Typical local repro |
|-----|---------------------|
| **quality** | `cd gitnexus && npx tsc --noEmit` |
| **unit-tests** | `cd gitnexus && npx vitest run test/unit` |
| **integration** | `cd gitnexus && npx vitest run test/integration` (see workflow matrix for groups) |
| **e2e** | Triggered when `gitnexus-web/` changes; `cd gitnexus-web && E2E=1 npx playwright test` (requires `gitnexus serve` + `npm run dev`) |
**Note:** Pushes that touch only certain markdown paths may be skipped by `paths-ignore` in CI — see workflow file for exact patterns.
---
## Memory / analyze crashes
Analyze re-execs Node with a **large old-space heap** when needed (`analyze.ts`). If you still OOM on huge repos, close other processes, avoid `--embeddings` for a first pass, or analyze a smaller path if supported by your workflow.
---
## LadybugDB / lock errors
Only one process should open a repo’s `.gitnexus/lbug` store at a time. If MCP and a second `analyze` run conflict, stop one process, then retry `analyze` or restart MCP.
---
## Where to dig deeper
- Architecture overview: [ARCHITECTURE.md](ARCHITECTURE.md)
- Agent safety rules: [GUARDRAILS.md](GUARDRAILS.md)
- Tests: [TESTING.md](TESTING.md)
+95
View File
@@ -0,0 +1,95 @@
# Testing — GitNexus
How we structure tests and which commands to run locally and in CI.
## Packages
| Package | Path | Runner | Notes |
| -------------- | -------------- | -------- | ------------------------------ |
| CLI + MCP core | `gitnexus/` | Vitest | Primary test surface in CI |
| Web UI | `gitnexus-web/`| Vitest | Unit/component tests |
| Web UI E2E | `gitnexus-web/`| Playwright | Run when changing UI flows |
## Commands (local)
From repository root, unless noted:
**`gitnexus` (CLI / library)**
```bash
cd gitnexus
npm install
npm run build
npm test # unit: vitest run test/unit
npm run test:integration # integration suite
npm run test:all
npm run test:coverage
npx tsc --noEmit # typecheck (matches CI)
```
**`gitnexus-web`**
```bash
cd gitnexus-web
npm install
npm test # unit tests (vitest)
npx tsc -b --noEmit # typecheck (matches CI)
npm run test:coverage
npm run test:e2e # Playwright (requires gitnexus serve + npm run dev)
```
## Pre-commit hook
A husky pre-commit hook (`.husky/pre-commit`) runs automatically on every `git commit`:
- **`gitnexus-web/` files staged** → `tsc -b --noEmit` + `vitest run`
- **`gitnexus/` files staged** → `tsc --noEmit` + `vitest run --project default`
Skip with `git commit --no-verify` (use sparingly).
## Test categories
- **Unit** — Pure logic, parsers, graph/query helpers; fast; no network.
- **Integration** — Real combinations (filesystem, MCP wiring, larger pipelines) as already organized under `gitnexus/test/integration`.
- **Eval-style / golden sets** — For agent- or classification-style behavior, keep labeled inputs and expected outputs (JSON or table-driven tests) and run them in CI when relevant.
- **E2E (web)** — Critical user paths only; prefer `data-testid` attributes for stable selectors. Tests run against real backend (`gitnexus serve`) and Vite dev server.
## Performance metrics (targets)
Set targets to match team expectations, then tune to this repo’s CI reality:
| Metric | Target (initial) | Notes |
| ------------------- | ---------------- | ------------------------------------------ |
| Unit coverage | Align with CI | CI runs Vitest with coverage in `gitnexus` |
| Unit wall time | Fast PR feedback | Use `vitest run test/unit` for tight loop |
| Integration duration| &lt; few minutes | Guard heavy tests with env flags if needed |
## Regression testing
Re-run the full relevant suite when:
- Prompt or agent-behavior documentation changes (if tests encode behavior)
- Model or embedding-related code paths change
- Graph schema, query contracts, or MCP tool shapes change
- Dependencies with parsing or runtime impact upgrade
## CI integration
GitHub Actions (`.github/workflows/ci.yml`) orchestrate:
- **`ci-quality.yml`** — `tsc --noEmit` for `gitnexus/` + `tsc -b --noEmit` for `gitnexus-web/`
- **`ci-tests.yml`** — `vitest run` with coverage (ubuntu) + cross-platform (macOS, Windows)
- **`ci-e2e.yml`** — Playwright E2E tests, gated on `gitnexus-web/**` changes
Local checks before pushing:
```bash
cd gitnexus && npx tsc --noEmit && npm test
cd ../gitnexus-web && npx tsc -b --noEmit && npm test
```
Or rely on the pre-commit hook which runs these automatically for staged files.
## User acceptance / beta (optional)
For staged releases or UI betas: deploy to a staging environment, collect structured feedback, watch errors and latency, then iterate before a wider release.
-375
View File
@@ -1,375 +0,0 @@
# Worker Pool Implementation Summary for Byterover
## 🎯 Project Context
**Project**: GitNexus - Client-side, edge-based code knowledge graph generator
**Implementation Date**: December 2024
**Primary Goal**: Massive performance improvement for large codebases through parallel processing
## 🚀 Performance Benefits Achieved
### **Expected Speedup by Codebase Size:**
- **Small codebases (< 100 files)**: 1.5-2x speedup
- **Medium codebases (100-1000 files)**: 2-4x speedup
- **Large codebases (1000+ files)**: 4-8x speedup
### **Key Performance Improvements:**
- **Parallel file parsing** - Multiple files processed simultaneously
- **Concurrent Tree-sitter operations** - AST generation in parallel
- **Better CPU utilization** - Leverages all available cores
- **Improved UI responsiveness** - Main thread freed up
## 📁 Files Created/Modified
### **Core Implementation Files:**
#### 1. `src/lib/web-worker-pool.ts` (NEW)
**Purpose**: Browser-compatible Web Worker Pool implementation
**Key Features**:
- Replaces Node.js `worker_threads` with standard Web Workers
- Manages worker lifecycle, task queuing, and error handling
- Supports progress tracking and batch processing
- Includes `FileProcessingPool` and `WebWorkerPoolUtils`
**Critical Code Patterns**:
```typescript
export class WebWorkerPool {
private workers: Worker[] = [];
private availableWorkers: Worker[] = [];
private taskQueue: WorkerTask<unknown, unknown>[] = [];
private activeTasks: Map<string, WorkerTask<unknown, unknown>> = new Map();
async execute<TInput, TOutput>(input: TInput): Promise<TOutput>
async executeWithProgress<TInput, TOutput>(inputs: TInput[], onProgress?: (completed: number, total: number) => void): Promise<TOutput[]>
async shutdown(): Promise<void>
}
```
#### 2. `src/core/ingestion/parallel-parsing-processor.ts` (NEW)
**Purpose**: Parallel file parsing using worker pool
**Key Features**:
- Replaces sequential `ParsingProcessor`
- Uses `tree-sitter-worker.js` for parallel AST parsing
- Integrates with `FunctionRegistryTrie` for optimized lookups
- Handles worker pool initialization and cleanup
**Critical Code Patterns**:
```typescript
export class ParallelParsingProcessor implements GraphProcessor<ParsingInput> {
private workerPool: WebWorkerPool;
async process(graph: KnowledgeGraph, input: ParsingInput): Promise<void>
private async processFilesInParallel(filePaths: string[], fileContents: Map<string, string>): Promise<ParallelParsingResult[]>
private async processResults(results: ParallelParsingResult[], graph: KnowledgeGraph): Promise<void>
}
```
#### 3. `src/core/ingestion/parallel-pipeline.ts` (NEW)
**Purpose**: Parallel 4-pass ingestion pipeline
**Key Features**:
- Replaces original `GraphPipeline`
- Integrates `ParallelParsingProcessor` for Pass 2
- Provides progress callbacks and performance logging
- Ensures proper worker resource cleanup
**Critical Code Patterns**:
```typescript
export class ParallelGraphPipeline {
private parsingProcessor: ParallelParsingProcessor;
public async run(input: PipelineInput): Promise<KnowledgeGraph>
public static isParallelProcessingSupported(): boolean
public static getOptimalWorkerCount(): number
}
```
### **Worker Scripts:**
#### 4. `public/workers/tree-sitter-worker.js` (NEW)
**Purpose**: Dedicated Tree-sitter parsing worker
**Key Features**:
- Initializes Tree-sitter and language parsers in worker context
- Supports TypeScript, JavaScript, Python parsing
- Extracts definitions using Tree-sitter queries
- Communicates results back to main thread
#### 5. `public/workers/generic-worker.js` (NEW)
**Purpose**: General-purpose processing worker
**Key Features**:
- Text analysis (word count, identifier extraction)
- File analysis (basic stats, language detection)
- Data processing (deduplication, filtering, transformation)
- Pattern matching and statistical analysis
#### 6. `public/workers/file-processing-worker.js` (NEW)
**Purpose**: Specialized file processing worker
**Key Features**:
- Leverages tree-sitter worker for parsing
- File structure analysis
- Dependency extraction (ES6 imports, CommonJS requires)
- Code complexity analysis
### **Configuration & Testing:**
#### 7. `src/config/feature-flags.ts` (MODIFIED)
**Changes**: Added worker pool feature flags
```typescript
// New flags added:
enableWorkerPool: boolean;
enableParallelParsing: boolean;
enableParallelProcessing: boolean;
// New methods:
enableWorkerPool(): void
disableWorkerPool(): void
```
#### 8. `src/lib/worker-pool-test.ts` (NEW)
**Purpose**: Comprehensive test suite
**Key Features**:
- Basic functionality tests
- File processing tests
- Performance benchmarking
- Error handling tests
- Browser console testing support
#### 9. `WORKER_POOL_IMPLEMENTATION_GUIDE.md` (NEW)
**Purpose**: Complete documentation
**Contents**:
- Performance benefits and benchmarks
- File structure and architecture
- Usage examples and configuration
- Testing instructions
- Migration guide from sequential to parallel
## 🔧 Technical Architecture
### **Worker Pool Design Pattern:**
```typescript
// Worker Pool Lifecycle
1. Initialize pool with optimal worker count
2. Queue tasks for processing
3. Distribute tasks to available workers
4. Collect results and handle errors
5. Recycle workers for next tasks
6. Shutdown and cleanup resources
```
### **Parallel Processing Flow:**
```typescript
// 4-Pass Pipeline with Parallel Pass 2
Pass 1: Structure Analysis (Sequential - lightweight)
Pass 2: Code Parsing (Parallel - CPU intensive) ← NEW
Pass 3: Import Resolution (Sequential - depends on Pass 2)
Pass 4: Call Resolution (Sequential - depends on Pass 3)
```
### **Worker Communication Pattern:**
```typescript
// Main Thread → Worker
worker.postMessage({
taskId: string,
input: TaskInput
});
// Worker → Main Thread
self.postMessage({
taskId: string,
result: TaskOutput | error: string
});
```
## 🎯 Integration Points
### **Feature Flag Integration:**
```typescript
// Check if worker pool is enabled
if (isWorkerPoolEnabled()) {
// Use parallel processing
const pipeline = new ParallelGraphPipeline();
} else {
// Fallback to sequential processing
const pipeline = new GraphPipeline();
}
```
### **Performance Monitoring:**
```typescript
// Worker pool statistics
const stats = workerPool.getStats();
console.log('Worker Pool Stats:', {
totalWorkers: stats.totalWorkers,
availableWorkers: stats.availableWorkers,
activeTasks: stats.activeTasks,
queuedTasks: stats.queuedTasks
});
```
## 🚨 Error Handling & Fallbacks
### **Worker Pool Error Handling:**
- Worker crashes are handled gracefully
- Failed workers are replaced automatically
- Task timeouts prevent hanging operations
- Fallback to sequential processing if workers fail
### **Browser Compatibility:**
- Checks for Web Worker support
- Graceful degradation for unsupported browsers
- Hardware concurrency detection
- Memory usage monitoring
## 📊 Performance Metrics
### **Benchmark Results:**
- **File Processing**: 4-8x faster for large codebases
- **Memory Usage**: Efficient worker recycling
- **CPU Utilization**: Near 100% on multi-core systems
- **UI Responsiveness**: Main thread remains responsive
### **Scalability:**
- **Worker Count**: Automatically optimized based on hardware
- **Task Distribution**: Intelligent load balancing
- **Memory Management**: Automatic cleanup and recycling
- **Error Recovery**: Robust error handling and recovery
## 🔄 Migration Strategy
### **From Sequential to Parallel:**
1. **Feature Flag**: Enable `enableWorkerPool` flag
2. **Pipeline Switch**: Replace `GraphPipeline` with `ParallelGraphPipeline`
3. **Processor Update**: Use `ParallelParsingProcessor` for Pass 2
4. **Testing**: Run comprehensive test suite
5. **Monitoring**: Track performance improvements
### **Backward Compatibility:**
- All existing APIs remain unchanged
- Feature flags control behavior
- Graceful fallback to sequential processing
- No breaking changes to existing code
## 🎯 Future Enhancements
### **Planned Improvements:**
1. **Dynamic Worker Scaling**: Adjust worker count based on load
2. **Advanced Caching**: Cache parsed ASTs for repeated processing
3. **Streaming Processing**: Process files as they're uploaded
4. **Priority Queuing**: Prioritize critical files for processing
5. **Distributed Processing**: Support for multiple browser tabs/workers
### **Performance Optimizations:**
1. **Worker Pool Pooling**: Reuse worker pools across sessions
2. **Memory Optimization**: Better memory management for large files
3. **Load Balancing**: Intelligent task distribution
4. **Preemptive Processing**: Start processing before all files are loaded
## 📝 Critical Implementation Details
### **Worker Script Loading:**
- Worker scripts are served from `/public/workers/`
- ES6 modules are used for better code organization
- Tree-sitter WASM files are loaded dynamically
- Error handling for missing worker scripts
### **Task Serialization:**
- Tasks are serialized for worker communication
- Complex objects are simplified for transfer
- Function references are converted to strings
- Results are deserialized on main thread
### **Memory Management:**
- Workers are recycled after task completion
- Large objects are transferred, not copied
- Memory usage is monitored and logged
- Automatic cleanup on pipeline shutdown
## 🔍 Testing Strategy
### **Test Coverage:**
- **Unit Tests**: Individual worker pool functions
- **Integration Tests**: End-to-end pipeline testing
- **Performance Tests**: Benchmarking with various file sizes
- **Error Tests**: Worker failure and recovery scenarios
- **Browser Tests**: Cross-browser compatibility
### **Test Commands:**
```typescript
// Browser console testing
window.testWorkerPoolBasic()
window.testFileProcessingPool()
window.testWorkerPoolPerformance()
window.runWorkerPoolTests()
```
## 📚 Documentation & Resources
### **Key Documentation Files:**
- `WORKER_POOL_IMPLEMENTATION_GUIDE.md` - Complete implementation guide
- `src/lib/worker-pool-test.ts` - Test suite with examples
- `public/workers/*.js` - Worker script documentation
### **Architecture Diagrams:**
- Worker Pool Lifecycle
- Parallel Processing Flow
- Error Handling Flow
- Performance Monitoring
## 🎯 Success Metrics
### **Performance Improvements:**
- ✅ 4-8x speedup for large codebases
- ✅ Improved UI responsiveness
- ✅ Better CPU utilization
- ✅ Reduced memory pressure
### **Code Quality:**
- ✅ Comprehensive error handling
- ✅ Extensive test coverage
- ✅ Clear documentation
- ✅ Backward compatibility
### **User Experience:**
- ✅ Progress tracking and feedback
- ✅ Graceful error recovery
- ✅ Automatic optimization
- ✅ Feature flag control
## 🔧 Configuration Options
### **Worker Pool Configuration:**
```typescript
const workerPool = new WebWorkerPool({
maxWorkers: navigator.hardwareConcurrency || 4,
workerScript: '/workers/tree-sitter-worker.js',
timeout: 60000, // 60 seconds
name: 'ParallelParsingPool'
});
```
### **Feature Flags:**
```typescript
// Enable all worker pool features
featureFlags.enableWorkerPool();
// Disable worker pool features
featureFlags.disableWorkerPool();
// Check worker pool status
const isEnabled = isWorkerPoolEnabled();
```
## 🚀 Deployment Notes
### **Production Considerations:**
- Worker scripts must be served from public directory
- Tree-sitter WASM files must be available
- Feature flags control rollout
- Performance monitoring is essential
- Error logging for debugging
### **Browser Support:**
- Modern browsers with Web Worker support
- ES6 module support required
- WASM support for Tree-sitter
- Hardware concurrency detection
This implementation represents a significant architectural improvement to GitNexus, providing massive performance benefits for large codebases while maintaining backward compatibility and robust error handling.
+57
View File
@@ -0,0 +1,57 @@
---
review_agents: [kieran-typescript-reviewer, pattern-recognition-specialist, architecture-strategist, data-integrity-guardian, security-sentinel, performance-oracle, code-simplicity-reviewer]
plan_review_agents: [kieran-typescript-reviewer, architecture-strategist, code-simplicity-reviewer]
voltagent_agents: [voltagent-lang:typescript-pro, voltagent-qa-sec:security-auditor, voltagent-data-ai:database-optimizer]
---
# Review Context
## Project Overview
GitNexus is a code intelligence tool that builds a knowledge graph from source code using tree-sitter AST parsing across 12 languages and KuzuDB for graph storage. Two packages: `gitnexus/` (CLI/MCP, TypeScript) and `gitnexus-web/` (browser).
## Cross-Language Pattern Consistency (pattern-recognition-specialist)
- 12 language-specific type extractors in `gitnexus/src/core/ingestion/type-extractors/` must follow identical patterns for: async unwrapping, constructor binding, namespace handling, nullable type stripping, for-loop element typing.
- Past bugs: C#/Rust missing `await_expression` unwrapping that TypeScript handled correctly; PHP backslash namespace splitting inconsistent with other languages' `::` / `.` splitting.
- When reviewing type extractor changes, verify the same pattern exists in ALL applicable language files — asymmetry is the #1 source of bugs.
## Data Integrity (data-integrity-guardian)
- KuzuDB graph operations: schema in `gitnexus/src/core/kuzu/schema.ts`, adapter in `kuzu-adapter.ts`.
- The ingestion pipeline writes symbols and relationships to the graph — changes to node/relation schemas or the ingestion pipeline can corrupt the index.
- Known issue: KuzuDB `close()` hangs on Linux due to C++ destructor — use `detachKuzu()` pattern.
- `lbug-adapter.ts` fallback path needs quote/newline escaping for Cypher injection prevention.
## Security (security-sentinel)
- Cypher query construction in `lbug-adapter.ts` and `kuzu-adapter.ts` — watch for injection via unescaped user-provided symbol names.
- CLI accepts `--repo` parameter and file paths — validate against path traversal.
- MCP server exposes tools to external AI agents — all tool inputs are untrusted.
## Performance (performance-oracle)
- Tree-sitter buffer size is adaptive (512KB–32MB) via `getTreeSitterBufferSize()` in `constants.ts`.
- The ingestion pipeline processes entire repositories — O(n) per file with potential O(n²) in cross-file resolution.
- KuzuDB batch inserts vs individual inserts matter for large repos.
## Architecture (architecture-strategist)
- Ingestion pipeline phases: structure → parsing → imports → calls → heritage → processes → type resolution.
- Shared modules: `export-detection.ts`, `constants.ts`, `utils.ts` — changes here have wide blast radius.
- `gitnexus-web` package drifts behind CLI — flag if a change should be mirrored.
## Voltagent Supplementary Agents
Invoke these via the Agent tool alongside `/ce:review` for deeper specialist analysis. These cover gaps that compound-engineering agents don't:
### voltagent-lang:typescript-pro
**When:** Changes touch type-resolution logic, generics, conditional types, or complex type-level programming in `type-env.ts`, `type-extractors/*.ts`, or `types.ts`.
**Why:** The type resolution system uses advanced TypeScript patterns (discriminated unions, mapped types, recursive generics) that benefit from deep TS type-system review beyond what kieran-typescript-reviewer covers.
### voltagent-qa-sec:security-auditor
**When:** Changes touch MCP tool handlers, Cypher query construction, CLI argument parsing, or any code that processes external input.
**Why:** GitNexus is an MCP server — all tool inputs come from untrusted AI agents. Systematic OWASP-level audit catches injection vectors that spot-checking misses. Past finding: `lbug-adapter.ts` fallback path had unescaped newlines in Cypher queries.
### voltagent-data-ai:database-optimizer
**When:** Changes touch `kuzu-adapter.ts`, `schema.ts`, `lbug-adapter.ts`, or any Cypher query construction/execution.
**Why:** No CE agent specializes in graph database optimization. KuzuDB batch insert patterns, index usage, and query planning directly affect analysis speed on large repos.
## Review Tooling
- Use `gitnexus_impact()` before approving changes to any symbol — check d=1 (WILL BREAK) callers.
- Use `gitnexus_detect_changes({scope: "compare", base_ref: "main"})` to map PR diffs to affected execution flows.
- Use claude-mem to surface past architectural decisions relevant to the code under review.
-1
View File
@@ -1 +0,0 @@
-1
View File
@@ -1 +0,0 @@
+100
View File
@@ -0,0 +1,100 @@
# COBOL Code Indexing
GitNexus indexes COBOL codebases using a **regex-only extraction** strategy, bypassing tree-sitter entirely. This document explains why, how the pipeline works, and links to detailed sub-documents.
## Why Regex-Only?
The tree-sitter-cobol grammar (v0.0.1) has three critical limitations that make it unusable for production indexing:
| Issue | Impact | Severity |
|-------|--------|----------|
| External scanner hangs on ~5% of files | No timeout mechanism exists for the C scanner; the process blocks indefinitely | **Blocking** |
| Only ~15% of paragraph headers detected | Most procedure-division paragraphs are invisible to the grammar | High |
| Patch markers in cols 1-6 cause parse errors | Enterprise COBOL uses non-standard sequence area content (e.g., `mzADD`, `estero`, `#FIX`) | High |
Because the external scanner hang cannot be interrupted (there is no `setTimeoutMicros` equivalent for tree-sitter), using tree-sitter-cobol would hang the indexing pipeline on a non-trivial fraction of real-world files.
The regex-only approach provides:
- **Speed**: ~1ms per file average extraction time
- **Reliability**: zero hangs, zero crashes across 13,000+ files
- **Coverage**: captures all critical symbols -- program name, paragraphs, sections, CALL, PERFORM, COPY, data items (01-77, 88-level), file declarations, FD entries, EXEC SQL/CICS blocks, ENTRY points, and MOVE statements
## Architecture
```mermaid
flowchart TD
A[Repository Scan] --> B{File Detection}
B -->|Extension match| C[COBOL file]
B -->|GITNEXUS_COBOL_DIRS match| C
B -->|No match| Z[Skip]
C --> D{Copybook?}
D -->|Yes| E[Add to Copybook Map]
D -->|No| F[Source Program]
E --> G[COPY Expansion Engine]
F --> G
G -->|Inline copybook content| H[Expanded Source]
H --> I[Patch Marker Cleanup]
I --> J[Regex State Machine]
J --> K[Extracted Symbols]
K --> L[Graph Model Builder]
L --> M[Knowledge Graph]
subgraph "Per-Chunk Processing"
G
H
I
J
K
L
end
subgraph "Post-Processing"
M --> N[Community Detection]
M --> O[Process Detection]
M --> P[Contract Detection]
end
style J fill:#e8f5e9,stroke:#2e7d32
style G fill:#e3f2fd,stroke:#1565c0
```
## COBOL vs Tree-Sitter Languages
| Feature | COBOL (Regex) | Tree-Sitter Languages |
|---------|--------------|----------------------|
| Parser | Single-pass regex state machine | tree-sitter grammar + queries |
| Speed | ~1ms/file | ~5ms/file |
| AST available | No | Yes |
| COPY expansion | Yes (pre-processing step) | N/A |
| Deep indexing | Data items, SQL, CICS, FD, ENTRY | Type annotations, generics, etc. |
| Call extraction | PERFORM (intra-file) + CALL (cross-program) | AST-based call site detection |
| Import extraction | COPY statements | `import`/`require`/`use`/`#include` |
| Coverage | All critical symbols | Language-dependent query coverage |
| Failure mode | Never hangs | External scanner can hang (COBOL only) |
## Sub-Documents
| Document | Description |
|----------|-------------|
| [File Detection](./file-detection.md) | Extension mapping, `GITNEXUS_COBOL_DIRS`, copybook classification |
| [COPY Expansion](./copy-expansion.md) | Copybook inlining, REPLACING transformations, cycle detection |
| [Regex Extraction](./regex-extraction.md) | State machine, regex patterns, line processing |
| [Deep Indexing](./deep-indexing.md) | Data items, EXEC SQL/CICS, file declarations, FD, ENTRY, MOVE |
| [Graph Model](./graph-model.md) | COBOL-specific node types, edge types, full annotated example |
| [Performance](./performance.md) | Benchmarks, worker pool tuning, caps, troubleshooting |
## Key Source Files
| File | Purpose |
|------|---------|
| `gitnexus/src/core/ingestion/cobol-preprocessor.ts` | Patch marker cleanup + regex extraction engine |
| `gitnexus/src/core/ingestion/cobol-copy-expander.ts` | COPY statement expansion with REPLACING |
| `gitnexus/src/core/ingestion/utils.ts` | `getLanguageFromPath`, `getLanguageFromFilename` |
| `gitnexus/src/core/ingestion/pipeline.ts` | `isCobolCopybook`, `expandCobolCopies`, `detectCrossProgamContracts` |
| `gitnexus/src/core/ingestion/workers/parse-worker.ts` | `processCobolRegexOnly` -- graph model builder |
| `gitnexus/src/core/ingestion/workers/worker-pool.ts` | Configurable sub-batch size for COBOL |
+157
View File
@@ -0,0 +1,157 @@
# COBOL COPY Expansion
The COPY statement is COBOL's include mechanism -- analogous to `#include` in C or `import` in modern languages. GitNexus expands COPY statements **before** regex extraction so that symbols defined inside copybooks (data items, paragraphs, etc.) are visible in the program's extracted graph.
## Supported Syntax
### Basic COPY
```cobol
COPY CPSESP.
COPY "WORKGRID.CPY".
```
Inlines the content of the named copybook, replacing the COPY line(s).
### COPY with REPLACING
```cobol
COPY CPSESP REPLACING "ANAZI-KEY" BY "LK-KEY".
COPY CPSESP REPLACING LEADING "ESP-" BY "LK-ESP-"
LEADING "KPSESPL" BY "LK-KPSESPL".
COPY LINKAGE REPLACING TRAILING "-IN" BY "-OUT".
```
Three REPLACING types are supported:
| Type | Syntax | Behavior | Example |
| ------------ | ------------------------------------ | --------------------------------------- | -------------------------------- |
| **EXACT** | `REPLACING "OLD" BY "NEW"` | Replace exact identifier matches | `ANAZI-KEY` becomes `LK-KEY` |
| **LEADING** | `REPLACING LEADING "PFX-" BY "NEW-"` | Replace prefix on all COBOL identifiers | `ESP-NAME` becomes `LK-ESP-NAME` |
| **TRAILING** | `REPLACING TRAILING "-IN" BY "-OUT"` | Replace suffix on all COBOL identifiers | `DATA-IN` becomes `DATA-OUT` |
Multiple REPLACING clauses can appear in a single COPY statement. They are applied in order to each COBOL identifier in the copybook content.
### Multi-Line COPY
COPY statements can span multiple lines (standard COBOL continuation rules apply):
```cobol
COPY CPSESP REPLACING
- LEADING "ESP-" BY "LK-ESP-"
- LEADING "KPSESPL" BY "LK-KPSESPL".
```
Continuation lines (indicator `-` in column 7) are merged before COPY statement scanning.
## Expansion Flow
```mermaid
sequenceDiagram
participant Pipeline
participant Expander as COPY Expander
participant Resolver
participant Reader
Pipeline->>Pipeline: Identify all COBOL files
Pipeline->>Pipeline: Classify copybooks vs programs
Pipeline->>Reader: Read all copybook content upfront
Reader-->>Pipeline: Copybook content map (name -> content)
loop For each source file in chunk
Pipeline->>Expander: expandCopies(content, filePath, resolveFile, readFile)
Expander->>Expander: Merge continuation lines
Expander->>Expander: Detect COPY statements via regex
loop For each COPY statement (reverse order)
Expander->>Resolver: resolveFile(copyTarget)
Resolver-->>Expander: Copybook key or null
alt Resolved successfully
Expander->>Reader: readFile(resolvedKey)
Reader-->>Expander: Copybook content
Expander->>Expander: Apply REPLACING transformations
Expander->>Expander: Recurse for nested COPYs (depth + 1)
Expander->>Expander: Splice expanded content into output
else Not resolved
Expander->>Expander: Keep original COPY line
end
end
Expander-->>Pipeline: Expanded content + resolution metadata
Pipeline->>Pipeline: Replace file content with expanded content
end
```
The return type `CopyExpansionResult` contains `expandedContent` and `copyResolutions`. The `expansionDepth` field has been removed from the return type (it was unused by callers).
COPY statement line numbers in `CopyResolution` are 1-based (consistent with the preprocessor's line numbering). The splice operation that replaces COPY lines with expanded content adjusts for 0-based array indexing internally.
## Cycle Detection
Circular COPY references (e.g., copybook A includes copybook B which includes copybook A) are detected and handled:
1. Each expansion chain maintains a `visited` set of resolved copybook paths
2. If a copybook path is already in the visited set, the expansion is skipped
3. A `warnedCircular` set (internal to `expandCopies()`, not a parameter) deduplicates warning messages within a single file expansion
Known circular copybooks in PROJECT-NAME: `ANAZI`, `ANDIP`, `QDIPE` (self-referential includes).
## Max Depth
Nested COPY expansion is limited to **10 levels** (`DEFAULT_MAX_DEPTH`). If a COPY chain exceeds this depth, a warning is logged and the remaining COPY statements are left unexpanded.
## Max Total Expansions
A breadth amplification guard caps the total number of COPY expansions across all branches within a single file to **500** (`MAX_TOTAL_EXPANSIONS`). This prevents exponential blowup from diamond-shaped COPY graphs where N copybooks each include N other copybooks. Once the limit is reached, further COPY statements in that file are left unexpanded and a single warning is logged.
## REPLACING Application Detail
The REPLACING engine works by scanning all COBOL identifiers (matching `\b[A-Z][A-Z0-9-]*\b`) in the copybook content and applying each replacement rule:
```
Original copybook content:
05 ESP-NAME PIC X(30).
05 ESP-CODE PIC X(10).
05 KPSESPL-FLAG PIC X(01).
After REPLACING LEADING "ESP-" BY "LK-ESP-" LEADING "KPSESPL" BY "LK-KPSESPL":
05 LK-ESP-NAME PIC X(30).
05 LK-ESP-CODE PIC X(10).
05 LK-KPSESPL-FLAG PIC X(01).
```
For LEADING replacements, the engine checks if each identifier starts with the `from` prefix (case-insensitive) and replaces only the prefix portion, preserving the rest of the identifier.
For TRAILING replacements, the same logic applies to suffixes.
For EXACT replacements, only identifiers that match the `from` value exactly (case-insensitive) are replaced.
## Copybook Resolution
The resolver tries multiple strategies to match a COPY target name to a copybook file:
1. **Exact match**: `COPY CPSESP` resolves to copybook named `CPSESP`
2. **Strip extension**: `COPY WORKGRID.CPY` strips `.CPY` and resolves to `WORKGRID`
3. **Add extension**: `COPY CPSESP` tries `CPSESP.CPY` and `CPSESP.COPY`
If no match is found, the COPY statement is left in place (unexpanded) and a resolution record with `resolvedPath: null` is created.
## Pipeline Integration
The expansion runs **per chunk**, after file content is read but before dispatch to worker threads:
1. All copybook files are read upfront (they are typically small, collectively under 100MB)
2. Per chunk, the copybook map is merged with chunk content (in case a chunk contains copybooks)
3. Only programs (not copybooks themselves) undergo expansion
4. The expanded content replaces the original content in-place before worker dispatch
## Inline Comment Handling
The copy expander's `stripInlineComment()` helper is quote-aware: pipe characters (`|`) inside single- or double-quoted strings are preserved. This matches the same quote-aware logic used by the preprocessor.
## Source Files
- `gitnexus/src/core/ingestion/cobol-copy-expander.ts` -- `expandCopies()`, `parseReplacingClause()`, `applyReplacing()`
- `gitnexus/src/core/ingestion/pipeline.ts` -- `expandCobolCopies()`, copybook map construction, chunk integration
+312
View File
@@ -0,0 +1,312 @@
# COBOL Deep Indexing
Beyond basic symbol extraction (program name, paragraphs, CALL, PERFORM, COPY), GitNexus performs deep indexing of COBOL-specific constructs: data items, EXEC SQL/CICS blocks, file declarations, FD entries, ENTRY points, and MOVE statements.
## Data Items
### Level Numbers
| Level Range | Meaning | Graph Node Type |
|-------------|---------|-----------------|
| 01 | Record (group item) | `Record` |
| 02-49 | Elementary/group items | `Property` |
| 66 | RENAMES | `Property` |
| 77 | Independent item | `Property` |
| 88 | Condition name | `Const` |
FILLER items are skipped (no useful name for the graph).
### Clauses Parsed
The `parseDataItemClauses()` function extracts these clauses from the trailing text of a data item declaration:
| Clause | Pattern | Example |
|--------|---------|---------|
| `PIC` / `PICTURE` | `\bPIC(?:TURE)?\s+(?:IS\s+)?(\S+)` | `PIC X(30)`, `PICTURE IS 9(5)V99` |
| `USAGE` | `\bUSAGE\s+(?:IS\s+)?(COMP\|BINARY\|...)` | `USAGE IS COMP-3`, `BINARY` |
| `REDEFINES` | `\bREDEFINES\s+([A-Z][A-Z0-9-]+)` | `REDEFINES WK-DATE-NUM` |
| `OCCURS` | `\bOCCURS\s+(\d+)` | `OCCURS 12 TIMES` |
Standalone COMP variants (without the `USAGE` keyword) are also detected: `COMP`, `COMP-1` through `COMP-6`, `COMP-X`, `BINARY`, `PACKED-DECIMAL`.
### Data Hierarchy
Data items form a hierarchical structure based on level numbers. The extractor uses a **stack algorithm**:
```
Processing order:
01 WK-RECORD -> push {01, WK-RECORD} -> parent: Module
05 WK-NAME -> push {05, WK-NAME} -> parent: WK-RECORD (01 < 05)
10 WK-FIRST -> push {10, WK-FIRST} -> parent: WK-NAME (05 < 10)
10 WK-LAST -> pop WK-FIRST, push -> parent: WK-NAME (05 < 10)
05 WK-CODE -> pop WK-LAST, WK-NAME -> parent: WK-RECORD (01 < 05)
88 WK-ACTIVE -> (88 handled separately) -> parent: WK-CODE
```
The stack maintains items where each entry's level is strictly less than the next. When a new item arrives with a level <= the top of stack, items are popped until the stack top has a smaller level. A `CONTAINS` edge is created from the stack top to the new item.
For 88-level condition names, the parent is the immediately preceding non-88 data item (found by scanning backwards).
### Annotated Example
```cobol
01 WK-EMPLOYEE.
05 WK-EMP-ID PIC 9(6).
05 WK-EMP-NAME PIC X(30).
05 WK-EMP-STATUS PIC X(01).
88 WK-ACTIVE VALUE "A".
88 WK-INACTIVE VALUE "I".
05 WK-SALARY PIC 9(7)V99 COMP-3.
05 WK-DEPT PIC X(04) OCCURS 3 TIMES.
```
Produces:
- `Record` node: `WK-EMPLOYEE` (level 01, section: working-storage)
- `Property` nodes: `WK-EMP-ID`, `WK-EMP-NAME`, `WK-EMP-STATUS`, `WK-SALARY`, `WK-DEPT`
- `Const` nodes: `WK-ACTIVE` (values: `A`), `WK-INACTIVE` (values: `I`)
- `CONTAINS` edges: `WK-EMPLOYEE -> WK-EMP-ID`, `WK-EMPLOYEE -> WK-EMP-NAME`, etc.
- `CONTAINS` edges: `WK-EMP-STATUS -> WK-ACTIVE`, `WK-EMP-STATUS -> WK-INACTIVE`
### Data Item Cap
A maximum of **500 data items per file** (`MAX_DATA_ITEMS_PER_FILE`) are processed. Some COBOL programs (especially after COPY expansion) can have 10,000+ data items, which would cause graph bloat and push the V8 relationship Map past its 16.7M entry limit across thousands of files.
The cap applies after extraction: the first 500 items in source order are kept. Since 01-level records appear first, critical top-level structure is preserved.
## EXEC SQL
EXEC SQL blocks are accumulated across lines between `EXEC SQL` and `END-EXEC`, then parsed as a unit.
### Operation Classification
The first SQL keyword determines the operation:
| First Keyword | Operation |
|---------------|-----------|
| `SELECT` | SELECT |
| `INSERT` | INSERT |
| `UPDATE` | UPDATE |
| `DELETE` | DELETE |
| `DECLARE` | DECLARE |
| `OPEN` | OPEN |
| `CLOSE` | CLOSE |
| `FETCH` | FETCH |
| *(anything else)* | OTHER |
### Table Extraction
Tables are extracted from SQL clauses:
| Clause Pattern | Example |
|----------------|---------|
| `FROM <table>` | `SELECT * FROM EMPLOYEES` |
| `INSERT INTO <table>` | `INSERT INTO EMPLOYEES` |
| `UPDATE <table>` | `UPDATE EMPLOYEES SET ...` |
| `JOIN <table>` | `LEFT JOIN DEPARTMENTS ON ...` |
Note: The `INTO` pattern is restricted to `INSERT INTO` to avoid false positives from `FETCH ... INTO :host-var` and `SELECT ... INTO :host-var` statements, where `INTO` introduces host variables rather than table names.
### Cursor Detection
```cobol
EXEC SQL
DECLARE C-EMPLOYEES CURSOR FOR
SELECT EMP-ID, EMP-NAME FROM EMPLOYEES
WHERE DEPT = :WK-DEPT
END-EXEC
```
Extracts: cursor `C-EMPLOYEES`, table `EMPLOYEES`, host variable `WK-DEPT`.
### Host Variables
Host variables are COBOL variables referenced in SQL with a `:` prefix. The colon is stripped:
```sql
WHERE EMP-ID = :WK-EMP-ID AND DEPT = :WK-DEPT
```
Extracts: `WK-EMP-ID`, `WK-DEPT`.
### Graph Output
- `CodeElement` node per table, with description `sql-table op:{OP}`
- `CodeElement` node per cursor, with description `sql-cursor`
- `ACCESSES` edge from Module to each CodeElement
- Deduplication: if the same table appears in multiple SQL blocks, only one node is created
## EXEC CICS
EXEC CICS blocks are accumulated and parsed similarly to SQL blocks.
### Command Detection
Two-word commands are detected first (matched against the block start):
```
SEND MAP, RECEIVE MAP, SEND TEXT, SEND CONTROL, READ NEXT, READ PREV
```
If no two-word command matches, the first word is used (e.g., `LINK`, `XCTL`, `RETURN`, `READ`, `WRITE`).
### Extraction
| Element | Pattern | Example |
|---------|---------|---------|
| MAP name | `MAP('name')` or `MAP("name")` | `EXEC CICS SEND MAP('EMPMENU')` |
| PROGRAM name | `PROGRAM('name')` or `PROGRAM("name")` | `EXEC CICS LINK PROGRAM('BGTABUP')` |
| TRANSID | `TRANSID('name')` or `TRANSID("name")` | `EXEC CICS START TRANSID('EMP1')` |
### Graph Output
- MAP: `CodeElement` node with description `cics-map cmd:{CMD}` + `ACCESSES` edge from Module
- PROGRAM: `CALLS` edge (cross-program call via CICS LINK/XCTL)
- TRANSID: `CodeElement` node with description `cics-transid cmd:{CMD}` + `ACCESSES` edge from Module
### Annotated Example
```cobol
EXEC CICS
SEND MAP('EMPMENU')
MAPSET('EMPSET')
FROM(WK-MAP-DATA)
ERASE
END-EXEC
```
Produces:
- `CodeElement` node: `EMPMENU` (description: `cics-map cmd:SEND MAP`)
- `ACCESSES` edge: Module -> `EMPMENU`
## File Declarations
SELECT statements in the INPUT-OUTPUT SECTION are accumulated across multiple lines (until a period terminator) and parsed for:
| Clause | Pattern | Example |
|--------|---------|---------|
| SELECT | `SELECT <name>` | `SELECT MASTER-FILE` |
| ASSIGN | `ASSIGN TO <file>` | `ASSIGN TO "MASTER.DAT"` |
| ORGANIZATION | `ORGANIZATION IS <type>` | `ORGANIZATION IS INDEXED` |
| ACCESS | `ACCESS MODE IS <mode>` | `ACCESS MODE IS DYNAMIC` |
| RECORD KEY | `RECORD KEY IS <field>` | `RECORD KEY IS WK-EMP-ID` |
| FILE STATUS | `FILE STATUS IS <field>` | `FILE STATUS IS WK-FILE-STATUS` |
### Graph Output
- `CodeElement` node with description containing all parsed clauses (e.g., `select org:INDEXED access:DYNAMIC key:WK-EMP-ID status:WK-FILE-STATUS assign:MASTER.DAT`)
- `RECORD_KEY_OF` edge: from Property node to CodeElement (confidence 0.8)
- `FILE_STATUS_OF` edge: from Property node to CodeElement (confidence 0.8)
## FD Entries
FD (File Description) entries associate a file name with its record layout:
```cobol
FD MASTER-FILE.
01 MASTER-RECORD.
05 MR-EMP-ID PIC 9(6).
05 MR-EMP-NAME PIC X(30).
```
The extractor tracks `pendingFdName` state: when an `FD` line is seen, the next 01-level data item becomes its record.
### Graph Output
- `CodeElement` node with description `fd record:{recordName}`
- `CONTAINS` edge: FD CodeElement -> Record node
- `CONTAINS` edge: SELECT CodeElement -> FD CodeElement (linking file declaration to file description)
## ENTRY Points
The `ENTRY` statement defines additional entry points into a COBOL program (in addition to the main program entry):
```cobol
ENTRY "SUBPROG" USING WK-PARAM-1 WK-PARAM-2.
```
### Graph Output
- `Constructor` node with description `entry params:{param1},{param2}` (or just `entry` if no parameters)
- `CONTAINS` edge: Module -> Constructor
- Symbol table entry (so the entry point is discoverable by name)
## PROCEDURE DIVISION USING
```cobol
PROCEDURE DIVISION USING WK-INPUT-REC WK-OUTPUT-REC.
```
The USING clause identifies parameters received by the program from its caller.
### Graph Output
- `RECEIVES` edge: Module -> Property (for each parameter name, confidence 0.8)
## MOVE Statements
MOVE statements produce `ACCESSES` edges in the graph:
```cobol
MOVE WK-NAME TO OUT-NAME.
MOVE CORRESPONDING WK-INPUT TO WK-OUTPUT.
MOVE CORR WK-IN TO WK-OUT.
```
### Extraction Details
- Source and target identifiers are captured
- `CORRESPONDING` and its abbreviation `CORR` are both recognized (bulk field-by-field move)
- Figurative constants (SPACES, ZEROS, LOW-VALUES, HIGH-VALUES, QUOTES, ALL) are skipped
- The enclosing paragraph (`caller`) is tracked for context
### MOVE CORRESPONDING / CORR Edge Reasons
MOVE CORRESPONDING (and CORR) produces distinct edge reasons to differentiate from simple MOVE:
| Edge | Reason (simple MOVE) | Reason (CORRESPONDING/CORR) |
|------|---------------------|-----------------------------|
| Read (source) | `cobol-move-read` | `cobol-move-corresponding-read` |
| Write (target) | `cobol-move-write` | `cobol-move-corresponding-write` |
This distinction allows queries to find bulk field-by-field moves separately from simple variable assignments.
## GO TO DEPENDING ON
The `GO TO` statement with multiple targets and a `DEPENDING ON` clause is a computed branch:
```cobol
GO TO PARA-1 PARA-2 PARA-3
DEPENDING ON WK-SELECTOR.
```
All target paragraph names are extracted and emitted as separate `gotos` entries. Each target produces a `CALLS` edge in the graph (same semantics as PERFORM). The `DEPENDING ON` variable is not currently tracked as a data-flow dependency.
## SORT INPUT/OUTPUT PROCEDURE
SORT and MERGE statements can specify procedural entry points instead of file-based I/O:
```cobol
SORT SORT-FILE ON ASCENDING KEY SORT-KEY
INPUT PROCEDURE IS PREPARE-INPUT
OUTPUT PROCEDURE IS FORMAT-OUTPUT.
```
`INPUT PROCEDURE IS` and `OUTPUT PROCEDURE IS` targets are extracted as control-flow targets (same as PERFORM). They produce `performs` entries and corresponding `CALLS` edges in the graph.
## Fixed-Format Literal Continuation
In fixed-format COBOL, string literals can span multiple lines using the continuation indicator (`-` in column 7). When a continuation line starts with a quote character, the extractor joins it with the predecessor by removing the trailing quote from the previous line and the opening quote from the continuation:
```
Line N: MOVE "THIS IS A LONG STRI
Line N+1 (cont): - "NG VALUE" TO WK-FIELD.
Merged: MOVE "THIS IS A LONG STRING VALUE" TO WK-FIELD.
```
The trailing `"` on line N and the opening `"` on line N+1 are both removed, producing a seamless literal. If no matching quote is found on the predecessor line, the continuation is appended as-is.
## Source Files
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- All extraction logic, clause parsers, EXEC block parsers
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `processCobolRegexOnly()`, graph node/edge emission
- `gitnexus/src/core/ingestion/parsing-processor.ts` -- Sequential fallback with same `MAX_DATA_ITEMS_PER_FILE` cap
+126
View File
@@ -0,0 +1,126 @@
# COBOL File Detection
GitNexus detects COBOL files through two mechanisms: extension-based mapping and directory-based override for extensionless files. This document covers both, plus the copybook/program classification logic.
## Extension Mapping
### Program Extensions
| Extension | Type |
|-----------|------|
| `.cbl` | COBOL program |
| `.cob` | COBOL program |
| `.cobol` | COBOL program |
### Copybook Extensions
| Extension | Type | Notes |
|-----------|------|-------|
| `.cpy` | Copybook | Standard |
| `.copy` | Copybook | Standard |
| `.gnm` / `.GNM` | Copybook | Enterprise (GnuCOBOL naming) |
| `.fd` / `.FD` | Copybook | File Description fragment |
| `.wrk` / `.WRK` | Copybook | Working-Storage fragment |
| `.sel` / `.SEL` | Copybook | SELECT clause fragment |
| `.open` / `.OPEN` | Copybook | File OPEN fragment |
| `.close` / `.CLOSE` | Copybook | File CLOSE fragment |
| `.ini` / `.INI` | Copybook | Initialization fragment |
| `.def` / `.DEF` | Copybook | Definition fragment |
All extension matching is case-sensitive in `getLanguageFromFilename` (the extensions above are matched as written, including uppercase variants like `.GNM`).
## Extensionless File Detection: `GITNEXUS_COBOL_DIRS`
Many enterprise COBOL repositories use extensionless files -- the filename alone identifies the program (e.g., `s/BGTABFL` is the source for program `BGTABFL`). GitNexus handles this via the `GITNEXUS_COBOL_DIRS` environment variable.
### Configuration
Set `GITNEXUS_COBOL_DIRS` to a comma-separated list of directory names:
```bash
# Files in s/, c/, and wfproc/ directories (at any depth) are treated as COBOL
export GITNEXUS_COBOL_DIRS=s,c,wfproc
```
The matching is **case-insensitive** and checks all path segments:
- `/repo/s/BGTABFL` -- matches segment `s` -- COBOL
- `/repo/src/c/CPSESP` -- matches segment `c` -- COBOL
- `/repo/wfproc/WF001` -- matches segment `wfproc` -- COBOL
- `/repo/docs/README` -- no matching segment -- skipped
### Decision Tree
```mermaid
flowchart TD
A[getLanguageFromPath] --> B[getLanguageFromFilename]
B --> C{Known extension?}
C -->|Yes .cbl/.cob/.cobol/.cpy/...| D[Return COBOL]
C -->|Yes .ts/.py/.java/...| E[Return other language]
C -->|No match| F{Has extension?}
F -->|"Has dot in basename"| G[Return null]
F -->|"No dot = extensionless"| H{GITNEXUS_COBOL_DIRS set?}
H -->|No| G
H -->|Yes| I{Any path segment<br/>matches a configured dir?}
I -->|Yes| D
I -->|No| G
style D fill:#e8f5e9,stroke:#2e7d32
style G fill:#ffebee,stroke:#c62828
```
### Implementation Detail
The `GITNEXUS_COBOL_DIRS` value is parsed once (on first call) and cached in a `Set<string>`:
```typescript
// From gitnexus/src/core/ingestion/utils.ts
const getCobolDirs = (): Set<string> => {
if (_cobolDirs) return _cobolDirs;
const raw = process.env.GITNEXUS_COBOL_DIRS;
_cobolDirs = raw
? new Set(raw.split(',').map(d => d.trim().toLowerCase()))
: new Set();
return _cobolDirs;
};
```
The path segment check splits the full path on `/` and tests each segment against the cached set.
## Copybook vs Program Classification
After a file is identified as COBOL, it must be classified as either a **program** (to be parsed for symbols) or a **copybook** (to be loaded into the copybook map for COPY expansion).
### Classification Rules
A COBOL file is classified as a **copybook** if ANY of these conditions is true:
1. It has a recognized copybook extension (`.cpy`, `.copy`, `.gnm`, `.fd`, `.wrk`, `.sel`, `.open`, `.close`, `.ini`, `.def`)
2. It is an extensionless file whose path contains a directory segment matching one of: `c`, `copy`, `copybooks`, `copylib`, `cpy`
A file is classified as a **program** if:
1. It has a program extension (`.cbl`, `.cob`, `.cobol`), OR
2. It is extensionless and does NOT match any copybook directory pattern
### Copybook Name Resolution
Copybook names are derived from the filename:
- Strip the extension (if any)
- Convert to uppercase
Examples:
- `c/CPSESP` -- name: `CPSESP`
- `copy/workgrid.cpy` -- name: `WORKGRID`
- `c/ANAZI.GNM` -- name: `ANAZI`
This name is used to resolve `COPY CPSESP.` statements during expansion.
## Source Files
- `gitnexus/src/core/ingestion/utils.ts` -- `getLanguageFromPath()`, `getLanguageFromFilename()`, `getCobolDirs()`
- `gitnexus/src/core/ingestion/pipeline.ts` -- `isCobolCopybook()`, `getCopybookName()`, `COPYBOOK_EXTENSIONS`, `COBOL_PROGRAM_EXTENSIONS`
+193
View File
@@ -0,0 +1,193 @@
# COBOL Graph Model
This document describes the graph nodes and edges that GitNexus creates for COBOL codebases. The COBOL graph model is richer than most tree-sitter languages because it captures domain-specific constructs: file declarations, FD entries, data hierarchies, SQL tables, CICS maps, and cross-program contracts.
## Entity-Relationship Diagram
```mermaid
erDiagram
File ||--o{ Module : DEFINES
File ||--o{ Function : DEFINES
File ||--o{ Namespace : DEFINES
File ||--o{ Record : DEFINES
File ||--o{ Property : DEFINES
File ||--o{ Const : DEFINES
File ||--o{ CodeElement : DEFINES
File ||--o{ Constructor : DEFINES
File }o--o{ File : IMPORTS
Module ||--o{ Record : CONTAINS
Module ||--o{ Constructor : CONTAINS
Module }o--o{ CodeElement : ACCESSES
Module }o--o{ Module : CALLS
Module }o--o{ Module : CONTRACTS
Module }o--o{ Property : RECEIVES
Record ||--o{ Property : CONTAINS
Record ||--o{ Const : CONTAINS
Record }o--o{ Record : REDEFINES
Property ||--o{ Property : CONTAINS
Property ||--o{ Const : CONTAINS
Property }o--o{ Property : REDEFINES
Property }o--o{ CodeElement : RECORD_KEY_OF
Property }o--o{ CodeElement : FILE_STATUS_OF
CodeElement ||--o{ CodeElement : CONTAINS
CodeElement ||--o{ Record : CONTAINS
Function }o--o{ Function : CALLS
```
## Node Types
| Node Type | COBOL Concept | Created From | Example |
|-----------|--------------|--------------|---------|
| `Module` | PROGRAM-ID | `PROGRAM-ID. BGTABFL` | Name: `BGTABFL`, description may include author and date |
| `Function` | Paragraph | `PROCESS-RECORD.` at column 8 | Name: `PROCESS-RECORD` |
| `Namespace` | Procedure section | `MAIN-LOGIC SECTION.` at column 8 | Name: `MAIN-LOGIC` |
| `Record` | 01-level data item | `01 WK-EMPLOYEE.` | Description: `level:01 section:working-storage` |
| `Property` | 02-49/66/77 data item | `05 WK-NAME PIC X(30).` | Description: `level:05 pic:X(30) section:working-storage` |
| `Const` | 88-level condition | `88 WK-ACTIVE VALUE "A".` | Description: `level:88 values:A` |
| `CodeElement` | SELECT, FD, SQL table, CICS map, cursor, transid | Various | Description varies by subtype |
| `Constructor` | ENTRY point | `ENTRY "SUBPROG" USING WK-DATA` | Description: `entry params:WK-DATA` |
### CodeElement Subtypes
CodeElement is used for multiple COBOL constructs, distinguished by their description prefix:
| Subtype | ID Pattern | Description Format | Example |
|---------|-----------|-------------------|---------|
| File SELECT | `CodeElement:{path}:SELECT:{name}` | `select org:INDEXED access:DYNAMIC ...` | `SELECT MASTER-FILE` |
| FD entry | `CodeElement:{path}:FD:{name}` | `fd record:{recordName}` | `FD MASTER-FILE` |
| SQL table | `CodeElement:{path}:sql-table:{name}` | `sql-table op:SELECT` | Table `EMPLOYEES` |
| SQL cursor | `CodeElement:{path}:sql-cursor:{name}` | `sql-cursor` | Cursor `C-EMPLOYEES` |
| CICS map | `CodeElement:{path}:cics-map:{name}` | `cics-map cmd:SEND MAP` | Map `EMPMENU` |
| CICS transid | `CodeElement:{path}:cics-transid:{name}` | `cics-transid cmd:START` | Transid `EMP1` |
## Edge Types
| Edge Type | Source | Target | Created By | Confidence | Example |
|-----------|--------|--------|-----------|------------|---------|
| `DEFINES` | File | any node | File defines its symbols | 1.0 | File -> Module `BGTABFL` |
| `CALLS` | Function | Function | `PERFORM X [THRU Y]` | (via call-processor) | `PROCESS-RECORD` -> `CALC-TAX` |
| `CALLS` | Module | Module | `CALL "BGTABUP"` | (via call-processor) | `BGTABFL` -> `BGTABUP` |
| `CALLS` | Module | Module | `EXEC CICS LINK PROGRAM('X')` | (via call-processor) | `BGTABFL` -> `BGTABUP` |
| `IMPORTS` | File | File | `COPY copybook` | (via import-processor) | Source file -> Copybook file |
| `CONTAINS` | Module | Record | Data hierarchy root | 1.0 | `BGTABFL` -> `WK-EMPLOYEE` |
| `CONTAINS` | Record | Property | Data hierarchy | 1.0 | `WK-EMPLOYEE` -> `WK-NAME` |
| `CONTAINS` | Property | Property | Nested data items | 1.0 | `WK-ADDRESS` -> `WK-CITY` |
| `CONTAINS` | Record/Property | Const | 88-level parent | 1.0 | `WK-STATUS` -> `WK-ACTIVE` |
| `CONTAINS` | CodeElement (FD) | Record | FD record link | 1.0 | `FD:MASTER-FILE` -> `MASTER-RECORD` |
| `CONTAINS` | CodeElement (SELECT) | CodeElement (FD) | SELECT-FD link | 0.9 | `SELECT:MASTER-FILE` -> `FD:MASTER-FILE` |
| `CONTAINS` | Module | Constructor | ENTRY in module | 1.0 | `BGTABFL` -> `SUBPROG` |
| `REDEFINES` | Record | Record | `01 X REDEFINES Y` | 1.0 | `WK-DATE-NUM` -> `WK-DATE-ALPHA` |
| `REDEFINES` | Property | Property | `05 X REDEFINES Y` | 1.0 | `WK-CODE-NUM` -> `WK-CODE-ALPHA` |
| `RECORD_KEY_OF` | Property | CodeElement (SELECT) | `RECORD KEY IS field` | 0.8 | `WK-EMP-ID` -> `SELECT:MASTER-FILE` |
| `FILE_STATUS_OF` | Property | CodeElement (SELECT) | `FILE STATUS IS field` | 0.8 | `WK-FS` -> `SELECT:MASTER-FILE` |
| `ACCESSES` | Module | CodeElement | EXEC SQL/CICS | 0.9 | `BGTABFL` -> `sql-table:EMPLOYEES` |
| `RECEIVES` | Module | Property | `PROCEDURE USING` | 0.8 | `BGTABFL` -> `WK-INPUT-REC` |
| `CONTRACTS` | Module | Module | Shared copybook detection | 0.9 | `BGTABFL` -> `BGTABUP` (via `CPSESP`) |
## Full Annotated Example
Given this COBOL program:
```cobol
IDENTIFICATION DIVISION.
PROGRAM-ID. EMPMAINT.
AUTHOR. Development Team.
ENVIRONMENT DIVISION.
INPUT-OUTPUT SECTION.
FILE-CONTROL.
SELECT EMP-FILE
ASSIGN TO "EMPLOYEE.DAT"
ORGANIZATION IS INDEXED
ACCESS MODE IS DYNAMIC
RECORD KEY IS EMP-ID
FILE STATUS IS WS-FILE-STATUS.
DATA DIVISION.
FILE SECTION.
FD EMP-FILE.
01 EMP-RECORD.
05 EMP-ID PIC 9(6).
05 EMP-NAME PIC X(30).
WORKING-STORAGE SECTION.
01 WS-FLAGS.
05 WS-FILE-STATUS PIC X(02).
05 WS-EOF-FLAG PIC X(01).
88 WS-EOF VALUE "Y".
LINKAGE SECTION.
01 LK-SEARCH-KEY PIC 9(6).
PROCEDURE DIVISION USING LK-SEARCH-KEY.
MAIN-LOGIC SECTION.
MAIN-START.
PERFORM OPEN-FILE
PERFORM PROCESS-RECORDS
PERFORM CLOSE-FILE
STOP RUN.
OPEN-FILE.
OPEN I-O EMP-FILE.
PROCESS-RECORDS.
MOVE LK-SEARCH-KEY TO EMP-ID
EXEC SQL
SELECT EMP_SALARY INTO :WS-SALARY
FROM EMPLOYEES
WHERE EMP_ID = :EMP-ID
END-EXEC
CALL "EMPREPORT".
CLOSE-FILE.
CLOSE EMP-FILE.
```
The graph produced contains:
**Nodes:**
- `Module`: EMPMAINT (description: `author:Development Team`)
- `Namespace`: MAIN-LOGIC
- `Function`: MAIN-START, OPEN-FILE, PROCESS-RECORDS, CLOSE-FILE
- `Record`: EMP-RECORD, WS-FLAGS, LK-SEARCH-KEY
- `Property`: EMP-ID, EMP-NAME, WS-FILE-STATUS, WS-EOF-FLAG
- `Const`: WS-EOF (values: Y)
- `CodeElement`: SELECT:EMP-FILE, FD:EMP-FILE, sql-table:EMPLOYEES
- (COPY imports, if any, would produce File IMPORTS edges)
**Edges:**
- `DEFINES`: File -> all nodes
- `CONTAINS`: EMPMAINT -> EMP-RECORD, EMPMAINT -> WS-FLAGS, EMPMAINT -> LK-SEARCH-KEY
- `CONTAINS`: EMP-RECORD -> EMP-ID, EMP-RECORD -> EMP-NAME
- `CONTAINS`: WS-FLAGS -> WS-FILE-STATUS, WS-FLAGS -> WS-EOF-FLAG
- `CONTAINS`: WS-EOF-FLAG -> WS-EOF
- `CONTAINS`: FD:EMP-FILE -> EMP-RECORD
- `CONTAINS`: SELECT:EMP-FILE -> FD:EMP-FILE
- `CALLS`: MAIN-START -> OPEN-FILE, MAIN-START -> PROCESS-RECORDS, MAIN-START -> CLOSE-FILE
- `CALLS`: EMPMAINT -> EMPREPORT (external CALL)
- `ACCESSES`: EMPMAINT -> sql-table:EMPLOYEES
- `RECEIVES`: EMPMAINT -> LK-SEARCH-KEY (PROCEDURE USING)
- `RECORD_KEY_OF`: EMP-ID -> SELECT:EMP-FILE
- `FILE_STATUS_OF`: WS-FILE-STATUS -> SELECT:EMP-FILE
## How COBOL Differs from Tree-Sitter Languages
| Aspect | COBOL | Tree-Sitter Languages |
|--------|-------|----------------------|
| Node variety | 8 types (Module, Function, Namespace, Record, Property, Const, CodeElement, Constructor) | Typically 4-6 (Function, Class, Method, Interface, Module, Const) |
| Domain edges | RECORD_KEY_OF, FILE_STATUS_OF, ACCESSES, RECEIVES, CONTRACTS, REDEFINES | Primarily CALLS, IMPORTS, EXTENDS, IMPLEMENTS |
| Data hierarchy | Deep CONTAINS chains (01 -> 05 -> 10 -> 88) | Flat class members |
| Cross-program calls | CALL "name" + CICS LINK PROGRAM | Import-based resolution |
| Contract detection | Shared COPY copybook between caller/callee | Not applicable |
| Metadata | AUTHOR, DATE-WRITTEN on Module | JSDoc/docstring (not indexed) |
## Source Files
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `processCobolRegexOnly()`, node/edge emission logic
- `gitnexus/src/core/ingestion/pipeline.ts` -- `detectCrossProgamContracts()` for CONTRACTS edges
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- `CobolRegexResults` interface (all extracted data)
+261
View File
@@ -0,0 +1,261 @@
# COBOL Performance and Tuning
This document covers real-world benchmarks, worker pool configuration, memory management, known limitations, and troubleshooting for COBOL indexing.
## PROJECT-NAME Benchmark
The PROJECT-NAME project is a large Italian payroll system written in COBOL. It serves as the primary benchmark for COBOL indexing performance.
### Input
| Metric | Value |
| --------------------------- | ---------------------------------------------------------------------------- |
| Paths scanned | 14,217 |
| Parseable files | 13,129 |
| Total source size | 224 MB |
| Chunks | 12 (at 20 MB budget) |
| Copybooks loaded | 2,976 |
| Copybooks used in expansion | 2,955 |
| Key directories | `s/` (7773 programs), `c/` (3036 copybooks), `wfproc/` (1973 workflow files) |
### Output
| Metric | Value |
| ---------------------- | ------ |
| Graph nodes | 2.79M |
| Graph edges | 5.67M |
| Clusters (communities) | 16,679 |
| Execution flows | 300 |
### Timing
| Phase | Duration |
| ------------------------------- | ----------------- |
| Total | ~251s |
| KuzuDB write | 132s |
| Full-text search indexing | 6.7s |
| Regex extraction (avg per file) | ~1ms |
| COPY expansion + deep indexing | Remainder (~112s) |
### Indexing Command
```bash
cd /path/to/PROJECT-NAME
GITNEXUS_COBOL_DIRS=s,c,wfproc GITNEXUS_VERBOSE=1 node --max-old-space-size=8192 \
/path/to/gitnexus/dist/cli/index.js analyze --force
```
## Open-Source Benchmarks
### CardDemo (AWS)
| Metric | Value |
| ------ | ----- |
| Graph nodes | 12,323 |
| Graph edges | 8,893 |
| Total time | 7.4s |
### ACAS
| Metric | Value |
| ------ | ----- |
| Graph nodes | 14,016 |
| Graph edges | 15,452 |
| Total time | 9.3s |
### Micro-Benchmark (Single-File Extraction)
| Metric | Value |
| ------ | ----- |
| Per-iteration | 0.65ms |
| Throughput | ~382K lines/sec |
## Worker Pool Tuning
### Sub-Batch Size
The worker pool splits each worker's chunk into sub-batches to bound peak memory per `postMessage` serialization. COBOL repos use a smaller sub-batch size than the default:
| Parameter | Default | COBOL Mode |
| --------------------- | ----------- | ------------------- |
| Sub-batch size | 1,500 files | 200 files |
| Per sub-batch timeout | 120s | 120s (configurable) |
**Why 200?** COBOL regex extraction + preprocessing takes ~1ms per file on average, but with COPY expansion and deep indexing the effective time is ~150ms per file. At sub-batch size 1500, that would be ~225s per sub-batch, exceeding the 120s timeout.
COBOL mode is activated automatically when `GITNEXUS_COBOL_DIRS` is set:
```typescript
// From pipeline.ts
const cobolSubBatch = process.env.GITNEXUS_COBOL_DIRS ? 200 : undefined;
workerPool = createWorkerPool(workerUrl, undefined, cobolSubBatch);
```
### Worker Count
Workers default to `min(8, cpus - 1)`. For COBOL repos, this is usually sufficient since regex extraction is CPU-bound but fast. The bottleneck is typically KuzuDB write, not extraction.
### Timeout Configuration
| Environment Variable | Default | Purpose |
| ------------------------------------ | --------------- | --------------------------------------------------- |
| `GITNEXUS_WORKER_TIMEOUT_MS` | 120,000 (2 min) | Per sub-batch processing timeout |
| `GITNEXUS_WORKER_STARTUP_TIMEOUT_MS` | 60,000 (1 min) | Worker initialization timeout (tree-sitter loading) |
For COBOL-only repos, worker startup is faster because tree-sitter native modules are loaded lazily (skipped entirely if only COBOL files are present).
## Data Item Cap
### Configuration
```typescript
const MAX_DATA_ITEMS_PER_FILE = 500;
```
This constant appears in both `parse-worker.ts` (worker path) and `parsing-processor.ts` (sequential fallback).
### Rationale
Some COBOL programs, especially after COPY expansion, can have 10,000+ data items. At that scale:
- The in-memory relationship Map (for CONTAINS, REDEFINES, etc.) approaches the V8 16.7M entry limit across thousands of files
- KuzuDB write time increases linearly with edge count
- Most deep-nested items (level 20+) are rarely queried individually
### Impact
The cap truncates data items beyond the 500th in source order. Since 01-level Records appear first in COBOL source, the cap preserves:
- All 01-level record definitions
- The most important 02-49 level items (those closest to the record root)
- 88-level conditions associated with early items
To increase the cap for specific needs, modify the `MAX_DATA_ITEMS_PER_FILE` constant in both files.
## Memory Management
### COPY Expansion Breadth Guard
A per-file `MAX_TOTAL_EXPANSIONS = 500` limit prevents exponential blowup from diamond-shaped COPY graphs (e.g., N copybooks each containing N COPY statements). Once the limit is reached, further COPY statements in that file are left unexpanded. See [copy-expansion.md](copy-expansion.md) for details.
### COPY Expansion Memory
All copybook content is loaded upfront into a Map before chunk processing begins. For PROJECT-NAME:
- 2,976 copybooks, typically under 100MB total
- The Map is shared (read-only) across chunk iterations
- Per-chunk, the copybook map is merged with chunk file content (in case a chunk contains copybooks not in the pre-loaded set)
- After all chunks are processed, the copybook map is freed (`cobolCopybookContents = undefined`)
### Chunk Budget
Source files are grouped into chunks of max 20MB (`CHUNK_BYTE_BUDGET`). Each chunk's lifecycle:
1. Read file content into memory
2. Expand COPY statements (mutates content in-place)
3. Dispatch to workers for extraction
4. Workers return serialized results
5. Merge results into graph
6. Chunk content goes out of scope (GC reclaims)
This ensures only ~20MB of source + ~200-400MB of working memory (ASTs, extracted records, serialization) is active at any time.
### Shared Warning Deduplication
The `warnedCircular` set (used by the COPY expansion engine) is shared across all files in a chunk. This prevents the same circular copybook warning (e.g., `ANAZI includes itself`) from being logged thousands of times.
## Known Limitations
| Limitation | Impact | Workaround |
| ---------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------- |
| tree-sitter-cobol hangs on ~5% of files | Cannot use tree-sitter for COBOL | Regex-only extraction (current approach) |
| Data item cap (500/file) | May miss deeply nested items in large programs | Increase `MAX_DATA_ITEMS_PER_FILE` in source |
| Circular copybooks (ANAZI, ANDIP, QDIPE) | Self-referential includes cannot be expanded | Detected and skipped with warning |
| wfproc/ files may not be pure COBOL | Workflow files may produce extraction noise | Exclude `wfproc` from `GITNEXUS_COBOL_DIRS` if problematic |
| No MOVE DATA_FLOW edges yet | Data flow between variables not in graph | Reserved for future release |
| Continuation line handling | Some complex multi-line continuations (especially in string literals spanning 3+ lines) may not merge correctly | Known edge case; affects <0.1% of lines |
| Single-line EXEC blocks | `EXEC SQL SELECT ... END-EXEC` on one line is handled, but pathological nesting is not | Extremely rare in practice |
| Extension case sensitivity | `.GNM` and `.gnm` are matched differently | Use the exact case from the codebase |
## Troubleshooting
### "COPY expansion failed"
```
[pipeline] COPY expansion failed for s/BGTABFL: Cannot read properties of null
```
**Cause:** A copybook referenced by a COPY statement cannot be found.
**Fix:**
1. Verify `GITNEXUS_COBOL_DIRS` includes the directory containing copybooks (typically `c`)
2. Check that copybook filenames match the COPY target (case-insensitive, after stripping extensions)
3. Ensure copybook files are not in `.gitignore`
### Worker sub-batch timeout
```
Worker 3 sub-batch timed out after 120s (chunk: 200 items)
```
**Cause:** A sub-batch took longer than the timeout. Typically happens when one file is extremely large (50,000+ lines after COPY expansion).
**Fix:** Increase the timeout:
```bash
GITNEXUS_WORKER_TIMEOUT_MS=300000 gitnexus analyze
```
### Memory errors (heap out of memory)
```
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
```
**Fix:** Increase Node.js heap size:
```bash
node --max-old-space-size=16384 /path/to/gitnexus/dist/cli/index.js analyze
```
For very large repos (>500MB source), consider `--max-old-space-size=32768`.
### Concurrent analyze corruption
**Rule:** Only ONE `gitnexus analyze` process should run at a time per repository. Concurrent writes to KuzuDB corrupt the database.
If corruption occurs:
```bash
# Remove the KuzuDB directory and re-index
rm -rf .gitnexus/kuzu
gitnexus analyze --force
```
### Slow KuzuDB write phase
The KuzuDB write phase (132s for PROJECT-NAME) is the bottleneck for large COBOL repos. This is proportional to the number of nodes and edges being written. Reducing `MAX_DATA_ITEMS_PER_FILE` or excluding non-essential directories from `GITNEXUS_COBOL_DIRS` can help.
### Verbose output
Enable verbose logging to see per-phase timing and statistics:
```bash
GITNEXUS_VERBOSE=1 gitnexus analyze
```
This outputs:
- Scan statistics (paths, parseable files, chunk count)
- Worker pool configuration (worker count, sub-batch size)
- COPY expansion statistics (copybooks loaded, files expanded)
- Community and process detection results
- Contract detection results
## Source Files
- `gitnexus/src/core/ingestion/workers/worker-pool.ts` -- `DEFAULT_SUB_BATCH_SIZE`, `SUB_BATCH_TIMEOUT_MS`, `WORKER_STARTUP_TIMEOUT_MS`
- `gitnexus/src/core/ingestion/pipeline.ts` -- `CHUNK_BYTE_BUDGET`, COBOL sub-batch configuration, chunk lifecycle
- `gitnexus/src/core/ingestion/workers/parse-worker.ts` -- `MAX_DATA_ITEMS_PER_FILE`, `processCobolRegexOnly()`
- `gitnexus/src/core/ingestion/parsing-processor.ts` -- Sequential fallback `MAX_DATA_ITEMS_PER_FILE`
@@ -0,0 +1,206 @@
# COBOL Regex Extraction
The `extractCobolSymbolsWithRegex()` function in `cobol-preprocessor.ts` performs single-pass, state-machine-driven extraction of all COBOL symbols. This document describes the state machine, line processing flow, and every regex pattern used.
## State Machine: Division Tracking
The extractor tracks which COBOL division is currently being processed. Division transitions are detected by the `RE_DIVISION` pattern.
```mermaid
stateDiagram-v2
[*] --> null : Start of file
null --> identification : IDENTIFICATION DIVISION
identification --> environment : ENVIRONMENT DIVISION
environment --> data : DATA DIVISION
data --> procedure : PROCEDURE DIVISION
note right of identification
Extracts: PROGRAM-ID, AUTHOR, DATE-WRITTEN
end note
note right of environment
Extracts: SELECT ... ASSIGN ... (file declarations)
end note
note right of data
Extracts: FD entries, data items (01-77, 88), COPY
end note
note right of procedure
Extracts: paragraphs, sections, PERFORM, CALL,
ENTRY, MOVE, EXEC SQL/CICS
end note
```
## State Machine: Data Section Tracking
Within the DATA DIVISION, a secondary state machine tracks the current section to tag data items with their origin.
```mermaid
stateDiagram-v2
[*] --> unknown : DATA DIVISION entered
unknown --> working_storage : WORKING-STORAGE SECTION
unknown --> linkage : LINKAGE SECTION
unknown --> file : FILE SECTION
unknown --> local_storage : LOCAL-STORAGE SECTION
working_storage --> linkage : LINKAGE SECTION
working_storage --> file : FILE SECTION
linkage --> working_storage : WORKING-STORAGE SECTION
file --> working_storage : WORKING-STORAGE SECTION
file --> linkage : LINKAGE SECTION
local_storage --> working_storage : WORKING-STORAGE SECTION
```
Within the ENVIRONMENT DIVISION, the `currentEnvSection` tracks whether we are in `INPUT-OUTPUT` or `CONFIGURATION` section. SELECT statement accumulation only occurs in `INPUT-OUTPUT`.
## Line Processing Flow
Each raw source line goes through this pipeline:
```
Raw line
|
v
Length < 7? ---------> Skip (flush pending if any)
|
v
Indicator col 7
|
+-- '*' or '/' -----> Comment: skip entirely
|
+-- '-' ------------> Continuation: append to pending line
|
+-- other ----------> Normal: flush pending, strip inline comments (|),
buffer as new pending logical line
```
After all lines are processed, the final pending line is flushed, along with any accumulated SELECT statement, SORT/MERGE accumulator, and any open EXEC block (truncated file without `END-EXEC`).
### Inline Comment Stripping
Enterprise COBOL (particularly Italian dialect) uses the pipe character `|` as an inline comment marker. The `stripInlineComment()` helper is **quote-aware**: it tracks whether the scan position is inside a single- or double-quoted string and only treats `|` as a comment marker when outside quotes. Pipe characters inside string literals are preserved.
Free-format `*>` inline comment stripping uses the same quote-aware approach: the scanner walks character by character, toggling quote state, and only recognizes `*>` as a comment marker when not inside a quoted string.
### Patch Marker Handling
The `preprocessCobolSource()` function (run before extraction in the worker) replaces non-standard content in columns 1-6. Standard COBOL expects spaces or digit sequence numbers in this area. If any letter or `#` character is found, the entire sequence area is replaced with 6 spaces:
```
Before: mzADD MOVE WK-AMT TO WK-TOTAL
After: MOVE WK-AMT TO WK-TOTAL
```
This preserves exact line count for position mapping.
## Regex Pattern Reference
All patterns are compiled once as module-level constants and reused across calls.
### Division and Section Detection
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_DIVISION` | `\b(IDENTIFICATION\|ENVIRONMENT\|DATA\|PROCEDURE)\s+DIVISION\b` | Division boundary | `PROCEDURE DIVISION` |
| `RE_SECTION` | `\b(WORKING-STORAGE\|LINKAGE\|FILE\|LOCAL-STORAGE\|INPUT-OUTPUT\|CONFIGURATION)\s+SECTION\b` | Section boundary | `WORKING-STORAGE SECTION` |
### IDENTIFICATION DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_PROGRAM_ID` | `\bPROGRAM-ID\.\s*([A-Z][A-Z0-9-]*)` | Program name | `PROGRAM-ID. BGTABFL` |
| `RE_AUTHOR` | `^\s+AUTHOR\.\s*(.+)` | Author metadata | `AUTHOR. D. Smith` |
| `RE_DATE_WRITTEN` | `^\s+DATE-WRITTEN\.\s*(.+)` | Date metadata | `DATE-WRITTEN. 2024-01-15` |
### ENVIRONMENT DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_SELECT_START` | `\bSELECT\s+(?:OPTIONAL\s+)?([A-Z][A-Z0-9-]+)` | File SELECT start (with optional `SELECT OPTIONAL` support) | `SELECT MASTER-FILE`, `SELECT OPTIONAL TRANS-FILE` |
SELECT statements are accumulated across multiple lines until a period terminator is found, then parsed for ASSIGN, ORGANIZATION, ACCESS, RECORD KEY, and FILE STATUS clauses.
### DATA DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_FD` | `^\s+FD\s+([A-Z][A-Z0-9-]+)` | File description | `FD MASTER-FILE` |
| `RE_DATA_ITEM` | `^\s+(\d{1,2})\s+([A-Z][A-Z0-9-]+)\s*(.*)` | Data item (01-77) | `05 WK-NAME PIC X(30)` |
| `RE_ANONYMOUS_REDEFINES` | `^\s+(\d{1,2})\s+REDEFINES\s+([A-Z][A-Z0-9-]+)` | Anonymous REDEFINES | `01 REDEFINES WK-REC` |
| `RE_88_LEVEL` | `^\s+88\s+([A-Z][A-Z0-9-]+)\s+VALUES?\s+(?:ARE\s+)?(.+)` | Condition name | `88 WK-ACTIVE VALUE "Y"` |
The trailing clauses of `RE_DATA_ITEM` are parsed by `parseDataItemClauses()` for PIC, USAGE, OCCURS, and REDEFINES.
### PROCEDURE DIVISION
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_PROC_SECTION` | `^ ([A-Z][A-Z0-9-]+)\s+SECTION\.\s*$` | Procedure section header | ` MAIN-LOGIC SECTION.` |
| `RE_PROC_PARAGRAPH` | `^ ([A-Z][A-Z0-9-]+)\.\s*$` | Paragraph header | ` PROCESS-RECORD.` |
| `RE_PERFORM` | `\bPERFORM\s+([A-Z][A-Z0-9-]+)(?:\s+THRU\s+([A-Z][A-Z0-9-]+))?` | PERFORM call | `PERFORM CALC-TAX THRU CALC-TAX-EXIT` |
| `RE_PROC_USING` | `\bPROCEDURE\s+DIVISION\s+USING\s+([\s\S]*?)(?:\.\|$)` | USING parameters | `PROCEDURE DIVISION USING WK-PARAM` |
| `RE_ENTRY` | `\bENTRY\s+"([^"]+)"(?:\s+USING\s+([\s\S]*?))?(?:\.\|$)` | ENTRY point | `ENTRY "SUBPROG" USING WK-DATA` |
| `RE_MOVE` | `\bMOVE\s+((?:CORRESPONDING\|CORR)\s+)?([A-Z][A-Z0-9-]+)\s+TO\s+(.+)` | MOVE statement (supports CORR abbreviation and multi-target) | `MOVE WK-NAME TO OUT-NAME`, `MOVE CORR WK-IN TO WK-OUT` |
The USING parameter list (`RE_PROC_USING`) is split on `\bRETURNING\b` before tokenization -- any RETURNING clause and everything after it is excluded from the parameter list (`.split(/\bRETURNING\b/i)[0]`).
Note: `RE_PROC_SECTION` and `RE_PROC_PARAGRAPH` require exactly 7 spaces of leading indentation (COBOL area A starting at column 8). This is the standard COBOL paragraph indentation.
### All-Division Patterns
These patterns are checked regardless of current division:
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_CALL` | `\bCALL\s+"([^"]+)"` | External program call | `CALL "BGTABUP"` |
| `RE_COPY_UNQUOTED` | `\bCOPY\s+([A-Z][A-Z0-9-]+)(?:\s\|\.)` | COPY (unquoted) | `COPY CPSESP.` |
| `RE_COPY_QUOTED` | `\bCOPY\s+"([^"]+)"(?:\s\|\.)` | COPY (quoted) | `COPY "WORKGRID.CPY".` |
### SORT/MERGE Support
| Constant | Purpose |
|----------|---------|
| `SORT_CLAUSE_NOISE` | Set of SORT/MERGE clause keywords filtered from USING/GIVING file lists: `ON`, `ASCENDING`, `DESCENDING`, `KEY`, `WITH`, `DUPLICATES`, `IN`, `ORDER`, `COLLATING`, `SEQUENCE`, `IS`, `THROUGH`, `THRU`, `INPUT`, `OUTPUT`, `PROCEDURE` |
SORT and MERGE statements are accumulated across multiple lines (like SELECT) until a period terminator is found, then parsed for USING/GIVING file lists and INPUT/OUTPUT PROCEDURE targets. The `flushSort()` helper encapsulates the flush-and-parse logic, mirroring the existing `flushSelect()` pattern. Both helpers are called at EOF to handle truncated files.
### GO TO Multi-Target
`RE_GOTO` captures all paragraph names in a `GO TO` statement, including the multi-target form `GO TO p1 p2 p3 DEPENDING ON x`. The captured group contains all target names (space-separated), which are split into individual targets. Each target produces a separate `gotos` entry.
### PROGRAM-ID Detection
PROGRAM-ID is detected regardless of the current division state. This handles sibling programs that appear after `END PROGRAM` and omit the `IDENTIFICATION DIVISION` header -- the extractor will still capture the PROGRAM-ID and push a new program boundary.
### EXEC Block Patterns
| Constant | Pattern | Purpose | Example Match |
|----------|---------|---------|---------------|
| `RE_EXEC_SQL_START` | `\bEXEC\s+SQL\b` | Start of EXEC SQL block | `EXEC SQL` |
| `RE_EXEC_CICS_START` | `\bEXEC\s+CICS\b` | Start of EXEC CICS block | `EXEC CICS` |
| `RE_END_EXEC` | `\bEND-EXEC\b` | End of EXEC block | `END-EXEC` |
EXEC blocks accumulate all lines between `EXEC SQL/CICS` and `END-EXEC`, then delegate to `parseExecSqlBlock()` or `parseExecCicsBlock()` for detailed extraction.
## Excluded Paragraph Names
The following names are excluded from paragraph detection to avoid false positives from division/section headers:
```
DECLARATIVES, END, PROCEDURE, IDENTIFICATION,
ENVIRONMENT, DATA, WORKING-STORAGE, LINKAGE,
FILE, LOCAL-STORAGE, COMMUNICATION, REPORT,
SCREEN, INPUT-OUTPUT, CONFIGURATION
```
Additionally, paragraph candidates containing `DIVISION` or `SECTION` as substrings are excluded.
## MOVE Skip List (Figurative Constants)
MOVE statements where the source is a figurative constant are skipped:
```
SPACES, ZEROS, ZEROES, LOW-VALUES, LOW-VALUE,
HIGH-VALUES, HIGH-VALUE, QUOTES, QUOTE, ALL
```
## Source Files
- `gitnexus/src/core/ingestion/cobol-preprocessor.ts` -- `preprocessCobolSource()`, `extractCobolSymbolsWithRegex()`, all regex constants
@@ -0,0 +1,326 @@
---
title: "feat: Complete COBOL language feature coverage for maximum knowledge graph value"
type: feat
status: active
date: 2026-03-26
origin: Feature audit from v3-integration-architect agent (session 8642401e)
---
## Enhancement Summary
**Deepened on:** 2026-03-26
**Research agents used:** COBOL expert (Phase 1+2), graph value analyst, codebase explorer
**Sections enhanced:** Phase 1 (5 features), Phase 2 (4 features), graph value ranking
### Key Improvements from Research
1. **CALL USING** is the #1 highest-value edge type (9.2/10) — fixes ~40% of missing caller references
2. **EXEC DLI** requires dual-interface support (EXEC DLI + CBLTDLI CALL) for full IMS coverage
3. **DECLARATIVES** is lowest-risk Phase 2 item — existing section/paragraph detection already captures structure
4. **SET TO TRUE** accounts for 80-90% of all SET statements — prioritize this form
5. **INSPECT** needs multi-line accumulator (like SORT) — can span 5+ continuation lines
6. **Graph value ranking**: cobol-call-using (9.2) > cobol-error-handler (9.0) > dli-gu (8.2) > cobol-string (6.2)
### New Edge Cases Discovered
- CALL USING supports mixed modes: `USING BY REFERENCE WS-A BY CONTENT WS-B BY VALUE WS-C`
- CALL USING `ADDRESS OF` and `OMITTED` must be filtered from parameter lists
- EXEC DLI can have multiple SEGMENT levels in hierarchical retrieval (use matchAll)
- DECLARATIVES can have multiple USE sections (one per file + catch-all for INPUT/OUTPUT/I-O/EXTEND)
- INSPECT TALLYING can have multiple counters in a single statement
- STRING/UNSTRING can span multiple lines (need accumulator pattern)
---
# Complete COBOL Language Feature Coverage
## Overview
Implement the remaining 25 unhandled COBOL language features and fix 10 partial features to achieve ~95% coverage (up from 71.9%). The goal is to build the richest possible knowledge graph from COBOL codebases, enabling a future `modernize` MCP command (out of scope for this plan) that would use the graph to assist with COBOL-to-modern-language migration.
## Problem Statement
The COBOL processor currently handles 54 of 89 applicable language features (71.9%). The 25 unhandled features represent real data loss in the knowledge graph:
- **Cross-program data flow** is invisible (CALL ... USING parameters not extracted)
- **IMS/DB programs** produce empty graphs (EXEC DLI not recognized)
- **String transformation logic** is invisible (STRING/UNSTRING/INSPECT not tracked)
- **SQL copybook dependencies** are missing (EXEC SQL INCLUDE not mapped)
- **Error handling flows** are lost (DECLARATIVES/USE AFTER not captured)
## Proposed Solution
Implement features in 4 phases, ordered by graph value density (edges created per LOC of implementation). Each phase is independently shippable and testable.
## Technical Approach
### Phase 1: High-Value Data Flow Edges (~150 LOC, ~8 new edge types)
The highest-ROI features: they create new ACCESSES and IMPORTS edges that directly improve impact analysis.
**Critical research finding**: Multi-line statement accumulation is the dominant challenge. CALL USING, STRING/UNSTRING, and multi-line data item clauses all span multiple lines in production COBOL. The free-format path processes each line independently — these features need statement accumulators (like SORT/SELECT) or the free-format path needs multi-line awareness. Estimated LOC increased from 110 to 150 to account for accumulator infrastructure.
#### 1.1 EXEC SQL INCLUDE -> IMPORTS edges
- **File:** `cobol-preprocessor.ts` (parseExecSqlBlock)
- **What:** Detect `INCLUDE` as the operation, extract member name, emit as a `copies[]` entry
- **Graph:** IMPORTS edge from File to included copybook/SQLCA with reason `sql-include`
- **Tests:** Unit test for `EXEC SQL INCLUDE SQLCA END-EXEC` and `EXEC SQL INCLUDE CUSTCOPY END-EXEC`
**Research insights (EXEC SQL INCLUDE):**
- DB2 member names can contain underscores: `EXEC SQL INCLUDE CUST_TBL_DCL END-EXEC` — regex must use `[A-Z][A-Z0-9_-]+`
- Quoted literal form: `EXEC SQL INCLUDE 'DBRMLIB.MEMBER' END-EXEC` (z/OS PDS qualified name)
- SQLCA/SQLDA are DB2 builtins — won't resolve to repo files. Emit unresolved IMPORTS edge (still valuable)
- No REPLACING support on EXEC SQL INCLUDE (unlike COPY)
- Add `INCLUDE` to `OP_MAP` in `parseExecSqlBlock`; extract member via `RE_SQL_INCLUDE = /^INCLUDE\s+(?:'([^']+)'|"([^"]+)"|([A-Z][A-Z0-9_-]+))/i`
#### 1.2 CALL ... USING parameter extraction -> ACCESSES edges (Graph value: 9.2/10)
- **File:** `cobol-preprocessor.ts` (processLogicalLine CALL section)
- **What:** After capturing CALL target, scan for USING clause. Extract parameter names (reuse USING_KEYWORDS filter). Store as `calls[].parameters: string[]`
- **Interface:** Add `parameters?: string[]` to calls array type in CobolRegexResults
- **File:** `cobol-processor.ts` (CALL edge block)
- **Graph:** For each USING parameter, create ACCESSES edge from caller to data item Property node with reason `cobol-call-using`
- **Tests:** `CALL 'AUDITLOG' USING CUST-ID WS-AMOUNT` -> 2 ACCESSES edges
**Research insights (CALL USING forms):**
- Mixed modes: `CALL 'PGM' USING BY REFERENCE WS-A BY CONTENT WS-B BY VALUE WS-C`
- Pointer passing: `CALL 'PGM' USING ADDRESS OF WS-A`
- Placeholder: `CALL 'PGM' USING OMITTED WS-B`
- Filter keywords: add `ADDRESS`, `OMITTED`, `LENGTH` to USING_KEYWORDS (already has BY/VALUE/REFERENCE/CONTENT)
- **Impact tool enhancement:** CALL-USING edges enable BFS traversal through parameter data flow — single most impactful edge type for COBOL impact analysis
#### 1.3 STRING/UNSTRING data flow -> ACCESSES edges
- **File:** `cobol-preprocessor.ts` (new section in extractProcedure)
- **What:** Accumulate multi-line STRING/UNSTRING until period or END-STRING/END-UNSTRING. Extract sources and INTO targets.
- **Interface:** Add `strings: Array<{ sources: string[]; target: string; type: 'string' | 'unstring'; line: number; caller: string | null }>` to CobolRegexResults
- **Graph:** read-ACCESSES on sources, write-ACCESSES on INTO target with reason `cobol-string-read` / `cobol-string-write`
- **Tests:** 2 unit tests + integration test assertions
**Research insights (STRING/UNSTRING):**
- **Needs statement accumulator** — STRING/UNSTRING always span multiple lines in production
- Terminate accumulation at: period, END-STRING/END-UNSTRING, or start of next COBOL verb
- STRING sources: identifiers before each `DELIMITED BY`. Filter: STRING, DELIMITED, BY, SIZE, ALL, INTO, WITH, POINTER, ON, OVERFLOW, NOT, END-STRING
- UNSTRING: source is first identifier after UNSTRING; INTO targets are identifiers after INTO. Filter: DELIMITER, IN, COUNT, TALLYING, OR
- WITH POINTER field is both read AND written (starting position updated)
- TALLYING IN / COUNT IN fields are write targets
- Literal sources (`'text'`) must be filtered — quote-aware tokenization needed
- **Edge case**: STRING terminated by next verb, not period — existing fixture has `STRING ... DISPLAY` without period between them
#### 1.4 OCCURS DEPENDING ON -> ACCESSES edge
- **File:** `cobol-preprocessor.ts` (parseDataItemClauses)
- **What:** Extend OCCURS regex to capture DEPENDING ON field, KEY fields, and INDEXED BY names
- **Interface:** Add `dependingOn?: string`, `occursMax?: number`, `occursKeys?: Array<{direction: string; fields: string[]}>`, `indexedBy?: string[]` to data items
- **Graph:** ACCESSES edge from table item to controlling field with reason `cobol-depends-on`
- **Tests:** `05 WS-TABLE OCCURS 100 DEPENDING ON WS-COUNT` -> edge
**Research insights (OCCURS):**
- IBM allows `OCCURS 0 TO n DEPENDING ON` (zero minimum) and `OCCURS UNBOUNDED DEPENDING ON` (V6.4)
- Subscripted controlling fields: `DEPENDING ON WS-COUNT(WS-IDX)` — strip subscripts before storing
- **Pre-existing gap**: Multi-line data item clauses without continuation indicator are NOT captured. `05 WS-TABLE\n OCCURS 100\n DEPENDING ON WS-COUNT.` — the current RE_DATA_ITEM only gets the first line, `rest` is empty. Fixing properly requires a data item accumulator (like SELECT). **Defer full fix to Phase 3; implement same-line capture now.**
- KEY IS fields: `ASCENDING KEY IS WS-KEY-1 WS-KEY-2` — capture for SEARCH ALL resolution
- INDEXED BY: `INDEXED BY IDX-1 IDX-2` — capture for SET/SEARCH context
#### 1.5 VALUE clause for standard data items
- **File:** `cobol-preprocessor.ts` (parseDataItemClauses)
- **What:** Extract VALUE using a pragmatic function that handles quoted strings, numerics, figurative constants, hex/national literals
- **Interface:** Already exists as `values?: string[]` on data items (currently only populated for 88-level)
- **Graph:** Stored in Property node description (no new edges)
- **Tests:** `01 WS-STATUS PIC X VALUE 'A'` -> values: ['A']
**Research insights (VALUE forms):**
- Hex literals: `VALUE X'F1F2F3F4'`, National: `VALUE N'text'`, DBCS: `VALUE G'text'`
- Figurative constants: SPACES, ZEROS, ZEROES, LOW-VALUES, HIGH-VALUES, QUOTES, NULL, NULLS
- ALL literal: `VALUE ALL '*'`
- Numeric with sign/decimal: `VALUE -123.45`, `VALUE +1`
- `VALUE IS` optional — both `VALUE 'A'` and `VALUE IS 'A'` valid
- **Decimal vs period ambiguity**: `VALUE 100.` — is `.` decimal or terminator? `parseDataItemClauses` already strips trailing period, so this is handled
- IBM V6.4: floating-point `VALUE 1.0E5` — extend numeric regex if needed
- Implementation: use a pragmatic `extractValue(rest)` function, not a single complex regex
### Phase 2: EXEC DLI + DECLARATIVES (~90 LOC, ~4 new edge types)
IMS/DB support and error handling flows.
#### 2.1 EXEC DLI (IMS/DB) -> ACCESSES edges (Graph value: 8.2/10)
- **File:** `cobol-preprocessor.ts` (processLogicalLine — add RE_EXEC_DLI_START check alongside SQL/CICS)
- **What:** Accumulate EXEC DLI blocks like EXEC SQL. Parse DLI verbs (GU, GN, GNP, GHU, GHN, GHNP, ISRT, DLET, REPL, CHKP, SCHD, TERM). Extract segment name, PCB number, INTO/FROM areas, WHERE fields, PSB name.
- **Interface:** Add `execDliBlocks: Array<{ line: number; verb: string; pcbNumber?: number; segmentName?: string; intoField?: string; fromField?: string; whereField?: string; psbName?: string }>` to CobolRegexResults
- **Graph:** CodeElement node + ACCESSES edge to `<ims>:<segmentName>` Record node with reason `dli-{verb}`; ACCESSES edges to INTO/FROM data areas; PSB ACCESSES for SCHD
- **Tests:** `EXEC DLI GU USING PCB(1) SEGMENT(CUSTOMER) INTO(WS-CUST) END-EXEC`
**Research insights (dual IMS interface):**
- **EXEC DLI**: Embedded command interface for CICS-DL/I programs only
- **CBLTDLI CALL**: Batch interface via `CALL 'CBLTDLI' USING function-code PCB io-area SSA1..SSA15`
- CBLTDLI is already captured as a CALL to 'CBLTDLI' — enrich with USING parameter semantics later
- Multiple SEGMENT levels in hierarchical retrieval — use `matchAll` on segment regex
- DLI verbs: GU (most common), GN, GNP, GHU, GHN, GHNP, ISRT, REPL, DLET, CHKP, SCHD, TERM, ROLL, ROLB
- **Edge case**: DLET/REPL have no SEGMENT clause (operate on current position)
- **Recommended order**: Implement AFTER DECLARATIVES and SET (lower risk, higher frequency)
#### 2.2 DECLARATIVES / USE AFTER STANDARD EXCEPTION (Graph value: 9.0/10)
- **File:** `cobol-preprocessor.ts` (processLogicalLine — detect DECLARATIVES keyword, track USE AFTER blocks)
- **What:** When `DECLARATIVES.` is encountered, switch to declaratives mode. Extract USE statements binding sections to files/modes.
- **Interface:** Add `declaratives: Array<{ sectionName: string; useType: 'error' | 'debug' | 'label' | 'reporting'; target: string; line: number }>` to CobolRegexResults
- **Graph:** ACCESSES edge from declarative Namespace to file Record with reason `cobol-declarative-error-handler`
- **Tests:** Unit test with DECLARATIVES section, integration test for error flow
**Research insights (DECLARATIVES syntax):**
- `USE AFTER STANDARD {EXCEPTION|ERROR} ON {file-name|INPUT|OUTPUT|I-O|EXTEND}`
- EXCEPTION and ERROR are synonymous; STANDARD is optional in IBM dialects
- Multiple USE sections allowed (one per file + catch-all for I/O modes)
- `END DECLARATIVES.` must NOT reset PROCEDURE DIVISION state
- `DECLARATIVES` is already in EXCLUDED_PARA_NAMES — no false paragraph risk
- Existing section/paragraph detection already captures structural elements — just need USE binding
- **Lowest risk Phase 2 item** — implement first
#### 2.3 SET statement -> ACCESSES edges
- **File:** `cobol-preprocessor.ts` (extractProcedure — new RE_SET regex)
- **Interface:** Add `sets: Array<{ targets: string[]; form: 'to-true'|'to-value'|'up-by'|'down-by'|'address-of'|'to-null'|'to-entry'; value?: string; entryTarget?: string; entryIsLiteral?: boolean; line: number; caller: string | null }>` to CobolRegexResults
- **Graph:** ACCESSES write edge with reason `cobol-set-condition` (TO TRUE), `cobol-set-index` (TO/UP/DOWN), `cobol-set-address` (ADDRESS OF). SET ENTRY with literal -> CALLS edge.
- **Tests:** `SET WS-EOF TO TRUE`, `SET IDX-1 TO 5`, `SET IDX-1 UP BY 1`
**Research insights (SET forms by frequency):**
- `SET condition TO TRUE` — 80-90% of all SET usage. Multiple targets: `SET COND-A COND-B TO TRUE`
- `SET index TO/UP BY/DOWN BY` — ~8%. Multiple indices: `SET IDX-1 IDX-2 UP BY 1`
- `SET pointer TO ADDRESS OF data-item` / `SET ADDRESS OF data-item TO pointer` — ~2%
- `SET proc-ptr TO ENTRY "PROGNAME"` — rare but creates CALLS edge (like dynamic CALL)
- Filter OF/IN qualifiers: `SET COND-A OF WS-RECORD TO TRUE` (strip OF WS-RECORD)
- **Prioritize**: SET TO TRUE alone covers 80-90% — implement this form first
#### 2.4 INSPECT -> ACCESSES edges
- **File:** `cobol-preprocessor.ts` (extractProcedure — new `inspectAccum` accumulator like SORT)
- **What:** Accumulate multi-line INSPECT until period. Extract inspected field + tally counters.
- **Interface:** Add `inspects: Array<{ inspectedField: string; counters: string[]; form: 'tallying'|'replacing'|'converting'|'tallying-replacing'; line: number; caller: string | null }>` to CobolRegexResults
- **Graph:** ACCESSES read on inspected field always; write if REPLACING/CONVERTING. Write edges for tally counters. Reason: `cobol-inspect-read`/`cobol-inspect-write`/`cobol-inspect-tally`
- **Tests:** `INSPECT WS-FIELD TALLYING WS-COUNT FOR ALL 'A'` -> read on WS-FIELD, write on WS-COUNT
**Research insights (INSPECT forms by frequency):**
- REPLACING (~60%): `INSPECT WS-STR REPLACING ALL 'A' BY 'B'`
- TALLYING (~25%): `INSPECT WS-STR TALLYING WS-CNT FOR ALL 'A'` — multiple counters possible
- CONVERTING (~10%): `INSPECT WS-STR CONVERTING 'abc' TO 'ABC'`
- Combined (~5%): TALLYING + REPLACING in single statement
- **Needs multi-line accumulator** — INSPECT frequently spans 3-5 lines in production
- Extract tally counters with `([A-Z][A-Z0-9-]+)\s+FOR\b` matchAll pattern
- Filter figurative constants (SPACES, ZEROS) using existing MOVE_SKIP set
### Phase 3: Completeness Fixes (~60 LOC)
Fix the 10 partial features and small gaps.
#### 3.1 CALL ... RETURNING extraction
- Extend RE_CALL processing to capture RETURNING target after the USING clause
- Store as `calls[].returning?: string`
- Graph: ACCESSES write edge with reason `cobol-call-returning`
#### 3.2 SELECT OPTIONAL flag preservation
- Store `isOptional: boolean` in FileDeclaration interface
- Include in Record node description
#### 3.3 ALTERNATE RECORD KEY extraction
- Add regex in parseSelectStatement: `/\bALTERNATE\s+RECORD\s+KEY\s+(?:IS\s+)?([A-Z][A-Z0-9-]+)/i`
- Store as `alternateKeys?: string[]`
#### 3.4 COMMON attribute on nested programs
- Extend RE_PROGRAM_ID: `/\bPROGRAM-ID\.\s*([A-Z][A-Z0-9-]+)(?:\s+IS\s+COMMON)?/i`
- Store `isCommon: boolean` on Module node
- Affects cross-program CALL resolution scope
#### 3.5 IS EXTERNAL / IS GLOBAL as first-class properties
- Change from usage string hack to proper boolean fields on data items
- Add `isExternal?: boolean`, `isGlobal?: boolean` to data item interface
#### 3.6 AUTHOR / DATE-WRITTEN mapped to Module node
- Already extracted as programMetadata — map to Module node properties
- `graph.addNode({ ..., properties: { ..., author, dateWritten } })`
#### 3.7 REPLACE statement
- Track REPLACE / REPLACE OFF state in preprocessor
- Apply text substitutions during preprocessing (before regex extraction)
- Complex: requires careful scoping rules
### Phase 4: Niche Features (~30 LOC)
Low-priority but nice for completeness.
#### 4.1 INITIALIZE statement -> write ACCESSES
- `/\bINITIALIZE\s+([A-Z][A-Z0-9-]+)/i`
- ACCESSES write edge with reason `cobol-initialize`
#### 4.2 Remaining IDENTIFICATION DIVISION paragraphs
- DATE-COMPILED, INSTALLATION, SECURITY, REMARKS
- Map to Module node description properties
#### 4.3 EXEC SQL INCLUDE -> IMPORTS edge (expansion)
- For EXEC SQL INCLUDE inside EXEC blocks that reference copybooks containing SQL
- Create IMPORTS edge similar to COPY
## Acceptance Criteria
### Functional Requirements
- [ ] Phase 1: All 5 features implemented with unit + integration tests
- [ ] Phase 2: All 4 features implemented with unit + integration tests
- [ ] Phase 3: All 7 partial features fixed
- [ ] Phase 4: At least 2 of 3 niche features implemented
- [ ] All existing 145 tests continue to pass
- [ ] TypeScript compiles cleanly
### Non-Functional Requirements
- [ ] No performance regression: CardDemo benchmark stays under 8s
- [ ] No file exceeds 1500 LOC (preprocessor currently 1326)
- [ ] ACAS benchmark shows increased node/edge counts (more data extracted)
- [ ] CardDemo benchmark shows increased edge counts (CALL USING, STRING, etc.)
### Quality Gates
- [ ] Each phase has its own commit
- [ ] Integration test assertions updated with exact counts per phase
- [ ] Benchmark run after each phase to track graph growth
## Dependencies & Risks
### Dependencies
- None. All changes are additive to existing COBOL processor code.
- No LanguageProvider changes needed.
- No graph schema changes needed (all new constructs map to existing node labels + edge types).
### Risks
- **preprocessor.ts size**: Currently 1326 LOC. Phase 1+2 adds ~200 LOC -> 1526 LOC. May need to extract helpers into a separate `cobol-data-flow.ts` module if it exceeds 1500.
- **REPLACE statement** (Phase 3.7) is the most complex feature — requires tracking text substitution state across logical lines. Consider deferring to a separate PR if it takes >100 LOC.
- **EXEC DLI** (Phase 2.1) is only testable against IMS codebases. Need fixture data or synthetic test cases.
## Graph Value Ranking by MCP Tool Impact
Research agent analyzed all 5 MCP tools (query, context, impact, detect_changes, rename) against planned edge types:
| Edge Type | QUERY | CONTEXT | IMPACT | DETECT | RENAME | **Overall** |
|-----------|-------|---------|--------|--------|--------|-------------|
| `cobol-call-using` | 4/5 | 5/5 | 5/5 | 4/5 | 4/5 | **9.2/10** |
| `cobol-error-handler` | 5/5 | 4/5 | 5/5 | 5/5 | 2/5 | **9.0/10** |
| `dli-*` (IMS verbs) | 4/5 | 4/5 | 5/5 | 4/5 | 2/5 | **8.2/10** |
| `cobol-string-*` | 4/5 | 3/5 | 3/5 | 3/5 | 2/5 | **6.2/10** |
**Key finding**: `cobol-call-using` alone would fix ~40% of missing caller references in COBOL graphs.
## Future Considerations
This plan provides the graph data foundation for a future `modernize` MCP command (out of scope) that would:
- Use CALL USING edges to map data contracts between programs
- Use STRING/UNSTRING edges to identify data transformation logic
- Use EXEC SQL/DLI edges to map database access patterns
- Use DECLARATIVES to understand error handling architecture
- Use the complete knowledge graph to generate migration plans
**MCP tool enhancements needed** (after this plan ships):
- Add `cobol-call-using`, `cobol-error-handler`, `dli-*` to IMPACT tool's default `relationTypes` for COBOL repos
- Add confidence floors for new edge types in `IMPACT_RELATION_CONFIDENCE`
- Register new edge types in `VALID_RELATION_TYPES` set (`local-backend.ts:52`)
## Sources & References
### Internal References
- Feature audit: session 8642401e (COBOL expert agent, 123 features audited)
- Prior plans: `docs/plans/2026-03-25-feat-cobol-100-percent-feature-coverage-plan.md`
- Architecture: `docs/code-indexing/cobol/` (7 documentation files)
### External References
- COBOL features reference: mainframestechhelp.com/tutorials/cobol/features.htm
- COBOL-85 standard: ISO/IEC 1989:1985
- IBM Enterprise COBOL reference
-28
View File
@@ -1,28 +0,0 @@
import js from '@eslint/js'
import globals from 'globals'
import reactHooks from 'eslint-plugin-react-hooks'
import reactRefresh from 'eslint-plugin-react-refresh'
import tseslint from 'typescript-eslint'
export default tseslint.config(
{ ignores: ['dist'] },
{
extends: [js.configs.recommended, ...tseslint.configs.recommended],
files: ['**/*.{ts,tsx}'],
languageOptions: {
ecmaVersion: 2020,
globals: globals.browser,
},
plugins: {
'react-hooks': reactHooks,
'react-refresh': reactRefresh,
},
rules: {
...reactHooks.configs.recommended.rules,
'react-refresh/only-export-components': [
'warn',
{ allowConstantExport: true },
],
},
},
)
+83
View File
@@ -0,0 +1,83 @@
import tsPlugin from '@typescript-eslint/eslint-plugin';
import tsParser from '@typescript-eslint/parser';
import unusedImports from 'eslint-plugin-unused-imports';
import reactHooks from 'eslint-plugin-react-hooks';
import prettierConfig from 'eslint-config-prettier';
export default [
// Global ignores
{
ignores: [
'**/dist/**',
'**/node_modules/**',
'**/coverage/**',
'gitnexus/vendor/**',
'gitnexus-web/src/vendor/**',
'gitnexus/test/fixtures/**',
'gitnexus-web/playwright-report/**',
'gitnexus-web/test-results/**',
'**/*.d.ts',
'.claude/**',
'.history/**',
],
},
// Base TypeScript config for all packages
{
files: ['**/*.{ts,tsx}'],
languageOptions: {
parser: tsParser,
parserOptions: {
ecmaVersion: 2022,
sourceType: 'module',
},
},
plugins: {
'@typescript-eslint': tsPlugin,
'unused-imports': unusedImports,
},
rules: {
// Unused imports — auto-fixable
'unused-imports/no-unused-imports': 'error',
'unused-imports/no-unused-vars': [
'warn',
{ vars: 'all', varsIgnorePattern: '^_', args: 'after-used', argsIgnorePattern: '^_' },
],
// TypeScript quality
'@typescript-eslint/no-unused-vars': 'off', // handled by unused-imports plugin
'no-unused-vars': 'off', // handled by unused-imports plugin
'@typescript-eslint/no-explicit-any': 'warn',
'@typescript-eslint/no-non-null-assertion': 'warn',
// General quality
'no-debugger': 'error',
'prefer-const': 'error',
'no-var': 'error',
eqeqeq: ['error', 'always', { null: 'ignore' }],
},
},
// CLI package — allow console.log (it's a CLI tool)
{
files: ['gitnexus/src/cli/**/*.ts', 'gitnexus/src/server/**/*.ts'],
rules: {
'no-console': 'off',
},
},
// React-specific rules for gitnexus-web
{
files: ['gitnexus-web/src/**/*.{ts,tsx}'],
plugins: {
'react-hooks': reactHooks,
},
rules: {
'react-hooks/rules-of-hooks': 'error',
'react-hooks/exhaustive-deps': 'warn',
},
},
// Disable formatting rules (prettier handles those)
prettierConfig,
];
+23
View File
@@ -0,0 +1,23 @@
# ─── GitNexus SWE-bench Eval — API Keys ───
# Copy this file to .env and fill in the keys you have.
# You only need keys for the models you plan to test.
# OpenRouter (covers Claude, MiniMax, GLM, and 200+ other models)
# Get yours at: https://openrouter.ai/keys
OPENROUTER_API_KEY=
# Anthropic (direct — optional if using OpenRouter)
# Get yours at: https://console.anthropic.com/
ANTHROPIC_API_KEY=
# ZhipuAI / GLM (direct — optional if using OpenRouter)
# Get yours at: https://open.bigmodel.cn/
ZHIPUAI_API_KEY=
# MiniMax (direct — optional if using OpenRouter)
MINIMAX_API_KEY=
# ─── Optional ───
# Cost tracking: set to "ignore_errors" if litellm can't find pricing for a model
# MSWEA_COST_TRACKING=ignore_errors
+16
View File
@@ -0,0 +1,16 @@
# Evaluation results (large, should not be committed)
results/
*.traj.json
preds.json
# Python
__pycache__/
*.pyc
*.egg-info/
.eggs/
dist/
build/
# Environment
.env
.venv/
+214
View File
@@ -0,0 +1,214 @@
# GitNexus SWE-bench Evaluation Harness
Evaluate whether GitNexus code intelligence improves AI agent performance on real software engineering tasks. Runs SWE-bench instances across multiple models and compares baseline (no graph) vs GitNexus-enhanced configurations.
## What This Tests
**Hypothesis**: Giving AI agents structural code intelligence (call graphs, execution flows, blast radius analysis) improves their ability to resolve real GitHub issues — measured by resolve rate, cost, and efficiency.
**Evaluation modes:**
| Mode | What the agent gets |
|------|-------------------|
| `baseline` | Standard bash tools (grep, find, cat, sed) — control group |
| `native` | Baseline + explicit GitNexus tools via eval-server (~100ms) |
| `native_augment` | Native tools + grep results automatically enriched with graph context (**recommended**) |
> **Recommended**: Use `native_augment` mode. It mirrors the Claude Code model — the agent gets both explicit GitNexus tools (fast bash commands) AND automatic enrichment of grep results with callers, callees, and execution flows. The agent decides when to use explicit tools vs rely on enriched search output.
**Models supported:**
- Claude 3.5 Haiku, Claude Sonnet 4, Claude Opus 4
- MiniMax M1 2.5
- GLM 4.7, GLM 5
- Any model supported by litellm (add a YAML config)
## Prerequisites
- Python 3.11+
- Docker (for SWE-bench containers)
- Node.js 18+ (for GitNexus)
- API keys for your chosen models
## Setup
```bash
cd eval
# Install dependencies
pip install -e .
# Set up API keys — copy the template and fill in your keys
cp .env.example .env
# Then edit .env and paste your key(s)
```
All models are routed through **OpenRouter** by default, so a single `OPENROUTER_API_KEY` is all you need. To use provider APIs directly (Anthropic, ZhipuAI, etc.), edit the model YAML in `configs/models/` and set the corresponding key in `.env`.
```bash
# Pull SWE-bench Docker images (pulled on-demand, but you can pre-pull)
docker pull swebench/sweb.eval.x86_64.django_1776_django-16527:latest
```
### Debug logging
Set `GITNEXUS_EVAL_DEBUG=1` to include full Python tracebacks in run summaries and logs. By default, errors are sanitized to avoid leaking host paths or stack traces.
## Quick Start
### Debug a single instance
```bash
# Fastest way to verify everything works
python run_eval.py debug -m claude-haiku -i django__django-16527 --subset lite
```
### Run a single configuration
```bash
# 5 instances, Claude Sonnet, native_augment mode (default)
python run_eval.py single -m claude-sonnet --subset lite --slice 0:5
# Baseline comparison (no GitNexus)
python run_eval.py single -m claude-sonnet --mode baseline --subset lite --slice 0:5
# Full Lite benchmark, 4 parallel workers
python run_eval.py single -m claude-sonnet --subset lite -w 4
```
### Run the full matrix
```bash
# All models x all modes
python run_eval.py matrix --subset lite -w 4
# Key comparison: baseline vs native_augment
python run_eval.py matrix -m claude-sonnet -m claude-haiku --modes baseline --modes native_augment --subset lite --slice 0:50
```
### Analyze results
```bash
# Summary table
python -m analysis.analyze_results results/
# Compare modes for a specific model
python -m analysis.analyze_results compare-modes results/ -m claude-sonnet
# GitNexus tool usage analysis
python -m analysis.analyze_results gitnexus-usage results/
# Export as CSV for further analysis
python -m analysis.analyze_results summary results/ --format csv > results.csv
# Run official SWE-bench test evaluation
python -m analysis.analyze_results summary results/ --swebench-eval
```
### List available configurations
```bash
python run_eval.py list-configs
```
## Architecture
```
eval/
run_eval.py # Main entry point (single, matrix, debug commands)
agents/
gitnexus_agent.py # GitNexusAgent: extends DefaultAgent with augmentation + metrics
environments/
gitnexus_docker.py # Docker env with GitNexus + eval-server + standalone tool scripts
bridge/
gitnexus_tools.sh # Bash wrappers (legacy — now standalone scripts are installed directly)
mcp_bridge.py # Legacy MCP bridge (kept for reference)
prompts/
system_baseline.jinja # System: persona + format rules
instance_baseline.jinja # Instance: task + workflow
system_native.jinja # System: + GitNexus tool reference
instance_native.jinja # Instance: + GitNexus debugging workflow
system_native_augment.jinja # System: + GitNexus tools + grep enrichment docs
instance_native_augment.jinja # Instance: + GitNexus workflow + risk assessment
configs/
models/ # Per-model YAML configs
modes/ # Per-mode YAML configs (baseline, native, native_augment)
analysis/
analyze_results.py # Post-run comparative analysis
results/ # Output directory (gitignored)
```
## How It Works
### Template structure
mini-swe-agent requires two Jinja templates:
- **system_template** → system message: persona, format rules, tool reference (static)
- **instance_template** → first user message: task, workflow, rules, examples (contains `{{task}}`)
Each mode has a `system_{mode}.jinja` + `instance_{mode}.jinja` pair. The agent loads both automatically based on the configured mode.
### Per-instance flow
1. Docker container starts with SWE-bench instance (repo at specific commit)
2. **GitNexus setup**: Node.js + gitnexus installed, `gitnexus analyze` runs (or restores from cache)
3. **Eval-server starts**: `gitnexus eval-server` daemon (persistent HTTP server, keeps LadybugDB warm)
4. **Standalone tool scripts installed** in `/usr/local/bin/` — works with `subprocess.run` (no `.bashrc` needed)
5. Agent runs with the configured model + system prompt + GitNexus tools
6. Agent's patch is extracted as a git diff
7. Metrics collected: cost, tokens, tool calls, GitNexus usage, augmentation stats
### Tool architecture
```
Agent → bash command → /usr/local/bin/gitnexus-query
→ curl localhost:4848/tool/query (fast path: eval-server, ~100ms)
→ npx gitnexus query (fallback: cold CLI, ~5-10s)
```
Each tool script in `/usr/local/bin/` is standalone — no sourcing, no env inheritance needed. This is critical because mini-swe-agent runs every command via `subprocess.run` in a fresh subshell.
### Eval-server
The eval-server is a lightweight HTTP daemon that:
- Keeps LadybugDB warm in memory (no cold start per tool call)
- Returns LLM-friendly text (not raw JSON — saves tokens)
- Includes next-step hints to guide tool chaining (query → context → impact → fix)
- Auto-shuts down after idle timeout
### Index caching
SWE-bench repos repeat (Django has 200+ instances at different commits). The harness caches GitNexus indexes per `(repo, commit)` hash in `~/.gitnexus-eval-cache/` to avoid redundant re-indexing.
### Grep augmentation (native_augment mode)
When the agent runs `grep` or `rg`, the observation is post-processed: the agent class calls `gitnexus-augment` on the search pattern and appends `[GitNexus]` annotations showing callers, callees, and execution flows for matched symbols. This mirrors the Claude Code / Cursor hook integration.
## Adding Models
Create a YAML file in `configs/models/`:
```yaml
# configs/models/my-model.yaml
model:
model_name: "openrouter/provider/model-name"
cost_tracking: "ignore_errors" # if not in litellm's cost DB
model_kwargs:
max_tokens: 8192
temperature: 0
```
The model name follows [litellm conventions](https://docs.litellm.ai/docs/providers).
## Metrics Collected
| Metric | Description |
|--------|-------------|
| Patch Rate | % of instances where agent produced a patch |
| Resolve Rate | % of instances where patch passes tests (requires --swebench-eval) |
| Total Cost | API cost across all instances |
| Avg Cost/Instance | Cost efficiency |
| API Calls | Number of LLM calls |
| GN Tool Calls | How many GitNexus tools the agent used |
| Augment Hits | How many grep/find results got enriched |
| Augment Hit Rate | % of search commands that got useful enrichment |
+1
View File
@@ -0,0 +1 @@
# GitNexus SWE-bench Evaluation Harness
View File
+199
View File
@@ -0,0 +1,199 @@
"""
GitNexus-Enhanced Agent for SWE-bench Evaluation
Extends mini-swe-agent's DefaultAgent with:
1. Native augment mode: GitNexus tools via eval-server + grep enrichment (recommended)
2. Native mode: GitNexus tools via eval-server only
3. Baseline mode: Pure mini-swe-agent (no GitNexus — control group)
The agent class itself is minimal — the heavy lifting is in:
- Prompt selection (system + instance templates per mode)
- Observation post-processing (grep result augmentation)
- Metrics tracking (which tools the agent actually uses)
Template structure (matches mini-swe-agent's expectations):
system_template → system message: persona + format rules + tool reference
instance_template → first user message: task + workflow + rules + examples
"""
import logging
import re
import time
from enum import Enum
from pathlib import Path
from constants import AUGMENT_TIMEOUT_SECONDS
from minisweagent import Environment, Model
from minisweagent.agents.default import AgentConfig, DefaultAgent
from tool_registry import BINARIES_BY_KEY, TOOL_METRIC_KEYS
logger = logging.getLogger("gitnexus_agent")
PROMPTS_DIR = Path(__file__).parent.parent / "prompts"
class GitNexusMode(str, Enum):
"""Evaluation modes for GitNexus integration."""
BASELINE = "baseline" # No GitNexus — pure mini-swe-agent
NATIVE = "native" # GitNexus tools via eval-server
NATIVE_AUGMENT = "native_augment" # Native tools + grep enrichment (recommended)
class GitNexusAgentConfig(AgentConfig):
"""Extended config for GitNexus evaluation agent."""
gitnexus_mode: GitNexusMode = GitNexusMode.BASELINE
augment_timeout: float = AUGMENT_TIMEOUT_SECONDS
augment_min_pattern_length: int = 3
track_gitnexus_usage: bool = True
class GitNexusAgent(DefaultAgent):
"""
Agent that optionally enriches its capabilities with GitNexus code intelligence.
In BASELINE mode, behaves identically to DefaultAgent.
In NATIVE mode, GitNexus tools are available as bash commands via eval-server.
In NATIVE_AUGMENT mode, GitNexus tools + automatic grep result enrichment.
"""
def __init__(self, model: Model, env: Environment, *, config_class: type = GitNexusAgentConfig, **kwargs):
mode = kwargs.get("gitnexus_mode", GitNexusMode.BASELINE)
if isinstance(mode, str):
mode = GitNexusMode(mode)
# Load system template
system_file = PROMPTS_DIR / f"system_{mode.value}.jinja"
if system_file.exists() and "system_template" not in kwargs:
kwargs["system_template"] = system_file.read_text()
# Load instance template
instance_file = PROMPTS_DIR / f"instance_{mode.value}.jinja"
if instance_file.exists() and "instance_template" not in kwargs:
kwargs["instance_template"] = instance_file.read_text()
super().__init__(model, env, config_class=config_class, **kwargs)
self.gitnexus_mode = mode
self.gitnexus_metrics = GitNexusMetrics()
def execute_actions(self, message: dict) -> list[dict]:
"""Execute actions with optional GitNexus augmentation and tracking."""
if self.config.track_gitnexus_usage:
self._track_tool_usage(message)
outputs = [self.env.execute(action) for action in message.get("extra", {}).get("actions", [])]
# Augment grep/find observations in NATIVE_AUGMENT mode
if self.gitnexus_mode == GitNexusMode.NATIVE_AUGMENT:
actions = message.get("extra", {}).get("actions", [])
for i, (action, output) in enumerate(zip(actions, outputs)):
augmented = self._maybe_augment(action, output)
if augmented:
outputs[i] = augmented
return self.add_messages(
*self.model.format_observation_messages(message, outputs, self.get_template_vars())
)
def _maybe_augment(self, action: dict, output: dict) -> dict | None:
"""
If the action is a search command (grep, find, rg, ag), augment the output
with GitNexus knowledge graph context.
"""
command = action.get("command", "")
if not command:
return None
pattern = self._extract_search_pattern(command)
if not pattern or len(pattern) < self.config.augment_min_pattern_length:
return None
start = time.time()
try:
augment_result = self.env.execute({
"command": f'gitnexus-augment "{pattern}" 2>&1 || true',
"timeout": self.config.augment_timeout,
})
elapsed = time.time() - start
self.gitnexus_metrics.augmentation_calls += 1
self.gitnexus_metrics.augmentation_time += elapsed
augment_text = augment_result.get("output", "").strip()
if augment_text and "[GitNexus]" in augment_text:
original_output = output.get("output", "")
output = dict(output)
output["output"] = f"{original_output}\n\n{augment_text}"
self.gitnexus_metrics.augmentation_hits += 1
return output
except Exception as e:
logger.debug(f"Augmentation failed for pattern '{pattern}': {e}")
self.gitnexus_metrics.augmentation_errors += 1
return None
@staticmethod
def _extract_search_pattern(command: str) -> str | None:
"""Extract the search pattern from a grep/find/rg command."""
patterns = [
r'(?:grep|rg|ag)\s+(?:-[a-zA-Z]*\s+)*["\']([^"\']+)["\']',
r'(?:grep|rg|ag)\s+(?:-[a-zA-Z]*\s+)*(\S+)',
]
for pat in patterns:
match = re.search(pat, command)
if match:
result = match.group(1)
if result.startswith("/") or result.startswith("."):
continue
if result.startswith("-"):
continue
return result
return None
def _track_tool_usage(self, message: dict):
"""Track which GitNexus tools the agent uses."""
for action in message.get("extra", {}).get("actions", []):
command = action.get("command", "")
for key, binary in BINARIES_BY_KEY.items():
if binary in command and key in self.gitnexus_metrics.tool_calls:
self.gitnexus_metrics.tool_calls[key] += 1
break
def serialize(self, *extra_dicts) -> dict:
"""Serialize with GitNexus-specific metrics."""
gitnexus_data = {
"info": {
"gitnexus": {
"mode": self.gitnexus_mode.value,
"metrics": self.gitnexus_metrics.to_dict(),
},
},
}
return super().serialize(gitnexus_data, *extra_dicts)
class GitNexusMetrics:
"""Tracks GitNexus-specific metrics during evaluation."""
def __init__(self):
self.tool_calls: dict[str, int] = {key: 0 for key in TOOL_METRIC_KEYS}
self.augmentation_calls: int = 0
self.augmentation_hits: int = 0
self.augmentation_errors: int = 0
self.augmentation_time: float = 0.0
self.index_time: float = 0.0
@property
def total_tool_calls(self) -> int:
return sum(self.tool_calls.values())
def to_dict(self) -> dict:
return {
"tool_calls": dict(self.tool_calls),
"total_tool_calls": self.total_tool_calls,
"augmentation_calls": self.augmentation_calls,
"augmentation_hits": self.augmentation_hits,
"augmentation_errors": self.augmentation_errors,
"augmentation_time_seconds": round(self.augmentation_time, 2),
"index_time_seconds": round(self.index_time, 2),
}
View File
+455
View File
@@ -0,0 +1,455 @@
#!/usr/bin/env python3
"""
Results Analyzer for GitNexus SWE-bench Evaluation
Reads evaluation results and generates comparative analysis:
- Resolve rate by model x mode
- Cost comparison (total, per-instance)
- Token/API call efficiency
- GitNexus tool usage patterns
- Augmentation hit rates
Usage:
python -m analysis.analyze_results /path/to/results
python -m analysis.analyze_results /path/to/results --format markdown
python -m analysis.analyze_results /path/to/results --swebench-eval # run actual test verification
"""
import json
import logging
import os
import subprocess
import sys
from pathlib import Path
from typing import Any
import typer
from rich.console import Console
from rich.table import Table
from tool_registry import TOOL_METRIC_KEYS
logger = logging.getLogger("analyze_results")
console = Console()
app = typer.Typer(rich_markup_mode="rich", add_completion=False)
def load_run_results(results_dir: Path) -> dict[str, dict]:
"""
Load all run results from the results directory.
Returns: {run_id: {summary, preds, instances}}
"""
runs = {}
for run_dir in sorted(results_dir.iterdir()):
if not run_dir.is_dir():
continue
run_id = run_dir.name
run_data: dict[str, Any] = {"run_id": run_id, "dir": run_dir}
# Load summary
summary_path = run_dir / "summary.json"
if summary_path.exists():
run_data["summary"] = json.loads(summary_path.read_text())
# Load predictions
preds_path = run_dir / "preds.json"
if preds_path.exists():
run_data["preds"] = json.loads(preds_path.read_text())
# Load individual trajectories for detailed metrics
run_data["trajectories"] = {}
for traj_dir in run_dir.iterdir():
if not traj_dir.is_dir():
continue
for traj_file in traj_dir.glob("*.traj.json"):
try:
traj = json.loads(traj_file.read_text())
instance_id = traj.get("instance_id", traj_dir.name)
run_data["trajectories"][instance_id] = traj
except Exception:
pass
if run_data.get("preds") or run_data.get("summary"):
runs[run_id] = run_data
return runs
def parse_run_id(run_id: str) -> tuple[str, str]:
"""Parse 'model_mode' into (model, mode) using known suffixes."""
# Match the longest known suffix first to avoid hyphen collisions in model names.
known_modes = [
"native_augment",
"native",
"baseline",
"mcp",
"augment",
"full",
]
for mode in known_modes:
suffix = f"_{mode}"
if run_id.endswith(suffix):
return run_id[: -len(suffix)], mode
return run_id, "unknown"
def compute_metrics(run_data: dict) -> dict:
"""Compute evaluation metrics for a single run."""
preds = run_data.get("preds", {})
summary = run_data.get("summary", {})
trajectories = run_data.get("trajectories", {})
n_instances = len(preds)
n_with_patch = sum(1 for p in preds.values() if p.get("model_patch", "").strip())
# Cost and API call metrics from trajectories
costs = []
api_calls = []
gn_tool_calls = []
gn_augment_hits = []
gn_augment_calls = []
for instance_id, traj in trajectories.items():
info = traj.get("info", {})
model_stats = info.get("model_stats", {})
costs.append(model_stats.get("instance_cost", 0))
api_calls.append(model_stats.get("api_calls", 0))
gn = info.get("gitnexus", {}).get("metrics", {})
if gn:
gn_tool_calls.append(gn.get("total_tool_calls", 0))
gn_augment_hits.append(gn.get("augmentation_hits", 0))
gn_augment_calls.append(gn.get("augmentation_calls", 0))
# Also try summary-level metrics
if not costs and summary:
results = summary.get("results", [])
for r in results:
costs.append(r.get("cost", 0))
api_calls.append(r.get("n_calls", 0))
gn = r.get("gitnexus_metrics", {})
if gn:
gn_tool_calls.append(gn.get("total_tool_calls", 0))
gn_augment_hits.append(gn.get("augmentation_hits", 0))
gn_augment_calls.append(gn.get("augmentation_calls", 0))
total_cost = sum(costs)
total_calls = sum(api_calls)
return {
"n_instances": n_instances,
"n_with_patch": n_with_patch,
"patch_rate": n_with_patch / max(n_instances, 1),
"total_cost": total_cost,
"avg_cost": total_cost / max(n_instances, 1),
"total_api_calls": total_calls,
"avg_api_calls": total_calls / max(n_instances, 1),
"total_gn_tool_calls": sum(gn_tool_calls),
"avg_gn_tool_calls": sum(gn_tool_calls) / max(len(gn_tool_calls), 1) if gn_tool_calls else 0,
"total_augment_hits": sum(gn_augment_hits),
"total_augment_calls": sum(gn_augment_calls),
"augment_hit_rate": sum(gn_augment_hits) / max(sum(gn_augment_calls), 1) if gn_augment_calls else 0,
}
def run_swebench_evaluation(results_dir: Path, run_id: str, subset: str = "lite") -> dict | None:
"""
Run the official SWE-bench evaluation on predictions.
Requires: pip install swebench
"""
preds_path = results_dir / run_id / "preds.json"
if not preds_path.exists():
return None
dataset_mapping = {
"lite": "princeton-nlp/SWE-Bench_Lite",
"verified": "princeton-nlp/SWE-Bench_Verified",
"full": "princeton-nlp/SWE-Bench",
}
try:
eval_output = results_dir / run_id / "swebench_eval"
cmd = [
sys.executable, "-m", "swebench.harness.run_evaluation",
"--dataset_name", dataset_mapping.get(subset, subset),
"--predictions_path", str(preds_path),
"--max_workers", "4",
"--run_id", run_id,
"--output_dir", str(eval_output),
]
logger.info(f"Running SWE-bench evaluation for {run_id}...")
result = subprocess.run(cmd, capture_output=True, text=True, timeout=600)
if result.returncode == 0:
# Parse evaluation results
report_path = eval_output / run_id / "results.json"
if report_path.exists():
return json.loads(report_path.read_text())
logger.error(f"SWE-bench eval failed: {result.stderr[:500]}")
return None
except Exception as e:
logger.error(f"SWE-bench eval error: {e}")
return None
# ─── CLI Commands ───────────────────────────────────────────────────────────
@app.command()
def summary(
results_dir: str = typer.Argument(..., help="Path to results directory"),
format: str = typer.Option("table", "--format", help="Output format: table, markdown, json, csv"),
swebench_eval: bool = typer.Option(False, "--swebench-eval", help="Run official SWE-bench test evaluation"),
subset: str = typer.Option("lite", "--subset", help="SWE-bench subset (for --swebench-eval)"),
):
"""Generate comparative analysis of evaluation results."""
results_path = Path(results_dir)
if not results_path.exists():
console.print(f"[red]Results directory not found: {results_path}[/red]")
raise typer.Exit(1)
runs = load_run_results(results_path)
if not runs:
console.print("[yellow]No evaluation results found[/yellow]")
raise typer.Exit(0)
console.print(f"\n[bold]Found {len(runs)} evaluation runs[/bold]\n")
# Compute metrics per run
all_metrics = {}
for run_id, run_data in runs.items():
model, mode = parse_run_id(run_id)
metrics = compute_metrics(run_data)
metrics["model"] = model
metrics["mode"] = mode
# Optionally run SWE-bench evaluation
if swebench_eval:
eval_result = run_swebench_evaluation(results_path, run_id, subset)
if eval_result:
metrics["resolved"] = eval_result.get("resolved", 0)
metrics["resolve_rate"] = eval_result.get("resolved", 0) / max(metrics["n_instances"], 1)
all_metrics[run_id] = metrics
if format == "table":
_print_table(all_metrics)
elif format == "markdown":
_print_markdown(all_metrics)
elif format == "json":
console.print(json.dumps(all_metrics, indent=2))
elif format == "csv":
_print_csv(all_metrics)
@app.command()
def compare_modes(
results_dir: str = typer.Argument(..., help="Path to results directory"),
model: str = typer.Option(..., "-m", "--model", help="Model to compare across modes"),
):
"""Compare modes for a specific model (baseline vs mcp vs augment vs full)."""
results_path = Path(results_dir)
runs = load_run_results(results_path)
# Filter to the specified model
model_runs = {
run_id: data for run_id, data in runs.items()
if parse_run_id(run_id)[0] == model
}
if not model_runs:
console.print(f"[yellow]No results found for model: {model}[/yellow]")
raise typer.Exit(1)
console.print(f"\n[bold]Mode comparison for {model}[/bold]\n")
metrics = {}
for run_id, run_data in model_runs.items():
_, mode = parse_run_id(run_id)
metrics[mode] = compute_metrics(run_data)
mode_order = [
mode
for mode in ["baseline", "native", "native_augment", "mcp", "augment", "full"]
if mode in metrics
] or sorted(metrics.keys())
# Print comparison table
table = Table(title=f"Mode Comparison: {model}")
table.add_column("Metric", style="bold")
for mode in mode_order:
table.add_column(mode, justify="right")
rows = [
("Instances", "n_instances", "d"),
("With Patch", "n_with_patch", "d"),
("Patch Rate", "patch_rate", ".1%"),
("Total Cost", "total_cost", "$.4f"),
("Avg Cost", "avg_cost", "$.4f"),
("Total API Calls", "total_api_calls", "d"),
("Avg API Calls", "avg_api_calls", ".1f"),
("GN Tool Calls", "total_gn_tool_calls", "d"),
("Augment Hits", "total_augment_hits", "d"),
("Augment Hit Rate", "augment_hit_rate", ".1%"),
]
for label, key, fmt in rows:
values = []
for mode in mode_order:
if mode in metrics:
v = metrics[mode].get(key, 0)
if fmt == ".1%":
values.append(f"{v:.1%}")
elif fmt == "$.4f":
values.append(f"${v:.4f}")
elif fmt == ".1f":
values.append(f"{v:.1f}")
else:
values.append(str(v))
table.add_row(label, *values)
# Add delta rows (improvement over baseline)
if "baseline" in metrics:
baseline_cost = metrics["baseline"]["avg_cost"]
baseline_calls = metrics["baseline"]["avg_api_calls"]
table.add_section()
for mode in mode_order:
if mode == "baseline":
continue
mode_cost = metrics[mode]["avg_cost"]
mode_calls = metrics[mode]["avg_api_calls"]
cost_delta = ((mode_cost - baseline_cost) / max(baseline_cost, 0.001)) * 100
calls_delta = ((mode_calls - baseline_calls) / max(baseline_calls, 1)) * 100
cost_str = f"{cost_delta:+.1f}%"
calls_str = f"{calls_delta:+.1f}%"
# Color-code: negative is good (cheaper/fewer calls)
cost_color = "green" if cost_delta < 0 else "red"
calls_color = "green" if calls_delta < 0 else "red"
console.print(f" {mode} vs baseline: cost [{cost_color}]{cost_str}[/{cost_color}], calls [{calls_color}]{calls_str}[/{calls_color}]")
console.print(table)
@app.command()
def gitnexus_usage(
results_dir: str = typer.Argument(..., help="Path to results directory"),
):
"""Analyze GitNexus tool usage patterns across all runs."""
results_path = Path(results_dir)
runs = load_run_results(results_path)
console.print("\n[bold]GitNexus Tool Usage Analysis[/bold]\n")
table = Table(title="Tool Usage by Run")
table.add_column("Run", style="bold")
for key in TOOL_METRIC_KEYS:
table.add_column(key, justify="right")
table.add_column("Total", justify="right")
table.add_column("Augment Hits", justify="right")
for run_id, run_data in sorted(runs.items()):
_, mode = parse_run_id(run_id)
if mode == "baseline":
continue
# Aggregate tool calls across trajectories
tool_totals: dict[str, int] = {key: 0 for key in TOOL_METRIC_KEYS}
augment_hits = 0
for traj in run_data.get("trajectories", {}).values():
gn = traj.get("info", {}).get("gitnexus", {}).get("metrics", {})
for tool, count in gn.get("tool_calls", {}).items():
tool_totals[tool] = tool_totals.get(tool, 0) + count
augment_hits += gn.get("augmentation_hits", 0)
# Also check summary
for r in run_data.get("summary", {}).get("results", []):
gn = r.get("gitnexus_metrics", {})
for tool, count in gn.get("tool_calls", {}).items():
tool_totals[tool] = tool_totals.get(tool, 0) + count
augment_hits += gn.get("augmentation_hits", 0)
total = sum(tool_totals.values())
if total > 0 or augment_hits > 0:
table.add_row(
run_id,
*[str(tool_totals.get(key, 0)) for key in TOOL_METRIC_KEYS],
str(total),
str(augment_hits),
)
console.print(table)
# ─── Output Formatters ─────────────────────────────────────────────────────
def _print_table(all_metrics: dict):
"""Print rich table summary."""
table = Table(title="Evaluation Results")
table.add_column("Run", style="bold")
table.add_column("Model")
table.add_column("Mode")
table.add_column("N", justify="right")
table.add_column("Patched", justify="right")
table.add_column("Rate", justify="right")
table.add_column("Cost", justify="right")
table.add_column("Calls", justify="right")
table.add_column("GN Tools", justify="right")
for run_id, m in sorted(all_metrics.items()):
resolved_str = ""
if "resolve_rate" in m:
resolved_str = f" ({m['resolve_rate']:.0%})"
table.add_row(
run_id,
m["model"],
m["mode"],
str(m["n_instances"]),
str(m["n_with_patch"]),
f"{m['patch_rate']:.0%}{resolved_str}",
f"${m['total_cost']:.2f}",
str(m["total_api_calls"]),
str(m["total_gn_tool_calls"]) if m["total_gn_tool_calls"] > 0 else "-",
)
console.print(table)
def _print_markdown(all_metrics: dict):
"""Print markdown table."""
print("| Run | Model | Mode | N | Patched | Rate | Cost | Calls | GN Tools |")
print("|-----|-------|------|---|---------|------|------|-------|----------|")
for run_id, m in sorted(all_metrics.items()):
gn = str(m["total_gn_tool_calls"]) if m["total_gn_tool_calls"] > 0 else "-"
print(f"| {run_id} | {m['model']} | {m['mode']} | {m['n_instances']} | {m['n_with_patch']} | {m['patch_rate']:.0%} | ${m['total_cost']:.2f} | {m['total_api_calls']} | {gn} |")
def _print_csv(all_metrics: dict):
"""Print CSV output."""
print("run_id,model,mode,n_instances,n_with_patch,patch_rate,total_cost,avg_cost,total_api_calls,avg_api_calls,total_gn_tool_calls,total_augment_hits,augment_hit_rate")
for run_id, m in sorted(all_metrics.items()):
print(
f"{run_id},{m['model']},{m['mode']},{m['n_instances']},{m['n_with_patch']},"
f"{m['patch_rate']:.4f},{m['total_cost']:.4f},{m['avg_cost']:.4f},"
f"{m['total_api_calls']},{m['avg_api_calls']:.1f},{m['total_gn_tool_calls']},"
f"{m['total_augment_hits']},{m['augment_hit_rate']:.4f}"
)
if __name__ == "__main__":
logging.basicConfig(level=logging.INFO)
app()
View File
+155
View File
@@ -0,0 +1,155 @@
#!/bin/bash
# GitNexus CLI tool wrappers for SWE-bench evaluation
#
# These functions call the GitNexus eval-server (HTTP daemon) for near-instant
# tool responses. The eval-server keeps KuzuDB warm in memory.
#
# If the eval-server is not running, falls back to direct CLI commands.
#
# Usage:
# gitnexus-query "how does authentication work"
# gitnexus-context "validateUser"
# gitnexus-impact "AuthService" upstream
# gitnexus-cypher "MATCH (n:Function) RETURN n.name LIMIT 10"
# gitnexus-overview
GITNEXUS_EVAL_PORT="${GITNEXUS_EVAL_PORT:-4848}"
GITNEXUS_EVAL_URL="http://127.0.0.1:${GITNEXUS_EVAL_PORT}"
_gitnexus_call() {
local tool="$1"
shift
local json_body="$1"
# Try eval-server first (fastest path — KuzuDB stays warm)
local result
result=$(curl -sf -X POST "${GITNEXUS_EVAL_URL}/tool/${tool}" \
-H "Content-Type: application/json" \
-d "${json_body}" 2>/dev/null)
if [ $? -eq 0 ] && [ -n "$result" ]; then
echo "$result"
return 0
fi
# Fallback: direct CLI (cold start, slower but always works)
case "$tool" in
query)
local q=$(echo "$json_body" | python3 -c "import sys,json; print(json.load(sys.stdin).get('query',''))" 2>/dev/null)
npx gitnexus query "$q" 2>&1
;;
context)
local n=$(echo "$json_body" | python3 -c "import sys,json; print(json.load(sys.stdin).get('name',''))" 2>/dev/null)
npx gitnexus context "$n" 2>&1
;;
impact)
local t=$(echo "$json_body" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('target',''))" 2>/dev/null)
local d=$(echo "$json_body" | python3 -c "import sys,json; d=json.load(sys.stdin); print(d.get('direction','upstream'))" 2>/dev/null)
npx gitnexus impact "$t" --direction "$d" 2>&1
;;
cypher)
local cq=$(echo "$json_body" | python3 -c "import sys,json; print(json.load(sys.stdin).get('query',''))" 2>/dev/null)
npx gitnexus cypher "$cq" 2>&1
;;
*)
echo "Unknown tool: $tool" >&2
return 1
;;
esac
}
gitnexus-query() {
local query="$1"
local task_context="${2:-}"
local goal="${3:-}"
if [ -z "$query" ]; then
echo "Usage: gitnexus-query <query> [task_context] [goal]"
echo "Search the code knowledge graph for execution flows related to a concept."
echo ""
echo "Examples:"
echo ' gitnexus-query "authentication flow"'
echo ' gitnexus-query "database connection" "fixing connection pool leak"'
return 1
fi
local args="{\"query\": \"$query\""
[ -n "$task_context" ] && args="$args, \"task_context\": \"$task_context\""
[ -n "$goal" ] && args="$args, \"goal\": \"$goal\""
args="$args}"
_gitnexus_call query "$args"
}
gitnexus-context() {
local name="$1"
local file_path="${2:-}"
if [ -z "$name" ]; then
echo "Usage: gitnexus-context <symbol_name> [file_path]"
echo "Get a 360-degree view of a code symbol: callers, callees, processes, file location."
echo ""
echo "Examples:"
echo ' gitnexus-context "validateUser"'
echo ' gitnexus-context "AuthService" "src/auth/service.py"'
return 1
fi
local args="{\"name\": \"$name\""
[ -n "$file_path" ] && args="$args, \"file_path\": \"$file_path\""
args="$args}"
_gitnexus_call context "$args"
}
gitnexus-impact() {
local target="$1"
local direction="${2:-upstream}"
if [ -z "$target" ]; then
echo "Usage: gitnexus-impact <symbol_name> [upstream|downstream]"
echo "Analyze the blast radius of changing a code symbol."
echo ""
echo " upstream = what depends on this (what breaks if you change it)"
echo " downstream = what this depends on (what it uses)"
echo ""
echo "Examples:"
echo ' gitnexus-impact "AuthService" upstream'
echo ' gitnexus-impact "validateUser" downstream'
return 1
fi
_gitnexus_call impact "{\"target\": \"$target\", \"direction\": \"$direction\"}"
}
gitnexus-cypher() {
local query="$1"
if [ -z "$query" ]; then
echo "Usage: gitnexus-cypher <cypher_query>"
echo "Execute a raw Cypher query against the code knowledge graph."
echo ""
echo "Schema: Nodes: File, Function, Class, Method, Interface, Community, Process"
echo "Edges via CodeRelation.type: CALLS, IMPORTS, EXTENDS, IMPLEMENTS, DEFINES, MEMBER_OF, STEP_IN_PROCESS"
echo ""
echo "Examples:"
echo " gitnexus-cypher 'MATCH (a)-[:CodeRelation {type: \"CALLS\"}]->(b:Function {name: \"save\"}) RETURN a.name, a.filePath'"
echo " gitnexus-cypher 'MATCH (n:Class) RETURN n.name, n.filePath LIMIT 20'"
return 1
fi
_gitnexus_call cypher "{\"query\": \"$query\"}"
}
gitnexus-overview() {
echo "=== Code Knowledge Graph Overview ==="
_gitnexus_call list_repos '{}'
}
# Export functions so they're available in subshells
export -f _gitnexus_call 2>/dev/null
export -f gitnexus-query 2>/dev/null
export -f gitnexus-context 2>/dev/null
export -f gitnexus-impact 2>/dev/null
export -f gitnexus-cypher 2>/dev/null
export -f gitnexus-overview 2>/dev/null
+378
View File
@@ -0,0 +1,378 @@
"""
MCP Bridge for GitNexus
Starts the GitNexus MCP server as a subprocess and provides a Python interface
to call MCP tools. Used by the bash wrapper scripts and the augmentation layer..
The bridge communicates with the MCP server via stdio using the JSON-RPC protocol.
"""
import json
import logging
import os
import subprocess
import sys
import threading
import time
from pathlib import Path
from typing import Any
from constants import (
MCP_FIND_GITNEXUS_FALLBACK_TIMEOUT_SECONDS,
MCP_FIND_GITNEXUS_TIMEOUT_SECONDS,
MCP_READ_TIMEOUT_SECONDS,
MCP_STOP_WAIT_SECONDS,
)
from utils.errors import is_debug_enabled, log_safe_exception
logger = logging.getLogger("mcp_bridge")
class MCPBridge:
"""
Manages a GitNexus MCP server subprocess and proxies tool calls to it.
Usage:
bridge = MCPBridge(repo_path="/path/to/repo")
bridge.start()
result = bridge.call_tool("query", {"query": "authentication"})
bridge.stop()
"""
def __init__(self, repo_path: str | None = None):
self.repo_path = repo_path or os.getcwd()
self.process: subprocess.Popen | None = None
self._request_id = 0
self._lock = threading.Lock()
self._started = False
def start(self) -> bool:
"""Start the GitNexus MCP server subprocess."""
if self._started:
return True
try:
# Find gitnexus binary
gitnexus_bin = self._find_gitnexus()
if not gitnexus_bin:
logger.error("GitNexus not found. Install with: npm install -g gitnexus")
return False
self.process = subprocess.Popen(
[gitnexus_bin, "mcp"],
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
cwd=self.repo_path,
text=False,
)
# Send initialize request
init_result = self._send_request("initialize", {
"protocolVersion": "2024-11-05",
"capabilities": {},
"clientInfo": {"name": "gitnexus-eval", "version": "0.1.0"},
})
if init_result is None:
logger.error("MCP server failed to initialize")
self.stop()
return False
# Send initialized notification
self._send_notification("notifications/initialized", {})
self._started = True
logger.info("MCP bridge started successfully")
return True
except Exception as e:
log_safe_exception(logger, "Failed to start MCP bridge", e, include_debug=is_debug_enabled())
self.stop()
return False
def stop(self):
"""Stop the MCP server subprocess."""
if self.process:
try:
if self.process.stdin:
self.process.stdin.close()
if self.process.stdout:
self.process.stdout.close()
if self.process.stderr:
self.process.stderr.close()
self.process.terminate()
self.process.wait(timeout=MCP_STOP_WAIT_SECONDS)
except Exception:
try:
self.process.kill()
except Exception:
pass
self.process = None
self._started = False
def call_tool(self, tool_name: str, arguments: dict[str, Any] | None = None) -> dict[str, Any] | None:
"""
Call a GitNexus MCP tool and return the result.
Returns the tool result content or None on error.
"""
if not self._started:
logger.error("MCP bridge not started")
return None
result = self._send_request("tools/call", {
"name": tool_name,
"arguments": arguments or {},
})
if result is None:
return None
# Extract text content from MCP response
content = result.get("content", [])
if content and isinstance(content, list):
texts = [item.get("text", "") for item in content if item.get("type") == "text"]
return {"text": "\n".join(texts), "raw": content}
return {"text": "", "raw": content}
def list_tools(self) -> list[dict]:
"""List available MCP tools."""
result = self._send_request("tools/list", {})
if result:
return result.get("tools", [])
return []
def read_resource(self, uri: str) -> str | None:
"""Read an MCP resource by URI."""
result = self._send_request("resources/read", {"uri": uri})
if result:
contents = result.get("contents", [])
if contents:
return contents[0].get("text", "")
return None
def _find_gitnexus(self) -> str | None:
"""Find the gitnexus CLI binary."""
# Check if npx is available (preferred - uses local install)
for cmd in ["npx"]:
try:
result = subprocess.run(
[cmd, "gitnexus", "--version"],
capture_output=True,
text=True,
timeout=MCP_FIND_GITNEXUS_TIMEOUT_SECONDS,
cwd=self.repo_path,
)
if result.returncode == 0:
return cmd # Will use "npx gitnexus mcp"
except Exception:
continue
# Check for global install
try:
result = subprocess.run(
["gitnexus", "--version"],
capture_output=True,
text=True,
timeout=MCP_FIND_GITNEXUS_FALLBACK_TIMEOUT_SECONDS,
)
if result.returncode == 0:
return "gitnexus"
except Exception:
pass
return None
def _next_id(self) -> int:
with self._lock:
self._request_id += 1
return self._request_id
def _send_request(self, method: str, params: dict) -> dict | None:
"""Send a JSON-RPC request and wait for response."""
if not self.process or not self.process.stdin or not self.process.stdout:
return None
request_id = self._next_id()
request = {
"jsonrpc": "2.0",
"id": request_id,
"method": method,
"params": params,
}
try:
message = json.dumps(request)
# MCP uses Content-Length header framing
header = f"Content-Length: {len(message.encode('utf-8'))}\r\n\r\n"
self.process.stdin.write(header.encode("utf-8"))
self.process.stdin.write(message.encode("utf-8"))
self.process.stdin.flush()
# Read response
response = self._read_response(timeout=MCP_READ_TIMEOUT_SECONDS)
if response and response.get("id") == request_id:
if "error" in response:
logger.error(f"MCP error: {response['error']}")
return None
return response.get("result")
return None
except Exception as e:
log_safe_exception(logger, "MCP request failed", e, include_debug=is_debug_enabled())
return None
def _send_notification(self, method: str, params: dict):
"""Send a JSON-RPC notification (no response expected)."""
if not self.process or not self.process.stdin:
return
notification = {
"jsonrpc": "2.0",
"method": method,
"params": params,
}
try:
message = json.dumps(notification)
header = f"Content-Length: {len(message.encode('utf-8'))}\r\n\r\n"
self.process.stdin.write(header.encode("utf-8"))
self.process.stdin.write(message.encode("utf-8"))
self.process.stdin.flush()
except Exception as e:
log_safe_exception(logger, "MCP notification failed", e, include_debug=is_debug_enabled())
def _read_content_length(self, deadline: float) -> int | None:
"""Read Content-Length header, returning the byte length or None."""
if not self.process or not self.process.stdout:
return None
header_line = b""
while time.time() < deadline:
byte = self.process.stdout.read(1)
if not byte:
return None
header_line += byte
if header_line.endswith(b"\r\n\r\n") or header_line.endswith(b"\n\n"):
break
if not header_line:
return None
header_str = header_line.decode("utf-8").strip()
for line in header_str.split("\r\n"):
if line.lower().startswith("content-length:"):
try:
return int(line.split(":", 1)[1].strip())
except (ValueError, IndexError):
return None
return None
def _read_body(self, content_length: int, deadline: float) -> bytes | None:
"""Read a response body of the expected length before deadline."""
if not self.process or not self.process.stdout:
return None
remaining = content_length
chunks: list[bytes] = []
while remaining > 0 and time.time() < deadline:
chunk = self.process.stdout.read(remaining)
if not chunk:
return None
chunks.append(chunk)
remaining -= len(chunk)
if remaining > 0:
return None
return b"".join(chunks)
def _read_response(self, timeout: float = MCP_READ_TIMEOUT_SECONDS) -> dict | None:
"""Read a JSON-RPC response from the MCP server."""
if not self.process or not self.process.stdout:
return None
try:
deadline = time.time() + timeout
while time.time() < deadline:
content_length = self._read_content_length(deadline)
if content_length is None:
continue
body = self._read_body(content_length, deadline)
if not body:
return None
message = json.loads(body.decode("utf-8"))
# Skip notifications (no id), return responses
if "id" in message:
return message
return None
except Exception as e:
log_safe_exception(logger, "Error reading MCP response", e, include_debug=is_debug_enabled())
return None
class MCPToolCLI:
"""
CLI wrapper that exposes MCP tools as simple command-line calls.
Used by the bash wrapper scripts inside Docker containers.
Usage from bash:
python -m bridge.mcp_bridge query '{"query": "authentication"}'
python -m bridge.mcp_bridge context '{"name": "validateUser"}'
"""
def __init__(self):
self.bridge = MCPBridge()
def run(self, tool_name: str, args_json: str = "{}") -> int:
"""Run a single tool call and print the result."""
try:
args = json.loads(args_json)
except json.JSONDecodeError:
# Try to parse as simple key=value pairs
args = self._parse_simple_args(args_json)
if not self.bridge.start():
print("ERROR: Failed to start GitNexus MCP bridge", file=sys.stderr)
return 1
try:
result = self.bridge.call_tool(tool_name, args)
if result:
print(result.get("text", ""))
return 0
else:
print("No results", file=sys.stderr)
return 1
finally:
self.bridge.stop()
@staticmethod
def _parse_simple_args(args_str: str) -> dict:
"""Parse 'key=value key2=value2' style arguments."""
args = {}
for part in args_str.split():
if "=" in part:
key, value = part.split("=", 1)
args[key] = value
return args
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: python -m bridge.mcp_bridge <tool_name> [args_json]", file=sys.stderr)
print("Tools: query, context, impact, cypher, list_repos, detect_changes, rename", file=sys.stderr)
sys.exit(1)
tool = sys.argv[1]
args_json = sys.argv[2] if len(sys.argv) > 2 else "{}"
cli = MCPToolCLI()
sys.exit(cli.run(tool, args_json))
+8
View File
@@ -0,0 +1,8 @@
# Claude Haiku 4.5 — fast, cheap, good baseline
# Via OpenRouter (set OPENROUTER_API_KEY in .env)
model:
model_name: 'openrouter/anthropic/claude-haiku-4.5'
cost_tracking: 'ignore_errors'
model_kwargs:
max_tokens: 8192
temperature: 0
+9
View File
@@ -0,0 +1,9 @@
# Claude Opus 4 — most capable, highest cost
# Via OpenRouter (set OPENROUTER_API_KEY in .env)
# To use Anthropic directly, change to: anthropic/claude-opus-4-20250514
model:
model_name: 'openrouter/anthropic/claude-opus-4'
cost_tracking: 'ignore_errors'
model_kwargs:
max_tokens: 16384
temperature: 0
+9
View File
@@ -0,0 +1,9 @@
# Claude Sonnet 4 — strong all-around model
# Via OpenRouter (set OPENROUTER_API_KEY in .env)
# To use Anthropic directly, change to: anthropic/claude-sonnet-4-20250514
model:
model_name: 'openrouter/anthropic/claude-sonnet-4'
cost_tracking: 'ignore_errors'
model_kwargs:
max_tokens: 16384
temperature: 0
+13
View File
@@ -0,0 +1,13 @@
model: deepseek-ai/deepseek-chat
provider: openrouter
cost:
input: 0.14 # per 1M tokens
output: 0.28 # per 1M tokens
# Native DeepSeek API (direct)
api_key: null
base_url: null
# For OpenRouter, uncomment below and comment out direct config above
# api_key: \${OPENROUTER_API_KEY}
# base_url: https://openrouter.ai/api/v1
+15
View File
@@ -0,0 +1,15 @@
model: deepseek-ai/DeepSeek-V3
provider: openrouter
cost:
input: 0.27 # per 1M tokens
output: 1.10 # per 1M tokens
# Native DeepSeek API (direct)
# Get your API key at: https://platform.deepseek.com/
# Or use OpenRouter with: OPENROUTER_API_KEY
api_key: null
base_url: null
# For OpenRouter, uncomment below and comment out direct config above
# api_key: \${OPENROUTER_API_KEY}
# base_url: https://openrouter.ai/api/v1
+7
View File
@@ -0,0 +1,7 @@
# GLM 4.7 — via OpenRouter (set OPENROUTER_API_KEY in .env)
model:
model_name: 'openrouter/zhipuai/glm-4.7'
cost_tracking: 'ignore_errors'
model_kwargs:
max_tokens: 8192
temperature: 0
+7
View File
@@ -0,0 +1,7 @@
# GLM 5 — via OpenRouter (set OPENROUTER_API_KEY in .env)
model:
model_name: 'openrouter/zhipuai/glm-5'
cost_tracking: 'ignore_errors'
model_kwargs:
max_tokens: 8192
temperature: 0
+7
View File
@@ -0,0 +1,7 @@
# MiniMax M1 2.5 — via OpenRouter (set OPENROUTER_API_KEY in .env)
model:
model_name: 'openrouter/minimax/minimax-m1-2.5'
cost_tracking: 'ignore_errors'
model_kwargs:
max_tokens: 8192
temperature: 0

Some files were not shown because too many files have changed in this diff Show More