fix(swift): preprocess indented conditional directives so class bodies survive parsing (#2771)
* fix(swift): preprocess indented conditional directives so class bodies survive parsing * fix(swift): make conditional-directive blanking comment-, string- and brace-aware (#2771) Addresses the review findings on PR #2771. The transform fired unconditionally, which turned valid Swift into parse errors while missing the most common shape it was written for. - The blank/keep decision now consults `blockCommentDepth`, so ` #endif */` — the result of commenting out a conditional block — keeps its comment terminator. Previously `hasError` went raw=false -> preprocessed=true and the rest of the file was swallowed. - The decision keys on the scanner's brace depth instead of indentation. A column-0 `#if` inside a class body is blanked (6 of 7 body shapes previously still lost the enclosing declaration) and an indented file-scope directive is not — matching what the doc comment already claimed. Bare-CR line endings, NBSP/ideographic indentation and a leading BOM are recognized too. - A group is blanked only when every branch is brace-balanced. An `#if`/`#else` that splits a declaration header leaves one unmatched `{` once both branches survive, which collapsed five top-level nodes into one and gave unrelated types fabricated `NetworkClient.` qualified names. Such a group now degrades to the pre-fix behavior. - Multiline strings honour `\"""` escapes, and a plain `"""` closes even when a `#` follows it, so the scanner no longer wedges in string state and silently stops blanking for the rest of the file. - The pound run is counted once per position and skipped. It was quadratic: 10.6s for one 64k-`#` line, well inside the 512 KB walker limit. - Extended regex literals (`#/.../#`) no longer open a phantom block comment. - Directive-free files return early, matching `stripUeMacros`. Worker parity: `emitSwiftScopeCaptures` and `emitCppScopeCaptures` re-apply their provider's `preprocessSource` on the parse-cache-miss path — Dart already did this — and the embedding parse in `ensureAndParse` applies the hook as well. Before this the worker and the scope-capture/embedding halves analyzed different programs, turning a consistent degradation into cold-run/warm-run non-determinism. A new parity test pins the equivalence for every provider that defines the hook. SCHEMA_BUMP 37 -> 38: this changes parse semantics, the chunk key hashes raw on-disk bytes, and `preprocessSource` runs after the key is computed — so a same-package-version warm cache would replay pre-fix Swift results verbatim, including across `--force`. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * refactor(ingestion): apply preprocessSource once in the scope bridge (#2771) Follow-up cleanup on the review fixes. The previous commit re-applied each provider's `preprocessSource` inside `emitSwiftScopeCaptures` and `emitCppScopeCaptures`, mirroring what Dart already did — three copies of the same rule, and a contract that asked every future emitter to remember it. `extractParsedFile` is the single funnel every `emitScopeCaptures` caller passes through (parse worker, scope-resolution run, Vue script extraction), and it already receives the provider. Applying the hook there on the cache-miss path covers all three languages and every future one, names no language in shared code, and drops Dart's unconditional transform on the cache-hit path. Verified the three emitters use `sourceText` for nothing but the parse, so the substitution is output-identical — which the parity test asserts directly. Also from the cleanup pass: - the parity test derives its language list from the provider registry, so a new provider adopting the hook fails until it adds a fixture - `ensureAndParse` resolves the provider from the language it already computed, instead of a second extension table (`getProviderForFile`) - the preprocessor returns `sourceText` unchanged when no group was blanked, which is the common case for files whose only directives are top-level - `split(/(\r\n|\n|\r)/)` replaces the hand-rolled line splitter, and the per-group brace bookkeeping is two scalars instead of an array - the hint regex is derived from the line regex so the two cannot drift - unit assertions compare the WHOLE preprocessed file against the expected blanking, replacing per-line spot checks; the pipeline tests share one `runFixture` helper and `getNodesForFile` in the resolver test helpers - `LanguageProvider.preprocessSource` documents the real call sites and says plainly that the set is not closed — `populateRangeBindings` still hands language helpers raw text Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * chore(autofix): apply prettier + eslint fixes via /autofix command --------- Co-authored-by: Gergő Magyar <gergomagyar@icloud.com> Co-authored-by: Gergo Magyar <gergomagyar0@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
This commit is contained in:
co-authored by
Claude Opus 5
Gergő Magyar
Gergo Magyar
github-actions[bot]
parent
d268f351d3
commit
639eb04b31
@@ -11,6 +11,7 @@ import {
|
||||
resolveLanguageKey,
|
||||
} from '../tree-sitter/parser-loader.js';
|
||||
import { parseSourceSafe } from '../tree-sitter/safe-parse.js';
|
||||
import { getProvider } from '../ingestion/languages/index.js';
|
||||
|
||||
const parserCache = new Map<string, any>();
|
||||
|
||||
@@ -30,7 +31,15 @@ export const ensureAndParse = async (content: string, filePath: string): Promise
|
||||
parserCache.set(parserKey, parserInstance);
|
||||
}
|
||||
|
||||
return parseSourceSafe(parserInstance, content);
|
||||
// Same text the ingestion worker parses — otherwise a provider whose
|
||||
// `preprocessSource` repairs a declaration (Swift conditional directives,
|
||||
// C++ UE macros, Dart extension types) would leave embeddings looking at an
|
||||
// error-recovered tree. Resolved from `language` so the transform and the
|
||||
// parser always come from the same provider. Length-preserving, so node
|
||||
// offsets still index `content`.
|
||||
const parseContent = getProvider(language).preprocessSource?.(content, filePath) ?? content;
|
||||
|
||||
return parseSourceSafe(parserInstance, parseContent);
|
||||
};
|
||||
|
||||
const FUNCTION_LIKE_TYPES = new Set([
|
||||
|
||||
@@ -114,9 +114,17 @@ interface LanguageProviderConfig {
|
||||
* The current C++ UE-macro preprocessor relies on the practical fact that
|
||||
* UE reflection macros and module-export tokens are ASCII-only.
|
||||
*
|
||||
* Must be a pure function — same input always yields the same output. Called
|
||||
* once per file, on every code path that re-parses (parsing-processor, import
|
||||
* processor, heritage processor, call processor, parse worker).
|
||||
* Must be a pure function — same input always yields the same output, and
|
||||
* re-applying it to its own output changes nothing.
|
||||
*
|
||||
* Applied by the parse worker (`parse-worker.ts`), by `extractParsedFile`
|
||||
* (`scope-extractor-bridge.ts`) on the parse-cache-miss path, and by the
|
||||
* embedding parse (`embeddings/ast-utils.ts`, which does not go through the
|
||||
* bridge). Any *new* path that re-parses a file must apply it too, or the two
|
||||
* halves of the pipeline analyze different programs — and note the set is not
|
||||
* closed today: language-owned re-parse helpers reached through other
|
||||
* provider hooks (e.g. `populateRangeBindings`) still see raw text.
|
||||
* `test/unit/preprocess-source-parity.test.ts` pins the bridge equivalence.
|
||||
*
|
||||
* Default: undefined (no preprocessing — `file.content` is parsed verbatim).
|
||||
*/
|
||||
|
||||
@@ -8,6 +8,7 @@ import {
|
||||
import { getCppParser, getCppScopeQuery } from './query.js';
|
||||
import { getTreeSitterBufferSize } from '../../constants.js';
|
||||
import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js';
|
||||
import { stripUeMacros } from '../../cpp-ue-preprocessor.js';
|
||||
import { normalizeQualifiedName } from '../../utils/qualified-name.js';
|
||||
import { splitCppInclude, splitCppUsingDecl } from './import-decomposer.js';
|
||||
import {
|
||||
@@ -159,8 +160,12 @@ export function emitCppScopeCaptures(
|
||||
): readonly CaptureMatch[] {
|
||||
let tree = cachedTree as ReturnType<ReturnType<typeof getCppParser>['parse']> | undefined;
|
||||
if (tree === undefined) {
|
||||
tree = parseSourceSafe(getCppParser(), sourceText, undefined, {
|
||||
bufferSize: getTreeSitterBufferSize(sourceText),
|
||||
// Idempotent re-application: `extractParsedFile` already preprocesses, but
|
||||
// direct emitter callers (benchmarks, capture goldens) must see the same
|
||||
// program the pipeline does.
|
||||
const parseText = stripUeMacros(sourceText);
|
||||
tree = parseSourceSafe(getCppParser(), parseText, undefined, {
|
||||
bufferSize: getTreeSitterBufferSize(parseText),
|
||||
});
|
||||
}
|
||||
|
||||
|
||||
@@ -39,13 +39,13 @@ import { computeDartArityMetadata } from './arity-metadata.js';
|
||||
import { synthesizeDartReceiverBinding } from './receiver-binding.js';
|
||||
import { synthesizeDartSignatureBindings } from './signature-bindings.js';
|
||||
import { getDartParser, getDartScopeQuery } from './query.js';
|
||||
import { preprocessDartExtensionTypes } from './extension-type-preprocess.js';
|
||||
import { recordCacheHit, recordCacheMiss } from './cache-stats.js';
|
||||
import { getTreeSitterBufferSize } from '../../constants.js';
|
||||
import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js';
|
||||
import { encodeMarker } from '../../utils/heritage-marker.js';
|
||||
import { DART_BUILT_INS } from './built-ins.js';
|
||||
import { synthesizeCallableFlowCaptures } from '../../utils/callable-flow-captures.js';
|
||||
import { preprocessDartExtensionTypes } from './extension-type-preprocess.js';
|
||||
import { synthesizeReceiverChainCapture } from '../../utils/receiver-chain-captures.js';
|
||||
|
||||
const FUNCTION_DECL_TAGS = [
|
||||
@@ -113,6 +113,9 @@ export function emitDartScopeCaptures(
|
||||
_filePath: string,
|
||||
cachedTree?: unknown,
|
||||
): readonly CaptureMatch[] {
|
||||
// Idempotent re-application: `extractParsedFile` already preprocesses, but
|
||||
// direct emitter callers (benchmarks, capture goldens) must see the same
|
||||
// program the pipeline does.
|
||||
const parseText = preprocessDartExtensionTypes(sourceText);
|
||||
let tree: Parser.Tree;
|
||||
if (cachedTree !== undefined && cachedTree !== null) {
|
||||
|
||||
@@ -37,6 +37,7 @@ import {
|
||||
swiftMergeBindings,
|
||||
swiftArityCompatibility,
|
||||
} from './swift/index.js';
|
||||
import { preprocessSwiftConditionalDirectives } from './swift/conditional-directive-preprocess.js';
|
||||
|
||||
/** Swift init/deinit declarations have special names and Constructor label. */
|
||||
const swiftExtractFunctionName = (
|
||||
@@ -178,6 +179,7 @@ const BUILT_INS: ReadonlySet<string> = new Set([
|
||||
export const swiftProvider = defineLanguage({
|
||||
id: SupportedLanguages.Swift,
|
||||
extensions: ['.swift'],
|
||||
preprocessSource: preprocessSwiftConditionalDirectives,
|
||||
entryPointPatterns: [
|
||||
/^viewDidLoad$/,
|
||||
/^viewWillAppear$/,
|
||||
|
||||
@@ -46,6 +46,7 @@ import { computeSwiftArityMetadata } from './arity-metadata.js';
|
||||
import { synthesizeSwiftReceiverBinding } from './receiver-binding.js';
|
||||
import { synthesizeSwiftSignatureBindings } from './signature-bindings.js';
|
||||
import { getSwiftParser, getSwiftScopeQuery } from './query.js';
|
||||
import { preprocessSwiftConditionalDirectives } from './conditional-directive-preprocess.js';
|
||||
import { recordCacheHit, recordCacheMiss } from './cache-stats.js';
|
||||
import { getTreeSitterBufferSize } from '../../constants.js';
|
||||
import { parseSourceSafe } from '../../../tree-sitter/safe-parse.js';
|
||||
@@ -89,10 +90,16 @@ export function emitSwiftScopeCaptures(
|
||||
cachedTree?: unknown,
|
||||
): readonly CaptureMatch[] {
|
||||
// Reuse the parse phase's cached Tree when available; otherwise parse.
|
||||
// `extractParsedFile` already applies `preprocessSource` on this path, but
|
||||
// this emitter is also called directly (benchmarks, capture goldens, the
|
||||
// scope-capture tripwire), and those callers must see the same program the
|
||||
// pipeline does. The transform is idempotent, so applying it twice is a
|
||||
// no-op; it is length-preserving, so offsets still index `sourceText`.
|
||||
let tree = cachedTree as ReturnType<ReturnType<typeof getSwiftParser>['parse']> | undefined;
|
||||
if (tree === undefined) {
|
||||
tree = parseSourceSafe(getSwiftParser(), sourceText, undefined, {
|
||||
bufferSize: getTreeSitterBufferSize(sourceText),
|
||||
const parseText = preprocessSwiftConditionalDirectives(sourceText);
|
||||
tree = parseSourceSafe(getSwiftParser(), parseText, undefined, {
|
||||
bufferSize: getTreeSitterBufferSize(parseText),
|
||||
});
|
||||
recordCacheMiss();
|
||||
} else {
|
||||
|
||||
@@ -0,0 +1,283 @@
|
||||
/**
|
||||
* A conditional-compilation directive occupying a whole line.
|
||||
*
|
||||
* `[^\S\r\n]` (horizontal whitespace) rather than `[ \t]` so NBSP /
|
||||
* ideographic-space indentation and a leading BOM are recognized too. Leading
|
||||
* whitespace is *not* required: nesting is decided from the scanner's brace
|
||||
* depth, not from indentation (a column-0 `#if` inside a class body is exactly
|
||||
* the shape that loses its enclosing declaration).
|
||||
*/
|
||||
const SWIFT_CONDITIONAL_DIRECTIVE_RE = /^[^\S\r\n]*#(if|elseif|else|endif)\b[^\r\n]*$/;
|
||||
|
||||
/**
|
||||
* Cheap whole-file precondition, mirroring `stripUeMacros`'s `HAS_UE_HINT`.
|
||||
* Derived from the line pattern so the two can never drift apart.
|
||||
*/
|
||||
const HAS_CONDITIONAL_DIRECTIVE_HINT = new RegExp(SWIFT_CONDITIONAL_DIRECTIVE_RE.source, 'm');
|
||||
|
||||
interface SwiftPreprocessScanState {
|
||||
blockCommentDepth: number;
|
||||
multilineStringPounds: number | null;
|
||||
/** Net `{` minus `}` seen in code position. May go negative on broken source. */
|
||||
braceDepth: number;
|
||||
}
|
||||
|
||||
/**
|
||||
* An `#if` … `#endif` run. Blanking is decided per group, so a group is either
|
||||
* fully blanked or fully preserved — never half-erased.
|
||||
*/
|
||||
interface DirectiveGroup {
|
||||
braceDepthAtStart: number;
|
||||
/** Indices into the alternating `parts` array of this group's directive lines. */
|
||||
directiveLines: number[];
|
||||
/** Net brace delta of every branch so far — what the parser sees once all survive. */
|
||||
totalDelta: number;
|
||||
/** False as soon as one branch is not brace-balanced on its own. */
|
||||
allBranchesBalanced: boolean;
|
||||
currentBranchDelta: number;
|
||||
}
|
||||
|
||||
function hasTripleQuoteAt(line: string, index: number): boolean {
|
||||
return line.startsWith('"""', index);
|
||||
}
|
||||
|
||||
/**
|
||||
* Length of the multiline terminator at `index`, or 0.
|
||||
*
|
||||
* A plain `"""` string always closes at `"""`, whatever follows it — an
|
||||
* adjacent `#` is the next token, not part of the delimiter. A raw `#"""`
|
||||
* string closes at `"""` followed by *at least* its own pound count.
|
||||
*/
|
||||
function matchingMultilineCloseLength(line: string, index: number, poundCount: number): number {
|
||||
if (!hasTripleQuoteAt(line, index)) return 0;
|
||||
if (poundCount === 0) return 3;
|
||||
return line.startsWith('#'.repeat(poundCount), index + 3) ? 3 + poundCount : 0;
|
||||
}
|
||||
|
||||
function skipRegularString(line: string, startIndex: number, rawPoundCount: number): number {
|
||||
const endPounds = '#'.repeat(rawPoundCount);
|
||||
let index = startIndex + (rawPoundCount > 0 ? rawPoundCount + 1 : 1);
|
||||
|
||||
while (index < line.length) {
|
||||
if (line[index] === '"') {
|
||||
if (rawPoundCount === 0) return index + 1;
|
||||
if (line.startsWith(endPounds, index + 1)) return index + 1 + rawPoundCount;
|
||||
}
|
||||
if (rawPoundCount === 0 && line[index] === '\\') index++;
|
||||
index++;
|
||||
}
|
||||
|
||||
return line.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* Skip an extended regex literal (`#/…/#`), whose body may legally contain
|
||||
* `/*` — which would otherwise open a block comment that never closes.
|
||||
* `slashIndex` points at the `/` that follows the opening pound run.
|
||||
*/
|
||||
function skipExtendedRegexLiteral(line: string, slashIndex: number, poundCount: number): number {
|
||||
const closePounds = '#'.repeat(poundCount);
|
||||
let index = slashIndex + 1;
|
||||
|
||||
while (index < line.length) {
|
||||
if (line[index] === '\\') {
|
||||
index += 2;
|
||||
continue;
|
||||
}
|
||||
if (line[index] === '/' && line.startsWith(closePounds, index + 1)) {
|
||||
return index + 1 + poundCount;
|
||||
}
|
||||
index++;
|
||||
}
|
||||
|
||||
return line.length;
|
||||
}
|
||||
|
||||
function scanSwiftLine(line: string, state: SwiftPreprocessScanState): void {
|
||||
let index = 0;
|
||||
|
||||
while (index < line.length) {
|
||||
if (state.blockCommentDepth > 0) {
|
||||
if (line.startsWith('/*', index)) {
|
||||
state.blockCommentDepth++;
|
||||
index += 2;
|
||||
} else if (line.startsWith('*/', index)) {
|
||||
state.blockCommentDepth--;
|
||||
index += 2;
|
||||
} else {
|
||||
index++;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (state.multilineStringPounds !== null) {
|
||||
const closeLength = matchingMultilineCloseLength(line, index, state.multilineStringPounds);
|
||||
if (closeLength > 0) {
|
||||
state.multilineStringPounds = null;
|
||||
index += closeLength;
|
||||
continue;
|
||||
}
|
||||
// Non-raw multiline strings honour backslash escapes, so `\"""` is string
|
||||
// data and not a terminator. Raw strings escape with `\#`, so a bare
|
||||
// backslash there is literal.
|
||||
index += state.multilineStringPounds === 0 && line[index] === '\\' ? 2 : 1;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (line.startsWith('//', index)) return;
|
||||
if (line.startsWith('/*', index)) {
|
||||
state.blockCommentDepth = 1;
|
||||
index += 2;
|
||||
continue;
|
||||
}
|
||||
|
||||
if (line[index] === '#') {
|
||||
// Count the pound run exactly once and always advance past it, so a long
|
||||
// run of bare `#` stays linear instead of being re-walked per character.
|
||||
let poundCount = 1;
|
||||
while (line[index + poundCount] === '#') poundCount++;
|
||||
const afterPounds = index + poundCount;
|
||||
|
||||
if (hasTripleQuoteAt(line, afterPounds)) {
|
||||
state.multilineStringPounds = poundCount;
|
||||
index = afterPounds + 3;
|
||||
} else if (line[afterPounds] === '"') {
|
||||
index = skipRegularString(line, index, poundCount);
|
||||
} else if (line[afterPounds] === '/') {
|
||||
index = skipExtendedRegexLiteral(line, afterPounds, poundCount);
|
||||
} else {
|
||||
index = afterPounds;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
|
||||
if (hasTripleQuoteAt(line, index)) {
|
||||
state.multilineStringPounds = 0;
|
||||
index += 3;
|
||||
continue;
|
||||
}
|
||||
if (line[index] === '"') {
|
||||
index = skipRegularString(line, index, 0);
|
||||
continue;
|
||||
}
|
||||
|
||||
if (line[index] === '{') state.braceDepth++;
|
||||
else if (line[index] === '}') state.braceDepth--;
|
||||
index++;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Blank nested Swift conditional-compilation directives before parsing.
|
||||
*
|
||||
* tree-sitter-swift 0.7.1 does not admit `directive` as a `class_body` child
|
||||
* (`vendor/tree-sitter-swift/src/node-types.json`), so error recovery can
|
||||
* discard the enclosing declaration. Replacing only the directive text with
|
||||
* spaces preserves `.length`, line endings, and every declaration offset.
|
||||
*
|
||||
* ponytail: delete this file (and the `preprocessSource` wiring in `swift.ts`)
|
||||
* once a vendored tree-sitter-swift release includes upstream PR #583 ("Allow
|
||||
* #if/#elseif/#else/#endif directives inside type bodies"); see also upstream
|
||||
* issue #298 and PR #599. `.github/vendored-grammars.json` has no `"hold"` on
|
||||
* swift, so the bump bot will land that release on its own.
|
||||
*
|
||||
* A directive group is blanked only when **all** of these hold:
|
||||
*
|
||||
* - it opens at `braceDepth > 0` — nesting, not indentation, is what the
|
||||
* grammar rejects; top-level directives are valid source-file members and
|
||||
* are left intact
|
||||
* - it is not inside a multiline string literal or a block comment — blanking
|
||||
* a line that carries a block-comment terminator would un-terminate the
|
||||
* comment and swallow the rest of the file, and blanking inside a literal
|
||||
* would rewrite program data
|
||||
* - every branch is brace-balanced. A group that splits a declaration header
|
||||
* (`#if` … `func f() async {` … `#else` … `func f() {` … `#endif`) leaves
|
||||
* one unmatched `{` once both branches survive, which re-parents every
|
||||
* later top-level declaration. Such a group degrades to the pre-fix
|
||||
* behavior instead.
|
||||
*
|
||||
* Known residuals, all of which degrade to "directive left in place" rather
|
||||
* than to corrupted output: string interpolation is not parsed with full Swift
|
||||
* expression fidelity, so a nested multiline string inside an interpolation can
|
||||
* confuse the scanner; a bare `/…/` regex literal containing `/*` opens a
|
||||
* phantom block comment; and a multiline extended regex literal is treated as
|
||||
* ending at its first line.
|
||||
*
|
||||
* Byte length is *not* preserved when a blanked directive line carries
|
||||
* non-ASCII trailing text (` #if os(iOS) // 日本語 🔥` is 23 UTF-16 code units
|
||||
* either way, but shrinks from 31 UTF-8 bytes to 23). C++ sidesteps this
|
||||
* because UE macro tokens are ASCII-only; a Swift directive line can carry any
|
||||
* trailing comment. Per the `LanguageProvider.preprocessSource` contract this is
|
||||
* safe only while no consumer slices the original UTF-8 bytes by `startIndex`:
|
||||
* node-tree-sitter reports UTF-16 code-unit indices, and every in-process
|
||||
* consumer slices the JS string, whose `.length` *is* preserved.
|
||||
*
|
||||
* Must stay pure and idempotent — see `LanguageProvider.preprocessSource`.
|
||||
*/
|
||||
export function preprocessSwiftConditionalDirectives(sourceText: string): string {
|
||||
if (!HAS_CONDITIONAL_DIRECTIVE_HINT.test(sourceText)) return sourceText;
|
||||
|
||||
// Alternating [line, terminator, line, terminator, …, line]; joining it back
|
||||
// reproduces the input byte for byte, including bare-`\r` endings.
|
||||
const parts = sourceText.split(/(\r\n|\n|\r)/);
|
||||
const state: SwiftPreprocessScanState = {
|
||||
blockCommentDepth: 0,
|
||||
multilineStringPounds: null,
|
||||
braceDepth: 0,
|
||||
};
|
||||
const openGroups: DirectiveGroup[] = [];
|
||||
let blankedAny = false;
|
||||
|
||||
for (let index = 0; index < parts.length; index += 2) {
|
||||
const text = parts[index]!;
|
||||
const insideLiteralOrComment =
|
||||
state.multilineStringPounds !== null || state.blockCommentDepth > 0;
|
||||
const match = insideLiteralOrComment ? null : SWIFT_CONDITIONAL_DIRECTIVE_RE.exec(text);
|
||||
|
||||
const braceDepthAtStart = state.braceDepth;
|
||||
scanSwiftLine(text, state);
|
||||
const braceDelta = state.braceDepth - braceDepthAtStart;
|
||||
|
||||
const openGroup = openGroups[openGroups.length - 1];
|
||||
if (match === null) {
|
||||
if (openGroup !== undefined) openGroup.currentBranchDelta += braceDelta;
|
||||
continue;
|
||||
}
|
||||
|
||||
// Directive lines are blanked as a unit, so their own braces (if any) never
|
||||
// reach the parser and must not count toward a branch's balance.
|
||||
if (match[1] === 'if') {
|
||||
openGroups.push({
|
||||
braceDepthAtStart,
|
||||
directiveLines: [index],
|
||||
totalDelta: 0,
|
||||
allBranchesBalanced: true,
|
||||
currentBranchDelta: 0,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
// A `#else`/`#endif` with no open `#if` is malformed source — leave it be.
|
||||
if (openGroup === undefined) continue;
|
||||
|
||||
openGroup.directiveLines.push(index);
|
||||
openGroup.totalDelta += openGroup.currentBranchDelta;
|
||||
openGroup.allBranchesBalanced &&= openGroup.currentBranchDelta === 0;
|
||||
openGroup.currentBranchDelta = 0;
|
||||
if (match[1] !== 'endif') continue;
|
||||
|
||||
openGroups.pop();
|
||||
const parentGroup = openGroups[openGroups.length - 1];
|
||||
// Every branch survives preprocessing, so the enclosing branch sees the sum
|
||||
// — not one branch's delta.
|
||||
if (parentGroup !== undefined) parentGroup.currentBranchDelta += openGroup.totalDelta;
|
||||
|
||||
if (openGroup.braceDepthAtStart > 0 && openGroup.allBranchesBalanced) {
|
||||
// Safe to mutate in place: every line of this group has already been
|
||||
// scanned, and the decision never reopens.
|
||||
for (const line of openGroup.directiveLines) parts[line] = ' '.repeat(parts[line]!.length);
|
||||
blankedAny = true;
|
||||
}
|
||||
}
|
||||
|
||||
return blankedAny ? parts.join('') : sourceText;
|
||||
}
|
||||
@@ -49,7 +49,16 @@ export function extractParsedFile(
|
||||
if (provider.emitScopeCaptures === undefined) return undefined;
|
||||
if (sourceText.trim().length === 0) return undefined;
|
||||
try {
|
||||
const captures = provider.emitScopeCaptures(sourceText, filePath, cachedTree, { sourceKind });
|
||||
// A provider that rewrites source before parsing must see the same text
|
||||
// here that the parse worker fed tree-sitter, or the two halves of the
|
||||
// pipeline analyze different programs. Only the cache-miss path re-parses;
|
||||
// with a cached tree the emitter ignores the text. The transform is
|
||||
// length-preserving, so every offset still indexes the original.
|
||||
const parseText =
|
||||
cachedTree === undefined
|
||||
? (provider.preprocessSource?.(sourceText, filePath) ?? sourceText)
|
||||
: sourceText;
|
||||
const captures = provider.emitScopeCaptures(parseText, filePath, cachedTree, { sourceKind });
|
||||
return extractScope(captures, filePath, provider);
|
||||
} catch (err) {
|
||||
const message = `scope extraction failed for ${filePath}: ${
|
||||
|
||||
@@ -149,7 +149,12 @@ import type { ParseWorkerResult } from '../core/ingestion/workers/parse-worker.j
|
||||
// v37: Java/Kotlin capture side-channels include Spring AOP owner/advice facts
|
||||
// (#2416). Warm cache entries at v36 do not carry those facts and would silently
|
||||
// omit ADVISED_BY evidence.
|
||||
const SCHEMA_BUMP = 37;
|
||||
// v38: Swift nested conditional-compilation directives are blanked before the
|
||||
// parse (#2771), so a class body that previously error-recovered away now
|
||||
// survives. The chunk key hashes raw on-disk bytes and `preprocessSource` runs
|
||||
// after it is computed, so unchanged Swift files would otherwise replay their
|
||||
// pre-fix `ParseWorkerResult` verbatim — including across `--force`.
|
||||
const SCHEMA_BUMP = 38;
|
||||
const GITNEXUS_PKG_VERSION = (() => {
|
||||
try {
|
||||
// package.json sits at gitnexus/package.json — two levels up from
|
||||
|
||||
@@ -12,6 +12,8 @@ import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { runPipelineFromRepo } from './resolvers/helpers.js';
|
||||
import type { PipelineResult } from '../../src/types/pipeline.js';
|
||||
import { isLanguageAvailable } from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
|
||||
function createTsRepo(): string {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'doc-desc-e2e-'));
|
||||
@@ -32,6 +34,27 @@ function createTsRepo(): string {
|
||||
return dir;
|
||||
}
|
||||
|
||||
function createSwiftRepo(): string {
|
||||
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'doc-desc-swift-e2e-'));
|
||||
fs.writeFileSync(
|
||||
path.join(dir, 'Widget.swift'),
|
||||
[
|
||||
'class Widget {',
|
||||
' /**',
|
||||
' Renders the widget, marker SWIFTDOC.',
|
||||
' Example:',
|
||||
' #if os(iOS)',
|
||||
' useUIKit()',
|
||||
' #endif',
|
||||
' */',
|
||||
' func render() {}',
|
||||
'}',
|
||||
'',
|
||||
].join('\n'),
|
||||
);
|
||||
return dir;
|
||||
}
|
||||
|
||||
describe('doc-comment description end-to-end (issue #2270)', () => {
|
||||
it('surfaces an exported function JSDoc as its node description through the pipeline', async () => {
|
||||
const result: PipelineResult = await runPipelineFromRepo(createTsRepo(), () => {}, {
|
||||
@@ -48,4 +71,26 @@ describe('doc-comment description end-to-end (issue #2270)', () => {
|
||||
expect(result.usedWorkerPool).toBe(true);
|
||||
expect(descriptions.get('Function:computeBalance')).toContain('EXPORTEDDOC');
|
||||
});
|
||||
|
||||
// Swift preprocessing blanks nested conditional directives before parsing;
|
||||
// doing that inside a doc comment would delete lines from `description` (#2771).
|
||||
it.skipIf(!isLanguageAvailable(SupportedLanguages.Swift))(
|
||||
'keeps directive lines inside a Swift doc comment in the node description',
|
||||
async () => {
|
||||
const result: PipelineResult = await runPipelineFromRepo(createSwiftRepo(), () => {}, {
|
||||
skipGraphPhases: true,
|
||||
workerThresholdsForTest: { minFiles: 1, minBytes: 1 },
|
||||
workerPoolSize: 2,
|
||||
});
|
||||
|
||||
const descriptions = new Map<string, unknown>();
|
||||
result.graph.forEachNode((node) => {
|
||||
descriptions.set(`${node.label}:${node.properties.name}`, node.properties.description);
|
||||
});
|
||||
|
||||
expect(descriptions.get('Function:render')).toBe(
|
||||
'Renders the widget, marker SWIFTDOC. Example: #if os(iOS) useUIKit() #endif',
|
||||
);
|
||||
},
|
||||
);
|
||||
});
|
||||
|
||||
@@ -95,6 +95,28 @@ export function getNodesByLabel(result: PipelineResult, label: string): string[]
|
||||
return names.sort();
|
||||
}
|
||||
|
||||
/**
|
||||
* Every node a single fixture file contributed, as sorted `Label:name` and
|
||||
* `Label:qualifiedName` strings — for exact `toEqual` assertions on the whole
|
||||
* file's output rather than spot checks.
|
||||
*/
|
||||
export function getNodesForFile(
|
||||
result: PipelineResult,
|
||||
filePathSuffix: string,
|
||||
): { names: string[]; labelled: string[]; qualified: string[] } {
|
||||
const names: string[] = [];
|
||||
const labelled: string[] = [];
|
||||
const qualified: string[] = [];
|
||||
result.graph.forEachNode((n) => {
|
||||
if (!String(n.properties.filePath ?? '').endsWith(filePathSuffix)) return;
|
||||
const name = n.properties.name;
|
||||
names.push(name);
|
||||
labelled.push(`${n.label}:${name}`);
|
||||
qualified.push(`${n.label}:${String(n.properties.qualifiedName ?? name)}`);
|
||||
});
|
||||
return { names: names.sort(), labelled: labelled.sort(), qualified: qualified.sort() };
|
||||
}
|
||||
|
||||
export function edgeSet(edges: Array<{ source: string; target: string }>): string[] {
|
||||
return edges.map((e) => `${e.source} → ${e.target}`).sort();
|
||||
}
|
||||
|
||||
@@ -0,0 +1,119 @@
|
||||
import { afterAll, describe, expect, it } from 'vitest';
|
||||
import fs from 'node:fs';
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { runPipelineFromRepo } from '../../src/core/ingestion/pipeline.js';
|
||||
import { isLanguageAvailable } from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { SupportedLanguages } from '../../src/config/supported-languages.js';
|
||||
import { getNodesForFile } from './resolvers/helpers.js';
|
||||
import { preprocessSwiftConditionalDirectives } from '../../src/core/ingestion/languages/swift/conditional-directive-preprocess.js';
|
||||
|
||||
const swiftFixture = `class Outer {
|
||||
enum A { case x }
|
||||
#if os(iOS)
|
||||
enum B { case y }
|
||||
#endif
|
||||
}
|
||||
`;
|
||||
|
||||
const swiftMultilineStringFixture = `class StringHolder {
|
||||
let payload = """
|
||||
#if string-data
|
||||
#elseif more-string-data
|
||||
#else
|
||||
#endif
|
||||
"""
|
||||
#if REAL_DIRECTIVE
|
||||
func afterString() {}
|
||||
#endif
|
||||
}
|
||||
`;
|
||||
|
||||
const swiftColumnZeroFixture = `class ColumnZero {
|
||||
enum A { case x }
|
||||
#if os(iOS)
|
||||
enum B { case y }
|
||||
#endif
|
||||
}
|
||||
`;
|
||||
|
||||
const swiftHeaderSplitFixture = `class NetworkClient {
|
||||
#if swift(>=5.5)
|
||||
func fetch() async {
|
||||
#else
|
||||
func fetch() {
|
||||
#endif
|
||||
perform()
|
||||
}
|
||||
}
|
||||
|
||||
struct SessionStore {}
|
||||
`;
|
||||
|
||||
const swiftAvailable = isLanguageAvailable(SupportedLanguages.Swift);
|
||||
const scratchDirs: string[] = [];
|
||||
|
||||
/** Analyze a one-file Swift repo through the real worker pool. */
|
||||
async function runFixture(prefix: string, source: string) {
|
||||
const repo = fs.mkdtempSync(path.join(os.tmpdir(), prefix));
|
||||
scratchDirs.push(repo);
|
||||
fs.writeFileSync(path.join(repo, 'Fixture.swift'), source, 'utf8');
|
||||
|
||||
const result = await runPipelineFromRepo(repo, () => {}, { workerPoolSize: 1 });
|
||||
return getNodesForFile(result, 'Fixture.swift');
|
||||
}
|
||||
|
||||
describe.skipIf(!swiftAvailable)('Swift conditional-directive pipeline regression', () => {
|
||||
afterAll(() => {
|
||||
for (const scratchDir of scratchDirs) fs.rmSync(scratchDir, { recursive: true, force: true });
|
||||
});
|
||||
|
||||
it('keeps Outer and both nested declarations in the real worker pipeline', async () => {
|
||||
const { names } = await runFixture('gitnexus-swift-directive-', swiftFixture);
|
||||
|
||||
expect(names).toEqual(['A', 'B', 'Fixture.swift', 'Outer', 'x', 'y']);
|
||||
}, 60000);
|
||||
|
||||
it('keeps a column-zero directive inside a class body from discarding the class', async () => {
|
||||
const { names } = await runFixture('gitnexus-swift-column-zero-', swiftColumnZeroFixture);
|
||||
|
||||
expect(names).toEqual(['A', 'B', 'ColumnZero', 'Fixture.swift', 'x', 'y']);
|
||||
}, 60000);
|
||||
|
||||
it('keeps later top-level types out of a class whose header is split across branches', async () => {
|
||||
// Blanking an unbalanced group re-parents unrelated declarations, which
|
||||
// shows up as a fabricated `NetworkClient.` qualified-name prefix.
|
||||
expect(preprocessSwiftConditionalDirectives(swiftHeaderSplitFixture)).toBe(
|
||||
swiftHeaderSplitFixture,
|
||||
);
|
||||
|
||||
const { qualified } = await runFixture('gitnexus-swift-header-split-', swiftHeaderSplitFixture);
|
||||
|
||||
expect(qualified).toEqual([
|
||||
'Class:NetworkClient',
|
||||
'File:Fixture.swift',
|
||||
'Function:fetch',
|
||||
'Function:fetch',
|
||||
'Struct:SessionStore',
|
||||
]);
|
||||
}, 60000);
|
||||
|
||||
it('preserves a multiline string property while blanking a real directive between strings', async () => {
|
||||
const rewritten = preprocessSwiftConditionalDirectives(swiftMultilineStringFixture);
|
||||
const opening = swiftMultilineStringFixture.indexOf('"""') + 3;
|
||||
const closing = swiftMultilineStringFixture.indexOf('"""', opening);
|
||||
expect([opening, closing]).toEqual([40, 105]);
|
||||
expect(rewritten.slice(opening, closing)).toBe(
|
||||
swiftMultilineStringFixture.slice(opening, closing),
|
||||
);
|
||||
|
||||
const { labelled } = await runFixture('gitnexus-swift-string-', swiftMultilineStringFixture);
|
||||
|
||||
expect(labelled).toEqual([
|
||||
'Class:StringHolder',
|
||||
'File:Fixture.swift',
|
||||
'Function:afterString',
|
||||
'Property:payload',
|
||||
]);
|
||||
}, 60000);
|
||||
});
|
||||
@@ -1,7 +1,11 @@
|
||||
import { describe, it, expect, beforeAll } from 'vitest';
|
||||
import fs from 'fs';
|
||||
import path from 'path';
|
||||
import { loadParser, loadLanguage } from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import {
|
||||
loadParser,
|
||||
loadLanguage,
|
||||
isLanguageAvailable,
|
||||
} from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { SupportedLanguages, getLanguageFromFilename } from 'gitnexus-shared';
|
||||
import { getProvider } from '../../src/core/ingestion/languages/index.js';
|
||||
import Parser from 'tree-sitter';
|
||||
@@ -493,6 +497,178 @@ describe('Tree-sitter multi-language parsing', () => {
|
||||
expect(defs.length).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
// Grammar-load failures are expected on the platform-sensitive matrix, so
|
||||
// skip rather than hard-fail — same guard the sibling tests apply inline.
|
||||
describe.skipIf(!isLanguageAvailable(SupportedLanguages.Swift))(
|
||||
'conditional-compilation directives',
|
||||
() => {
|
||||
const provider = getProvider(SupportedLanguages.Swift);
|
||||
const preprocess = (content: string): string =>
|
||||
provider.preprocessSource?.(content, 'Fixture.swift') ?? content;
|
||||
|
||||
beforeAll(async () => {});
|
||||
|
||||
it('captures a class whose body contains indented conditional directives after preprocessing', () => {
|
||||
const content = [
|
||||
'class Outer {',
|
||||
' enum A { case x }',
|
||||
' #if os(iOS)',
|
||||
' enum B { case y }',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
const { tree, matches } = parseAndQuery(
|
||||
parser,
|
||||
preprocess(content),
|
||||
provider.treeSitterQueries,
|
||||
);
|
||||
const defs = extractDefinitions(matches);
|
||||
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
expect(defs).toEqual([
|
||||
{ type: 'definition.class', name: 'Outer' },
|
||||
{ type: 'definition.enum', name: 'A' },
|
||||
{ type: 'definition.property', name: 'x' },
|
||||
{ type: 'definition.enum', name: 'B' },
|
||||
{ type: 'definition.property', name: 'y' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('captures a class whose body contains column-zero conditional directives', () => {
|
||||
const content = [
|
||||
'class Outer {',
|
||||
' enum A { case x }',
|
||||
'#if os(iOS)',
|
||||
' enum B { case y }',
|
||||
'#endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
const { tree, matches } = parseAndQuery(
|
||||
parser,
|
||||
preprocess(content),
|
||||
provider.treeSitterQueries,
|
||||
);
|
||||
const defs = extractDefinitions(matches);
|
||||
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
expect(defs).toEqual([
|
||||
{ type: 'definition.class', name: 'Outer' },
|
||||
{ type: 'definition.enum', name: 'A' },
|
||||
{ type: 'definition.property', name: 'x' },
|
||||
{ type: 'definition.enum', name: 'B' },
|
||||
{ type: 'definition.property', name: 'y' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('leaves top-level conditional directives intact while capturing their declarations', () => {
|
||||
const content = [
|
||||
'#if os(iOS)',
|
||||
'struct PlatformValue {',
|
||||
' let value: Int = 1',
|
||||
'}',
|
||||
'#else',
|
||||
'struct PlatformValue {',
|
||||
' let value: Int = 2',
|
||||
'}',
|
||||
'#endif',
|
||||
].join('\n');
|
||||
const parseContent = preprocess(content);
|
||||
const { tree, matches } = parseAndQuery(parser, parseContent, provider.treeSitterQueries);
|
||||
const defs = extractDefinitions(matches);
|
||||
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
expect(parseContent).toBe(content);
|
||||
expect(defs).toEqual([
|
||||
{ type: 'definition.struct', name: 'PlatformValue' },
|
||||
{ type: 'definition.property', name: 'value' },
|
||||
{ type: 'definition.struct', name: 'PlatformValue' },
|
||||
{ type: 'definition.property', name: 'value' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('keeps source that comments out a conditional block parseable', () => {
|
||||
const content = [
|
||||
'class Foo {',
|
||||
' /* temporarily disabled:',
|
||||
' #if DEBUG',
|
||||
' func f() {}',
|
||||
' #endif */',
|
||||
' func g() {}',
|
||||
'}',
|
||||
].join('\n');
|
||||
const parseContent = preprocess(content);
|
||||
const { tree, matches } = parseAndQuery(parser, parseContent, provider.treeSitterQueries);
|
||||
const defs = extractDefinitions(matches);
|
||||
|
||||
// Erasing the comment terminator would swallow `g()` and the rest.
|
||||
expect(parseContent).toBe(content);
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
expect(defs).toEqual([
|
||||
{ type: 'definition.class', name: 'Foo' },
|
||||
{ type: 'definition.function', name: 'g' },
|
||||
]);
|
||||
});
|
||||
|
||||
it('does not re-parent later declarations when branches split a declaration header', () => {
|
||||
const content = [
|
||||
'class NetworkClient {',
|
||||
' #if swift(>=5.5)',
|
||||
' func fetch() async {',
|
||||
' #else',
|
||||
' func fetch() {',
|
||||
' #endif',
|
||||
' perform()',
|
||||
' }',
|
||||
'}',
|
||||
'struct SessionStore {}',
|
||||
'enum Unrelated { case a }',
|
||||
].join('\n');
|
||||
const parseContent = preprocess(content);
|
||||
const { tree } = parseAndQuery(parser, parseContent, provider.treeSitterQueries);
|
||||
const topLevelTypes = tree.rootNode.namedChildren.map((child) => child.type);
|
||||
|
||||
// Blanking both markers would leave `NetworkClient` unterminated and
|
||||
// collapse every later top-level declaration into it (5 nodes -> 1).
|
||||
// `struct`/`enum` both surface as `class_declaration` in this grammar.
|
||||
expect(parseContent).toBe(content);
|
||||
expect(topLevelTypes).toEqual([
|
||||
'class_declaration',
|
||||
'directive',
|
||||
'function_declaration',
|
||||
'class_declaration',
|
||||
'class_declaration',
|
||||
]);
|
||||
});
|
||||
|
||||
it('keeps a declaration from every branch once the directives are blanked', () => {
|
||||
const content = [
|
||||
'class Themed {',
|
||||
' #if os(iOS)',
|
||||
' func accent(alpha: Int) {}',
|
||||
' #else',
|
||||
' func accent() {}',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
const { tree, matches } = parseAndQuery(
|
||||
parser,
|
||||
preprocess(content),
|
||||
provider.treeSitterQueries,
|
||||
);
|
||||
const defs = extractDefinitions(matches);
|
||||
|
||||
// Branch selection is not modelled: mutually exclusive declarations
|
||||
// both reach the graph. Pinned so changing it shows up as a diff.
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
expect(defs).toEqual([
|
||||
{ type: 'definition.class', name: 'Themed' },
|
||||
{ type: 'definition.function', name: 'accent' },
|
||||
{ type: 'definition.function', name: 'accent' },
|
||||
]);
|
||||
});
|
||||
},
|
||||
);
|
||||
|
||||
it('gracefully handles missing tree-sitter-swift', async () => {
|
||||
// If Swift is NOT available, loadLanguage should throw
|
||||
// If it IS available, this test just passes
|
||||
|
||||
@@ -21,7 +21,10 @@ vi.mock('../../src/core/tree-sitter/safe-parse.js', async () => {
|
||||
return buildSafeParseMock(parseSourceSafeSpy);
|
||||
});
|
||||
|
||||
vi.mock('gitnexus-shared', () => ({
|
||||
// Partial mock: `ast-utils` now resolves the LanguageProvider registry to apply
|
||||
// `preprocessSource`, and that graph needs the real shared exports (#2771).
|
||||
vi.mock('gitnexus-shared', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('gitnexus-shared')>()),
|
||||
getLanguageFromFilename,
|
||||
}));
|
||||
|
||||
|
||||
@@ -22,7 +22,10 @@ vi.mock('../../src/core/tree-sitter/parser-loader.js', () => ({
|
||||
),
|
||||
}));
|
||||
|
||||
vi.mock('gitnexus-shared', () => ({
|
||||
// Partial mock: `ast-utils` now resolves the LanguageProvider registry to apply
|
||||
// `preprocessSource`, and that graph needs the real shared exports (#2771).
|
||||
vi.mock('gitnexus-shared', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('gitnexus-shared')>()),
|
||||
getLanguageFromFilename,
|
||||
}));
|
||||
|
||||
|
||||
@@ -22,7 +22,10 @@ const { getLanguageFromFilename } = vi.hoisted(() => ({
|
||||
getLanguageFromFilename: vi.fn().mockReturnValue('typescript'),
|
||||
}));
|
||||
|
||||
vi.mock('gitnexus-shared', () => ({
|
||||
// Partial mock: `ast-utils` now resolves the LanguageProvider registry to apply
|
||||
// `preprocessSource`, and that graph needs the real shared exports (#2771).
|
||||
vi.mock('gitnexus-shared', async (importOriginal) => ({
|
||||
...(await importOriginal<typeof import('gitnexus-shared')>()),
|
||||
getLanguageFromFilename,
|
||||
}));
|
||||
|
||||
|
||||
@@ -101,9 +101,9 @@ describe('fileContentHash', () => {
|
||||
});
|
||||
|
||||
describe('PARSE_CACHE_VERSION', () => {
|
||||
// 36 -> 37 for Java/Kotlin Spring AOP capture side-channels (#2416).
|
||||
it('pins SCHEMA_BUMP to 37 so concurrent bumps cannot silently collide (#2416)', () => {
|
||||
expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(37);
|
||||
// 37 -> 38 for the Swift conditional-directive parse-semantics change (#2771).
|
||||
it('pins SCHEMA_BUMP to 38 so concurrent bumps cannot silently collide (#2771)', () => {
|
||||
expect(Number(PARSE_CACHE_VERSION.split('+', 1)[0])).toBe(38);
|
||||
});
|
||||
|
||||
it('embeds the gitnexus package version (so upgrades invalidate the cache)', () => {
|
||||
|
||||
@@ -0,0 +1,87 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { SupportedLanguages } from 'gitnexus-shared';
|
||||
import { providers, getProvider } from '../../src/core/ingestion/languages/index.js';
|
||||
import { extractParsedFile } from '../../src/core/ingestion/scope-extractor-bridge.js';
|
||||
import { isLanguageAvailable } from '../../src/core/tree-sitter/parser-loader.js';
|
||||
import { ensureAndParse } from '../../src/core/embeddings/ast-utils.js';
|
||||
|
||||
/**
|
||||
* Every provider that defines `preprocessSource` must produce the same
|
||||
* `ParsedFile` whether it is handed raw source or already-preprocessed source.
|
||||
*
|
||||
* The parse worker applies the hook, but `emitScopeCaptures` re-parses on a
|
||||
* parse-cache miss and the embedding pipeline parses independently — so unless
|
||||
* those paths see the same transform the halves of the pipeline analyze
|
||||
* different programs and the graph depends on whether the run was warm (#2771).
|
||||
*
|
||||
* Fixtures are keyed by language and cross-checked against the registry, so a
|
||||
* new provider adopting the hook fails here until it adds one.
|
||||
*/
|
||||
const FIXTURES: Partial<Record<SupportedLanguages, { filePath: string; source: string }>> = {
|
||||
[SupportedLanguages.Swift]: {
|
||||
filePath: 'Fixture.swift',
|
||||
source: [
|
||||
'class Outer {',
|
||||
' enum A { case x }',
|
||||
' #if os(iOS)',
|
||||
' enum B { case y }',
|
||||
' #endif',
|
||||
'}',
|
||||
'',
|
||||
].join('\n'),
|
||||
},
|
||||
[SupportedLanguages.CPlusPlus]: {
|
||||
filePath: 'Actor.cpp',
|
||||
source: [
|
||||
'UCLASS()',
|
||||
'class MYGAME_API AGameActor : public AActor {',
|
||||
' GENERATED_BODY()',
|
||||
'public:',
|
||||
' UPROPERTY(EditAnywhere) int Health;',
|
||||
' UFUNCTION(BlueprintCallable) void Tick(float DeltaTime) { Health = 1; }',
|
||||
'};',
|
||||
'',
|
||||
].join('\n'),
|
||||
},
|
||||
[SupportedLanguages.Dart]: {
|
||||
filePath: 'meters.dart',
|
||||
source: ['extension type Meters(int value) {', ' int get raw => value;', '}', ''].join('\n'),
|
||||
},
|
||||
};
|
||||
|
||||
const languagesWithHook = Object.entries(providers)
|
||||
.filter(([, provider]) => provider.preprocessSource !== undefined)
|
||||
.map(([language]) => language)
|
||||
.sort();
|
||||
|
||||
describe('LanguageProvider.preprocessSource parity', () => {
|
||||
it('has a fixture for every provider defining the hook', () => {
|
||||
expect(Object.keys(FIXTURES).sort()).toEqual(languagesWithHook);
|
||||
});
|
||||
|
||||
describe.each(languagesWithHook)('%s', (language) => {
|
||||
const provider = getProvider(language as SupportedLanguages);
|
||||
const { filePath, source } = FIXTURES[language as SupportedLanguages]!;
|
||||
|
||||
describe.skipIf(!isLanguageAvailable(language as SupportedLanguages))(
|
||||
'with the grammar',
|
||||
() => {
|
||||
it('extracts the same ParsedFile from raw and preprocessed source', () => {
|
||||
const preprocessed = provider.preprocessSource!(source, filePath);
|
||||
|
||||
expect(preprocessed).not.toBe(source);
|
||||
expect(preprocessed).toHaveLength(source.length);
|
||||
expect(extractParsedFile(provider, source, filePath, () => {})).toEqual(
|
||||
extractParsedFile(provider, preprocessed, filePath, () => {}),
|
||||
);
|
||||
});
|
||||
|
||||
it('parses the preprocessed text on the embedding path too', async () => {
|
||||
const tree = await ensureAndParse(source, filePath);
|
||||
|
||||
expect(tree.rootNode.hasError).toBe(false);
|
||||
});
|
||||
},
|
||||
);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,334 @@
|
||||
import { describe, expect, it } from 'vitest';
|
||||
import { preprocessSwiftConditionalDirectives } from '../../src/core/ingestion/languages/swift/conditional-directive-preprocess.js';
|
||||
|
||||
/**
|
||||
* Assert the WHOLE output: `source` with exactly `blankedLines` replaced by
|
||||
* spaces of the same width. Widths come from the source line, so a wrong-length
|
||||
* blank fails, and an unexpected blank anywhere else fails too.
|
||||
*/
|
||||
function expectBlanked(source: string, blankedLines: readonly number[], separator = '\n'): void {
|
||||
const lines = source.split(separator);
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(source).split(separator)).toEqual(
|
||||
lines.map((line, index) => (blankedLines.includes(index) ? ' '.repeat(line.length) : line)),
|
||||
);
|
||||
}
|
||||
|
||||
describe('Swift conditional-directive preprocessing', () => {
|
||||
it('blanks every directive of a nested group and leaves the top-level one alone', () => {
|
||||
const source = [
|
||||
'#if os(macOS)',
|
||||
'class TopLevel {}',
|
||||
'#endif',
|
||||
'class Outer {',
|
||||
' #if os(iOS) // platform branch',
|
||||
' enum A { case x }',
|
||||
'\t#elseif DEBUG && canImport(UIKit) // fallback',
|
||||
' enum B { case y }',
|
||||
' #else',
|
||||
' enum C { case z }',
|
||||
' #endif // end branch',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [4, 6, 8, 10]);
|
||||
});
|
||||
|
||||
it('preserves JavaScript string length and newline count', () => {
|
||||
const source = '#if DEBUG\nclass Outer {\n #else\n}\n#endif\n';
|
||||
const rewritten = preprocessSwiftConditionalDirectives(source);
|
||||
|
||||
expect(rewritten).toHaveLength(source.length);
|
||||
expect(rewritten.match(/\n/g)?.length ?? 0).toBe(source.match(/\n/g)?.length ?? 0);
|
||||
expectBlanked(source, []);
|
||||
});
|
||||
|
||||
it('returns directive-free Swift source unchanged', () => {
|
||||
const source = 'class Plain {\n var value: Int = 0\n func read() -> Int { value }\n}\n';
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
|
||||
it('preserves CRLF line endings and offsets', () => {
|
||||
const source = 'class Outer {\r\n\t#if os(iOS)\r\n\tenum A { case x }\r\n\t#endif\r\n}\r\n';
|
||||
const rewritten = preprocessSwiftConditionalDirectives(source);
|
||||
|
||||
expect(rewritten).toHaveLength(source.length);
|
||||
expect(rewritten.indexOf('enum A')).toBe(source.indexOf('enum A'));
|
||||
expectBlanked(source, [1, 3], '\r\n');
|
||||
});
|
||||
|
||||
it('treats a bare carriage return as a line terminator', () => {
|
||||
const source = 'class Outer {\r #if os(iOS)\r enum A { case x }\r #endif\r}\r';
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toHaveLength(source.length);
|
||||
expectBlanked(source, [1, 3], '\r');
|
||||
});
|
||||
|
||||
it('leaves regular and raw multiline string interiors byte-identical', () => {
|
||||
const source = [
|
||||
'struct Strings {',
|
||||
' let regular = """',
|
||||
' #if os(iOS)',
|
||||
' #elseif DEBUG',
|
||||
' #else',
|
||||
' #endif',
|
||||
' """',
|
||||
' #if REAL_DIRECTIVE',
|
||||
' let between = true',
|
||||
' #endif',
|
||||
' let raw = #"""',
|
||||
' #if raw(iOS)',
|
||||
' #elseif raw(DEBUG)',
|
||||
' #else',
|
||||
' #endif',
|
||||
' """#',
|
||||
' let doubleRaw = ##"""',
|
||||
' #if double-raw-string-data',
|
||||
' #endif',
|
||||
' """##',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [7, 9]);
|
||||
});
|
||||
|
||||
it('does not let an unterminated multiline string blank later lines', () => {
|
||||
const source = ['let text = """', ' #if this-is-string-data', ' still string data'].join(
|
||||
'\n',
|
||||
);
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
|
||||
it('leaves non-conditional hash directives untouched', () => {
|
||||
const source = [
|
||||
'class Directives {',
|
||||
' #warning("warning")',
|
||||
' #error("error")',
|
||||
' #available(iOS 17, *)',
|
||||
' #selector(getter: Directives.value)',
|
||||
' #if DEBUG',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [5, 6]);
|
||||
});
|
||||
|
||||
it('keeps nested block comments out of string state and never blanks inside them', () => {
|
||||
const source = [
|
||||
'/*',
|
||||
' #if in-comment',
|
||||
' /* nested comment */',
|
||||
' #endif',
|
||||
'*/',
|
||||
'let text = """',
|
||||
' #if in-string',
|
||||
'"""',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, []);
|
||||
});
|
||||
|
||||
it('keeps a block-comment terminator that shares its line with a directive', () => {
|
||||
const source = [
|
||||
'class Foo {',
|
||||
' /* temporarily disabled:',
|
||||
' #if DEBUG',
|
||||
' func f() {}',
|
||||
' #endif */',
|
||||
' func g() {}',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
// Blanking ` #endif */` would un-terminate the comment and swallow `g()`.
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
|
||||
it('blanks a column-zero directive nested inside a class body', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' enum A { case x }',
|
||||
'#if os(iOS)',
|
||||
' enum B { case y }',
|
||||
'#endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [2, 4]);
|
||||
});
|
||||
|
||||
it('leaves an indented directive that is still at file scope intact', () => {
|
||||
const source = [' #if DEBUG', ' struct Debugged {}', ' #endif', ''].join('\n');
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
|
||||
it('recognizes non-ASCII indentation and a leading byte-order mark', () => {
|
||||
const nbspSource = [
|
||||
'class Outer {',
|
||||
' #if os(iOS)',
|
||||
' enum A { case x }',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
const bomSource = `class Outer {\n #if os(iOS)\n enum A { case x }\n #endif\n}\n`;
|
||||
|
||||
expectBlanked(nbspSource, [1, 3]);
|
||||
expectBlanked(bomSource, [1, 3]);
|
||||
});
|
||||
|
||||
it('refuses to blank a group whose branches split a declaration header', () => {
|
||||
const source = [
|
||||
'class NetworkClient {',
|
||||
' #if swift(>=5.5)',
|
||||
' func fetch() async {',
|
||||
' #else',
|
||||
' func fetch() {',
|
||||
' #endif',
|
||||
' perform()',
|
||||
' }',
|
||||
'}',
|
||||
'struct SessionStore {}',
|
||||
].join('\n');
|
||||
|
||||
// Both branch bodies open a brace and only one closes; blanking would leave
|
||||
// `NetworkClient` unterminated and re-parent `SessionStore` under it.
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
|
||||
it('blanks nested balanced groups at every level', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' #if os(iOS)',
|
||||
' func inner() {',
|
||||
' #if DEBUG',
|
||||
' log()',
|
||||
' #endif',
|
||||
' }',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [1, 3, 5, 7]);
|
||||
});
|
||||
|
||||
it('lets an unbalanced nested group also block its enclosing group', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' #if os(iOS)',
|
||||
' func inner() {',
|
||||
' #if DEBUG',
|
||||
' if x {',
|
||||
' #else',
|
||||
' if y {',
|
||||
' #endif',
|
||||
' log()',
|
||||
' }',
|
||||
' }',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
// Both `if` branches survive blanking, so the enclosing branch is +1 too.
|
||||
// Conservative propagation degrades the whole nest to pre-fix behavior.
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
|
||||
it('does not wedge on an escaped triple quote inside a multiline string', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' let text = """',
|
||||
' escaped \\""" still string data',
|
||||
' #if in-string',
|
||||
' """',
|
||||
' #if REAL_DIRECTIVE',
|
||||
' func after() {}',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [5, 7]);
|
||||
});
|
||||
|
||||
it('closes a plain multiline string whose terminator is followed by a pound', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' let text = """',
|
||||
' body',
|
||||
' """#hashAfterClose',
|
||||
' #if REAL_DIRECTIVE',
|
||||
' func after() {}',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [4, 6]);
|
||||
});
|
||||
|
||||
it('closes a raw multiline string terminated by extra pounds', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' let raw = #"""',
|
||||
' body',
|
||||
' """##',
|
||||
' #if REAL_DIRECTIVE',
|
||||
' func after() {}',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [4, 6]);
|
||||
});
|
||||
|
||||
it('does not let an extended regex literal open a phantom block comment', () => {
|
||||
const source = [
|
||||
'class Outer {',
|
||||
' let pattern = #/a/*b/#',
|
||||
' #if REAL_DIRECTIVE',
|
||||
' func after() {}',
|
||||
' #endif',
|
||||
'}',
|
||||
].join('\n');
|
||||
|
||||
expectBlanked(source, [2, 4]);
|
||||
});
|
||||
|
||||
it('stays linear on a long run of bare pound signs', () => {
|
||||
// The pre-fix scanner re-walked the whole run at every index: ~10.6s for
|
||||
// n=64000. The bound is a per-test timeout rather than a measured
|
||||
// elapsed-time assertion; the fixed scanner runs this in ~1ms.
|
||||
const source = `class Outer {\n let s = ${'#'.repeat(64000)}\n #if REAL_DIRECTIVE\n func after() {}\n #endif\n}\n`;
|
||||
|
||||
expectBlanked(source, [2, 4]);
|
||||
}, 2000);
|
||||
|
||||
it('preserves JavaScript length on a directive carrying non-ASCII comment text', () => {
|
||||
const source = ['class Outer {', ' #if os(iOS) // 日本語 🔥', ' #endif', '}'].join('\n');
|
||||
const rewritten = preprocessSwiftConditionalDirectives(source);
|
||||
|
||||
// UTF-16 length is preserved; UTF-8 byte length is not (56 -> 48).
|
||||
// Documented as safe because no consumer slices the original bytes by
|
||||
// `startIndex` — node-tree-sitter reports UTF-16 code-unit indices.
|
||||
expect(rewritten).toHaveLength(source.length);
|
||||
expect(Buffer.byteLength(source, 'utf8')).toBe(56);
|
||||
expect(Buffer.byteLength(rewritten, 'utf8')).toBe(48);
|
||||
expectBlanked(source, [1, 2]);
|
||||
});
|
||||
|
||||
it('is idempotent', () => {
|
||||
const source = ['class Outer {', ' #if os(iOS)', ' enum A { case x }', ' #endif', '}'].join(
|
||||
'\n',
|
||||
);
|
||||
const once = preprocessSwiftConditionalDirectives(source);
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(once)).toBe(once);
|
||||
});
|
||||
|
||||
it('leaves an unmatched directive untouched', () => {
|
||||
const source = ['class Outer {', ' #endif', ' #if NEVER_CLOSED', '}'].join('\n');
|
||||
|
||||
expect(preprocessSwiftConditionalDirectives(source)).toBe(source);
|
||||
});
|
||||
});
|
||||
Reference in New Issue
Block a user