From 0c7d0f7d422aadbe5b78366e926823afaa5e7bd0 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 16:09:30 +0000 Subject: [PATCH 01/23] =?UTF-8?q?perf(resolution):=20remove=20the=20Python?= =?UTF-8?q?=20indexing=20regressions=20since=201.6.2=20=E2=80=94=20graph?= =?UTF-8?q?=20byte-identical=20(#2332)=20(#2344)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Indexing CPython took about three times as long on 1.6.2 as at 290e03f (193 s vs 59 s wall, 8.3 GB vs 3.5 GB peak on a 16-thread Windows box). A CPU profile put ~290 s of the resolver workers' 412 s on two Python checks that first shipped in 1.6.2: - pythonExternalWrites (#2291) read every .py file once per module global its own module types. CPython has 2,375 Python files against the resolver's 1,000-entry content cache, so every read went to disk (184 s). - isPythonLocallyBound (#2198, #2219) re-stripped the calling file for every (function, name), and, asked once per same-named candidate by fitsPythonCallShape, re-derived the enclosing function each time: 3.55M calls for 53K distinct answers (104 s). Now: - A global's own module is read first; when its bindings already leave the type unknown (319 of 379 globals on CPython), no other module is read. - One pass per resolution indexes which Python files spell a name after a dot or a quote, and which write through .__dict__; the write scan visits only those files, in the same path order. Comment stripping only blanks text, so any write the scan accepts is spelled that way in the raw text; a non-ASCII name still reads every file. Index keys are flat copies (a regex capture would pin its whole file). - The repo files an import names are resolved once per mapping, in a WeakMap that lives as long as the resolver's import cache keeps it. - isPythonLocallyBound keeps the current file's stripped lines and per-name module-level answer, and answers a ref once. Verification: nodes, edges and unresolved_refs dumps of CPython and pretix are byte-identical to 6560052a. Medians of 3 interleaved CPython runs on a loaded machine: 290e03f 100 s / 271 s CPU / 3.3 GB, 6560052a 322 s / 801 s / 7.9 GB, this change 123 s / 361 s / 5.5 GB. In the CPU profile pythonExternalWrites drops 184 s -> 16 s and isPythonLocallyBound 104 s -> 13 s. Most of the remaining gap to 290e03f on CPython is JS checks running on its bundled minified d3 (matchDestructuredCallResult, #2334; jsFunctionLocalScope from #2226). __tests__/python-resolution-work.test.ts counts the work instead of timing it: reads of untouched files, strips of the calling file, and node lookups with 16 same-named candidates (4 / 2 / 8 with the fix, 19 / 18 / 128 on main). Each count fails when only its own part of the fix is reverted. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 +- __tests__/python-resolution-work.test.ts | 102 +++++++++++++++ src/resolution/name-matcher.ts | 155 +++++++++++++++++++---- 3 files changed, 231 insertions(+), 28 deletions(-) create mode 100644 __tests__/python-resolution-work.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index f280890b4f..68d7cc062b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -20,7 +20,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - +- Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/python-resolution-work.test.ts b/__tests__/python-resolution-work.test.ts new file mode 100644 index 0000000000..f4faf445b5 --- /dev/null +++ b/__tests__/python-resolution-work.test.ts @@ -0,0 +1,102 @@ +import { afterEach, describe, expect, it, vi } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { ReferenceResolver } from '../src/resolution'; +import { stripCommentsForRegex } from '../src/resolution/strip-comments'; + +// Pass-through, so a test can count how often a file's text is stripped. +vi.mock('../src/resolution/strip-comments', async (importOriginal) => { + const actual = await importOriginal(); + return { ...actual, stripCommentsForRegex: vi.fn(actual.stripCommentsForRegex) }; +}); + +/** + * #2332: resolving Python does a bounded amount of work per file and per call. + * The cross-module write scan for a module global read every Python file once + * per global; the local-binding check stripped the calling file once per + * function, and looked up the function around a call once per same-named + * candidate. On CPython that was minutes of CPU. Counted, never timed. + */ +describe('Python resolution work (#2332)', () => { + let tmpDir: string | undefined; + let cg: CodeGraph | undefined; + + afterEach(() => { + vi.restoreAllMocks(); + vi.mocked(stripCommentsForRegex).mockClear(); + cg?.close(); + cg = undefined; + if (tmpDir) fs.rmSync(tmpDir, { recursive: true, force: true, maxRetries: 5 }); + tmpDir = undefined; + }); + + /** Index `files`, counting per file what resolution reads, strips and looks up. */ + async function indexCounting(files: Record) { + tmpDir = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-py-work-')); + for (const [file, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(tmpDir, file)), { recursive: true }); + fs.writeFileSync(path.join(tmpDir, file), content); + } + cg = CodeGraph.initSync(tmpDir); + const context = (cg as unknown as { resolver: ReferenceResolver }).resolver.getResolutionContext(); + const reads = vi.spyOn(context, 'readFile'); + const lookups = vi.spyOn(context, 'getNodesInFile'); + await cg.indexAll(); + return { + graph: cg, + reads: (file: string) => reads.mock.calls.filter(([f]) => f === file).length, + lookups: (file: string) => lookups.mock.calls.filter(([f]) => f === file).length, + strips: (file: string) => vi.mocked(stripCommentsForRegex).mock.calls.filter(([text]) => text === files[file]).length, + }; + } + + it('reads no file once per module global, and strips no file once per function', async () => { + const SCALE = 16; + const conns = Array.from({ length: SCALE }, (_, i) => `conn${i}`); + const files: Record = { + // Globals typed by their own module's writes: each one's type also + // depends on what every other module writes to it. + 'store.py': 'class Store:\n def fetch(self, ids):\n return ids\n', + 'settings.py': `from store import Store\n${conns.map(c => `${c} = None\n`).join('')}\n` + + `def init():\n global ${conns.join(', ')}\n${conns.map(c => ` ${c} = Store()\n`).join('')}`, + 'consumer.py': `import settings\n${conns.map((c, i) => `\ndef cb${i}(pool):\n pool.submit(settings.${c}.fetch)\n`).join('')}`, + // A bare call in every function: each asks whether its function binds the name. + 'helpers.py': 'def helper():\n return 1\n', + 'callers.py': Array.from({ length: SCALE }, (_, i) => `def f${i}():\n return helper()\n\n`).join('') + + 'def shadowed(make):\n helper = make()\n return helper()\n', + }; + // Files no reference touches. + const fillers = Array.from({ length: 20 }, (_, i) => `pkg/filler${i}.py`); + fillers.forEach((file, i) => { files[file] = `def filler${i}(x):\n return x + ${i}\n`; }); + const { graph, reads, strips } = await indexCounting(files); + + // Both paths ran: every global's method value reached Store.fetch, and the + // function that binds `helper` itself calls its own value. + const fetch = graph.getNodesByName('fetch').find(n => n.qualifiedName === 'Store::fetch')!; + expect(graph.getIncomingEdges(fetch.id).filter(e => e.metadata?.fnRef === true) + .map(e => graph.getNode(e.source)?.name).sort()).toEqual(conns.map((_, i) => `cb${i}`).sort()); + const helper = graph.getNodesByName('helper').find(n => n.kind === 'function')!; + expect(graph.getIncomingEdges(helper.id).filter(e => e.kind === 'calls') + .map(e => graph.getNode(e.source)?.name).sort()).toEqual(Array.from({ length: SCALE }, (_, i) => `f${i}`).sort()); + // A few whole-project passes read each file once; the write scan read every + // file once more per global. + expect(Math.max(...fillers.map(reads))).toBeLessThan(SCALE / 2); + // Stripped once for the pass, not once per function. + expect(strips('callers.py')).toBeLessThan(SCALE / 2); + }); + + it('finds the function around a call once, however many functions share its name', async () => { + const CANDIDATES = 16; + const CALLS = 8; + const files: Record = { + 'callers.py': Array.from({ length: CALLS }, (_, i) => `def f${i}():\n return helper()\n\n`).join(''), + }; + for (let k = 0; k < CANDIDATES; k++) files[`lib${k}/helpers.py`] = 'def helper():\n return 1\n'; + const { lookups } = await indexCounting(files); + // Each same-named candidate asks whether the caller binds `helper`; the + // answer is the call's, so the caller's nodes are read per call. + expect(lookups('callers.py')).toBeLessThanOrEqual(2 * CALLS); + }); +}); diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index e0b4ad28b8..2984a43464 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -7,7 +7,7 @@ import * as fs from 'fs'; import * as path from 'path'; import { Language, Node } from '../types'; -import { UnresolvedRef, ResolvedRef, ResolutionContext, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; +import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSupertypeTarget, CPP_DEFINE_SIGNATURE, isInheritanceRef, isImportableKind } from './types'; import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; @@ -659,26 +659,13 @@ function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: Resoluti let memo = PYTHON_GLOBAL_CLASSES.get(context); if (!memo) { memo = new Map(); PYTHON_GLOBAL_CLASSES.set(context, memo); } if (memo.has(global.id)) return memo.get(global.id)!; - const file = global.filePath; - const external = pythonExternalWrites(global, context); + const own = pythonOwnGlobalWrites(global, context); + const external = own && pythonExternalWrites(global, context); // Each type is resolved in the file that wrote it (its imports name the class). - const writes: Array<{ type: string; file: string }> = [...external.writes]; - let classes: Node[] | null = external.unknown || pythonDynamicGlobalWrite(global.name, file, context) ? null : []; - for (const b of classes ? pythonGlobalBindings(global.name, file, context) : []) { - if (b.kind !== 'assign') { classes = null; break; } - const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; - if (b.type) { - // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. - if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) { classes = null; break; } - writes.push({ type: b.type, file }); - continue; - } - if (b.value === 'None') continue; - if (!constructor) { classes = null; break; } - writes.push({ type: constructor, file }); - } + const writes = external && !external.unknown ? [...external.writes, ...own!] : null; + let classes: Node[] | null = writes && []; const seen = new Set(); - for (const write of classes ? writes : []) { + for (const write of writes ?? []) { const cls = pythonRefClass(write.type, { ...ref, filePath: write.file }, context); if (!cls) { classes = null; break; } if (!seen.has(cls.id)) { seen.add(cls.id); classes!.push(cls); } @@ -687,6 +674,33 @@ function pythonGlobalClasses(global: Node, ref: UnresolvedRef, context: Resoluti return classes; } +/** + * The types the global's own module writes to it, or null when a binding + * there leaves its type unknown — whatever other modules write, so they are + * not read (#2332). + */ +function pythonOwnGlobalWrites(global: Node, context: ResolutionContext): Array<{ type: string; file: string }> | null { + return pythonNameScan(context, `own\0${global.id}`, () => { + const file = global.filePath; + if (pythonDynamicGlobalWrite(global.name, file, context)) return null; + const writes: Array<{ type: string; file: string }> = []; + for (const b of pythonGlobalBindings(global.name, file, context)) { + if (b.kind !== 'assign') return null; + const constructor = b.value && b.value !== 'None' ? pythonConstructorCall(b.value) : null; + if (b.type) { + // `conn: Base = make()` trusts the annotation; `conn: A = B()` contradicts it. + if (constructor && constructor.split('.').pop() !== b.type.split('.').pop()) return null; + writes.push({ type: b.type, file }); + continue; + } + if (b.value === 'None') continue; + if (!constructor) return null; + writes.push({ type: constructor, file }); + } + return writes; + }); +} + /** * The repo files a Python module path can name from `fromFile`. Relative * paths (`..settings`) resolve exactly; absolute ones match a file path @@ -722,9 +736,7 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso const aliases = new Set(); const ambiguous = new Set(); for (const m of context.getImportMappings(filePath, 'python')) { - const dotted = m.isNamespace ? m.source - : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; - const files = pythonModuleFiles(dotted, filePath, context); + const files = pythonImportedFiles(m, filePath, context); if (!files.includes(moduleFile)) continue; // The mapping cannot tell `import a.b` from `import a.b as b`; the source line can. // Exactly this module (not `other.a.b`), outside string literals. @@ -736,6 +748,24 @@ function pythonModuleAliases(filePath: string, moduleFile: string, context: Reso return { aliases: [...aliases], ambiguous: [...ambiguous] }; } +const PYTHON_IMPORTED_FILES = new WeakMap>(); +/** + * The repo files an import of `filePath` can name. Every global's write scan + * asks again of the same files (#2332); kept for as long as the resolver keeps + * the mapping itself, so the memo never outlives its import cache. + */ +function pythonImportedFiles(m: ImportMapping, filePath: string, context: ResolutionContext): string[] { + let memo = PYTHON_IMPORTED_FILES.get(context); + if (!memo) PYTHON_IMPORTED_FILES.set(context, (memo = new WeakMap())); + let files = memo.get(m); + if (!files) { + const dotted = m.isNamespace ? m.source + : /^\.+$/.test(m.source) ? `${m.source}${m.exportedName}` : `${m.source}.${m.exportedName}`; + memo.set(m, (files = pythonModuleFiles(dotted, filePath, context))); + } + return files; +} + const PYTHON_MAIN_GUARD = /^if\s+(?:__name__\s*==\s*(['"])__main__\1|(['"])__main__\2\s*==\s*__name__)\s*:/; /** Line indexes inside a top-level `if __name__ == "__main__":` block — script code, not module state. */ function pythonMainBlockLines(filePath: string, context: ResolutionContext): Set { @@ -775,8 +805,8 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write const out = { writes: [] as Array<{ type: string; file: string }>, unknown: false, writers: new Set() }; const name = global.name; const escape = (s: string) => s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); - for (const file of context.getAllFiles()) { - if (file === global.filePath || !/\.pyi?$/.test(file) || !context.readFile(file)?.includes(name)) continue; + for (const file of pythonWriteCandidates(name, context)) { + if (file === global.filePath) continue; const test = isPythonTestFile(file); const { aliases, ambiguous } = pythonModuleAliases(file, global.filePath, context); if (!test && !aliases.length && !ambiguous.length) continue; @@ -812,6 +842,41 @@ function pythonExternalWrites(global: Node, context: ResolutionContext): { write }); } +/** + * The Python files that can write a module global named `name`, in path + * order: those that spell it after a dot (`settings.conn = X`) or a quote + * (`setattr(settings, "conn", X)`), and those that write through `.__dict__`, + * whose statement may spell it anywhere. Comment stripping only blanks text, + * so whatever a stripped statement spells, the file's text spells too. The + * files are indexed once per pass, instead of every global reading every + * Python file (#2332). + */ +function pythonWriteCandidates(name: string, context: ResolutionContext): string[] { + const index = pythonNameScan(context, 'write-candidates', () => { + const files = context.getAllFiles().filter(f => /\.pyi?$/.test(f)); + const spelled = new Map(); + const dict: Array<{ at: number; words: string }> = []; + files.forEach((file, at) => { + const source = context.readFile(file) ?? ''; + const names = new Set(); + for (const m of source.matchAll(/[.'"](\w+)/g)) names.add(m[1]!); + for (const n of names) { + const list = spelled.get(n); + // A capture is a sliced view that would pin the file's whole text: key a flat copy. + if (list) list.push(at); else spelled.set(Buffer.from(n).toString(), [at]); + } + // A word-only name is in the text exactly when it is in one of its words. + if (names.has('__dict__')) dict.push({ at, words: [...new Set(source.match(/\w+/g))].join('\n') }); + }); + return { files, spelled, dict }; + }); + // The index holds ASCII words; any other name is looked for in every file's text. + if (!/^\w+$/.test(name)) return index.files.filter(f => context.readFile(f)?.includes(name)); + const hits = new Set(index.spelled.get(name)); + for (const { at, words } of index.dict) if (words.includes(name)) hits.add(at); + return [...hits].sort((a, b) => a - b).map(at => index.files[at]!); +} + /** * Whether the global's own module can write it through its namespace dict: * `globals()` / `vars()` / `sys.modules[__name__]` used as anything but a @@ -853,6 +918,8 @@ function pythonDynamicGlobalWrite(name: string, filePath: string, context: Resol * as a base-typed receiver does. Otherwise, no edge. */ function pythonGlobalMembers(global: Node, member: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + // Unknown from its own module alone: no edge, and no other module to read. + if (!pythonOwnGlobalWrites(global, context)) return []; // A test that installs its own double sees the double, not the production type. if (pythonExternalWrites(global, context).writers.has(ref.filePath)) return []; const classes = pythonGlobalClasses(global, ref, context); @@ -2538,6 +2605,14 @@ function isDecoratedFixture(n: Node, context: ResolutionContext): boolean { } const PY_LOCAL_BINDS = new WeakMap>(); +/** + * The file isPythonLocallyBound last read: its code lines, and per name + * whether its module binds it. Refs arrive grouped by file, so one file per + * context spares re-stripping the file for every function and name (#2332). + */ +const PY_LOCAL_FILE = new WeakMap }>(); +/** The ref isPythonLocallyBound last answered, and the answer. */ +const PY_LOCAL_LAST = new WeakMap(); /** * Whether the function around a Python call — or its module, at top level — @@ -2547,6 +2622,17 @@ const PY_LOCAL_BINDS = new WeakMap>(); * every such call went to one test file's `def view`. */ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { + // fitsPythonCallShape asks once per same-named candidate, and finding the + // function around the call reads every node in the file: answer a ref once (#2332). + const last = PY_LOCAL_LAST.get(context); + if (last?.ref === ref && last.name === name) return last.bound; + const bound = pythonLocalBinding(name, ref, context); + PY_LOCAL_LAST.set(context, { ref, name, bound }); + return bound; +} + +/** isPythonLocallyBound's answer, kept per calling function and name. */ +function pythonLocalBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): boolean { const fn = context.getNodesInFile(ref.filePath) .filter((f) => (f.kind === 'function' || f.kind === 'method') && f.startLine <= ref.line && f.endLine >= ref.line) .sort((a, b) => (a.endLine - a.startLine) - (b.endLine - b.startLine))[0]; @@ -2561,7 +2647,12 @@ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: Resolut return false; } // Code only: `{% user_display user as user_display %}` in a docstring binds nothing. - const lines = stripCommentsForRegex(context.readFile(ref.filePath) ?? '', 'python').split(/\r?\n/); + let file = PY_LOCAL_FILE.get(context); + if (file?.filePath !== ref.filePath) { + const lines = stripCommentsForRegex(context.readFile(ref.filePath) ?? '', 'python').split(/\r?\n/); + PY_LOCAL_FILE.set(context, (file = { filePath: ref.filePath, lines, module: new Map() })); + } + const lines = file.lines; const n = name; const assigns = new RegExp(`^\\s*(?:[\\w\\s,*()\\[\\]]*,\\s*)?\\(?\\*?${n}\\)?\\s*(?:,[\\w\\s,*()\\[\\]]*)?(?::[^=]+)?=(?!=)`); const targets = new RegExp(`\\bfor\\s+[\\w\\s,()]*\\b${n}\\b[\\w\\s,()]*\\s+in\\b|\\bas\\s+${n}\\b`); @@ -2582,8 +2673,15 @@ function isPythonLocallyBound(name: string, ref: UnresolvedRef, context: Resolut } } // A module-level binding (`view = api_view(['GET'])(handler)`). - const top = new RegExp(`^(?:[\\w,\\s]*,\\s*)?${n}\\s*(?:,[\\w\\s,]*)?(?::[^=]+)?=(?!=)`); - for (let line = 0; !bound && line < lines.length; line++) bound = top.test(lines[line] ?? ''); + if (!bound) { + let module = file.module.get(n); + if (module === undefined) { + const top = new RegExp(`^(?:[\\w,\\s]*,\\s*)?${n}\\s*(?:,[\\w\\s,]*)?(?::[^=]+)?=(?!=)`); + module = lines.some(line => top.test(line)); + file.module.set(n, module); + } + bound = module; + } memo.set(key, bound); return bound; } @@ -6606,6 +6704,7 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { PYTHON_STATEMENT_STARTS.delete(context); PYTHON_GLOBAL_CLASSES.delete(context); PYTHON_NAME_SCANS.delete(context); + PYTHON_IMPORTED_FILES.delete(context); AWAITED_TYPE_MEMO.delete(context); AWAITED_FILES.delete(context); C_STATIC_MEMO.delete(context); @@ -6664,6 +6763,8 @@ export function clearNameMatcherMemos(context: ResolutionContext): void { JVM_PACKAGES.delete(context); MINIFIED_SCRIPTS.delete(context); PY_LOCAL_BINDS.delete(context); + PY_LOCAL_FILE.delete(context); + PY_LOCAL_LAST.delete(context); OVERLOAD_SETS.delete(context); PHP_FILE_SCOPES.delete(context); JAVA_STATIC_IMPORTS.delete(context); From 355afc70b14c22713fee491eafb881b577d34ef1 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:20:15 +0000 Subject: [PATCH 02/23] fix(csharp): walk field and property initializers; a constant never names a type (#2337) (#2348) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * fix(csharp): walk field and property initializers as the member Field declarators and property `= initializer` values were skipped by both extractors, so `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` linked nothing, and a method or class used only from an initializer looked unused. - tree-sitter.ts / csharp.rs: walk each C# field declarator whole with its field on the stack, and every property `value:` (not only `=> expr`) with the property on the stack. Calls, instantiations, static reads, lambda bodies and fn-ref candidates (varinit included) belong to the member; the fn-ref scan skips the walked subtrees, so each candidate is captured once. Attributes stay unwalked. - A target-typed `new()` that is the initializer instantiates the declared type (`List _items = new();` -> List). In bodies it stays invisible, as before. - name-matcher.ts: a .NET type position never names a `constant` (the kind `const` / `static readonly` fields get). Walking initializers exposed `new Version(5, 18)` binding to a `const string Version` and `static readonly Meter Meter = new(...)` instantiating itself. Kernel/wasm parity: 0 diffs on serilog, Newtonsoft.Json and jellyfin, and full-index dumps byte-identical between the two arms. No EXTRACTION_VERSION bump: #2345 already moved it to 28 this release. Co-Authored-By: Claude Opus 5.5 * docs(changelog): credit the report and contributor behind the constant change The type-position change here is what issue #2337 reports (`new Station { … }` unresolved beside a `private const string Station`) and the same one-line change as @drakeo338's #2339. Verified on the issue's own three-file repro: the `instantiates` edge to `Demo.Core::Station`, missing on main, is present with this branch. Co-Authored-By: Claude Opus 5.5 --------- Co-authored-by: Claude Opus 5.5 --- CHANGELOG.md | 2 + __tests__/csharp-member-initializers.test.ts | 155 +++++++++++++++++++ __tests__/csharp-type-position-refs.test.ts | 42 +++++ __tests__/kernel-csharp-parity.test.ts | 40 ++++- codegraph-kernel/src/csharp.rs | 94 +++++++++-- docs/design/csharp-kernel-port-checklist.md | 96 ++++++++---- src/extraction/tree-sitter.ts | 103 +++++++++--- src/resolution/name-matcher.ts | 8 +- 8 files changed, 469 insertions(+), 71 deletions(-) create mode 100644 __tests__/csharp-member-initializers.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index 68d7cc062b..da460b5b77 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -17,6 +17,8 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, every member of a `Structure` is now indexed, including its fields, properties, methods, constructors and nested enums. Before, only the first member was, so the rest could not be found and their callers looked empty. - In VB.NET and C#, what a property's `Get` and `Set` code calls, creates and reads now belongs to that property, as do C#'s `get => …` accessors and `=> …` property bodies. Before, it was dropped, so a method used only from a property looked unused. - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. +- In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. +- In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) diff --git a/__tests__/csharp-member-initializers.test.ts b/__tests__/csharp-member-initializers.test.ts new file mode 100644 index 0000000000..f5125b8321 --- /dev/null +++ b/__tests__/csharp-member-initializers.test.ts @@ -0,0 +1,155 @@ +/** + * C# field and property initializers belong to the member they initialize. + * `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` and + * `public List Items { get; } = new List();` used to be skipped by + * both extractors, so the calls, instantiations and static reads written + * there were lost, and a method passed as a value there was the class's + * reference rather than the member's. + * + * A target-typed `new()` names no type of its own; as an initializer it + * constructs the member's declared type. + * + * Runs against the native kernel (when built) and the wasm extractor, which + * must agree. + */ +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; +import type { Node } from '../src/types'; + +const FILES: Record = { + 'Lib.cs': `namespace Lib +{ + public class Crate { } +} +public interface ILogger { } +public static class LogManager +{ + public static ILogger GetLogger(System.Type t) => null; +} +public static class Helper +{ + public static int Compute(int n) => n; + public static int Seed() => 1; + public static void Register(System.Action a) { } + public static System.Action Wrap(System.Action a) => a; +} +public class Widget +{ + public Widget() { } + public Widget(int n) { } + public int Size { get; set; } +} +public class Bag { } +public static class Defaults +{ + public const string Name = "x"; + public static readonly Widget Empty = new Widget(); +} +`, + 'Box.cs': `using Lib; + +public class Box +{ + private readonly ILogger _log = LogManager.GetLogger(typeof(Box)); + private int _a = 1, _b = Helper.Compute(2); + private static readonly int Max = Helper.Seed(); + private readonly Widget _made = new Widget(3); + private readonly Widget _typed = new() { Size = Helper.Seed() }; + private Widget? _maybe = new(); + private readonly Bag _bag = new(); + private readonly Lib.Crate _crate = new(); + private readonly string _label = Defaults.Name; + private readonly System.Func _square = x => Helper.Compute(x * x); + private readonly System.Action _later = () => Helper.Register(Handle); + private readonly System.Action _direct = Handle; + [System.Obsolete("use Helper.Seed()")] private int _flagged; + public Bag Items { get; } = new Bag(); + public Widget Made { get; set; } = new(); + public Widget Copy { get; } = Defaults.Empty; + public System.Action Wrapped { get; } = Helper.Wrap(Handle); + public int Seeded { get; } = Helper.Seed(); + private static void Handle() { } +} +`, +}; + +describe('C# field and property initializers', () => { + let root = ''; + let cg: CodeGraph | undefined; + let kernel: string | undefined; + + beforeEach(() => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-initializers-')); + for (const [rel, content] of Object.entries(FILES)) fs.writeFileSync(path.join(root, rel), content); + kernel = process.env.CODEGRAPH_KERNEL; + }); + + afterEach(() => { + cg?.close(); + cg = undefined; + fs.rmSync(root, { recursive: true, force: true }); + if (kernel === undefined) delete process.env.CODEGRAPH_KERNEL; + else process.env.CODEGRAPH_KERNEL = kernel; + }); + + it.each(['default', 'wasm'])('are walked as the member they initialize (%s)', async (backend) => { + if (backend === 'wasm') process.env.CODEGRAPH_KERNEL = '0'; + else delete process.env.CODEGRAPH_KERNEL; + cg = await CodeGraph.init(root, { index: true }); + const graph = cg; + const member = (qualifiedName: string): Node => { + const node = graph.getNodesInFile('Box.cs').find((n) => n.qualifiedName === qualifiedName); + expect(node, qualifiedName).toBeDefined(); + return node!; + }; + const targets = (node: Node, kind: string): string[] => + graph + .getOutgoingEdgesFrom([node.id]) + .filter((e) => e.kind === kind) + .map((e) => graph.getNode(e.target)!.qualifiedName) + .sort(); + + // Calls, per declarator. + expect(targets(member('Box::_log'), 'calls')).toEqual(['LogManager::GetLogger']); + expect(targets(member('Box::_a'), 'calls')).toEqual([]); + expect(targets(member('Box::_b'), 'calls')).toEqual(['Helper::Compute']); + expect(member('Box::Max').kind).toBe('constant'); + expect(targets(member('Box::Max'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box::Seeded'), 'calls')).toEqual(['Helper::Seed']); + + // Instantiations, including a target-typed `new()` of the declared type. + expect(targets(member('Box::_made'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_typed'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_typed'), 'calls')).toEqual(['Helper::Seed']); + expect(targets(member('Box::_maybe'), 'instantiates')).toEqual(['Widget']); + expect(targets(member('Box::_bag'), 'instantiates')).toEqual(['Bag']); + expect(targets(member('Box::_crate'), 'instantiates')).toEqual(['Lib::Crate']); + expect(targets(member('Box::Items'), 'instantiates')).toEqual(['Bag']); + expect(targets(member('Box::Made'), 'instantiates')).toEqual(['Widget']); + + // Static reads. + expect(targets(member('Box::_label'), 'references')).toContain('Defaults'); + expect(targets(member('Box::Copy'), 'references')).toContain('Defaults'); + + // Lambda bodies belong to the member the lambda initializes. + expect(targets(member('Box::_square'), 'calls')).toEqual(['Helper::Compute']); + expect(targets(member('Box::_later'), 'calls')).toEqual(['Helper::Register']); + + // A method passed as a value is the member's reference, captured once — + // not the class's as well. + expect(targets(member('Box::_later'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box::_direct'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box::Wrapped'), 'calls')).toEqual(['Helper::Wrap']); + expect(targets(member('Box::Wrapped'), 'references')).toContain('Box::Handle'); + expect(targets(member('Box'), 'references')).not.toContain('Box::Handle'); + + // Attribute arguments are not an initializer, and the class itself + // calls and creates nothing. + expect(targets(member('Box::_flagged'), 'calls')).toEqual([]); + expect(targets(member('Box'), 'calls')).toEqual([]); + expect(targets(member('Box'), 'instantiates')).toEqual([]); + }); +}); diff --git a/__tests__/csharp-type-position-refs.test.ts b/__tests__/csharp-type-position-refs.test.ts index c000d92105..ccfba566b9 100644 --- a/__tests__/csharp-type-position-refs.test.ts +++ b/__tests__/csharp-type-position-refs.test.ts @@ -69,4 +69,46 @@ describe('C#: a type position names a type', () => { cg.close(); } }); + + it('never a constant that shares the name', async () => { + // `const` and `static readonly` fields are constants: jellyfin's `new + // Version(5, 18)` (System.Version) bound to a `const string Version` + // claim name, and serilog's `static readonly Meter Meter = new(…)` + // instantiated itself. + const root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-cs-type-refs-')); + roots.push(root); + const files: Record = { + 'src/Claims.cs': `namespace App; +public static class ClaimTypes { + public const string Version = "v"; +} +`, + 'src/Encoder.cs': `namespace App; +using System; +public class ValueFormatter { } +public class Encoder { + private static readonly Version MinVersion = new Version(5, 18); + private static readonly Meter Meter = new("app"); + private static readonly ValueFormatter ValueFormatter = new(); + public void Check() { var v = new Version(1, 0); } +} +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + const cg = await CodeGraph.init(root, { index: true }); + try { + const members = cg.getNodesInFile('src/Encoder.cs').filter((n) => n.kind !== 'file'); + const targets = cg + .getOutgoingEdgesFrom(members.map((n) => n.id), ['references', 'instantiates', 'type_of']) + .map((e) => `${cg.getNode(e.source)!.name} ${e.kind} ${cg.getNode(e.target)!.kind}:${cg.getNode(e.target)!.qualifiedName}`) + .sort(); + expect(targets.filter((t) => t.includes(' constant:'))).toEqual([]); + expect(targets).toContain('ValueFormatter instantiates class:App::ValueFormatter'); + } finally { + cg.close(); + } + }); }); diff --git a/__tests__/kernel-csharp-parity.test.ts b/__tests__/kernel-csharp-parity.test.ts index 3d09c19d98..7a15371891 100644 --- a/__tests__/kernel-csharp-parity.test.ts +++ b/__tests__/kernel-csharp-parity.test.ts @@ -8,12 +8,12 @@ * * - Torture.cs — block namespace + nested/second-namespace quirks, * base_list shapes, records, properties (incl. the bare-identifier - * signature loss, accessor bodies walked as the property and - * never-walked initializers), fields/constants, + * signature loss; accessor bodies and initializers walked as the + * property), fields/constants (initializers walked as the field), * events/operators/indexer/destructor (no nodes, calls → class), ctor * initializer hole, explicit interface impl, local functions, the call * zoo (raw member-access texts, chained re-encode, `(myDel)(x)` conv, - * `nameof`), instantiation shapes (incl. invisible `new()`/`new {}`/ + * `nameof`), instantiation shapes (incl. invisible body `new()`/`new {}`/ * arrays), static value reads, C# type refs, fn-ref candidates * (`+=` subscription, `this.X` bare-name form, initializer lists), * value-ref targets + local shadow prune, preprocessor passthrough. @@ -169,10 +169,9 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { minNodes: 2, }, { - // Accessor bodies and `=> expr` are walked as the property (calls, - // instantiations, static reads, fn-ref candidates); an `= initializer` - // is only scanned for candidates, attributed to the class. - name: 'property bodies are walked as the property; initializers only scanned', + // Accessor bodies, `=> expr` and `= initializer` are walked as the + // property (calls, instantiations, static reads, fn-ref candidates). + name: 'property bodies and initializers are walked as the property', source: [ 'public class C {', ' void H(int v) { }', @@ -185,6 +184,33 @@ describe.skipIf(!kernelBuilt)('kernel C# extraction parity', () => { ].join('\n'), minNodes: 7, }, + { + // Each field declarator and each property `= initializer` is walked + // with its member on the stack — lambdas included — and its fn-ref + // candidates (varinit too) are the member's, captured once. A + // target-typed `new()` there instantiates the declared type; attribute + // arguments stay with the class's candidates-only scan. + name: 'field and property initializers are walked as the member', + source: [ + 'public class C {', + ' static void H() { }', + ' private readonly ILogger _log = LogManager.GetLogger(typeof(C));', + ' int a = 1, b = Calc.Max(2);', + ' private static readonly List Table = new() { H };', + ' private Widget? _w = new(1) { Size = Make() };', + ' private global::Ns.Box _g = new(), _h = (new());', + ' private (int, int) _t = new();', + ' Action g = () => Register(H);', + ' Del d = H;', + ' public Widget P { get; } = new();', + ' public Action R { get; } = Wrap(H);', + ' public string S { get; } = Defaults.Name;', + ' [Attr(Make(H))] public int T { get; set; } = 3;', + '}', + '', + ].join('\n'), + minNodes: 17, + }, ]; for (const m of MICROS) { diff --git a/codegraph-kernel/src/csharp.rs b/codegraph-kernel/src/csharp.rs index 04e20ca5a5..c10696ec99 100644 --- a/codegraph-kernel/src/csharp.rs +++ b/codegraph-kernel/src/csharp.rs @@ -5,8 +5,8 @@ //! path, bug-for-bug, verified by scripts/kernel-parity.mjs and the full-index //! dump-diff gate. The authoritative quirk list is //! docs/design/csharp-kernel-port-checklist.md — including every deliberate -//! emission hole (field/property initializers, constructor initializers, -//! delegates/events/operators/indexers, top-level locals) and garbage ref +//! emission hole (constructor initializers, delegates/events/operators/ +//! indexers, top-level locals) and garbage ref //! (`(repo)` primary-ctor extends, `: byte` enum extends, `nameof` calls) //! this file preserves on purpose. Positions in UTF-16 code units. Files whose //! parse tree contains ERRORS defer to the wasm extractor. @@ -572,16 +572,18 @@ impl<'t> Walker<'t> { self.extract_enum(node); skip_children = true; } else if kind == "property_declaration" && self.inside_class_like() { - // The code a property runs — its accessor bodies and `=> expr` — - // is walked with the property on the stack (propertyBodies). The - // candidates-only scan covers the rest (an `= initializer`), + // The code a property runs — its accessor bodies, `=> expr` and + // `= initializer` — is walked with the property on the stack + // (propertyBodies). The candidates-only scan covers the rest, // skipping what the body walk captured. let walked: Vec = match self.extract_property(node) { Some((row, name)) => { let bodies = property_bodies(node); if !bodies.is_empty() { self.stack.push(Scope { row, kind: "property", name }); + let declared = node.child_by_field_name("type"); for body in &bodies { + self.extract_target_typed_new(Some(*body), declared); self.visit_function_body(*body); } self.stack.pop(); @@ -593,8 +595,11 @@ impl<'t> Walker<'t> { self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "field_declaration" && self.inside_class_like() { - self.extract_field(node); - self.scan_fn_ref_subtree(node, 0, &[]); + // Each declarator, `= initializer` included, is walked with its + // field on the stack (extract_field); the candidates-only scan + // covers the rest, skipping the declarators that walk captured. + let walked = self.extract_field(node); + self.scan_fn_ref_subtree(node, 0, &walked); skip_children = true; } else if kind == "local_declaration_statement" && !self.inside_class_like() { // Top-level statements: extractVariable's generic fallback finds no @@ -864,8 +869,11 @@ impl<'t> Walker<'t> { } /// extractField (2046) — field_declaration; each declarator becomes a - /// field/constant node anchored at the DECLARATOR. - fn extract_field(&mut self, node: Node<'t>) { + /// field/constant node anchored at the DECLARATOR, and the declarator — + /// its `= initializer` — is walked with that node on the stack. Returns + /// the walked declarators, for the fn-ref scan to skip. + fn extract_field(&mut self, node: Node<'t>) -> Vec { + let mut walked = Vec::new(); let docstring = preceding_docstring(node, self.src); let visibility = Some(self.visibility_of(node)); let is_static = Some(self.is_static(node)); @@ -932,6 +940,15 @@ impl<'t> Walker<'t> { // multi-declarator fields emit the type refs once PER // declarator, each from its own field node. self.extract_csharp_type_refs(node, row); + // The initializer is the declarator's last, unnamed child: + // its calls, instantiations, static reads and fn-ref + // candidates (varinit included) are the field's. + self.stack.push(Scope { row, kind: field_kind, name }); + let declared = var_decl.and_then(|vd| vd.child_by_field_name("type")); + self.extract_target_typed_new(last_named_child(decl), declared); + self.visit_function_body(decl); + self.stack.pop(); + walked.push(decl.id()); } } } else { @@ -951,6 +968,44 @@ impl<'t> Walker<'t> { ); } } + walked + } + + /// extractTargetTypedNew (tree-sitter.ts) — a target-typed `new()` + /// (implicit_object_creation_expression) names no type, so it is no + /// instantiation kind; as a field's or property's initializer it + /// constructs the declared type (`List _items = new();` → List). + /// Emitted from the stack top (the member) at the `new()`. + fn extract_target_typed_new(&mut self, value: Option>, declared: Option>) { + let Some(value) = value else { return }; + if value.kind() != "implicit_object_creation_expression" || self.stack.is_empty() { + return; + } + let Some(class_name) = declared.and_then(|t| self.class_type_name(t)) else { return }; + let from = self.top_row(); + self.push_ref_at(from, &class_name, edge_kind_index("instantiates").unwrap(), value); + } + + /// csharpClassTypeName (tree-sitter.ts) — the class a declared type + /// names, as `new T()` would name it: `List` → List, `Ns.Foo` / + /// `global::Foo` → Foo, `Foo?` → Foo. Predefined, array, tuple and pointer + /// types name no class. + fn class_type_name(&self, node: Node) -> Option { + match node.kind() { + "identifier" => { + let text = self.text(node); + (!text.is_empty()).then(|| text.to_string()) + } + "generic_name" => { + let ident = (0..node.named_child_count()) + .filter_map(|i| node.named_child(i)) + .find(|c| c.kind() == "identifier")?; + self.class_type_name(ident) + } + "qualified_name" | "alias_qualified_name" => self.class_type_name(node.child_by_field_name("name")?), + "nullable_type" => self.class_type_name(node.child_by_field_name("type")?), + _ => None, + } } /// extractMethod (1737) — method_declaration + constructor_declaration. @@ -1142,7 +1197,8 @@ impl<'t> Walker<'t> { let Some(ctor) = ctor else { return }; // `new List()` → `List`; `new Ns.Foo()` → `Foo`. Target-typed // `new()` / anonymous `new { }` / arrays `new T[n]` never reach here - // (not in INSTANTIATION_KINDS) — invisible by design. + // (not in INSTANTIATION_KINDS) — invisible by design, except a `new()` + // that initializes a field or property (extract_target_typed_new). let class_name = strip_generic_and_qualifier(self.text(ctor)); if !class_name.is_empty() { let from = self.top_row(); @@ -1492,8 +1548,9 @@ impl<'t> Walker<'t> { return; } // functionTypes is EMPTY for C#; the literal halt list applies — - // lambda_expression IS C#'s lambda, so initializer lambdas stop the - // scan; anonymous_method_expression is NOT listed and scans through. + // lambda_expression IS C#'s lambda, so a lambda the scan reaches (a + // top-level statement's) stops it; anonymous_method_expression is NOT + // listed and scans through. Member initializers are walked instead. if depth > 0 && matches!( node.kind(), @@ -1643,8 +1700,9 @@ impl<'t> Walker<'t> { } /// propertyBodies (tree-sitter.ts) — the parts of a property that run code: -/// each accessor's `body` (a block or `=> expr`) and an expression-bodied -/// property's `=> …` value. An `= initializer` value is not a body. +/// each accessor's `body` (a block or `=> expr`), then the property's +/// `value` — an expression body's `=> …` or an `= initializer`. Attributes +/// are not walked. fn property_bodies(node: Node) -> Vec { let mut bodies = Vec::new(); if let Some(accessors) = node.child_by_field_name("accessors") { @@ -1659,13 +1717,15 @@ fn property_bodies(node: Node) -> Vec { } } if let Some(value) = node.child_by_field_name("value") { - if value.kind() == "arrow_expression_clause" { - bodies.push(value); - } + bodies.push(value); } bodies } +fn last_named_child(node: Node) -> Option { + node.named_child(node.named_child_count().checked_sub(1)?) +} + fn find_anonymous_class_body(node: Node) -> Option { for i in 0..node.named_child_count() { if let Some(child) = node.named_child(i) { diff --git a/docs/design/csharp-kernel-port-checklist.md b/docs/design/csharp-kernel-port-checklist.md index e817968c95..919e767687 100644 --- a/docs/design/csharp-kernel-port-checklist.md +++ b/docs/design/csharp-kernel-port-checklist.md @@ -234,8 +234,8 @@ all PRESERVE): | `enum_declaration` | enumTypes:1064 → extractEnum:1914 | body `enum_member_declaration_list` required (bodiless → no node); extractInheritance sees `base_list` → **the underlying type `: byte` emits an `extends` ref named `byte`** (quirk, §inheritance); `enum_member_declaration` children → extractEnumMembers:1958 — `name` field path: ONE `enum_member` node per member, positioned at the member node (attributes included in its span), values/attributes ignored; non-member children (preproc_*, comment) → visitNode (no-op) | | `method_declaration` | methodTypes:1027 → extractMethod:1737 | classifyMethodNode absent → always extractMethod. Gate 1747 passes via class-like (a method_declaration outside a type does not occur in non-erroring C# — top-level `void M(){}` parses as local_function_statement, probed); bodyless interface/partial signatures mint nodes with no body walk; **expression-bodied methods have `body: arrow_expression_clause` (a real body FIELD, probed) → walked** | | `constructor_declaration` | methodTypes → extractMethod | name field = the class-name identifier → **method node named like the class**; returnType undefined; **`constructor_initializer` (`: base(args)` / `: this(args)`) is a sibling of the body field → NEVER walked → calls inside initializer args are LOST** (probed); expression-bodied ctor body = arrow_expression_clause → walked | -| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`) and an expression-bodied `=> expr` `value:` — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; changed 2026-10-04, previously never walked), then scanFnRefSubtree (capture-only, attributed to the class) over the rest — **the `= initializer` stays unwalked** — skipping the walked bodies, + skipChildren. §property below | -| `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator + scanFnRefSubtree + skipChildren → **field initializers emit no calls/instantiates/static-member refs** (fn-ref candidates only). §field below | +| `property_declaration` (inside class-like) | propertyTypes:1075 → extractProperty:1986 | property node, then **propertyBodies — each accessor's `body` (`get { … }`, `set => …`), then the `value:` (an expression body's `=> expr` or an `= initializer`) — walked by visitFunctionBody with the property pushed** (calls, instantiates, static reads and fn-ref candidates attribute to the property; accessor/arrow bodies changed 2026-10-04, `= initializer` 2026-10-05 — previously never walked); a target-typed `= new()` value instantiates the declared type (extractTargetTypedNew); then scanFnRefSubtree (capture-only, attributed to the class) over the rest (attributes, type) skipping the walked bodies, + skipChildren. §property below | +| `field_declaration` (inside class-like) | fieldTypes:1084 → extractField:2046 | field/constant nodes per declarator, **each declarator walked whole by visitFunctionBody with its field pushed** (its `= initializer`: calls, instantiates, static reads, fn-ref candidates incl. varinit attribute to the field; changed 2026-10-05, previously never walked); a target-typed `= new()` instantiates the declared type; then scanFnRefSubtree (capture-only, attributed to the class) skipping the walked declarators, + skipChildren. §field below | | `local_declaration_statement` | variableTypes:1098 (only reachable at top level — global statements; body locals go through visitFunctionBody instead) | not class-like → extractVariable:2538 → **generic fallback (2863-2881) finds no direct `identifier`/`variable_declarator` children (the declarator nests inside `variable_declaration`, probed) → ZERO nodes minted**; isClassScopeConstantAssignment (1508) needs node.type `assignment` → never true. skipChildren=true + scanFnRefSubtree → **a top-level `var builder = WebApplication.CreateBuilder(args);` produces NO node, NO calls ref, NO instantiates** — only fn-ref candidates. PRESERVE | | `using_directive` | importTypes:1209 → extractImport:3170 | hook (§config) → import node + ONE generic `imports` ref {fromNodeId: nodeStack top (namespace node if present, else file), referenceName: moduleName, line/col of the directive}; **no per-binding emitter** (the TS/py/rust/php/ruby ladder at 3197-3234 excludes csharp) | | `invocation_expression` (top level — global statements) | callTypes:1248 → extractCall:3684 | fires with caller = file/namespace node; children still visited (no skipChildren) so nested invocations recurse | @@ -306,9 +306,10 @@ for C#) or bare `name`. QUIRKS (probed, PRESERVE): children [modifier, predefined_type, identifier, **arrow_expression_clause (`value:` field)**] → typeNode = predefined_type → signature `"int Computed"`; the arrow clause is a property body (walked as the property, below). -- `{ get; } = new();` initializers: the `value:` implicit_object_creation is - NOT a body → no refs, no instantiates (candidates-only scan, attributed to - the class); the accessor_list is excluded from the type scan. +- `{ get; } = new();` initializers: the `value:` is the + implicit_object_creation_expression itself (no type child) → walked as a + body, and extractTargetTypedNew emits `instantiates` named for the + declared `type` (below); the accessor_list is excluded from the type scan. Then extractDecoratorsFor (no-op) and **extractTypeAnnotations (2037) → extractCsharpTypeRefs** — the `type` field IS walked for refs (so `public @@ -316,10 +317,22 @@ List Items` emits references `List` + `Foo` even though the signature kept the raw text). The returned node is pushed while propertyBodies (tree-sitter.ts; csharp.rs `property_bodies`) are walked: every `accessor_declaration`'s `body` field (block or arrow_expression_clause, in -accessor order), then the property's `value:` when it is an -arrow_expression_clause. The fn-ref scan that follows skips those subtrees -by node id, so a candidate is captured once — from the property when it sits -in a body, from the class when it sits in an initializer. (The +accessor order), then the property's `value:` whatever it is (an +arrow_expression_clause or an `= initializer` expression). Each body is +preceded by **extractTargetTypedNew(body, `type` field)** (csharp.rs +`extract_target_typed_new`): when the body IS an +`implicit_object_creation_expression` it emits ONE `instantiates` ref from +the property at the `new()`'s position, named by csharpClassTypeName +(csharp.rs `class_type_name`): `identifier` → its text; `generic_name` → its +identifier (`List` → `List`); `qualified_name` / `alias_qualified_name` +→ recurse into the `name` field (`Ns.Foo` → `Foo`, `global::Foo` → +`Foo`, `Outer.Inner` → `Inner`); `nullable_type` → recurse into `type` +(`Foo?` → `Foo`); anything else (predefined/array/tuple/pointer) → nothing. +Only a `new()` that IS the value counts — `(new())`, `c ? new() : null` and +nested `new()`s emit nothing. The fn-ref scan that follows skips the walked +bodies by node id, so a candidate is captured once — from the property; only +what is left (attribute arguments, e.g. `[Attr(Make(H))]` → `H` from the +class) is the class's. Attributes are never walked for calls. (The classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) ### extractField (2046) — field_declaration @@ -342,6 +355,23 @@ classifyMethodNode initializer-walk path at 1031-1047 is TS-only.) the variable_declaration's `type` field (5905-5909) → **multi-declarator fields (`Foo A, B;`) emit the type refs ONCE PER DECLARATOR**, each from its own field node. +- Then, per created field node (changed 2026-10-05 — initializers used to + emit nothing but fn-ref candidates, attributed to the class): push the + field, **extractTargetTypedNew(the declarator's LAST named child, the + variable_declaration's `type` field)** (§property above — a target-typed + `private readonly List _items = new();` instantiates `List`), then + **visitFunctionBody(the whole declarator)**, pop. The C# declarator has no + `value` field — the initializer is its last, unnamed child (probed: + [name, expression]; a fixed buffer is [name, bracketed_argument_list]) — + so the declarator is walked whole, like VB.NET's: maybeCaptureFnRefs fires + on it first (varinit: `Del d = Handler;` → candidate `Handler` FROM THE + FIELD), then the name (inert) and the initializer — calls, instantiates, + static reads (`Defaults.Name` → `Defaults`) and lambda bodies (no halt: + `Func f = x => Compute(x)` → the field calls `Compute`) attribute + to the field. extractField returns the walked declarator ids and the + dispatcher's scanFnRefSubtree skips them, so each candidate is captured + once, by the field. (Java's walk of its `value` field reports nothing back + — its class-level scan still captures those candidates too, unchanged.) - docstring/visibility/isStatic computed once from the outer declaration, shared by all declarators. The PHP property_element and bare-fallback branches (2078-2154) are unreachable for C#. @@ -425,16 +455,20 @@ statements at top level) AND visitFunctionBody:5145. QUIRKS, PRESERVE: **`implicit_object_creation_expression` (`new()`) and `anonymous_object_creation_expression` (`new { X = 1 }`) and `array_creation_expression` (`new Widget[10]`) are NOT in INSTANTIATION_KINDS -→ no instantiates refs** (target-typed `new()` — everywhere in modern C# — is -invisible); object/collection initializer args and `new[] { Mk() }` contents -still recurse to their own calls. Top-level `var w = new Widget();` emits -nothing at all (§local_declaration_statement). +→ no instantiates refs** (a target-typed `new()` in a body — `Widget c = +new();`, an argument, a return — is invisible); the one exception is a +`new()` that IS a field's or property's initializer, which +extractTargetTypedNew names for the declared type (§property, §field). +Object/collection initializer args and `new[] { Mk() }` contents still +recurse to their own calls. Top-level `var w = new Widget();` emits nothing +at all (§local_declaration_statement). ### extractStaticMemberRef (4750) — csharp ∈ STATIC_MEMBER_LANGS (345) Called for EVERY node in visitFunctionBody (5218) — body walker only (never -visitNode, so class-level field initializers and top-level statements emit no -static refs). Node gate: MEMBER_ACCESS_TYPES (323) contains +visitNode, so top-level statements emit no static refs; field and property +initializers ARE body-walked, so `= Defaults.Name` reads attribute to the +member). Node gate: MEMBER_ACCESS_TYPES (323) contains `member_access_expression`. Skip when the access IS a call's callee (4772-4779: parent ∈ callTypes && callee.startIndex === node.startIndex — so `Console.WriteLine(…)`'s access is skipped but `DoThing(Constants.MAX)`'s @@ -621,14 +655,17 @@ layers: `argument`→null (descend named children). special: nothing (not idTypes/special). - Capture fires from visitNode:990, visitFunctionBody:5137, and scanFnRefSubtree (property/field/variable declarations + top-level - statements, depth ≤12, capture-only). **scanFnRefSubtree's halt list - (tree-sitter.ts:606-612) includes the literal type `lambda_expression` — - which IS C#'s lambda node** — so at depth>0 the scan STOPS at a lambda in a - field/property initializer (`Action A = () => Register(H);` yields no - candidates from inside the lambda), while `anonymous_method_expression` - (`delegate() { … }`) is NOT in the list and is scanned through. Method-BODY - lambdas are unaffected (visitFunctionBody recursion has no halt — capture - fires per node). + statements, depth ≤12, capture-only — skipping the subtrees the body + walker already went through: property bodies/initializers and field + declarators, whose candidates are the member's). **scanFnRefSubtree's halt + list (tree-sitter.ts:606-612) includes the literal type `lambda_expression` + — which IS C#'s lambda node** — so at depth>0 the scan STOPS at a lambda it + reaches (a top-level statement's), while `anonymous_method_expression` + (`delegate() { … }`) is NOT in the list and is scanned through. Body-walked + lambdas — methods, accessors, and since 2026-10-05 field/property + initializers (`Action A = () => Register(H);` → candidate `H` from `A`) — + are unaffected (visitFunctionBody recursion has no halt — capture fires per + node). - Flush gate (flushFnRefCandidates:639): generated-file skip; `this.`-prefixed names skip the gate (C# never produces them — its this-forms are bare); otherwise name ∈ definedHere ∪ importedNames. **definedHere = same-file @@ -762,10 +799,14 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. `struct Fwd;` (NO node); interface with bodyless method + property + default-impl arrow method; enum with `: byte` (extends `byte` quirk), attributed member, valued members; const + static-readonly (→ `constant`) - + multi-declarator + instance fields (signatures `Type name`); + + multi-declarator + instance fields (signatures `Type name`), field + initializers (→ the field: `= new() { TargetCb }` instantiates the declared + `List` and the list candidate is the field's); `protected internal` (→ protected); property shapes: predefined-type, bare-identifier type (signature loses type), generic type, expression-bodied - (`=>` calls → the property), `{ get; } = new();` (initializer LOST), + (`=>` calls → the property), `{ get; } = new();` (→ the property, + instantiates the declared `Widget`), `{ get; } = () => Register(H)` (the + lambda's call and candidate → the property), accessor bodies with calls (→ the property); event_field_declaration + event_declaration with add/remove bodies (no nodes; accessor calls → class); operator + conversion operator + indexer + @@ -779,7 +820,7 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. `Foo.Create(1).Bar()` (re-encode + inner both) + `GetThing().Bar()`, `(myDel)(x)` (conv regex → `myDel`), `nameof(Widget)` (calls ref `nameof`); `new Widget(…) { … }` (instantiates + initializer calls) + `new Ns.Foo()` - (strip both) + `new()` / `new { }` / `new Widget[10]` (all NOTHING); + (strip both) + body `new()` / `new { }` / `new Widget[10]` (all NOTHING); static value reads (`ReadType.ReadAsDouble` → `ReadType`; `Outer.Inner.DEEP` → `Outer`; skip-as-callee; lowercase skip); type refs: params (nullable/array/tuple-element/generic/qualified/`dynamic`), returns @@ -792,7 +833,8 @@ AspNetCore refs / Program.cs / Startup.cs / controller-source scan. function + trailing `partial class Program`; fn-refs: `Register(HandleThing)`, `Register(this.HandleThing)` (bare name), `Register(C.StaticHandler)` (nothing), `Click += OnClick`, initializer_expression list, varinit - (`Action g = () => …` no candidate; `Del d = Handler;` candidate), + (`Action g = () => …` no candidate; `Del d = Handler;` candidate — from + the method for a local, from the field for a field initializer), `this.x = x` param-storage skip; value-refs: const target + reader methods + a `var MaxItems = …` local shadow (prune) + `static readonly` multi-target; preprocessor: `#region`/`#endregion`/`#pragma`/`#nullable`/`#define` diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 9a10862475..222291f14c 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -239,6 +239,29 @@ function scalaBaseTypeName(node: SyntaxNode | null, source: string): string | nu } } +/** + * The class a C# declared type names, as `new T()` would name it: `Foo`, + * `List` → `List`, `Ns.Foo` / `global::Foo` → `Foo`, `Foo?` → `Foo`. + * Predefined, array, tuple and pointer types name no class → null. Mirrored + * in the kernel (csharp.rs class_type_name). + */ +function csharpClassTypeName(node: SyntaxNode | null, source: string): string | null { + if (!node) return null; + switch (node.type) { + case 'identifier': + return getNodeText(node, source) || null; + case 'generic_name': + return csharpClassTypeName(node.namedChildren.find((c: SyntaxNode) => c.type === 'identifier') ?? null, source); + case 'qualified_name': + case 'alias_qualified_name': + return csharpClassTypeName(getChildByField(node, 'name'), source); + case 'nullable_type': + return csharpClassTypeName(getChildByField(node, 'type'), source); + default: + return null; + } +} + /** * Resolve the declared identifier inside a C declarator. A `declaration`'s * `declarator` field nests the name through `init_declarator` (with value), @@ -1287,22 +1310,27 @@ export class TreeSitterExtractor { const bodies = propNode ? this.propertyBodies(node) : []; if (propNode && bodies.length > 0) { this.nodeStack.push(propNode.id); - for (const body of bodies) this.visitFunctionBody(body, propNode.id); + const declaredType = getChildByField(node, 'type'); + for (const body of bodies) { + this.extractTargetTypedNew(body, declaredType); + this.visitFunctionBody(body, propNode.id); + } this.nodeStack.pop(); } - // Whatever the body walk didn't cover (a C# `= initializer`, any other - // language's whole declaration) is scanned for function-as-value + // Whatever the body walk didn't cover (a C# property's attributes, any + // other language's whole declaration) is scanned for function-as-value // candidates (#756); the bodies captured their own. this.scanFnRefSubtree(node, 0, new Set(bodies.map((b) => b.id))); skipChildren = true; } // Check for class fields (e.g. Java field_declaration, C# field_declaration) else if (this.extractor.fieldTypes?.includes(nodeType) && this.isInsideClassLikeNode()) { - this.extractField(node); - // Field initializers aren't walked — scan for function-as-value - // candidates (#756): Java `List table = List.of(Main::cb)`, - // C# `List> table = new() { TargetCb }`. - this.scanFnRefSubtree(node, 0); + const walked = this.extractField(node); + // Scan the declaration for function-as-value candidates (#756): Java + // `List table = List.of(Main::cb)`. A C# declarator + // extractField walked captured its own (`List> table = + // new() { TargetCb }` is the field's), so the scan skips it. + this.scanFnRefSubtree(node, 0, walked); skipChildren = true; } // Check for variable declarations (const, let, var, etc.) @@ -2324,9 +2352,9 @@ export class TreeSitterExtractor { * VB.NET writes its `Get` / `Set` blocks, `= initializer` and `As New T` * as children of the declaration itself, so the declaration is walked * whole, the way its methods are (resolveBody). C# runs code in each - * accessor's body (`get { … }`, `set => …`) and in an expression-bodied - * property's `=> …`; its `= initializer`, like a field's, stays unwalked. - * Mirrored in the kernel (csharp.rs property_bodies). + * accessor's body (`get { … }`, `set => …`) and in the property's `value`: + * an expression body's `=> …` or an `= initializer`. Its attributes are + * not walked. Mirrored in the kernel (csharp.rs property_bodies). */ private propertyBodies(node: SyntaxNode): SyntaxNode[] { if (this.language === 'vbnet') return [node]; @@ -2337,16 +2365,41 @@ export class TreeSitterExtractor { if (body) bodies.push(body); } const value = getChildByField(node, 'value'); - if (value?.type === 'arrow_expression_clause') bodies.push(value); + if (value) bodies.push(value); return bodies; } + /** + * A C# target-typed `new()` names no type, which is why INSTANTIATION_KINDS + * leaves `implicit_object_creation_expression` out. As a field's or + * property's initializer, though, it constructs the declared type: + * `private readonly List _items = new();` instantiates List, as + * `new List()` does. Emitted from the node-stack top (the member). + * Mirrored in the kernel (csharp.rs extract_target_typed_new). + */ + private extractTargetTypedNew(value: SyntaxNode | null, declaredType: SyntaxNode | null): void { + if (this.language !== 'csharp' || value?.type !== 'implicit_object_creation_expression') return; + const className = declaredType ? csharpClassTypeName(declaredType, this.source) : null; + const fromNodeId = this.nodeStack[this.nodeStack.length - 1]; + if (!className || !fromNodeId) return; + this.unresolvedReferences.push({ + fromNodeId, + referenceName: className, + referenceKind: 'instantiates', + line: value.startPosition.row + 1, + column: value.startPosition.column, + }); + } + /** * Extract a class field declaration (e.g. Java field_declaration, C# field_declaration). * Extracts each declarator as a 'field' kind node inside the owning class. + * Returns the C# declarators it walked, for the function-as-value scan to + * skip. */ - private extractField(node: SyntaxNode): void { - if (!this.extractor) return; + private extractField(node: SyntaxNode): Set { + const walked = new Set(); + if (!this.extractor) return walked; const docstring = this.docstringFor(node); const visibility = this.extractor.getVisibility?.(node); @@ -2401,7 +2454,7 @@ export class TreeSitterExtractor { isStatic, }); } - return; + return walked; } } @@ -2447,11 +2500,24 @@ export class TreeSitterExtractor { // edge at all and `target` looked callerless. Keyed on the `value` // FIELD, which only Java's `variable_declarator` carries. VB.NET // writes `= expr` (the declarator's `initializer`) or `As New T(…)` - // (inside its as_clause), so its whole declarator is walked. C# - // and PHP spell their initializer differently and are untouched. - const valueNode = this.language === 'vbnet' ? decl : getChildByField(decl, 'value'); + // (inside its as_clause), and C# writes `= expr` as the + // declarator's last, unnamed child, so both walk the whole + // declarator. PHP spells its initializer differently and is + // untouched. + const wholeDeclarator = this.language === 'vbnet' || this.language === 'csharp'; + const valueNode = wholeDeclarator ? decl : getChildByField(decl, 'value'); if (valueNode) { this.nodeStack.push(fieldNode.id); + if (this.language === 'csharp') { + this.extractTargetTypedNew( + decl.namedChild(decl.namedChildCount - 1), + varDecl ? getChildByField(varDecl, 'type') : null, + ); + // Its function-as-value candidates are the field's, captured + // here once. (VB.NET captures none; Java's scan still takes + // its initializers for the class as well.) + walked.add(decl.id); + } this.visitFunctionBody(valueNode, fieldNode.id); this.nodeStack.pop(); } @@ -2470,6 +2536,7 @@ export class TreeSitterExtractor { }); } } + return walked; } /** diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index 2984a43464..f721a2cc43 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -2438,10 +2438,14 @@ function isDotNetTypeRef(ref: UnresolvedRef, context: ResolutionContext): boolea * method (a constructor is one) or an enum case shares the type's name, not * its meaning: AutoMapper's `Type sourceType` bound to an attribute's `Type` * property and `TypeMap typeMap` to a `TypeMap` property beside the `TypeMap` - * class. A field never names a type either. + * class. A field never names a type either, nor does a constant — the kind a + * C# `const` / `static readonly` field gets: jellyfin's `new Version(5, 18)` + * bound to a claim-name `const string Version`, and serilog's `static + * readonly Meter Meter = new(…)` to itself. */ function canNameInTypePosition(n: Node): boolean { - return !(n.kind === 'property' || n.kind === 'method' || n.kind === 'enum_member' || n.kind === 'field'); + return !(n.kind === 'property' || n.kind === 'method' || n.kind === 'enum_member' || n.kind === 'field' || + n.kind === 'constant'); } /** From c9944147bbd579869a6f3eee3cbc3533e3167f44 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:32:53 +0000 Subject: [PATCH 03/23] fix(vbnet): resolve receiver calls by declared type; guard .NET names, bare calls and ties (#2351) VB.NET's resolver guessed a receiver call's target from the receiver's name whenever no class matched it, without any of the guards C# has. Indexing Structure members (#2345) gave those guesses new targets: 23 SCrawler `xxxFile.Delete()` calls on PersonalUtilities' SFile went to a nested struct's `TempFileConversion::Delete`, and staxrip's main app sent 90 calls to its AutoCrop tool's duplicate `ColorHSL`. - Declared types (new src/resolution/vbnet-receivers.ts): a receiver's `As` type decides the call (local, parameter, field, property, `For Each`, `As New`, ...). The candidates are that type's member, one it inherits, or an extension method written for it. A type from outside the project gets no link. - .NET standard method names (`Add`, `Contains`, `Clear`, `Dispose`, `ToString`, ...) on an untyped receiver need a receiver named after the owner, as in C#. VB.NET matches them case-insensitively. - A bare call, or one on `Me`, reaches the enclosing class, what it inherits, or a Module. A nested type is matched by its bare name only from inside its owner. Types resolve through enclosing namespaces, `Imports` (aliases included) and the caller's project. A tie between equally good guesses goes to the caller's file, then its project, then the nearer directory, or gets no link. - vbReceiverOf reads the name at or after the reference's column, so `Me.Size = New System.Drawing.Size(...)` is read through `System.Drawing`, not `Me`. Before/after on main (nodes unchanged): SCrawler 15,092 -> 14,893 edges, staxrip 28,002 -> 26,647. Of the non-containment edges removed, 855 / 2,802 are the same link with new resolver metadata. The rest: - .NET names on untyped receivers no longer guessed: 196 / 1,378. 731 of staxrip's are StringBuilder `sb.Append` calls that had gone to `LogBuilder::Append`. - Other receiver guesses dropped: 86 / 103, including the SFile.Delete calls. - Bare calls out of scope: 38 / 834. 590 of staxrip's are `New Point(...)` that had gone to a class nested in `ButtonEx`; others are WinForms `Refresh()`, `Focus()` and `Activate()`. - Retargeted: 227 / 532, e.g. 208 bare `Add(...)` calls in encoder classes now reach the inherited `CommandLineParams::Add`. 121 / 958 call sites resolve that did not before, e.g. `cms.Add` -> ContextMenuStripEx and `td.AddButton` -> TaskDialog. Designer -> Size links fall from 5 / 1 to 0; designer -> Add stays 0. Known trade-off, shared with C#'s guard: about 31 staxrip calls like `switches.Join(BR)` that may be the project's own `MiscExtensions` extension methods are no longer linked. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 5 + __tests__/vbnet-bare-member-scope.test.ts | 100 ++ __tests__/vbnet-member-access.test.ts | 35 + __tests__/vbnet-project-ties.test.ts | 143 +++ __tests__/vbnet-receiver-types.test.ts | 465 +++++++ __tests__/vbnet-std-methods.test.ts | 75 ++ src/resolution/index.ts | 2 + src/resolution/name-matcher.ts | 81 +- src/resolution/vbnet-receivers.ts | 1335 +++++++++++++++++++++ 9 files changed, 2235 insertions(+), 6 deletions(-) create mode 100644 __tests__/vbnet-bare-member-scope.test.ts create mode 100644 __tests__/vbnet-project-ties.test.ts create mode 100644 __tests__/vbnet-receiver-types.test.ts create mode 100644 __tests__/vbnet-std-methods.test.ts create mode 100644 src/resolution/vbnet-receivers.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index da460b5b77..b5476b128f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -19,6 +19,11 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - In VB.NET, a field or property initializer like `= Compute()` or `As New List(Of Order)` now links what it calls and creates, and so do a `Custom Event`'s `AddHandler`, `RemoveHandler` and `RaiseEvent` blocks. Re-index VB.NET and C# projects after upgrading. - In C#, a field or property initializer like `private readonly ILogger _log = LogManager.GetLogger(typeof(X));` or `public List Items { get; } = new();` now links what it calls, creates and reads, including inside a lambda, and a target-typed `new()` there counts as creating the declared type. Before, initializers were skipped, so a method or class used only from one — like a converter created in a static list — looked unused, and a method passed as a value there was credited to the whole class instead of the field or property. - In C# and VB.NET, a constant no longer stands in for a type with the same name: `new Station { … }` links to class `Station` even when another class declares `private const string Station`, `new Version(…)` no longer links to a `const string Version`, and a field like `static readonly Meter Meter` no longer points at itself. Thanks @EvanYu1980 for the report and @drakeo338. (#2337) +- In VB.NET, a call on a variable, parameter, field or property now reaches a method of the type it is declared with — one that type inherits, or an extension method written for it — and nothing when that type comes from outside your project, instead of any project method that merely shares the name. +- In VB.NET, a call to one of .NET's own methods such as `Add`, `Contains`, `Clear` or `Dispose` on a value whose type isn't known is no longer linked to a project method of that name, unless the value is named after that method's class. +- In VB.NET, a type name is looked up the way VB.NET does it — through the namespaces around it, the file's and project's `Imports` (aliases included), and the caller's own project — so a class declared in several namespaces or projects no longer draws every call to whichever copy was indexed first, and two candidates nothing tells apart get no link at all. +- In VB.NET, a call with no receiver, or on `Me`, now reaches only a member of the class it is written in, of a class it inherits, or of a Module, and no longer a nearby class's member of the same name. +- VB.NET designer code such as `New System.Drawing.Point(…)` or `Me.Size = New System.Drawing.Size(…)` is no longer linked to a project type or member that only shares the name, and a class nested inside another is matched by its bare name only from inside that class or one that inherits it. - Upgrading CodeGraph while an agent session is open no longer leaves the old version's background server in charge of your project: the first session started from the new install stops it and starts a current one in its place, even while sessions opened before the upgrade are still running. That old server could no longer load the language parsers the upgrade removed, so it saved every file it re-indexed with no symbols, while new sessions could only read the index beside it without keeping it up to date. A background server from a newer install is never stopped, and sessions opened before the upgrade keep the old version until you restart them. Thanks @lipchey for the report. (#2335) - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) diff --git a/__tests__/vbnet-bare-member-scope.test.ts b/__tests__/vbnet-bare-member-scope.test.ts new file mode 100644 index 0000000000..a2601671b1 --- /dev/null +++ b/__tests__/vbnet-bare-member-scope.test.ts @@ -0,0 +1,100 @@ +/** + * A VB.NET call with no receiver — or with `Me` / `MyClass` / `MyBase`, + * which the extractor drops — is a member of the type it is written in, of a + * type around it, or of one they inherit; a `Module`'s members are reached + * from anywhere. Never another class's same-named member, however near its + * file: SCrawler's `{ToString()}` in the Instagram `UserData` (which + * `Inherits UserDataBase`) went to a nearby structure's `ToString`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-bare-')); + const files: Record = { + 'API/Base/UserDataBase.vb': `Namespace API.Base + Friend MustInherit Class UserDataBase + Public Overrides Function ToString() As String + Return "user" + End Function + Protected Sub Refresh() + End Sub + End Class +End Namespace +`, + 'API/Instagram/MediaItem.vb': `Namespace API.Instagram + Friend Class MediaItem + Public Overrides Function ToString() As String + Return "media" + End Function + Friend Sub Refresh() + End Sub + End Class +End Namespace +`, + 'API/Instagram/UserData.vb': `Namespace API.Instagram + Friend Class UserData : Inherits API.Base.UserDataBase + Friend Sub SetTagsLimit() + Dim aStr$ = $"Enter the number of posts from user {ToString()}" + Me.Refresh() + Log(aStr) + End Sub + Private Class Counter + Friend Sub Tick() + Report() + End Sub + End Class + Private Shared Sub Report() + End Sub + End Class +End Namespace +`, + 'Tools/Logger.vb': `Public Module Logger + Public Sub Log(ByVal Text As String) + End Sub +End Module +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of one method. */ +function callsOf(file: string, qualifiedName: string): string[] { + const from = cg.getNodesInFile(file).find((n) => n.qualifiedName === qualifiedName)!; + return cg + .getOutgoingEdgesFrom([from.id]) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET receiver-less calls', () => { + it('reach the members the class inherits, not a nearer class’s, and a module’s', () => { + expect(callsOf('API/Instagram/UserData.vb', 'API.Instagram::UserData::SetTagsLimit')).toEqual([ + 'API.Base::UserDataBase::Refresh', + 'API.Base::UserDataBase::ToString', + 'Logger::Log', + ]); + }); + + it('reach a member of the class around a nested one', () => { + expect(callsOf('API/Instagram/UserData.vb', 'API.Instagram::UserData::Counter::Tick')).toEqual([ + 'API.Instagram::UserData::Report', + ]); + }); +}); diff --git a/__tests__/vbnet-member-access.test.ts b/__tests__/vbnet-member-access.test.ts index d8643ecc52..668cc18aac 100644 --- a/__tests__/vbnet-member-access.test.ts +++ b/__tests__/vbnet-member-access.test.ts @@ -7,6 +7,12 @@ * (`GetService(Of Notifier).Notify()`). SCrawler's designer code sent * `New System.Drawing.Size(…)` to a nested enum's `Size` case 713 times and * `Controls.Add(…)` to a collection class's `Add` 547 times. + * + * The name read is the one the reference starts at (or after) — in + * `Me.Size = New System.Drawing.Size(…)` that is `System.Drawing.Size`, not + * `Me.Size` — and a type is one its written qualifier names: staxrip's + * designer `New System.Drawing.Point(…)` went to a nested `Point` class 550 + * times. A type nested in a class is named bare only inside it. */ import { describe, it, expect, afterAll, beforeAll } from 'vitest'; import * as fs from 'fs'; @@ -56,6 +62,24 @@ End Module Private Sub Refresh() End Sub End Class +`, + 'UI/ButtonEx.vb': `Public Class ButtonEx + Public Class SymbolDrawer + Public Class Point + End Class + Public Sub Draw() + Dim p As New Point() + End Sub + End Class +End Class +`, + 'Forms/AppsForm.vb': `Public Class AppsForm + Private Sub InitializeComponent() + Me.Size = New System.Drawing.Size(315, 205) + Me.ToolStrip.Location = New System.Drawing.Point(473, 10) + Me.lDescription.Location = New Point(4, 285) + End Sub +End Class `, }; for (const [rel, content] of Object.entries(files)) { @@ -84,4 +108,15 @@ describe('VB.NET member access', () => { expect(targets).toContain('Notifier::Notify'); expect(targets).toContain('MainForm::Refresh'); }); + + it('reads the name the reference starts at, and a type its qualifier names', () => { + const targetsOf = (file: string) => cg + .getOutgoingEdgesFrom(cg.getNodesInFile(file).map((n) => n.id)) + .filter((e) => e.kind === 'calls' || e.kind === 'instantiates') + .map((e) => `${e.kind} ${cg.getNode(e.target)!.qualifiedName}`) + .sort(); + // `Me.Size = New System.Drawing.Size(…)`, `New System.Drawing.Point(…)`, `New Point(…)` outside ButtonEx. + expect(targetsOf('Forms/AppsForm.vb')).toEqual([]); + expect(targetsOf('UI/ButtonEx.vb')).toEqual(['instantiates ButtonEx::SymbolDrawer::Point']); + }); }); diff --git a/__tests__/vbnet-project-ties.test.ts b/__tests__/vbnet-project-ties.test.ts new file mode 100644 index 0000000000..9d8e2fcc0e --- /dev/null +++ b/__tests__/vbnet-project-ties.test.ts @@ -0,0 +1,143 @@ +/** + * A VB.NET solution often carries the same type in two projects: staxrip's + * main app and its separate AutoCrop tool (its own AutoCrop.vbproj) each + * declare `ColorHSL`, `FrameServerFactory` and `DirectFrameServer`. A call + * in the main app means its own project's copy — about 90 calls such as + * `_backColor.AddLuminance(0.025)` went to the AutoCrop copy, whichever was + * indexed first. Between equally good guesses, the caller's project decides, + * then the nearer directory; when neither does, there is no guess. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const VBPROJ = ` + + WinExe + + +`; + +const COLOR_HSL = `Public Class ColorHSL + Public Sub New(ByVal h As Double, ByVal s As Double, ByVal l As Double, ByVal a As Double) + End Sub + Public Function AddLuminance(ByVal offset As Single) As ColorHSL + Return Me + End Function +End Class +`; + +const FRAME_SERVER_FACTORY = `Public Class FrameServerFactory + Public Shared Function Create(ByVal path As String) As Object + Return Nothing + End Function +End Class +`; + +const HELPER = `Public Class Helper + Public Sub Run() + End Sub +End Class +`; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-ties-')); + const files: Record = { + 'Source/StaxRip.vbproj': VBPROJ, + 'Source/Tools/AutoCrop/AutoCrop.vbproj': VBPROJ, + 'Source/Tools/AutoCrop/Main.vb': COLOR_HSL + FRAME_SERVER_FACTORY, + 'Source/UI/ColorHSL.vb': COLOR_HSL, + 'Source/Video/FrameServer.vb': FRAME_SERVER_FACTORY, + 'Source/General/ThemeManager.vb': `Public Class ThemeManager + Public Sub Apply(ByVal palette As Object) + Dim _backColor As ColorHSL = New ColorHSL(0, 0.01, 0.1, 1) + Dim _controlBackColor As ColorHSL = _backColor.AddLuminance(0.025) + For Each backgroundColor In palette + backgroundColor.AddLuminance(-0.1) + Next + Dim server = FrameServerFactory.Create("video.mkv") + End Sub +End Class +`, + 'Source/UI/TipProvider.vb': `Public Class TipProvider + Public Sub SetTip(ByVal text As String) + End Sub + Public Sub SetTip(ByVal text As String, ByVal title As String) + End Sub +End Class +`, + 'Source/General/Tips.vb': `Public Class Tips + Public Sub Apply(ByVal providers As Object) + For Each tipProvider In providers + tipProvider.SetTip("x") + Next + End Sub +End Class +`, + 'Plugins/A/A.vbproj': VBPROJ, + 'Plugins/A/Helper.vb': HELPER, + 'Plugins/B/B.vbproj': VBPROJ, + 'Plugins/B/Helper.vb': HELPER, + 'App/App.vbproj': VBPROJ, + 'App/Main.vb': `Public Class Main + Public Sub Start(ByVal helpers As Object) + For Each runHelper In helpers + runHelper.Run() + Next + End Sub +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member @ file` of every call / instantiation edge out of a file. */ +function targetsFrom(file: string, kind: 'calls' | 'instantiates'): string[] { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === kind) + .map((e) => { + const target = cg.getNode(e.target)!; + return `${target.qualifiedName} @ ${target.filePath}`; + }) + .sort(); +} + +describe('VB.NET duplicate types across projects', () => { + it('resolve a call to the caller’s own project’s copy', () => { + expect(targetsFrom('Source/General/ThemeManager.vb', 'calls')).toEqual([ + 'ColorHSL::AddLuminance @ Source/UI/ColorHSL.vb', + 'ColorHSL::AddLuminance @ Source/UI/ColorHSL.vb', + 'FrameServerFactory::Create @ Source/Video/FrameServer.vb', + ]); + }); + + it('resolve a construction to the caller’s own project’s copy', () => { + expect(targetsFrom('Source/General/ThemeManager.vb', 'instantiates')).toEqual([ + 'ColorHSL @ Source/UI/ColorHSL.vb', + ]); + }); + + it('make no guess between copies nothing tells apart', () => { + expect(targetsFrom('App/Main.vb', 'calls')).toEqual([]); + }); + + it('take one type’s overloads as one guess, not a tie', () => { + expect(targetsFrom('Source/General/Tips.vb', 'calls')).toEqual(['TipProvider::SetTip @ Source/UI/TipProvider.vb']); + }); +}); diff --git a/__tests__/vbnet-receiver-types.test.ts b/__tests__/vbnet-receiver-types.test.ts new file mode 100644 index 0000000000..0a81309747 --- /dev/null +++ b/__tests__/vbnet-receiver-types.test.ts @@ -0,0 +1,465 @@ +/** + * A VB.NET call through a local, a parameter, a field or a property is a call + * on the type it is declared with — `Dim x As T`, `ByVal x As T`, + * `x As New T`, `Private File As SFile`, `Property File As SFile`, a type + * character (`Dim name$`), or what `Dim x = obj.GetString(…)` returns — + * never a guess at a same-named method of some project class. Keywords are + * matched without regard to case. A declared type the project does not define + * means nothing of the project's, except an extension method declared for it. + * + * - SCrawler's `ThumbnailFile.Delete(SFO.File, …)` on an `SFile` (from the + * external PersonalUtilities library) went to a nested `TempFileConversion` + * class's `Delete`, `GroupFile.Delete()` to a download group's, + * `UserUpdatedEventHandlers.Add(e)` on a `List(Of …)` to `UserDataBind.Add` + * and `TotalSize.CompareTo(…)` on a `Double` to a plugin's `VSize.CompareTo`; + * - staxrip's `timestampFontColorValue.ToColor(…)`, a String from + * `settings.GetString(…)`, went to a `ColorHSL.ToColor()` instead of the + * String extension `StringExtensions.ToColor`. + * + * The type a name means is the one VB.NET's lookup finds where it is written: + * the namespaces around it (SCrawler declares a `SiteSettings` and an `M3U8` + * per site namespace), a base class's nested types, the file's `Imports` + * aliases, and the type arguments a subclass gives its base. The declared + * type wins over whatever is assigned later, a `For Each` variable is an + * element of what it loops over, and an interface a class implements lends + * it no members to read bare. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-receiver-')); + const files: Record = { + 'Download/Groups/DownloadGroup.vb': `Namespace DownloadObjects.Groups + Friend Class DownloadGroup + Friend Sub Delete() + End Sub + End Class +End Namespace +`, + 'YouTube/Objects/YouTubeMediaContainerBase.vb': `Namespace API.YouTube.Objects + Public MustInherit Class YouTubeMediaContainerBase + Public ReadOnly Property ThumbnailFile As SFile + Private Class TempFileConversion + Friend Sub Delete(ByVal Mode As Integer) + End Sub + End Class + End Class +End Namespace +`, + 'Download/Groups/DownloadGroupCollection.vb': `Namespace DownloadObjects.Groups + Friend Class DownloadGroupCollection + Private ReadOnly GroupFile As SFile = "Settings\\Groups.xml" + Friend Sub Update() + GroupFile.Delete() + End Sub + Friend Sub Reset(byval groupFile as SFile) + groupFile.Delete() + End Sub + End Class +End Namespace +`, + 'Hosts/DownloadableMediaHost.vb': `Namespace Hosts + Friend Class DownloadableMediaHost : Inherits API.YouTube.Objects.YouTubeMediaContainerBase + Friend Sub Clean() + ThumbnailFile.Delete(1) + End Sub + End Class +End Namespace +`, + 'API/UserDataBind.vb': `Namespace API + Friend Class UserDataBind + Friend Sub Add(ByVal User As Object) + End Sub + End Class +End Namespace +`, + 'API/Base/UserDataBase.vb': `Namespace API.Base + Friend MustInherit Class UserDataBase + Private ReadOnly UserUpdatedEventHandlers As List(Of UserUpdatedEventHandler) + Friend Sub AddUpdateHandler(ByVal e As UserUpdatedEventHandler) + UserUpdatedEventHandlers.Add(e) + End Sub + Friend Sub Download() + End Sub + End Class +End Namespace +`, + 'API/Instagram/UserData.vb': `Namespace API.Instagram + Friend Class UserData : Inherits API.Base.UserDataBase + End Class +End Namespace +`, + 'API/Instagram/Downloader.vb': `Namespace API.Instagram + Friend Class Downloader + Friend Sub Run(ByVal u As UserData) + u.Download() + End Sub + End Class +End Namespace +`, + 'Plugin/UserData.vb': `Friend Class VSize + Public Function CompareTo(ByVal Other As VSize) As Integer + Return 0 + End Function +End Class +`, + 'Editors/UsersInfoForm.vb': `Friend Class UsersInfoForm + Private NotInheritable Class UserOpt + Friend Property TotalSize As Double = 0 + Friend Function CompareTo(ByVal Other As UserOpt) As Integer + Return TotalSize.CompareTo(Other.TotalSize) * -1 + End Function + End Class +End Class +`, + 'UI/ColorHSL.vb': `Public Class ColorHSL + Public Sub New(ByVal h As Double, ByVal s As Double, ByVal l As Double, ByVal a As Double) + End Sub + Public Function AddLuminance(ByVal offset As Single) As ColorHSL + Return Me + End Function + Public Function ToColor() As Integer + Return 0 + End Function +End Class +`, + 'UI/BackColorAdjuster.vb': `Public Class BackColorAdjuster + Public Function AddLuminance(ByVal offset As Single) As Integer + Return 0 + End Function +End Class +`, + 'UI/ThemeManager.vb': `Public Class ThemeManager + Public Sub Apply() + Dim _backColor As ColorHSL = New ColorHSL(0, 0.01, 0.1, 1) + Dim _controlBackColor As ColorHSL = _backColor.AddLuminance(0.025) + End Sub +End Class +`, + 'UI/Popup.vb': `Public Class Popup + Public Sub Show() + End Sub +End Class +`, + 'UI/MainForm.vb': `Public Class MainForm + Inherits Form + + Public Sub OpenSettings() + Dim settingsForm As New MainForm + settingsForm.Show() + End Sub +End Class +`, + 'General/Extensions.vb': `Imports System.Runtime.CompilerServices + +Module StringExtensions + + Function ToColor(ByVal str As String, Optional ByVal defaultColor As Integer = 0) As Integer + Return 0 + End Function + + Function Join(ByVal instance As IEnumerable(Of String), ByVal delimiter As String) As String + Return "" + End Function + + Function Sort(Of T)(ByVal instance As IEnumerable(Of T)) As IEnumerable(Of T) + Return instance + End Function +End Module +`, + 'General/ObjectStorage.vb': `Public Class ObjectStorage + Public Function GetString(ByVal key As String, Optional ByVal defaultValue As String = Nothing) As String + Return defaultValue + End Function +End Class +`, + 'General/Thumbnailer.vb': `Public Class Thumbnailer + Public Sub Run(ByVal settings As ObjectStorage) + Dim timestampFontColorValue = settings.GetString("TimestampFontColor", "#fff") + Dim timestampFontColor = timestampFontColorValue.ToColor(1) + Dim colorText As String = "#000" + Dim outline = colorText.ToColor() + Dim name$ = "x" + Dim named = name.ToColor() + End Sub +End Class +`, + 'General/Lists.vb': `Public Class Lists + Public Sub Run() + Dim names As New List(Of String) + names.Sort() + Dim joined = names.Join(", ") + End Sub +End Class +`, + 'Base/DownDetector.vb': `Namespace API.Base + Friend NotInheritable Class DownDetector + Friend MustInherit Class Checker(Of T) + Protected ReadOnly Property Source As T + End Class + End Class +End Namespace +`, + 'Sites/Bluesky/SiteSettings.vb': `Namespace API.Bluesky + Friend Class SiteSettings + Friend Function IsMyUser(ByVal url As String) As Boolean + Return False + End Function + Friend Function AvailableTrueValue() As Boolean + Return False + End Function + End Class + Friend NotInheritable Class M3U8 + Friend Shared Sub Download(ByVal url As String) + End Sub + End Class +End Namespace +`, + 'Sites/Reddit/SiteSettings.vb': `Namespace API.Reddit + Friend Class SiteSettings + Friend Function IsMyUser(ByVal url As String) As Boolean + Return True + End Function + Friend Function AvailableTrueValue() As Boolean + Return True + End Function + Private Class MyDownDetector : Inherits API.Base.DownDetector.Checker(Of SiteSettings) + Friend Sub Check() + Source.AvailableTrueValue() + End Sub + End Class + End Class + Friend NotInheritable Class M3U8 + Friend Shared Sub Download(ByVal url As String) + End Sub + End Class +End Namespace +`, + 'Sites/Reddit/UserData.vb': `Namespace API.Reddit + Friend Class UserData + Private ReadOnly Property MySettings As SiteSettings + Friend Sub Check(ByVal url As String) + MySettings.IsMyUser(url) + M3U8.Download(url) + End Sub + End Class +End Namespace +`, + 'Encoding/VideoEncoder.vb': `Public MustInherit Class VideoEncoder + Public Class MenuList + Public Sub Add(ByVal text As String) + End Sub + End Class +End Class +Public Class BatchEncoder + Inherits VideoEncoder + Public Function GetMenu() As Object + Dim ret As New MenuList + ret.Add("Codec Configuration") + Return ret + End Function +End Class +`, + 'YouTube/YouTubeFunctions.vb': `Friend Interface IContainer + Sub Parse() +End Interface +Friend Class Channel : Implements IContainer + Friend Sub Parse() Implements IContainer.Parse + End Sub +End Class +Friend Module YouTubeFunctions + Friend Sub Load() + Dim item As IContainer + item = New Channel + item.Parse() + End Sub +End Module +`, + 'UI/Labels.vb': `Public Class LabelUI + Public Function AddLabel(ByVal text As String) As Object + Return Nothing + End Function +End Class +Public Class SimpleUI + Public Function AddLabel(ByVal text As String) As Object + Return Nothing + End Function + Public Function AddLabel(ByVal text As String, ByVal width As Integer) As Object + Return Nothing + End Function +End Class +Public Class SimpleSettingsForm + Friend WithEvents SimpleUI As SimpleUI +End Class +`, + 'UI/AudioForm.vb': `Public Class AudioForm + Public Sub ShowAdvanced() + Using form As New SimpleSettingsForm() + Dim ui = form.SimpleUI + ui.AddLabel("EBU R128") + End Using + End Sub +End Class +`, + 'UI/Theme.vb': `Public Class ButtonLabel + Public Sub ApplyTheme() + End Sub +End Class +Public Class ToggleButtonLabel + Public Sub ApplyTheme() + End Sub +End Class +Public Class ThemeApplier + Public Sub Apply(ByVal controls As Object) + For Each control In controls.OfType(Of ToggleButtonLabel) + control.ApplyTheme() + Next + End Sub + Public Sub ApplyAll(ByVal labels As List(Of ToggleButtonLabel)) + For Each label In labels + label.ApplyTheme() + Next + End Sub +End Class +`, + 'Download/TDownloader.vb': `Namespace App.Download + Friend Class TDownloader + Friend Class Job + Friend Sub Start() + End Sub + End Class + End Class + Friend Class Worker + Friend Sub Start() + End Sub + End Class +End Namespace +`, + 'Download/DownloadProgress.vb': `Imports TDJob = App.Download.TDownloader.Job + +Namespace App.Download + Friend Class DownloadProgress + Friend ReadOnly Property Worker As TDJob + Friend Sub Run() + Worker.Start() + End Sub + End Class +End Namespace +`, + 'Plugin/IPluginContentProvider.vb': `Namespace Plugin + Public Interface ISiteSettings + End Interface + Public Interface IPluginContentProvider + Property Settings As ISiteSettings + End Interface +End Namespace +`, + 'MainMod.vb': `Friend Module MainMod + Friend Settings As SettingsCLS +End Module +`, + 'SettingsCLS.vb': `Friend Class SettingsCLS + Friend Sub UpdateUsersList() + End Sub +End Class +`, + 'API/Base/UserDataProvider.vb': `Namespace API.Base + Friend MustInherit Class UserDataProvider : Implements Plugin.IPluginContentProvider + Friend Property MySettings As Plugin.ISiteSettings Implements Plugin.IPluginContentProvider.Settings + Friend Sub Delete() + Settings.UpdateUsersList() + End Sub + End Class +End Namespace +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of a file. */ +function callsFrom(file: string): string[] { + const ids = cg.getNodesInFile(file).map((n) => n.id); + return cg + .getOutgoingEdgesFrom(ids) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET calls through declared receivers', () => { + it('reach nothing of the project on a type it does not define', () => { + expect(callsFrom('Download/Groups/DownloadGroupCollection.vb')).toEqual([]); + expect(callsFrom('Hosts/DownloadableMediaHost.vb')).toEqual([]); + expect(callsFrom('API/Base/UserDataBase.vb')).toEqual([]); + }); + + it('reach nothing of the project on a VB.NET built-in type', () => { + expect(callsFrom('Editors/UsersInfoForm.vb')).toEqual([]); + }); + + it('reach the declared type’s own method, or the one it inherits', () => { + expect(callsFrom('UI/ThemeManager.vb')).toEqual(['ColorHSL::AddLuminance']); + expect(callsFrom('API/Instagram/Downloader.vb')).toEqual(['API.Base::UserDataBase::Download']); + }); + + it('reach nothing of the project for a method the declared type inherits from outside it', () => { + expect(callsFrom('UI/MainForm.vb')).toEqual([]); + }); + + it('reach an extension method declared for the type, through what a call returns too', () => { + expect(callsFrom('General/Thumbnailer.vb')).toEqual([ + 'ObjectStorage::GetString', + 'StringExtensions::ToColor', + 'StringExtensions::ToColor', + 'StringExtensions::ToColor', + ]); + }); + + it('reach a .NET type’s own method before an extension of that name, and an extension it lacks', () => { + // `List(Of T)` has its own `Sort()`, but no `Join`. + expect(callsFrom('General/Lists.vb')).toEqual(['StringExtensions::Join']); + }); + + it('name the type their declaration’s namespace sees, through a base class’s type argument too', () => { + // SCrawler declares a `SiteSettings` and an `M3U8` in each site's namespace. + expect(callsFrom('Sites/Reddit/UserData.vb')).toEqual([ + 'API.Reddit::M3U8::Download', + 'API.Reddit::SiteSettings::IsMyUser', + ]); + // `Inherits DownDetector.Checker(Of SiteSettings)` makes its `Source As T` a SiteSettings. + expect(callsFrom('Sites/Reddit/SiteSettings.vb')).toEqual(['API.Reddit::SiteSettings::AvailableTrueValue']); + // A type nested in a base class is named bare in a subclass: staxrip's `Dim ret As New MenuList`. + expect(callsFrom('Encoding/VideoEncoder.vb')).toEqual(['VideoEncoder::MenuList::Add']); + }); + + it('take the declared type over what is assigned, and an import alias for what it names', () => { + expect(callsFrom('YouTube/YouTubeFunctions.vb')).toEqual(['IContainer::Parse']); + expect(callsFrom('Download/DownloadProgress.vb')).toEqual(['App.Download::TDownloader::Job::Start']); + }); + + it('read a field through what a call returns, and a loop variable through what it loops over', () => { + expect(callsFrom('UI/AudioForm.vb')).toEqual(['SimpleUI::AddLabel']); + // `For Each control In controls.OfType(Of ToggleButtonLabel)`, `For Each label In labels` over a `List(Of ToggleButtonLabel)`. + expect(callsFrom('UI/Theme.vb')).toEqual(['ToggleButtonLabel::ApplyTheme', 'ToggleButtonLabel::ApplyTheme']); + }); + + it('see a module’s field, not the property of an interface the class implements', () => { + expect(callsFrom('API/Base/UserDataProvider.vb')).toEqual(['SettingsCLS::UpdateUsersList']); + }); +}); diff --git a/__tests__/vbnet-std-methods.test.ts b/__tests__/vbnet-std-methods.test.ts new file mode 100644 index 0000000000..8c29b1e775 --- /dev/null +++ b/__tests__/vbnet-std-methods.test.ts @@ -0,0 +1,75 @@ +/** + * VB.NET runs on the same .NET base library as C#, so a call such as + * `x.Contains(…)`, `x.Add(…)` or `x.Dispose()` on a receiver whose type is not + * known is far more likely the library's method than the one project method + * that happens to share the name. As in C#, such a guess is kept only when the + * receiver is named after the method's owner. staxrip's + * `SupportedInput.Contains(ret)` — a String array — went to `VideoScript`'s + * `Contains`. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-std-')); + const files: Record = { + 'Video/VideoScript.vb': `Public Class VideoScript + Public Function Contains(ByVal value As String) As Boolean + Return False + End Function +End Class +`, + 'Audio/AudioProfile.vb': `Public Class AudioProfile + Public Function IsInputSupported(ByVal inputs As Object, ByVal ext As String) As Boolean + For Each supported In inputs + If supported.Contains(ext) Then Return True + Next + Return False + End Function + + Public Function HasScript(ByVal scripts As Object, ByVal name As String) As Boolean + For Each script In scripts + If script.Contains(name) Then Return True + Next + Return False + End Function +End Class +`, + }; + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +/** `Owner::member` of every call edge out of one method. */ +function callsOf(qualifiedName: string): string[] { + const from = cg.getNodesInFile('Audio/AudioProfile.vb').find((n) => n.qualifiedName === qualifiedName)!; + return cg + .getOutgoingEdgesFrom([from.id]) + .filter((e) => e.kind === 'calls') + .map((e) => cg.getNode(e.target)!.qualifiedName) + .sort(); +} + +describe('VB.NET standard-library method names', () => { + it('are not a project method on a receiver that does not name its owner', () => { + expect(callsOf('AudioProfile::IsInputSupported')).toEqual([]); + }); + + it('still reach the project method when the receiver names its owner', () => { + expect(callsOf('AudioProfile::HasScript')).toEqual(['VideoScript::Contains']); + }); +}); diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 1eaf14b145..4c9fb45f2c 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -25,6 +25,7 @@ import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, is import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; +import { clearVbnetReceiverMemos } from './vbnet-receivers'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; @@ -457,6 +458,7 @@ export class ReferenceResolver { clearNameMatcherMemos(this.context); clearCppMacroVisibility(this.context); clearSwiftTypeVisibility(this.context); + clearVbnetReceiverMemos(this.context); clearTypeParameterMemos(this.context); } } diff --git a/src/resolution/name-matcher.ts b/src/resolution/name-matcher.ts index f721a2cc43..0ba9c946d4 100644 --- a/src/resolution/name-matcher.ts +++ b/src/resolution/name-matcher.ts @@ -11,6 +11,7 @@ import { UnresolvedRef, ResolvedRef, ResolutionContext, ImportMapping, isSuperty import { blankStringContents, stripCommentsForRegex } from './strip-comments'; import { JS_BUILT_INS, JS_BUILTIN_METHODS, TS_PRIMITIVE_TYPES } from './js-builtins'; import { SWIFT_TYPE_PATH_CALL, resolveSwiftTypePathCall } from './swift-type-visibility'; +import { breakVbTie, isVbMemberInScope, isVbNestedTypeInScope, isVbTypeQualifiedBy, matchVbTypedCall, preferVbProject, sameVbProject } from './vbnet-receivers'; import { isTestPath } from '../search/query-utils'; import { isMinifiedContent } from '../extraction/generated-detection'; import { getCargoWorkspaceCrateMap } from './frameworks/cargo-workspace'; @@ -3112,6 +3113,9 @@ const CSHARP_STD_METHODS: ReadonlySet = new Set([ 'ForEach', 'Sort', 'Reverse', 'Clone', 'Seek', 'SetLength', ]); +/** The same .NET names as VB.NET writes them — in any case. */ +const VBNET_STD_METHODS: ReadonlySet = new Set([...CSHARP_STD_METHODS].map((m) => m.toLowerCase())); + /** * A request handler the web framework dispatches to: a Django / DRF / Flask * view's `get` / `post` / …, a controller's `index` / `store` / `update` / @@ -3156,11 +3160,17 @@ function stdMethodNames(language: string): ReadonlySet | null { case 'rust': return RUST_STD_METHODS; case 'kotlin': return KOTLIN_STD_METHODS; case 'csharp': return CSHARP_STD_METHODS; + case 'vbnet': return VBNET_STD_METHODS; case 'dart': return DART_STD_METHODS; default: return null; } } +/** Whether `name` is one of `language`'s standard-library method names (VB.NET's in any case). */ +function isStdMethodName(language: string, name: string): boolean { + return stdMethodNames(language)?.has(language === 'vbnet' ? name.toLowerCase() : name) ?? false; +} + /** Methods of Dart's String, List, Iterable, Map and Set — names a project type rarely carries itself. */ const DART_STD_METHODS: ReadonlySet = new Set([ 'endsWith', 'startsWith', 'contains', 'split', 'substring', 'trim', 'trimLeft', 'trimRight', 'toLowerCase', @@ -4657,7 +4667,13 @@ function vbReceiverOf(ref: UnresolvedRef, context: ResolutionContext): string | const name = ref.referenceName.toLowerCase(); let start = lower.startsWith(name, ref.column) ? ref.column : -1; if (start < 0) { - const m = new RegExp(`(?>>(); /** @@ -5536,6 +5568,8 @@ export function matchByExactName( const cfmlBare = (ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const vbReceiver = ref.language === 'vbnet' && (ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && /^\w+$/.test(ref.referenceName) ? vbReceiverOf(ref, context) : null; + const vbScoped = isVbScopedCall(ref, vbReceiver, context); + const vbUnqualified = isVbUnqualifiedName(ref, vbReceiver, context); const objcShape = ref.language === 'objc' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*:*(?:\w+:)*$/.test(ref.referenceName) ? objcCallShape(ref, context) : null; const csharpBare = ref.language === 'csharp' && (ref.referenceKind === 'calls' || ref.referenceKind === 'references') && /^[A-Za-z_]\w*$/.test(ref.referenceName); @@ -5557,6 +5591,9 @@ export function matchByExactName( !(objcShape === 'self-send' && !isObjcSelfSendTarget(n, ref, context)) && !(objcShape === 'super-send' && !isObjcSelfSendTarget(n, ref, context, true)) && !(vbReceiver !== null && !isVbMemberReachable(n, vbReceiver)) && + !(vbReceiver && !/^(?:me|mybase|myclass)$/i.test(vbReceiver) && !isVbTypeQualifiedBy(n, vbReceiver, ref.filePath, context)) && + !(vbScoped && !isVbMemberInScope(n, ref, context)) && + !(vbUnqualified && !isVbNestedTypeInScope(n, ref, context)) && !(rubyBare && n.kind === 'method' && !isRubyMethodInScope(n, ref, context)) && !(cfmlBare && n.kind === 'method' && !isCfmlMethodInScope(n, ref, context)) && !(javaBare && n.kind === 'method' && !isJavaMethodInScope(n, ref, context)) && @@ -7817,6 +7854,14 @@ export function matchMethodCall( // shared source-based inferrer. resolveMethodOnType validates the method // exists on the inferred type, so a mis-inference produces no edge. if (inferableReceiver) { + // A VB.NET receiver's declared type decides the call, or that there is no + // project method to call: SCrawler's `ThumbnailFile.Delete(…)` on an + // external `SFile` went to a nested class's `Delete` by a shared word. + if (ref.language === 'vbnet' && dotMatch) { + const typed = nmTimedT('mc-vbtyped', ref, () => + matchVbTypedCall(objectOrClass!, methodName!, ref, context, (name) => isStdMethodName('vbnet', name))); + if (typed !== undefined) return typed; + } let inferredType = nmTimedT('mc-infer', ref, () => ref.language === 'cpp' ? inferCppReceiverType(objectOrClass!, ref, context) @@ -8024,6 +8069,9 @@ export function matchMethodCall( const visible = classCandidates.filter((c) => c.language !== 'csharp' || isCsharpTypeVisible(c, typeRef, context)); classCandidates = [...visible, ...classCandidates.filter((c) => !visible.includes(c))]; } + // A VB.NET type declared in two projects is the caller's own project's: + // staxrip's `FrameServerFactory.Create(…)` went to its AutoCrop tool's copy. + if (ref.language === 'vbnet') classCandidates = preferVbProject(classCandidates, ref, context); for (const classNode of classCandidates) { // Skip cross-language class matches @@ -8213,10 +8261,10 @@ export function matchMethodCall( // `json` of its `MockedResponse`. !isUnnamedTestDouble(targetMethods[0]!, objectOrClass!, ref, context) && !((ref.language === 'lua' || ref.language === 'luau') && isLuaLibraryCall(objectOrClass!, methodName!, ref, targetMethods[0]!)) && - // Rust / Go / Kotlin / C#: a standard-library method name on an - // untyped receiver (`sym.map(…)`, `w.Header().Get(…)`, + // Rust / Go / Kotlin / C# / VB.NET: a standard-library method name on + // an untyped receiver (`sym.map(…)`, `w.Header().Get(…)`, // `reader.Value.ToString()`) is the library type's. - !(stdMethodNames(ref.language)?.has(methodName!) && + !(isStdMethodName(ref.language, methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!) && !receiverNamesOwner(receiverLink(objectOrClass!), targetMethods[0]!, context)) && !(UNTYPED_RECEIVER_LANGUAGES.has(ref.language) && !/^(?:self|self\.class|this|super|weak_?self|strong_?self)$/i.test(objectOrClass!) && !sharesReceiverWord(objectOrClass!, targetMethods[0]!) && @@ -8238,11 +8286,12 @@ export function matchMethodCall( const head = receiverWords[receiverWords.length - 1]?.toLowerCase(); let bestMatch: typeof targetMethods[0] | undefined; let bestScore = 0; + let tied: typeof targetMethods = []; // Same-file candidates first, so a score tie (`score > bestScore` keeps // the first seen) resolves to the call site's own file rather than the // first-indexed duplicate (#1079). - const std = stdMethodNames(ref.language)?.has(methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!); + const std = isStdMethodName(ref.language, methodName!) && !/^(?:self|Self|this|base)$/.test(objectOrClass!); for (const method of preferCallSiteFile(targetMethods, ref.filePath)) { if (std && !receiverNamesOwner(receiverLink(objectOrClass!), method, context)) continue; // The owner type's own name — not its namespace (`eShop.ClientApp…` @@ -8265,8 +8314,16 @@ export function matchMethodCall( if (score > bestScore) { bestScore = score; bestMatch = method; + tied = [method]; + } else if (score === bestScore) { + tied.push(method); } } + // VB.NET: between equally good guesses, the caller's own file, then its + // project, then the nearer directory — and no guess when none of them + // decides. staxrip's main app and its AutoCrop tool each declare a + // `ColorHSL`, and the first indexed took about 90 of the app's calls. + if (ref.language === 'vbnet' && tied.length > 1 && bestScore >= 2) bestMatch = breakVbTie(tied, ref, context) ?? undefined; // A wrapper handing its call on — BookStack's `FileStorage::delete` doing // `$storage->delete($path)`, `CommentRepo::delete` doing @@ -9229,7 +9286,7 @@ function computePathProximity(filePath1: string, filePath2: string): number { function findBestMatch( ref: UnresolvedRef, candidates: Node[], - _context: ResolutionContext + context: ResolutionContext ): Node | null { // Prioritization rules: // 1. Same file > different file @@ -9269,6 +9326,13 @@ function findBestMatch( // Directory proximity bonus — strongly prefer same module/package score += pathProximityFromDirs(refDirs, candidate.filePath); + // A VB.NET project compiles its own files: the caller's project weighs as + // much as the nearest a directory can be. staxrip's `New ColorHSL(…)` went + // to its AutoCrop tool's copy. + if (ref.language === 'vbnet' && candidate.language === 'vbnet' && sameVbProject(candidate.filePath, ref.filePath, context)) { + score += 80; + } + // Language matching: strongly prefer same language, penalize cross-language if (candidate.language === ref.language) { score += 50; @@ -9353,6 +9417,8 @@ export function matchFuzzy( const cfmlBare = (ref.language === 'cfml' || ref.language === 'cfscript') && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*$/.test(ref.referenceName); const vbReceiver = ref.language === 'vbnet' && (ref.referenceKind === 'calls' || ref.referenceKind === 'instantiates') && /^\w+$/.test(ref.referenceName) ? vbReceiverOf(ref, context) : null; + const vbScoped = isVbScopedCall(ref, vbReceiver, context); + const vbUnqualified = isVbUnqualifiedName(ref, vbReceiver, context); const objcShape = ref.language === 'objc' && ref.referenceKind === 'calls' && /^[A-Za-z_]\w*:*(?:\w+:)*$/.test(ref.referenceName) ? objcCallShape(ref, context) : null; const csharpBare = ref.language === 'csharp' && (ref.referenceKind === 'calls' || ref.referenceKind === 'references') && /^[A-Za-z_]\w*$/.test(ref.referenceName); @@ -9388,6 +9454,9 @@ export function matchFuzzy( !(rubyBare && n.kind === 'method' && !isRubyMethodInScope(n, ref, context)) && !(cfmlBare && n.kind === 'method' && !isCfmlMethodInScope(n, ref, context)) && !(vbReceiver !== null && !isVbMemberReachable(n, vbReceiver)) && + !(vbReceiver && !/^(?:me|mybase|myclass)$/i.test(vbReceiver) && !isVbTypeQualifiedBy(n, vbReceiver, ref.filePath, context)) && + !(vbScoped && !isVbMemberInScope(n, ref, context)) && + !(vbUnqualified && !isVbNestedTypeInScope(n, ref, context)) && !(objcShape === 'c-call' && OBJC_MEMBER_KINDS.has(n.kind)) && !(objcShape === 'self-send' && !isObjcSelfSendTarget(n, ref, context)) && !(objcShape === 'super-send' && !isObjcSelfSendTarget(n, ref, context, true)) && diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts new file mode 100644 index 0000000000..07732927e1 --- /dev/null +++ b/src/resolution/vbnet-receivers.ts @@ -0,0 +1,1335 @@ +/** + * VB.NET receivers, projects and receiver-less calls. + * + * The name matcher guesses a `receiver.Method()` call it cannot type by the + * method's name alone (matchMethodCall's Strategy 3: a unique name, or the + * receiver sharing words with the owner), and a receiver-less call by the + * nearest same-named member. VB.NET brought none of the evidence C# does to + * those guesses, so a call landed on whichever project method shared its + * name: SCrawler's `ThumbnailFile.Delete(…)` on an external `SFile` went to a + * nested `TempFileConversion.Delete`, staxrip's main app called its AutoCrop + * tool's copy of `ColorHSL`. What this module reads, from the source: + * + * 1. A receiver's declared type — `Dim x As T`, `ByVal x As T`, `x As New T`, + * a type character (`Dim name$`), `Dim x = New T(…)`, a `For Each` over a + * typed collection, a field or property of the class, of one it inherits + * (with the type arguments the subclass gives) or of a `Module`, or what + * `Dim x = obj.GetString(…)` returns. A typed receiver's call is the + * type's own method or one it inherits, else an extension method declared + * for the type, else nothing of the project's: a type the project does + * not define (`SFile`, `String`, `List(Of T)`) has none of its methods. + * 2. Which type a name means where it is written: the namespaces around it, + * outward, then its file's and project's `Imports` (aliases included), + * then the project a file belongs to — the `.vbproj` above it, whose + * `RootNamespace` its namespaces are inside. Between same-named types, and + * between guesses nothing else tells apart, the caller's own project's + * comes first; a written qualifier (`System.Drawing.Point`) must match. + * 3. A receiver-less call (or one on `Me` / `MyClass` / `MyBase`, which the + * extractor drops) is a member of a type around it, of what those inherit, + * or of a `Module`; a type nested in a class is named bare only inside it. + * + * VB.NET's keywords and names are matched without regard to case. Nothing + * here runs for another language. + */ +import * as fs from 'fs'; +import * as path from 'path'; +import type { Node } from '../types'; +import type { ResolutionContext, ResolvedRef, UnresolvedRef } from './types'; + +/** A VB.NET declared type: its simple name (no namespace, no type arguments) and whether it is an array. */ +interface VbType { + name: string; + array: boolean; + /** The namespaces or types written before the name (`API.Base` in `API.Base.UserDataBase`), lowercased. */ + qualifier?: string[]; + /** Where the name is written, which decides the namespaces it is looked up in. */ + file?: string; + line?: number; + /** A supertype named in an `Implements` statement: its members are not the type's own. */ + implemented?: boolean; + /** The type arguments written with it: `SiteSettings` in `Checker(Of SiteSettings)`. */ + args?: VbType[]; +} + +/** What a statement says a local is: its type, the call whose result it holds, or a binding that names no type. */ +type VbBinding = + | { kind: 'type'; type: VbType } + | { kind: 'call'; receiver: string | null; member: string } + | { kind: 'each'; collection: string } + | { kind: 'unknown' }; + +/** .NET's collections of one element type: `For Each x In list` over a `List(Of T)` binds a `T`. */ +const VB_ELEMENT_COLLECTIONS = /^(?:List|IList|IEnumerable|ICollection|IReadOnlyList|IReadOnlyCollection|HashSet|SortedSet|Queue|Stack|LinkedList|ObservableCollection|Collection|ReadOnlyCollection|BindingList|ConcurrentBag|ConcurrentQueue|ConcurrentStack|BlockingCollection)$/i; + +/** The type of what a `For Each` over a value of type `t` binds: an array's or a .NET collection's element type. */ +function elementType(t: VbType): VbType | null { + if (t.array) return { ...t, array: false }; + return t.args?.length === 1 && VB_ELEMENT_COLLECTIONS.test(t.name) && t.args[0]!.name !== '?' ? t.args[0]! : null; +} + +/** VB.NET's built-in types, by keyword and by .NET name, keyed to the keyword. */ +const VB_BUILTIN_TYPES: ReadonlyMap = new Map([ + ...['boolean', 'byte', 'char', 'date', 'decimal', 'double', 'integer', 'long', 'object', 'sbyte', 'short', 'single', + 'string', 'uinteger', 'ulong', 'ushort'].map((k): [string, string] => [k, k]), + ['int16', 'short'], ['int32', 'integer'], ['int64', 'long'], ['uint16', 'ushort'], ['uint32', 'uinteger'], + ['uint64', 'ulong'], ['datetime', 'date'], +]); + +/** The type a type character declares: `Dim aStr$`, `For i% = 0 …`. */ +const VB_TYPE_CHARS: Readonly> = { + $: 'String', '%': 'Integer', '&': 'Long', '!': 'Single', '#': 'Double', '@': 'Decimal', +}; + +/** VB.NET's conversion functions and the built-in type each returns. */ +const VB_CONVERSIONS: Readonly> = { + cbool: 'Boolean', cbyte: 'Byte', cchar: 'Char', cdate: 'Date', cdbl: 'Double', cdec: 'Decimal', cint: 'Integer', + clng: 'Long', cobj: 'Object', csbyte: 'SByte', cshort: 'Short', csng: 'Single', cstr: 'String', cuint: 'UInteger', + culng: 'ULong', cushort: 'UShort', +}; + +/** VB.NET type nodes a member can belong to (a `Module` is indexed as a class). */ +const VB_TYPE_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface']); +const VB_VALUE_KINDS: ReadonlySet = new Set(['field', 'property', 'constant', 'variable']); +const VB_MEMBER_KINDS: ReadonlySet = new Set(['method', 'property', 'field', 'enum_member', 'constant', 'variable']); + +/** A declaration keyword right before a name: the name is a member being declared, not a variable. */ +const VB_MEMBER_HEAD = /\b(?:Function|Sub|Property|Event|Operator|Declare|Delegate|Class|Structure|Module|Interface|Enum|Namespace)\s+$/i; + +function escapeRegex(s: string): string { + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); +} + +/** A type's key for comparing two of them: the built-in keyword or the lowercased name, `[]` for an array. */ +function typeKey(t: VbType): string { + const lower = t.name.toLowerCase(); + return `${VB_BUILTIN_TYPES.get(lower) ?? lower}${t.array ? '[]' : ''}`; +} + +function isBuiltin(t: VbType): boolean { + return !t.array && VB_BUILTIN_TYPES.has(t.name.toLowerCase()); +} + +/** The index of the `)` closing the `(` at `open`, or -1. */ +function closeParen(text: string, open: number): number { + let depth = 0; + for (let i = open; i < text.length; i++) { + if (text[i] === '(') depth++; + else if (text[i] === ')' && --depth === 0) return i; + } + return -1; +} + +/** The top-level items of the parenthesized list opening at `open`, or null when it does not close. */ +function splitArgs(text: string, open: number): string[] | null { + const close = closeParen(text, open); + if (close < 0) return null; + const args: string[] = []; + let depth = 0; + let cur = ''; + for (const ch of text.slice(open + 1, close)) { + if (ch === '(' || ch === '{') depth++; + else if (ch === ')' || ch === '}') depth--; + if (ch === ',' && depth === 0) { + args.push(cur); + cur = ''; + } else cur += ch; + } + args.push(cur); + return args.map((a) => a.trim()); +} + +/** + * The type written at `text[at]` — `List(Of Foo)`, `String()`, `Integer?`, + * `Global.A.B` — by its last name; null when no type starts there. + */ +function readType(text: string, at: number): VbType | null { + const head = /^\s*([A-Za-z_][\w.]*)/.exec(text.slice(at)); + if (!head) return null; + let i = at + head[0].length; + let args: VbType[] | undefined; + if (/^\s*\(\s*Of\b/i.test(text.slice(i))) { + const open = text.indexOf('(', i); + const close = closeParen(text, open); + if (close < 0) return null; + args = (splitArgs(text, open) ?? []).map((a) => readType(a.replace(/^Of\s+/i, ''), 0) ?? { name: '?', array: false }); + i = close + 1; + } + const segments = head[1]!.split('.'); + const name = segments.pop()!; + if (!/^[A-Za-z_]\w*$/.test(name) || /^(?:New|Of|As|In|Out|From|With)$/i.test(name)) return null; + const qualifier = segments.map((s) => s.toLowerCase()).filter((s, i) => !(i === 0 && s === 'global')); + return { + name, + array: /^\s*\??\s*\(\s*,*\s*\)/.test(text.slice(i)), + ...(qualifier.length > 0 ? { qualifier } : {}), + ...(args ? { args } : {}), + }; +} + +/** `t` — and the type arguments written inside it — written at `file:line`. */ +function sited(t: VbType | null, file: string, line: number): VbType | null { + return t ? { ...t, file, line, ...(t.args ? { args: t.args.map((a) => sited(a, file, line)!) } : {}) } : null; +} + +/** A VB.NET line's code: its comment dropped and every string literal emptied to `""`. */ +function vbCode(line: string): string { + // Most lines have neither: the line itself, without a copy. + if (!/["'‘’\r]/.test(line)) return line; + let out = ''; + for (let i = 0; i < line.length; i++) { + const ch = line[i]!; + if (ch === '"') { + let j = i + 1; + for (; j < line.length; j++) { + if (line[j] !== '"') continue; + if (line[j + 1] === '"') j++; + else break; + } + out += '""'; + i = j; + continue; + } + if (ch === "'" || ch === '\u2018' || ch === '\u2019' || ch === '\r') break; + out += ch; + } + return out; +} + +const VB_CODE_LINES = new WeakMap>(); + +/** A file's lines as code (see vbCode), read once per file. */ +function codeLines(file: string, context: ResolutionContext): string[] { + let memo = VB_CODE_LINES.get(context); + if (!memo) VB_CODE_LINES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const lines = (context.getFileLines?.(file) ?? context.readFile(file)?.split(/\r?\n/) ?? []).map(vbCode); + if (memo.size >= 1024) memo.delete(memo.keys().next().value!); + memo.set(file, lines); + return lines; +} + +interface VbPatterns { + /** The name standing where a binding names it — not as `name.Member` nor `name(args)`, which only use it. */ + declared: RegExp; + /** `name As T`, `name() As T`, `name As New T`: a variable, parameter, loop or catch variable. */ + asClause: RegExp; + /** `Dim name, other As T`: a declarator sharing the next one's type. */ + listed: RegExp; + /** `Dim name$`, `ByVal name%`, `For i% = …`. */ + typeChar: RegExp; + /** `Dim name = …`: the value's type. */ + inferred: RegExp; + /** `For Each name In …`: the collection's element type, when it says it. */ + forEach: RegExp; + /** A binding that names no type: `For name = …`, `Function(name)`, `From name In`, `Catch name`. */ + loose: RegExp; +} + +const VB_PATTERNS = new Map(); + +function patternsFor(name: string): VbPatterns { + const key = name.toLowerCase(); + const hit = VB_PATTERNS.get(key); + if (hit) return hit; + const r = escapeRegex(key); + const patterns: VbPatterns = { + declared: new RegExp(`(?= 4096) VB_PATTERNS.delete(VB_PATTERNS.keys().next().value!); + VB_PATTERNS.set(key, patterns); + return patterns; +} + +/** + * What every binding of a name has on its line: a declaring keyword, or a + * type character. Most lines naming a variable just use it (`sb.Append(…)`). + */ +const VB_MAY_BIND = /\b(?:As|Dim|Static|Const|For|Each|Function|Sub|Catch|Using|From|Let|Aggregate)\b|\w[$%&!#@]/i; + +/** What one line of code says `name` is, or undefined when it doesn't bind the name. */ +function bindingOn(code: string, name: string): VbBinding | undefined { + if (!VB_MAY_BIND.test(code)) return undefined; + const p = patternsFor(name); + if (!p.declared.test(code)) return undefined; + p.asClause.lastIndex = 0; + for (let m = p.asClause.exec(code); m; m = p.asClause.exec(code)) { + if (VB_MEMBER_HEAD.test(code.slice(0, m.index))) continue; + const type = readType(code, m.index + m[0].length); + if (!type) return { kind: 'unknown' }; + // `As New T(…)`: the parentheses are the constructor's. + return { kind: 'type', type: m[2] ? { ...type, array: false } : { ...type, array: type.array || !!m[1] } }; + } + const listed = p.listed.exec(code); + if (listed) { + const type = readType(code, listed.index + listed[0].length); + return type ? { kind: 'type', type } : { kind: 'unknown' }; + } + const typeChar = p.typeChar.exec(code); + if (typeChar) return { kind: 'type', type: { name: VB_TYPE_CHARS[typeChar[1]!]!, array: !!typeChar[2] } }; + // An assignment (`x = New Channel`) says nothing: the declared type + // (`Dim x As IYouTubeMediaContainer`) is what the call binds to. + const inferred = p.inferred.exec(code); + if (inferred) return valueBinding(code.slice(inferred.index + inferred[0].length)); + // `For Each c In controls.OfType(Of ButtonLabel)`: the loop's elements are + // that type; `For Each user In users`: the elements of what `users` is. + const loop = p.forEach.exec(code); + if (loop) { + const collection = loop[1]!.trim(); + const elements = /\.\s*(?:OfType|Cast)\s*\(\s*Of\s+/i.exec(collection); + const type = elements && /^[^()]*\)\s*(?:\(\s*\))?\s*$/.test(collection.slice(elements.index + elements[0].length)) + ? readType(collection, elements.index + elements[0].length) : null; + if (type) return { kind: 'type', type }; + return /^[A-Za-z_]\w*$/.test(collection) ? { kind: 'each', collection } : { kind: 'unknown' }; + } + return p.loose.test(code) ? { kind: 'unknown' } : undefined; +} + +/** What a local initialized with `expr` (`Dim x = expr`) is. */ +function valueBinding(expr: string): VbBinding { + const e = expr.split(/\s:(?!=)/)[0]!.trim(); + const unknown: VbBinding = { kind: 'unknown' }; + const created = /^New\s+/i.exec(e); + if (created) { + const type = readType(e, created[0].length); + return type ? { kind: 'type', type: { ...type, array: false } } : unknown; + } + const cast = /^(?:DirectCast|TryCast|CType)\s*\(/i.exec(e); + if (cast) { + const args = splitArgs(e, cast[0].length - 1); + const type = args?.length === 2 ? readType(args[1]!, 0) : null; + return type ? { kind: 'type', type } : unknown; + } + if (/^\$?""/.test(e)) return { kind: 'type', type: { name: 'String', array: false } }; + const conversion = /^(C[A-Za-z]+)\s*\(/.exec(e); + const converted = conversion ? VB_CONVERSIONS[conversion[1]!.toLowerCase()] : undefined; + if (converted) return { kind: 'type', type: { name: converted, array: false } }; + // `obj.Member(…)`, `Member(…)`, `Me.Member`: a call or a read as the whole value. + const call = /^(?:([A-Za-z_]\w*)\s*\.\s*)?([A-Za-z_]\w*)\s*/.exec(e); + if (call) { + let rest = e.slice(call[0].length); + if (rest.startsWith('(')) { + const close = closeParen(rest, 0); + rest = close < 0 ? 'unclosed' : rest.slice(close + 1).trim(); + } + if (rest === '') return { kind: 'call', receiver: call[1] ?? null, member: call[2]! }; + } + return unknown; +} + +/** The 0-based line a call's own declarations start on: its member's first line, which holds a method's parameters. */ +function scopeStartLine(ref: UnresolvedRef, context: ResolutionContext): number { + const from = context.getNodeById?.(ref.fromNodeId); + if (from && from.filePath === ref.filePath && from.startLine <= ref.line && from.endLine >= ref.line && + (from.kind === 'method' || from.kind === 'function' || from.kind === 'property' || from.kind === 'field')) { + return from.startLine - 1; + } + let start = -1; + for (const n of context.getNodesInFile(ref.filePath)) { + if (n.kind !== 'method' && n.kind !== 'function' && n.kind !== 'property') continue; + if (n.startLine <= ref.line && n.endLine >= ref.line && n.startLine - 1 > start) start = n.startLine - 1; + } + return start < 0 ? ref.line - 1 : start; +} + +const VB_WORD_LINES = new WeakMap>>(); + +/** + * The 0-based lines of a file's code each lowercased word appears on, in + * order: a call's receiver is looked for only on the lines that name it, + * not on every line back to its method's start (a long method's calls each + * rescanned it). + */ +function wordLines(file: string, context: ResolutionContext): Map { + let memo = VB_WORD_LINES.get(context); + if (!memo) VB_WORD_LINES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const index = new Map(); + const words = /[A-Za-z_]\w*/g; + const lines = codeLines(file, context); + for (let i = 0; i < lines.length; i++) { + const code = lines[i]!.toLowerCase(); + words.lastIndex = 0; + for (let m = words.exec(code); m; m = words.exec(code)) { + let at = index.get(m[0]); + if (!at) index.set(m[0], (at = [])); + if (at[at.length - 1] !== i) at.push(i); + } + } + if (memo.size >= 256) memo.delete(memo.keys().next().value!); + memo.set(file, index); + return index; +} + +const VB_BINDINGS = new WeakMap>>(); + +/** + * Every statement of a file that binds `name`, by 0-based line — read once: + * staxrip's `GetArgs` methods call `sb.Append(…)` on hundreds of lines, each + * of which would otherwise walk back over all the others to the `Dim sb`. + */ +function bindingsOf(name: string, file: string, context: ResolutionContext): Array<{ line: number; binding: VbBinding }> { + let memo = VB_BINDINGS.get(context); + if (!memo) VB_BINDINGS.set(context, (memo = new Map())); + const key = `${file}|${name.toLowerCase()}`; + const hit = memo.get(key); + if (hit) return hit; + const lines = codeLines(file, context); + const found: Array<{ line: number; binding: VbBinding }> = []; + for (const i of wordLines(file, context).get(name.toLowerCase()) ?? []) { + const binding = bindingOn(lines[i]!, name); + if (binding) found.push({ line: i, binding }); + } + if (memo.size >= 65536) memo.delete(memo.keys().next().value!); + memo.set(key, found); + return found; +} + +/** The nearest statement above a call (in its member) that binds `name`, and its 1-based line. */ +function localBinding(name: string, ref: UnresolvedRef, context: ResolutionContext): { binding: VbBinding; line: number } | undefined { + const found = bindingsOf(name, ref.filePath, context); + // The last binding at or above the call's line. + let lo = 0; + let hi = found.length; + while (lo < hi) { + const mid = (lo + hi) >>> 1; + if (found[mid]!.line <= ref.line - 1) lo = mid + 1; + else hi = mid; + } + const nearest = found[lo - 1]; + return nearest && nearest.line >= scopeStartLine(ref, context) ? { binding: nearest.binding, line: nearest.line + 1 } : undefined; +} + +const VB_FILE_TYPES = new WeakMap>(); + +/** The VB.NET types a line is written inside, innermost first. */ +function typesAround(file: string, line: number, context: ResolutionContext): Node[] { + let memo = VB_FILE_TYPES.get(context); + if (!memo) VB_FILE_TYPES.set(context, (memo = new Map())); + let types = memo.get(file); + if (!types) { + types = context.getNodesInFile(file) + .filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)) + .sort((a, b) => b.startLine - a.startLine); + if (memo.size >= 1024) memo.delete(memo.keys().next().value!); + memo.set(file, types); + } + return types.filter((n) => n.startLine <= line && n.endLine >= line); +} + +/** The project's VB.NET types (classes, modules, structures, interfaces) named `name`, case aside. */ +function projectTypesNamed(name: string, context: ResolutionContext): Node[] { + return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); +} + +const VB_MEMBERS = new WeakMap>(); + +/** Members named `name` declared directly in `type` (any partial part of it), case aside. */ +function membersNamed(type: Node, name: string, context: ResolutionContext): Node[] { + let memo = VB_MEMBERS.get(context); + if (!memo) VB_MEMBERS.set(context, (memo = new Map())); + const qn = `${type.qualifiedName}::${name}`.toLowerCase(); + const hit = memo.get(qn); + if (hit) return hit; + const members = context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && n.qualifiedName.toLowerCase() === qn); + if (memo.size >= 65536) memo.delete(memo.keys().next().value!); + memo.set(qn, members); + return members; +} + +const VB_MODULES = new WeakMap>(); + +/** Whether a VB.NET owner (by qualified name) is a `Module`, whose members are reached without a qualifier. */ +function isModule(ownerQn: string, context: ResolutionContext): boolean { + let memo = VB_MODULES.get(context); + if (!memo) VB_MODULES.set(context, (memo = new Map())); + const hit = memo.get(ownerQn); + if (hit !== undefined) return hit; + // Its own head line or the two after it (attributes may come first); the raw + // lines, as the whole file need not be read as code for this. + const module = context.getNodesByQualifiedName(ownerQn).some((n) => n.language === 'vbnet' && n.kind === 'class' && + (context.getFileLines?.(n.filePath) ?? context.readFile(n.filePath)?.split(/\r?\n/) ?? []).slice(n.startLine - 1, n.startLine + 2) + .some((l) => /^\s*(?:<[^>]*>\s*)*(?:(?:Public|Friend|Private|Partial)\s+)*Module\s/i.test(l))); + memo.set(ownerQn, module); + return module; +} + +/** The members named `name` of the project's Modules, which code reaches without a qualifier — the caller's project's first. */ +function moduleMembersNamed(name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] { + const members = context.getNodesByLowerName(name.toLowerCase()).filter((n) => { + const cut = n.qualifiedName.lastIndexOf('::'); + return n.language === 'vbnet' && cut > 0 && isModule(n.qualifiedName.slice(0, cut), context); + }); + return preferVbProject(members, ref, context); +} + +const VB_SUPERS = new WeakMap>(); + +/** + * The types a VB.NET type's declarations (every `Partial` part) inherit or + * implement: the `Inherits` / `Implements` statements that open its body, + * including the `Class X : Inherits Y` form. + */ +function supertypesOf(type: Node, context: ResolutionContext): VbType[] { + let memo = VB_SUPERS.get(context); + if (!memo) VB_SUPERS.set(context, (memo = new Map())); + const key = type.qualifiedName.toLowerCase(); + const hit = memo.get(key); + if (hit) return hit; + const supers: VbType[] = []; + const parts = context.getNodesByQualifiedName(type.qualifiedName).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); + for (const part of parts.length > 0 ? parts : [type]) { + const lines = codeLines(part.filePath, context); + scan: for (let i = part.startLine - 1; i < Math.min(lines.length, part.startLine + 12, part.endLine); i++) { + for (const statement of lines[i]!.split(/:(?!=)/)) { + const s = statement.trim(); + // Blank, an attribute, a directive, or the declaration's own head. + if (s === '' || /^<.*>$/.test(s) || s.startsWith('#') || /\b(?:Class|Structure|Interface|Module)\s+[A-Za-z_]/i.test(s) && !/^(?:Inherits|Implements)\b/i.test(s)) continue; + const clause = /^(Inherits|Implements)\s+(.+)$/i.exec(s); + if (!clause) break scan; + for (const sup of splitArgs(`(${clause[2]})`, 0) ?? []) { + const t = sited(readType(sup, 0), part.filePath, part.startLine); + if (t) supers.push(/^Implements$/i.test(clause[1]!) ? { ...t, implemented: true } : t); + } + } + } + } + memo.set(key, supers); + return supers; +} + +/** A type a VB.NET type inherits, with what its type parameters stand for there (`T` → `SiteSettings`). */ +interface VbAncestor { + node: Node; + args: Map; +} + +const VB_ANCESTRIES = new WeakMap>(); + +/** `t` with a type parameter replaced by what `args` says it stands for. */ +function substitute(t: VbType, args: ReadonlyMap): VbType { + return (!t.qualifier && !t.array && args.get(t.name.toLowerCase())) || t; +} + +/** + * A VB.NET type and the project types it inherits, nearest first: a class's + * base classes, an interface's base interfaces — whose members are the + * type's own. Each carries the type arguments `Inherits Checker(Of + * SiteSettings)` gives its parameters. An interface a class implements lends + * it no members: they are reached through the class's own `… Implements + * IFoo.Bar` ones. + */ +function ancestry(type: Node, context: ResolutionContext): VbAncestor[] { + let memo = VB_ANCESTRIES.get(context); + if (!memo) VB_ANCESTRIES.set(context, (memo = new Map())); + const hit = memo.get(type.id); + if (hit) return hit; + const out: VbAncestor[] = []; + const seen = new Set(); + const queue: VbAncestor[] = [{ node: type, args: new Map() }]; + while (queue.length > 0 && out.length < 16) { + const current = queue.shift()!; + const key = current.node.qualifiedName.toLowerCase(); + if (seen.has(key)) continue; + seen.add(key); + out.push(current); + for (const sup of supertypesOf(current.node, context)) { + if (sup.implemented) continue; + const found = typesNamedAt(sup, context); + if (found.ambiguous) continue; + const given = (sup.args ?? []).map((a) => substitute(a, current.args)); + for (const base of found.owners) { + const params = [...typeParametersOf(base, context).keys()]; + queue.push({ node: base, args: new Map(params.flatMap((p, i): Array<[string, VbType]> => (given[i] ? [[p, given[i]!]] : []))) }); + } + } + } + memo.set(type.id, out); + return out; +} + +/** A VB.NET type and the project types it inherits, nearest first (see ancestry). */ +function hierarchy(type: Node, context: ResolutionContext): Node[] { + return ancestry(type, context).map((a) => a.node); +} + +const VB_PROJECTS = new WeakMap>(); + +/** The directory of the nearest `.vbproj` at or above `dir` (`''` for the root), or null outside any project. */ +function projectOfDir(dir: string, context: ResolutionContext, memo: Map): string | null { + const hit = memo.get(dir); + if (hit !== undefined) return hit; + let found: string | null = null; + try { + if (fs.readdirSync(path.join(context.getProjectRoot(), dir)).some((e) => /\.vbproj$/i.test(e))) found = dir; + } catch { + // Unreadable: no project file here. + } + if (found === null && dir !== '') { + const cut = dir.lastIndexOf('/'); + found = projectOfDir(cut < 0 ? '' : dir.slice(0, cut), context, memo); + } + memo.set(dir, found); + return found; +} + +/** The project (the directory of the nearest `.vbproj` above it) a file belongs to, or null. */ +function projectOf(file: string, context: ResolutionContext): string | null { + let memo = VB_PROJECTS.get(context); + if (!memo) VB_PROJECTS.set(context, (memo = new Map())); + const cut = file.lastIndexOf('/'); + return projectOfDir(cut < 0 ? '' : file.slice(0, cut), context, memo); +} + +/** Whether two files are in the same VB.NET project — false when either is in none. */ +export function sameVbProject(a: string, b: string, context: ResolutionContext): boolean { + const project = projectOf(a, context); + return project !== null && project === projectOf(b, context); +} + +interface VbProjectInfo { + /** The root namespace every file of the project declares its namespaces inside, as segments. */ + root: string[]; + /** The namespaces the project imports into every file (``). */ + imports: string[]; +} + +const VB_PROJECT_INFO = new WeakMap>(); + +/** A file's project's root namespace (``, else the project's name) and project-wide imports, lowercased. */ +function projectInfo(file: string, context: ResolutionContext): VbProjectInfo { + const dir = projectOf(file, context); + if (dir === null) return { root: [], imports: [] }; + let memo = VB_PROJECT_INFO.get(context); + if (!memo) VB_PROJECT_INFO.set(context, (memo = new Map())); + const hit = memo.get(dir); + if (hit) return hit; + const info: VbProjectInfo = { root: [], imports: [] }; + try { + const abs = path.join(context.getProjectRoot(), dir); + const project = fs.readdirSync(abs).find((e) => /\.vbproj$/i.test(e)); + if (project) { + const text = fs.readFileSync(path.join(abs, project), 'utf8'); + const root = /\s*([\w.]*)\s*<\/RootNamespace>/i.exec(text)?.[1] ?? project.replace(/\.vbproj$/i, ''); + info.root = root.toLowerCase().split('.').filter((s) => s !== ''); + for (const m of text.matchAll(/ s.split('.'))]; +} + +/** A file's lines before its first declaration: where its `Option` and `Imports` statements are. */ +function headerLines(file: string, context: ResolutionContext): string[] { + const lines = codeLines(file, context); + const end = lines.findIndex((l) => + /^\s*(?:Namespace|Module|Class|Structure|Interface|Enum|Delegate|Public|Friend|Private|Protected|Partial|NotInheritable|MustInherit)\b/i.test(l)); + return end < 0 ? lines : lines.slice(0, end); +} + +const VB_FILE_IMPORTS = new WeakMap>(); + +/** The namespaces a file imports — its own `Imports` and its project's — lowercased. */ +function importedNamespaces(file: string, context: ResolutionContext): string[] { + let memo = VB_FILE_IMPORTS.get(context); + if (!memo) VB_FILE_IMPORTS.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const imports = [...projectInfo(file, context).imports]; + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([\w.]+)\s*$/i.exec(line); + if (m) imports.push(m[1]!.toLowerCase().replace(/^global\./, '')); + } + memo.set(file, imports); + return imports; +} + +const VB_ALIASES = new WeakMap>>(); + +/** A file's import aliases: `Imports TDJob = SCrawler.DownloadObjects.TDownloader.Job`, by lowercased alias. */ +function importAliases(file: string, context: ResolutionContext): Map { + let memo = VB_ALIASES.get(context); + if (!memo) VB_ALIASES.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const aliases = new Map(); + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([A-Za-z_]\w*)\s*=\s*(.+)$/i.exec(line); + const target = m ? readType(m[2]!, 0) : null; + if (target) aliases.set(m![1]!.toLowerCase(), target); + } + memo.set(file, aliases); + return aliases; +} + +/** `t` with an import alias it is written with — as itself or as its first qualifier — spelled out. */ +function unalias(t: VbType, file: string, context: ResolutionContext): VbType { + const aliases = importAliases(file, context); + if (aliases.size === 0) return t; + if (!t.qualifier) { + const target = aliases.get(t.name.toLowerCase()); + return target ? { ...t, name: target.name, ...(target.qualifier ? { qualifier: target.qualifier } : {}) } : t; + } + const target = aliases.get(t.qualifier[0]!); + return target ? { ...t, qualifier: [...(target.qualifier ?? []), target.name.toLowerCase(), ...t.qualifier.slice(1)] } : t; +} + +/** Whether `tail` is the end of `full`. */ +function endsWith(full: string[], tail: string[]): boolean { + return tail.length <= full.length && tail.every((s, i) => full[full.length - tail.length + i] === s); +} + +const VB_TYPE_LIKE_KINDS: ReadonlySet = new Set(['class', 'struct', 'interface', 'enum', 'type_alias']); + +/** The class, structure or interface (not a `Module`) a VB.NET type is nested in, or null. */ +function enclosingType(n: Node, context: ResolutionContext): Node | null { + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return null; + const parentQn = n.qualifiedName.slice(0, cut); + const parent = context.getNodesByQualifiedName(parentQn).find((p) => p.language === 'vbnet' && VB_TYPE_KINDS.has(p.kind)); + return parent && !isModule(parentQn, context) ? parent : null; +} + +function isNestedInType(n: Node, context: ResolutionContext): boolean { + return enclosingType(n, context) !== null; +} + +const VB_BASE_NAMES = new WeakMap>>(); + +/** + * The lowercased names of the types around a line and of the classes they + * inherit, walked by name (a few levels): the types whose nested types code + * there names bare — `Dim ret As New MenuList` in a `VideoEncoder` subclass. + */ +function baseTypeNamesAround(file: string, line: number, context: ResolutionContext): Set { + const around = typesAround(file, line, context); + let memo = VB_BASE_NAMES.get(context); + if (!memo) VB_BASE_NAMES.set(context, (memo = new Map())); + const key = around.map((t) => t.id).join('|'); + const hit = memo.get(key); + if (hit) return hit; + const names = new Set(); + let frontier = around; + for (let depth = 0; depth < 5 && frontier.length > 0; depth++) { + const next: Node[] = []; + for (const t of frontier) { + if (names.has(t.name.toLowerCase())) continue; + names.add(t.name.toLowerCase()); + for (const sup of supertypesOf(t, context)) if (!sup.implemented) next.push(...projectTypesNamed(sup.name, context)); + } + frontier = next; + } + memo.set(key, names); + return names; +} + +/** + * Whether a type is one a name written with this qualifier can mean: its + * namespaces and outer types end with what is written before the name. + * Designer code's `New System.Drawing.Point(3, 3)` is not staxrip's nested + * `ButtonEx.SymbolDrawer.Point`; `New API.Base.UserDataBase(…)` is that one. + */ +export function isVbTypeQualifiedBy(n: Node, qualifier: string, file: string, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_TYPE_LIKE_KINDS.has(n.kind)) return true; + const written = readType(`${qualifier}.${n.name}`, 0); + if (!written?.qualifier) return true; + const t = unalias(written, file, context); + return !t.qualifier || endsWith(fullSegments(n, context).slice(0, -1), t.qualifier); +} + +/** + * The project types a type name written at a site means, as VB.NET looks it + * up: one its written qualifier names, nested in or declared in the namespace + * of a type around the site — the nearest first — else nested in a class + * those inherit, else in a namespace the file or project imports, else in + * the site's own project (a class's nested type only where it is in scope). + * SCrawler declares + * a `SiteSettings` in each site's namespace (`API.Pinterest`, `API.Bluesky`, + * …); a member typed `SiteSettings` in `API.Pinterest.UserData` is + * Pinterest's. `ambiguous` when what is left are different types. + */ +function typesNamedAt(written: VbType, context: ResolutionContext): { owners: Node[]; ambiguous: boolean } { + const t = written.file ? unalias(written, written.file, context) : written; + let candidates = projectTypesNamed(t.name, context); + // `System.Drawing.Color` is not the project's `Color`; `API.Base.UserDataBase` is that one. + if (t.qualifier) candidates = candidates.filter((c) => endsWith(fullSegments(c, context).slice(0, -1), t.qualifier!)); + if (candidates.length === 0 || !t.file) return { owners: candidates, ambiguous: false }; + const file = t.file; + const site = typesAround(file, t.line ?? 0, context)[0]; + const sitePath = site ? fullSegments(site, context) : projectInfo(file, context).root; + let tier: Node[] = []; + let best = -1; + for (const c of candidates) { + const container = fullSegments(c, context).slice(0, -1); + if (container.length > sitePath.length || container.some((s, i) => s !== sitePath[i])) continue; + if (container.length > best) { + best = container.length; + tier = [c]; + } else if (container.length === best) tier.push(c); + } + // A type nested in a class the site's types inherit. + if (tier.length === 0 && !t.qualifier) { + const bases = baseTypeNamesAround(file, t.line ?? 0, context); + tier = candidates.filter((c) => bases.has(enclosingType(c, context)?.name.toLowerCase() ?? '')); + } + if (tier.length === 0) { + const imports = importedNamespaces(file, context); + tier = candidates.filter((c) => imports.includes(fullSegments(c, context).slice(0, -1).join('.'))); + } + // Else any the project declares in a namespace (the root namespaces and + // project imports this cannot see) — but not a class's nested type, which + // is named bare only inside it or a class deriving from it. + if (tier.length === 0) tier = t.qualifier ? candidates : candidates.filter((c) => !isNestedInType(c, context)); + // A project's own declaration over another project's of the same name + // (staxrip's main app and its AutoCrop tool each declare a `ColorHSL`). + const own = tier.filter((c) => sameVbProject(c.filePath, file, context)); + if (own.length > 0) tier = own; + // One type's partial parts are one type. + const distinct = new Set(tier.map((c) => `${projectOf(c.filePath, context)}|${c.qualifiedName.toLowerCase()}`)); + return { owners: [...tier.filter((c) => c.filePath === file), ...tier.filter((c) => c.filePath !== file)], ambiguous: distinct.size > 1 }; +} + +/** The call site's own file first, then its project's, the rest after, each in its given order. */ +export function preferVbProject(nodes: Node[], ref: UnresolvedRef, context: ResolutionContext): Node[] { + if (nodes.length < 2) return nodes; + const file: Node[] = []; + const project: Node[] = []; + const rest: Node[] = []; + for (const n of nodes) { + if (n.filePath === ref.filePath) file.push(n); + else if (sameVbProject(n.filePath, ref.filePath, context)) project.push(n); + else rest.push(n); + } + return [...file, ...project, ...rest]; +} + +/** How many leading directories two files share. */ +function sharedDirs(a: string, b: string): number { + const da = a.split('/').slice(0, -1); + const db = b.split('/').slice(0, -1); + let i = 0; + while (i < da.length && i < db.length && da[i] === db[i]) i++; + return i; +} + +/** + * The one of several equally good guesses a call means: the one in its own + * file, else in its own project, else in the nearest directory — or null + * when none of these tells them apart. + */ +export function breakVbTie(tied: Node[], ref: UnresolvedRef, context: ResolutionContext): Node | null { + // One type's overloads (and partial parts) are one guess, not a tie: the first stands for them. + const seen = new Set(); + const distinct = tied.filter((n) => { + const key = `${projectOf(n.filePath, context)}|${n.qualifiedName.toLowerCase()}`; + if (seen.has(key)) return false; + seen.add(key); + return true; + }); + if (distinct.length === 1) return distinct[0]!; + let pool = distinct.filter((n) => n.filePath === ref.filePath); + if (pool.length === 0) { + const own = distinct.filter((n) => sameVbProject(n.filePath, ref.filePath, context)); + pool = own.length > 0 ? own : distinct; + } + if (pool.length === 1) return pool[0]!; + let best = -1; + let winners: Node[] = []; + for (const n of pool) { + const shared = sharedDirs(ref.filePath, n.filePath); + if (shared > best) { + best = shared; + winners = [n]; + } else if (shared === best) winners.push(n); + } + return winners.length === 1 ? winners[0]! : null; +} + +/** The type a field or property declares itself with, read from its declaration. */ +function memberDeclaredType(member: Node, context: ResolutionContext): VbType | null { + const code = codeLines(member.filePath, context).slice(member.startLine - 1, member.startLine + 2).join(' '); + const r = escapeRegex(member.name); + const property = new RegExp(`\\bProperty\\s+${r}\\b\\s*`, 'i').exec(code); + if (property) { + let at = property.index + property[0].length; + if (code[at] === '(') { + const close = closeParen(code, at); + if (close < 0) return null; + at = close + 1; + } + const as = /^\s*As\s+(New\s+)?/i.exec(code.slice(at)); + const type = as ? readType(code, at + as[0].length) : null; + return type && as?.[1] ? { ...type, array: false } : type; + } + const binding = bindingOn(code, member.name); + if (binding?.kind === 'type') return binding.type; + // `Private x = New Foo()`: a field's initializer is what it holds. + const init = new RegExp(`(? = new Map()): VbType | null { + const type = sited(member.kind === 'method' ? declaredReturnType(member, context) : memberDeclaredType(member, context), + member.filePath, member.startLine); + const given = type ? substitute(type, args) : null; + return given !== type ? given : resolveTypeParameter(type, member, context); +} + +/** + * The type of what a receiver-less `name` reads where a call is: a member of + * a class around the call or of one it inherits, else of a `Module`. Null when + * it declares no type the source shows, undefined when nothing has the name. + */ +function memberType(name: string, ref: UnresolvedRef, context: ResolutionContext): VbType | null | undefined { + const fits = (n: Node) => VB_VALUE_KINDS.has(n.kind) || n.kind === 'method'; + for (const around of typesAround(ref.filePath, ref.line, context)) { + for (const { node, args } of ancestry(around, context)) { + const member = membersNamed(node, name, context).find(fits); + if (member) return valueMemberType(member, context, args); + } + } + const global = moduleMembersNamed(name, ref, context).find(fits); + return global ? valueMemberType(global, context) : undefined; +} + +/** The type a method, function or property returns, read from its declaration; null for a `Sub` or none written. */ +function declaredReturnType(n: Node, context: ResolutionContext): VbType | null { + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 6).join(' '); + const head = new RegExp(`\\b(Function|Property|Sub)\\s+${escapeRegex(n.name)}\\b`, 'i').exec(code); + if (!head || /^Sub$/i.test(head[1]!)) return null; + let at = head.index + head[0].length; + // `(Of T)`, then the parameter list. + for (let group = 0; group < 2; group++) { + const open = /^\s*\(/.exec(code.slice(at)); + if (!open) break; + const close = closeParen(code, at + open[0].length - 1); + if (close < 0) return null; + at = close + 1; + } + const as = /^\s*As\s+/i.exec(code.slice(at)); + return as ? readType(code, at + as[0].length) : null; +} + +const VB_TYPE_PARAMS = new WeakMap>>(); + +/** The type parameters a declaration's head names (`Class Repo(Of T As Entity)`), each with its constraint or null. */ +function typeParametersOf(n: Node, context: ResolutionContext): Map { + let memo = VB_TYPE_PARAMS.get(context); + if (!memo) VB_TYPE_PARAMS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit) return hit; + const params = readTypeParameters(n, context); + memo.set(n.id, params); + return params; +} + +function readTypeParameters(n: Node, context: ResolutionContext): Map { + const params = new Map(); + const code = codeLines(n.filePath, context).slice(n.startLine - 1, n.startLine + 2).join(' '); + const head = new RegExp(`\\b(?:Class|Structure|Interface|Module|Function|Sub)\\s+${escapeRegex(n.name)}\\s*(?=\\(\\s*Of\\b)`, 'i').exec(code); + if (!head) return params; + for (const p of splitArgs(code, head.index + head[0].length) ?? []) { + const m = /^(?:Of\s+)?(?:In\s+|Out\s+)?([A-Za-z_]\w*)(?:\s+As\s+(.+))?$/i.exec(p); + if (!m) continue; + const constraints = (m[2] ?? '').replace(/^\{|\}$/g, '').split(',').map((c) => c.trim()) + .filter((c) => c !== '' && !/^(?:New|Class|Structure)$/i.test(c)); + params.set(m[1]!.toLowerCase(), constraints.length > 0 ? sited(readType(constraints[0]!, 0), n.filePath, n.startLine) : null); + } + return params; +} + +/** + * What a type means where it is written: a type parameter of the method + * (`owner`) or of a type around it stands for its constraint — null when it + * has none, as its value's type is then unknown — and any other name for itself. + */ +function resolveTypeParameter(t: VbType | null, owner: Node | null, context: ResolutionContext): VbType | null { + if (!t || t.array || isBuiltin(t) || t.qualifier || !t.file) return t; + const key = t.name.toLowerCase(); + for (const decl of [...(owner && owner.kind === 'method' ? [owner] : []), ...typesAround(t.file, t.line ?? 0, context)]) { + const params = typeParametersOf(decl, context); + if (params.has(key)) return params.get(key) ?? null; + } + return t; +} + +/** + * The project types `t` names where it is written, null when the name leaves + * different types (a guess between them would be no better than none), or + * none for an array or a built-in type. + */ +function ownersOf(t: VbType, context: ResolutionContext): Node[] | null { + if (t.array || isBuiltin(t)) return []; + const found = typesNamedAt(t, context); + return found.ambiguous ? null : found.owners; +} + +/** + * A method — or, `withValues`, a field or property — named `name` on one of + * `owners` (the type `typed` names) or a project type they inherit, with what + * the type parameters of the type declaring it stand for. + */ +function memberOn( + owners: Node[], + name: string, + ref: UnresolvedRef, + context: ResolutionContext, + withValues: boolean, + typed?: VbType, +): { node: Node; args: Map } | null { + const fits = (n: Node) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)); + for (const level of [0, 1]) { + for (const owner of preferVbProject(owners, ref, context)) { + // What the owner's own parameters are, as the receiver's type writes them (`Repo(Of Foo)`). + const given = new Map([...typeParametersOf(owner, context).keys()].flatMap((p, i): Array<[string, VbType]> => + (typed?.args?.[i] ? [[p, typed.args[i]!]] : []))); + const ancestors = ancestry(owner, context); + for (const { node, args } of level === 0 ? ancestors.slice(0, 1) : ancestors.slice(1)) { + const found = preferVbProject(membersNamed(node, name, context).filter(fits), ref, context)[0]; + if (found) return { node: found, args: level === 0 ? given : new Map([...args].map(([p, t]) => [p, substitute(t, given)])) }; + } + } + } + return null; +} + +/** The type of the value `Member(…)` / `obj.Member(…)` gives, from the member's declaration. */ +function callResultType( + call: { receiver: string | null; member: string }, + ref: UnresolvedRef, + context: ResolutionContext, + depth: number, +): VbType | null { + let member: { node: Node; args: Map } | null = null; + if (call.receiver === null || /^(?:Me|MyClass|MyBase)$/i.test(call.receiver)) { + const owners = typesAround(ref.filePath, ref.line, context).slice(0, 1); + member = owners.length > 0 ? memberOn(owners, call.member, ref, context, true) : null; + if (!member && call.receiver === null) { + const global = moduleMembersNamed(call.member, ref, context).find((n) => n.kind === 'method' || VB_VALUE_KINDS.has(n.kind)); + member = global ? { node: global, args: new Map() } : null; + } + } else { + const type = receiverType(call.receiver, ref, context, depth + 1); + // Not a variable: a type or module named for a shared call (`ObjectStorage.Load()`). + const owners = type === undefined ? ownersOf({ name: call.receiver, array: false, file: ref.filePath, line: ref.line }, context) + : type ? ownersOf(type, context) : null; + member = owners && owners.length > 0 ? memberOn(owners, call.member, ref, context, true, type ?? undefined) : null; + } + return member ? valueMemberType(member.node, context, member.args) : null; +} + +/** + * What a receiver named `name` is declared as where the call is: a local or + * parameter of its member, else a field or property of its class, of one the + * class inherits, or of a `Module`. Null when the receiver is bound but its + * type isn't known; undefined when nothing binds the name (a type or module). + */ +function receiverType(name: string, ref: UnresolvedRef, context: ResolutionContext, depth: number): VbType | null | undefined { + if (depth > 2) return null; + const local = localBinding(name, ref, context); + if (local) { + const { binding, line } = local; + const owner = context.getNodeById?.(ref.fromNodeId) ?? null; + if (binding.kind === 'type') return resolveTypeParameter(sited(binding.type, ref.filePath, line), owner, context); + if (binding.kind === 'call') return callResultType(binding, { ...ref, line }, context, depth); + if (binding.kind === 'each') { + const collection = receiverType(binding.collection, { ...ref, line }, context, depth + 1); + return collection ? elementType(collection) : null; + } + return null; + } + return memberType(name, ref, context); +} + +interface VbExtension { + node: Node; + /** The type it extends; null when that is one of its own type parameters (it extends anything). */ + param: VbType | null; +} + +const VB_EXTENSIONS = new WeakMap>(); + +/** A method's extension-method declaration (` Function F(s As String, …)`), or null when it isn't one. */ +function extensionOf(n: Node, context: ResolutionContext): VbExtension | null { + if (n.kind !== 'method' || n.language !== 'vbnet') return null; + let memo = VB_EXTENSIONS.get(context); + if (!memo) VB_EXTENSIONS.set(context, (memo = new Map())); + const hit = memo.get(n.id); + if (hit !== undefined) return hit; + const extension = readExtension(n, context); + memo.set(n.id, extension); + return extension; +} + +function readExtension(n: Node, context: ResolutionContext): VbExtension | null { + const code = codeLines(n.filePath, context).slice(Math.max(0, n.startLine - 2), n.startLine + 6).join(' '); + const decl = new RegExp(`\\b(?:Function|Sub)\\s+${escapeRegex(n.name)}\\b`, 'i').exec(code); + if (!decl) return null; + const attributes = /((?:<[^<>]*>\s*)+)(?:(?:Public|Friend|Private|Protected|Shared|Overloads|Async|Iterator)\s+)*$/i + .exec(code.slice(0, decl.index))?.[1]; + if (!attributes || !/\bExtension(?:Attribute)?\b/i.test(attributes)) return null; + let at = decl.index + decl[0].length; + const typeParams = new Set(); + if (/^\s*\(\s*Of\b/i.test(code.slice(at))) { + const open = code.indexOf('(', at); + for (const p of splitArgs(code, open) ?? []) { + const tp = /^(?:Of\s+)?(?:In\s+|Out\s+)?([A-Za-z_]\w*)/i.exec(p); + if (tp) typeParams.add(tp[1]!.toLowerCase()); + } + const close = closeParen(code, open); + if (close < 0) return null; + at = close + 1; + } + const open = /^\s*\(/.exec(code.slice(at)); + const first = open ? splitArgs(code, at + open[0].length - 1)?.[0]?.replace(/<[^<>]*>/g, '') : undefined; + const as = first ? /\bAs\s+/i.exec(first) : null; + const type = as ? readType(first!, as.index + as[0].length) : null; + if (!type) return null; + return { node: n, param: typeParams.has(type.name.toLowerCase()) && !type.array ? null : type }; +} + +/** `Object`'s instance methods, which every type has. */ +const OBJECT_METHODS = ['tostring', 'equals', 'gethashcode', 'gettype']; + +/** + * The instance methods of .NET's everyday types: a call one of them answers + * is the type's own, whatever extension of that name the project declares + * (`list.Sort()` on a `List(Of T)`), and a name one of them lacks is no + * instance method there (`list.Join(", ")` is an extension's). + */ +const BCL_INSTANCE_METHODS: ReadonlyMap> = new Map(Object.entries({ + string: ['clone', 'compareto', 'contains', 'copyto', 'endswith', 'getenumerator', 'gettypecode', 'indexof', 'indexofany', + 'insert', 'isnormalized', 'lastindexof', 'lastindexofany', 'normalize', 'padleft', 'padright', 'remove', 'replace', 'split', + 'startswith', 'substring', 'tochararray', 'tolower', 'tolowerinvariant', 'toupper', 'toupperinvariant', 'trim', 'trimend', + 'trimstart'], + stringbuilder: ['append', 'appendformat', 'appendjoin', 'appendline', 'clear', 'copyto', 'ensurecapacity', 'getchunks', 'insert', + 'remove', 'replace'], + list: ['add', 'addrange', 'asreadonly', 'binarysearch', 'clear', 'contains', 'convertall', 'copyto', 'exists', 'find', 'findall', + 'findindex', 'findlast', 'findlastindex', 'foreach', 'getenumerator', 'getrange', 'indexof', 'insert', 'insertrange', + 'lastindexof', 'remove', 'removeall', 'removeat', 'removerange', 'reverse', 'sort', 'toarray', 'trimexcess', 'trueforall'], + dictionary: ['add', 'clear', 'containskey', 'containsvalue', 'ensurecapacity', 'getenumerator', 'remove', 'trimexcess', 'tryadd', + 'trygetvalue'], + hashset: ['add', 'clear', 'contains', 'copyto', 'exceptwith', 'getenumerator', 'intersectwith', 'ispropersubsetof', + 'ispropersupersetof', 'issubsetof', 'issupersetof', 'overlaps', 'remove', 'removewhere', 'setequals', 'symmetricexceptwith', + 'trimexcess', 'trygetvalue', 'unionwith'], + '[]': ['clone', 'copyto', 'getenumerator', 'getlength', 'getlonglength', 'getlowerbound', 'getupperbound', 'getvalue', + 'initialize', 'setvalue'], +}).map(([type, methods]) => [type, new Set([...methods, ...OBJECT_METHODS])])); + +/** The instance methods of a .NET type the table above knows; undefined for any other. */ +function bclInstanceMethods(t: VbType): ReadonlySet | undefined { + return BCL_INSTANCE_METHODS.get(t.array ? '[]' : typeKey(t)); +} + +/** + * Whether a value of type `t` might reach an extension declared for `param` + * through a conversion nothing here can check: `Object`, an array's + * collection interfaces, an interface a built-in type implements, or any + * type an outside base type may lead to. + */ +function mayExtend(t: VbType, open: boolean, param: VbType): boolean { + if (typeKey(param) === 'object') return true; + if (param.array !== t.array) { + return t.array && /^(?:IEnumerable|IList|ICollection|IReadOnlyList|IReadOnlyCollection|Array)$/i.test(param.name); + } + if (isBuiltin(param)) return false; + if (isBuiltin(t)) return /^I[A-Z]/.test(param.name); + return open; +} + +/** + * The extension method a call on a value of type `t` reaches: one declared + * for `t` or for a type it inherits — else, when exactly one other extension + * of that name could apply and the name is no instance method of `t`'s + * (one of .NET's own names, for a type nothing here describes), that one. + */ +function extensionFor( + t: VbType, + owners: Node[], + method: string, + ref: UnresolvedRef, + context: ResolutionContext, + isStdMethod: (name: string) => boolean, +): ResolvedRef | null { + const instanceMethods = owners.length === 0 ? bclInstanceMethods(t) : undefined; + if (instanceMethods?.has(method.toLowerCase())) return null; + const extensions = context.getNodesByLowerName(method.toLowerCase()) + .map((n) => extensionOf(n, context)) + .filter((e): e is VbExtension => e !== null); + if (extensions.length === 0) return null; + // Every name the type goes by — its own, its base types', the interfaces + // they implement — and whether one of them is an outside type, whose own + // ancestry nothing here shows. + const names = new Set([typeKey({ name: t.name, array: false })]); + let open = owners.length === 0 && !isBuiltin(t) && !t.array; + for (const owner of owners) { + for (const type of hierarchy(owner, context)) { + names.add(type.name.toLowerCase()); + for (const sup of supertypesOf(type, context)) { + names.add(typeKey(sup)); + const found = typesNamedAt(sup, context).owners; + if (found.length === 0) open = true; + else if (sup.implemented) for (const i of found) for (const h of hierarchy(i, context)) names.add(h.name.toLowerCase()); + } + } + } + const exact = extensions.filter((e) => e.param !== null && + (typeKey(e.param) === typeKey(t) || (!t.array && !e.param.array && names.has(typeKey(e.param))))); + if (exact.length > 0) { + const target = preferVbProject(exact.map((e) => e.node), ref, context)[0]!; + return { original: ref, targetNodeId: target.id, confidence: 0.85, resolvedBy: 'instance-method' }; + } + const loose = extensions.filter((e) => e.param === null || mayExtend(t, open, e.param)); + if (loose.length === 1 && (instanceMethods !== undefined || !isStdMethod(method))) { + return { original: ref, targetNodeId: loose[0]!.node.id, confidence: 0.7, resolvedBy: 'instance-method' }; + } + return null; +} + +/** + * Resolve `receiver.method()` through the receiver's declared type: the + * type's own method or one it inherits, else an extension method for it, + * else null — a typed receiver is never a guess. A receiver naming one of the + * project's types or modules is a shared call on it. Undefined when the type + * isn't known, or is `Object` (a late-bound call), for the name strategies. + */ +export function matchVbTypedCall( + receiver: string, + method: string, + ref: UnresolvedRef, + context: ResolutionContext, + isStdMethod: (name: string) => boolean, +): ResolvedRef | null | undefined { + if (!/^[A-Za-z_]\w*$/.test(receiver)) return undefined; + const type = receiverType(receiver, ref, context, 0); + if (type === undefined) { + // A type or module named for a shared call — `M3U8.Download(…)` in SCrawler's + // `API.Reddit.UserData` is `API.Reddit.M3U8`'s, not another site's: its own + // member or one it inherits. + const named = typesNamedAt({ name: receiver, array: false, file: ref.filePath, line: ref.line }, context); + if (named.owners.length === 0) return undefined; + if (named.ambiguous) return null; + const shared = memberOn(named.owners, method, ref, context, false); + return shared ? { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' } : null; + } + if (!type || typeKey(type) === 'object') return undefined; + const owners = ownersOf(type, context); + // A type name that leaves two of the project's types: no guess between them. + if (owners === null) return null; + if (owners.length > 0) { + const own = memberOn(owners, method, ref, context, false, type); + if (own) return { original: ref, targetNodeId: own.node.id, confidence: 0.9, resolvedBy: 'instance-method' }; + } + return extensionFor(type, owners, method, ref, context, isStdMethod); +} + +const VB_TYPE_IMPORTS = new WeakMap>>(); + +/** The last names of a file's `Imports` (a type imported this way lends its shared members; an alias lends none). */ +function importedNames(file: string, context: ResolutionContext): Set { + let memo = VB_TYPE_IMPORTS.get(context); + if (!memo) VB_TYPE_IMPORTS.set(context, (memo = new Map())); + const hit = memo.get(file); + if (hit) return hit; + const names = new Set(); + for (const line of headerLines(file, context)) { + const m = /^\s*Imports\s+([\w.]+)\s*$/i.exec(line); + if (m) names.add(m[1]!.split('.').pop()!.toLowerCase()); + } + memo.set(file, names); + return names; +} + +const VB_SCOPES = new WeakMap | null>>(); + +/** The lowercased names of the types around a call and of the project types they inherit; null outside any type. */ +function scopeOwners(ref: UnresolvedRef, context: ResolutionContext): Set | null { + let memo = VB_SCOPES.get(context); + if (!memo) VB_SCOPES.set(context, (memo = new WeakMap())); + const hit = memo.get(ref); + if (hit !== undefined) return hit; + const around = typesAround(ref.filePath, ref.line, context); + const owners = around.length === 0 ? null + : new Set(around.flatMap((t) => hierarchy(t, context)).map((t) => t.name.toLowerCase())); + memo.set(ref, owners); + return owners; +} + +/** + * Whether an unqualified VB.NET type name can mean `n`: a type nested in a + * class, structure or interface is named bare only inside it or a type + * deriving from it — designer code's `New Point(4, 285)` is not staxrip's + * nested `ButtonEx.SymbolDrawer.Point`. A `Module`'s types belong to its + * namespace; a file can import the outer type, or alias the nested one + * (`Imports UserMediaD = SCrawler.DownloadObjects.TDownloader.UserMediaD`). + * Not judged when no type is around the name. + */ +export function isVbNestedTypeInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_TYPE_LIKE_KINDS.has(n.kind)) return true; + const parent = enclosingType(n, context); + if (!parent) return true; + const owners = scopeOwners(ref, context); + if (owners === null || owners.has(parent.name.toLowerCase())) return true; + const alias = importAliases(ref.filePath, context).get(ref.referenceName.toLowerCase()); + if (alias && alias.name.toLowerCase() === n.name.toLowerCase() && + (!alias.qualifier || endsWith(fullSegments(n, context).slice(0, -1), alias.qualifier))) return true; + return importedNames(ref.filePath, context).has(parent.name.toLowerCase()); +} + +/** + * Whether a receiver-less VB.NET call (or one through `Me` / `MyClass` / + * `MyBase`) can mean member `n`: one of a type around the call or of a type + * those inherit, of a `Module`, or of a type the file imports. Not judged + * when no type is around the call, or `n` is not a member of a type. + */ +export function isVbMemberInScope(n: Node, ref: UnresolvedRef, context: ResolutionContext): boolean { + if (n.language !== 'vbnet' || !VB_MEMBER_KINDS.has(n.kind)) return true; + const cut = n.qualifiedName.lastIndexOf('::'); + if (cut < 0) return true; + const owners = scopeOwners(ref, context); + if (owners === null) return true; + const ownerQn = n.qualifiedName.slice(0, cut); + const owner = ownerQn.split(/::|\./).pop()!.toLowerCase(); + return owners.has(owner) || isModule(ownerQn, context) || importedNames(ref.filePath, context).has(owner); +} + +/** + * Drop this module's per-context memos — file lines and what was read from + * them — with the resolver's own caches (ReferenceResolver.clearCaches), so a + * sync never reads a changed file's old declarations. + */ +export function clearVbnetReceiverMemos(context: ResolutionContext): void { + for (const memo of [VB_CODE_LINES, VB_WORD_LINES, VB_BINDINGS, VB_FILE_TYPES, VB_MEMBERS, VB_MODULES, VB_SUPERS, VB_ANCESTRIES, + VB_PROJECTS, VB_PROJECT_INFO, VB_FILE_IMPORTS, VB_ALIASES, VB_BASE_NAMES, VB_TYPE_PARAMS, VB_EXTENSIONS, VB_TYPE_IMPORTS, + VB_SCOPES] as Array>) { + memo.delete(context); + } +} From 26e8488aedd632ff8102e2626808d71435a602a2 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 17:37:39 +0000 Subject: [PATCH 04/23] fix(windows): the kernel-parity sweep loads the engine as file:// URLs (#2352) scripts/kernel-parity.mjs handed import() a path from path.join(). On Windows that is `C:\...`, which Node's ESM loader rejects with ERR_UNSUPPORTED_ESM_URL_SCHEME, so the sweep could not run there at all and contributors kept patched scratch copies. The three engine imports now go through pathToFileURL(...).href, as the agent-eval probe scripts already do. On macOS and Linux an ordinary path resolves to the same file:// URL as before. Checked on Windows 11 after npm ci, npm run build and scripts/build-kernel.sh: the unmodified script threw at its first import; with the fix, the C# fixtures (3/3), src/search (4/4), all of src/ as TypeScript (261/263, 2 deferred to wasm) and every fixture language (42/42) run to the summary line and exit 0. Co-authored-by: Claude Opus 5.5 --- scripts/kernel-parity.mjs | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/scripts/kernel-parity.mjs b/scripts/kernel-parity.mjs index 5209369d8c..98c347eefd 100644 --- a/scripts/kernel-parity.mjs +++ b/scripts/kernel-parity.mjs @@ -25,7 +25,7 @@ import * as fs from 'node:fs'; import * as path from 'node:path'; -import { fileURLToPath } from 'node:url'; +import { fileURLToPath, pathToFileURL } from 'node:url'; const ROOT = path.resolve(path.dirname(fileURLToPath(import.meta.url)), '..'); const dist = (p) => path.join(ROOT, 'dist', p); @@ -106,9 +106,11 @@ if (files.length === 0) { } // --- load the built engine --------------------------------------------------- -const { extractFromSource } = await import(dist('extraction/tree-sitter.js')); -const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(dist('extraction/grammars.js')); -const kernel = await import(dist('extraction/kernel/index.js')); +// As file:// URLs: import() rejects a bare `C:\...` path on Windows +// (ERR_UNSUPPORTED_ESM_URL_SCHEME). +const { extractFromSource } = await import(pathToFileURL(dist('extraction/tree-sitter.js')).href); +const { initGrammars, loadGrammarsForLanguages, detectLanguage } = await import(pathToFileURL(dist('extraction/grammars.js')).href); +const kernel = await import(pathToFileURL(dist('extraction/kernel/index.js')).href); await initGrammars(); await loadGrammarsForLanguages([...KERNEL_LANGS]); From 69ee3b9a8597940b2a12ea5fe26c48ab0c9c0650 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 19:03:03 +0000 Subject: [PATCH 05/23] fix(ui): fall back past a port Windows refuses with EACCES, and say why (#2299) (#2353) On Windows, a port another program holds exclusively (SO_EXCLUSIVEADDRUSE) or one inside an excluded port range fails listen() with EACCES rather than EADDRINUSE. The `codegraph ui` port fallback only moved on after EADDRINUSE, so it stopped at the first such port. The error then blamed the POSIX privileged-port rule ("Ports below 1024 usually need elevated privileges") for ports like 4747 or 49912. Confirmed on Windows 11: a .NET listener on 0.0.0.0:P with ExclusiveAddressUse makes Node's listen(P, '127.0.0.1') fail with EACCES. Fix: - shouldTryNextPort moves past EADDRINUSE on every platform. It moves past EACCES at any port on Windows, which has no privileged ports, and from 1024 up elsewhere. - Below 1024 off Windows, EACCES is the privileged-port rule, and the next port falls under it too. The walk stops there with the message it always gave, rather than trying 20 privileged ports. - describeBindFailure takes the platform: - Windows names the exclusive hold or reserved range and how to list the ranges (netsh int ipv4 show excludedportrange protocol=tcp). - POSIX keeps the privileged-port text below 1024. Above it, it says the system refused permission. - A walk that ran out says "in use or reserved" (Windows) or "in use or not allowed" when any port was refused rather than in use. - Both helpers take the platform as a parameter, like browserOpenCommand, so the tests check every platform's answer on any host. This builds on PR #2319 by @sx4im, which advanced on EACCES, used the Windows wording and added the issue's listen-spy test. That PR advanced on EACCES on every platform and printed "Windows refused port N ... netsh" on Linux and macOS too. On POSIX it would also walk 20 privileged ports and end with "Ports 80-99 are all in use or reserved". This change restricts both to where they are true. The changelog entry goes in the viewer-launch file, because the viewer is not released yet. Verification: - __tests__/ui-server.test.ts gains three kinds of test: - The issue's end-to-end test: port N is taken and N+1 is refused through a listen spy, so the server must land past N+1 and serve 200. - A pinned-port message test. - Unit tests of the decision and the messages for win32, linux and darwin. - Results on __tests__/ui-server.test.ts: - Main source: 9 failed / 40 passed. The walk fails with "Not allowed to listen on port 57423. Ports below 1024 usually need elevated privileges". - With the fix: 49 passed / 2 skipped (existing POSIX-gated tests). - Built CLI on Windows, with 0.0.0.0: held exclusively: - `codegraph ui` while 4747 is held: main exits 1 with the privileged-port message. With the fix it serves on 4748 (GET / 200). - `codegraph ui --port `: prints the Windows message. - A 3-port walk over held ports reports "in use or reserved". - Re-listening on the same server after a real EACCES works on Node 22.22 and 24.16. - tsc --noEmit is clean and npm run build succeeds. The 20 viewer test files pass (439 passed, 5 skipped). - Full suite: 5732 passed, 10 failed, 319 skipped. The 10 failures are timeouts and EBUSY teardown under load in function-ref, git-index-currency, index-daemon-rebuild and rust-self-owner. All four files pass when run alone. Issue #2299 reported by @ijbranch. Co-authored-by: Saim Shafique Co-authored-by: Claude Opus 5.5 --- __tests__/ui-server.test.ts | 160 +++++++++++++++++++++++++++++++- docs/viewer-launch-changelog.md | 2 + src/ui-server/index.ts | 92 +++++++++++++++--- 3 files changed, 238 insertions(+), 16 deletions(-) diff --git a/__tests__/ui-server.test.ts b/__tests__/ui-server.test.ts index df25858a84..4b4e63fc8c 100644 --- a/__tests__/ui-server.test.ts +++ b/__tests__/ui-server.test.ts @@ -12,7 +12,7 @@ * name in undici, and forging it is the whole point of half these cases. */ -import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; import * as http from 'http'; import * as fs from 'fs'; import * as os from 'os'; @@ -21,12 +21,14 @@ import { browserOpenCommand, cacheControlFor, contentTypeFor, + describeBindFailure, isAllowedHost, isAllowedOrigin, isSafeRequestPath, PathRefusalError, resolveProjectFile, resolveStaticAsset, + shouldTryNextPort, startUiServer, type UiServerHandle, } from '../src/ui-server'; @@ -69,6 +71,31 @@ function request( }); } +/** A failed `listen()` as node reports it. */ +function bindError(code: string, port: number): NodeJS.ErrnoException { + return Object.assign(new Error(`listen ${code}: 127.0.0.1:${port}`), { code, syscall: 'listen' }); +} + +/** + * Make `listen()` on `refused` fail with EACCES, the way Windows refuses a port + * another program holds exclusively or one inside a reserved range, without + * needing either on the machine running the test (#2299). Every other port is + * a real bind. + */ +function refuseListenOn(...refused: number[]): { mockRestore(): void } { + const realListen = http.Server.prototype.listen; + return vi + .spyOn(http.Server.prototype, 'listen') + .mockImplementation(function (this: http.Server, ...args: unknown[]) { + const port = args[0]; + if (typeof port === 'number' && refused.includes(port)) { + process.nextTick(() => this.emit('error', bindError('EACCES', port))); + return this; + } + return (realListen as (...a: unknown[]) => http.Server).apply(this, args); + }); +} + describe('codegraph ui server', () => { let tempDir: string; let viewerDir: string; @@ -198,6 +225,50 @@ describe('codegraph ui server', () => { await new Promise((resolve) => blocker.close(() => resolve())); } }); + + /** + * #2299: on Windows a port another program holds exclusively, or one in a + * reserved range, fails with EACCES rather than EADDRINUSE, and the walk + * used to stop there. Port N is really taken, N+1 is refused, so the server + * has to land beyond both. N+1 is an ephemeral port, above 1023, so this + * holds on every platform. + */ + it('falls back past a port refused with EACCES', async () => { + const blocker = http.createServer(() => {}); + await new Promise((resolve) => blocker.listen(0, '127.0.0.1', resolve)); + const taken = (blocker.address() as { port: number }).port; + const refusal = refuseListenOn(taken + 1); + + try { + const second = await startUiServer({ projectRoot, viewerDir, port: taken }); + try { + expect(second.port).toBeGreaterThan(taken + 1); + // …and it actually works on the port it landed on. + const res = await request(second.port, '/'); + expect(res.status).toBe(200); + } finally { + await second.close(); + } + } finally { + refusal.mockRestore(); + await new Promise((resolve) => blocker.close(() => resolve())); + } + }); + + it('blames a refused pinned port on what refused it, not on privileged ports', async () => { + // The port from the report: nowhere near 1024. + const refusal = refuseListenOn(49912); + try { + const failure = startUiServer({ projectRoot, viewerDir, port: 49912, portFallback: false }); + await expect(failure).rejects.toThrow(/49912/); + const message = await failure.catch((err: Error) => err.message); + expect(message).not.toMatch(/below 1024|elevated privileges/); + expect(message).not.toMatch(/already in use/); + expect(message).toMatch(/refused/); + } finally { + refusal.mockRestore(); + } + }); }); describe('Host allowlist (DNS rebinding)', () => { @@ -538,6 +609,93 @@ describe('security helpers', () => { }); }); +/** + * What a bind failure means depends on the platform (#2299): Windows has no + * privileged ports and refuses a taken or reserved one with EACCES, while + * elsewhere EACCES below 1024 is the privileged-port rule. The platform is a + * parameter, so every host checks every platform's answer. + */ +describe('port fallback on each platform', () => { + const LIST_RESERVED = 'netsh int ipv4 show excludedportrange protocol=tcp'; + + it('moves past a port in use everywhere, and past EACCES on Windows at any port', () => { + for (const platform of ['win32', 'linux', 'darwin'] as const) { + expect(shouldTryNextPort('EADDRINUSE', 80, platform), platform).toBe(true); + expect(shouldTryNextPort('EADDRINUSE', 4747, platform), platform).toBe(true); + } + expect(shouldTryNextPort('EACCES', 49912, 'win32')).toBe(true); + // Windows has no privileged-port rule: a low port is held, not forbidden. + expect(shouldTryNextPort('EACCES', 80, 'win32')).toBe(true); + }); + + it('stops at EACCES below 1024 elsewhere, where the next port needs the same privilege', () => { + expect(shouldTryNextPort('EACCES', 80, 'linux')).toBe(false); + expect(shouldTryNextPort('EACCES', 1023, 'darwin')).toBe(false); + // From 1024 up the refusal is about that port, and the next may be allowed. + expect(shouldTryNextPort('EACCES', 1024, 'linux')).toBe(true); + expect(shouldTryNextPort('EACCES', 4747, 'darwin')).toBe(true); + }); + + it('never moves past any other failure', () => { + for (const platform of ['win32', 'linux'] as const) { + expect(shouldTryNextPort('EADDRNOTAVAIL', 4747, platform), platform).toBe(false); + expect(shouldTryNextPort('EPERM', 4747, platform), platform).toBe(false); + expect(shouldTryNextPort(undefined, 4747, platform), platform).toBe(false); + } + }); + + it('says why Windows refused a pinned port, without the POSIX privileged-port story', () => { + for (const port of [49912, 80]) { + const message = describeBindFailure(bindError('EACCES', port), port, { port, fallback: false }, 'win32').message; + expect(message).toMatch( + new RegExp(`^Windows refused port ${port}: another program holds it, or it is in a reserved range`) + ); + expect(message).toContain(LIST_RESERVED); + expect(message).toContain('omit --port'); + expect(message).not.toMatch(/below 1024|elevated privileges/); + } + }); + + it('keeps the privileged-port explanation off Windows, and only below 1024', () => { + for (const fallback of [false, true]) { + const message = describeBindFailure(bindError('EACCES', 80), 80, { port: 80, fallback }, 'linux').message; + expect(message).toBe( + 'Not allowed to listen on port 80. Ports below 1024 usually need elevated privileges — pick a higher one with --port.' + ); + } + const high = describeBindFailure(bindError('EACCES', 4747), 4747, { port: 4747, fallback: false }, 'darwin').message; + expect(high).toMatch(/^Not allowed to listen on port 4747: the system refused permission\./); + expect(high).not.toMatch(/below 1024|elevated privileges|Windows|netsh/); + }); + + it('says a walk that ran out met refusals, not only ports in use', () => { + const walk = { port: 4747, fallback: true }; + const inUse = bindError('EADDRINUSE', 4766); + const refused = bindError('EACCES', 4766); + + // Nothing refused: the message it always gave. + expect(describeBindFailure(inUse, 4766, walk, 'win32').message).toBe( + 'Ports 4747–4766 are all in use. Free one, or pick another with --port.' + ); + const windows = describeBindFailure(refused, 4766, walk, 'win32').message; + expect(windows).toMatch(/^Ports 4747–4766 are all in use or reserved/); + expect(windows).toContain(LIST_RESERVED); + // An earlier port was refused even though the last one was only in use. + expect(describeBindFailure(inUse, 4766, walk, 'win32', true).message).toBe(windows); + const posix = describeBindFailure(refused, 4766, walk, 'linux').message; + expect(posix).toMatch(/^Ports 4747–4766 are all in use or not allowed\./); + expect(posix).not.toMatch(/reserved|netsh/); + }); + + it('leaves the pinned in-use message and other failures as they were', () => { + expect( + describeBindFailure(bindError('EADDRINUSE', 8080), 8080, { port: 8080, fallback: false }, 'win32').message + ).toBe('Port 8080 is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.'); + const other = bindError('EADDRNOTAVAIL', 4747); + expect(describeBindFailure(other, 4747, { port: 4747, fallback: true }, 'linux')).toBe(other); + }); +}); + describe('browserOpenCommand', () => { it('uses the platform opener', () => { expect(browserOpenCommand('http://x', 'darwin')).toEqual({ command: 'open', args: ['http://x'] }); diff --git a/docs/viewer-launch-changelog.md b/docs/viewer-launch-changelog.md index 593ba632c0..84ed815051 100644 --- a/docs/viewer-launch-changelog.md +++ b/docs/viewer-launch-changelog.md @@ -205,6 +205,8 @@ These describe `codegraph ui` and its screens. They were taken out of `## [Unrel - A viewer trail now holds up to 64 hops everywhere — the trail bar, saved trails and "Read as flow". Thanks @inth3shadows for the report and @danusha2345. (#1976) +- **`codegraph ui` on Windows moves on from a port another program holds.** Windows refuses a port that another program keeps for itself, or one inside a range it reserves, with a different error than a port that is simply in use, so the viewer gave up at the first such port instead of trying the next one, and blamed ports below 1024 needing elevated privileges — a rule Windows doesn't have. It now moves on to the next free port, and when none is left, or the port you pinned with `--port` is refused, it says what Windows did and how to list the reserved ranges. Thanks @ijbranch for the report and @sx4im. (#2299) + ## Entries whose graph and `codegraph_explore` half already shipped The release notes carry a reworded version of each of these. Keep only the viewer half at launch. diff --git a/src/ui-server/index.ts b/src/ui-server/index.ts index 40c170f703..e5d0cced8a 100644 --- a/src/ui-server/index.ts +++ b/src/ui-server/index.ts @@ -397,9 +397,10 @@ function safeDecode(value: string): string { /** * Bind the first free port at or after `port`, on loopback only. * - * Only `EADDRINUSE` advances to the next port — a permission failure or a bad - * address will not get better one port over, and retrying twenty times would - * only bury the real error. + * Only a port that is taken advances to the next one (see + * {@link shouldTryNextPort}) — a permission failure every port shares, or a + * bad address, will not get better one port over, and retrying twenty times + * would only bury the real error. */ async function listenWithFallback( server: http.Server, @@ -407,6 +408,9 @@ async function listenWithFallback( ): Promise { // Port 0 means "any free port", so there is nothing to fall back from. const attempts = opts.port === 0 || !opts.fallback ? 1 : Math.max(1, opts.attempts); + const platform = process.platform; + // Whether any port was refused rather than in use — the message says so. + let refused = false; for (let i = 0; i < attempts; i++) { const candidate = opts.port === 0 ? 0 : opts.port + i; @@ -419,8 +423,9 @@ async function listenWithFallback( return address.port; } catch (err) { const code = (err as NodeJS.ErrnoException).code; - if (code !== 'EADDRINUSE' || i === attempts - 1) { - throw describeBindFailure(err, candidate, opts); + if (code === 'EACCES') refused = true; + if (!shouldTryNextPort(code, candidate, platform) || i === attempts - 1) { + throw describeBindFailure(err, candidate, opts, platform, refused); } } } @@ -434,7 +439,8 @@ async function listenWithFallback( * The same `http.Server` is reused across attempts: a `listen()` that failed * with EADDRINUSE never took a handle, so it can be listened on again directly * (verified on Node 20 and 22 — `server.listening` is still `false` afterwards, - * and `close()` on a never-listening server would itself throw). + * and `close()` on a never-listening server would itself throw). The same goes + * for a port Windows refused with EACCES (checked on Node 22 and 24). */ function listenOnce(server: http.Server, port: number): Promise { return new Promise((resolve, reject) => { @@ -452,22 +458,78 @@ function listenOnce(server: http.Server, port: number): Promise { }); } -/** Turn a bind failure into something a user can act on. */ -function describeBindFailure( +/** + * Whether a `listen()` on `port` that failed with `code` leaves the next port + * worth trying. + * + * `EADDRINUSE` does, and so does `EACCES` on Windows: a port another program + * holds exclusively (`SO_EXCLUSIVEADDRUSE`), or one inside a range the system + * reserves (Hyper-V and WinNAT reserve whole blocks of them), fails there with + * EACCES rather than EADDRINUSE. Elsewhere, EACCES below 1024 is the + * privileged-port rule, which the next port is under too, so walking on would + * only bury it. From 1024 up it is something refusing that one port — a + * security policy, say — and the next one may be allowed. + */ +export function shouldTryNextPort( + code: string | undefined, + port: number, + platform: NodeJS.Platform = process.platform +): boolean { + if (code === 'EADDRINUSE') return true; + return code === 'EACCES' && !isPrivilegedPort(port, platform); +} + +/** + * Whether `port` is one only an administrator may bind: below 1024, on every + * platform but Windows, which has no such rule. + */ +function isPrivilegedPort(port: number, platform: NodeJS.Platform): boolean { + return platform !== 'win32' && port > 0 && port < 1024; +} + +/** The command that lists the port ranges Windows reserves. */ +const LIST_RESERVED_PORTS = '`netsh int ipv4 show excludedportrange protocol=tcp`'; + +/** + * Turn a bind failure into something a user can act on, and that is true where + * it is read: Windows refuses ports for reasons POSIX doesn't have, and has no + * privileged ports. + * + * `refused` says whether any port a fallback walk tried was refused (EACCES) + * rather than in use; it only changes the message for a walk that ran out. + */ +export function describeBindFailure( err: unknown, port: number, - opts: { port: number; fallback: boolean; attempts: number } + opts: { port: number; fallback: boolean }, + platform: NodeJS.Platform = process.platform, + refused = false ): Error { const code = (err as NodeJS.ErrnoException).code; + const windows = platform === 'win32'; + if (opts.fallback && shouldTryNextPort(code, port, platform)) { + // A failure that moves on only ends the walk when there is nowhere left. + const taken = + !refused && code === 'EADDRINUSE' + ? 'in use' + : windows + ? `in use or reserved (${LIST_RESERVED_PORTS} lists the reserved ranges)` + : 'in use or not allowed'; + return new Error(`Ports ${opts.port}–${port} are all ${taken}. Free one, or pick another with --port.`); + } if (code === 'EADDRINUSE') { - return opts.fallback - ? new Error( - `Ports ${opts.port}–${port} are all in use. Free one, or pick another with --port.` - ) - : new Error(`Port ${port} is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.`); + return new Error(`Port ${port} is already in use. Pick another with --port, or omit --port to let CodeGraph find a free one.`); } if (code === 'EACCES') { - return new Error(`Not allowed to listen on port ${port}. Ports below 1024 usually need elevated privileges — pick a higher one with --port.`); + if (isPrivilegedPort(port, platform)) { + return new Error(`Not allowed to listen on port ${port}. Ports below 1024 usually need elevated privileges — pick a higher one with --port.`); + } + return new Error( + (windows + ? `Windows refused port ${port}: another program holds it, or it is in a reserved range (${LIST_RESERVED_PORTS} lists them).` + : `Not allowed to listen on port ${port}: the system refused permission.`) + + ' Pick another with --port, or omit --port to let CodeGraph find a free one.' + ); } return err instanceof Error ? err : new Error(String(err)); } From 6c61dd74ca672dd8aa4adc86718406641967e109 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:05:18 +0000 Subject: [PATCH 06/23] fix(vbnet): link Shared fields and properties read through their class (#2305) (#2355) VB.NET emitted a reference for calls only, so a value read or write through a type name -- `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level` -- linked nothing, and `codegraph callers` on the field, the property and the class came back empty. The reporter's central session class showed 6 callers against ~460 referencing files. - Extraction (TS only; VB.NET is not kernel-routed): the static-member pass now covers VB.NET. A value read through a Capitalized name is sent as one `references` ref, `Name.Member`, the receiver kept as a call's is. Namespace roots (`System`, `Microsoft`, `My`, `Global`) and the built-in type keywords (`String.Empty`) are not sent. - Resolution (vbnet-receivers.ts matchVbMemberRead, ahead of the framework, import and name strategies): the receiver is looked up with #2351's VB.NET scoping, case-insensitively. A local, parameter, field, property or Module member of that name holds a value and links nothing; one typed as the type of its own name ("Color Color") reads that type. Otherwise the class, Module, Structure, Interface or Enum the namespaces, Imports and aliases around the read see is the type: the read links the member it declares or inherits (field, property, constant, event, Enum value, nested type; Shared before a same-named instance member, never the member the read is in) and, from outside the type, the type itself. A method named this way is a call made without parentheses (`calls`) unless AddressOf / NameOf only names it. - `AppSession.Items(0)`, an index into a Shared field that VB.NET writes as a call, now reads the field and the type instead of linking nothing. Contributor PR #2321 (@ChrisPrapas) had the right idea -- enable the static-member pass for VB.NET with a member-level ref -- and its test scenarios are adopted. Its resolution is replaced: it name-matched the receiver's bare name and checked a member only against the receiver's spelling. Ported onto main it passes its own tests, but binds a parameter `appSession As OtherSession` to AppSession's members, misses import aliases, and adds 8,652 / 13,554 edges on SCrawler / staxrip, about 2,000 per repo binding a receiver to another class's same-named member (a plugin form's local `Dim CONTAINER_MAIN` went to another form's control field). Verification: new vbnet-shared-member-refs.test.ts (7 tests) fails 6/7 on main and passes; the other VB.NET suites pass; tsc and build clean; the issue's repro lists Consumer.Run and Consumer2.Describe for SessionId, CurrentUser and AppSession. Validation, main vs this branch: nodes unchanged, 0 edges removed, no self-loops. SCrawler 14,893 -> 17,437 edges, staxrip 26,647 -> 32,568: Enum values 803 / 1,448, Shared fields and properties 363 / 1,459, the type read through 1,374 / 2,879, Shared functions called without parentheses 3 / 126, AddressOf / NameOf 1 / 9. Spot-checked against source: per-site SiteSettings, import aliases, escaped `[Date]` / `[New]`, nested enums, "Color Color" fields; designer `Point.Empty` and staxrip's AutoCrop duplicates stay unlinked. Index time within noise; value reads that link nothing are kept for re-resolution like failed calls (db 22.6 -> 28.9 MB, 28.3 -> 35.8 MB). Not covered: reads through Me / MyBase, unqualified reads, instance reads through a typed variable, Module variables as receivers, namespace-qualified receivers and With blocks. Co-authored-by: Claude Opus 5.5 Co-authored-by: ChrisPrapas <129202926+ChrisPrapas@users.noreply.github.com> --- CHANGELOG.md | 1 + __tests__/vbnet-shared-member-refs.test.ts | 218 +++++++++++++++++++++ src/extraction/tree-sitter.ts | 43 +++- src/resolution/index.ts | 8 +- src/resolution/vbnet-receivers.ts | 140 +++++++++++-- 5 files changed, 396 insertions(+), 14 deletions(-) create mode 100644 __tests__/vbnet-shared-member-refs.test.ts diff --git a/CHANGELOG.md b/CHANGELOG.md index b5476b128f..9c5f79e4c4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -28,6 +28,7 @@ and adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). - A file is no longer saved with no symbols when its language parser can't be loaded, which is what happened to every file a background server re-indexed after an upgrade removed its install: the file keeps what it had and is indexed again once the parser loads, and files an earlier version emptied this way are re-indexed by the next sync. A background server also exits on its own once its install is upgraded or removed, so the next session starts one from the current install. Thanks @lipchey for the report. (#2335) - `codegraph status` no longer says the index is up to date while indexed files are missing their symbols: it now names files the parser couldn't read and files stored without their symbols (which `codegraph sync` repairs), `status --json` counts both, and `codegraph files --json` lists each file's recorded errors. Thanks @lipchey for the report. (#2336) - Indexing large Python projects is much faster again and needs less memory: since 1.6.2, resolving Python references re-read source files over and over, so a project the size of CPython took several times as long to index. The graph it builds is unchanged. Thanks @bompus for the report. (#2332) +- In VB.NET, reading or setting a `Shared` field or property through its class or module name, like `AppSession.SessionId`, `AppSession.CurrentUser = "demo"` or `AppSession.Items(0)`, now counts as a use of that member and of the class, and so do reading an `Enum` value like `Mode.Fast` and calling a `Shared` function without parentheses. Before, only calls written with parentheses were linked, so `codegraph callers` on such a field, property or class came back empty and impact missed most of the code that depends on it. A local, parameter or field that only shares a class's name is not mistaken for the class. Thanks @serkanince for the report and @ChrisPrapas. (#2305) ## [1.6.2] - 2026-10-03 diff --git a/__tests__/vbnet-shared-member-refs.test.ts b/__tests__/vbnet-shared-member-refs.test.ts new file mode 100644 index 0000000000..bbcfa77ffe --- /dev/null +++ b/__tests__/vbnet-shared-member-refs.test.ts @@ -0,0 +1,218 @@ +/** + * VB.NET: a value read or written through a type or module name — + * `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level`, + * `Mode.Fast` — is a use of the member and of the type (#2305). Before, only + * calls linked, so `codegraph callers` on a Shared field or property, and on + * the class itself, came back empty. + * + * The name is looked up as VB.NET does, without regard to case: a local, a + * parameter or a field of that name holds a value, and links nothing here (a + * member typed as its own name's type, `Property Settings As Settings`, reads + * that type's members either way); a type is the one the namespaces around + * the read, its `Imports` and aliases see — SCrawler declares a + * `SiteSettings` in every site's namespace. A method named without + * parentheses is called, unless `AddressOf` only names it, and an index into + * a Shared field (`AppSession.Items(0)`), which VB.NET writes as a call, reads + * the field. + */ +import { describe, it, expect, afterAll, beforeAll } from 'vitest'; +import * as fs from 'fs'; +import * as os from 'os'; +import * as path from 'path'; +import { CodeGraph } from '../src'; + +let root = ''; +let cg: CodeGraph; + +const files: Record = { + // The issue's three files. + 'AppSession.vb': `Public Class AppSession + Public Shared SessionId As Guid = Guid.NewGuid() + Public Shared Property CurrentUser As String + Public Shared Items As New List(Of String) + Public Shared Function GetGreeting() As String + Return "Hello " & CurrentUser + End Function + Public Shared Sub OnTick(ByVal sender As Object, ByVal e As EventArgs) + End Sub + Public Shared Function Describe() As String + Return AppSession.CurrentUser + End Function +End Class +`, + 'Consumer.vb': `Public Class Consumer + Public Sub Run() + Dim id As Guid = AppSession.SessionId + AppSession.CurrentUser = "demo" + Console.WriteLine(AppSession.GetGreeting()) + End Sub +End Class +`, + 'Consumer2.vb': `Public Class Consumer2 + Public Function Describe() As String + Return AppSession.CurrentUser & AppSession.SessionId.ToString() + End Function +End Class +`, + 'OtherSession.vb': `Public Class OtherSession + Public Shared SessionId As Guid = Guid.NewGuid() + Public Property Size As Integer +End Class +`, + 'Size.vb': `Public Class Size +End Class +`, + 'Logger.vb': `Public Module Logger + Public Level As Integer +End Module +`, + 'Mode.vb': `Public Enum Mode + Fast + Slow +End Enum +`, + 'Form1.vb': `Public Class Form1 + Private Panel1 As Panel + Public Sub Setup() + Logger.Level = 3 + Dim a = Panel1.Size + Me.Panel1.Size = New System.Drawing.Size(1, 2) + Dim speed = Mode.Fast + Dim greeting = AppSession.GetGreeting + AddHandler Timer1.Tick, AddressOf AppSession.OnTick + Dim first = AppSession.Items(0) + Dim empty = String.Empty + End Sub + Public Sub Shadowed(ByVal appSession As OtherSession) + Dim id = AppSession.SessionId + End Sub +End Class +`, + // A type declared in each site's namespace, and one read through an import alias. + 'Sites/Reddit/SiteSettings.vb': `Namespace API.Reddit + Friend Class SiteSettings + Friend Const Header As String = "r" + End Class + Friend Class UserData + Friend Function Key() As String + Return SiteSettings.Header + End Function + End Class +End Namespace +`, + 'Sites/Twitter/SiteSettings.vb': `Namespace API.Twitter + Friend Class SiteSettings + Friend Const Header As String = "t" + End Class + Friend Class UserData + Friend Function Key() As String + Return SiteSettings.Header + End Function + End Class +End Namespace +`, + 'Sites/Facebook/UserData.vb': `Imports RS = API.Reddit.SiteSettings +Namespace API.Facebook + Friend Class UserData + Friend Function Key() As String + Return RS.Header + End Function + End Class +End Namespace +`, + // A property typed as the type of its own name; a property initializer. + 'Settings.vb': `Public Class Settings + Public Property Theme As String +End Class +`, + 'Window.vb': `Public Class Window + Public Property Settings As Settings + Public Property Title As String = AppSession.CurrentUser + Public Function CurrentTheme() As String + Return Settings.Theme + End Function +End Class +`, + // CRLF line endings and multibyte text before the read. + 'Weird.vb': [ + 'Public Class Weird', + ' Public Sub Go()', + ' Dim s = "SessionId éééé 😀" & AppSession.SessionId', + ' End Sub', + 'End Class', + '', + ].join('\r\n'), +}; + +beforeAll(async () => { + root = fs.mkdtempSync(path.join(os.tmpdir(), 'cg-vb-shared-')); + for (const [rel, content] of Object.entries(files)) { + fs.mkdirSync(path.dirname(path.join(root, rel)), { recursive: true }); + fs.writeFileSync(path.join(root, rel), content); + } + cg = await CodeGraph.init(root, { index: true }); +}); + +afterAll(() => { + cg?.close(); + if (root) fs.rmSync(root, { recursive: true, force: true }); +}); + +function nodeId(qualifiedName: string): string { + const found = cg.getNodesInFiles(Object.keys(files)).filter((n) => n.qualifiedName === qualifiedName); + if (found.length !== 1) throw new Error(`${found.length} nodes named ${qualifiedName}`); + return found[0]!.id; +} + +/** The qualified names of what links to `qualifiedName` with an edge of `kind`. */ +function linkedFrom(qualifiedName: string, kind: 'references' | 'calls'): string[] { + return cg + .getIncomingEdgesTo([nodeId(qualifiedName)], [kind]) + .map((e) => cg.getNode(e.source)!.qualifiedName) + .sort(); +} + +describe('VB.NET Shared members read through their type (#2305)', () => { + it('link a Shared field and property to the methods that read or write them', () => { + expect(linkedFrom('AppSession::SessionId', 'references')).toEqual(['Consumer2::Describe', 'Consumer::Run', 'Weird::Go']); + expect(linkedFrom('AppSession::CurrentUser', 'references')) + .toEqual(['AppSession::Describe', 'Consumer2::Describe', 'Consumer::Run', 'Window::Title']); + }); + + it('link the class to the code outside it that reads through it', () => { + expect(linkedFrom('AppSession', 'references')) + .toEqual(['Consumer2::Describe', 'Consumer2::Describe', 'Consumer::Run', 'Consumer::Run', 'Form1::Setup', 'Weird::Go', 'Window::Title']); + expect(cg.getCallers(nodeId('AppSession')).map((c) => c.node.qualifiedName).sort()) + .toEqual(['Consumer2::Describe', 'Consumer::Run', 'Form1::Setup', 'Weird::Go', 'Window::Title']); + }); + + it('keep the call to a Shared method, and call one named without parentheses', () => { + expect(linkedFrom('AppSession::GetGreeting', 'calls')).toEqual(['Consumer::Run', 'Form1::Setup']); + expect(linkedFrom('AppSession::OnTick', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('AppSession::OnTick', 'calls')).toEqual([]); + }); + + it('read a Shared field indexed like a call, a Module variable and an Enum value', () => { + expect(linkedFrom('AppSession::Items', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Logger::Level', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Mode::Fast', 'references')).toEqual(['Form1::Setup']); + expect(linkedFrom('Mode', 'references')).toEqual(['Form1::Setup']); + }); + + it('link nothing through a local, parameter or field, whatever its case', () => { + expect(linkedFrom('OtherSession::SessionId', 'references')).toEqual([]); + expect(linkedFrom('OtherSession::Size', 'references')).toEqual([]); + expect(linkedFrom('Size', 'references')).toEqual([]); + expect(linkedFrom('OtherSession', 'references')).toEqual([]); + }); + + it('read the type the namespaces and imports around the read see', () => { + expect(linkedFrom('API.Reddit::SiteSettings::Header', 'references')) + .toEqual(['API.Facebook::UserData::Key', 'API.Reddit::UserData::Key']); + expect(linkedFrom('API.Twitter::SiteSettings::Header', 'references')).toEqual(['API.Twitter::UserData::Key']); + }); + + it('read through a member typed as the type of its own name', () => { + expect(linkedFrom('Settings::Theme', 'references')).toEqual(['Window::CurrentTheme']); + }); +}); diff --git a/src/extraction/tree-sitter.ts b/src/extraction/tree-sitter.ts index 222291f14c..17b0b02caf 100644 --- a/src/extraction/tree-sitter.ts +++ b/src/extraction/tree-sitter.ts @@ -361,7 +361,7 @@ const PHP_TYPE_NODES: ReadonlySet = new Set([ */ const MEMBER_ACCESS_TYPES: ReadonlySet = new Set([ 'field_access', // java (`Foo.BAR`) - 'member_access_expression', // c# (`Foo.Bar`) + 'member_access_expression', // c# / vb.net (`Foo.Bar`) 'navigation_expression', // kotlin / swift (`Foo.bar`) 'field_expression', // scala (`Foo.bar`) 'class_constant_access_expression', // php (`Foo::CONST`, `Foo::class`) @@ -380,11 +380,21 @@ const MEMBER_ACCESS_TYPES: ReadonlySet = new Set([ * static read is pure duplication) — while adding real graph noise (+1813 edges / * +2448 `references` on excalidraw, the retrieval-perf benchmark, all pointing at * already-covered types). Don't re-add `member_expression`/`attribute` here. + * VB.NET (#2305) sends the member with its receiver instead, and its resolver + * decides whether the receiver is a type (see extractVbMemberRead). */ const STATIC_MEMBER_LANGS: ReadonlySet = new Set([ - 'java', 'csharp', 'kotlin', 'swift', 'scala', 'dart', 'php', 'cpp', + 'java', 'csharp', 'kotlin', 'swift', 'scala', 'dart', 'php', 'cpp', 'vbnet', ]); +/** + * VB.NET receivers no project type can be named: the namespace roots + * (`System.IO.Path`, `My.Settings`, `Global.X`) and the built-in type keywords + * (`String.Empty`, `Integer.MaxValue`). A read through one is never sent. + */ +const VB_NON_TYPE_RECEIVERS = + /^(?:Global|System|Microsoft|My|Boolean|Byte|Char|Date|Decimal|Double|Integer|Long|Object|SByte|Short|Single|String|UInteger|ULong|UShort)$/i; + /** * Tree-sitter node kinds that represent constructor invocations * (`new Foo()` and friends). Used by extractInstantiation to emit @@ -5549,6 +5559,10 @@ export class TreeSitterExtractor { getChildByField(node, 'scope') ?? node.namedChild(0); if (!recv) return; + if (this.language === 'vbnet') { + this.extractVbMemberRead(node, recv, ownerId); + return; + } const t = recv.type; if ( t === 'identifier' || t === 'type_identifier' || t === 'simple_identifier' || @@ -5569,6 +5583,31 @@ export class TreeSitterExtractor { }); } + /** + * VB.NET: a value read or write through a name — `AppSession.SessionId`, + * `AppSession.CurrentUser = "demo"`, `Logger.Level`, `Mode.Fast` — is a use + * of the member as well as of what the name names (#2305). One `references` + * ref carries both, as `Name.Member` (the receiver kept, as a call's is); + * the resolver links the member and the type when the name means a project + * type or module there, and nothing when it holds a value (a local, a + * parameter, a field), which only it can tell in case-insensitive VB.NET + * (see vbnet-receivers' matchVbMemberRead). Its types are Capitalized all + * the same, so a lowercase receiver — a local, nearly always — is skipped. + */ + private extractVbMemberRead(node: SyntaxNode, recv: SyntaxNode, ownerId: string): void { + const member = getChildByField(node, 'member'); + if (recv.type !== 'identifier' || member?.type !== 'identifier') return; + const name = getNodeText(recv, this.source); + if (!/^[A-Z]\w*$/.test(name) || VB_NON_TYPE_RECEIVERS.test(name)) return; + this.unresolvedReferences.push({ + fromNodeId: ownerId, + referenceName: `${name}.${getNodeText(member, this.source)}`, + referenceKind: 'references', + line: node.startPosition.row + 1, + column: node.startPosition.column, + }); + } + /** * Find a `class_body` child of an `object_creation_expression` — the * marker for an anonymous class (`new T() { ... }`). Returns the body diff --git a/src/resolution/index.ts b/src/resolution/index.ts index 4c9fb45f2c..fcd47b7720 100644 --- a/src/resolution/index.ts +++ b/src/resolution/index.ts @@ -25,7 +25,7 @@ import { isPythonSelfCall, matchJsStoreBindingCall, isUnresolvedJsMemberCall, is import { isVisibleCppMacro, clearCppMacroVisibility } from './cpp-macro-visibility'; import { isCppConstructorRef, matchCppConstructor } from './cpp-constructor'; import { gateSwiftTypeTarget, clearSwiftTypeVisibility, swiftExtendedConformances } from './swift-type-visibility'; -import { clearVbnetReceiverMemos } from './vbnet-receivers'; +import { clearVbnetReceiverMemos, isVbMemberRead, matchVbMemberRead } from './vbnet-receivers'; import { gateTypeParameter, clearTypeParameterMemos } from './type-parameters'; import { resolveViaImport, resolvePhpImportedStaticCall, resolvePhpQualifiedClassRef, resolveJvmImport, extractImportMappings, extractReExports, loadCppIncludeDirs, isPhpIncludePathRef, isCobolCopybookRef, isNixPathImportRef, isJsPathImportRef, isBoundToOutOfRepoImport, clearImportResolverMemos, resolveImportPath, isExternalImport } from './import-resolver'; import { ResolverPool, minRefsForPool, shouldEngageAdaptively } from './resolver-pool'; @@ -1134,6 +1134,12 @@ export class ReferenceResolver { return this.gateLanguage(matchJsStoreBindingCall(ref, this.context), ref); } + // A VB.NET value read through a name (`AppSession.SessionId`, #2305) means + // what VB.NET's scoping says the name is — a project type, whose member + // and the type itself it links, or a value, which links nothing here — and + // no framework, import or name strategy guesses past that. + if (isVbMemberRead(ref)) return this.gateLanguage(matchVbMemberRead(ref, this.context), ref); + // Function-as-value refs (#756) get a dedicated, strictly-gated path: // import-based resolution first (an imported callback resolves through its // import, the most precise cross-file signal), then matchFunctionRef diff --git a/src/resolution/vbnet-receivers.ts b/src/resolution/vbnet-receivers.ts index 07732927e1..f252dfc979 100644 --- a/src/resolution/vbnet-receivers.ts +++ b/src/resolution/vbnet-receivers.ts @@ -432,9 +432,9 @@ function typesAround(file: string, line: number, context: ResolutionContext): No return types.filter((n) => n.startLine <= line && n.endLine >= line); } -/** The project's VB.NET types (classes, modules, structures, interfaces) named `name`, case aside. */ -function projectTypesNamed(name: string, context: ResolutionContext): Node[] { - return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && VB_TYPE_KINDS.has(n.kind)); +/** The project's VB.NET types (classes, modules, structures, interfaces — or `kinds`) named `name`, case aside. */ +function projectTypesNamed(name: string, context: ResolutionContext, kinds: ReadonlySet = VB_TYPE_KINDS): Node[] { + return context.getNodesByLowerName(name.toLowerCase()).filter((n) => n.language === 'vbnet' && kinds.has(n.kind)); } const VB_MEMBERS = new WeakMap>(); @@ -768,11 +768,16 @@ export function isVbTypeQualifiedBy(n: Node, qualifier: string, file: string, co * SCrawler declares * a `SiteSettings` in each site's namespace (`API.Pinterest`, `API.Bluesky`, * …); a member typed `SiteSettings` in `API.Pinterest.UserData` is - * Pinterest's. `ambiguous` when what is left are different types. + * Pinterest's. `ambiguous` when what is left are different types. `kinds` + * widens the types looked for (an `Enum` a value is read through). */ -function typesNamedAt(written: VbType, context: ResolutionContext): { owners: Node[]; ambiguous: boolean } { +function typesNamedAt( + written: VbType, + context: ResolutionContext, + kinds: ReadonlySet = VB_TYPE_KINDS, +): { owners: Node[]; ambiguous: boolean } { const t = written.file ? unalias(written, written.file, context) : written; - let candidates = projectTypesNamed(t.name, context); + let candidates = projectTypesNamed(t.name, context, kinds); // `System.Drawing.Color` is not the project's `Color`; `API.Base.UserDataBase` is that one. if (t.qualifier) candidates = candidates.filter((c) => endsWith(fullSegments(c, context).slice(0, -1), t.qualifier!)); if (candidates.length === 0 || !t.file) return { owners: candidates, ambiguous: false }; @@ -993,9 +998,9 @@ function ownersOf(t: VbType, context: ResolutionContext): Node[] | null { } /** - * A method — or, `withValues`, a field or property — named `name` on one of - * `owners` (the type `typed` names) or a project type they inherit, with what - * the type parameters of the type declaring it stand for. + * A method — or, `withValues`, a field or property; or what `fits` — named + * `name` on one of `owners` (the type `typed` names) or a project type they + * inherit, with what the type parameters of the type declaring it stand for. */ function memberOn( owners: Node[], @@ -1004,8 +1009,8 @@ function memberOn( context: ResolutionContext, withValues: boolean, typed?: VbType, + fits: (n: Node) => boolean = (n) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)), ): { node: Node; args: Map } | null { - const fits = (n: Node) => n.kind === 'method' || (withValues && VB_VALUE_KINDS.has(n.kind)); for (const level of [0, 1]) { for (const owner of preferVbProject(owners, ref, context)) { // What the owner's own parameters are, as the receiver's type writes them (`Repo(Of Foo)`). @@ -1238,7 +1243,11 @@ export function matchVbTypedCall( if (named.owners.length === 0) return undefined; if (named.ambiguous) return null; const shared = memberOn(named.owners, method, ref, context, false); - return shared ? { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' } : null; + if (shared) return { original: ref, targetNodeId: shared.node.id, confidence: 0.85, resolvedBy: 'qualified-name' }; + // `AppSession.Items(0)`: an index into a shared field or property, which + // VB.NET writes as a call — a read of the member and of its type (#2305). + const indexed = memberOn(named.owners, method, ref, context, true); + return indexed ? readThrough(indexed.node, named.owners, ref, context, { edgeKind: 'references' }) : null; } if (!type || typeKey(type) === 'object') return undefined; const owners = ownersOf(type, context); @@ -1251,6 +1260,115 @@ export function matchVbTypedCall( return extensionFor(type, owners, method, ref, context, isStdMethod); } +/** The types a value is read through: those a shared call is made on, and an `Enum`. */ +const VB_READ_TYPE_KINDS: ReadonlySet = new Set([...VB_TYPE_KINDS, 'enum']); + +/** + * What a read through a type names: a value it holds or gives (a field, a + * property, a constant, an `Enum` case, an event), a method — which VB.NET + * runs when it is named without parentheses — or a type nested in it. + */ +const VB_READ_KINDS: ReadonlySet = new Set([...VB_VALUE_KINDS, 'enum_member', 'method', ...VB_TYPE_LIKE_KINDS]); + +/** `Name.Member`, as the extractor sends a VB.NET value read through a name (extractVbMemberRead). */ +const VB_MEMBER_READ = /^([A-Za-z_]\w*)\.(\[?[A-Za-z_]\w*\]?)$/; + +/** Whether a reference is a VB.NET value read through a name, which matchVbMemberRead alone resolves. */ +export function isVbMemberRead(ref: UnresolvedRef): boolean { + return ref.language === 'vbnet' && ref.referenceKind === 'references' && VB_MEMBER_READ.test(ref.referenceName); +} + +/** + * The project types `name` means where a value is read through it + * (`AppSession` in `AppSession.SessionId`): a class, module, structure, + * interface or enum. Null when the name holds a value there — a local, a + * parameter, a member of a type around the read or of a module — or names + * no project type, or two. A member typed as the type of its own name + * (`Public Property Settings As Settings`) reaches that type's members + * either way, as VB.NET's "Color Color" rule has it. + */ +function typesReadThrough(name: string, ref: UnresolvedRef, context: ResolutionContext): Node[] | null { + const bound = receiverType(name, ref, context, 0); + const written: VbType | null = bound === undefined ? { name, array: false, file: ref.filePath, line: ref.line } + : bound && !bound.array && bound.name.toLowerCase() === name.toLowerCase() ? bound : null; + if (!written) return null; + const found = typesNamedAt(written, context, VB_READ_TYPE_KINDS); + return found.ambiguous || found.owners.length === 0 ? null : found.owners; +} + +/** + * A read of `member` through one of `owners` (the parts of the type its + * receiver names): the member, and the type as a second target — unless the + * read is written inside that type, which doesn't depend on itself. + */ +function readThrough( + member: Node, + owners: Node[], + ref: UnresolvedRef, + context: ResolutionContext, + extra: Partial = {}, +): ResolvedRef | null { + if (member.id === ref.fromNodeId) return null; + const owner = owners.find((o) => o.filePath === member.filePath) ?? owners[0]!; + const from = context.getNodeById?.(ref.fromNodeId)?.qualifiedName.toLowerCase(); + const own = owner.qualifiedName.toLowerCase(); + const inside = owner.id === ref.fromNodeId || (from !== undefined && (from === own || from.startsWith(`${own}::`))); + return { + original: ref, + targetNodeId: member.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + ...extra, + ...(inside ? {} : { alsoTargets: [{ targetNodeId: owner.id }] }), + }; +} + +/** Whether a method is named at a reference without being run: `AddressOf Type.Method`, `NameOf(Type.Method)`. */ +function namesWithoutRunning(ref: UnresolvedRef, context: ResolutionContext): boolean { + const line = context.getFileLines?.(ref.filePath)?.[ref.line - 1] ?? context.readFile(ref.filePath)?.split(/\r?\n/)[ref.line - 1] ?? ''; + return /\b(?:AddressOf\s+|NameOf\s*\(\s*)$/i.test(line.slice(0, Math.max(0, ref.column))); +} + +/** + * Resolve a VB.NET value read or write through a name (#2305) — + * `AppSession.SessionId`, `AppSession.CurrentUser = "demo"`, `Logger.Level`, + * `Mode.Fast` — to the member that the type the name means declares or + * inherits, and to the type. A method named this way is called (no type is + * linked, as for a call written with parentheses), unless `AddressOf` or + * `NameOf` only names it. Null when the name holds a value there (an + * instance's members are not read through its type), names no project type, + * or names one without that member: `Color.Red` is not a project `Color`'s. + */ +export function matchVbMemberRead(ref: UnresolvedRef, context: ResolutionContext): ResolvedRef | null { + const m = VB_MEMBER_READ.exec(ref.referenceName); + if (!m) return null; + const owners = typesReadThrough(m[1]!, ref, context); + if (!owners) return null; + // Not the member the read is written in: staxrip's `Overrides ReadOnly + // Property Package` returns the class's `Shared ReadOnly Property Package`. + const fits = (n: Node) => VB_READ_KINDS.has(n.kind) && n.id !== ref.fromNodeId; + // `MySettings.Default` reads the property declared `[Default]`, and back. + const name = m[2]!.replace(/^\[(.*)\]$/, '$1'); + let member = (memberOn(owners, name, ref, context, true, undefined, fits) ?? + memberOn(owners, `[${name}]`, ref, context, true, undefined, fits))?.node; + if (!member) return null; + // Of a `Shared` member and an instance one of the same name, a type's name reads the `Shared` one. + if (!member.isStatic) { + const found = member; + member = context.getNodesByQualifiedName(found.qualifiedName).find((n) => n.isStatic && n.filePath === found.filePath && fits(n)) ?? found; + } + if (member.kind === 'method') { + return { + original: ref, + targetNodeId: member.id, + confidence: 0.85, + resolvedBy: 'qualified-name', + ...(namesWithoutRunning(ref, context) ? {} : { edgeKind: 'calls' as const }), + }; + } + return readThrough(member, owners, ref, context, { confidence: 0.9 }); +} + const VB_TYPE_IMPORTS = new WeakMap>>(); /** The last names of a file's `Imports` (a type imported this way lends its shared members; an alias lends none). */ From a192df40f96cd3fee5bc86100e5555eeac9316a7 Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:06:42 +0000 Subject: [PATCH 07/23] fix(telemetry): count late index uploads, treat usage as live ingest, purge before catch-up (#2333) (#2356) Three bugs in the self-hosted telemetry services (telemetry-worker/ and telemetry-dashboard/), reported against 6560052a and all still on main. 1. An index run that uploads late never counted toward activation. The dashboard's funnel reads machine_first_seen.first_index_day, which only the nightly rollup wrote, and the rollup re-rolls just the last three days plus days with no daily_machines row. Ingest accepts timestamps up to 30 days old and the client keeps an event's original timestamp when it re-queues it, so an index event 4+ days late landed on a day nothing revisited and first_index_day stayed NULL. The ingest upsert of machine_first_seen now lowers first_index_day the way it lowers first_day: coalesce(min(old, new), old, new), because SQLite's min() is NULL if either side is. It is the same statement and row, with no new index and no migration (the column exists since 0002). The rollup still re-derives it from raw events, which covers events stored before this change. 2. ingest_stalled read only max(day) FROM events, but usage counters have lived in usage_daily since 0003, so a day with usage and no lifecycle events read as "The ingest worker ... is not storing anything". /api/meta now also reads max(day) FROM usage_daily (a primary-key lookup) and returns latest_usage_day and latest_ingest_day. Ingest counts as stalled only when there is no lifecycle event from yesterday or later and no usage counter from the day before yesterday or later, because clients upload a day's counters only after that day ends. The banner names latest_ingest_day. 3. runNightly rolled up the 3 regular days, then up to 31 missed days, and ran purgeOldEvents last. Each day first folds its legacy usage_rollup rows, which takes minutes on a heavy day, so a backlog could run past Cloudflare's 15-minute Cron Trigger limit and skip the purge and the summary line night after night. The purge now runs first: it is bounded and touches only days past the window, which no rollup reads. After that, rollups stop starting new days or fold chunks 10 minutes in (NIGHTLY_BUDGET_MS). A day stopped mid-fold gets no rollup at all, so it stays a missed day and the next night continues from its last committed chunk. The summary line counts those days as `deferred`, and `caught_up` now counts missed days actually rolled up. Verification: __tests__/telemetry-services.test.ts (new; the root vitest config picks it up) runs the worker's fetch/cron code and the dashboard API against the checked-in migrations in in-memory node:sqlite through a small D1-shaped adapter. With the source fix reversed, all 5 tests fail (activation 0, first_index_day null, ingest_stalled true, and "nightly rollup incomplete: 4 day(s) failed, purge failed" once statements pass the 15-minute mark). With the fix, all 5 pass. `npm run check` is clean in both packages. Against wrangler dev, run locally only: smoke:rollup 71/71, smoke:cutover 62/62, smoke:api 122/122 and render-check 87/87. smoke (ingest) is 46/47; the one failure is the 70 KB oversized-body curl argument hitting the Windows command-line limit, and main fails it the same way. The new smoke assertions fail on main. Takes effect only once both workers are redeployed; no migration. Co-authored-by: Claude Opus 5.5 --- __tests__/telemetry-services.test.ts | 438 ++++++++++++++++++++++ docs/design/telemetry.md | 6 +- telemetry-dashboard/README.md | 9 +- telemetry-dashboard/public/app.js | 3 +- telemetry-dashboard/scripts/fixture.sql | 1 + telemetry-dashboard/scripts/smoke-api.sh | 14 + telemetry-dashboard/src/api.ts | 51 ++- telemetry-worker/README.md | 32 +- telemetry-worker/scripts/smoke-cutover.sh | 11 +- telemetry-worker/scripts/smoke-ingest.sh | 8 + telemetry-worker/scripts/smoke-rollup.sh | 9 +- telemetry-worker/src/index.ts | 20 +- telemetry-worker/src/rollup.ts | 127 +++++-- 13 files changed, 662 insertions(+), 67 deletions(-) create mode 100644 __tests__/telemetry-services.test.ts diff --git a/__tests__/telemetry-services.test.ts b/__tests__/telemetry-services.test.ts new file mode 100644 index 0000000000..657681cc78 --- /dev/null +++ b/__tests__/telemetry-services.test.ts @@ -0,0 +1,438 @@ +/** + * The self-hosted telemetry services — `telemetry-worker/` (ingest + nightly rollup) + * and `telemetry-dashboard/` (the read API) — run against the checked-in D1 + * migrations in an in-memory node:sqlite database (#2333). + * + * D1 is SQLite, so every statement here is the SQL production runs. The one stand-in + * is a thin adapter giving node:sqlite D1's prepare/bind/first/all/run/batch shape, + * with `batch()` as one transaction the way D1 runs it. No wrangler, no workerd, no + * network: the end-to-end versions of these checks are the packages' own smoke + * suites (`npm run smoke`, `smoke:rollup`, `smoke:cutover`, `smoke:api`), which boot + * `wrangler dev`. + * + * npx vitest run __tests__/telemetry-services.test.ts + */ + +import { describe, it, expect, beforeEach, afterEach, vi } from 'vitest'; +import * as fs from 'fs'; +import * as path from 'path'; +import worker from '../telemetry-worker/src/index'; +import { runNightly } from '../telemetry-worker/src/rollup'; +import { handleApi } from '../telemetry-dashboard/src/api'; + +let nodeSqlite: typeof import('node:sqlite') | null = null; +try { + nodeSqlite = require('node:sqlite') as typeof import('node:sqlite'); +} catch { + /* Node < 22.5 — skipped below */ +} + +const MIGRATIONS = path.join(__dirname, '..', 'telemetry-worker', 'migrations'); +const MINUTE = 60_000; +/** Cloudflare ends a Cron Trigger invocation after 15 minutes of wall-clock time. */ +const CRON_LIMIT_MS = 15 * MINUTE; + +type Row = Record; + +// --------------------------------------------------------------------------- +// D1 over node:sqlite +// --------------------------------------------------------------------------- + +/** Binds integral numbers as INTEGER, as D1 does (node:sqlite would bind them as REAL). */ +const bindable = (v: unknown): unknown => (typeof v === 'number' && Number.isInteger(v) ? BigInt(v) : v); + +class SqliteD1Statement { + constructor( + private readonly d1: SqliteD1, + readonly sql: string, + private readonly params: unknown[] = [], + ) {} + + bind(...params: unknown[]): SqliteD1Statement { + return new SqliteD1Statement(this.d1, this.sql, params); + } + + execute(): { success: true; results: Row[]; meta: { changes: number } } { + this.d1.onStatement?.(this.sql); + const stmt = this.d1.db.prepare(this.sql); + const params = this.params.map(bindable) as never[]; + if (/^\s*(SELECT|WITH)\b/i.test(this.sql)) { + return { success: true, results: stmt.all(...params) as Row[], meta: { changes: 0 } }; + } + const info = stmt.run(...params); + return { success: true, results: [], meta: { changes: Number(info.changes) } }; + } + + async first(): Promise { + return (this.execute().results[0] as T | undefined) ?? null; + } + + async all(): Promise> { + return this.execute(); + } + + async run(): Promise> { + return this.execute(); + } +} + +class SqliteD1 { + /** Runs before every statement — the cron tests use it to model elapsed wall-clock time. */ + onStatement: ((sql: string) => void) | null = null; + + constructor(readonly db: import('node:sqlite').DatabaseSync) {} + + prepare(sql: string): SqliteD1Statement { + return new SqliteD1Statement(this, sql); + } + + async batch(statements: SqliteD1Statement[]): Promise[]> { + this.db.exec('BEGIN'); + try { + const results = statements.map((s) => s.execute()); + this.db.exec('COMMIT'); + return results; + } catch (err) { + this.db.exec('ROLLBACK'); + throw err; + } + } +} + +// --------------------------------------------------------------------------- +// Harness +// --------------------------------------------------------------------------- + +describe.skipIf(!nodeSqlite)('telemetry services against the D1 schema (#2333)', () => { + let db: import('node:sqlite').DatabaseSync; + let d1: SqliteD1; + let env: never; + let logged: Row[]; + + const at = (iso: string): void => { + vi.setSystemTime(new Date(iso)); + }; + + const one = (sql: string, ...params: unknown[]): Row | undefined => + db.prepare(sql).get(...(params.map(bindable) as never[])) as Row | undefined; + + /** POST /v1/events the way a client does, then wait for the off-response-path write. */ + async function post(machineId: string, events: Row[]): Promise { + const body = JSON.stringify({ + machine_id: machineId, + codegraph_version: '1.6.2', + os: 'linux', + arch: 'x64', + node_major: 22, + ci: false, + schema_version: 2, + events, + }); + const pending: Promise[] = []; + const request = new Request('https://telemetry.test/v1/events', { + method: 'POST', + headers: { 'content-type': 'application/json', 'content-length': String(Buffer.byteLength(body)) }, + body, + }); + const ctx = { waitUntil: (p: Promise) => void pending.push(p), passThroughOnException: () => {} }; + const response = await worker.fetch(request as never, env, ctx as never); + await Promise.all(pending); + return response.status; + } + + /** GET one dashboard endpoint, as the signed-in page does. */ + async function api(pathAndQuery: string): Promise { + const result = await handleApi(env, new URL(`https://stats.test${pathAndQuery}`)); + expect(result.status ?? 200).toBe(200); + return result.body as Row; + } + + /** + * One cron invocation at `iso`. `chunkMs` is the wall-clock time each legacy-usage + * fold chunk is charged; a statement that starts after Cloudflare's 15-minute limit + * throws, the way the runtime ends the invocation there. + */ + async function nightly(iso: string, chunkMs = 0): Promise<{ latestStartMs: number; summary: Row | undefined }> { + at(iso); + const started = Date.now(); + let latestStartMs = 0; + d1.onStatement = (sql) => { + const elapsed = Date.now() - started; + if (elapsed > CRON_LIMIT_MS) { + throw new Error(`statement started ${elapsed / MINUTE} min into the cron run — past the 15-minute limit`); + } + latestStartMs = Math.max(latestStartMs, elapsed); + if (/INSERT INTO usage_daily/.test(sql) && /FROM events/.test(sql)) vi.setSystemTime(Date.now() + chunkMs); + }; + const before = logged.length; + try { + await runNightly(env, Date.parse(iso)); + } finally { + d1.onStatement = null; + } + const summary = logged.slice(before).find((line) => line.msg === 'nightly rollup'); + return { latestStartMs, summary }; + } + + beforeEach(() => { + vi.useFakeTimers({ toFake: ['Date'] }); + db = new nodeSqlite!.DatabaseSync(':memory:'); + for (const file of fs.readdirSync(MIGRATIONS).filter((f) => f.endsWith('.sql')).sort()) { + db.exec(fs.readFileSync(path.join(MIGRATIONS, file), 'utf8')); + } + d1 = new SqliteD1(db); + env = { + DB: d1, + RETENTION_DAYS: 90, + MACHINE_RATE_LIMITER: { limit: async () => ({ success: true }) }, + ADMIN_RATE_LIMITER: { limit: async () => ({ success: true }) }, + } as never; + logged = []; + const capture = (line: unknown): void => { + try { + logged.push(JSON.parse(String(line)) as Row); + } catch { + /* not one of the worker's JSON lines */ + } + }; + vi.spyOn(console, 'log').mockImplementation(capture); + vi.spyOn(console, 'error').mockImplementation(capture); + }); + + afterEach(() => { + vi.restoreAllMocks(); + vi.useRealTimers(); + db.close(); + }); + + const M1 = '00000000-0000-4000-8000-000000000001'; + const M2 = '00000000-0000-4000-8000-000000000002'; + const M3 = '00000000-0000-4000-8000-000000000003'; + const install = (ts: string): Row => ({ event: 'install', ts, props: { scope: 'local', kind: 'fresh' } }); + const index = (ts: string): Row => ({ event: 'index', ts, props: { languages: ['typescript'] } }); + const usage = (day: string, count: number): Row => ({ + event: 'usage_rollup', + ts: `${day}T12:00:00.000Z`, + props: { kind: 'mcp_tool', name: 'codegraph_explore', count, client_name: 'Claude Code' }, + }); + + // ------------------------------------------------------------------------- + // 1. Activation counts an index run that uploads late + // ------------------------------------------------------------------------- + + describe('a late index upload', () => { + it('counts toward activation even when it arrives after the nightly run stopped revisiting its day', async () => { + // Sep 25: the machine installs. Its index run fails to upload and stays in the + // client's queue, keeping its original timestamp. + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + await nightly('2026-09-26T00:30:00Z'); + + // Oct 3: the queued index run finally goes out — eight days late, inside the + // 30 days ingest accepts, but long past the three days each nightly run re-rolls. + at('2026-10-03T10:00:00Z'); + expect(await post(M1, [index('2026-09-25T14:05:00Z')])).toBe(204); + // Someone else's activity, so the rollup's coverage moves past the cohort's window. + expect(await post(M2, [install('2026-10-03T09:00:00Z')])).toBe(204); + await nightly('2026-10-04T00:30:00Z'); + + at('2026-10-04T08:00:00Z'); + const funnel = await api('/api/activation?from=2026-09-25&to=2026-09-25&window=7'); + expect(funnel).toMatchObject({ installs: 1, activated: 1, dropped: 0, covered_through: '2026-10-03' }); + expect(one('SELECT first_day, first_index_day FROM machine_first_seen WHERE machine_id = ?', M1)).toEqual({ + first_day: '2026-09-25', + first_index_day: '2026-09-25', + }); + }); + + it('is applied as it is stored, and only ever moves the first index day earlier', async () => { + const firstSeen = (machineId: string): Row | undefined => + one('SELECT first_day, first_index_day FROM machine_first_seen WHERE machine_id = ?', machineId); + + at('2026-10-03T10:00:00Z'); + // A batch without an index run says nothing about indexing. + expect(await post(M1, [install('2026-10-02T10:00:00Z')])).toBe(204); + expect(await post(M1, [usage('2026-10-01', 4)])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: null }); + + // The earliest index run in a batch wins, with no rollup involved. + expect(await post(M1, [index('2026-10-03T09:00:00Z'), index('2026-10-02T11:00:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: '2026-10-02' }); + + // A later index run leaves it alone; a backdated one lowers it, and first_day with it. + expect(await post(M1, [index('2026-10-03T09:30:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-10-01', first_index_day: '2026-10-02' }); + expect(await post(M1, [index('2026-09-20T08:00:00Z')])).toBe(204); + expect(firstSeen(M1)).toEqual({ first_day: '2026-09-20', first_index_day: '2026-09-20' }); + + // A machine whose very first batch carries an index run. + expect(await post(M2, [install('2026-10-03T08:00:00Z'), index('2026-10-03T08:01:00Z')])).toBe(204); + expect(firstSeen(M2)).toEqual({ first_day: '2026-10-03', first_index_day: '2026-10-03' }); + + // The nightly rollup agrees with what ingest already wrote. + await nightly('2026-10-04T00:30:00Z'); + expect(firstSeen(M1)).toEqual({ first_day: '2026-09-20', first_index_day: '2026-09-20' }); + expect(firstSeen(M2)).toEqual({ first_day: '2026-10-03', first_index_day: '2026-10-03' }); + }); + }); + + // ------------------------------------------------------------------------- + // 2. Usage-only days are not a stalled ingest + // ------------------------------------------------------------------------- + + describe('the stalled-ingest flag', () => { + it('counts usage counters as ingest that is still storing', async () => { + // The last lifecycle event is Sep 25; after that only usage counters arrive. + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + // Clients upload a day's counters only once it is over: Oct 3's arrive on Oct 4. + at('2026-10-04T06:00:00Z'); + expect(await post(M2, [usage('2026-10-03', 12)])).toBe(204); + + at('2026-10-04T08:00:00Z'); + const meta = await api('/api/meta'); + expect(meta).toMatchObject({ latest_raw_day: '2026-09-25', latest_ingest_day: '2026-10-03', ingest_stalled: false }); + + // Just past midnight nobody has sent Oct 4's counters yet; that is not a stall. + at('2026-10-05T00:10:00Z'); + expect(await api('/api/meta')).toMatchObject({ ingest_stalled: false }); + + // A whole day later with nothing stored at all, it is — and the banner date is + // the last day anything was stored, not the last lifecycle event. + at('2026-10-06T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-10-03', ingest_stalled: true }); + }); + + it('still reports a stall when nothing at all has arrived, and not on an empty database', async () => { + at('2026-10-04T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: null, ingest_stalled: false }); + + at('2026-09-25T15:00:00Z'); + expect(await post(M1, [install('2026-09-25T14:00:00Z')])).toBe(204); + at('2026-10-04T08:00:00Z'); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-09-25', ingest_stalled: true }); + + // A lifecycle event from yesterday is fresh, as before. + expect(await post(M1, [index('2026-10-03T22:00:00Z')])).toBe(204); + expect(await api('/api/meta')).toMatchObject({ latest_ingest_day: '2026-10-03', ingest_stalled: false }); + }); + }); + + // ------------------------------------------------------------------------- + // 3. A long catch-up stays inside the cron's wall-clock limit, and the purge runs + // ------------------------------------------------------------------------- + + describe('the nightly run under a catch-up backlog', () => { + /** The rollup folds legacy usage rows 50,000 per transaction (rollup.ts, LEGACY_CHUNK_ROWS). */ + const CHUNK_ROWS = 50_000; + /** + * Each fold chunk is charged 4 minutes, so a backlog of legacy-heavy days costs + * several minutes apiece, as it did in production. The six missed days below hold + * seven chunks: 28 minutes of folding, against a 15-minute limit. + */ + const CHUNK_MS = 4 * MINUTE; + const NIGHT_1 = '2026-10-05T00:30:00Z'; + /** Missed days, newest first — the order catch-up takes them in. Sep 21 needs two chunks. */ + const MISSED = ['2026-09-23', '2026-09-22', '2026-09-21', '2026-09-20', '2026-09-19', '2026-09-18']; + const LEGACY_ROWS: Record = Object.fromEntries( + MISSED.map((day) => [day, day === '2026-09-21' ? CHUNK_ROWS + 3 : 3]), + ); + const PAST_WINDOW = '2026-06-01'; + + /** Legacy rows as they were stored before migrations/0003: one `events` row per upload, count 1. */ + function seedLegacyDay(day: string, rows: number): void { + db.prepare( + `WITH RECURSIVE n(i) AS (SELECT 1 UNION ALL SELECT i + 1 FROM n WHERE i < ?1) + INSERT INTO events (received_at, ts, day, event, machine_id, codegraph_version, os, arch, + node_major, ci, schema_version, props) + SELECT ?2 || 'T13:00:00.000Z', ?2 || 'T12:00:00.000Z', ?2, 'usage_rollup', + '00000000-0000-4000-8000-00000000010' || ((i - 1) % 5), '1.5.0', 'linux', 'x64', 22, 0, 1, + '{"kind":"mcp_tool","name":"codegraph_explore","count":1,"error_count":0,"client_name":"Claude Code"}' + FROM n`, + ).run(BigInt(rows), day); + for (let m = 0; m < Math.min(rows, 5); m++) { + const machine = `00000000-0000-4000-8000-00000000010${m}`; + db.prepare('INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, 1)').run(machine, day); + db.prepare( + `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) + ON CONFLICT (machine_id) DO UPDATE SET first_day = min(first_day, excluded.first_day)`, + ).run(machine, day); + } + } + + /** Where each seeded usage count sits right now, and whether the day has been rolled up. */ + function ledger(day: string) { + const folded = Number(one('SELECT coalesce(sum(count), 0) AS n FROM usage_daily WHERE day = ?', day)?.n); + const unfolded = Number( + one( + `SELECT coalesce(sum(json_extract(props, '$.count')), 0) AS n FROM events + WHERE day = ? AND event = 'usage_rollup'`, + day, + )?.n, + ); + const rolledUsage = one(`SELECT count FROM daily_event_counts WHERE day = ? AND event = 'usage_rollup'`, day); + const rolledUp = one('SELECT machines FROM daily_machines WHERE day = ?', day) !== undefined; + return { folded, unfolded, rolledUsage: rolledUsage ? Number(rolledUsage.count) : null, rolledUp }; + } + + function expectConsistent(): void { + for (const day of MISSED) { + const l = ledger(day); + // Nothing lost and nothing counted twice, however far the fold got. + expect(l.folded + l.unfolded, day).toBe(LEGACY_ROWS[day]); + if (l.rolledUp) { + expect(l, day).toMatchObject({ unfolded: 0, rolledUsage: LEGACY_ROWS[day] }); + } else { + // A day whose fold did not finish gets no rollup at all, so it stays "missed". + expect(l.rolledUsage, day).toBeNull(); + } + } + } + + beforeEach(() => { + for (const day of MISSED) seedLegacyDay(day, LEGACY_ROWS[day]!); + // Past the 90-day window: the purge must take these. + db.prepare( + `INSERT INTO events (received_at, ts, day, event, machine_id, props) + VALUES ('2026-06-01T10:00:00.000Z', NULL, ?1, 'install', ?2, '{}')`, + ).run(PAST_WINDOW, M3); + db.prepare( + `INSERT INTO usage_daily (day, machine_id, kind, name, count) VALUES (?, ?, 'cli_command', 'index', 2)`, + ).run(PAST_WINDOW, M3); + }); + + it('purges on schedule, stops before the limit, and resumes the backlog the next nights', async () => { + // The three days the run re-rolls every night have ordinary, current traffic. + at('2026-10-04T12:00:00Z'); + expect(await post(M1, [install('2026-10-04T11:00:00Z'), usage('2026-10-03', 5)])).toBe(204); + expect(await post(M2, [index('2026-10-02T11:00:00Z')])).toBe(204); + + const first = await nightly(NIGHT_1, CHUNK_MS); + expect(first.latestStartMs).toBeLessThanOrEqual(CRON_LIMIT_MS); + // The purge ran, and the summary line was written. + expect(one('SELECT count(*) AS n FROM events WHERE day < ?', '2026-07-07')?.n).toBe(0); + expect(one('SELECT count(*) AS n FROM usage_daily WHERE day < ?', '2026-07-07')?.n).toBe(0); + expect(first.summary).toMatchObject({ purged: 1, usage_purged: 1, failed: 0 }); + // Tonight's regular days were rolled up first. + for (const day of ['2026-10-04', '2026-10-03', '2026-10-02']) { + expect(one('SELECT machines FROM daily_machines WHERE day = ?', day), day).toEqual({ machines: 1 }); + } + // The backlog did not fit: Sep 21 was left part-folded and un-rolled, the rest untouched. + expect(ledger('2026-09-21')).toMatchObject({ folded: CHUNK_ROWS, unfolded: 3, rolledUp: false }); + expect(MISSED.filter((day) => !ledger(day).rolledUp)).toEqual(MISSED.slice(2)); + expect(first.summary).toMatchObject({ caught_up: 2, deferred: 4 }); + expectConsistent(); + + // The next nights pick up exactly where this one stopped. + let nights = 1; + for (let day = 6; MISSED.some((d) => !ledger(d).rolledUp) && nights < 5; day++, nights++) { + const night = await nightly(`2026-10-0${day}T00:30:00Z`, CHUNK_MS); + expect(night.latestStartMs).toBeLessThanOrEqual(CRON_LIMIT_MS); + expect(night.summary).toMatchObject({ failed: 0 }); + expectConsistent(); + } + expect(nights).toBe(3); + for (const day of MISSED) expect(ledger(day), day).toMatchObject({ unfolded: 0, rolledUp: true }); + }); + }); +}); diff --git a/docs/design/telemetry.md b/docs/design/telemetry.md index 530600d93c..bf5deed9cc 100644 --- a/docs/design/telemetry.md +++ b/docs/design/telemetry.md @@ -232,9 +232,9 @@ Full documentation is [`telemetry-dashboard/README.md`](../../telemetry-dashboar `telemetry-worker/scripts/smoke-cutover.sh` exists to cover — a mismatch there is silent, showing up as a panel that reads zero forever rather than as an error. - **Reads rollups, not raw events**, so a chart stays correct for days whose raw rows have - been purged. `/api/activation` is the one exception — "did this machine ever run an index" - is not a daily aggregate — so it reads raw `events` and is bounded by the retention window, - which it reports as `raw_events_from`. + been purged. The activation funnel's "did this machine ever run an index" is not a daily + aggregate, so it reads `machine_first_seen.first_index_day` instead, which the ingest Worker + lowers as each index event is stored and the nightly cron re-derives from raw `events`. - **Auth is a shared password and a signed cookie**, sized for exactly two people: `ADMIN_PASSWORD` + `SESSION_SECRET` as Worker secrets, constant-time compare, HMAC-signed cookie with no session store, everything except `/login` and `robots.txt` gated. Rotating diff --git a/telemetry-dashboard/README.md b/telemetry-dashboard/README.md index 26feaac2df..10c1fb3fc2 100644 --- a/telemetry-dashboard/README.md +++ b/telemetry-dashboard/README.md @@ -40,7 +40,7 @@ renders. Bad input is a `400` with a message, never a guess. Chart data carries | Endpoint | Answers | |---|---| -| `/api/meta` | How current the data is: `today`, the rollup's last day (`latest_rollup_day`), the last day an event arrived, machines active yesterday, and two flags — `ingest_stalled` and `rollup_behind` — that put a warning above the panels when either writer stops. Cached 60 s. | +| `/api/meta` | How current the data is: `today`, the rollup's last day (`latest_rollup_day`), the last day ingest stored anything (`latest_ingest_day` — a lifecycle event or a usage counter), machines active yesterday, and two flags — `ingest_stalled` and `rollup_behind` — that put a warning above the panels when either writer stops. Cached 60 s. | | `/api/summary` | Big numbers: production users, active machines, new machines, installs, uninstalls, indexing runs, tool calls. | | `/api/timeseries?metric=` | `installs_uninstalls`, `new_installs`, `production_users`, `indexing_activity`, `tool_calls`, `duration_buckets`. One dense point per day — a day with nothing is a zero, not a gap. Days after the rollup's last day are `null` ("not counted yet") and `covered_through` says where that is; `new_installs` is written live and runs through today. | | `/api/breakdown?dim=` | `os`, `arch`, `codegraph_version`, `node_major`, `language`, `file_count_bucket`, `duration_bucket`, `target`, `scope`, `kind`, `name`, `client_name`, `name_error`. Optional `&event=`, `&metric=count\|machines`, `&limit=`. | @@ -53,7 +53,8 @@ kept forever, so a chart stays correct for days whose raw events have been purge panel reads raw `events`.** D1 runs one query at a time per database: the activation funnel used to join every cohort machine against `events`, which took ~55 s for one week of cohorts in production and failed every panel queued behind it. It now reads -`machine_first_seen.first_index_day`, which the nightly rollup maintains. +`machine_first_seen.first_index_day`, which the ingest worker lowers as each index event is +stored and the nightly rollup re-derives from raw events. The presets end on **today** (UTC — every event and rollup is keyed on the UTC day). Live numbers — production users, new machines, retention — run through today; rolled-up ones @@ -138,6 +139,10 @@ npm run smoke:render # the panels, in a browser (79 assertions) Each suite starts its own throwaway `wrangler dev` on its own port and cleans up after itself, so they can be run in any order (`DASH_PORT` overrides the port). +The repo's own test suite also runs `src/api.ts`, with the ingest worker feeding it, against +the writer's migrations in an in-memory SQLite database — no wrangler needed. From the repo +root (it is part of `npm test` there): `npx vitest run __tests__/telemetry-services.test.ts`. + **`smoke-auth.sh`** is the regression net for the gate: unauthenticated requests reach nothing (pages, API *and* static assets), the cookie is persistent and correctly flagged, flipped/truncated/forged cookies are all rejected, brute force is capped, and rotating the diff --git a/telemetry-dashboard/public/app.js b/telemetry-dashboard/public/app.js index b29506aada..abb2f46efa 100644 --- a/telemetry-dashboard/public/app.js +++ b/telemetry-dashboard/public/app.js @@ -408,7 +408,8 @@ function drawDataWarning() { if (meta?.ingest_stalled) { lines.push([ - `No new events since ${shortDay(meta.latest_raw_day)}.`, + // The last day anything was stored, lifecycle event or usage counter. + `No new events since ${shortDay(meta.latest_ingest_day ?? meta.latest_raw_day)}.`, ' The ingest worker at telemetry.getcodegraph.com is not storing anything; its logs and the D1 database are where to look.', ]); } diff --git a/telemetry-dashboard/scripts/fixture.sql b/telemetry-dashboard/scripts/fixture.sql index 9e2ddef020..e9ca7c151d 100644 --- a/telemetry-dashboard/scripts/fixture.sql +++ b/telemetry-dashboard/scripts/fixture.sql @@ -36,6 +36,7 @@ DELETE FROM daily_machines; DELETE FROM machine_days; DELETE FROM machine_first_seen; DELETE FROM events; +DELETE FROM usage_daily; -- --------------------------------------------------------------------------- -- install — 12, one per machine on its first day diff --git a/telemetry-dashboard/scripts/smoke-api.sh b/telemetry-dashboard/scripts/smoke-api.sh index 67008c47b4..8dda68403b 100755 --- a/telemetry-dashboard/scripts/smoke-api.sh +++ b/telemetry-dashboard/scripts/smoke-api.sh @@ -105,6 +105,7 @@ field "retention window" retention_days 14 "$META" # The fixture stops on 07-10, months ago: nothing has arrived since, but the # rollup did cover every day that saw activity, so it is not behind. field "ingest stalled (no events since 07-10)" ingest_stalled true "$META" +field "…named by the last day anything was stored" latest_ingest_day 2026-07-10 "$META" field "rollup not behind (it covered every active day)" rollup_behind false "$META" field "nobody active yesterday" machines_yesterday 0 "$META" @@ -307,6 +308,19 @@ field "…and yesterday's machine is counted" machines_yesterday 1 "$META" npx wrangler d1 execute codegraph-telemetry --local \ --command "DELETE FROM machine_days WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 +echo +echo "Usage counters alone are not a stalled ingest" +# Usage lands in usage_daily, not events, so a day of tool calls with no install or +# index run must not read as "not storing anything". Removed again straight after. +npx wrangler d1 execute codegraph-telemetry --local \ + --command "INSERT INTO usage_daily (day, machine_id, kind, name, count) VALUES ('$YESTERDAY', '$STALE_ID', 'mcp_tool', 'codegraph_explore', 3)" >/dev/null 2>&1 +META="$(get "/api/meta")" +field "not stalled while usage arrives" ingest_stalled false "$META" +field "…the usage day is the last day stored" latest_ingest_day "$YESTERDAY" "$META" +field "…though the last lifecycle event is 07-10" latest_raw_day 2026-07-10 "$META" +npx wrangler d1 execute codegraph-telemetry --local \ + --command "DELETE FROM usage_daily WHERE machine_id = '$STALE_ID'" >/dev/null 2>&1 + echo printf '%d passed, %d failed\n' "$PASS" "$FAIL" [[ "$FAIL" -eq 0 ]] diff --git a/telemetry-dashboard/src/api.ts b/telemetry-dashboard/src/api.ts index 63b77556c6..24deb33522 100644 --- a/telemetry-dashboard/src/api.ts +++ b/telemetry-dashboard/src/api.ts @@ -6,8 +6,8 @@ * - **Rollups only.** Every panel is answered from `daily_*`, `machine_days` and * `machine_first_seen`, which are kept forever. No panel reads raw `events`: * D1 runs one query at a time per database, so one slow scan there fails every - * panel queued behind it. (/api/meta reads the table's first and last day, one - * indexed lookup each.) + * panel queued behind it. (/api/meta reads the table's first and last day, and + * `usage_daily`'s last, one indexed lookup each.) * - **Today is in range; uncounted days are not zeros.** Rolled-up numbers stop * at the nightly rollup's last day and come back null after it, so a chart * ending today draws a gap where the count has not happened yet, not a cliff. @@ -214,6 +214,7 @@ interface MetaRow { earliest_active_day: string | null; earliest_raw_day: string | null; latest_raw_day: string | null; + latest_usage_day: string | null; machines_yesterday: number | null; } @@ -246,6 +247,7 @@ async function meta(env: Env): Promise { (SELECT min(day) FROM machine_days) AS earliest_active_day, (SELECT min(day) FROM events) AS earliest_raw_day, (SELECT max(day) FROM events) AS latest_raw_day, + (SELECT max(day) FROM usage_daily) AS latest_usage_day, (SELECT count(*) FROM machine_days WHERE day = ?) AS machines_yesterday`, ) .bind(yesterday) @@ -253,11 +255,22 @@ async function meta(env: Env): Promise { const latestRollup = row?.latest_rollup_day ?? null; const latestRaw = row?.latest_raw_day ?? null; + const latestUsage = row?.latest_usage_day ?? null; const latestActive = row?.latest_active_day ?? null; - - // Nothing at all since before yesterday. (Client clocks may run a few minutes - // ahead, so the latest day can be tomorrow — that is fresh, not stale.) - const ingestStalled = latestRaw !== null && latestRaw < yesterday; + // Ingest stores lifecycle events in `events` and usage counters in `usage_daily`, so + // either one arriving means it is working — a day of usage and no installs or index + // runs is quiet, not stalled. + const latestIngest = + latestUsage !== null && (latestRaw === null || latestUsage > latestRaw) ? latestUsage : latestRaw; + + // Stalled: nothing stored since before yesterday. (Client clocks may run a few + // minutes ahead, so the latest day can be tomorrow — that is fresh, not stale.) Usage + // counters get a day more, because a client uploads a day's counters only once that + // day is over: just after midnight UTC the newest one can be the day before yesterday. + const ingestStalled = + latestIngest !== null && + (latestRaw === null || latestRaw < yesterday) && + (latestUsage === null || latestUsage < addDays(today, -2)); // The 00:30 UTC run rolls up yesterday, so the day before that must always be in // by now. Only "behind" if there was activity after the last rolled-up day — a day // nobody used codegraph would be a silent rollup, not a missed one. @@ -274,7 +287,12 @@ async function meta(env: Env): Promise { latest_rollup_day: latestRollup, latest_active_day: latestActive, earliest_raw_day: row?.earliest_raw_day ?? null, + /** The last day of a stored lifecycle event (install, index, uninstall). */ latest_raw_day: latestRaw, + /** The last day of a stored usage counter. */ + latest_usage_day: latestUsage, + /** The later of the two: the last day ingest stored anything, which the stall banner names. */ + latest_ingest_day: latestIngest, machines_yesterday: row?.machines_yesterday ?? 0, rollup_behind: rollupBehind, ingest_stalled: ingestStalled, @@ -670,17 +688,18 @@ interface ActivationRow { * reinstalls does not re-enter the funnel, which is what makes this a * conversion rate rather than an install-event ratio. * - * "Ran an index" is `machine_first_seen.first_index_day`, which the nightly rollup - * keeps at the earliest day each machine indexed. That makes this a range read over - * one small table. It used to be a join against raw `events` — on production volume - * ~55 s per week of cohorts, which held D1's single query lane long enough to fail - * every other panel waiting behind it. A first index day is never before the first - * day (the ingest path keeps first_day at the machine's earliest event), so "within - * the window" is just `first_index_day <= first_day + window`. + * "Ran an index" is `machine_first_seen.first_index_day`, the earliest day each + * machine indexed. The ingest worker lowers it as each index event is stored — so a + * run that uploads days late still counts — and the nightly rollup re-derives it from + * raw events. That makes this a range read over one small table. It used to be a + * join against raw `events` — on production volume ~55 s per week of cohorts, which + * held D1's single query lane long enough to fail every other panel waiting behind + * it. A first index day is never before the first day (the ingest path keeps + * first_day at the machine's earliest event), so "within the window" is just + * `first_index_day <= first_day + window`. * - * Because first_index_day is rolled up, cohorts after the rollup's last day have no - * conversions counted yet. They are left out of the totals and drawn as gaps — - * counting them would show a drop in conversion that is only a lag. + * Cohorts after the rollup's last day are left out of the totals and drawn as gaps, + * like every other rolled-up number on the page. */ async function activation(env: Env, url: URL, range: Range): Promise { const rawWindow = url.searchParams.get('window'); diff --git a/telemetry-worker/README.md b/telemetry-worker/README.md index 296ec2251d..077d0812f0 100644 --- a/telemetry-worker/README.md +++ b/telemetry-worker/README.md @@ -80,7 +80,8 @@ nothing looks broken from the outside. That ran from 2026-08-11 to October 2026. ## Rollups & retention (nightly cron) -`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things. +`src/rollup.ts` runs on a Cron Trigger at **00:30 UTC** and does two things, the purge +first. **Rolls up** the day that just ended into `daily_machines`, `daily_event_counts`, `daily_dim_counts` and `machine_first_seen.first_index_day`, then re-runs the two days before @@ -88,7 +89,12 @@ it — offline clients ship completed-day rollups late, so a day keeps growing a It then **catches up** on any earlier day that saw activity but never got a rollup (a `machine_days` day with no `daily_machines` row — a night the run failed or the database refused writes), newest first, up to 31 a night. An outage heals on the first good night -instead of leaving a hole someone has to notice. The aggregation is one +instead of leaving a hole someone has to notice. Cloudflare ends a Cron Trigger after 15 +minutes of wall-clock time, and a day that still holds millions of legacy usage rows (below) +takes minutes, so the run starts no new day and no new fold chunk once 10 minutes have +passed (`NIGHTLY_BUDGET_MS`). A day it stops partway gets no rollup at all, so it is still a +missed day the next night, which carries on from the last folded chunk; the summary line +counts those days as `deferred`. The aggregation is one `INSERT … SELECT … ON CONFLICT DO UPDATE` per table or dimension, so it happens inside D1 and no event row crosses the wire. Every write overwrites the recomputed value rather than adding to it: **re-running a day is a no-op, never a double count.** Two things the SQL is careful @@ -100,7 +106,9 @@ rollup over old days is the whole migration. Adding a breakdown is a line in `RO never a migration — that is what the generic `(dim, value)` shape buys. **Purges** raw `events` and `usage_daily` rows older than `RETENTION_DAYS` (90, a var in -`wrangler.jsonc`) in bounded `DELETE` batches, and logs one line of counts. `machine_days` and `machine_first_seen` are +`wrangler.jsonc`) in bounded `DELETE` batches, before any rollup: behind a long catch-up it +would be the part the 15-minute limit cuts off, night after night, while the database +grows. The run ends with one line of counts. `machine_days` and `machine_first_seen` are never purged — retention cohorts need the full history and they are two orders of magnitude smaller. Rollups are kept forever, so shortening the window costs ad-hoc drill-back, never a chart. @@ -122,10 +130,13 @@ each day is a full scan of that day's events, and the request has a wall-clock b ### Backfilling first_index_day -`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads, and only -the rollup writes it. Days rolled up before the migration left it NULL, and days stored before +`first_index_day` (migration `0002`) is what the dashboard's activation funnel reads. The +ingest worker lowers it as each index event is stored, so an index run that uploads days late +still counts, and the rollup re-derives it from the raw events of every day it rolls up. Days +rolled up before the migration left it NULL, an index event stored before the ingest worker +began setting it (#2333) counts only once its day is rolled up again, and days stored before `0003` still hold their usage as one `events` row per upload. Re-running the rollup over every -day that still has raw events fixes both — it folds that day's legacy usage rows into +day that still has raw events fixes all three — it folds that day's legacy usage rows into `usage_daily`, sets `first_index_day`, and recomputes the day's rollups. It is idempotent, so overlapping or repeating a range is harmless. A day with millions of legacy rows takes several minutes, so go a day at a time: @@ -300,6 +311,15 @@ To drive the cron body by hand, run `wrangler dev --test-scheduled` and hit `.dev.vars.example` to `.dev.vars` — without an `ADMIN_TOKEN` the route 404s, exactly as a deploy that never set the secret does. +The repo's own test suite also covers this worker and the dashboard API without wrangler: +it runs their source against these migrations in an in-memory SQLite database, including +late uploads and a nightly run whose catch-up outlasts the cron's time limit. From the repo +root (it is part of `npm test` there): + +```bash +npx vitest run __tests__/telemetry-services.test.ts +``` + ## Changing the schema The allowlist in `src/index.ts` mirrors `docs/design/telemetry.md` (and the user-facing diff --git a/telemetry-worker/scripts/smoke-cutover.sh b/telemetry-worker/scripts/smoke-cutover.sh index f0dff12443..6d55f8c79a 100755 --- a/telemetry-worker/scripts/smoke-cutover.sh +++ b/telemetry-worker/scripts/smoke-cutover.sh @@ -235,11 +235,14 @@ ts "tool calls (sums the prop)" tool_calls '[20]' '[2]' MET=$(api "meta") is "meta reports the rolled-up day" "$DAY" "$(jget "$MET" latest_day)" is "meta reports the rollup ran" "$DAY" "$(jget "$MET" latest_rollup_day)" +# Usage lands in usage_daily rather than events; the stalled-ingest check reads both. +is "meta reports the last day ingest stored" "$DAY" "$(jget "$MET" latest_ingest_day)" -# The funnel reads machine_first_seen.first_index_day, which only the nightly -# rollup writes (from raw `index` events). This is the seam that pins it: ingest -# writes the events, the rollup sets the column, the dashboard reads it. If the -# rollup stopped setting it, "activated" here would read 0. +# The funnel reads machine_first_seen.first_index_day. The ingest worker sets it as +# each index event is stored, and the nightly rollup re-derives it from raw `index` +# events (smoke-rollup.sh pins that half, on events it seeds past the ingest path). +# This is the seam: ingest writes the column, the dashboard reads it. If neither +# writer set it, "activated" here would read 0. # # Its denominator is FIRST-SEEN MACHINES, not `install` events (api.ts: "a machine # that reinstalls does not re-enter the funnel"). m3 is the discriminator: it never diff --git a/telemetry-worker/scripts/smoke-ingest.sh b/telemetry-worker/scripts/smoke-ingest.sh index 458123443a..9fd2b87fe4 100755 --- a/telemetry-worker/scripts/smoke-ingest.sh +++ b/telemetry-worker/scripts/smoke-ingest.sh @@ -214,6 +214,14 @@ is "machine_days: each backdated batch gets its own day" "$BACK_DAY,$RECENT_DAY" is "machine_first_seen recorded" "$RECENT_DAY" "$(q "select first_day from machine_first_seen where machine_id='$M_OK'")" is "machine_first_seen only moves earlier" "$BACK_DAY" \ "$(q "select first_day from machine_first_seen where machine_id='$M_BACK'")" +# The activation funnel's input is kept here too, not only by the nightly rollup: an +# index run that uploads after the cron stopped re-rolling its day must still count. +is "first_index_day recorded as the index run is stored" "$RECENT_DAY" \ + "$(q "select first_index_day from machine_first_seen where machine_id='$M_OK'")" +is "first_index_day only moves earlier, like first_day" "$BACK_DAY" \ + "$(q "select first_index_day from machine_first_seen where machine_id='$M_BACK'")" +is "a machine that never sent an index run has none" "null" \ + "$(q "select coalesce(first_index_day, 'null') from machine_first_seen where machine_id='$M_USE'")" echo echo "usage counters add up instead of piling up" diff --git a/telemetry-worker/scripts/smoke-rollup.sh b/telemetry-worker/scripts/smoke-rollup.sh index 294b4378f8..2befab4eb8 100755 --- a/telemetry-worker/scripts/smoke-rollup.sh +++ b/telemetry-worker/scripts/smoke-rollup.sh @@ -16,7 +16,8 @@ # * the purge deletes only rows past the window, and leaves machine_days / # machine_first_seen alone # * each machine's first_index_day is its earliest index, and survives the purge -# * the cron catches up on a day that saw activity but was never rolled up +# * the cron catches up on a day that saw activity but was never rolled up, and +# logs its one summary line # * usage comes from usage_daily; a legacy usage row still in `events` is folded # into it (counts added, row deleted) before the day is rolled up # * /admin/rollup does not exist without ADMIN_TOKEN, and rejects a wrong one @@ -196,6 +197,12 @@ is "rollup $DAY_RESET with reset → 200" 200 "$(roll "day=$DAY_RESET&reset=1")" # rolled up), and purges everything past the window. is "cron trigger → 200" 200 "$(curl -s -o /dev/null -w '%{http_code}' "$BASE/__scheduled?cron=30+0+*+*+*")" sleep 2 +# Its one summary line: written once the purge and every rollup are done, and with +# nothing this small left for the next night by the time budget. +is "the cron logs its summary, nothing failed or deferred" "0/0" \ + "$(grep -ao '{"msg":"nightly rollup".*}' "$LOG" | tail -1 | + node -e 'let s="";process.stdin.on("data",d=>s+=d).on("end",()=>{ + const r=JSON.parse(s||"{}");console.log(`${r.failed}/${r.deferred}`);})')" # Rolling a purged day with reset=1 must NOT blank the rollups it already has: past # the window the reset is ignored, so the delete-then-rebuild can't find zero events. diff --git a/telemetry-worker/src/index.ts b/telemetry-worker/src/index.ts index 9a6efa3394..3c4ac7cde3 100644 --- a/telemetry-worker/src/index.ts +++ b/telemetry-worker/src/index.ts @@ -200,9 +200,18 @@ const INSERT_EVENT = `INSERT INTO events ( const UPSERT_MACHINE_DAY = `INSERT INTO machine_days (machine_id, day, prod) VALUES (?, ?, ?) ON CONFLICT (machine_id, day) DO UPDATE SET prod = max(machine_days.prod, excluded.prod)`; -// A late-arriving offline buffer can move a machine's first day earlier, never later. -const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day) VALUES (?, ?) - ON CONFLICT (machine_id) DO UPDATE SET first_day = min(machine_first_seen.first_day, excluded.first_day)`; +// A late-arriving offline buffer can move a machine's first day earlier, never later — +// and its first index day (the activation funnel's input) the same way. That one is set +// here as well as by the nightly rollup because the rollup only re-rolls the last three +// days, while this path accepts events up to 30 days old: an index run that uploads a +// week late would otherwise never count. SQLite's multi-argument min() is NULL when +// either side is, hence the coalesce: a batch with no index run leaves the stored day +// alone, and the first index run on record sets it. +const UPSERT_FIRST_SEEN = `INSERT INTO machine_first_seen (machine_id, first_day, first_index_day) VALUES (?, ?, ?) + ON CONFLICT (machine_id) DO UPDATE SET + first_day = min(machine_first_seen.first_day, excluded.first_day), + first_index_day = coalesce(min(machine_first_seen.first_index_day, excluded.first_index_day), + machine_first_seen.first_index_day, excluded.first_index_day)`; // usage_rollup counters ADD into one row per machine × day × tool (migrations/0003): // clients upload the same counter many times over — once per process — so storing @@ -305,10 +314,13 @@ async function writeToD1( } } + // The earliest day this batch indexed on, if it did — see UPSERT_FIRST_SEEN. + let firstIndexDay: string | null = null; for (const e of batch) { if (e.event === 'usage_rollup') continue; const day = (e.ts ?? receivedAt).slice(0, 10); days.add(day); + if (e.event === 'index' && (firstIndexDay === null || day < firstIndexDay)) firstIndexDay = day; stmts.push( insertEvent.bind( receivedAt, @@ -328,7 +340,7 @@ async function writeToD1( const firstDay = [...days].sort()[0]; if (firstDay !== undefined) { - stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay)); + stmts.push(env.DB.prepare(UPSERT_FIRST_SEEN).bind(machineId, firstDay, firstIndexDay)); } await env.DB.batch(stmts); diff --git a/telemetry-worker/src/rollup.ts b/telemetry-worker/src/rollup.ts index 87072a59a9..9502210984 100644 --- a/telemetry-worker/src/rollup.ts +++ b/telemetry-worker/src/rollup.ts @@ -9,12 +9,13 @@ * 1. ROLL UP the just-completed UTC day into `daily_machines`, `daily_event_counts`, * `daily_dim_counts` and `machine_first_seen.first_index_day` — plus the two days * before it, because clients buffer offline and ship completed-day rollups late, so - * a day keeps growing after it ends, plus any earlier day a failed run missed. Every - * write is an upsert that OVERWRITES the recomputed value rather than adding to it - * (or, for first_index_day, only ever lowers it), so re-running a day is a no-op - * and never double-counts. + * a day keeps growing after it ends, plus any earlier day a failed run missed, as + * many as fit in the run's time budget. Every write is an upsert that OVERWRITES + * the recomputed value rather than adding to it (or, for first_index_day, only ever + * lowers it), so re-running a day is a no-op and never double-counts. * - * 2. PURGE raw `events` past the retention window, in bounded batches. Rollups are + * 2. PURGE raw `events` past the retention window, in bounded batches — first, so a + * long catch-up can never push it past the cron's wall-clock limit. Rollups are * kept forever, so only ad-hoc drill-down has a horizon; `machine_days` and * `machine_first_seen` are never purged, because retention cohorts need the full * history and they are two orders of magnitude smaller than the raw rows. @@ -32,6 +33,15 @@ export const ROLLUP_LOOKBACK_DAYS = 3; export const MAX_MANUAL_DAYS = 31; /** Most missed days one nightly run catches up on, newest first; the next night takes the rest. */ export const MAX_CATCHUP_DAYS = 31; +/** + * How long into a nightly run it still starts rollup work. Cloudflare ends a Cron + * Trigger after 15 minutes of wall-clock time, and catching up a day that still holds + * millions of legacy usage rows takes minutes, so a backlog can run straight into that + * limit. Past this point the run starts no new day and no new fold chunk; the 5 + * minutes left cover the one statement already in flight and the summary line. What + * did not fit stays a missed day, so the next night carries on from there. + */ +export const NIGHTLY_BUDGET_MS = 10 * 60_000; /** Rows per purge DELETE — bounded so one statement stays well inside D1's limits. */ const PURGE_BATCH_ROWS = 5_000; @@ -148,6 +158,10 @@ const DAILY_MACHINES = `INSERT INTO daily_machines (day, machines, prod_machines * it straight off `machine_first_seen`, which keeps the funnel off raw `events` (a * cohort join there took most of a minute per week of cohorts) and past the purge. * + * The ingest path lowers it the same way as each index event is stored (src/index.ts), + * because a run that uploads late lands on a day this cron has stopped re-rolling. This + * statement is what fills it for events stored before that, and what a backfill re-runs. + * * Only rows that actually move are written, so re-running a day is free. `?1` is the * day — still the one bound parameter, used three times. */ @@ -221,6 +235,11 @@ export interface DayResult { rows: number; /** Day is past the retention window — a `reset` on it is ignored (see below). */ pastRetention: boolean; + /** + * The deadline passed before the day's legacy usage rows were all folded, so nothing + * was rolled up: the next run finishes the fold and rolls the day up then. + */ + deferred: boolean; } /** @@ -236,14 +255,20 @@ export interface DayResult { * exists would otherwise linger. It is IGNORED past the retention window, where it * would delete rows and then find no events to rebuild them from: silently blanking a * real day is the one irreversible thing this file could do. + * + * `deadline` (epoch ms) bounds the legacy fold that runs first. If it stops the fold + * partway, the day is not rolled up at all — usage counted from a half-folded day would + * be short, and its `daily_machines` row would stop the nightly run from ever coming + * back to it. The chunks already folded are committed, so the next call carries on. */ export async function rollupDay( env: Env, day: string, - opts: { cutoff: string; reset?: boolean }, + opts: { cutoff: string; reset?: boolean; deadline?: number }, ): Promise { const pastRetention = day < opts.cutoff; - await foldLegacyUsage(env, day); + const fold = await foldLegacyUsage(env, day, opts.deadline); + if (!fold.complete) return { day, rows: 0, pastRetention, deferred: true }; const statements: D1PreparedStatement[] = []; if (opts.reset && !pastRetention) { @@ -257,7 +282,7 @@ export async function rollupDay( const results = await env.DB.batch(statements); const rows = results.reduce((total, r) => total + (r.meta?.changes ?? 0), 0); - return { day, rows, pastRetention }; + return { day, rows, pastRetention, deferred: false }; } // --------------------------------------------------------------------------- @@ -298,15 +323,32 @@ const LEGACY_FOLD = `INSERT INTO usage_daily ( const LEGACY_DELETE = `DELETE FROM events WHERE day = ?1 AND event = 'usage_rollup' AND id < ?2`; -/** Folds any legacy usage rows for `day` into usage_daily. Returns how many rows it moved. */ -export async function foldLegacyUsage(env: Env, day: string): Promise { +export interface FoldResult { + /** Legacy rows moved into usage_daily by this call. */ + moved: number; + /** False when the deadline stopped it with rows still to fold; the next call carries on. */ + complete: boolean; +} + +/** + * Folds any legacy usage rows for `day` into usage_daily, a chunk per transaction. + * With a `deadline` (epoch ms) it starts no chunk once that has passed: each chunk + * commits on its own and always takes the oldest rows left, so stopping between two + * loses nothing and the next call simply continues. + */ +export async function foldLegacyUsage( + env: Env, + day: string, + deadline = Number.POSITIVE_INFINITY, +): Promise { const any = await env.DB.prepare(`SELECT 1 AS hit FROM events WHERE day = ? AND event = 'usage_rollup' LIMIT 1`) .bind(day) .first<{ hit: number }>(); - if (!any) return 0; + if (!any) return { moved: 0, complete: true }; let moved = 0; for (let chunk = 0; chunk < LEGACY_MAX_CHUNKS; chunk++) { + if (Date.now() >= deadline) return { moved, complete: false }; // Always the oldest remaining rows: everything below the next chunk's first id. // Rows folded by earlier chunks are gone, so there is no cursor to carry. const next = await env.DB.prepare( @@ -320,7 +362,7 @@ export async function foldLegacyUsage(env: Env, day: string): Promise { env.DB.prepare(LEGACY_DELETE).bind(day, below), ]); moved += deleted?.meta?.changes ?? 0; - if (!next) return moved; + if (!next) return { moved, complete: true }; } throw new Error(`legacy usage fold for ${day} did not finish within ${LEGACY_MAX_CHUNKS} chunks`); } @@ -399,56 +441,81 @@ async function missedDays(env: Env, cutoff: string, before: string): Promise { const started = Date.now(); + const deadline = started + NIGHTLY_BUDGET_MS; const keepDays = retentionDays(env); const cutoff = retentionCutoff(atMs, keepDays); - const days: string[] = []; - for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) days.push(utcDay(atMs - back * DAY_MS)); + let purge: PurgeResult | null = null; + try { + purge = await purgeOldEvents(env, cutoff); + } catch (err) { + console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) })); + } + + const regular: string[] = []; + for (let back = 1; back <= ROLLUP_LOOKBACK_DAYS; back++) regular.push(utcDay(atMs - back * DAY_MS)); // A failure to find the missed days must not cost tonight's regular rollup. - let caughtUp = 0; + let missed: string[] = []; try { - const missed = await missedDays(env, cutoff, days[days.length - 1] ?? utcDay(atMs)); - days.push(...missed); - caughtUp = missed.length; + missed = await missedDays(env, cutoff, regular[regular.length - 1] ?? utcDay(atMs)); } catch (err) { console.error(JSON.stringify({ msg: 'missed-day scan failed', err: String(err) })); } const rolled: string[] = []; const failed: string[] = []; + const deferred: string[] = []; + let caughtUp = 0; let rows = 0; - for (const day of days) { + for (const day of [...regular, ...missed]) { + if (Date.now() >= deadline) { + deferred.push(day); + continue; + } try { - rows += (await rollupDay(env, day, { cutoff })).rows; + const result = await rollupDay(env, day, { cutoff, deadline }); + if (result.deferred) { + deferred.push(day); + continue; + } + rows += result.rows; rolled.push(day); + if (!regular.includes(day)) caughtUp++; } catch (err) { failed.push(day); console.error(JSON.stringify({ msg: 'rollup day failed', day, err: String(err) })); } } - let purge: PurgeResult | null = null; - try { - purge = await purgeOldEvents(env, cutoff); - } catch (err) { - console.error(JSON.stringify({ msg: 'purge failed', cutoff, err: String(err) })); - } - console.log( JSON.stringify({ msg: 'nightly rollup', days: rolled, + /** Missed days rolled up tonight. */ caught_up: caughtUp, + /** Days the time budget left for the next run. */ + deferred: deferred.length, rows, failed: failed.length, retention_days: keepDays, From 8998697f1d87fc670771817a11943868d4821e1a Mon Sep 17 00:00:00 2001 From: Colby Mchenry Date: Mon, 5 Oct 2026 20:17:51 +0000 Subject: [PATCH 08/23] fix(vue,ts): link calls in Vue templates and in top-level destructuring (#2340) (#2357) `codegraph callers ` missed most of the components using it in a Nuxt app (Issue #2340), through two extraction gaps: - A destructuring declaration at module or ` +`; + const code = ending === 'CRLF' ? lf.replace(/\n/g, '\r\n') : lf; + const result = extractFromSource('Card.vue', code); + const component = result.nodes.find((n) => n.kind === 'component')!; + const calls = result.unresolvedReferences + .filter((r) => r.referenceKind === 'calls') + .map((r) => `${r.fromNodeId === component.id ? 'component' : r.fromNodeId}:${r.referenceName}@${r.line}`) + .sort(); + expect(calls).toEqual([ + 'component:label@3', + 'component:useBar@2', + 'component:useBar@2', + 'component:useFoo@7', + 'component:useFoo@8', + ]); + }); + it('should extract calls from Vue Options API object methods', () => { const code = `