Natives Text/Search Pipeline

This document maps the @musepi/pi-natives text/search/code surface from generated JS/TS exports to Rust N-API modules and back to JS result objects.

Terminology follows docs/natives-architecture.md:

• Generated binding: public API in packages/natives/native/index.d.ts.
• Rust module layer: N-API exports in crates/pi-natives/src/*.
• Shared scan cache: fs_cache-backed directory-entry cache used by discovery/search flows.

Implementation files

• packages/natives/native/index.d.ts
• crates/pi-natives/src/grep.rs
• crates/pi-natives/src/glob.rs
• crates/pi-natives/src/glob_util.rs
• crates/pi-natives/src/fs_cache.rs
• crates/pi-natives/src/fd.rs
• crates/pi-natives/src/ast.rs
• crates/pi-natives/src/text.rs
• crates/pi-natives/src/highlight.rs
• crates/pi-natives/src/tokens.rs

JS API ↔ Rust export mapping

Pipeline overview by subsystem

1) Regex search (grep, search, hasMatch)

Input/options flow

1. Callers invoke generated native exports directly; there is no package-local TS wrapper that renames search to searchContent.
2. Rust option structs in grep.rs deserialize camelCase fields (ignoreCase, maxCount, contextBefore, contextAfter, maxColumns, timeoutMs).
3. grep creates CancelToken from timeoutMs + AbortSignal and runs inside task::blocking("grep", ...).
4. search and hasMatch operate on provided string/Uint8Array content and do not scan the filesystem.

Execution branches

• In-memory branch
  • search -> search_sync / search helpers over provided content bytes.
  • hasMatch compiles/checks pattern against provided content and returns a boolean.
  • No filesystem scan, no fs_cache.
• Single-file branch
  • grep resolves path, checks metadata is file, and searches that file.
• Directory branch
  • Optional cache lookup via fs_cache::get_or_scan when cache: true.
  • Fresh scan via fs_cache::force_rescan when cache: false.
  • Optional empty-result recheck when cached results are older than the empty-result recheck threshold.
  • Entry filtering: file-only + optional glob filter (glob_util) + optional type filter mapping (js, ts, rust, etc.).

Search/collection semantics

• Regex engine: grep_regex::RegexMatcherBuilder with ignoreCase and multiline.
• Context resolution:
  • contextBefore/contextAfter override legacy context.
  • Non-content modes do not collect context.
• Output modes:
  • content -> one GrepMatch per hit.
  • count and filesWithMatches map to count-style entries (lineNumber=0, line="", matchCount set).
  • offset and maxCount are applied during aggregation across sorted file results.
  • Directory searches use parallel filesystem walking/searching, then aggregate per-file results to preserve global offset/limit semantics in the returned result and callback stream.

Result shaping back to JS

• Rust SearchResult/GrepResult fields map to TS interfaces via N-API object conversion.
• Counters are clamped before crossing N-API where needed.
• GrepResult.limitReached is optional and emitted when true.
• Streaming callback receives each shaped GrepMatch for content or count-style entries.

Failure behavior

• search returns SearchResult.error for regex/search failures instead of throwing.
• grep rejects on hard errors such as invalid path, invalid glob/regex, or cancellation timeout/abort.
• hasMatch returns a boolean on success and throws on invalid pattern/UTF-8 conversion errors.
• File open/search errors in multi-file scans are skipped per-file; scan continues.

Malformed regex handling

grep.rs sanitizes braces before regex compile:

• Invalid repetition-like braces are escaped ({/} -> \{/\}) when they cannot form {N}, {N,}, {N,M}.
• This prevents common literal-template fragments (for example ${platform}) from failing as malformed repetition.
• After brace sanitization, a compile error reporting an unclosed/unopened group triggers one retry with unescaped parentheses escaped, so literal snippets like fetchAnthropicProvider( still search instead of erroring.
• Remaining invalid regex syntax still returns a regex error.

2) File discovery (glob) and fuzzy path search (fuzzyFind)

glob and fuzzyFind share fs_cache scans; matching logic differs.

glob flow

1. Caller passes GlobOptions directly. pattern and path are required in the generated type.
2. Rust resolves the search path and compiles pattern via glob_util::compile_glob.
3. Entry source:
   • cache=true -> get_or_scan + optional stale-empty force_rescan.
   • cache=false -> force_rescan(..., store=false) (fresh only).
4. Filtering:
   • skip .git always;
   • skip node_modules unless requested (includeNodeModules) or pattern mentions node_modules;
   • apply glob match;
   • apply file-type filter; symlink file/dir filters resolve target metadata.
5. Optional sort by mtime descending (sortByMtime) before truncating to maxResults.

fuzzyFind flow

1. Rust implementation lives in fd.rs; generated export is fuzzyFind.
2. Shared scan source from fs_cache with the same cache/no-cache split and stale-empty recheck policy.
3. Scoring:
   • exact / starts-with / contains / subsequence-based fuzzy score;
   • separator/punctuation-normalized scoring path;
   • directory bonus and deterministic tie-break (score desc, then path asc).
4. Symlink entries are excluded from fuzzy results.

Failure behavior

• Invalid glob pattern returns an error from glob_util::compile_glob.
• Search root must resolve to an existing directory for directory discovery flows.
• Cancellation/timeouts propagate as abort errors via CancelToken::heartbeat() checks in loops.

Malformed glob handling

glob_util::build_glob_pattern is tolerant:

• normalizes \ to /,
• auto-prefixes simple recursive patterns with **/ when recursive=true,
• auto-closes unbalanced {... alternation groups before compile.

3) AST search/match/edit (astGrep, astMatch, astEdit)

ast.rs exposes syntax-aware code search and rewrite operations.

• astGrep(options) returns matches with byte/line/column coordinates and optional metavariable bindings.
• astMatch(options) runs the same patterns against an in-memory source string instead of files; lang is required (there is no path to infer it from), and the result keeps matches, totalMatches, limitReached, and parse errors but omits the file-count fields.
• astEdit(options) returns replacement changes, per-file counts, searched/touched file counts, parse errors, and whether edits were applied.
• dryRun defaults to true for edit options in the generated documentation.
• Options include language override, path/glob/selector, strictness, limits, parse-error policy, signal, and timeoutMs.

These exports are direct native APIs used by tooling; they are not mediated by a TS wrapper in packages/natives.

4) Shared scan/cache lifecycle (fs_cache)

fs_cache stores scan results as normalized relative entries (path, fileType, optional mtime and regular-file size) keyed by:

• canonical search root,
• include_hidden,
• use_gitignore,
• skip_node_modules,
• scan detail (Minimal vs Full).

follow_links affects a fresh scan but is not currently part of the cache key.

Cache state transitions

1. Miss / disabled
   • TTL is 0 or key absent/expired -> fresh collection.
2. Hit
   • Entry age is within TTL -> return cached entries + cache_age_ms.
3. Stale-empty recheck
   • If query yields zero matches and cache age exceeds the empty-result threshold, force one rescan.
4. Invalidation
   • invalidateFsScanCache(path?):
     • no arg: clear all keys;
     • path arg: remove keys for roots affected by that path.

Stale-result tradeoff

• Cache favors low-latency repeated scans over immediate consistency.
• TTL window can return stale positives/negatives.
• Empty-result recheck reduces stale negatives for older cached scans at the cost of one extra scan.
• Explicit invalidation is the intended correctness hook after file mutations.

5) ANSI text utilities (text)

These are pure, in-memory utilities.

Boundaries and responsibilities

• text.rs owns terminal-cell semantics:
  • ANSI sequence parsing,
  • grapheme-aware width and slicing,
  • wrap/truncate/slice behavior,
  • explicit tab-width parameter on width-sensitive APIs.
• grep.rs line truncation (maxColumns) is separate:
  • simple character-boundary truncation of matched lines with ...,
  • not ANSI-state-preserving and not terminal-cell width aware.

Key behaviors

• wrapTextWithAnsi: wraps by visible width, carries active SGR codes across wrapped lines.
• truncateToWidth: visible-cell truncation with ellipsis policy (Unicode, Ascii, Omit), optional right padding.
• sliceWithWidth: column slicing with optional strict width enforcement.
• extractSegments: extracts before/after segments around an overlay while restoring ANSI state for the after segment.
• sanitizeText (ANSI/control/surrogate stripping with line-ending normalization) no longer lives in text.rs; it moved to @musepi/pi-utils as a pure-JS implementation in packages/utils/src/sanitize-text.ts. The native binding was removed in the same change because the JS version was competitive on the benchmarked workloads, and keeping a Rust copy forced every caller (including pi-utils) to pull in @musepi/pi-natives.
• visibleWidth: counts visible terminal cells using caller-supplied tab width.

Failure behavior

Text functions generally return deterministic transformed output; errors are limited to N-API argument/string conversion boundaries.

6) Syntax highlighting (highlight)

highlight.rs is pure transformation; it does not use the filesystem scan cache.

Flow

1. Caller passes code, optional lang, and ANSI color palette.
2. Rust resolves syntax by token/name lookup, extension lookup, alias table fallback, then plain-text fallback.
3. Each line is parsed with syntect ParseState and scope stack.
4. Scopes map to semantic color categories and ANSI color codes are injected/reset.

Failure behavior

• Per-line parse failure does not fail the call: that line is appended unhighlighted and processing continues.
• Unknown/unsupported language falls back to plain text syntax.

7) Token counting (tokens)

countTokens(input, encoding?) is an in-memory utility.

• input may be a single string or an array of strings.
• Arrays return one aggregate count and are encoded in parallel in Rust.
• Default encoding is O200kBase; Cl100kBase is also available.
• The implementation uses ordinary tokenization, not special-token handling.

Pure utility vs filesystem-dependent flows

End-to-end lifecycle summary

1. Caller invokes generated native export with typed options.
2. Rust validates/normalizes options and builds matcher/search config.
3. For filesystem flows, entries are scanned (cache hit/miss/rescan where applicable) then filtered/scored/searched.
4. Worker loops periodically call cancel heartbeat; timeout/abort can terminate execution.
5. Rust shapes outputs into N-API objects (lineNumber, matchCount, limitReached, etc.).
6. Generated bindings return typed JS objects and optional per-match callbacks for grep/glob.