Natives Architecture

@musepi/pi-natives is a two-layer package around an ESM loader:

1. ESM loader/package entrypoint resolves and loads the correct .node addon with createRequire, validates the release sentinel outside workspace-dev loads, and re-exports generated classes/functions plus enum runtime objects as explicit named ESM exports.
2. Rust N-API module layer implements the exported functions/classes and emits the generated TypeScript declarations.

This document is the foundation for deeper module-level docs.

Implementation files

• packages/natives/native/index.js
• packages/natives/native/index.d.ts
• packages/natives/native/loader-state.js
• packages/natives/native/embedded-addon.js
• packages/natives/scripts/build-native.ts
• packages/natives/scripts/embed-native.ts
• packages/natives/scripts/gen-enums.ts
• packages/natives/package.json
• crates/pi-natives/src/lib.rs

Package entrypoint and public surface

packages/natives/package.json points at generated native artifacts:

• main: ./native/index.js
• types: ./native/index.d.ts
• exports["."].types: ./native/index.d.ts
• exports["."].import: ./native/index.js

There is no current packages/natives/src TypeScript wrapper layer. Consumers import functions/classes/enums directly from @musepi/pi-natives; the type contract is the generated native/index.d.ts plus the explicit named exports generated into native/index.js by scripts/gen-enums.ts.

Current capability groups in the generated API include:

• Search/text/code primitives: grep, search, hasMatch, fuzzyFind, glob, astGrep, astEdit, blockRangeAt, summarizeCode, text width/slicing/wrapping/sanitization, syntax highlighting, token counting.
• Execution/process/terminal primitives: executeShell, Shell, PtySession, Process, key parsing, bash fixups.
• System/media/isolation/conversion primitives: clipboard, SIXEL encoding, HTML-to-Markdown, macOS appearance/power helpers, work profiling, workspace scanning, isolation backend helpers (iso*).

Loader layer

packages/natives/native/index.js is the package entrypoint; it calls loadNative() from loader-state.js, which owns runtime addon selection and optional embedded extraction.

Candidate resolution model

• Platform tag is ${process.platform}-${process.arch}.
• Supported tags are currently:
  • linux-x64
  • linux-arm64
  • darwin-x64
  • darwin-arm64
  • win32-x64
• x64 can use CPU variants:
  • modern (AVX2-capable)
  • baseline (fallback)
• Non-x64 uses the default filename without a variant suffix.

Filename strategy:

• Default: pi_natives.<platform>-<arch>.node
• x64 variant: pi_natives.<platform>-<arch>-modern.node or ...-baseline.node
• x64 runtime fallback includes the unsuffixed default filename after variant candidates.

Platform-specific variant detection

For x64, variant selection uses:

• Linux: /proc/cpuinfo
• macOS: sysctl -n machdep.cpu.leaf7_features, then machdep.cpu.features
• Windows: PowerShell check for System.Runtime.Intrinsics.X86.Avx2

PI_NATIVE_VARIANT can force modern or baseline; invalid values are ignored.

Binary distribution and extraction model

The published @musepi/pi-natives package ships only the loader layer in native/: the ESM loader (index.js), generated declarations (index.d.ts), the loader-state.js/.d.ts helpers, and the embedded-addon manifest stub (embedded-addon.js). It carries no .node binaries.

Each platform’s prebuilt .node is published as a separate optional-dependency leaf package — @musepi/pi-natives-<platform>-<arch>, one per supported tag — which the core lists in optionalDependencies at the lockstep version during publish. npm/bun install only the leaf whose os/cpu match the host. The working-tree package keeps built .node files under native/ for local dev; the release-publish rewrite (prepareNativeCorePackage in scripts/ci-release-publish.ts) strips them from the core tarball, and the leaves are generated by packages/natives/scripts/gen-npm-packages.ts (LEAF_TARGETS). Adding a build target therefore requires a matching LEAF_TARGETS entry, or the binary never reaches npm users.

For compiled binaries, loader behavior is:

1. Check versioned user cache path: <getNativesDir()>/<packageVersion>/....
2. Check legacy compiled-binary location:
   • Windows: %USERPROFILE%\.musepi
   • non-Windows: ~/.local/bin
3. Fall back to packaged native/ and executable directory candidates.

getNativesDir() uses $XDG_DATA_HOME/musepi/natives when $XDG_DATA_HOME/musepi exists; otherwise it uses ~/.musepi/natives.

If a populated embedded addon manifest is present, it is also treated as a compiled-binary signal. Current embedded manifests point at a gzip-compressed tar archive (embedded-addons.<tag>.tar.gz) that contains one or more matching .node files. The loader extracts the archive into the versioned cache directory, validates the selected file by size, and prepends that cache path before normal candidate probing.

For npm/bun installs (non-compiled), loader-state.js resolves the platform leaf directory via require.resolve("@musepi/pi-natives-<tag>/package.json") and probes its .node before the core package’s native/ directory and the executable directory. The optional-dependency binary is therefore preferred over any .node left in the core (e.g. a stale local-dev build). On Windows node_modules installs, the loader first stages the selected leaf/core addon into <getNativesDir()>/<packageVersion>/... and prepends that staged path so running processes do not lock the node_modules copy during global updates.

Failure modes

Loader failures are explicit:

• Unsupported platform tag: after failed probing, throws with supported platform list.
• No loadable candidate: throws with all attempted paths and remediation hints.
• Embedded/staging errors: directory/write/archive/staging failures are recorded and included in final load diagnostics if no candidate loads.
• Release mismatch: outside workspace-dev loads, a candidate that loads but lacks the version sentinel export for package.json#version is rejected with a reinstall hint.

Rust N-API module layer

crates/pi-natives/src/lib.rs declares exported module ownership:

• appearance
• ast
• block
• clipboard
• crash_handler
• fd
• fs_cache
• glob
• glob_util
• grep
• highlight
• html
• iso
• keys
• language (re-exported from pi_ast)
• power
• prof
• ps
• pty
• shell
• sixel
• snapcompact
• summary
• task
• text
• tokens
• utils (crate-private helpers)
• workspace

N-API exports are generated from Rust #[napi] functions/classes/objects/enums. Snake_case Rust names are exposed as camelCase JavaScript names unless explicitly configured by napi-rs.

Ownership boundaries

• Loader/package ownership (packages/natives/native, packages/natives/scripts)
  • runtime binary selection
  • CPU variant selection and override handling
  • compiled-binary embedded archive extraction
  • Windows node_modules addon staging
  • generated TypeScript declarations and explicit ESM export/enum patching
• Rust ownership (crates/pi-natives/src)
  • algorithmic and system-level implementation
  • platform-native behavior and performance-sensitive logic
  • N-API symbol implementation consumed directly by package callers
• Consumer ownership (packages/coding-agent, packages/tui)
  • user-facing policy and fallbacks that are not built into the native API
  • higher-level rendering, artifact, shell-session, and command behavior

For the contributor-facing crate map covering pi-natives, pi-shell, pi-ast, pi-iso, pi-walker, pi_uu_grep, pi-uutils-ctx, and the vendored brush-* crates, see native-crates.md. The root-docs inclusion policy that keeps internal Rust crates under native architecture docs unless promoted as user-facing also lives in user-facing-packages.md.

Runtime flow (high level)

1. Consumer imports from @musepi/pi-natives.
2. native/index.js computes platform/arch/variant and candidate paths.
3. Optional embedded archive extraction or Windows node_modules staging can prepend a versioned-cache candidate.
4. Each candidate is require(...)d; install/compiled loads must expose the package-version sentinel.
5. The loaded addon object is bound to explicit named ESM exports, including generated enum objects.
6. Caller invokes generated N-API functions/classes directly.

Glossary

• Native addon: A .node binary loaded via Node-API (N-API).
• Platform tag: Runtime tuple platform-arch (for example darwin-arm64).
• Platform leaf package: Per-platform npm package @musepi/pi-natives-<tag> that carries one platform’s prebuilt .node. The core depends on every leaf via optionalDependencies; the package manager installs only the host-matching one (os/cpu).
• Variant: x64 CPU-specific build flavor (modern AVX2, baseline fallback).
• Generated binding declaration: native/index.d.ts emitted by napi-rs during build-native.ts.
• Version sentinel: Rust export named from the package version (for example __piNativesV16_0_3) that lets the loader reject a .node from a different release.
• Compiled binary mode: Runtime mode where the CLI is bundled and native addons are resolved from embedded/cache paths before package-local paths.
• Embedded addon: Build artifact metadata and archive reference generated into native/embedded-addon.js so compiled binaries can extract matching .node payloads.