Technical Details¶

This document describes emend’s internal design, the rationale behind key architectural choices, and the dependencies that underpin each subsystem.

Overview¶

emend is structured around two complementary refactoring primitives that share a common execution model:

Structured edits — component-level surgery on symbol metadata (parameters, return types, decorators, bases, body) addressed via a selector grammar.
Pattern transforms — code-pattern search and replace using metavariable capture syntax ($X, $...ARGS) compiled to a unified structural matcher.

Both systems parse source files into a concrete syntax tree (CST) using Tree-sitter via a Rust backend. Transformations produce a sequence of byte-range edits that are applied to the original source, ensuring that untouched code is reproduced character-for-character while maintaining high performance.

Two-layer architecture¶

Rust layer: `emend_core`¶

The emend_core extension is a PyO3 / maturin Rust crate compiled directly into the emend wheel. It handles all performance-critical AST analysis and manipulation:

File discovery — parallel directory walk via rayon.
Unified Scope Resolver — a multi-language-capable engine that builds scope trees and resolves qualified names. Scoping and binding rules are driven by a language configuration file (e.g., src/emend/languages/python/config.toml).
Structural Matcher — a generic matcher that executes structural queries and captures metavariables directly on the Tree-sitter AST.
Mutation Engine — PyFileTransform manages a set of non-overlapping byte-range replacements, providing the foundation for all code edits.
Symbol collection — extracts definition trees with rich metadata (signatures, return types, visibility) for the search --output summary command.
Reference analysis — resolves all identifiers and attributes to qualified names, supporting refs, rename, and deadcode.

Python layer: CLI and Orchestration¶

The Python layer provides the user-facing CLI and orchestrates complex refactoring workflows:

CLI Entry Point — command definitions and argument parsing using Typer.
Query Parsing — Lark grammars for the selector and pattern languages.
Transform Orchestration — higher-level refactoring logic (e.g., move, copy-to) that combines primitives like symbol extraction and byte-range edits.
Type Oracle integration — adapters for external type inference engines (Pyright, Pyrefly) that provide optional semantic metadata.

Language Configuration¶

emend’s scoping and binding logic is language-agnostic and data-driven. Each supported language is defined by a TOML configuration file that specifies:

Scope Creators — AST node types that create new lexical scopes (functions, classes, comprehensions).
Binding Rules — rules for how names are introduced (assignments, loop variables, parameters).
Import Resolution — strategies for resolving cross-file qualified names.
Visibility Rules — language-specific conventions for public vs. private API detection.

Adding a new language¶

To add support for a new language, create a directory under src/emend/languages/ with two files:

src/emend/languages/<lang_name>/
├── config.toml       # scope resolver configuration
└── symbols.scm       # tree-sitter query for symbol extraction

config.toml¶

The configuration file drives the scope resolver and qualified-name builder. Use src/emend/languages/python/config.toml as a reference. The file contains the following sections:

[language]: Top-level metadata: language name, the tree_sitter_grammar crate to use for parsing, file_extensions (e.g., [".py"]), and keywords that the pattern compiler should treat as reserved.
[scoping]: Declares which AST node types create new scopes (scope_creators) and per- kind rules. Each scope kind has an is_closure_boundary flag that controls whether name lookups propagate outward (e.g., Python class scopes are closure boundaries, so inner functions cannot see class-level names directly).
[bindings]: Rules for how names are introduced into a scope: assignments, loop targets, function/class parameters, with targets, exception handler names, and definition nodes (functions, classes). Each rule maps an AST node type to the child field that holds the bound name.
[imports]: Describes the structure of import statements so the resolver can extract module paths and bound names. resolution selects one of the built-in resolvers (see below).
[qualified_names]: Controls how qualified names are assembled: module_separator ("." for Python), class_member_prefix and nested_function_prefix flags, and the locals_marker string for nested scopes.
[exports]: Public API conventions such as __all__ membership or naming patterns that mark a symbol as an entry point for dead-code analysis.
[builtins]: A list of names that are always in scope and do not need resolution (e.g., print, len, True in Python).

symbols.scm¶

A tree-sitter query file that defines which AST nodes constitute symbols for the search --output summary and deadcode commands. The Python query captures function definitions, class definitions, decorated definitions, and top-level variable assignments. Each captured node should use a @name capture for the symbol’s identifier and a top-level capture (e.g., @function, @class, @variable) for the full node.

Import resolution strategies¶

The Rust backend provides built-in import resolution strategies selected by the resolution field in [imports]:

"python" — follows importlib semantics: sys.path, src/ layout detection, relative imports, and namespace packages.
"node" — implements Node.js module resolution for JavaScript and TypeScript: node_modules lookup, index.js defaults, and package.json main/exports fields.

Adding a new resolution strategy requires changes to the Rust emend_core crate.

Testing¶

After adding a new language configuration, run the full test suite to verify that the scope resolver, symbol collection, and reference analysis all behave correctly:

make test

Pay particular attention to qualified-name construction and import resolution, as errors in those areas propagate to refs, rename, and deadcode.

Caching and indexing¶

emend maintains a cache at .emend/cache/parse.db (SQLite, WAL mode). Syntax caches reuse content-addressed results; search projections also track file paths and module-resolution configuration. A .gitignore and .dockerignore are auto-generated inside .emend/cache/ to prevent the database from being checked in. Each git worktree owns its cache because the manifest, editor projections, and fact graph describe one mutable source snapshot.

Overview of cache tables¶

Table	Key	Contents
`qn_index`	(absolute path, content/module-context hash)	Compressed-pickled `set[str]` of every qualified name in the file. Used by `visit_project()` to skip files that cannot reference a target symbol.
`type_cache`	content MD5 (TEXT)	Compressed-pickled `FileTypes` from the type oracle. Avoids re-running pyrefly / pyright / ty on unchanged files.
`file_manifest`	(worktree_id, absolute path)	`(content_hash, scope_hash)`. Records the source content and module context used for each file’s search projection, scoped to the worktree.
`symbol_index`	(file_path, name, …)	One row per symbol definition (function, class, method). Stores name, qualified name, kind, line range, depth, parent, signature, return type, and bases. Indexed on `name`, `qualified_name`, `file_path`, and `kind` for fast lookups.
`reference_index`	(target_qn, file_path, line, col)	One row per reference to a qualified name. Each row records the reference kind (`read`, `write`, `import`, `call`). Indexed on `target_qn` for fast find-references.
`index_meta`	key name (TEXT)	Key-value pairs: `schema_version`, `git_head:<worktree_id>`, `indexed_at:<worktree_id>`. Per-worktree keys are scoped by the worktree’s resolved root path.

How caches are populated¶

emend tool index calls warm_caches(), which scans the project in parallel using a ThreadPoolExecutor. --jobs controls file-worker concurrency (default: CPU count), not the separate type engine’s worker count. Each worker derives and caches a file’s search projections:

QN resolution — PyScopeResolver → compressed pickle → qn_index.
Symbol collection — emend_core.collect_symbols_from_str() → symbol_index rows (name, kind, line, signature, etc.).
Reference collection — PyScopeResolver reference output → reference_index rows (target QN, line, column, ref_kind).

All analysis is handled by the Rust tree-sitter backend.

Workers also produce content-addressed native facts. A bounded queue feeds these directly to one fact builder while subsequent files are still being processed, without reloading their extraction results. Cold fact graphs are built in memory and bulk-saved to SQLite before atomic publication; incremental updates use disk-backed private generations. Cold builds therefore trade higher temporary memory use for fewer disk writes.

After extraction supplies the import inventory, the configured type engine runs in a separate subprocess, overlapping the remaining fact writes and publication. It stores its results in type_cache. Indexing also updates:

File manifest — writes (worktree_id, path, content_hash, scope_hash) to file_manifest. Each worktree maintains its own set of manifest rows.
Git HEAD — runs git rev-parse HEAD and stores the SHA in index_meta under the key git_head:<worktree_id>.
Editor search — populates the full-text search index.

Workers write per-file search rows through their own SQLite connections. WAL mode allows readers alongside a writer; SQLite still serializes writers. Unchanged extraction and search results are reused. Ordinary indexing does not precompute duplicate analysis; duplicate detection prepares it on demand.

How caches are invalidated¶

AnalysisStore owns source revision detection. _scan_manifest() compares that snapshot’s content hashes and module-resolution context against file_manifest. Git HEAD changes are reported separately; an unchanged HEAD does not imply an unchanged working tree.

When a file changes, its QN marker, symbol rows, and reference rows are replaced in one transaction, keyed by file path. A module-root configuration change also refreshes these projections even if source text is unchanged. Canonical imports and other analysis facts are maintained separately in facts.db.

This check is implemented in _scan_manifest() and exposed through _ensure_index_fresh(), which commands call before querying the index. If fewer than 50 files are stale, they are re-indexed inline; otherwise the caller falls back to the cold path or advises running emend tool index.

How caches are cleaned¶

emend does not aggressively prune old entries. Content-hash keyed tables such as type_cache accumulate entries across branch switches, which is intentional: switching back to an earlier branch reuses those entries.

Path-indexed rows are kept consistent by the re-index cycle described above: when a file’s content changes, its old rows (keyed by file path) are deleted before new rows are inserted. Deleted files are removed from file_manifest and their derived rows are cleaned up during _ensure_index_fresh().

To reclaim disk space or force a full rebuild:

# Delete the entire cache and rebuild from scratch:
rm -rf .emend/cache/
emend tool index

# Or just rebuild (existing entries are overwritten):
emend tool index

The emend tool index --status command reports the number of indexed files, symbols, and references, plus how many files are stale. It does not build analysis facts to collect these statistics.

Warm-path query acceleration¶

Editor symbol search and reference lookup use the SQLite search projections. Semantic symbol, reference, and import queries use the canonical facts through AnalysisStore. The old SQLite import and DSL tables are no longer populated; DSL commands extract their data on demand.

Existing SQLite caches migrate in place. Obsolete columns and tables are dropped while symbol row IDs are preserved for the editor’s full-text index. The shared symbol schema retains module names and decorators for dependency lookup; project search does not populate those fields.

Git worktree support¶

When emend runs inside a git worktree, parse.db and facts.db stay under that worktree’s own .emend/cache/. Content hashes are safe to share, but the databases also contain mutable manifests, full-text projections, and a project-wide fact snapshot; sharing those structures lets one branch overwrite another branch’s analysis.

User-managed mappings are different: they are durable input rather than a recomputable snapshot. For those files, emend follows the worktree’s commondir reference and stores the mapping data in the main checkout’s .emend/ directory.

SQLite WAL mode still protects concurrent readers and writers operating on the same worktree cache.

Selector grammar¶

Selectors are parsed by a Lark grammar (grammars/selector.lark) into ExtendedSelector dataclasses. The grammar handles:

dotted symbol paths with wildcard segments (Class.*, Test*)
component accessors ([params], [returns], [decorators], [bases], [body], [imports])
by-name and by-index sub-accessors ([ctx], [0])
pseudo-classes (:KEYWORD_ONLY, :POSITIONAL_ONLY, :POSITIONAL_OR_KEYWORD)
line-range selectors (file.py:42-100)
file glob expansion (src/**/*.py::*[params])

Pattern grammar¶

Patterns are parsed by a second Lark grammar (grammars/pattern.lark) into Pattern dataclasses containing MetaVar objects. compile_pattern_to_rust_ir() translates a Pattern into a JSON IR consumed by the Rust structural matcher engine, which performs matching directly on the Tree-sitter AST.

Metavariable types:

$X — captures any expression node
$_ — anonymous, matches any expression and discards the capture
$...ARGS — variadic capture (sequence of arguments)
$X:str / $X:int / $X:call / $X:attr — syntactic type-constrained capture
$X:type[T] / $X:returns[T] — inferred type constraint (requires TypeOracle; see below)

TypeOracle layer¶

type_oracle.py provides a pluggable type inference adapter used by find (for :type[X] / :returns[X] pattern constraints and --returns lookup filtering) and the analyze types command.

Abstract interface: `TypeOracle`¶

TypeOracle is an abstract base class with four abstract methods:

infer_file(path, project_root) → FileTypes — return all type bindings for a file
type_at(path, line, col) → TypeBinding | None — return the binding at a position
clear_cache() — evict cached results
is_available() → bool — check if the backing tool is installed

Results are returned as FileTypes (a list of TypeBinding objects with positional and name indexes built by FileTypes.build_index()). TypeBinding records the name, source location, raw_type string from the engine, and a parsed TypeDescriptor tree.

Backends¶

Three backends are provided:

PyreflyAdapter — shells out to pyrefly check --debug-info and parses the JSON binding dump. This is the most comprehensive source of type information (full binding dump, not just diagnostics) but requires pyrefly to be installed. Supports infer_batch() for multi-file queries in one subprocess call.
PyrightAdapter — starts pyright-langserver via the LSP protocol and queries textDocument/hover for each identifier collected by _collect_symbols(). Type strings are extracted from the hover markdown and parsed into TypeDescriptor trees. The LSP process is started once and reused across calls.
TyAdapter — same approach as PyrightAdapter but using ty lsp.

All three cache results keyed on the MD5 of the file’s content (_FileTypeCache, bounded at 256 entries with FIFO eviction) so unchanged files are not re-analyzed.

Type string parsing¶

parse_type_string(raw) converts type strings from any backend into a TypeDescriptor tree, handling named types, parameterized generics (list[int], dict[str, int]), union types (str | None), callable signatures ((x: int) -> str), and Self@ClassName prefixes from Pyrefly. TypeDescriptor.matches(constraint) performs structural matching: an unknown constraint acts as a wildcard, a named constraint matches parameterized types by base name, and union types match if any member satisfies the constraint.

Integration with pattern matching¶

:type[X] and :returns[X] constraint tokens are matched syntactically by m.DoNotCare() (any node passes) and then post-filtered by _filter_matches_by_type_oracle() in transform.py. This keeps the Rust fast-path bypass simple: any oracle constraint is post-filtered after the Rust structural matcher returns positional match data.

Engine autodetection¶

detect_type_engine(project_root) checks for config files in order (pyrightconfig.json → ty.toml → pyrefly.toml → pyproject.toml sections), then falls back to tool availability on PATH (pyrefly → ty → pyright). create_type_oracle(engine="auto") combines detection and instantiation in one call.

Lint engine¶

lint.py loads rules from .emend/rules.yaml (with legacy fallback to .emend/patterns.yaml). Each rule specifies a match pattern, a message, an optional not-within constraint, and an optional fix pattern for --fix mode.

The lint engine applies a two-tier scan:

Batch Rust path — rules whose match pattern compiles to Rust IR are batched into a single find_multi_patterns_in_files call. This handles the common case of simple pattern rules (function calls, attribute accesses) with no structural scope constraint.
Single-file Rust path — rules with complex patterns or not-within constraints are evaluated per-file using find_pattern() with the Rust structural matcher.

# noqa suppression is implemented by tokenizing the source for # noqa comments, then mapping each comment to its enclosing statement range. A suppressed statement suppresses all matches inside it.

Free-threaded Python¶

Python 3.13 introduced an experimental free-threaded build (--disable-gil) and 3.14 continues this as a supported configuration. emend is designed to take full advantage:

emend_core is registered gil_used = false, so all Rust functions release the GIL immediately and can run on multiple OS threads simultaneously.
Cross-file operations (rename, refs) use a ThreadPoolExecutor on free-threaded Python, with the Rust extension performing GIL-free analysis across threads.

To enable free-threaded speedups, install emend with a free-threaded Python:

uv tool install --python 3.14t emend

Fact identities and language boundaries¶

Qualified names remain human-readable (for example, api.common), but are not globally unique: Python and TypeScript can both define that name. FactGraph schema 11 stores symbols by (qualified_name, file_path) and records each file’s effective configured language in file_namespace. JavaScript and TypeScript share a namespace; Rust and Python do not. This metadata survives incremental updates and JSON serialization, including project-configured file extensions.

Built-in call, reference, dead-code and impact queries keep recursive edges within a language namespace. File selectors supply that namespace; bare-name read-only queries aggregate namespaces without connecting them. Destructive cascade planning rejects ambiguous names. Same-language module-resolution ambiguities are not resolved by this language boundary.

Custom CozoScript queries intentionally see all stored facts. Join file_namespace[file_path, namespace] explicitly when relating names across files. decorator_on now has columns symbol_qn, decorator, file_path; func_summary has func_qn, param_name, file_path, flows_to_return, flows_to_sink, sink_label. Public metadata constructors accept a trailing file_path; omitting it requires an unambiguous definition owner. Existing persistent fact caches rebuild automatically after the schema change.

Build system¶

The project uses maturin as its build backend. pyproject.toml declares build-backend = "maturin" and a [tool.maturin] section that points at rust/Cargo.toml and sets module-name = "emend.emend_core". maturin compiles the Rust crate and packages the resulting shared library alongside the Python source into a single platform wheel. Users receive one wheel with no additional binary dependencies.

Version numbers are stored in rust/Cargo.toml and propagated to Python metadata automatically by maturin (dynamic = ["version"] in pyproject.toml).