Unified lexical and token frontend

Purpose and scope

Aims to give the formatter and parser one lexical source. Replacing line/regex structural reading with a token frontend is an internal architecture change, distinct from introducing new language syntax.

Core design rules

• Lossless CST tokens retain text, trivia, comments and UTF-16 locations.
• A bounded structural reader identifies blocks from token ranges; semantic lowering produces the existing NMParseResult contract.
• Shadow execution, independent semantic parity, controlled switching and legacy removal are separate stages.

Example from the original

This example illustrates the design recorded in the original. It is not by itself a claim of executable or stable support; check required options and the current version.

The original does not provide a short source example for this topic. Read its detailed data contracts, evaluation rules and review gates in the original-document view.

Limits and interpretation

• Partial shadow coverage cannot be presented as parity over the complete mandatory corpus.
• Production switching requires performance, OS/Node matrix, review and rollback evidence; silent legacy fallback is not accepted.

Status and implementation boundary

The original records proposed status and staged internal evidence. Local results in its revision history do not establish a changed production default; inspect current frontend selection separately.