Performance measurements and retention guarantees

Historical benchmark artifacts referenced on this page are no longer included in the documentation.

For application-value extraction, see the phase and allocation profile. It separates recognition/CST costs from first and cached value projection.

Phase 4 records evidence rather than promising a universal performance win. The measurements below were collected locally on Windows, PHP 8.2.10 and PCRE 10.40, with three iterations per case in fresh worker processes. JSON artifacts include runtime, input bytes, source kind, output mode, min/median/max time, source-setup time, allocated PHP process peak and reproduction commands.

Reproduce

php tools/benchmark.php --iterations=3 --size=2048
php tools/benchmark.php --source=memory-byte --iterations=3 --size=2048
php tools/benchmark.php --source=file-byte --iterations=3 --size=2048
php tools/benchmark.php --source=file-utf8 --iterations=3 --size=2048
php tools/benchmark.php --source=file-utf8 --iterations=3 --size=65536 --case=regex
php tools/benchmark.php --source=file-utf8 --iterations=3 --size=65536 --case=regex-window
php tools/benchmark.php --source=file-utf8 --iterations=3 --size=1048576 --case=regex-window --mode=recognition
php tools/benchmark.php --iterations=3 --size=2048 --case=shared-prefix --mode=recognition --profile

Add --output=path.json to save a report. --case and --mode select focused measurements. Cases include text, Unicode, RFC URI, consuming parenthesized recursion, shared-prefix alternatives, malformed input, chunked large text and regex. Source setup is timed separately; file setup includes creation of the fixture, disk snapshot copying and UTF-8 validation. The generated input string is retained in the worker, so process peaks include fixture storage too.

Modes are recognition (which always includes CST construction), CST array serialization, named AST projection and a cached semantic mapping to byte length. They measure a complete parse plus that requested output, not isolated output costs. Failed cases do not project a tree/value. The harness verifies acceptance before accepting a measurement. Small cases and three samples are noisy; these results are engineering baselines, not platform-independent guarantees. Byte and UTF-8 source results should only be compared under the same encoding mode.

Bounded UTF-8 validation and counting

Utf8FileSource::validateContent() validates and counts complete-code-point chunks with mb_check_encoding() and mb_strlen(). Each proposed chunk is at most 8192 bytes, with one extra peek byte. Inspecting continuation bits retreats its endpoint by at most three bytes; native validation remains authoritative before counting. Rejected chunks use the original scalar state machine to retain the exact first-invalid-byte or truncated-EOF diagnostic. This rescan is bounded to the rejected chunk rather than the complete file.

Construction still copies the original into a stable private snapshot before validation. It still restores borrowed stream positions on success and failure. Preparation remains O(input bytes), temporary validation storage is bounded by the chunk plus the configured source page, and snapshot disk use remains O(input bytes). The native valid path makes two passes over each chunk, with bounded overlap/copying at its endpoint. It trades a small fixed memory increase for removing per-byte PHP dispatch. In-memory Utf8Source preparation and all recognition, CST, diagnostics, semantics, effects and execution limits are unchanged.

Reproduce with fixtures generated directly into a temporary stream in bounded blocks; fixture generation and correctness checks are outside the setup timer:

php -d pcov.enabled=0 tools/benchmark-utf8-validation.php --case=ascii --size=9437184 --iterations=3
php -d pcov.enabled=0 tools/benchmark-utf8-validation.php --case=unicode --size=9437184 --iterations=3
php -d pcov.enabled=0 tools/benchmark-utf8-validation.php --case=malformed-start --size=9437184 --iterations=3
php -d pcov.enabled=0 tools/benchmark-utf8-validation.php --case=malformed-end --size=9437184 --iterations=3

--source=file-byte selects the unchanged snapshot-copy control. --size=0 selects empty input, and --page=4 exercises minimum-size pages. The timer covers complete source construction, including snapshot copying and validation/counting; it excludes release. Expected counts, exact malformed-input errors and borrowed stream restoration are checked on every observation. No full fixture string or code-point index is retained by the harness.

For baseline comparisons, save the previous Utf8FileSource.php as Utf8FileSource.before.php in a directory and add --baseline=that-directory. The saved class, alternating-worker reports and a generated differential driver are locally retained under ignored build/utf8-validation/; while retained, use --baseline=build/utf8-validation. The differential driver compares 3,000 deterministic inputs across both variants, including random malformed octets, Unicode suffixes and chunk/page edges. Acceptance, counts and error messages matched exactly in this run.

Windows/PHP 8.2.10/PCRE 10.40, native mbstring loaded, PCOV disabled, three fresh workers per variant with alternating before/after order, one warmup and three observations per worker. Medians of worker medians, milliseconds:

Complete source setup Before After
1 MiB ASCII 66.451 7.461
Approximately 1 MiB mixed Unicode 98.209 5.689
9 MiB ASCII 604.319 68.791
Approximately 9 MiB mixed Unicode 873.596 53.102
9 MiB plus an invalid surrogate sequence at the start 16.189 16.230
9 MiB plus a truncated four-byte sequence at EOF 596.496 69.245
8,190 Unicode bytes, 4-byte pages 14.710 13.658
9 MiB byte-source copy control 14.779 13.661

Unicode repeats ASCII and two-, three- and four-byte code points, rounded down to complete repetitions. Early errors still pay snapshot-copy cost, and tiny pages remain I/O-heavy. A one-byte fixture measured 0.587/0.579 ms, with noisy worker medians; no small-file benefit is established. Host load was uncontrolled, and native/platform implementations can change these results. These measurements do not establish a recognition-throughput improvement.

Incremental used-PHP setup peaks for valid default-page fixtures increased from 34,648 bytes to 46,936 bytes for ASCII and 59,224 bytes for Unicode. Those values were the same at approximately 1 MiB and 9 MiB; they are not RSS measurements. There is no new retained source-size cache or full-input materialization.

Utf8FileSourceTest covers complete counts and conversions, all UTF-8 widths, minimum/default page sizes, chunk-edge alignments, exact invalid-byte diagnostics, overlongs, surrogates, out-of-range values, truncation, stream restoration and bounded memory on growing files. Reject the optimization if any of those contracts diverges or temporary validation memory grows with file size.

Checked file-byte access and cached subject readability

Utf8FileSource::readUnit() now checks and decodes a single lead byte through ByteFileSource::readByteValue(). Boundary queries use the same checked page access. This removes duplicate one-byte slicing, repeated offset/readability checks and temporary strings without changing byte spans, range errors or closed-source behavior. The page loader still publishes only complete reads and retains one bounded page.

ParseSession::regexSubject() checks exact built-in file readability using an empty slice. It no longer loads page zero merely to test whether a cached subject can be read. Custom sources, including file subclasses, retain the original boundary probe; immutable built-in memory sources still skip it. The cache cap, full-subject regex context, logical work and session ownership are unchanged.

Reproduce current behavior with the maintained harness:

php -d pcov.enabled=0 tools/benchmark-file-access.php --case=units --iterations=3
php -d pcov.enabled=0 tools/benchmark-file-access.php --case=units --unicode --page=4 --iterations=3
php -d pcov.enabled=0 tools/benchmark-file-access.php --case=subject --size=1048576 --iterations=3
php -d pcov.enabled=0 tools/benchmark-file-access.php --case=regex --size=1048576 --iterations=3

--source=file-byte selects the byte-mode control. For fresh-process before/after comparisons, save the baseline ByteFileSource.php, Utf8FileSource.php and ParseSession.php as ByteFileSource.before.php, Utf8FileSource.before.php and ParseSession.before.php in one directory, then add --baseline=that-directory. Saved baseline classes and raw observations from this change are locally retained under ignored build/file-byte-access/; while retained, use --baseline=build/file-byte-access. Alternate variant order across workers.

Windows/PHP 8.2.10/PCRE 10.40, PCOV disabled, three fresh workers per variant, one warmup and three timed observations per worker. Medians of worker medians, milliseconds:

Fixture Before After
64 KiB ASCII, UTF-8 file units + raw text 140.032 75.001
65,530 bytes mixed Unicode, UTF-8 file units + raw text 60.451 31.269
Same Unicode fixture, 4-byte pages 118.457 86.985
64 KiB ASCII, byte-file control units + raw text 62.913 62.613
1 MiB UTF-8 file, 1,000 cached subject / distant-boundary cycles 7.933 0.762
1 MiB UTF-8 file, 1,000 regex recognitions near EOF 14.312 6.321
9 MiB UTF-8 file, 256 cached subject / distant-boundary cycles 1.931 0.199

Unicode repeats ASCII and two-, three- and four-byte code points. Unit timings include raw-span reads and checksums, not parser/CST construction. Subject cycles include a distant boundary query before requesting the cached subject. Regex timings include checked terminal recognition/CST construction at a nonzero offset in one session, with success checks and result release. Logical work deltas must remain identical across observations. Setup and first subject materialization are reported separately, outside warm timings. The harness sets the subject cap to the fixture size, including 9 MiB; this does not measure the default uncached path above 8 MiB. Fixture storage and the full warmed subject are retained by the harness and excluded from incremental access peaks.

Used-PHP unit-access peaks stayed at 12,384 bytes for ASCII and 12,416 bytes for Unicode on default pages, before and after. Warm cached-subject cycles stopped allocating replacement pages (12,288 to zero incremental bytes in this probe). These are not RSS measurements or reductions in the retained full-subject cache. Preparation complexity, one-page storage bounds and snapshot disk use are unchanged. Host load was uncontrolled; the unchanged byte control illustrates timing variation. No general application-throughput or setup improvement is claimed.

Regressions compare all byte boundaries near page edges with memory sources, including UTF-8 width extremes, interior offsets, EOF, extreme invalid offsets, closure precedence, arbitrary octets, cached-page retention and custom-source hooks. Reject these optimizations if byte spans, exceptions, source lifetime, diagnostics, work limits or semantic/effect behavior differs.

Chunked file scanning

ReadRun accelerates exact built-in file sources for case-sensitive literal ReadUntil, ReadWhileSet and ReadWhileRange scans without finite work budgets. It uses 8192-byte chunks, complete UTF-8 endpoints and bounded delimiter overlap. Generated atoms retain the existing 8192-byte cap; temporary subjects are bounded by the chunk plus delimiter lookahead and three UTF-8 boundary bytes. No complete subject cache, source-size index or alternative recognition runtime is introduced. Finite work budgets and unsupported cases retain the scalar loop. A failed PCRE attempt resumes that loop at the first uncharged chunk.

Reproduce with the maintained harness, running each command in a fresh process:

php -d pcov.enabled=0 tools/benchmark-file-scanning.php --mode=scalar --iterations=3
php -d pcov.enabled=0 tools/benchmark-file-scanning.php --mode=chunked --iterations=3
php -d pcov.enabled=0 tools/benchmark-file-scanning.php --mode=chunked --unicode --iterations=3
php -d pcov.enabled=0 tools/benchmark-file-scanning.php --source=file-byte --size=8192 --page=4 --delimiter-bytes=128 --mode=chunked --iterations=3

Use both modes for every fixture. --case=until|set|range, --source=file-byte|file-utf8, --size, --page and --delimiter-bytes select additional cases. Scalar mode forces the authoritative unit loop with a generous finite budget; it is not a historical checkout. Each process checks acceptance, byte length, CST arrays and logical work against an in-memory scalar reference. Timers include a fresh session and recognition/CST construction; verification, source setup, serialization and result release are outside them. One warmup precedes measured observations. Setup includes snapshot copying and UTF-8 validation, excluding fixture creation. The harness retains its input and reference tree, so its total process memory does not model bounded file ingestion.

Local Windows/PHP 8.2.10/PCRE 10.40 measurements, PCOV disabled, one worker and three observations per fixture/mode; medians in milliseconds:

Fixture, absent delimiter Scalar Chunked Chunked incremental PHP peak
256 KiB ASCII, byte file 394.191 0.626 51,248 bytes
256 KiB ASCII, UTF-8 file 705.663 2.750 51,248 bytes
262,140 bytes mixed Unicode, byte file 393.277 0.762 51,248 bytes
262,140 bytes mixed Unicode, UTF-8 file 297.661 2.153 63,504 bytes
8 KiB ASCII, byte file, 4-byte pages, 128-byte delimiter 805.213 11.708 34,864 bytes

Mixed Unicode repeats ASCII plus two-, three- and four-byte code points; byte mode inspects octets, while UTF-8 mode inspects code points. Peaks are incremental used PHP memory during recognition, not RSS. Scalar peaks were about 14 KiB on default pages. Raw local observations are retained under ignored build/source-research/chunk-comparison.json. Host load was uncontrolled and mode order was not alternated; these are engineering observations, not general speed guarantees. Repeat in alternating fresh workers before making application claims.

Chunking removes PHP per-unit calls and repeated source-page lookahead. It adds bounded string copies and PCRE preparation; short scans may not benefit. Tempered literal matching can still perform work proportional to input times delimiter length; this is not a linear-time search algorithm. Patterns that exceed PCRE limits retain scalar performance. Source construction and validation are unchanged.

FileScanningTest compares Unicode/page/chunk crossings, overlapping and partial delimiters, min/max failures, CSTs, diagnostics and work with scalar recognition. It also checks finite-budget offsets, temporary-memory bounds, source closure, custom-source and callback fallbacks, separate file/memory plans, cached values and explicit effects. Reject acceleration if those contracts diverge or temporary subject memory grows with file size.

Recorded comparisons

Direct facade snapshots (2026-10-23)

FrozenGrammar::from($facade, $flags) now copies the facade's owned parser directly into the resolving destination snapshot. Previously it first created a detached authoring export and then copied that export again. Comprehend::parser() keeps its normal detached, non-resolving behavior; an optional internal destination context lets the freeze factory avoid the intermediate graph. The factory still validates flags, resolves the optimizer when requested, isolates the graph, checks safety and prepares recognition. It creates fresh snapshots without consulting facade caches or dispatching a custom root freeze() override, as before.

This removes one O(nodes + edges) authoring-copy pass over the exported graph, including its temporary graph storage and snapshot indexes. It does not eliminate the final detached snapshot or cache mutable definitions. Factory-returned facades still export separately: sharing their internal roots across named productions could merge diagnostic metadata or parser identities. Ordinary facade freeze() already avoided this intermediate export and is unchanged.

Three fresh Windows/PHP 8.2.10 workers per variant alternated before/after order, with PCOV disabled, one warmup and 31 observations per fixture/flag pair. Timers cover complete FrozenGrammar::from($facade, $flags) preparation, excluding facade construction, source setup, correctness checks, recognition and release. Medians of worker medians, in microseconds; memory is incremental used PHP bytes from a separate warmed preparation, not RSS:

Fixture / flags Preparation before / after Peak bytes before / after
Text / NONE 5.2 / 3.7 4,584 / 4,136
Text / OPTIMIZE 6.8 / 5.5 4,584 / 4,136
512 flat Text operands / NONE 1,323.9 / 948.1 834,136 / 592,032
512 flat Text operands / OPTIMIZE 5,036.0 / 4,525.1 1,033,432 / 791,288
128 nested Sequences / NONE 558.2 / 397.5 306,080 / 228,832
128 nested Sequences / OPTIMIZE 6,483.1 / 6,158.2 489,664 / 407,296
128 references / NONE 722.1 / 645.8 203,840 / 151,688
128 references / OPTIMIZE 2,599.4 / 2,454.1 391,624 / 339,432
RFC URI / NONE 1,015.0 / 989.7 293,368 / 292,984
RFC URI / OPTIMIZE 2,315.2 / 2,237.9 293,424 / 293,040

The reference fixture uses one Text production 128 times. The URI facade's unresolved export is small, so the removed pass saves little compared with resolution and preparation. The flat graph measured about 28% faster preparation without optimization and 10% with it, with about 29% and 23% lower peaks. Retained graph memory was effectively unchanged in most fixtures; a nested optimized fixture retained 328,264 bytes before and 323,144 after. No general retained-memory, cold-process or recognition-throughput improvement is claimed. Host activity was uncontrolled, and class loading was warmed before observations.

Baseline source, worker and raw reports are under ignored build/direct-snapshot/. While retained, reproduce in fresh alternating workers:

php -d pcov.enabled=0 -d memory_limit=512M build/direct-snapshot/measure.php before
php -d pcov.enabled=0 -d memory_limit=512M build/direct-snapshot/measure.php after

Workers compare valid, truncated and trailing-input acceptance, nonzero-offset CSTs, diagnostics and values against ordinary exported-parser freezing. New FrozenGrammarTest cases verify one resolving copy per shared participant, public export isolation, custom freeze dispatch, fresh factory resolution, old snapshots after redefinition, failed-copy retries, byte spans, cached values and explicit effects under both flag sets. Reject the optimization if ownership, freshness, custom dispatch, diagnostics or execution lifecycle differs.

Compact analysis ancestry (2026-10-23)

GraphAnalysis stores one first-discovery parent ID and the original collection label per node instead of a full path array for each node. It reconstructs paths only for warnings and errors. Leading-cycle detection also uses one active map, restored in finally, instead of copying the ancestry map at each recursive call. Warning order, exact error text, first-discovery paths through shared graphs and custom analysis call order remain unchanged. Nullability scheduling is unchanged.

Three fresh Windows/PHP 8.2.10 workers per variant used alternating before/after order, PCOV disabled, one warmup and 21 timed validations per fixture. Graph construction, resolution, correctness checks and object release are outside the analysis timer. Medians of worker medians, in microseconds:

Graph Analysis before / after Retained analysis bytes before / after Incremental peak bytes before / after
64 consuming wrappers 140.0 / 116.3 86,080 / 33,104 224,488 / 60,464
256 consuming wrappers 771.2 / 468.9 1,355,840 / 82,256 3,665,640 / 191,376
1,024 consuming wrappers 15,285.1 / 1,884.9 15,239,232 / 328,016 44,324,584 / 934,424
512 nullable wrappers 3,061.7 / 1,455.2 4,585,536 / 164,176 12,236,520 / 405,784
128 branches sharing 256 wrappers 1,176.1 / 782.1 1,407,960 / 82,256 3,747,136 / 259,144
512 flat Text operands 458.6 / 445.1 233,952 / 164,176 295,936 / 226,192
Resolved RFC URI 521.7 / 480.5 153,368 / 49,488 228,480 / 124,600

Consuming chains end in one Text; nullable chains end in Nothing. Memory uses a separate warmed validation and keeps the analysis object alive. These are PHP used bytes above the constructed-graph baseline, not RSS or frozen-grammar retention. The 1,024-wrapper case measured about 8.1x faster analysis with about 98% less retained analysis memory and incremental peak memory. Host activity was uncontrolled. These measurements do not establish a complete-freeze or recognition speedup; optimizer preparation and CST construction are separate costs.

Ancestry storage is O(nodes), with an O(depth) active map. Reconstructing a diagnostic costs O(path length). Many deep opaque warnings still require output proportional to their full paths. Recursive PHP call stacks, nullability dependency tables and optimizer plans retain their separate costs and bounds.

The baseline class, worker and raw reports are under ignored build/analysis-ancestry/. While retained, reproduce in fresh alternating workers:

php -d pcov.enabled=0 -d memory_limit=512M build/analysis-ancestry/measure.php before
php -d pcov.enabled=0 -d memory_limit=512M build/analysis-ancestry/measure.php after

The focused GraphAnalysis and AnalysisContract tests pass against both the saved baseline and current class. New checks pin exact shared-graph warning paths and order, captured labels, complete structural/progress error ancestry, cycle paths after completed siblings, and retries after failed analysis. Reject the change if diagnostic ordering/text, custom analysis behavior or graph safety checks diverge, or representative overhead outweighs the deep-graph memory improvement.

Shared optimizer preparation (2026-10-23)

Optimizer preparation now shares successful literal and lexical-lookahead classifications within one resolved, safety-checked graph. Descriptors include structural height, so reuse under a deeper caller still obeys the original classification depth limits. Failed classifications are not cached. Terminal classification remains direct; bounded lexical tables remain independently owned by their combinators.

Literal compilation also reuses source-free, operand-relative syntax templates when the same parser occurs on multiple graph edges. Recognition still constructs fresh CST nodes and execution records for every occurrence. A temporary weak-keyed context owns the descriptors and template index; it is released after preparation, including when preparation throws. There is no cross-freeze definition cache or shared recognition session, semantic projection or effect execution.

Three fresh Windows/PHP 8.2.10 workers per variant alternated before/after order, with PCOV disabled, one warmup and 31 observations per fixture. Preparation includes snapshot copying, resolution, safety analysis and optimization; source construction, correctness comparisons and release are outside preparation timers. Recognition includes the CST. Medians of worker medians, in microseconds:

Fixture Preparation before / after Reused recognition before / after
One Text 4.9 / 5.4 5.7 / 5.5
128 distinct flat Text operands 1066.9 / 1120.3 193.4 / 196.2
128 nested Sequences 6670.4 / 6396.6 475.5 / 477.3
128 occurrences of one literal subgraph 5378.7 / 329.6 1359.2 / 1349.1
RFC URI 2671.3 / 2308.2 852.8 / 840.7

The shared subgraph is Sequence(Text('ab'), Repeat::exact(Text('c'), 8)). Its retained prepared graph fell from 343,352 to 20,264 used PHP bytes, with incremental preparation peaks falling from 354,376 to 39,752 bytes. The flat control retained the same 97,896 bytes but its peak rose from 183,240 to 206,616 bytes. Other retained graph measurements were unchanged. These are PHP used bytes, not RSS. The flat control also shows approximately five percent preparation overhead; this is a tradeoff for overlapping graphs, not a universal cold-parse win. Host activity was uncontrolled, and no general recognition speedup is established.

Successful descriptor classification can reuse a shared subtree rather than traverse it for each parent. Graph occurrence counting and temporary maps add O(nodes + edges) traversal work and O(nodes) indexing space. Descriptor strings and expanded syntax remain subject to their existing byte/node/depth bounds. Template sharing reduces duplicate retained recipes, but each input occurrence still requires its full CST. Deep unshared templates can still dominate preparation.

The local baseline sources, worker and raw reports are retained under ignored build/preparation-reuse/. While retained, run these in fresh alternating workers:

php -d pcov.enabled=0 -d memory_limit=512M build/preparation-reuse/measure.php before
php -d pcov.enabled=0 -d memory_limit=512M build/preparation-reuse/measure.php after

Workers compare valid, truncated and trailing-input acceptance, byte-offset CSTs, diagnostics and values with ordinary freezing. PreparationTest adds depth-bound, expanded-size, cycle and source/session-release checks. PreparationReuseTest compares cold/warm trees, named ASTs, diagnostics and every tested exhausted work/depth budget on byte and UTF-8 sources at a nonzero offset. Existing lifecycle tests cover definition refresh, pure recognition, cached values and explicit effects. Reject this reuse if any of those contracts diverge, sources or execution state are retained, or representative preparation overhead outweighs the overlapping-graph gains.

Choice dispatch and prefix-plan comparison (2026-10-23)

Optimizer\ChoiceDispatch provides conservative first-byte rejection for exact built-in ASCII Text alternatives outside existing bounded lexical tables. Plans require at least two distinct eligible first bytes and leave the first alternative on its ordinary path. Unknown, nullable and executable operands are never inferred through. Rejected operands still merge their original diagnostics in alternative order and charge their original entry/mismatch work. Matching candidates retain checked recognition. Finite budgets, active spacing, case-insensitive scopes and custom choice subclasses retain their prior paths. This is not general first-set analysis or a way to remove the ordered alternative loop.

The maintained tools/compare-choice-plans.php compares the previous optimized Choice, production dispatch, and two research-only plans: a byte trie and a compressed prefix tree (internal prefix factoring). Both prototypes classify flat literal alternatives, replay failure observations in original order through the ordinary Choice selector, and recognize successful operands normally. They retain the original CST; rewriting P a | P b as a new authored P (a | b) would change tree boundaries and can change guards, scopes and ranked selection. That grammar rewrite is not an equivalent benchmark or a production optimization here.

Fresh PHP 8.2.10/Windows workers used PCOV disabled, three warmups and 31 timed recognition/CST observations. Three workers per variant alternated forward/reverse variant order. Timings exclude source setup, correctness checks, result release and preparation. The test suite ran afterward. Host activity was uncontrolled. The fixture has 128 alternatives, a 512-byte prefix plus three decimal suffix bytes; distinct prepends one of 90 initial bytes. These are medians of worker medians in milliseconds, not application throughput guarantees:

FIRST-policy fixture Previous lexical Dispatch Byte trie Factored prefix
Diverse initial bytes, last alternative 0.641 0.325 0.459 0.270
Shared prefix, last alternative 0.836 0.845 0.540 0.387
Shared prefix, late malformed input 0.863 0.842 0.531 0.355
First alternative succeeds 0.0110 0.0109 0.1990 0.0314
Eight-byte prefix, warm lexical tables 0.488 0.473 0.285 0.282
Same short literals, saturated tables 0.893 0.904 0.282 0.284

Diverse-input worker medians ranged 0.624–0.646 ms before and 0.322–0.329 ms with dispatch. Checked parser entries fell from 129 to 4; work remained 1,285 units. Shared-prefix entries/work remained 129/65,829. Its small apparent dispatch slowdown is an overhead to watch, not evidence that first-byte filtering helps shared prefixes. The LONGEST shared-prefix control measured 0.841/0.862/0.560/0.385 ms respectively. Short warmed tables already reduce checked entries to one; prototypes instead charge ordinary Text work, so their unlimited counters differ. The saturated fixture fills each table with 160 distinct failing lookaheads before the measured input. Production dispatch does not replace these tables or their capacity/saturation behavior.

For diverse literals, preparation retained 64,568 used bytes before and 160,656 with dispatch, including static expectations. A dispatch-disabled Choice gains one empty array property; the shared fixture retained the same measured 64,568 bytes in this run. The byte trie retained 28,657,784 bytes across 46,545 nodes; factoring retained 184,280 bytes across 167 nodes. Shared-prefix plans retained 455,592/141,432 bytes across 656/145 nodes respectively. These are PHP used bytes, not RSS or allocator peaks. The report also records preparation and parse peaks, first-parse time, and retained result memory. All four diverse-input variants retained the same 2,592-byte selected result in this probe. Preparation timings for production include freezing, while prototypes omit snapshot/safety analysis; their absolute preparation times must not be treated as equivalent cold parsing. Diverse-fixture preparation medians were 0.618 ms before and 0.805 ms with dispatch, versus 38.907 ms for trie construction and 0.342 ms for factored-plan construction. First-parse results and reuse counts are needed to assess amortization for an application; warm recognition alone does not establish a cold-parse gain.

Dispatch adds preparation proportional to inspected literal bytes and retains source-free failure recipes. Recognition still visits O(alternatives) entries to preserve ordered diagnostics; it replaces full-literal attempts with one-byte probes where rejection is proven. Prototypes classify the matched prefix plus candidate outcomes, but still replay O(alternatives) failure observations. Trie storage grows with distinct byte prefixes; factoring compresses unary paths into string comparisons. Their eager whole-choice classification explains the severe early-success regressions. Neither prototype is enabled in production. Factoring is the better further experiment for long late-failing chains. The lazy research variant below investigates classification cost; broader eligibility/lifecycle proofs remain necessary before promotion.

Each worker performs 216 cold/warm differential checks against ordinary freezing: all policies, byte/UTF-8 sources, nonzero offsets, EOF, malformed input, prefix overlap and ties, prefix/all-input entry points, diagnostics, CST/AST and values. The new unit/feature regressions additionally check guards, cached collections, explicit effects, string keys, file pages, inherited scopes, authoring copies, partial failure progress and exact exhausted budgets. No memoization is added; none of this demonstrates context-equivalent repeated recognition for a cache.

Reproduce the current comparison (the lexical control disables dispatch through reflection after preparing the current graph):

php -d pcov.enabled=0 tools/compare-choice-plans.php --case=distinct --workers=3
php -d pcov.enabled=0 tools/compare-choice-plans.php --case=shared --workers=3
php -d pcov.enabled=0 tools/compare-choice-plans.php --case=first --workers=3
php -d pcov.enabled=0 tools/compare-choice-plans.php --case=shared --policy=longest --workers=3

Other cases are short, saturated, malformed, prefixes, second and empty. To reproduce the original implementation control used above, save b30235b:extensions/Optimizer/Parser/Choice.php as build/choice-dispatch/original-choice.php and add --baseline-choice=build/choice-dispatch/original-choice.php; the control worker loads that class before discovery. Raw observations are retained under ignored build/choice-dispatch/ while available. Reject dispatch if diagnostic ordering, selection, work or lifecycle comparisons diverge, or representative regressions outweigh the targeted benefit. Reject either eager prefix plan as a general replacement based on the measured early-success and preparation-memory costs.

Lazy compressed-prefix research plan (2026-10-23)

tools/choice-plans/ChoicePlanProbe.php now also supports lazy-factored in the comparison harness. It prepares the same compressed tree eagerly, with parent/byte-offset and literal-to-terminal indexes. Recognition attempts the first alternative normally. A later alternative examines only its ancestry, caching each examined edge's mismatch offset or complete match within that parse. It never classifies unrelated sibling subtrees. The subject window is copied on the first planned attempt, bounded by the longest literal and remaining input. Per-parse comparisons and the window are restored in finally, including on exceptions; neither is retained between parses. This is comparison reuse within one choice, not parser-position memoization.

The ordinary Choice loop still selects alternatives and merges failure observations in their original order. Successful alternatives use their original checked Text recognizers and construct fresh CSTs. Finite work/depth limits, active spacing and case-insensitive scopes use ordinary recognition. The research harness supplies flat ASCII Text alternatives; this does not establish support for arbitrary grammar graphs or subsequent mutation of those alternatives. This research variant itself does not change production parsing; the restricted production implementation is described below.

Fresh Windows/PHP 8.2.10/PCRE 10.40 workers, PCOV disabled, three workers per variant with forward/reverse variant order, three warmups and 31 observations. No test-suite run overlapped these timings. Host load was uncontrolled. Each fixture has 128 alternatives; long prefixes contain 512 bytes. Medians of worker medians for recognition including CST construction, milliseconds:

Fixture / policy Production dispatch Eager factored Lazy factored
Shared prefix, last success / FIRST 0.8600 0.3802 0.4262
Shared prefix, late mismatch / FIRST 0.8313 0.3701 0.4063
First alternative succeeds / FIRST 0.0112 0.0301 0.0116
Second alternative succeeds / FIRST 0.0239 0.0326 0.0335
Diverse initial bytes, last success / FIRST 0.3298 0.2791 0.3807
Short literals, warm lexical tables / FIRST 0.5057 0.2936 0.3311
Short literals, saturated tables / FIRST 0.8671 0.2880 0.3303
Overlapping prefix, first success / FIRST 0.0127 0.0273 0.0139
Empty input / FIRST 0.4172 0.3652 0.4266
Shared prefix, last success / LONGEST 0.8602 0.3803 0.4301
Shared prefix, last success / SHORTEST 0.8430 0.3903 0.4420

The lazy plan retains the long-prefix benefit and avoids most of the eager first-success penalty. It is slower than eager factoring on these late-success fixtures, and slower than production for second success and diverse initial bytes. These results reject it as a general production replacement. They do not demonstrate an application-throughput improvement or justify automatic selection.

For the shared-prefix fixture, production/eager/lazy preparation retained 76,536/141,624/183,808 used PHP bytes. Eager/lazy construction medians were 0.4137/0.4535 ms; production complete freezing was 0.7529 ms. Prototype construction omits snapshot and safety analysis, so these are not comparable cold-parse costs. Recognition incremental peaks were 172,480/145,680/145,928 used PHP bytes; retained results were identical across variants. These are not RSS measurements. The extra ancestry indexes cost O(plan nodes + alternatives) storage; per-parse edge results cost O(visited nodes), and the subject window costs O(longest literal). The ordered alternative loop and diagnostic replay remain; matching candidates also pay ordinary Text recognition. Preparing the plan is still eager, and there is no general linear-time guarantee.

Reproduce each table row with the corresponding case/policy:

php -d pcov.enabled=0 tools/compare-choice-plans.php --case=shared --workers=3 --iterations=31
php -d pcov.enabled=0 tools/compare-choice-plans.php --case=first --workers=3 --iterations=31
php -d pcov.enabled=0 tools/compare-choice-plans.php --case=second --workers=3 --iterations=31
php -d pcov.enabled=0 tools/compare-choice-plans.php --case=shared --policy=longest --workers=3 --iterations=31

Raw reports are locally retained under ignored build/lazy-choice-CASE.json; ranked shared-prefix reports use longest and shortest filenames. Each worker runs 216 differential checks against ordinary freezing before timing. The focused tests/Feature/authoring/LazyChoicePlanTest.php additionally compares overlapping literals, ties, empty/multibyte input, byte offsets, inherited diagnostics and work; it checks scoped fallbacks, exact exhausted budgets, file-window avoidance on first success, exception restoration, source release, fresh cached values and explicit idempotent effects. These research measurements alone do not establish bounded preparation, safe graph-copy behavior or comparable complete-freeze costs.

Restricted production prefix plans (2026-10-23)

Optimizer\ChoicePrefix::prepare() builds a source-free compressed plan only for 4–256 exact built-in ASCII Text alternatives, each longer than the 32-byte lexical table limit and sharing at least 32 prefix bytes. Executable operand metadata, case-insensitive literals and custom parser subclasses exclude the entire plan. Preparation caps each literal at 65,536 bytes, total literal bytes at 262,144, compressed depth at 32 levels and nodes at 4,096. A rejected graph keeps its existing lexical/dispatch paths. These are conservative eligibility bounds rather than claims of optimal thresholds for every workload.

The optimized Choice leaves its first two alternatives on ordinary recognition, avoiding the research variant's second-success classification penalty. Later attempts examine only needed ancestor edges, caching outcomes until that choice returns. Each edge reads its own bounded source interval rather than copying a longest-literal subject. Successful candidates still use checked recognition and fresh CST construction. Negative recipes preserve partial progress, alternative order, original expectations/rule paths and ordinary Text work charges. No semantic projection or effect execution is shared. Finite work/depth budgets, active spacing, case-insensitive scopes, custom sources and custom optimized Choice subclasses retain existing recognition. Only exact built-in memory/file byte/UTF-8 sources use the plan. finally restores recognition-local comparisons; graph copies clear the prepared plan and prepare their own replacement.

Three fresh workers per variant, alternating forward/reverse order, three warmups and 31 observations, Windows/PHP 8.2.10/PCRE 10.40, PCOV disabled. No test suite overlapped timing; host load was uncontrolled. Recognition/CST medians of worker medians in milliseconds:

Fixture / policy Prefix disabled Production prefix
Shared prefix, last success / FIRST 0.8540 0.4361
Shared prefix, late mismatch / FIRST 0.8395 0.4329
Shared prefix, first success / FIRST 0.0112 0.0119
Shared prefix, second success / FIRST 0.0242 0.0263
Diverse initial bytes, last success / FIRST 0.3377 0.3301
Short literals, warm tables / FIRST 0.4849 0.4939
Short literals, saturated tables / FIRST 0.8557 0.8878
Shared prefix, last success / LONGEST 0.8639 0.4668

The shared fixture has 128 alternatives with a 512-byte prefix and a three-byte suffix. Checked entries fell from 129 to 4 with work unchanged at 65,829 units. Diverse and short fixtures are ineligible; their entry/work counters remain unchanged. Early shared successes still pay scope/plan checks even though they do not compare planned edges. The small control timing differences are overhead and noise to monitor, not proof of a general throughput improvement.

Shared-fixture retained prepared memory increased from 76,600 to 231,488 used PHP bytes. Incremental recognition peaks fell from 172,480 to 83,104 bytes. Retained result deltas fell from 6,432 to 1,952 bytes because diagnostic strings can reference the prepared recipes; their storage has moved into the larger prepared graph. These are PHP used-memory deltas, not RSS or a general total-memory reduction. Ordered selection/diagnostic replay remains O(alternatives); edge comparisons and their temporary outcomes scale with the visited compressed paths. The complete tree still requires storage proportional to selected CST nodes.

The dispatch control discards the prefix plan after freezing, so its reported preparation time includes prefix construction. A separate saved pre-change Choice control measured complete freeze at 0.7686 ms versus 1.2681 ms for production prefix, with peaks of 199,256/296,584 used bytes. First parses measured 0.8763/0.4746 ms. Those separate control workers did not alternate with production workers; treat these preparation figures as exploratory. Reuse frequency and early-success rates determine whether preparation amortizes. No cold-parse speedup is established.

tools/compare-choice-plans.php now includes prefix (production), dispatch (prefix disabled) and lexical (both prefix and dispatch disabled). Explicit --variant selects one variant; otherwise each worker alternates the full set. Reproduce with --case=shared, shared-first, second, distinct, short, saturated or malformed, and --workers=3 --iterations=31. Raw reports are locally retained under ignored build/production-prefix-CASE.json. Use --variant=lexical --baseline-choice=path/to/saved-original-choice.php for a pre-change Choice control. The saved control in this run is 67d1cc4:extensions/Optimizer/Parser/Choice.php, retained locally as build/choice-prefix-before.php.

Focused ChoicePrefix unit and feature tests cover eligibility bounds, exact negative work/diagnostics, string keys, ties, overlapping literals, immutable trees, byte and UTF-8 memory/file sources, scoped/custom-source fallbacks, exact exhausted budgets, source errors, copy isolation, cached values and explicit idempotent effects. Reject or narrow the plan if those contracts diverge, bounded preparation is exceeded, or representative preparation/memory and early-success costs outweigh late shared-prefix gains.

Flat compiled-literal failures (2026-10-23)

When a compiled literal run contains only exact built-in ASCII Text operands, its mismatch progress identifies the failing operand. Recognition now skips successful preceding operands and invokes the original failing Text for its expectation, rule path and partial progress. Their parser-entry and literal-unit work charges remain, including the existing speculative compiled comparison. Finite work/depth limits, active spacing and case-insensitive scopes still use ordinary recognition. Nested runs retain their existing failure fallback.

The 128-alternative research fixture shares a 2,048-byte Text prefix, followed by a distinct three-byte Text suffix, and accepts the final alternative. Warmed optimized parser entries fell from 511 to 384, while work remained 523,076 units. The selected CST matched the ordinary graph. Three fresh workers per variant used alternating before/after order, three warmups and 31 observations, with PCOV disabled on Windows/PHP 8.2.10. These initial timing workers overlapped test-suite execution. A follow-up after the suite completed used the same protocol: long-prefix worker medians ranged 10.657–10.875 ms before and 7.501–9.443 ms after; medians of worker medians were 10.814 and 7.635 ms. Host activity remained uncontrolled; these synthetic observations do not establish application throughput or a universal speedup. Short-prefix lexical-table and distinct-literal controls retained one checked parser entry in both versions.

Preparation retains an additional list of Text references for each eligible run; other runs retain null. Eligibility is determined during existing classification. There is no source/result cache. Failure recovery walks operand boundaries and still scans the failing operand, but avoids rescanning and building discarded successes for preceding operands. Successful recognition and retained CST complexity are unchanged. No retained-memory improvement is established.

The ignored research driver and reports are under build/choice-research/. While retained, reproduce the current probe with:

php -d pcov.enabled=0 build/choice-research/probe.php
php -d pcov.enabled=0 -d auto_prepend_file=build/choice-research/original-literal.php build/choice-research/probe.php

CompiledLiteralFailureTest checks all choice policies, cold/warm observations, ordered diagnostics, byte/UTF-8 sources, nonzero offsets, operand boundaries, EOF, file pages, exact failure progress, preserved work and exhausted limits. Existing mixed-run tests check preceding guards, lazy values and explicit effects. Reject this specialization if diagnostic ordering, scope restoration, execution limits or lifecycle behavior differs, or isolated representative workloads show regressions that outweigh the removed successful-prefix recognition.

Separated-list staging buffers (2026-10-22)

SeparatedList::attempt() appends selected matches and concrete syntax only after its operand succeeds. A failed item therefore leaves the staged separator and its preceding trivia unchanged. The list now retains that existing pair buffer for optional trailing acceptance, removing the separate separator-array snapshots that forced copy-on-write when a successful item was appended. At a finite item bound, the final separator attempt appends directly to the selected and concrete arrays; failure leaves those arrays unchanged. Pair staging remains in place where a separator may need to roll back with its following item.

A source counter on the saved original 128-ID nested scanner-spacing fixture recorded 266 removed snapshot assignments. Successful item appends followed nonempty shared prefixes containing 258 array slots in total. These count source operations and logical array slots, not PHP heap allocations or allocated bytes. The change introduces no new preparation, cache or persistent state. Retained tree complexity and parsing complexity are unchanged.

Fresh Windows/PHP 8.2.10 timing workers disabled PCOV, warmed three times and measured 31 recognition/CST observations each, alternating original/current order. Preparation, source setup, correctness, values, serialization, release and collection were outside timing. Five workers per variant initially measured scanner-spacing medians of 5.398 versus 5.387 ms and repeated-character medians of 6.046 versus 6.756 ms. A seven-worker follow-up reversed the apparent repeated-character slowdown: medians were 8.353 versus 7.984 ms, with scanner spacing at 6.623 versus 6.001 ms. Worker ranges overlap substantially; uncontrolled host variation establishes no consistent throughput improvement or regression.

Retained recognition memory and incremental parse peaks were unchanged: 388,600 / 400,512 used bytes for scanner spacing and 389,608 / 403,320 for repeated-character spacing. CST hashes, parser entries and successful work counts agreed. Removing short-lived array copies does not establish lower retained memory or a different asymptotic bound.

A 52,992-case original/current comparison matched acceptance, byte consumption, ordered diagnostics, CST/AST, cached values, effects and execution-limit outcomes. New regressions check bounded and unbounded trailing acceptance, partial separator failure, discarded item spacing, zero-width trivia, separator collections and exact commit ordering in both freeze modes. Original sources, counters, workers and raw reports are under ignored build/list-staging/. While retained, reproduce:

php -d pcov.enabled=0 build/list-staging/count.php
php -d pcov.enabled=0 build/list-staging/before.php --spacing=scan
php -d pcov.enabled=0 build/spacing-research/experiment.php --spacing=scan
php -d pcov.enabled=0 build/list-staging/differential.php --original-list
php -d pcov.enabled=0 build/list-staging/differential.php

Use fresh alternating workers and --spacing=set for the repeated-character control. Reject the optimization if staged data leaks after failure, separator permissions or rollback differ, effects reorder, limits change, or stable common-workload regressions outweigh the removed array traffic.

Automatic-spacing dispatch (2026-10-22)

ParseSession::skipSpacingSyntax() now owns its syntax-returning path rather than forwarding through recognizeTrivia(). It still shares the per-session wrapper cache with ordinary skipSpacing(), suppresses recursive spacing and restores that scope in finally. Prepared nullable trivia caches the two static CST kind names once. No scanner, trivia node, child match or diagnostic is removed.

A fresh trace of core and extension copies on the 128-ID nested application fixture with scanner spacing counted 22,816 source method calls before and 21,774 after, 1,042 fewer. recognizeTrivia() calls fell from 272 to 11, concreteKind() calls from 827 to 307, and the 261 syntax-only results no longer call Span::byteLength(). Parser entries, work, source allocation counts and the retained 820-node CST hash agree. Instrumented times are not performance data.

Five fresh Windows/PHP 8.2.10 workers per variant used alternating order, three warmups and 31 observations each, with PCOV disabled. Recognition/CST median of worker medians was 5.134 versus 4.765 ms for scanner spacing and 5.279 versus 5.066 ms for repeated-character spacing. Worker-median ranges overlap in both cases; these observations establish no consistent throughput improvement. Preparation, source setup, values, correctness checks, serialization, result release and collection are outside timing. Host activity was uncontrolled.

Retained recognition memory remains 388,600 bytes for scanner spacing and 389,608 for repeated-character spacing. The two cached kind references increase each prepared wrapper's size; the observed incremental recognition peak rose 64 used bytes in each fixture. Automatic wrappers live with the session, while explicit prepared Trivia can retain those fields with the grammar. There is no new cache, preparation pass or asymptotic change. First wrapper preparation does two extra kind lookups, replacing two per successful syntax-only attempt.

Original files, full-source trace reports and timing workers are retained under ignored build/spacing-dispatch/, using the independent-oracle research driver under build/spacing-research/. While retained, run each in fresh processes and alternate order:

php -d pcov.enabled=0 build/spacing-dispatch/before.php --spacing=scan
php -d pcov.enabled=0 build/spacing-research/experiment.php --spacing=scan

Use --spacing=set for the repeated-character control. The existing trivia regressions compare prepared/plain sequence, Repeat and SeparatedList behavior, including byte/UTF-8 sources, rollback, boundaries, effects and exhausted limits. A session regression covers custom syntax hooks throwing and falling back to ordinary matches while sharing one cached wrapper. Reject this change if trees, ordered diagnostics, work, scope restoration, effects or custom hooks differ, or stable common-workload regressions outweigh the dispatch reduction.

Composed scanner dispatch (2026-10-22)

Prepared sequences batch adjacent metadata-free ReadWhileSet and ReadWhileRange operands using their existing scanLength() implementation. Every scanner still constructs its original CST leaf, including empty leaves. This fuses dispatch, not the scanning patterns. Metadata separates runs; active spacing, enclosing case-insensitivity and finite work/depth limits retain the ordinary path. Failed runs retry only their original callback-free operands for diagnostics, so unlimited failed parses can charge additional speculative work. Finite budgets never enter this path. Earlier executable operands are not retried.

A fresh Windows/PHP 8.2.10 comparison used five workers per variant, alternating order, three warmups and 31 observations per worker, with PCOV disabled. The 128-ID nested application grammar used scanner spacing and composed scanner tokens. Recognition includes CST construction; preparation, source construction, value checks, serialization, result release and cycle collection are outside timing. Medians of worker medians were 12.488 ms before and 11.205 ms after (10.3% lower). Worker medians ranged 12.191–12.828 and 10.778–11.341 ms. Host activity was uncontrolled; this is not a universal speed guarantee.

Frozen parser entries fell from 816 to 558. The 1,336-node CST hash and retained recognition memory of 723,552 used bytes were unchanged. Preparation includes autoload/discovery and varied; no preparation-time improvement is established. Plans add one small grammar object and operand-reference array per eligible run; they retain no source or per-session results. Recognition remains proportional to scanned input and constructed syntax, with unchanged retained-tree complexity. Short scanner compositions still retain more nodes than the regex-token grammar; this optimization does not establish that replacing regex tokens is beneficial.

The saved pre-change Sequence, driver and raw observations are under ignored build/scanner-fusion/, using the research driver in build/spacing-research/. The maintained profiler now accepts --scan-tokens; compare matching revisions in fresh workers with --scan-spacing --scan-tokens --case=nested --size=128 --iterations=31. Its parser-entry metrics use an unfrozen graph and must not be used to infer prepared dispatch counts. Correctness tests compare same-graph CST, AST, ordered diagnostics, byte offsets, rollback, cached values, effects, file snapshots, scope restoration and exhausted execution limits.

Scanner sequence execution-record allocations (2026-10-24)

Prepared metadata-free scanner batches now return their original immutable CST leaves directly. They no longer create a temporary Success execution record, bind diagnostic metadata to it and discard it when the parent builds its CST. Literal batches already return nodes, so both batch paths use the same parent assembly loop. Child metadata, custom scanner subclasses, active spacing, enclosing case-insensitivity and finite budgets retain their existing eligibility and ordinary recognition paths. Failed scanner runs still retry only their callback-free operands for diagnostics.

Reproduce with the maintained benchmark's scanners case, which composes 128 single-unit scanners:

php tools/benchmark-literal-batches.php --case=scanners --iterations=31
php tools/benchmark-literal-batches.php --case=scanners --iterations=9 --trace

For a baseline comparison, save the previous ScannerSequence.php, LiteralSequence.php and Parser/Sequence.php in one directory as ScannerSequence.php, LiteralSequence.php and Sequence.php, then pass --baseline=that-directory. Existing literal-only baseline directories remain supported: if they omit ScannerSequence.php, the current scanner batch is used.

A Windows/PHP 8.2.10 comparison alternated three fresh workers per variant, with three warmups and 31 observations per worker, PCOV disabled and Xdebug unloaded. Each observation verifies the complete CST hash, diagnostics and cached value against unoptimized recognition; explicit effects are checked separately. Preparation, source setup, correctness projection, release and collection are outside parse timing. Separate method tracing counted 129 Success constructors before and one after per parse: only the parent execution record remains.

Untraced incremental used-PHP parse peaks fell from 69,600 to 46,968 bytes, about 33% lower on this fixture. Retained result memory stayed at 40,664 bytes. These are PHP allocator measurements, not RSS, and they do not change the retained CST's size or asymptotic memory bound. Host load was uncontrolled; traced timing and memory are distorted and are not used for allocation-byte comparisons. The saved baseline and raw reports are locally retained under ignored build/scanner-allocations/ while available.

Regressions cover independent empty and consuming CST leaves, named-result operations between batches, lazy cached outer values, validation and explicit commit. Existing scanner-sequence tests also compare trees, ASTs, diagnostics, work, rollback, scope restoration, file spans and exhausted budgets.

Separated-list checkpoint reuse (2026-10-22)

SeparatedList::recognize() reuses its loop-entry diagnostic checkpoint for an absent optional leading separator. Only local variables are initialized between loop entry and the separator attempt, so the second checkpoint captured identical session state. Pair termination, bounds, spacing and deferred failures retain their existing restoration paths.

A fresh PHP 8.2.10 instrumented comparison counted checkpoint captures on a successful eight-item list: 17 before versus 9 after, or 18 versus 9 with leading separators enabled. Both freeze modes produced the same counts and matching CST/value/diagnostic fingerprints. A separate 52,992-case baseline/current comparison matched parser outputs and execution-limit outcomes. Regression tests cover a partial optional leading separator observing farther input before its failure is discarded, followed by successful or required-failing item recognition.

This removes transient outer checkpoint arrays, not deep copies of their shared contents. No state, preparation pass or asymptotic change is introduced. No throughput or retained-memory improvement is established. The saved original SeparatedList, counter and differential driver are under ignored build/list-checkpoint/; while retained, reproduce from the repository root:

php -d pcov.enabled=0 build/list-checkpoint/count.php --baseline
php -d pcov.enabled=0 build/list-checkpoint/count.php
php -d pcov.enabled=0 -d memory_limit=512M build/list-checkpoint/differential.php --baseline
php -d pcov.enabled=0 -d memory_limit=512M build/list-checkpoint/differential.php

Deferred-list union sharing (2026-10-22)

ParseSession now retains tied, unobserved deferred-list observations as shared Runtime\DeferredDiagnostic unions. Their ordered operands are logically immutable. Reporting flattens shared branches once, caches a normal Diagnostic with fixed arrays, and releases the materialized node's operand references. Already observed active/deferred combinations reuse their cached output and keep eager merging; rebuilding a lazy history on that path would add unnecessary work. Farther observations still win without flattening nearer histories.

Flattening preserves left-before-right ordering, including both nested deferral boundaries: the checkpoint's deferred details precede newly deferred details, while ordinary details precede deferred details when exposed. Numeric-looking path components retain the original pairwise SORT_REGULAR merges. Numeric and lexical string comparisons can interact, so an arbitrary flattened union must not silently substitute strict path equality or regroup those operations. Identical normalized observations share their existing snapshot, and pending chain depth is capped at 128 by periodic materialization. Explicit checkpoints can still retain older histories; this is not constant-memory parsing.

Checkpoint arrays retain five fields, with the deferred field typed as Diagnostic|DeferredDiagnostic|null. Save and restore them as opaque snapshots; diagnostic() and diagnostic(true) remain the observation APIs. Public result Diagnostics retain their existing class, ordered readonly arrays and byte offsets. Work/depth accounting, parser delegation, selected trees, semantic caches and explicit effects are unchanged. Cache keys publish only after materialization succeeds, allowing a failed source check to retry without returning stale output.

A Windows/PHP 8.2.10 comparison used five fresh workers per variant in alternating order, one warmup and 11 observations per synthetic timing case, with PCOV disabled. Instrumented workers counted allocations and candidate entries separately; their times are not performance evidence. These medians are medians of worker medians:

512 tied deferred probes Before ms After ms
Accumulate without exposing the final union 140.701 4.726
Accumulate and expose the final union 141.680 5.128
Repeated identical singleton details 1.658 1.441

The distinct probes have different expectation labels and different nonnumeric paths at one byte offset. Before, both distinct cases constructed 1,534 Diagnostics and processed 393,470 expectation entries plus 393,470 path entries through tied unions. After, pending accumulation constructed 526 Diagnostics and 1,022 small union nodes, processing 5,474 entries of each kind during 14 compactions. Requesting the final union raised that to 527 Diagnostics, 5,986 entries of each kind and 15 materializations. These are logical input slots, not allocated bytes or unique output entries. Total object count does not decrease: cheap union nodes replace much more expensive repeated array merging. Identical singleton probes create no pending nodes. Numeric-path probes retain their eager operations; the 128-probe numeric control measured 10.044 versus 10.172 ms, showing a small overhead.

The benefit depends on whether intermediate unions are observable. A longest Choice over 512 nullable SeparatedLists, followed by a successful literal, creates tied termination failures while succeeding. With FreezeFlags::NONE, its Diagnostic count fell from 1,534 to 526 and expectation input slots from 393,470 to 5,474. Three fresh timing workers per variant measured 16.367 versus 12.068 ms; a trailing-input failure variant measured 17.091 versus 12.189 ms. The worker ranges did not overlap in these synthetic cases. With default optimization, failed item metadata already exposes each union: the cache-reuse path retains the original 1,534 Diagnostics and input-slot counts. A focused five-worker comparison with 51 observations per worker measured 17.153 versus 17.289 ms for success, with overlapping ranges. Do not infer the isolated session fixture's roughly 30-fold gain for ordinary application parsing.

The nested application control used five fresh workers and 101 observations each. Recognition medians were 4.464 versus 4.457 ms with overlapping worker ranges; retained recognition memory remained 389,704 bytes and the CST hash was unchanged. No application throughput improvement or consistent regression is established.

Separate memory workers measured PHP used-byte deltas above each iteration's pre-call baseline, with the source and grammar already prepared. Keeping the 512-probe session pending retained 219,592 versus 220,376 bytes; peak usage rose from 289,336 to 348,312 bytes. Exposing the final Diagnostic retained 168,208 bytes in both versions, but still incurred the higher peak. Shared nodes, traversal bookkeeping and materialization chunks trade temporary memory for less repeated union work. The default-optimized successful grammar retained 1,144 bytes and peaked at 425,160 bytes in both versions. These are neither RSS nor cumulative allocation measurements.

There is no new freeze pass or retained grammar cache; first use can autoload the new runtime class. Cold preparation/first-parse differences were not established. Flattening visits shared branches once per materialization, but candidate arrays still need merging/deduplication. Periodic compaction and repeatedly observed growing prefixes can still produce quadratic total traffic. Numeric paths keep the original complexity. No general linear-time or bounded-total-memory guarantee is introduced.

Baseline/current fingerprints matched 52,992 parser cases and 14,306 randomized session observations, including checkpoint forks, restore/relabel/defer ordering, numeric and binary paths, byte/UTF-8 sources, both freeze modes, all choice policies, values, effects and work/depth exceptions. Focused tests cover shared branches, operand reuse, immediate operand release, duplicate/long histories, numeric comparison boundaries and retry after failed materialization. Reject the change if ordered payloads, rollback, retained snapshots or limit outcomes differ; if pending memory exceeds the application's budget; or if stable common-workload regressions outweigh the gains on deferred-heavy inputs.

Saved original core/extension sources, drivers and raw reports are under ignored build/deferred-unions/. While retained, run each variant in fresh processes, alternating order; add --baseline to select the saved original sources:

php -d pcov.enabled=0 -d memory_limit=512M build/deferred-unions/measure.php --iterations=11
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-unions/measure.php --trace
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-unions/measure.php --memory
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-unions/measure.php --no-optimize --case=grammar --iterations=11
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-unions/session-differential.php
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-unions/differential.php
php build/deferred-unions/summarize-published.php

Correctness checks and serialization are outside synthetic timers and memory observations. Grammar preparation is separate; sessions/parse results are returned from timed calls, and result release occurs afterward. The pending-session memory case deliberately retains that session before observing its final union. Instrumented counts stop before correctness inspection. profile.php wraps the maintained extraction profiler with matching extension sources; baseline runs also pass --source-dir=build/deferred-unions/baseline/src. Final timing artifacts are summary-published.jsonl, summary-control.json and summary-none.json; earlier exploratory runs remain alongside them. Host activity is uncontrolled.

Discarded terminal failure snapshots (2026-10-22)

The shared checked recognition path now permits exact built-in Choice, Repeat and SeparatedList consumers to omit Diagnostic snapshots on terminal failures whose records they discard without inspecting. Ordinary recognition still records every failure, expectation and rule path. The owning combinator materializes the observable diagnostic, including deferred list failures and relabeling. Successful child records and guard rejections retain their snapshots. Custom terminals, combinator subclasses and other consumers retain eager metadata by default. The eligible ordinary terminals are exact Any, Char, Range, Regex, Set and Text classes; overriding their failure hooks does not opt a subclass in.

Optimizer Choice and Repeat apply the same rule to lexical-table replay. First observations still materialize diagnostics needed to prepare relative recipes. No new recognition entry point, lazy public Diagnostic proxy, persistent history, session cache or grammar preparation pass is introduced. Byte progress remains available without constructing an active/deferred diagnostic union. The internal delegation option is local to one attempt, so it requires no mutable scope to restore after exceptions. Deferred-list accumulation and checkpoint representation are unchanged.

A fresh Windows/PHP 8.2.10 comparison instrumented source copies separately from time and memory workers, with PCOV disabled. A fixture tries 512 distinct failing Text alternatives before accepting az; repeat and list fixtures accept eight items. Warmed frozen optimization produced these Diagnostic construction counts:

Successful fixture Before After
Choice 512 1
Repeat of Choice 10 9
SeparatedList of Choice 4,098 10

The already optimized repeat has little additional opportunity. The nested application-value fixture still constructs four Failures and four Diagnostics, retains 822 CST nodes and uses 389,704 bytes after recognition. There is no application allocation reduction in that control.

Separate warmed memory workers measured the 512-alternative choice's PHP used recognition peak at 73,888 versus 61,816 bytes above the prepared baseline. Retained result memory was unchanged at 14,080 bytes. The 128-alternative list fixture's peak was 56,552 versus 54,320 bytes, with unchanged retained memory of 37,480 bytes. These are used bytes, not RSS, process allocation peaks or cumulative allocation. Temporary object elimination does not remove retained CSTs or selected semantic state.

Timing used five fresh workers per variant, alternating order, three warmups and 101 observations per worker. One run's median of worker medians improved the 512-alternative choice from 2.952 to 2.209 ms; a follow-up measured 3.188 versus 3.252 ms with widely overlapping worker ranges. The nested control measured 5.317 versus 5.344 ms initially, and 7.521 versus 5.208 ms in the follow-up, again with overlapping ranges. Tiny literal/sequence controls varied too. These noisy runs establish no consistent throughput improvement or regression. Do not infer application speedups from allocation counts or instrumented timings.

There is no additional retained grammar or session data structure. Preparation complexity is unchanged; a cold preparation-time difference was not established. Recognition still visits the same parsers and records the same failures. Avoided snapshots reduce temporary allocation and some copy-on-write pressure, while ordered expectation/path accumulation and observable array materialization remain. Selected successful children can still retain different historical snapshots.

A 52,992-case baseline/current comparison matched acceptance, byte consumption, ordered diagnostics with formatted locations/excerpts, CST/AST, cached values, explicit effects and exact work/depth exception offsets and paths. Cases include byte/UTF-8 sources, nonzero entry positions, all choice policies, both freeze modes, labels, malformed nested lists and optional/lookahead rollback. Regression tests cover successful/guard-rejected operands, retained custom terminal failures, combinator subclass hooks through cold and warmed lexical tables, immutable observations and deferred farthest offsets. The full runtime suite also passes. Reject this optimization if any observer loses metadata, a retained snapshot changes, failure ordering or rollback differs, effects execute speculatively, limit accounting changes, or reproducible common-workload regressions outweigh the temporary allocation benefit.

The original sources, experiment drivers and raw reports are under ignored build/deferred-attempts/. While retained, reproduce baseline and current in fresh processes, alternating their order:

php -d pcov.enabled=0 -d memory_limit=512M build/deferred-attempts/measure.php --baseline
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-attempts/measure.php
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-attempts/measure.php --memory
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-attempts/experiment.php --trace --frozen-opt
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-attempts/differential.php --baseline
php -d pcov.enabled=0 -d memory_limit=512M build/deferred-attempts/differential.php

Add --baseline to trace or memory commands for the saved source snapshot. profile.php wraps the maintained extraction profiler with matching extension sources; baseline workers additionally pass --source-dir=build/deferred-attempts/baseline/src. Reports under idle/ retain the follow-up timings. Neither run controls unrelated host activity.

Internal effect lifetimes (2026-10-22)

Parser::bindCommitEffects() now stores a CommitEffects registration snapshot instead of a closure capturing its owning Success. CallbackTrait supplies that owner only when executing the binding. Ordinary custom callbacks retain their zero-argument contract and position in the same callback list. Assignments still precede effects; one ParseResult wrapper is shared by the effects in each binding. No additional ExecutionState fields, weak references or per-recognition caches are needed. One binding object replaces one closure; this is primarily a lifetime improvement, not a reduction in the number of objects constructed.

A fresh local Windows/PHP 8.2.10 comparison used 128 successful longest-choice alternatives, each with a cached constant mapping and a no-op commit effect. Separate instrumented source copies counted allocations and live objects with GC disabled. Both versions created 129 Successes, 129 ExecutionStates, 128 Spans, 129 ConcreteNodes and 128 SemanticProjections. Before explicit GC, live Successes fell from 129 to 2, live projections from 128 to 1, and live nodes from 129 to 2. The new version created 128 binding objects, with only one still live at return; the old trace excludes its 128 closure allocations. After GC the selected graphs have the same object counts. Existing span reuse, sparse state and projection caching remain in place.

Uninstrumented memory workers measured these used-byte deltas from the prepared grammar/source baseline, consistently across five workers and 21 observations:

Phase Before After
Recognition return, before explicit GC 245,104 2,632
Recognition return, after explicit GC 3,296 2,632
Recognition peak 291,400 50,832
After value read and result release, before GC 1,904 0
After final GC 0 0

Elapsed time was measured separately without tracing or memory-phase GC. The first five-worker run was noisy and appeared slower. A follow-up with 11 fresh workers per variant, alternating order, three warmups and 101 observations gave median recognition times of 0.955 / 0.946 ms. Worker-median ranges overlap (0.855–1.191 / 0.866–1.530 ms). The no-effect control took 0.876 / 0.866 ms, also overlapping. These results establish no throughput improvement. Commit throughput was not measured; dispatch adds an instanceof check per callback. Preparation retains the same grammar registrations, with one new class to autoload; cold preparation-time differences were not established. Asymptotic recognition, tree retention and commit complexity are unchanged. Destruction work moves from cycle collection to reference release during recognition.

CST/diagnostic fingerprints matched across variants and measurement modes. Regression tests disable cycle collection and check immediate source/record release, losing alternatives, registration snapshots, assignment order, shared effect wrapper identity, zero-argument callbacks and independent commits. Existing tests cover cached exceptions, partial commit failure, validation and recovery. User callbacks, cached values and exception traces may still form cycles. Reject this change if selected state is released early, speculative effects execute, snapshots/order/errors change, or stable representative workloads show an unacceptable recognition or commit regression.

The saved source baseline, driver and raw reports are under ignored build/effect-cycles-20261022/. While those artifacts remain, reproduce each phase separately (use --source-root=build/effect-cycles-20261022/baseline for the original version), alternating variants in fresh processes:

php -d pcov.enabled=0 build/effect-cycles-20261022/experiment.php --case=ranked-effects --size=128 --mode=trace --iterations=1
php -d pcov.enabled=0 build/effect-cycles-20261022/experiment.php --case=ranked-effects --size=128 --mode=memory --iterations=21
php -d pcov.enabled=0 build/effect-cycles-20261022/experiment.php --case=ranked-effects --size=128 --mode=time --iterations=101

Single-unit terminal allocations (2026-10-22)

Exact built-in Char, Range, Set and Any terminals now reuse a successful Source::readUnit() span instead of constructing an equal match span. Both success factories use the same metadata, collection and effect binding code. Subclasses retain the existing success() and makeMatch() hooks. Char validates its literal encoding directly; construction already established its unit count. Range bounds and Set membership preparation likewise validate strings without temporary Utf8Source objects. Invalid literals still fail at recognition, including EOF.

A local Windows/PHP 8.2.10/PCRE 10.40 comparison used complete saved original core/extension sources and the changed checkout. It isolates this change. Separate allocation traces covered copies of both trees; elapsed time and memory came from uninstrumented workers with PCOV disabled. Fourteen scenarios used five fresh workers per variant in alternating order, three warmups and 21 timed observations; separate memory workers used three observations. Grammar/source preparation, correctness checks, serialization, result release and explicit GC were outside recognition timers. Medians of worker medians for 1,024-unit repeats:

Scalar terminal/source Recognition ms before / after Source Span allocations before / after
Char, Unicode UTF-8 memory 7.112 / 6.194 2,050 / 1,026
Char, ASCII UTF-8 memory, optimization disabled 6.754 / 6.094 2,050 / 1,026
Char, byte memory, optimization disabled 6.129 / 5.567 2,050 / 1,026
Range, Unicode UTF-8 memory 7.010 / 6.790 2,050 / 1,026
Set, Unicode UTF-8 memory, optimization disabled 6.593 / 6.260 2,050 / 1,026
Any, UTF-8 memory, optimization disabled 5.481 / 5.190 2,050 / 1,026
Char, Unicode UTF-8 file 10.115 / 9.574 2,050 / 1,026

The Unicode Char fixture also removed 1,025 temporary Utf8Sources, including its EOF attempt. Warm Range/Set already cached their validation, so their corresponding source counts were zero in both recognition traces. Counts describe source new expressions, excluding PHP internals, arrays and closure creation. The resulting CST still has 1,025 nodes and retains 1,025 spans. Retained recognition memory remained 316,184 used bytes and the parse peak remained 535,448 bytes in the Char fixture: this removes short-lived objects, not the retained tree/execution graph. First/cached value allocation and semantic behavior are unchanged.

The larger first-run application controls were noisy, so a separate comparison used 11 workers per variant and 101 parses each. Unicode Char recognition took 7.429 versus 6.624 ms (about 11% faster), with nonoverlapping worker-median ranges in that run. Nested application recognition took 4.395 versus 4.476 ms, the record took 0.651 versus 0.645 ms, and an unoptimized Text repeat took 5.730 versus 5.818 ms. Those control ranges overlap; the added common binding call may impose a small cost on ordinary successes. No general application speedup is established. Already optimized ASCII/scanner repetitions keep their existing allocation counts and have little to gain from this scalar change. Preparation adds no caches or state fields, and recognition/tree-retention complexity is unchanged.

Baseline/current workers agreed on CST/diagnostic hashes and all 21 application acceptance/domain oracles. A separate 8,968-case comparison matched byte/UTF-8 sources, every tested byte position, include/exclude and case policies, invalid literals, both freeze modes, metadata, collections, values, assignments, explicit commit, AST/CST output, formatted diagnostics and exact work/depth failures. Regression tests also cover custom forwarding sources, subclass hooks and scope restoration on exceptions, page-crossing file units, closed snapshots, zero-width failures and independent recognitions sharing one immutable source span.

The source snapshot, drivers and raw reports are under ignored build/single-unit-allocations-20261022/. While those artifacts remain:

powershell -NoProfile -ExecutionPolicy Bypass -File build/single-unit-allocations-20261022/run.ps1
php build/single-unit-allocations-20261022/summarize.php
powershell -NoProfile -ExecutionPolicy Bypass -File build/single-unit-allocations-20261022/run-controls.ps1
php build/single-unit-allocations-20261022/summarize-controls.php

Nullability preparation (2026-10-21)

Graph analysis now records lookup dependencies during each stable nullability round and schedules calculations whose observed estimates changed. This includes self-reads, custom reads of collected descendants and operands discovered as short circuits open. Updates are published after the round's lookup is released, avoiding an entire estimate-map copy on each round. The conservative ordering false < unknown < true, progress checks and leading-cycle checks still apply. The dependency tables are temporary; frozen grammars retain no analysis cache.

A local Windows/PHP 8.2.10 comparison loaded saved original GraphAnalysis and NullabilityLookup classes or their changed versions against the same remaining checkout. Each of 32 scenarios used three fresh workers per variant in alternating order, a first-use freeze, two warmups and five timed observations. PCOV was disabled. Builder construction, source setup, recognition, correctness checks and release were outside freeze timers. Timed phases separated snapshot copying/resolution, analysis and optimizer preparation, alongside complete Parser::freeze() calls. Deferred factory construction remains part of snapshot copying/resolution. Medians of warmed worker medians, with default optimization enabled:

Grammar Complete freeze before (us) After (us)
One Text terminal 4.8 5.2
128 nested nullable Sequences 8405.7 3586.2
1,024 nested nullable Sequences 396236.8 46157.1
512 branches sharing a nullable chain 21388.7 16888.0
RFC URI 2880.4 2687.1
Mail address list 1172.2 1138.8
XML document 3165.2 2982.4
JSON text 456.5 459.8
Application-value extraction 94.2 91.4
512 consuming sequence operands 4096.6 4375.8

Both FreezeFlags::NONE and OPTIMIZE were measured. With optimization disabled, the 1,024-wrapper freeze took 363.31 versus 19.32 ms. URI analysis alone took 0.883 versus 0.557 ms without optimization, and 0.890 versus 0.578 ms with it. JSON and extraction changes were small; the one-terminal and consuming controls show overhead without a reduction in nullability calculations. Cold samples are retained separately: initial freezes can include provider discovery and class loading, while some authoring paths incur those costs during construction. No general cold-start or warmed recognition speedup is claimed.

Separate instrumented runs counted all nullability calculations on the 1,024-wrapper graph: 1,051,650 before and 2,049 after; the Sequence-only counts are 1,050,624 and 2,048. Fixed-point rounds were still 1,026 versus 1,025: the gain comes from avoiding whole-graph visits each round. URI calculations fell from 1,750 to 281, and the 512-branch shared graph from 10,600 to 1,059. Instrumented times are not used as performance measurements.

The reverse dependency map costs O(V + E) temporary storage for observed reads. Each estimate increases at most twice, bounding scheduling notifications by O(E); calculation cost still depends on operand scans and custom hooks. A simple chain now needs O(V) nullability calculations, but collected paths and cycle traversal still have quadratic costs on deep chains. The optimized URI analysis used-memory peak rose from 176,264 to 230,376 bytes; the 512-operand nullable sequence rose from 312,296 to 485,576 bytes. The deep-chain analysis peak stayed about 44.3 MB because paths dominate. These are incremental PHP used-memory peaks above prepared snapshot baselines, not process allocation peaks or RSS. Retained frozen-grammar memory was unchanged in these scenarios.

All workers checked independent acceptance and source round trips, with exact application values for extraction. Cold/warm frozen graphs and the phase-split prepared roots produced matching CST, AST, value and diagnostic fingerprints across variants. Library inputs included valid/malformed cases and generated sizes 1/8/64 (repetitions or payload length); this preparation change does not depend on input length. The same targeted safety tests passed against original and changed classes. Regression coverage includes stable-round reads, unknown propagation, changing read dependencies, non-monotone self-reads, recursion and both freeze modes.

The driver, original classes, raw cold/warm observations, trace copies and summary are under ignored build/nullability-preparation/. While those snapshots remain:

php -d pcov.enabled=0 -d memory_limit=512M build/nullability-preparation/measure.php
php -d pcov.enabled=0 -d memory_limit=512M build/nullability-preparation/measure.php --trace
php -d pcov.enabled=0 build/nullability-preparation/summarize.php

Three workers and small timings have host/allocator noise. Reject the optimization if safety decisions, warnings, paths, snapshot isolation or deferred operations differ, or if the application's preparation/memory cost regresses beyond its budget. Large chain gains do not imply gains for already shallow or reused graphs.

Bounded diagnostic scanning (2026-10-21)

Diagnostic::location() now scans only its source prefix, with one-byte lookahead when that prefix ends in CR. excerpt() searches forward for the current line's terminator in 8 KiB chunks. A diagnostic retains its coordinates and line boundaries, replacing the per-diagnostic array of every line start. Repeated locations retain the closed-file check without rereading prefix bytes. Default array/JSON output continues to read no source bytes.

A local Windows/PHP 8.2.10 comparison loaded the saved original Diagnostic or the changed class in fresh workers against the same remaining checkout. PCOV was disabled. Each of 32 scenarios used three workers per variant in alternating order, a separate first-use observation, two warmups and seven timed observations. The sources were in-memory and file-backed UTF-8; fixtures had 64, 4,096, 65,536 or 350,000 short CRLF lines, or a 64 KiB/1 MiB Unicode line. Offsets covered the start, middle and EOF. Source preparation, fixture construction, assertions and release were outside formatting timers. Medians of warmed worker medians:

Source and requested position, 1,050,001-byte short-line fixture First location before After
UTF-8 memory, start 62.07 ms 0.0007 ms
UTF-8 memory, middle 62.42 ms 24.82 ms
UTF-8 memory, EOF 63.17 ms 49.41 ms
UTF-8 file, start 63.08 ms 0.0008 ms
UTF-8 file, middle 64.04 ms 26.09 ms
UTF-8 file, EOF 64.01 ms 50.09 ms

All workers matched independent expected locations, excerpts, messages and JSON; before/after output fingerprints agreed in every scenario. Separate read-counting workers requested 1,050,001 logical bytes for the original byte-zero location and zero nonempty bytes afterward. Requesting its first excerpt then read 8,194 bytes (one search window and the two-byte line). These are Source slice requests, not physical disk I/O. Instrumented timings are not used as performance data.

Location retained used memory on that in-memory fixture fell from 10,486,344 to 536 bytes, measured above prepared-source baselines. For the single-line fixture, it rose from 504 to 536 bytes. These deltas include the diagnostic object and returned coordinates; they are PHP used bytes, not allocated process peaks or RSS. No preparation or parsing speedup is claimed.

First location remains O(requested byte offset), with O(1) retained position metadata. First excerpt additionally scans the remaining current line and copies the requested line; repeated excerpts still copy it. A first excerpt after a known location can cost more than the old index lookup: the file/middle case above took 15.6 versus 10.8 us. The 1 MiB Unicode line at EOF showed no meaningful first-location improvement (memory: 53.51 versus 53.44 ms; file: 54.35 versus 54.25 ms). Cached location access avoids recounting long prefixes (memory/EOF: 1.95 ms to 0.00044 ms in that fixture). Small timings and three workers are noisy; these measurements establish bounded reads and workload-specific gains.

The driver, original class snapshot, raw observations and summary are under ignored build/diagnostic-line-scanning/. While that snapshot is retained, reproduce with:

php -d pcov.enabled=0 -d memory_limit=512M build/diagnostic-line-scanning/experiment.php
php -d pcov.enabled=0 -d memory_limit=512M build/diagnostic-line-scanning/experiment.php --reads
php -d pcov.enabled=0 build/diagnostic-line-scanning/summarize.php

Timing workers separately measure construction, first/cached location access, first/cached excerpt after location, and JSON after both caches are populated. The first-use observation follows source setup and is not cold disk I/O. Regression tests cover every byte position in mixed-newline/Unicode/binary fixtures, CRLF at chunk/page boundaries, read bounds, failed-read retry and closed snapshots. Any changed coordinates, excerpt bytes, source-lifetime behavior or unbounded early-position reads would reject this optimization.

Automatic trivia syntax (2026-10-21)

Eligible prepared sequence, repeat and separated-list implementations now retain nullable automatic trivia as its original immutable syntax without constructing a temporary wrapper execution record. Required-minimum trivia, metadata, custom classes, explicit trivia and finite work/depth limits keep their existing paths. The public session skipSpacing() method still returns independent matches.

A local Windows/PHP 8.2.10 comparison used complete core/extension snapshots, five fresh workers per variant/scenario, alternating before/after order, three warmups and 30 timed parses per worker (seven for the 1,024-ID fixture). PCOV was disabled. Timings include fresh UTF-8 source setup, complete recognition/CST and the first domain value. Preparation, assertions, serialization, result release and cycle collection are outside timing. Medians of worker medians, microseconds:

Fixture Before After Speedup
Nested 128 IDs, repeated-character spacing 5453.8 4513.2 1.21x
Nested 128 IDs, scanner spacing 5010.4 4206.1 1.19x
Late malformed nested list, scanner spacing 5060.6 4170.8 1.21x
Nested 1,024 IDs, scanner spacing 40396.3 32447.8 1.24x

Every application worker checked all 21 independent oracle cases, and all before/after workers agreed on CST/diagnostic fingerprints. No-spacing sequence and repeat controls were approximately unchanged. A longer small-record run used 15 workers per variant and 100 parses each: repeated spacing took 785.8 versus 751.9 us; scanner spacing took 713.6 versus 716.2 us. The latter is effectively neutral. Host variation and small-input overhead preclude a universal speed claim. Cold/preparation observations are retained in the raw report; class loading and extension discovery affect those costs, and no preparation speedup is established.

A separate trace of both core and extension methods on the scanner-spacing 128-ID fixture counted 429 to 168 checked parser entries and 425 to 164 Success allocations. ConcreteNode allocations remained 826, spans remained 555, and the retained tree still had 820 nodes and the same serialized hash. Retained used memory after value access was unchanged at 392,536 bytes. Incremental used-memory parse peaks fell from 449,312 to 400,480 bytes; the 1,024-ID peaks fell from 3,338,016 to 2,896,032 bytes. These are PHP used bytes above a warmed baseline, not allocated process peaks or OS RSS. The rewrite reduces temporary records, while CST retention and recognition complexity remain unchanged.

The driver, full snapshots and reports are under ignored build/trivia-record-optimization/. Run php -d pcov.enabled=0 -d memory_limit=512M build/trivia-record-optimization/measure.php to repeat the timing comparison while that local snapshot is retained. Trace workers use --worker --variant=before|after --case=nested --spacing=scanner --iterations=1 --trace separately; instrumented times are not performance data.

Prepared trivia and direct list punctuation (2026-10-21)

Prepared plain trivia can scan its operand and construct the same child syntax without a temporary child execution record. Eligible ASCII literal list separators can likewise create their final CST node directly. The ordinary list loop retains bounds, rollback and lazy item values. Custom/decorated operands, deferred trivia and finite execution limits retain the corresponding ordinary paths. See grammar optimization for eligibility and fallbacks.

Local Windows/PHP 8.2.10 measurements compare saved source and extension snapshots with whitespace specialization alone and both changes. Each variant/case used six fresh workers in rotating/reversed order, three warmups and 30 timed parses per worker. The table reports medians of worker medians, in microseconds. Parsing starts from a string, requires complete input and obtains the domain value; preparation, assertions, result release and cycle collection are outside timing. PCOV is disabled. The nested fixture contains 128 generated IDs.

Fixture Before Trivia alone Both changes Combined speedup
Record, repeated-character spacing 773.6 671.1 704.8 1.10x
Record, scanner spacing 711.3 626.8 637.1 1.12x
Nested list, repeated-character spacing 7338.4 5840.2 5482.4 1.34x
Nested list, scanner spacing 6807.0 5579.1 5189.2 1.31x

Every worker checked all 21 independent acceptance/output oracle cases. CST and diagnostic fingerprints agreed across all workers and variants. Retained CST counts and used-memory deltas were unchanged. The list specialization improved the larger fixture beyond trivia alone, but its extra list bookkeeping made the record slightly slower than trivia alone in this run. These workload-specific measurements do not establish universal gains or individual CPU attribution. Separate traces on the scanner-spacing nested fixture counted 830 parser entries before and 429 after, and 826 temporary Success allocations before and 425 after. The retained tree still has 820 nodes and the same serialized hash. Tracing covers core source methods, adds overhead and is used for call/allocation counts only; new expressions in uninstrumented extension methods are not included. The local driver, snapshots and raw reports are under ignored build/trivia-cst-optimization/.

The Extraction application-value comparison checks two independent grammars against one output oracle and records extraction timings, memory retention, diagnostics and static-analysis observations. It includes Comprehend's mandatory CST cost and an isolated optional dependency.

Runtime allocation and dispatch (2026-10-20)

Optimized repetition now creates its original per-unit CST nodes directly. Its shared scan primitive returns acceptance and byte progress without constructing a discarded scanner tree or temporary child execution records. Parser metadata and deferred operations on the repeat still use the ordinary preparation path.

Sequence preparation compiles adjacent eligible literal runs around ordinary operands such as regexes and decorated parsers. Failed runs retry only their own original operands, preserving diagnostics without repeating earlier recognition callbacks. Active spacing, case-insensitive scopes and finite work/depth limits retain the ordinary sequence path.

Diagnostic accumulation keeps ordered expectation/path lists with membership indexes. Restoring a changed checkpoint invalidates the corresponding index; the next insertion rebuilds it from the restored list. Binary labels, strict path identity, insertion order and the existing checkpoint shape remain intact. Semantic projections also consult their value/error cache before constructing a callback-facing ParseResult. Nulls, object identity, recursive evaluation errors and explicit commit behavior retain their existing contracts.

Local Windows/PHP 8.2.10 measurements compare a saved source snapshot with these four changes together. Each case/variant used five fresh workers, alternating before/after order. Workers used two warmups and seven measured batches; the table reports the median of worker medians. Parse fixtures use in-memory byte sources. Preparation and source construction are excluded; parsing includes full CST construction, acceptance checks and result release.

Fixture Before After Speedup
Repeat 4,096 set characters, original character nodes retained 8.271 ms 3.677 ms 2.25x
32 literal children, regex, 32 literal children 0.405 ms 0.152 ms 2.66x
Accumulate 4,000 distinct expectations and paths at one position 205.025 ms 3.858 ms 53.15x
One cached scalar value read 0.664 us 0.305 us 2.18x
Nested application-value fixture, recognition and CST 9.896 ms 8.625 ms 1.15x

Scanner/mixed batches contain 20 parses, nested batches five parses, diagnostics one accumulation and cached-value batches 100,000 reads. The diagnostic fixture isolates accumulation before requesting its Diagnostic; it does not model the cost of creating and retaining a diagnostic snapshot after every insertion. Cached-value evaluation is warmed before measurement. Parse CST hashes agree across all workers and variants, and the nested fixture checks expected domain values outside timing. These synthetic, combined measurements do not establish individual attribution, a general throughput guarantee or a preparation-time win.

Separate traces over copies of both source and extension methods counted 4,098 Success allocations before and one after for the repeated-character fixture. Its retained tree still contains 4,097 nodes; ConcreteNode allocations fell from 4,098 to 4,097. Ten warmed value reads allocated ten ParseResult wrappers before and zero after. Tracing adds overhead and is used only for allocation evidence. The fixture driver, source hashes, worker reports and traces are saved locally under ignored build/architecture-optimizations/.

Recursive literal sequences (2026-10-20)

The optimizer can compile eligible nested literal sequences and fixed repetitions to one internal Text recognizer, retaining every original CST node through source-free templates. See grammar optimization for the eligibility rules and fallbacks. This reduces parser dispatch and temporary match records; it does not remove CST allocation or change character-node granularity.

Local Windows/PHP 8.2.10 measurements compare the previous optimizer with literal compilation enabled, rather than comparing all optimization against NONE:

Fixture Bytes Previous optimizer ms Literal compilation ms Speedup
Flat sequence of 512 one-byte texts 512 3.460 0.908 3.81×
16 sequences of 32 one-byte texts 512 3.666 0.601 6.10×
Exact repeat of 256 ab texts, then ! 513 1.126 0.297 3.79×

Each result is the median of three fresh-worker medians. Before/after worker order alternated; workers used two warmups and seven batches of 20 parses per prepared grammar. Timings include recognition, full CST construction, acceptance checks and result release, and exclude source construction and grammar preparation. CST hashes matched across all workers and both optimizer versions for each fixture. The benchmark also checked the unoptimized CST and alternated unoptimized/optimized batch order within each worker. These synthetic results are not general latency guarantees; they do not establish gains for variable mail or XML grammars.

Compilation adds preparation work: optimized preparation medians increased from 30.55 to 34.58 ms for the flat fixture, 1.49 to 9.81 ms for nested sequences, and 1.15 to 3.03 ms for the fixed repeat. The first fixture also includes cold optimizer class loading. Reuse frozen grammars to amortize classification and template creation. Raw local reports are saved under ignored build/performance-review/.

The specialized benchmark used for these historical measurements has been removed.

Library lexical tables (2026-10-17)

Freeze-time optimization now configures bounded lexical tables after references are resolved and graph safety checks pass. Repeat operands, Choice alternatives and Not probes can reuse source-free syntax/diagnostic recipes for known finite ASCII patterns. Named-rule nodes, selected branches and diagnostics remain intact. The first observation uses the original runtime; executable metadata and custom parsers always execute normally. Tables hold at most 128 entries per operand, with at most 32 bytes of proven lookahead. FreezeFlags::NONE disables them.

The historical benchmark ran each case/flag pair in a fresh PHP worker. It recorded grammar preparation, first-parse time, warm min/median/max time, retained memory, allocated process peak and warmed parser visits. Before accepting results it checked complete consumption and identical CST/diagnostic fingerprints across flags and between the first and warmed parses. Recognition includes full CST construction; source construction and serialization used for verification are outside the timed loop. The specialized benchmark has been removed.

Measured on Windows with PHP 8.2.10, validated in-memory UTF-8 sources, seven samples of 30 parses per prepared grammar:

Case Bytes NONE median ms OPTIMIZE median ms Speedup Warm parser visits, NONE → OPTIMIZE
Named URI port digits 300 4.215 1.682 2.51× 604 → 2
ABNF quoted text / range choice 322 9.264 2.323 3.99× 890 → 7
URI with repeated percent escapes 468 19.289 5.751 3.35× 2260 → 147
URI with varied hashed path segments 343 13.412 8.054 1.67× 1699 → 521
XML with a long CDATA section 445 29.549 25.349 1.17× 2764 → 1895
Mail address list 322 26.208 26.508 0.99× 3432 → 3432
JSON array 121 5.243 5.165 1.02× 623 → 623
IPv6 2001:db8::1 11 5.441 2.636 2.06× 741 → 270

These are workload-specific warm results, not general speed guarantees. The mail and JSON list controls are effectively unchanged: variable-size items retain their original loops; no automatic SeparatedList substitution or separator-loop fusion was added. XML still runs its character guards; finite delimiter probes and ASCII regex alternatives account for its smaller improvement. IPv6 retains its original ordered/ranked forms, while their finite group patterns reuse recipes.

Tables trade memory and initial classification work for repeated recognition. Relative post-warm retained memory increased by about 20 KB for the port case, 49 KB for ABNF, 199 KB for escaped URIs, 339 KB for varied URIs, 45 KB for XML and 163 KB for IPv6; process peaks were unchanged or up to 2 MiB higher. These process measurements include lazy class/runtime initialization as well as table storage. The cold IPv6 parse took 18.685 ms optimized versus 9.500 ms without optimization; its preparation also increased in this run. The JSON report retains these cold and preparation measurements. Prefer explicit frozen-grammar reuse for these warm gains, or disable optimization when a single parse and minimum retention matter more. Non-ASCII lookahead, active spacing and work/depth limits use the ordinary recognition path.

Small facades and URI grammar (2026-10-16)

Owned built-in facade graphs now retain one prepared FrozenGrammar per immutable expression. Deferred rules, Definition implementations and custom classes without an explicit reuse capability prepare freshly. No recognition results or semantic/effect state are shared by this cache. Session setup also reuses immutable default limits, recursion tracking uses integer parser/position keys, default diagnostic labels are cached, and plain leaf execution state avoids empty child filtering. Already-frozen tiny parsers showed no substantial, consistent isolated speedup from these smaller changes.

The URI grammar uses cheaper lexical implementations for unreserved, scheme, h16 and dec_octet, with the original combinator rules preserved in comments. The accompanying RFC fixes cover IPv6 compression, hexadecimal case, IPvFuture, generic host ambiguity and standalone path selection. See URI syntax for scope, named rule overrides and the changed lexical CST interiors.

Representative valid inputs, microseconds per parse:

Input and entry point Before (0ea7613) After
alice@example.com, facade 142.43 53.49
+31612345678, facade 30.73 14.72
https://example.com, facade 4805.26 3658.99
https://example.com, frozen 2961.66 1891.01
alice@example.com, frozen 56.35 57.23
+31612345678, frozen 15.85 15.31

Windows/PHP 8.2.10/PCRE 10.40. Each table entry is the median of three worker medians. For each worker/input there are ten warmups and seven batches of ten measured calls. Baseline and current workers alternate for each scenario; the baseline autoloader reads copies of the eight changed implementation files from 0ea7613, with unchanged files shared. Timings include fresh UTF-8 sources, full recognition/CST construction, acceptance assertions and result release. They exclude PHP startup, class loading and initial grammar construction. The facade rows include automatic preparation behavior; frozen rows reuse an explicit snapshot. Email/phone fixtures are intentionally limited application grammars, not comprehensive protocol validation.

The first non-alternating run was visibly affected by host variation, so it was not used for this table. Even the alternating frozen-email worker medians ranged 56.86–111.23 microseconds after the changes, compared with 55.56–58.14 before. Treat small differences as noise, not latency guarantees. URI improvements do not demonstrate transparent caching of mutable references: those remain fresh. The before URI grammar rejected the intended-valid IPv6 fixture; that input is reported as an acceptance mismatch, not included as successful validation.

Raw alternating-worker reports include the fixture driver, source-copy manifest, ranges and reproduction steps. Run php docs/examples/uri.php for a quick acceptance/source-round-trip check.

Sparse execution state and session subjects (2026-10-16)

Successful syntax remains an immutable CST. Mutable collection, value, validation and effect state now lives in a lazily allocated ExecutionState. Once a parent has built its CST and completed guards, it retains only execution children with deferred state. Plain child execution records can be released while their syntax nodes remain available. Selected-child and validation-child relationships stay distinct; semantic errors, cached values and commit transitions are still local to each recognition. Grammars with operations on every node still need per-node state, so savings depend on the grammar.

General Regex and IntegerNumber share a session-owned full subject for sources up to ParseLimits::maxRegexSubjectBytes, 8 MiB by default. A zero cap disables caching; larger sources retain uncached reads. Literal windows do not materialize this subject. Each full-subject attempt still charges its original logical work, including cache hits. See file input for ownership and memory limits.

Local Windows/PHP 8.2.10/PCRE 10.40 measurements:

Workload Before After
64 KiB, one Text child per byte: retained used memory 44.01 MiB 19.00 MiB
Same fixture: median parse 683.00 ms 689.98 ms
256 KiB UTF-8 file, general regex chunks: median parse 1079.29 ms 123.30 ms

The retention probe uses two warmups and seven measured samples, and reports incremental memory_get_usage(false) over a prepared source and grammar. These are retained used bytes, not allocator peaks or OS RSS. It keeps the complete ParseResult live; keeping only the CST uses 19.00 MiB both before and after. The fixture has no per-node semantic/effect registrations. There is no demonstrated parse-speed improvement for that character-node fixture.

The regex benchmark uses seven samples with no explicit warmup, excludes source setup and result release, and includes CST construction. Its allocated process peak remains 6 MiB at PHP's allocation granularity. Literal-window matching on the same file measured 77.44 ms after these changes versus 69.55 ms in the earlier run; the subject cache is a benefit for arbitrary full-subject matching, not a replacement for bounded literal windows. Before/after runs are separate local samples, not isolated attribution or latency guarantees.

Raw reports include ranges and the retention probe source. Reproduce the standard fixture with:

php tools/benchmark.php --source=file-utf8 --case=regex --size=262144 --iterations=7 --mode=recognition
php tools/benchmark.php --source=memory --case=text --size=65536 --iterations=7 --mode=recognition
php docs/examples/file-input.php

Inner-loop preparation and accounting (2026-10-16)

The runtime now batches literal work charges while preserving partial work counts and exact byte offsets when a budget is exhausted. Case-sensitive literal failures use a byte-prefix comparison that retreats to a complete UTF-8 unit. ASCII case-insensitive successes have a bulk path; other case-insensitive matches retain per-unit folding with cached literal units. Set membership, regex patterns/windows and range bounds cache preparation separately for source mode and effective case policy. Validation remains lazy at recognition time.

Checked delegation reuses valid immutable spans, unmapped semantic projections are created on first value access, and Repeat shares its child list until trivia requires a distinct concrete list. Custom-parser boundary checks, diagnostics, tree shape, cached values and explicit commit behavior remain in place.

Windows/PHP 8.2.10/PCRE 10.40, two warmups and nine samples per case:

Workload Before (06c7328) After
64 KiB exact literal, UTF-8 memory 9.409 ms 0.008 ms
64 KiB literal, first-character failure 0.017 ms 0.011 ms
64 KiB literal, last-character failure 99.613 ms 0.320 ms
64 KiB ASCII literal, case-insensitive success 138.033 ms 0.035 ms
64 KiB, 64-unit Text children 17.278 ms 6.844 ms
64 KiB, 64-unit literal-window regex children 18.676 ms 8.272 ms
1,024 units, 1,024-member Set, member last 200.380 ms 7.911 ms

These combined changes were measured in separate runs on successive days, not isolated per-optimization experiments. Earlier repeat-heavy timings varied substantially, and tiny exact-match timings are especially sensitive to PHP string sharing: that fixture gives the source and terminal the same backing string. The literal/folding cases contain ASCII through a UTF-8 source; the Set contains unique Unicode members. Cold preparation and source setup are outside timing. Literal/repeat timings include result release; Set timings exclude it. No file or non-ASCII folding speedup is inferred from this table.

Raw before/after data records ranges, runtime, timings and commands. The historical sweep checked acceptance and late-literal failure positions, reported literal/repeat and Set sweeps, and included an accounting-only microbenchmark. The specialized benchmark has been removed. The accounting-only microbenchmark is not a parsing speedup or an exhausted-budget equivalence test. Regression tests cover partial budgets, Unicode mismatch boundaries, byte/text cache separation, policy changes, invalid UTF-8, frozen graphs and retained trivia/effect order.

Literal and callback allocation improvements (2026-10-15)

On Windows, PHP 8.2.10 and PCRE 10.40, seven samples per case produced the following local medians. Source setup is excluded; recognition includes the CST. The before measurements use revision 7c0f17b; after measurements include the literal preparation/comparison and empty callback registration optimizations.

Workload Before After Allocated peak before / after
2 KiB UTF-8 memory, one Text per byte 19.91 ms 15.34 ms 10 / 4 MiB
64 KiB UTF-8 memory, 64-byte Text chunks 108.82 ms 18.01 ms 6 / 4 MiB
2 KiB UTF-8 file, 64-byte Text chunks 8.49 ms 0.67 ms 4 / 4 MiB

These are combined changes, not isolated attribution of either optimization. Text caches its immutable literal units separately for byte and UTF-8 sources. Successful case-sensitive literals use one bounded byte comparison; mismatch and case-insensitive recognition retain the unit path. Work charges and limit error positions remain per unit. Empty local collection/effect registrations no longer allocate closures, while registered callbacks still bind to each recognition.

The new text-chunked benchmark chooses runs of 1–64 a characters instead of one node per character. At 64 KiB in UTF-8 memory, it took 9.58 ms and 4 MiB peak, versus 557.96 ms and 62 MiB for the optimized character-node grammar. The latter had a 536.48–2085.63 ms range, so its timing remains noisy. Before these changes, the character-node case exhausted the default 128 MiB limit.

Coarser terminals are an explicit application grammar choice: they change CST children and diagnostic expectations. The runtime never merges existing nodes. Use them only when individual characters do not need names, guards, mappings or effects. The runnable example checks short and partial final runs, malformed input, node counts and CST/AST source round trips:

php docs/examples/chunked-text.php
php tools/benchmark.php --case=text-chunked --mode=recognition --iterations=7 --size=65536

This example uses general regex over in-memory input. Its bounded match length does not bound file subject reads; arbitrary regex still uses the complete source. The acceptance language matches nonempty repeated Text('a'), but the CST shape and mismatch expectations deliberately differ. Root-only AST and byte-length semantic projections in the harness are inexpensive and do not model rich outputs.

Raw before/after reports retain commands, ranges, peaks and source setup times. Samples share a worker process, have no explicit warmup and include allocator retention and fixture storage. These results are workload-specific baselines, not latency guarantees.

Earlier file-window comparisons

Case and source Median parse + CST PHP process peak
2 KiB text, in-memory, pre-change 19.28 ms 8 MiB
2 KiB text, in-memory, phase 4 21.91 ms 8 MiB
64 KiB UTF-8 file, full-subject regex chunks 63.47 ms 6 MiB
64 KiB UTF-8 file, bounded literal regex chunks 20.88 ms 6 MiB
1 MiB UTF-8 file, bounded literal regex chunks 406.75 ms 52 MiB
1 MiB UTF-8 memory, bounded literal regex chunks 293.17 ms 52 MiB

The 64 KiB window comparison recognizes the same 64-byte literal chunks, with the same node counts and output mode. It demonstrates reduced file rereads for this workload; the full-subject version reads the entire file for every chunk. There was no visible peak-memory difference at PHP's allocation granularity. At 2 KiB in memory, windowed matching was slower (0.66 ms versus 0.37 ms), so it is an explicit facility rather than a blanket replacement. Source setup for the 1 MiB file was 128.80 ms versus 5.74 ms in memory, excluded from parse time. Budget bookkeeping and I/O have costs; no overall speedup is claimed.

Artifacts: pre-change baseline, UTF-8 memory, byte memory, byte file, UTF-8 file, large text file, large general regex, large literal window, megabyte file, megabyte memory.

Memoization decision

Memoization is deferred. The historical shared-prefix profile recorded seven parser entries, one repeated parser/byte position and 4105 work units. The lightweight ParseMetrics still omits scope context. That count is a lead, not a cache benefit. The opt-in instrumentation below now separates weaker policy repetitions from stricter context repetitions and measures their actual logical work. No result cache, semantic-state replay or speculative effects are introduced.

If pursued, keys must include frozen parser identity, source/session identity, byte position, case/choice/trivia/lexeme policy and relevant guard dependencies. Only pure guards and frozen graphs qualify, and selected execution records/value caches/effects cannot be replayed as shared mutable cache entries. A bounded cache with explicit file/history eviction and cached/uncached tests would be a separate increment. Parser/byte-position counts alone do not justify those risks.

Selective memoization instrumentation (2026-10-23)

Runtime\MemoizationProfile can observe existing frozen entry points:

use Vanderlee\Comprehend\Runtime\MemoizationProfile;

$frozen = $grammar->freeze();
$profile = new MemoizationProfile(capacity: 4096, maxDepth: 128, maxContextBytes: 16384);
$result = $frozen->parseAll($source, memoizationProfile: $profile);
$counts = $profile->snapshot();

FrozenGrammar::parse() and matchPrefix() accept the same optional argument. They register the already isolated, resolved and safety-checked graph, then use the ordinary Parser/ParseSession recognition path. Object Parser entry points and ParseSession can receive the observer too, but do not automatically qualify a mutable graph: unregistered nodes supply entry/position counts, not reuse candidates. Facades can freeze first. ParseMetrics remains the cheaper observer with its existing snapshot schema. The default recognition path creates no profile, context key or outcome summary.

Eligibility uses an exact-class allowlist of inspected built-ins and optimizer implementations. Executable metadata, guards, callbacks, unknown/custom classes, recursive dependency cycles and unsupported descendants disqualify a subtree. Inspected children can still be eligible when their parent carries metadata. Finite work/depth limits and active automatic spacing exclude reuse candidates; they continue to be observed for total work and exceptions. Guard dependencies are deliberately not guessed. Optimizer-internal recognizers outside the frozen graph are unregistered, while bypassed lexical-table attempts generate no checked entries. Neither absence nor repeated parser offsets establish cache usefulness.

Every position key includes monotonic weak identities for session, source and parser, plus byte offset. Source identity also distinguishes its mode and stable snapshot. Policy keys add effective case/choice, trivia identity, lexeme depth and failure-diagnostic retention requirements. Strict keys additionally capture caller rule paths, active ancestry identities/offsets, recursion membership, depth, full policy stacks, active diagnostics and failure revision, pending diagnostic identity, regex subject-cache presence and resource policy. Pending diagnostics are not materialized. Actual caller identities prevent recycled PHP object IDs from aliasing observations. Comparisons never cross sessions, even with the same Source.

policy_repeats and strict_repeats count completed visits to existing keys. *_summary_matches further requires the same acceptance, byte progress, farthest failure, root kind and immediate child count. These bounded summaries do not compare complete trees or prove diagnostic-recipe equivalence. Policy summary conflicts can be expected when caller diagnostics differ; strict conflicts would invalidate that partition. Matching summaries still are not cache hits. *_repeated_work_inclusive measures matching-summary repeated calls including descendants. *_repeated_work_nonoverlap removes nested overlap; per-class inclusive totals must not be added to estimate savings. Work units are cooperative runtime charges, not CPU time, allocated bytes or guaranteed removable work.

Profiles retain bounded scalar keys, summaries and class totals, never a match, tree, Source, effect or semantic projection. Weak identities and registered-node records do not keep their objects alive. Each position/context table and graph classification is capped by capacity; identity bookkeeping is capped at three times that capacity. Frames are bounded per active session by maxDepth and are removed on completion or exception. Context serialization is size-checked before allocation; oversized policy stacks are not expanded. Bounds and all dropped observation counters appear in the report. Saturation, conservative eligibility and context-size truncation can hide candidates, so zero is not proof of absence.

The maintained harness uses fresh workers, 256-byte synthetic prefixes, eight lookahead probes, ASCII application fixtures, byte Sources and separate cold/warm observations. Three workers per case/flag confirmed these warm counts on Windows, PHP 8.2.10, with PCOV disabled. All counts below are plain / optimized frozen graphs:

Fixture Entries Repeated positions Policy repeats Strict repeats Nonoverlapping strict repeated work
Shared prefix, two alternatives 7 / 6 1 / 0 1 / 0 0 / 0 0 / 0
Eight repeated long lookaheads 34 / 34 28 / 28 28 / 21 28 / 21 1,827 / 3,633
Short lexical alternatives 7 / 1 0 / 0 0 / 0 0 / 0 0 / 0
Different case scopes 9 / 9 1 / 0 0 / 0 0 / 0 0 / 0
Guarded lookahead prefix 34 / 34 28 / 28 7 / 7 7 / 7 14 / 14
URI with repeated escapes 1,179 / 49 15 / 0 15 / 0 0 / 0 0 / 0
JSON array of objects 651 / 603 18 / 18 18 / 18 0 / 0 0 / 0
ABNF quoted text 650 / 7 0 / 0 0 / 0 0 / 0 0 / 0

The plain lookahead's inclusive repeated work is 5,460, but its nonoverlapping total is 1,827 of 2,347 observed work units. Optimized nested literal failure fallback adds speculative comparisons: total work is 4,411 and nonoverlapping repeated work is 3,633. Some internal compiled terminals are unregistered, so their position repetitions are not eligible candidates themselves. The guarded variant excludes the expensive guarded prefix and its parents; only seven cheap suffix failures, two units each, qualify. This demonstrates why raw repetition counts overstate selective opportunities.

Shared-prefix and ordinary URI policy keys also expose inherited-diagnostic summary conflicts; their matching parser positions are not interchangeable execution records. JSON's default 16 KiB context cap drops some observations. A separate 128 KiB run eliminates those context-size drops and still finds zero strict repetitions in this sample. UTF-8 file workers exercise the same fixtures against 64-byte pages, and validate outputs against an independent unprofiled graph. These are small samples with conservative keys: relative diagnostic/CST recipes might allow broader reuse, but this profile does not prove that equivalence.

Instrumentation traverses graph eligibility, hashes scoped arrays and observes outcomes; it is intentionally expensive. Its elapsed time is profiler overhead, not evidence of a parsing optimization. The harness records collector/grammar-cache used-memory deltas, recognition peaks before correctness projection, cold/warm counts, and separate unprofiled recognition timings. Warm used-memory deltas in the byte-source sample ranged from 6,232 bytes for optimized short lexical choice to 1,567,000 bytes for ordinary URI recognition. These retain the collector and any new grammar caches, not a hypothetical result cache. Registration adds graph inspection only when a profile is supplied; context hashing is per observed eligible visit. Bounded records avoid unbounded input-position history, but do not make instrumentation free or establish a suitable production cache size. A five-worker alternating original/current comparison with profiling disabled measured ABNF worker medians of 4.439–4.774 ms before and 4.305–5.010 ms after; medians of worker medians were 4.488 and 4.409 ms. Ranges overlap: no consistent default throughput improvement or regression is established. The control loads Parser and ParseSession from revision 4e8502f; host activity was uncontrolled.

Reproduce:

php docs/examples/memoization-profile.php
php -d pcov.enabled=0 tools/profile-memoization.php --size=256 --workers=3
php -d pcov.enabled=0 tools/profile-memoization.php --source=file-utf8 --size=256
php -d pcov.enabled=0 tools/profile-memoization.php --case=json --context-bytes=131072
php -d pcov.enabled=0 tools/profile-memoization.php --no-profile --case=abnf --flags=0

Optional --capacity, --case, --flags=0|1, --iterations and --source=memory-byte|memory-utf8|file-utf8 control the experiment. To compare the previous disabled path, save that revision's src/Parser/Parser.php and src/Runtime/ParseSession.php under build/memoization-profile/ and add --baseline-runtime=build/memoization-profile to the --no-profile command. Raw local reports are retained under that ignored directory while available. Regressions compare profiled/plain CST/AST, values, ordered diagnostics, work, guard/effect lifecycle, source release, deferred diagnostics and exact exhausted limits. The runnable example checks a repeated lookahead at a nonzero byte offset.

The next justified cache experiment would target a bounded, expensive pure lookahead with frequent context matches, comparing uninstrumented cached/uncached workers. It must prove full relative diagnostics and tree reconstruction, fresh semantic/effect ownership and exact work/limit behavior. Reject it if broader context checks erase the hits, bounded recipes/lookup cost exceed recognition, or memory and preparation outweigh reuse. Current URI/ABNF/JSON samples supply no strict repeated-work evidence for a general cache; that is not a proof that all relative-context memoization is unhelpful. No cache is enabled by this instrumentation.

Batched literal repetition

Frozen optimized grammars batch repetitions of plain, case-sensitive ASCII Text operands between 2 and 8192 bytes. For example, Repeat::exact(new Text('ab'), 32768)->freeze() compares bounded blocks instead of entering the terminal for every pair. Each block contains complete operands and occupies at most 8192 bytes. Preparation retains one block per eligible repetition; recognition does not materialize a complete file subject.

The original per-operand CST children and byte spans remain intact. Tree memory therefore still grows with the number of matches. Operand metadata and callbacks, active trivia, case-insensitive sessions, custom sources and finite work/depth limits retain ordinary recognition. Minimum failures also use that path for exact progress and diagnostics. Successful batches account for original terminal entry and byte work; a terminating failed operand runs normally with its diagnostics restored after the minimum has been met. Single-unit repetitions retain their existing scanner optimization.

Reproduce with coverage disabled in fresh workers:

php tools/benchmark-repeat-batching.php --iterations=9
php tools/benchmark-repeat-batching.php --iterations=9 --source=file

The harness alternates three fresh workers per variant when given --baseline=path/to/Repeat.before.php, a saved copy of the previous extensions/Optimizer/Parser/Repeat.php. It excludes source setup, grammar preparation, two warmups, correctness checks and release from parse timing. Every observation checks acceptance, value and a CST hash against unoptimized recognition. Used-PHP retained bytes are measured before value projection.

On Windows/PHP 8.2.10, PCOV disabled, nine observations per worker, medians of worker medians for 64 KiB of repeated ab were 139.811 to 19.146 ms in UTF-8 memory and 160.822 to 19.175 ms from a UTF-8 file. Both variants retained 9,966,360 used-PHP bytes after recognition. These are approximately 7.3x and 8.4x throughput improvements on this fixture, not reductions in CST memory or general application guarantees. Host load was uncontrolled. The saved baseline and raw reports are locally retained under ignored build/repeat-batching/.

Direct spans for one-byte repetitions

After a prepared repetition scans a run successfully, GraphOptimizer::scanRepeat() builds its original per-unit CST children directly for exact built-in byte sources. For exact built-in UTF-8 sources, it also does so for case-sensitive ASCII literals, included characters, included ASCII sets and included ranges with an ASCII upper bound. These runs have known one-byte intervals, so constructing each retained span does not need another readUnit() call to rediscover its end.

Custom sources and file-source subclasses retain their unit-access hooks. Other eligible UTF-8 runs use the bounded boundary decoding described below, including case-insensitive runs: Unicode case folding can match a multibyte character against an ASCII operand. An explicit local case-sensitive policy can still qualify inside a case-insensitive scope. Finite work/depth limits, active spacing and operand metadata keep their existing eligibility and fallbacks. Recognition work accounting is unchanged; tree memory still grows with matched units.

The repetition benchmark accepts --literal (default ab) to measure this path:

php tools/benchmark-repeat-batching.php --literal=a --iterations=9
php tools/benchmark-repeat-batching.php --literal=a --iterations=9 --source=file

For before/after comparisons, save the previous extensions/Optimizer/GraphOptimizer.php and pass --baseline=path/to/GraphOptimizer.before.php. The harness alternates three fresh workers per variant, with two warmups and nine observations per worker. Source setup, graph preparation, release and correctness projection are outside parse timing; every observation checks acceptance, value and a CST hash against unoptimized parsing.

On Windows/PHP 8.2.10 with PCOV disabled and Xdebug unloaded, medians of worker medians for 32 KiB of repeated a were:

UTF-8 source Before (ms) After (ms)
Memory 70.649 56.785
File 32.294 17.798

These are approximately 20% and 45% lower parse times on this fixture. Both variants retained 9,966,360 used-PHP bytes. Host load was uncontrolled; these figures do not establish general application speedups. Separate Xdebug 3.5.3 captures of five optimized parses of 2,048 ASCII literals showed 10,240 readCodePoint() calls before and none after. Instrumented elapsed times are not used for the throughput comparison. The saved baseline and raw reports are locally retained under ignored build/one-byte-spans/ while available.

Regression tests compare original and optimized trees, ASTs, values and diagnostics across memory/file byte and UTF-8 sources, nonzero offsets, small pages, empty and bounded repetitions, minimum failures and Unicode case policies. They also check custom file-source hooks, arbitrary octet children and unchanged work during child construction. Existing limit and lifecycle tests cover the retained ordinary paths.

Bounded UTF-8 repetition boundaries

After an eligible repetition has recognized its byte extent, exact built-in Utf8Source and Utf8FileSource inputs decode remaining unit boundaries in chunks of at most 8192 bytes. Each chunk end retreats by at most three bytes to a code-point boundary. The sources already guarantee valid UTF-8, so the loop can determine widths from lead bytes and construct every original CST leaf and span directly. It releases each chunk before reading the next; there is no input-sized boundary index or additional full-file subject for reconstruction. The scanner's existing source/regex-subject retention rules still apply.

Known one-byte runs keep their direct-span path. Custom sources keep readUnit() hooks. Scanner eligibility, minimum-failure fallback, finite work/depth limits, spacing, metadata, diagnostics and lazy semantics are unchanged. Boundary reconstruction adds no recognition work. Retained tree memory remains proportional to the number of matches; bounded decoding does not make the parser tree-free.

The repetition benchmark's --any option repeats Any over the generated UTF-8 fixture instead of matching each complete literal as a unit:

php tools/benchmark-repeat-batching.php --any --literal='aé€🙂' --count=8192 --iterations=9
php tools/benchmark-repeat-batching.php --any --literal='aé€🙂' --count=8192 --iterations=9 --source=file

For before/after comparisons, save the previous GraphOptimizer.php and pass --baseline=path/to/GraphOptimizer.before.php. On Windows/PHP 8.2.10, with PCOV disabled and Xdebug unloaded, three fresh workers per variant alternated order, with two warmups and nine observations each. Medians of worker medians for 80 KiB of mixed-width input (32,768 retained units) were:

UTF-8 source Before (ms) After (ms)
Memory 32.312 22.866
File 33.269 22.869

These are approximately 29% and 31% lower parse times on this fixture. Both variants retained 9,966,360 used-PHP bytes. Source setup, preparation, release and correctness projection are outside timing. Every observation checks acceptance, value and a CST hash against unoptimized recognition. Host load was uncontrolled; these measurements do not establish general application speedups. A separate Xdebug 3.5.3 capture of three parses, each containing 4,000 mixed-width units, counted 12,000 Utf8Source::readUnit() calls before and none after. Instrumented elapsed times are not used for throughput comparisons. Saved baselines and raw reports are locally retained under ignored build/utf8-repeat-chunks/.

Regressions compare full CST fingerprints, ASTs, diagnostics and independent code-point intervals for memory and file sources, nonzero offsets, four-byte file pages, two-/three-/four-byte chunk splits, exact chunk ends, empty input and bounded matches. They also verify custom UTF-8 source hooks and unchanged work accounting.

List spacing scope bookkeeping

ParseSession now checks and updates its private lexeme-depth counter directly inside automatic spacing recognition, including the prepared syntax-only path used by lists. This removes forwarding through isLexeme(), pushLexeme() and popLexeme() for each trivia attempt. The final session still owns the counter; the increment is paired with a decrement in finally, including exceptions from custom wrapper creation and recognition. Public scope operations, nested lexemes, spacing policies, wrapper caching and list rollback behavior are unchanged.

A source-method trace of the nested 128-ID application fixture with scanner spacing counted 272 calls each to those three methods before, and none after: 816 fewer method entries. Both variants consumed 57,188 work units and retained the same 820-node CST hash. This count covers instrumented core methods; extension methods are not instrumented by profile-extraction.php.

Three fresh Windows/PHP 8.2.10 workers per variant alternated before/after order, with two warmups and 15 observations, PCOV disabled and Xdebug unloaded. Medians of worker medians for recognition/CST were 3.798 versus 3.735 ms with scanner spacing and 4.517 versus 4.442 ms with repeated-character spacing. Worker ranges overlap, and uncontrolled host load establishes no consistent throughput gain. Retained used-PHP memory was unchanged at 388,696 and 389,704 bytes respectively. Preparation, source setup, correctness checks, values and result release are outside recognition timing. Traced elapsed times are not throughput measurements.

Reproduce current measurements and method counts with:

php -d pcov.enabled=0 tools/profile-extraction.php --case=nested --size=128 --iterations=15 --scan-spacing
php -d pcov.enabled=0 tools/profile-extraction.php --case=nested --size=128 --iterations=1 --scan-spacing --trace

Omit --scan-spacing for repeated-character spacing. To compare a saved baseline, place its SpacingContextTrait.php under Runtime/Scope/ in a source directory and add --source-dir=that-directory. The saved baseline and raw reports are locally retained under ignored build/list-overhead/. Regression tests exercise nested suppression, explicit null policies, custom wrappers, exception restoration, file sources, rejected list pairs, effects and exact exhausted budgets. No spacing result cache or probe elimination is introduced.

Lifetime and memory guarantees

  • File source copying and validation use bounded chunks; random reads retain one configured page. The stable snapshot occupies O(input size) temporary disk.
  • Requested byte slices allocate their requested size. General Regex and Integer allocate complete subjects, shared within the configured session cache cap; bounded literal regex reads a finite safe window.
  • Every accepted parse retains its CST and deferred execution/semantic/effect records where needed. Plain execution children can be released after tree construction. Tree memory grows with matched nodes; the earlier 1 MiB example's 52 MiB peak demonstrates why this is not constant-memory streaming.
  • ASTs, arrays, JSON/XML, raw text and semantic values add output-dependent retention. Diagnostics scan their prefix and current line lazily, retaining constant position metadata; a requested long line/prefix can still be large.
  • Results and spans retain their source and snapshot lifetime. Explicit close invalidates later data reads. See File input for ownership.
  • Runtime budgets count cooperative recognition work/depth, not wall-clock CPU, source construction or arbitrary semantic/effect callbacks.

Optional CST retention is deferred: ASTs, raw source reconstruction, serialization, validation and semantic callbacks currently rely on that tree contract. A recognition-only mode needs its own output/diagnostic contract before removing nodes. The benchmark does not label the existing runtime as tree-free parsing.