Seekable file input and runtime limits¶
ByteFileSource and Utf8FileSource implement the same Source interface as
in-memory sources. They accept a readable path or borrowed seekable stream:
use Vanderlee\Comprehend\Source\Utf8FileSource;
use Vanderlee\Comprehend\Parser\Terminal\Regex;
use Vanderlee\Comprehend\Runtime\ParseLimits;
$source = new Utf8FileSource('input.txt');
$result = Regex::literalWindow('hello')->matchPrefix($source,
limits: new ParseLimits(maxDepth: 100, maxWork: 100000));
$result->rawText();
// Close only after finishing every result, span, diagnostic and projection read.
$source->close();
Ownership and stability¶
Construction copies input to a private temporary disk snapshot in bounded chunks. A path's input handle is opened/closed by the source. A borrowed stream remains open and its position is restored, even if validation fails. Input must be seekable/readable and stable during copying; nonseekable streams are unsupported. After construction, edits or closure of the original cannot alter source spans.
The source owns its snapshot until explicit idempotent close() or destruction
after the last source/result/span reference is released. Sources cannot be
cloned. Closing invalidates data reads, including retained result raw text,
serialization, AST text, diagnostics and code-point conversion; cached metadata
such as byte length remains available. Releasing a local source variable does
not close a snapshot still retained by a result. Do not close it before committing
effects that read source text. Temporary disk use is proportional to input size.
Random byte slicing uses one read page (8192 bytes by default; minimum 4).
sliceBytes() returns exactly the requested interval and can allocate a large
string if explicitly asked. Bytes are never normalized or stripped. Byte sources
accept arbitrary octets; UTF-8 files validate and count bounded complete-code-point
chunks with native mbstring. A one-byte peek and at most three retreats keep valid
code points together. Rejected chunks use scalar validation to preserve exact
invalid-byte diagnostics for overlong sequences, surrogates, out-of-range code
points and truncation. Reads crossing page/validation boundaries
preserve complete code points. UTF-8 position conversion scans units without a
full code-point index, trading bounded auxiliary storage for linear scan time.
Bounded scanners¶
Case-sensitive readUntil() with a literal delimiter, readWhileSet() and
readWhileRange() accelerate exact built-in byte/UTF-8 file sources using
8192-byte scan chunks. Subjects include bounded delimiter lookahead and complete
UTF-8 endpoints; they never materialize the complete file or use the regex subject
cache. Temporary subject storage stays bounded independently of file size.
The scanners retain the same byte spans, CST, diagnostics and logical work.
Finite work budgets, predicates, case-insensitive scans, custom sources/scanners, maximum counts above 65535 and oversized generated patterns retain unit scanning. PCRE failures resume unit scanning at the current chunk without replaying work already charged. Snapshot ownership and explicit close behavior are unchanged. This acceleration applies to these scanners, not to arbitrary regex patterns. See performance measurements for reproduction.
Other character encodings¶
EncodedFileSource(pathOrStream, encoding, pageSize = 8192) validates an explicit
mbstring character encoding and converts a stable copy into a UTF-8 disk snapshot.
Aliases such as SJIS are accepted; unknown encodings and transfer encodings such
as Base64 are rejected. Malformed encoded input throws rather than substituting
replacement characters. There is no encoding detection. BOM interpretation
follows the selected mbstring encoding; use explicit endian encodings as needed.
All spans, byte offsets, diagnostics and rawText() refer to normalized UTF-8.
inputEncoding reports the selected canonical encoding and originalByteLength
reports the original file size. Original-file byte offsets are not mapped.
Borrowed stream position/ownership, snapshot stability and close semantics match
the other file sources. All source-bound spans retain the EncodedFileSource itself.
Transcoding currently validates and allocates the complete original input plus the UTF-8 output in memory during construction, supporting stateful encodings without unsafe chunk boundaries. Subsequent reads use bounded pages. This is not constant-memory decoding; ParseLimits apply to parsing, not transcoding.
use Vanderlee\Comprehend\Comprehend;
use Vanderlee\Comprehend\Source\EncodedFileSource;
$source = new EncodedFileSource('legacy.txt', 'Windows-1252');
$result = Comprehend::text('café')->parse($source);
// Keep the source open until all result, tree and diagnostic reads are finished.
$source->close();
Regex boundaries¶
General Regex uses the complete original subject, preserving offsets, ^, $,
lookbehind and other context-sensitive behavior. IntegerNumber uses a full-subject
regex too. Both share one lazily materialized subject per ParseSession when the
source fits ParseLimits::maxRegexSubjectBytes (8 MiB by default). The cache is
released with the session; ordinary results retain the source snapshot, not the
session's cached string. Explicit source closure still invalidates reads.
Only the final built-in ByteSource and Utf8Source classes skip the repeated
readability probe, because their immutable memory buffers cannot be closed.
Exact built-in file sources check readability with an empty byte slice, which
does not load page zero or evict the active read page. Custom sources, including
ByteFileSource subclasses, retain their boundary probe on every subject request.
Set new ParseLimits(maxRegexSubjectBytes: 0) to disable caching, or set a
nonnegative byte cap appropriate to the application. Sources above the cap use
uncached full-subject reads. This cap bounds retained cache storage, not the
temporary memory needed by a single regex match. Work still charges the complete
subject on every attempt, even when cached. Arbitrary regex is not bounded-memory
matching; choose literal windows when bounded subject reads are required.
Regex::literalWindow(string) is an explicit opt-in for quoted literals only.
It escapes metacharacters, anchors at the supplied start and reads a proven
finite window: literal byte length in byte mode, at most four bytes per literal
code point in UTF-8 mode. The latter permits Unicode caseless variants with
different byte lengths. Window ends stay on code-point boundaries. It preserves
literal prefix acceptance at nonzero offsets without pretending that arbitrary
regex patterns can be truncated safely. There is no generic window setting;
unsupported patterns keep the complete-subject path.
Per-session budgets¶
Parser and FrozenGrammar matchPrefix()/parseAll() accept an optional third
ParseLimits, also available as named limits:. maxDepth and maxWork are
positive integers or null; null disables that limit. Each public parse creates
fresh counters. Depth counts simultaneously active parser recognitions. Work
counts parser entries, source-unit requests through ParseSession, regex subject
bytes and each digit inspected during IntegerNumber conversion and range trimming.
Failed/speculative attempts count;
diagnostic suppression never refunds work. Full-subject regex work is charged
before reading the subject, so a small budget can reject a large allocation.
Exhaustion throws ParseLimitException with limit, maximum, canonical
byteOffset and rulePath. It is distinct from syntax rejection and produces no
committable root. Parser scopes/active entries restore on exceptions; work is not
reset inside a reused session. Limits do not bound source construction, semantic
callbacks, output projection or arbitrary native PCRE CPU work. PCRE's own
limits still apply. Custom recognizers should charge long loops through
consumeWork() and use readSourceUnit(); uncooperative user PHP code cannot be
preempted. Limits are cooperative execution bounds, not wall-clock deadlines.
ParseMetrics is optional bounded profiling via an explicitly supplied
ParseSession. It records entries and repeated parser/byte positions, keeping at
most 4096 positions by default. Scope context is intentionally omitted: repeats
are investigation candidates, not valid memoization keys. No recognition-result
cache is introduced. See Performance measurements for the
recorded memoization decision and retention tradeoffs.
Run the checked file example with
php docs/examples/file-input.php.