FTS Performance: Reproducible Measurement

Before this revision, the course quoted figures from another machine (Apple M1 Max, Node 24.6), which the reader could not verify. A number without a reproduction method is not a fact, but a claim on faith. Further — harness examples/fts/benchmark/, with which you measure your own milliseconds on your own machine, rather than copying others into notes.

Minimal example

The compiler from static/js/vendor/fts/browser.js — is what measures the harness, and is what is executed in the course sandbox. The small model below compiles and passes its own examples:

категория «Тарификация»

  объект Заявка
    объём является числом
    «премиум клиент» является признаком

  утилита «Рассчитать стоимость»
    принимает Заявка
    возвращает деньги
    начинает с 100

    правило «Крупный объём»
      если объём не меньше 50
      то добавить 40

    правило «Премиум скидка»
      если «премиум клиент» равен да
      то добавить -20

    свойство «Стоимость не ниже минимального тарифа»
      результат не меньше 80

    пример «Стандартная заявка»
      дано объём равен 10
      дано «премиум клиент» равен нет
      ожидается результат равен 100

    пример «Крупная премиум заявка»
      дано объём равен 80
      дано «премиум клиент» равен да
      ожидается результат равен 120

compile parses such text into a document, validate checks links and types, executeUtility/testUtilities execute the rules, generateTypeScript prints the .ts-implementation and test. It is exactly these four operations that the harness measures.

What the compiler does

Walkthrough — not parsing in a vacuum: natural Russian syntax (natural-parser.js) unfolds into the same document as the explicit JSON document. Validate checks that structures, fields, and functors exist and types are consistent, but does not execute rules. Execute goes through the list of utility rules and computes the result for a specific input. Generate prints the ready TypeScript and test on node:test — without accessing the disk and without tsc.

All four operations are pure JS code on an already parsed or not yet parsed string. None of them touch the network, file system, build formats, or child processes.

Measurement methodology

measure.mjs — not a one-time console.time. Three solutions explain why:

• Warm-up. Before the measurement, 5 dry runs are executed — JIT is not yet warmed up, and the first calls are systematically slower than those seen by a real process that has been running for a while.
• Batches with a minimum time of 2 ms. A single call to compile on a model with 10 fields takes tenths of a millisecond — on this scale, the timer’s resolution and the function call overhead become comparable to the measured quantity. The harness selects the batch size so that the total time of one measurement is no less than 2 ms, and divides it by the batch size.
• Median, not average. For a series of batches, the harness sorts the results and takes the median. The average is pulled by a single outlier — a GC pause, JIT deoptimization, a neighboring process that stole a CPU core. The median is stable against such noise. min/max/p95 in the report show the spread: if p95 is far from the median, the measurement is unstable and a single number is not reliable.

Models are generated programmatically, with 10, 100 and 1000 fields/rules — the same way as in upstream scripts/benchmark.mjs: deterministic text, without manual .fts files, meaning anyone can generate exactly the same model on their own.

Transpilation of the generated TypeScript is out of the measurement scope: the typescript package is not present in this repository — it is a learning repository, not a production build of FTS. The harness does not fake the digit and does not omit the skip: in the absence of the package, it prints an explicit string transpile: { measured: false, note: "..." }. If typescript is available in your environment, the harness will connect it through a dynamic import and measure the step separately.

Your digits for one team

node examples/fts/benchmark/measure.mjs

Measured output on the machine where this module was built (node v24.18.0, linux/x64, AMD EPYC 9354 32-Core Processor, 8 cores), median in milliseconds:

At scale 1000 compile + validate together take about 7.01 ms. A full JSON report with mean/min/max/p95 for each operation — in examples/fts/benchmark/baseline-linux-x64-node24.json, captured by the same team with the flag --out.

Growing a model from scale 10 to scale 1000 (100 times) increases compile approximately 88 times, validate — approximately 81 times: both operations grow almost linearly with the model size, slightly slower than it. Execute grows differently: from 0.0015 to 0.018 ms, that is, 12 times with a 100-fold increase in the number of rules — because the loop over rules is cheap by itself, each additional rule costs a fraction of a microsecond. Even with 1000 matched rules, execute remains two orders of magnitude cheaper than compile of the same scale. Parsing syntax and building document structures costs orders of magnitude more than traversing an already prepared list of rules — and this is exactly why “the model grew” almost always means “compile became more expensive”, not “execution became more expensive”.

Comparison with baseline

node examples/fts/benchmark/compare.mjs my.json examples/fts/benchmark/baseline-linux-x64-node24.json

compare.mjs prints the median deviation in percentages operation by operation. Comparing the baseline from another machine literally is not possible — absolute milliseconds do not transfer between different hardware, the tool prints an explicit warning about this. Below — the actual output of comparing the measurement from this machine and a foreign baseline from upstream (Apple M1 Max, Node 24.6.0, taken from benchmarks/baseline-darwin-arm64-node24.json upstream FTS, not measured here):

ВНИМАНИЕ: baseline снят на другом железе или ОС. Абсолютные миллисекунды не сравнимы —
смотрите на то, как растёт median между scale 10/100/1000, а не на разницу процентов ниже.

operation             scale   current_ms   baseline_ms   diff_%
compile               10      0.0705       0.0485        +45.4%
compile               100     0.5743       0.3742        +53.5%
compile               1000    6.1858       3.8279        +61.6%
validate              1000    0.8217       0.9477        -13.3%
execute               1000    0.018        0.0146        +23.3%
generate_typescript   1000    0.5771       0.3442        +67.7%

Different hardware gives different percentages for each operation — this is expected and not a reason to panic. A useful signal — compare today’s report with yesterday’s on your machine: the same CPU, the same Node, the same method. If compile on scale 1000 suddenly triples on the same hardware — this is a regression, not noise.

What these digits do not mean

Harness measures clean runtime FTS in Node.js — four functions over an already loaded string in memory. It does not measure:

• build Vite, webpack or esbuild around the generated code;
• full tsc with type graph of the whole project, incremental cache, plugins and sourcemaps — instead this transpilation step is honestly marked as skipped;
• network, disk, Node process start — the measurement itself starts already inside a warmed-up process;
• behavior under concurrent load — the harness is single-threaded and sequential, while the prod-service can compile models in parallel.

It is also appropriate to mention the order of magnitudes here. A cold rebuild of a frontend project on Vite or webpack usually takes from hundreds of milliseconds to several seconds; type checking the entire project via tsc — seconds. Compile + validate + generate for a model with 1000 fields and rules on this machine takes 7.8 ms — three orders of magnitude less. For realistic models (in course projects — dozens of fields, not thousands) the performance of FTS stops being an issue before the reader has time to open the profiler: it drowns in the noise of the rest of the build. It is worth measuring explicitly only when the model is several orders of magnitude larger — 1000 fields or more — or when compile is called on every request, not once at the start of the process.

Measurement in CI

Regression is caught not by an absolute threshold (noisy CI runners on shared CPU give variation, due to which a fixed percentage will either falsely trigger or catch nothing), but by comparing with the previous report on the same runner:

- run: node examples/fts/benchmark/measure.mjs --out current.json
- run: node examples/fts/benchmark/compare.mjs current.json baseline.json
# baseline.json — артефакт предыдущего успешного прогона на master,
# сохранённый через actions/cache или actions/upload-artifact
- run: node examples/fts/benchmark/compare.mjs current.json baseline.json --json > diff.json

Next - a small wrapper script (not included in this harness), which reads diff.json and fails if diff_percent on the required operation and scale exceeds the selected threshold with a margin for runner noise - for example, x2, not x1.1. The point is not in the exact number, but in catching jumps by an order of magnitude, not bothering the team for every percent.

Practice in the sandbox

On the “Check” tab, the workbench also shows time: “Model is valid · walkthrough N ms”. This is not the same measurement that measure.mjs does, and here’s why: in the workbench, this is one call of compile+validate in the reader’s browser, without warming up and without batches — exactly what really happened when you opened the tab. This number is useful as a sense of “right now, on this device, this is fast”, but is not suitable for comparison between runs: one cold call noises more than the median of half a dozen warmed-up batches. The harness is meant for something else — a stable trend over time, which can be placed in CI and compared day by day.

Checklist

• Digitize measure.mjs on your own machine — do not treat others’ as your own.
• Compare compare.mjs first with the previous report on the same hardware; others’ baseline — only by growth pattern, not by absolute figures.
• Do not measure what the harness does not measure: tsc, bundler, network — they require a separate application-level measurement.
• Clearly mark skipped steps (like transpilation without typescript) instead of either falsifying the number or hiding it silently.
• For CI, compare with the previous run on the same runner and take a threshold with a noise margin, not a rigid constant.

Cases in the catalog on this topic

Next: End-to-End Project