File Intelligence release fixtures — 41 published checks

Measured 2026-07-29 · Node.js release gate using the same deterministic intelligence-core.js shipped to browsers. Inputs are synthetic strings and byte fixtures; no visitor files or personal records are collected.

Question: Do the eight File Intelligence tools still produce the exact findings, checksums, abstentions and evidence behavior described on their public pages?

Findings

  • All 41 deterministic fixtures pass against version 1.0.0 of the public browser engine.
  • Every tool publishes the same case and pass count enforced by the release test; the build fails if those numbers drift.
  • The sensitive-data suite includes checksum validation and a negative control, while the document assistant includes a required abstention case.
  • The searchable-PDF contract verifies the OCR/PDF engine exports and that the produced output is reopened for text extraction.

The data

ToolDatasetCasesPassedPass ratePublished scopeEngine versionRelease date
file-health-checkFH-INT-SYNTH-2026-0766100%PDF/JPEG/PNG signatures, mismatch detection, JSON validity and CSV row-width fixtures1.0.02026-07-29
metadata-privacy-checkerFH-INT-SYNTH-2026-0744100%PDF Info fields, PNG text chunks, JPEG EXIF marker and clean-file control1.0.02026-07-29
pdf-accessibility-checkerFH-INT-SYNTH-2026-0755100%Tag, language, title, text-density and outline indicator fixtures1.0.02026-07-29
searchable-pdf-makerFH-INT-SYNTH-2026-0733100%Synthetic scan render, local OCR completion and output text re-extraction1.0.02026-07-29
sensitive-data-finderFH-INT-SYNTH-2026-0777100%Email, phone, IBAN, Luhn card, Turkish-ID checksum, IP and negative control1.0.02026-07-29
invoice-receipt-extractorFH-INT-SYNTH-2026-0777100%Vendor, number, date, subtotal, tax, total and currency labelled-field fixtures1.0.02026-07-29
semantic-document-compareFH-INT-SYNTH-2026-0755100%Aligned change, addition, removal, numeric-token change and identical-control fixtures1.0.02026-07-29
document-ai-assistantFH-INT-SYNTH-2026-0744100%Summary coverage, supported answer, abstention threshold and evidence ranking fixtures1.0.02026-07-29

⬇ Download raw data (CSV) ⬇ Download report (JSON)

Method

  1. Generate or load only synthetic fixtures for signatures, damaged structures, metadata, accessibility indicators, sensitive-data patterns, invoice fields, document changes and retrieval questions.
  2. Run each fixture through the exported browser-analysis core without changing thresholds or post-processing the result.
  3. Compare actual checks, severities, extracted fields and abstention behavior with the published per-tool contract.
  4. Fail the release if a tool reports fewer or more cases than its public benchmark, or if any expected assertion fails.

Limitations

  • This is a regression dataset for published deterministic behavior, not an independent benchmark against competing products.
  • The dataset is intentionally small and synthetic; it does not prove exhaustive coverage of every valid or damaged file.
  • Browser-level OCR and searchable-PDF creation are additionally checked in release QA because model loading and rendering need a browser runtime.

Takeaway

The dataset supports a narrow claim: version 1.0.0 behaves as published on these 41 synthetic regression cases. It does not prove universal accuracy, which is why every tool page separately lists unsupported and false-result conditions.

Advertisements

← All Labs reports · standards in the testing methodology