We tested the tools
that test you.
Two free, open-source EU AI Act scanners, run in August 2026 against our corpus using the workflows their own documentation recommends. They are useful tools, given away by people solving a hard problem early. Every finding below has been reproduced from a clean install using the documented commands, and each is written up so its maintainer can reproduce it too. A third tool was excluded as a different category rather than counted as a failure, and two more we could not install are recorded as untested — “we could not install it” is a note, not a result.
It would not start
A dependency declared with a floor and no ceiling. The dependency released a major version, moved a module, and a clean install fails at import before doing anything. Still true on a fresh machine when we last re-ran it, in August 2026. Pinning the older version fixes it in a second, once you know.
A compliance tool that will not start is a compliance tool nobody ran, and the failure presents as your fault rather than the package’s.
A perfect score over an empty set
A declaration-based scanner reported no AI frameworks, then printed a green tick and 100%. Its newer release detects the AI and still prints the same headline — 0 systems scanned, 0 of 0 obligations met, 100% — with the detection sitting in a warning underneath.
Any metric that reports perfection on an empty set will eventually be shown to somebody who believes it.
Two commands, two answers
The same tool’s initialiser reports no AI frameworks on a directory where its scanner reports three. Same directory, same version, same minute. The initialiser is the command the documentation tells you to run first, and it decides what ends up in your config.
Run every command a tool offers, not the one in the quickstart.
Dead code counted as live systems
One live file plus copies under backups/ and deprecated/ returned three systems. All three contain AI, so the tool was not wrong about what it read — but an inventory that is too large fails about as expensively as one too small, because every entry generates a classification, an owner and a retention period.
Present is not running.
We also published a finding that was false, and it is worth more than the four above.
We reported that one scanner could not detect a self-hosted image model — the case where running open weights makes you the provider, with the heaviest obligation set in the Act. It detects it correctly. Our test fixture was a Python dictionary containing two strings and no imports: not a pipeline, a data structure that resembled one if you already knew what it was meant to be.
We had written that a benchmark testing tools in configurations nobody uses is not a benchmark but a hit piece. Then we did exactly that.
Rebuilding the fixture honestly exposed the same blind spot in our own detector, which had been matching a single string and had no pattern for the ordinary way anyone loads open weights. The bad fixture had hidden a real gap in our own tool for a week. Both are fixed, and the retraction stays published — the case it concerns is in the suite, with the reasoning, rather than quietly removed.
When your test case is the only evidence that somebody else's tool is broken, check the case first.