AI & Agents
Umwelt
Last updated 2026-11-04 · benchmark measured 2026-11-20 — deterministic & reproducible
First Silicon: a benchmark testing whether AI models can find real hardware bugs.
Umwelt at a glance
Umwelt's First Silicon is a benchmark dataset of 86 real hardware bug fixes from open-source chip designs, used to test how well AI language models can detect genuine defects without raising false alarms.
Good for
- AI researchers and model developers wanting to measure how well a language model finds real hardware bugs
- Chip design teams or formal verification specialists checking whether an AI model can be trusted to review RTL code
- Teams wanting an independent, blind evaluation of their own AI model or verification tool against real bug fixes
Questions people ask
What does the First Silicon benchmark actually test?
It gives an AI model one version (buggy or fixed) of a real hardware module and asks it to write SystemVerilog assertions describing the intended behavior; a formal checker (Yosys and SymbiYosys) then checks those assertions against both the buggy and fixed versions to see if the model truly caught the bug without raising a false alarm.
How well do current AI models perform on this benchmark?
The best-performing configuration, Claude Opus 5.5 at maximum effort, caught only 16 of 86 cases cleanly, and most bugs escaped detection by every model tested.
Where do the bugs in the dataset come from?
The 86 cases are real bug fixes taken from open-source chip projects: 70 from OpenTitan (Google's open-source security chip) and 16 from 10 other open-source projects.
Can I get my own AI model or verification tool tested on First Silicon?
Yes, Umwelt offers to run your model or verification tool on First Silicon and share the results only with you, or to check your own RTL design for bugs, which you can arrange by contacting them.
Is the benchmark data publicly available?
Umwelt publishes the outcome for every bug along with five bugs in full detail and a salted SHA-256 commitment for each bug and prompt, but keeps the other 81 bugs private so the benchmark stays usable for evaluating future models.
Written by Legit.Show from umwelt.vision on 2026-11-09. It does not change the score.
Are you the maker of Umwelt? Claim this listing. It is free, takes a meta tag, a DNS record or GitHub admin rights, and never changes the score.
Launched something? Add your product, free.
To cite this score: legit.show/s/umwelt-vision/2026-11-20. That address never changes; this page moves with every re-measure.
Legit.Show scored Umwelt 74/100 on 2026-11-20, measured across all 7 frames from its public surface. legit.show/s/umwelt-vision/2026-11-20
Is Umwelt production-ready?
Legit.Show scores Umwelt 74 out of 100 — the simple average of its 7 measured frames. Legit.Show ran its deterministic 7-Frame production-readiness benchmark on Umwelt (public-surface assessment), measured from the public surface with no LLM in the scoring path. Its strongest frame is Discoverability; its weakest is Privacy. Every frame it averages is published with its evidence on the Legit.Show listing.
The 7 Frames
- Performance — 88/100
- Accessibility — 99/100
- Security — 25/100How this compares across the catalogue: The Web Security Baseline
- Privacy — 25/100How this compares across the catalogue: The Privacy Gap
- Reliability — 92/100
- Standards — 92/100
- Discoverability — 100/100
What we measured
- No Content-Security-Policy and no HSTS.
- Served over HTTPS with a valid certificate.
- Real Lighthouse performance run — 969 ms to first byte.
- Returns a proper 404 for unknown routes.
- 0 of 0 sampled routes reachable.
- No privacy policy found.
- Sets cookies / loads scripts with no consent prompt.
- Discoverable: structured data, sitemap, OpenGraph image, canonical URL.
Who it's for
AI researchers · Hardware engineers · Chip design teams · AI model developers · Formal verification specialists
Sources and updates
- Description
- Taken from umwelt.vision's own website on 2026-11-04.
- Benchmark
- Measured by Legit.Show from the live site, the way any visitor sees it — with no access to its code or accounts. Same method for every product, and no AI decides the score. Last checked 2026-11-20.
Put together from public information, without Umwelt's involvement. If anything here is wrong, tell us and a person will check it.
Visit Umwelt → · Alternatives to Umwelt → · How this was measured →