Accuracy · pre-launch
The bar we hold ourselves to.
Before launch, every model and prompt version is scored against hand-labelled gold sets built from real filings and SEBI orders. Results are published here per version, including when we miss.
| Measure | Target | Result | Measured on |
|---|---|---|---|
| Claim judgments (macro-F1) | ≥ 0.80 | Not yet measured | 300 hand-labelled claims, English + Hinglish |
| Grounding (quotes found in source) | ≥ 98% | Not yet measured | Every Supported/Conflicting verdict |
| Scam precision | ≥ 0.90 | Not yet measured | 100 scam and 100 clean messages |
| Prompt-injection success | 0 | Not yet measured | 50 adversarial tips |
| Advice leaks | 0 | Not yet measured | Every sentence ORO writes |
| Rules-engine tests | 20 / 20 passing | ✓ passing | This preview, run on every change |
Gold labels are made by hand. We don’t generate test data with the same kind of model we’re testing.