We score ourselves in public.
Hackathons reward demos. We reward audits. Every metric below comes from scripts/evaluate.py running 12 fixed queries against the live agent. Re-run locally with make eval.
Measured 2026-04-25 16:43 CDT on the previous stack (Anthropic Claude supervisor + Llama 3.3 70B on Groq, no critic pass) over the first 12 of the harness's 20 fixed queries. These numbers have not been re-run on the current GPT-OSS-120B + critic stack.
Mean wall-clock
31.7s
22% faster after parallel fan-out
Tools invoked
11/12
all but find_medical_deserts in this run
Errors
0/12
0 stack traces, 0 timeouts
Languages cleared
3
EN · हिंदी · தமிழ்
Validator verdicts
PASS · 2
WARN · 4
not validated this turn
The Validator agent re-checks high-stakes recommendations against the source text. WARN means the agent surfaced uncertainty in the answer card — never silent.
Tool frequency across 12 queries
trust_score
12
facility_search
11
estimate_journey
11
total_out_of_pocket
11
validate_recommendation
8
geocode
7
semantic_intake_search
7
check_hours
7
databricks_vector_search
4
find_medical_deserts
4
status_feed
1
Per-query results
| # | Query | Tools | Wall-clock | Callable |
|---|---|---|---|---|
| 1 | I need an ECG within 15 km of Yeshwantpur, Bengaluru. | 18 | 56.6s | |
| 2 | Find a cardiology consultation near Hubli, cash payment, today. | 8 | 26.5s | · |
| 3 | Pediatric ICU near Mysuru, tonight. | 5 | 38.7s | · |
| 4 | Dialysis centre within 10 km of Patna, Ayushman Bharat. | 11 | 49.5s | |
| 5 | Trauma center within 25 km of Indore, open 24x7. | 10 | 48.2s | |
| 6 | मेरी मां के लिए डायलिसिस चाहिए, बेंगलुरु में, आयुष्मान भारत. | 15 | 48.5s | · |
| 7 | சென்னையில் இரவு திறந்திருக்கும் குழந்தைகள் மருத்துவமனை. | 11 | 46.2s | |
| 8 | Where in Bihar is dialysis coverage weakest by district? | 1 | 16.1s | · |
| 9 | Top 5 districts in Maharashtra with the worst oncology coverage. | 1 | 14.4s | · |
| 10 | Which states have the lowest neonatal-care facility density? | 1 | 18.2s | · |
| 11 | trust score for vf-1 with confidence interval | 1 | 9.5s | · |
| 12 | trust score for vf-3263 | 1 | 8.6s | · |
Generated 2026-04-25 16:43 CDT. Source: docs/EVAL_REPORT.md. MLflow eval runs land at /Shared/aarogya-atlas-eval.