Health Memory
Arena

Evaluates a healthcare AI agent's memory and reasoning over long-term user health data, spanning the full pipeline from basic information extraction to complex root-cause localization.

A synthetic test set covering diverse users and scenarios, with verifiable answers for automated scoring.

Two-stage scoring routed by answer type: numeric / boolean / list answers are matched programmatically with tolerance, while free-form text is graded 0–1 by an LLM-as-Judge against key-point rubrics. The five capability dimensions share the same protocol with equal weight; the behavioral Inducement dimension is ranked by F1 of resist-false / accept-true premise rates.

Per-dimension sub-scores and a weighted aggregate, enabling horizontal comparisons across models and agent architectures.

Evaluates whether a medical AI generates plausible-sounding but wrong, fabricated, or untraceable information, covering classic and data hallucination categories with multiple sub-types under each.

Medical-AI evaluation items spanning multiple clinical specialties, each with structured ground truth or expert annotations.

Category-routed dual-channel evaluation: classic hallucinations use LLM-as-Judge (citation cases additionally verified via PubMed / CrossRef), while data hallucinations combine LLM extraction with programmatic checks — lethal-grade errors weighted ×2.

Per-type detection rates across all hallucination categories plus an overall hallucination score (0–1). Data hallucinations also emit per-field match details for cross-model comparison.

Evaluates whether a health AI holds its safety boundaries under inducement — refusing to issue definitive diagnoses, prescribe drugs, contradict a treating physician, or give advice that delays urgent care.

50 single-turn inducement questions across 5 red-team dimensions, derived from real health profiles, anglicized and stripped of PII.

A server-side LLM-as-Judge scores each reply against compliance red lines; results are aggregated per dimension and overall.

Per-dimension compliance scores plus an overall score (0–100), comparable across models and agents.

Evaluates whether an agent that starts almost blind, in an environment where every action costs budget, knows what to ask, what to refuse, when to say it does not know, and whether it backs down when the evidence contradicts it.

Interactive cases compiled from real medical records (growing), each with a multi-stage timeline and rubric points on five tracks: data sufficiency, risk recognition, differential diagnosis, safe action, closed-loop verification.

Three-step scoring: LLMs may extract and normalise, but every score is computed by code. Prospective and outcome axes are reported separately; nine safety gates multiply into both and cannot be compensated. The leaderboard uses the gated per-track scores; the total is the equal-weight mean of the five dimensions.

Five gated track scores plus a total (0–100).