Dataset

Open evaluation datasets for healthcare AI agents, covering memory retrieval and hallucination detection

Synthetic Health Profiles

An event-driven mechanism generates longitudinal synthetic health profiles spanning demographics, chronic-disease history, medication records, wearable-device metric streams, periodic exam results, and structured life-event logs. Each virtual user carries a complete health trajectory spanning months to years, simulating real-world long-term health changes, anomaly progression, and multi-factor interactions.

Generated through a programmatic three-stage pipeline (user initialization → event-driven simulation → structured export), with verifiable ground truth and event-dependency relations automatically constructed — supporting standardized benchmark evaluation of healthcare AI agents on memory, reasoning, trend analysis, anomaly detection, and medical-safety tasks.

User Profile

Demographics, chronic conditions, lifestyle, medication history

Health Timeline

Daily device indicators, exam records, unified temporal view

Event Log

Life events with temporal kernels and per-indicator effects

Data hosted on HuggingFace, fully open-source with community oversight.

Sample Patients

Two synthetic patients from the current ESL-Bench batch. Everything below is shown as it appears in the data files — profile, three years of wearable trajectories (weekly means), exam results and life events. This is exactly what the agent under test sees.

Commercial Beekeeper

#5300Explore full data

54 y · Male · Rural

Type 2 diabetes · diagnosed at 47

A 54-year-old male commercial beekeeper presenting with poorly controlled Type 2 diabetes and diabetic dyslipidemia.

Device trajectories · 3.0 years · weekly means

68 indicator kinds recorded; 6 selected by condition. Grey line spans 2023-08-22 to 2026-08-20; the value on the right is the latest week.

  • BodyWeight80.9 kg
  • RestingHeartRate-RHR71.7 bpm
  • DailyStepCount10,849 steps
  • FastingBloodGlucose-FBG7.1 mmol/L
  • MeanGlucose-MG7.6 mmol/L
  • GlucoseTimeinRange-TIR74.4 %
Latest exam · 2026-02-19

194 indicators, 25 out of range. Numeric abnormal items:

Microalbuminuria-MAU45.4mg/LPhysicalExamination-SystolicBloodPressure129mmHgPhysicalExamination-DiastolicBloodPressure83.6mmHgGlycatedAlbumin-GA19.4%
Recent events
  • 2026-08-20health eventReviewing his stable digital glucose trends on the shed workbench at the close of the August harvest1 d
  • 2026-08-13health eventUpgrading to a ventilated mesh beekeeping suit to efficiently exhaust his core engine heat8 d
  • 2026-08-08diet changeFiring up the rusty grill for lean poultry to deliberately keep his Saturday night glucose gauge flat2 d
1095 days tracked198 events6 exams57711 device readings100 evaluation queries

Retired Archivist

#5303Explore full data

78 y · Female · Urban

Primary hypertension · diagnosed at 55Chronic kidney disease · diagnosed at 71

A 78-year-old retired archivist presenting with moderately controlled primary hypertension and Stage 3b chronic kidney disease.

Device trajectories · 3.0 years · weekly means

72 indicator kinds recorded; 6 selected by condition. Grey line spans 2023-08-22 to 2026-08-20; the value on the right is the latest week.

  • BodyWeight58.1 kg
  • RestingHeartRate-RHR65.5 bpm
  • DailyStepCount3,639 steps
  • 24-HourAverageSystolicBP138 mmHg
  • 24-HourAverageDiastolicBP81.8 mmHg
  • TimeSpentBelow90%SpO214.6 minutes
Latest exam · 2026-02-19

191 indicators, 34 out of range. Numeric abnormal items:

Microalbuminuria-MAU175mg/L24-HourUrineProtein-24h-UP0.4g/24hBeta-2Microglobulin-β2-MG4.7mg/LGlomerularFiltrationRate-eGFR40.4mL/min/1.73m²
Recent events
  • 2026-08-19health eventThree-day meticulous compilation and binding of her comprehensive three-year health ledgers2 d
  • 2026-08-14health eventFour-week meticulous transcription of 1880s regional postmaster correspondence to gracefully anchor her late-summer routine7 d
  • 2026-08-08diet changeFourteen-day precise integration of chilled pear slices to safely quench her late-summer thirst13 d
1095 days tracked205 events6 exams70865 device readings100 evaluation queries

Update Mechanism

Datasets follow a rolling release cadence; each benchmark ships new batches on its own schedule (batch IDs use YYYYMM — e.g. 202603 = March 2026).

1
Batch Generation — Each benchmark produces a new batch of cases via its own synthesis or construction pipeline (see each dataset tab's "Construction" field)
2
Publish to HuggingFace — manifest.json is updated and the new batch is uploaded to the repository; older batches are kept for traceability and longitudinal reproduction
3
Platform Auto-sync — HMA detects manifest changes automatically, syncs the latest version info, and re-evaluates listed agents against the new dataset

Rolling updates refresh evaluation data periodically to prevent overfitting on a fixed question set; historical versions remain available for longitudinal comparison.

ESL-Bench

An event-driven longitudinal health-memory benchmark that evaluates an agent's ability to memorize and reason over long-term user health data.

Theme

Longitudinal health-memory dataset. Event streams simulate real users' multi-year health changes, stressing an Agent's ability to recall and reason over long-term, multi-source data.

Composition

20 synthetic users × 1–5 years of health trajectory → 320 evaluation cases. Each case includes baseline profile, event timeline (history), evaluation prompt, and ground truth.

Construction

User profiles are synthesized from a knowledge graph; health events are injected into the timeline following medical causal relations. Ground truth is programmatically derived from the graph — fully reproducible and auditable.

Coverage

Five health-ability dimensions evaluated under a unified schema; answer types span numeric, boolean, list, and free-form explanation.

LookupTrendComparisonAnomalyExplanationInducement

Version List

BatchRelease DateUsersCasesDownload
2026092026-09-3020320HuggingFace
202608latest2026-08-3120320HuggingFace
2026072026-07-2920280HuggingFace
2026062026-06-2840200HuggingFace
2026052026-04-2850200HuggingFace
2026042026-03-2445—HuggingFace
2026032026-03-04120—HuggingFace

Data Sample

One representative test case from the dataset.

{
  "id": "kg_trend_042",
  "title": "ESL-Bench Trend Dimension Sample",
  "user": {
    "type": "manual",
    "strict_inputs": [
      "In which month did resting heart rate show the largest month-over-month change?"
    ]
  },
  "eval": {
    "evaluator": "semantic",
    "dimension": "trend",
    "expected_value": "2024-07",
    "answer_type": "date",
    "key_points": [
      "Month-level aggregation required",
      "Rate of change computation"
    ],
    "source_data": "timeline.json (resting_heart_rate)"
  }
}

Dataset File Structure

Dataset hosted on HuggingFace, organized in batch / user two-level directories. Each batch named <codeYYYYMM/> (e.g., <code202603/>), representing the latest evaluation data for that month. Only the latest batch is used for evaluation; historical batches are preserved for version tracking and longitudinal comparison.

# HuggingFace repository root
manifest.json# Version index: release dates, user lists, checksums for all batches
data/
202603/# Latest batch (each batch contains multiple virtual-user subdirectories)
user100_AT_demo/# Virtual user directory (directory name = user identifier)
profile.json
timeline.json
kg_evaluation_queries.json
user101_AT_demo/# Same structure, 6 files per user
...
202602/# Historical batch (preserved for version tracking, not used in current evaluation)
...

Per-User File Details

profile.json

User Profile (Profile p_i): demographics, chronic conditions, lifestyle, medication history in structured JSON

timeline.json

Complete Health Timeline: chronologically sorted unified temporal view of all device indicators, exam data, and health events

kg_evaluation_queries.json

Evaluation Questions: five-dimension questions (Lookup/Trend/Comparison/Anomaly/Explanation), each with expected_value, answer_type, key_points, and source_data references

manifest.json is the repository's version index file, recording creation time, user lists, and checksums for all batches. The HMA platform automatically detects dataset updates by reading this file.