We benchmarked how AI performs on nursing operational tasks.
- Dr Ron Li
- Kenneth Sng
- Oliver Chan
- Cameron Yeo
Coordination takes up a lot of a hospital-at-home nurse’s day: building schedules, making intake calls, writing handoff notes. It’s tedious work, but every detail matters. So we ran a benchmark study to see how AI performs on it, using real hospital-at-home programs and scenarios from licensed clinicians, with 92 yes-or-no criteria based on their definition of good. Experienced RNs set the baseline, and we measured a general-purpose AI agent and Bedside against it.
1. Results
In the study, Bedside significantly outperformed general-purpose AI, while matching the nurse baseline.
Performance index
- Nurses: 1.00, nurses = 1.00.
- General-purpose AI: 1.03, nurses = 1.00.
- Bedside: 1.29, nurses = 1.00.
Performance index based on overall criteria achieved.
* General-purpose AI showed no statistically significant difference from nurses; Bedside performed significantly better than nurses.
Figure 1. Representation of overall results.
On the 24 safety-critical criteria in the baseline, Bedside met every one.
2. Methodology
Every task was either real data from a running hospital-at-home program or a simulated task written by licensed clinicians.
Each criterion is a yes-or-no statement of good practice, based on real hospital programs’ definition of good. Criteria where a miss was judged to harm the patient are flagged as safety-critical, such as a missed medication dose, diagnosing a symptom on the call instead of leaving it to the provider, or leaving a patient’s IV antibiotic out of the handoff note. Criteria that at least half of the experienced RNs got right were used to set the baseline.
Sample tasks
Scheduling
Each participant plans one program’s full day of home visits from its census, roster and orders: every dose inside its window and every admission, lab and physician call, placed on the program’s own nurses. The requirements reflect the program’s actual census, staffing and SOP, such as its visit scheduling guidelines and required dosing intervals.
Nurse baseline
Met
General-purpose AI
Missed
Bedside
Met
CriterionAs per SOP, morning rounding visits are scheduled for 90 minutes, afternoon/evening visits for 60 minutes, and ad hoc visits for 30 minutes.
Figure 2. Diagram of the scheduling task.
Home intake call
A patient has just come home from hospital, and the participant phones her to run a program’s intake protocol: check how she is, take her vital signs with her home kit, go through her medicines and her home, and write a note for the provider visiting that afternoon. The patient is played by an AI voice and, like a real patient, mentions some problems only when asked the right question. The nurses spoke with her by phone and the agents by text.
Nurse baseline
Caller: Do you have a working refrigerator and working AC and heat?
Patient: The fridge works and has room. Heat and air conditioning both work.
Caller: Do you have enough food at home?
Patient: There’s plenty.
Met.
General-purpose AI
Caller: Where are the antibiotic bags being kept right now — are they in the fridge?
Patient: They’re on the kitchen counter. I don’t know if they should be in the fridge.
Caller: That’s alright. To be safe, could you put the bags in the fridge for now? Please don’t freeze them.
Patient: Sure, I’ll put it in the fridge.
Asks about the fridge only for the IV bags, and omits food and heating.
Missed.
Bedside
Caller: Could you put that bag in your refrigerator for now? Is your fridge working?
Patient: Sure, I’ll put it in the fridge. The fridge works fine.
Caller: Is the heating and cooling in the apartment working okay?
Patient: Yes, both work fine.
Caller: And do you have enough food in the house for the next several days?
Patient: Yes, Marcus does the shopping and there’s plenty.
Met.
CriterionAsks whether she has a working fridge, enough food, and working heating or cooling.
Figure 3. Illustration of the calling task.
3. Key learnings
An agent trained on the program’s SOP can perform nursing operational tasks effectively
Bedside met 100% of the nurse baseline, including every safety-critical criterion, and it did so consistently. Every schedule it built for the first program gave each patient all the visits their orders required, with every IV dose inside its window and no new patient visited before they had got home. It planned each program’s full day of visits in under 4 minutes, time a nurse could spend with patients instead.
Program-specific training has a positive effect on agents
Working without the program’s SOP, general-purpose AI met 71% of the nurse baseline. Most of what it missed is written in the program’s own procedures. The intake protocol asks the caller to take a full set of vital signs with the patient’s home kit and to check she has food, a working fridge and heating: Bedside did both on every call, general-purpose AI on none. The first program’s SOP keeps the last half hour of each staff nurse’s shift free so no visit is rushed: Bedside always kept it free, general-purpose AI never did.
Without training, an agent can miss safety-critical steps
In one scheduling scenario, a patient’s daily IV antibiotic was due around noon, with two hours’ leeway either side. General-purpose AI scheduled it after 3 pm every time, whereas Bedside always kept it inside the window.
In another, the SOP set morning visits at 90 minutes and afternoon visits at 60. Every general-purpose AI schedule cut some of these visits short or left them out; every Bedside schedule gave them their full time.
And in an intake call, the patient had a sore on her foot. The protocol is to record it in her own words and leave the assessment to the provider visiting that afternoon. In most of its notes, general-purpose AI diagnosed it instead, as a probable ulcer and a possible source of her leg infection.
Taken together, the results show that an agent trained on a program’s own SOP, like Bedside, can carry the coordination work of hospital at home to the standard experienced nurses set, safety-critical steps included. That leaves nurses’ time for patients.
4. Case study
A full schedule by Bedside
Bedside completed this schedule in 1.1 minutes, with all medication doses timed within their windows and all visits scheduled within staff availability.
Bedside
7:00Patient C
Cefepime IV 0700 dose; daily morning labs + blood cultures + wound cultures; respiratory assessment (Spanish via interpreter line)
8:00Patient D
Vancomycin IV 0800 dose; daily morning lab draw; cellulitis assessment (interpreter line)
8:00Patient G
Cefazolin IV 0800 dose; daily morning lab draw; begin graduation teaching
8:15Patient B
Initial enrollment (90 min): tech kit setup, room safety check, first teaching, admission assessment (age 95; ceftazidime 0800 already given in hospital — no home dose)
11:00Patient A
Initial enrollment (90 min): tech kit setup (confirm supply kit/IV meds delivered), home safety assessment, first teaching, LLE cellulitis assessment (vancomycin given 0930 in hospital — next dose 2130)
11:30Patient E
Daily lab draw; ertapenem IV daily dose ~1200 (30-min infusion); diabetic foot assessment
13:00Patient F
Furosemide IV ~1300 dose; daily weight, volume status/HF assessment
15:00Patient C
Cefepime IV 1500 dose; respiratory reassessment; follow up midday lab results (interpreter line)
18:00Patient G
Cefazolin IV 1800 final dose; IV removal after dose complete; discharge education; after-visit summary — graduation
19:00Patient F
Furosemide IV ~1900 dose; HF assessment, intake/output review
19:15Patient B
Ceftazidime IV 2000 dose; pyelonephritis/fall-risk assessment; reinforce teaching
20:00Patient D
Vancomycin IV 2000 dose; daily RLE wound care (son translating)
20:30Patient A
Vancomycin IV 2130 dose (due 12h after 0930 hospital dose); LLE cellulitis check; reinforce teaching
22:00Patient C
Cefepime IV 2300 dose (given ~2200–2215, within ±2h window); respiratory assessment (interpreter line)
Nurse 1
7:00–8:00Patient C60 min
8:15–9:45Patient B90 min
11:00–12:30Patient A90 min
Nurse 2
15:00–16:00Patient C60 min
19:15–20:15Patient B60 min
Nurse 3
20:30–21:30Patient A60 min
22:00–23:00Patient C60 min
Nurse 4On call
8:00–9:00Patient D60 min
20:00–21:00Patient D60 min
Nurse 5On call
8:00–9:00Patient G60 min
18:00–19:15Patient G75 min
Nurse 6On call
11:30–12:30Patient E60 min
13:00–14:00Patient F60 min
19:00–20:00Patient F60 min
Figure 4. Bedside’s schedule for one day at a hospital-at-home program.
Frequently asked questions
Why did Bedside conduct this study?
Bedside builds AI for hospital-at-home operations. Programs deciding whether to trust AI with this work had no evidence drawn from their own workflows, only vendors’ claims. We built a benchmark from real program days and the programs’ own definition of good, had experienced RNs set the baseline, and measured a general-purpose AI agent and Bedside against it, so the answer rests on that evidence rather than on our word.
How did we keep the study authentic and accurate?
Authentic: every task was either real data from a running hospital-at-home program, with its census, roster, orders and SOP, or a simulated task written by licensed clinicians. The criteria were written and edited with the programs’ nurse leads and those clinicians, and each program’s SOP and protocol were used unaltered.
Accurate: the criteria were crafted in consultation with the programs and licensed clinicians, and every answer was graded by a separate AI model, as described under “Who grades the answers”. The results were inspected before publication.
How were the nurses recruited?
Recruitment was open, and every nurse who took part is a real, practicing nurse. Each worked through the assigned tasks remotely, at their own pace, with the same task description, SOP and tools: the study app, a calendar for the schedules and a microphone for the call. Only each nurse’s first graded attempt at a task counts.
Are these real patients and real programs?
Tasks built from a program’s real day use its own census, roster and orders, with names and places changed. The patients in simulated tasks are fictional, written by licensed clinicians. Program names and locations have been anonymized.
Who is the patient on the intake call?
An AI voice playing a patient written by licensed clinicians. Like a real patient, she answers what she is asked and brings up some problems only when asked the right question, so the call shows what the caller thinks to ask.
How did the AI agents take the tasks?
For each schedule, each agent received the same information a nurse sees, as text: the task, every patient’s case card, the staff on duty and, for Bedside, the program’s SOP. It returned the whole day as one structured schedule, every visit with its nurse, patient, task and times, and that schedule was graded exactly as a nurse’s was. For the intake call it spoke by text with the same AI patient the nurses called, one line at a time, and ended the call by writing a handoff note for the visiting clinician. Each run started fresh, with no memory of earlier runs and no fine-tuning on any program.
Who grades the answers, and can an AI grade fairly?
A separate AI model, independent of the agents under test, grades every answer without knowing who produced it. Each criterion is a yes-or-no question about the answer. The grader casts three votes on each schedule, with the majority recorded, and one vote on each intake call. Drive times and clock comparisons are computed in advance and handed to it, so it compares rather than calculates, and verdicts that contradict their own reasoning are regraded.
Published work supports grading of this kind. Strong model judges agree with human judges over 80 percent of the time, as often as humans agree with one another (Zheng et al., 2023). In health, a model deciding whether each written criterion was met agreed with physicians about as often as physicians agreed with each other (Arora et al., 2025), and a jury of models rating clinical answers agreed with clinicians slightly more closely than clinicians agreed with one another (Bedi et al., 2025).
Considerations
These results are not conclusive: nurses who took part, though experienced RNs, had no prior experience with these SOPs. They were graded on each program’s own rules with only its SOP to go on, while a program’s own nurses would also know its patients, staff and routines.