AI Agent Evaluator
AI AGENT DEVELOPMENT & EVALUATION
Primary Care, Acute Symptom Triage & Travel Health | Contract, up to 10 hrs/week
ABOUT THE ROLE
We are seeking Nurses & Nurse Practitioners with clinical experience in one or more of the following areas:
- acute symptom management
- CHF and hypertension management
- diabetes management
- travel health
to serve as a subject matter expert supporting the evaluation of AI-driven clinical agents. In this role, you will evaluate agent conversations and recommendations spanning acute symptom triage and escalation decisions, insulin/diabetes titration, and pre-travel health guidance, ensuring clinical accuracy, guideline adherence, and patient safety. This is a part-time, remote contract position (1099) requiring up to 10 hours per week, with flexible scheduling.
KEY RESPONSIBILITIES
- Review recorded and simulated AI agent interactions involving acute symptom triage and assess the appropriateness of escalation and de-escalation decisions.
- Review agent conversations related to insulin and diabetes medication titration for clinical accuracy and guideline adherence.
- Evaluate agent-delivered travel health guidance, including pre-travel counseling, vaccination recommendations, and destination-specific risk assessment.
- Identify clinical safety concerns, edge cases, and gaps in the agent's triage logic or clinical reasoning.
- Score and annotate agent performance using structured evaluation rubrics.
- Provide clear, actionable written feedback to clinical and product teams to inform agent refinement.
- Participate in periodic case review and calibration sessions with other clinical SMEs to ensure consistent evaluation standards.
- Help develop and refine test scenarios used in ongoing agent quality assurance.
QUALIFICATIONS
- Registered Nurses: active, unrestricted RN License Nurse Practitioners: active, unrestricted Nurse Practitioner license in the United States, with current board certification (...