Responsibilities
- Develop and maintain monitoring systems for deployed machine learning and generative AI models, including automated workflows, dashboards, and alert mechanisms.
- Define and oversee key model performance indicators—such as accuracy, precision, recall, AUC, calibration, and drift for traditional ML models, and output quality, grounding, relevance, latency, cost, and compliance for GenAI systems.
- Identify and troubleshoot issues like data drift, concept drift, performance degradation, and data integrity problems, then escalate to responsible teams as needed.
- Set meaningful alert thresholds that enable timely detection of model issues while minimizing unnecessary notifications, and maintain clear documentation of expected behaviors and response procedures.
- Collaborate with data science and machine learning engineering teams to embed monitoring practices into model development and deployment pipelines, including CI/CD and MLOps/LLMOps frameworks.
- Assist in model governance by generating audit-compliant reports, monitoring records, and documentation that meet enterprise risk management standards.
- Evaluate model outcomes in relation to business performance metrics to identify opportunities for optimization or retraining.
- Present monitoring insights and model health assessments clearly to both technical and non-technical audiences through regular reporting and review sessions.
Work Arrangement
Hybrid — Boston, MA, Portsmouth, NH, Seattle, WA, Columbus, OH, Plano, TX
Work Arrangement
Hybrid — Boston, MA, Portsmouth, NH, Seattle, WA, Columbus, OH, Plano, TX
Other
- Candidates who live within 50 miles of Boston, MA; Portsmouth, NH; Seattle, WA; Columbus, OH; or Plano, TX will follow a hybrid schedule, coming into the office two days per week.
- Otherwise, this role is remote with occasional travel.
