Responsibilities
- Monitor live AI workflows for irregularities, trust concerns, and declining performance
- Analyze signals tied to output accuracy, fallback triggers, and low-confidence predictions
- Report critical issues impacting user confidence, business integrity, or financial outcomes
- Measure usage trends, adoption levels, and engagement with AI tools
- Examine incidents involving output quality, system coordination failures, and prompt or workflow regressions
- Differentiate between actual software bugs and issues stemming from model behavior, data quality, or process gaps
- Collaborate with Engineering, QA, and business teams on complex incident resolution
- Support human-in-the-loop processes, fallback mechanisms, and risk-mitigated operational responses
- Establish and enforce review protocols for high-risk AI workflows in production
- Record evidence and recurring problem patterns to inform testing and release strategies
- Apply consistent oversight to third-party AI solutions, including managing vendor SLAs, update schedules, and incident trends
- Spot repeated failures and suggest enhancements to monitoring systems and feedback mechanisms
- Channel real-world operational insights back to QA, Product, and Engineering teams
- Supply structured, trackable data to analytics platforms, including performance indicators, incident trends, and adoption metrics
- Serve as the product owner for dashboards tracking AI tool usage and adoption
Work Arrangement
Hybrid — US
