Responsibilities
- Collaborate with AI product, engineering, and domain specialists to establish primary and safeguard metrics for key AI initiatives, emphasizing adoption, task success, quality, and business impact.
- Work closely with data science teams on experiment design, analysis, and causal reasoning when launching new AI products.
- Analyze how metrics, raw data, documentation, and APIs interact with agents to ensure data and analytics engineering needs are clear, organized, and verifiable.
- Create and manage evaluation sets, reference datasets, scoring criteria, and regression tests to validate agent outputs for accuracy, thoroughness, safety, and utility.
- Categorize errors, tool misuse, context deficiencies, and data anomalies systematically, using insights to improve prompts, retrieval systems, tools, and data pipelines.
- Track real-time usage of agentic AI analytics workflows, monitoring success rates, drop-offs, fallback behaviors, and manual intervention to guide engineering priorities.
- Maintain internal guides on agent usage, necessary human oversight points, and uncertainty communication, while aiding training and leadership narratives on AI analytics.
- Partner with executives, lead analytics engineers, and data engineering groups to translate AI features into quantifiable business results.
- Produce regular measurement outputs like dashboards, reviews, or reports for leadership to inform roadmap decisions and reduce high-impact agent errors.
Benefits
- Salary range of $165,000 to $225,000 based on experience and performance, with potential adjustments for cost of living.
- Eligibility for a year-end performance bonus and equity compensation.
- Comprehensive medical, dental, and vision insurance tailored to individual needs.
- Flexible vacation policy encouraging time off as needed.
- Pet discount programs and a 401K retirement plan with employer matching.
- Opportunity to collaborate with skilled, driven colleagues on innovative challenges with global impact.
