We're developing advanced AI platforms that reshape how technology insights are generated and used. As an Observability Engineer, you'll play a central role in ensuring these systems operate with clarity, efficiency, and measurable quality.
What You’ll Do
You’ll build and refine the core observability infrastructure that monitors AI agent behavior across the full pipeline. This includes designing dashboards that track key metrics like response times, token consumption, cost per query, and error rates. You’ll implement alerting mechanisms to detect anomalies in performance or spending, enabling rapid response.
You’ll extend tracing systems to expose granular component-level behavior, helping teams understand system dynamics. You’ll also develop A/B testing capabilities to evaluate prompt engineering strategies and pipeline modifications under real-world conditions. Working closely with evaluation specialists, you’ll ensure data from experiments informs ongoing optimization.
What We’re Looking For
- At least three years of experience in observability, monitoring, or data visualization systems
- Hands-on work with tools like Prometheus, Grafana, or Datadog for metrics and alerting
- Proficiency in Python and familiarity with distributed tracing concepts
- Proven experience building or contributing to experimentation platforms
- Ability to process and interpret large-scale telemetry data
Nice to Have
- Experience with LLM-specific observability tools such as Langfuse or LangSmith
- Background in cost analysis or resource efficiency for AI systems
- Knowledge of statistical methods for evaluating experiment outcomes
- History of developing internal tools or custom dashboards
- Platform or infrastructure engineering experience
Environment & Culture
This role operates in a hybrid model, with minimal office requirements—around 1–2 days per month. You’ll collaborate with a team of skilled professionals in a culture that values intellectual curiosity, transparency, and shared progress. Initiative is recognized, and there’s room to explore beyond your core focus. The organization emphasizes long-term growth, meaningful contributions, and a work environment where perspectives are welcomed and impact is clear.
