At MEDvidi, we are scaling AI-powered systems across critical parts of our product and operations.

As these systems become more autonomous and deeply integrated into user workflows, understanding how they behave in production becomes a core business challenge.

We are hiring a Product Analyst to help us evaluate, monitor, and improve AI agents and LLM-powered workflows operating at scale.

This role is focused on analyzing real AI system behavior in production environments:

• understanding product and operational impact,
• identifying failure patterns on generational tasks,
• measuring quality and reliability of outputs,
• helping teams make accurate decisions under ambiguity.

Responsibilities

• Analyze production behavior of AI agents, automation systems, and LLM-powered workflows,
• Investigate hallucinations, inconsistencies, failure modes, and quality degradation across automated user journeys,
• Work directly with logs, conversations, raw system outputs, and operational datasets to identify hidden patterns and actionable insights,
• Design evaluation approaches and analytical frameworks for measuring AI system quality and reliability,
• Build monitoring systems and dashboards for AI performance visibility,
• Identify operational and product risks introduced by automation,
• Measure the impact of AI systems on business metrics, operational efficiency, and user experience,
• Detect data quality issues, shifts, and anomalies affecting AI system performance,
• Partner closely with Product, Engineering, QA, and Operations teams on AI-related initiatives,
• Support rapid decision-making in fast-changing and ambiguous environments.

Requirements