We're growing our team and looking for a Senior/Principal QA Engineer to own quality across our AI voice assistant: from prompt and agent behavior testing to mobile, TV, and backend.
Requirements
- Senior / Principal, 5+ years of manual QA - a high level of ownership and self-direction: the ability to take on the project as a whole, be fully accountable for quality, and get up to speed on new things independently.
- Experience testing AI agents / assistants: checking scenario and prompt following (prompt / instruction following), response quality and relevance.
- Experience testing "under the hood" via AI observability / tracing tools (LangSmith and analogs): analyzing an agent's traces - tool calls, memory handling, the model that ran, inputs/outputs, token consumption, spotting bottlenecks.
- Experience with mobile and/or TV-app testing; experience specifically with TV devices is less critical.
- Experience testing backends and integrations: API testing (REST / gRPC), verifying the seams between services, reading logs and traces (Kibana / Grafana / Loki, etc.).
- Solid QA fundamentals: test design, test documentation, defect handling.
- Hands-on experience using AI tools in the day-to-day QA process.
Nice to have
- Experience testing speech technologies (ASR / TTS) and working with audio.
- Evals - building and running AI-assistant quality evaluations: scenario-based self-play, LLM-as-judge, manual validation as human-in-the-loop.
- CI/CD / deployment / test environments: configuring stands, building and deploying applications (GitHub / GitLab, etc.).
- Device testing experience / ADB.
- Willingness to move into test automation over time.
Responsibilities
AI assistant:
- Assessing assistant quality: prompt / instruction following, compliance with requirements; catching regressions between prompt and model versions.
