PravoTech creates products for lawyers. We are developing our product — Pravo(Agent), which transfers legal work to AI agents. The team is small and fast: changes to the agents are released daily, and we are looking for an engineer who will take ownership of the agents' quality.
What you will do
- Monitor changes in agents, skills, prompts, and RAG, and test affected scenarios before release.
- Create E2E scenarios of a lawyer's work in the chat and run them through test agents: tests run in parallel, reports are collected automatically.
- Evaluate model responses using LLM-as-a-judge and calibrate assessments based on lawyer annotations.
- Test RAG: search quality, accuracy of links, consistency of responses with sources.
- Analyze traces in Langfuse and telemetry: where the agent made a mistake, the cost and duration of each step.
- Prepare short reports: what broke, who is fixing it, how to make the agent faster and cheaper.
- Write API and UI autotests, manage regression in CI, create bug reports.
What is important to us
- From 3 years of experience in AQA or backend development in Python.
- From 1 year of practice with LLM agents or LLM applications: testing or developing agent systems.
- Experience with LLM-as-a-judge and RAG evaluation.
- Langfuse or similar (LangSmith, Phoenix, OpenTelemetry): traces, datasets, token and cost calculation.
- Python, pytest, API and UI autotests (Playwright or similar), Docker, CI.
- Understanding of agent architecture: tool calls, context, prompt cache, reasoning tokens.
- Independence and speed: you notice changes yourself, test them, and provide a report.
Will be a plus
- Backend development experience in Python.
- Understanding of hybrid RAG architecture.
- Pet projects with agent systems.
- Relevant IT education.
