• Build online and offline evaluation systems for LLM agents, including pipelines that use golden datasets, ground-truth data, human review workflows, and experiment results.
• Create automated quality gates so changes to prompts, context, models, or agent logic can be tested before reaching production.
• Analyze large volumes of agent traces and executions in columnar and analytical databases such as BigQuery or ClickHouse to identify failure modes, quality regressions, latency issues, reliability gaps, and cost optimization opportunities.
• Build reliable data retention and replay mechanisms for long-term analysis of production agent behavior.
• Manage observability tools for tracing, monitoring, debugging, and experiment management of our audit agents.
• Team up with backend engineers to improve the speed and reliability of our retrieval and reasoning agents.

You will fit into the role if you...

• Have strong Python and/or backend engineering experience.
• Have a solid understanding of how LLM and agent systems are evaluated—including deterministic checks, ground truth, LLM-as-judge, human review, and quality metrics—and can reason about when each approach is appropriate.
• Have deployed and operated systems in the cloud, ideally on GCP.
• Have hands-on experience building end-to-end retrieval or ML pipeline evaluation systems and using LLM observability or experimentation tools such as Braintrust, MLflow, Langfuse, or Weights & Biases.
• Are comfortable working with analytical databases, data warehouses, columnar stores, and high-volume event or trace data.
• Understand system design, reliability, observability, monitoring, logging, debugging, and operational trade-offs.
• Bring senior-level engineering judgment: you can make architectural decisions, communicate trade-offs, and build systems that other engineers can extend.
• Are comfortable with ambiguity, able to reason from first principles, and excited to build infrastructure for AI systems that are actively used in production.

Nice-to-haves that are a plus:

• Designing data pipelines, ETL/ELT workflows, event-processing systems, or feedback loops for production data.
• Building infrastructure around LLM-based products or agentic systems, including optimizing LLM usage, context windows, reasoning tokens, or model selection.
• Working with production traces from complex distributed systems.
• Building internal platforms for engineers, domain experts, or operations teams.
• Using workflow orchestration systems such as Temporal or similar.
• Familiarity with audit, finance, compliance, or other high-accuracy domains.
• Experience in an early-stage startup or fast-moving engineering environment.

No-one checks every box. If you’ve shipped retrieval systems and like owning evaluations and pipelines, let’s talk.

What we offer

• High impact & growth: Shape strategy at a scaling AI startup from day one.
• Mission-driven culture: Ambitious team valuing first-principles thinking and bold ideas.
• Attractive compensation: competitive salary plus significant equity.
• Best tools for the job: Generous coding tools budget so you can use the best tools for the job.
• Startup perks: Flexible vacation, team lunches, retreats, central Berlin office.

Interview process

• First Call — Intro to Cortea with Liza
• Second Call — Technical interview with Vlad
• Third Call — Deep dive into our culture with our Co-Founder Philipp
• On-site Day (Berlin) — Meet the team and work on a real problem together