Hi! 👋
We are a team developing corporate AI agents and RAG systems for smart search within internal documentation. Some of our solutions are already in production, and some are under active development.
We are currently looking for a specialist to focus on the quality aspect of our systems. Your task will be to improve RAG quality, build methodologies for evaluation, testing, and protection, as well as engage in prompt engineering.
Important: this is an applied position – we are not training models, but using ready-made ones through a corporate gateway. Therefore, the entire focus will be on engineering, evaluation, and fine-tuning agent behavior.
If you are driven by making AI solutions reliable, measurable, and useful – welcome to the team!
What you will be doing:
- Improving the quality of RAG agents in three areas: search within technical documentation, regulatory base, and accounting documentation;
- Prompt engineering and context engineering for all team agents;
- Preparing benchmark datasets for quality evaluation;
- Implementing LLM-as-a-judge as a methodology for automatic regression testing;
- Configuring guards for production agents: prompt injection filters, output structure and content validation, PII leak protection, anti-hallucination mechanisms;
- A/B testing prompts and models to select the best configurations;
- Tuning quality based on user feedback signals and traces from Langfuse.
Requirements:
- Python with 2+ years of commercial development experience;
- Practical experience with RAG systems in production: embeddings, vector databases (Qdrant, FAISS, or pgvector), reranking, chunking. Experience in building and supporting at least one RAG solution from start to finish, not a prototype;
- Practical experience in evaluating the quality of LLM systems: preparing benchmark datasets, offline metrics, LLM-as-a-judge, regression testing. Experience with evaluation frameworks (Ragas, DeepEval, or similar);
- Practical experience in configuring guards for LLM applications: prompt injection protection, output structure and content validation, personal data leak protection;
- Experience in prompt engineering and context engineering in real projects: iterative prompt tuning, structured output, function calling;
- Understanding of RAG architecture: document splitting strategies, metadata, embedding model selection, reranking, source attribution accuracy;
- Practical experience with at least one LLM framework: LangChain, LangGraph, PydanticAI, OpenAI API, or similar;
- Practical experience in A/B testing prompts and models in production;
- Experience with agent protocols (MCP) or a custom tool-layer for agents;
- Basic SQL and relational database work.
