LLM Ops Engineer
Internal Support Automations is an internal development team that creates automations and services for the support cluster. The main goal is to ensure that automation works well and does not distract support from caring for our clients.
What needs to be done
- Design and build an LLM stack in Go: agent architecture, tool-use, skills/scenarios framework (build services through which the assistant will do its job);
- Solve build-vs-adopt for the platform: evaluate Dify/LangFuse/OpenSearch (make decisions on whether to adopt ready-made solutions or build our own, and be responsible for the choice);
- Develop and maintain LLM integrations: RAG over knowledge bases, agent pipelines, tools secured by role-based access control and authorization, connect and proxy existing integrations;
- Design a system of skills and scenarios: destructive scenarios involving financial data and flight tickets (returns, exchanges, voids);
- Build LLMOps processes, monitor quality: deployment, versioning of models and prompts, reproducibility, CI/CD, golden sets, A/B testing where possible;
- Participate in scheduled on-call rotations for production services.
What we expect from you
- Commercial development experience in Go or Python for at least three years;
- Experience with Kafka, MongoDB, PostgreSQL;
- Experience operating LLM/ML systems in production (monitoring quality and cost, drift, evals);
- Experience with LLM integrations: RAG, retrieval, embeddings, prompt construction (chain-of-thought / ReAct), tool-use / agent architectures;
- Engineering maturity and autonomy – ability to evaluate platform tools and make choices: adopt, extend, replace;
- Experience with containerization (Docker), Kubernetes.
Will be a plus
- Experience with LLMOps platforms: LangFuse / LangSmith, prompt registry, tracing, observability;
- Experience with LLM pipeline orchestrators: Dify, LangGraph or similar;
- Building golden sets and LLM eval in CI (tracking degradation);
- Experience with ClickHouse, OpenSearch, Trino.
How we work
- from anywhere in the world: we are not tied to a location, we pay in dollars, we love to travel;
- no bureaucracy: comfortable, modern processes, horizontal and open communication, quick discussion of ideas and decision-making;
- compensation: voluntary medical insurance, psychotherapy or foreign language courses, sports activities, and sick leave coverage;
- on a paradise island: you can work from Phuket with an ocean view.
