Our NLP team works on tasks related to information extraction from documents, dialogues, and texts of various kinds.
Our main goal is to create ready-made products with LLM under the hood, as well as to create self-service for the business, where everyone can create the scenario (skill) they need in a No-code mode.
Responsibilities
- design and calibrate judge pipelines: rubrics, reference labeling, human agreement, combating positional bias and length bias
- build RAG search for duplicates and contradictions: chunking, hybrid search, reranking, checking contradictions at the statement level
- perform knowledge regression tests: golden set, thresholds, run on every change of prompt or model
- catch hallucinations: statement extraction, linking to source, verification
- fine-tune and distill judge models to reduce cost and latency without loss of quality
- assemble multi-agent pipelines with guardrails and run cost control.
Requirements
- higher education: mathematics, physics, computer science, or a relevant data science degree
- 5+ years in ML, of which at least one and a half years with LLM in production
- RAG in production, not a demo: with quality metrics, load, and improvement history
- Fine-tuning: SFT, LoRA/QLoRA, DPO preferably
- experience in building and calibrating LLM-as-judge
- Guardrails and hallucination detection
- multi-agent pipelines: orchestration, failure handling, cost limitation
- Python: writing code that goes into production
- English for reading articles and documentation.
Will be a plus
- backend Python experience: FastAPI, asynchronous programming, queues, PostgreSQL, vector databases
- knowledge graphs and ontologies
- NLI, fact extraction, working with annotators
- publications, presentations, open source.
Conditions
- hybrid work format
- annual bonus
- corporate gym and recreation areas
- over 400 educational programs from SberUniversity for professional and career development
- regular meetups and a developed DS-community
- extended VHI (Voluntary Health Insurance), preferential insurance for family, and a corporate pension program
- flexible mortgage discount equal to 1/3 of the Central Bank's key rate
- Prime subscription with the possibility of sharing with three close people
- reward for recommending friends to the Sber team.
