Analyst-Developer AliceAI (Education)

Our team is developing Yandex's large language model, Alice AI. We want Alice to become a top-tier assistant in Russia, helping users daily with their studies and work, and engaging in friendly conversation. To achieve this, we define key product aspects of responses, then translate them into metrics and prepare the data necessary for quality improvement.

A major current focus for us is improving the quality of responses in the education domain. We aim to make Alice even more useful for school students, so she can not only help with homework but also explain concepts clearly and correctly. Our current main focuses are mathematics, Russian and English languages, history, and literature, but we don't forget about other subjects either. We believe that in this domain, we can catch up with and surpass our main global competitors.

We are looking for both middle+ level specialists and team leads ready to join our team.

What tasks await you

Creating Quality Metrics One of the main tasks is to evaluate model quality. For this, we use various approaches: from benchmarks and LLM-as-a-judge to crowdsourcing and highly qualified AI trainers. You will be responsible for formulating criteria, experimenting with approaches, and creating stable metrics for continuous improvement.

Analyzing Problematic Slices The analyst's task is to proactively search for problems the model currently cannot handle and, together with the development team, devise a solution plan, prepare data for model training, and then evaluate the effectiveness of the changes.

Data Collection Good data is crucial. You will search for high-quality instructional sources for training and validation, identify issues in existing data, formulate criteria for reference responses, and meticulously monitor compliance with these criteria.

We expect you to have

• Experience working with metrics in ML
• Proficient in SQL and Python
• Enjoy and are skilled at analyzing textual data, have built effective data processing workflows
• Knowledge of probability theory and mathematical statistics
• Are independent and ready to tackle new tasks for which there is no ready-made solution
• Are interested in LLM development and would like to delve into this topic

It will be a plus if you

• Have experience working with LLMs
• Have worked with Toloka or other crowdsourcing platforms