Lead of Expert Analytics at Alice AI

Our team develops the quality of the company's key LLM technology — Alice AI LLM. We work to make our models smarter, more useful, and more reliable, and we are looking for a Lead of Expert Analytics who will help us move even faster.

We create new metrics, design data collection and validation processes, deeply analyze model behavior, and bring them to new levels of quality. We pay special attention to improving quality in expert domains — areas where the correctness of a response can be assessed by only a small number of specialists with relevant education and experience. In such tasks, it's important not only to evaluate the model's output but also to determine authoritative sources, verify the correctness of the argumentation, and propose a visualization of the ideal, reference answer.

What tasks await you

Strategy and Leadership You will shape the analytical strategy for developing the model's LLM skills, build a team of analysts, engage in mentoring, and plan resources. You will also have to develop approaches to evaluating LLM quality in different company products within expert domains.

Metrics and Evaluation System You will develop comprehensive metrics that reflect the real quality of models, create evaluation methodologies in highly complex expert domains — formulate criteria, select authoritative sources, describe ideal answers. You will also need to compare our own models with SOTA approaches on the market and automate evaluations using LLM (LLM-as-a-judge) and hybrid methods.

Data Collection You will launch complex data annotation processes for fine-tuning and RLHF/RLAIF, work with narrow expert groups, build pipelines involving domain specialists, accelerate processes using LLM tools, as well as control data quality and prepare datasets for research tasks.

Implementing Best Practices You will monitor and analyze new research, benchmarks, and methodologies, conduct experiments with synthetic data, LLM feedback, active learning, expert-in-the-loop, and implement research findings into production processes.

More about analytics at Yandex — in the Yandex for Analytics channel

We expect that you

• Have managed a team of analysts for at least three years
• Love working with data and know how to extract practical results from it
• Know mathematical statistics and probability theory
• Know how to write code in Python and know algorithms
• Are self-sufficient and ready to take on new tasks for which there is no ready-made solution
• Are interested in the development of LLMs and want to dive deep into the topic