Lead for the Offline KPI Ranking Metrics Team

The offline KPI ranking metric is used to evaluate the quality of Yandex's search results based on individual signals: the relevance of web pages to a query, their authority, etc. The metric must fully reflect the quality of results for any topic of queries, types of web pages, and other aspects and set the development vector for Yandex Search.

We constantly improve the quality of signals for the metric and use LLMs. The metric must adapt to these changes and extract the maximum from improving signal quality for the final ranking.

You will take on the area of responsibility for selecting the Search KPI metric and, together with the team, build a regular process for its improvement.

What tasks await you

Improving the offline ranking metric You will research, formulate, and quantify problems in the metric and the final ranking, devise ways to fix them, and implement them. The result directly impacts production, as the offline metric is the target for the ranking model in Search.

Building and maintaining analytical models for assessing Search quality The offline KPI metrics team is responsible not only for the form of the offline metric itself but also for many offline search quality assessment processes: acceptance of new ML ranking models, compiling 'baskets' (sets of queries for assessing search quality), building monitoring quality dashboards and deriving insights from them about problem areas and methods for solving them. You will need to solve many important product and analytical questions and lead complex projects.

Working with offline metrics for all components of search results Yandex's search results consist not only of links to web pages (so-called organic results) but also of advertisements, thematic blocks. For each of these components, there is also an offline signal. In the future, you will be responsible for integrating these offline signals into a unified offline metric for all results and improving both the quality of the signals themselves and the processes around them.

More about analytics at Yandex — in the channel Yandex for Analytics

We expect that you

• Have been a lead or mentor in analytics
• Have deep knowledge of mathematics
• Know how to build stable and transparent work processes

Will be a plus

• Understand the field of machine learning, have implemented models
• Participated and won in olympiads in mathematics, physics, programming
• Read and understand specialized texts in English