Lead of Offline Metrics Analytics at Yandex Pictures

The core metric for the quality of picture ranking is built based on assessor ratings of individual signals: the relevance of the picture, the quality and relevance of its source, etc. It sets the target for the search algorithms ranking pictures by query in the Pictures service. Good picture ranking is required both for picture search and for enriching search results and Alice.

We are constantly improving the key metrics of Search, and the team faces many challenges: to improve relevance on complex search slices, to learn how to measure and grow the diversity, relevance, and specialization of results.

You will lead a team of analytics developers in the role of a "player-coach": 50% — team leadership and managerial tasks, 50% — solving applied tasks.

What tasks await you

Improving offline metrics You will delve into the structure of the offline metric and help improve its quality across the entire pipeline: from assessor ratings of individual signals to the final form of the metric. You will research, formulate, and quantify growth points in the metric and the final ranking, devise and implement ways to improve it.

Automation You will research and integrate LLMs, VLMs, and other models to automate the data labeling process, which is traditionally performed by assessors. This is a new area: it will be necessary to devise ways to correctly implement and evaluate the quality of signals when using models, find optimal configurations and implement them, thereby reducing labeling costs, as well as increasing its quality and efficiency.

Evaluating metric quality and its changes Together with the Pictures team, you will seek answers to the questions: how to correctly measure improvements in the metric or its individual components and how to assess how far the current metric is from ideal?

More about analytics at Yandex — in the channel Yandex for Analytics

We expect that you

• Have analyzed data using Python or SQL
• Confidently master probability theory and mathematical statistics
• Can solve non-standard mathematical problems
• Are capable of calmly and substantively defending your point of view and listening to others'

Will be a plus

• Have leadership or mentoring experience in analytics
• Love mathematics
• Participated and won in olympiads in mathematics, physics, or programming
• Read and understand specialized texts in English
• Possess knowledge in the field of machine learning, have implemented models