Analyst for the Ranking Relevance Group

The relevance signal measures which pages best respond to the user's query: we want to bring the most complete, accurate, and convenient documents to the top. Our work directly affects both the monitoring of search quality and the training of those very ranking models that are called upon for every query to compile search results. We are actively implementing LLMs to save costs and improve quality.

Our team supports the full cycle of analytics and signal acceptance - from raw data to monitoring in production. At the same time, we face significant challenges - due to the scale, even small changes in the signal have a large effect, so both manual labeling and LLMs require advanced monitoring tools. We are threatened by fraud from performers, model hallucinations, data drift, and much more - to cope with this, we are looking for a strong analyst.

What tasks await you

Improve data quality You will find and fix problems in labeling and aggregation pipelines, build quality metrics and automatic checks to catch degradations even before the data enters training and monitoring. Another important task is to filter out bad performers and identify good ones.

Track non-obvious model problems You will need to identify hidden degradations: data drift, shifts by query and document types, LLM hallucinations, and fraud in labeling, which are not visible in high-level metrics. For this, you will set up alerts, build dashboards, and develop status transition processes.

Develop the signal You will propose and validate improvements to the signal itself: corrections to instructions, labeling templates, and LLM prompts.

More about analytics at Yandex - in the Yandex for Analytics channel

We expect that you

• Possess an analytical mindset, have worked with data
• Know the basics of probability theory and mathematical statistics
• Are proficient in Python and SQL for data analysis and processing
• Act proactively within your area of responsibility

Will be a plus

• Have experience with search engines
• Have experience with crowd platforms: Yandex Tasks, Toloka, MTurk, or equivalents
• Are familiar with prompt engineering and the principles of LLM operation
• Have participated and won in Olympiads in mathematics, physics, programming