Researcher-Developer for the AI Tools Effectiveness Evaluation Team in Advertising

The Metrics team for Advertising develops key metrics for Yandex's Advertising product, one of the company's most technologically advanced and profitable businesses.

Our Main Projects

• The primary acceptance metric for Advertising A/B experiments — the main KPI for many technical teams in Advertising. In developing this metric, we have already solved tasks related to proper conversion attribution, evaluating conversion value using AI, and deprecating ineffective placements.
• Developing and maintaining over 6000 Advertising metrics — working with a high-load distributed system, processing several petabytes of data per day, and conducting research.
• KPI Dashboard — a convenient analytics tool for all Advertising A/B experiments. Using the KPI Dashboard, goals are set for teams and results are approved.
• Evaluating the effectiveness of AI development tools — a new project for which we are seeking a researcher-developer.

Within the team, we work on tasks across the entire development cycle: from research and hypothesis testing to implementing tools and deploying them to production.

What tasks await you

Developing metrics for evaluating the effectiveness of AI development tools According to recent studies, tools based on AI technologies accelerate the development process, but not as significantly as predicted. At Yandex, we are developing metrics that objectively show the benefit from using breakthrough technologies at different stages of implementing new features.

Researching promising directions for optimizing the development process within the company An analytical approach to developing metrics will help you more accurately assess the benefits of technology adoption and find insights.

Visualizing metrics and presenting results to technical leaders of business groups You will create a convenient and fast dashboard for the development process efficiency, which will help departments set optimization goals and evaluate the effects of implementing new tools.

More about Backend at Yandex — in the channel Yandex for Backend

We expect that you

• Confidently write fast and reliable code in Python
• Are proficient in YQL and can optimize queries for large volumes
• Have a good knowledge of mathematics and statistics: correctly evaluate effects, understand the physical meaning of metrics
• Can research and test hypotheses on data (formulation → calculation → conclusions)
• Are interested in modern AI development tools and want to understand what exactly needs to be measured in them

It will be a plus

• Have worked with big data and high-load systems
• Have designed and maintained metrics, including dashboards and monitoring
• Possess an ML/DS background (can evaluate models and metrics, understand sources of bias)