Advertising is one of the most high-load services at Yandex, handling up to 700K RPS and operating with billions of ads.

We are involved in the preparation, processing, and delivery of data for the serving runtime, as well as rapid statistics collection: calculation of neural network models, banner generation, and building smart k-NN indexes. Our systems process tens of gigabytes of compressed data per second and operate with a state of hundreds of terabytes.

Some time ago, we made a quantum leap when we migrated key components from the MapReduce paradigm to streaming data processing.

Our conference talks:

• Efficient state updates in a database from streaming processing services * How Yandex Advertising generates titles for three billion ads using GPT neural networks

What you will get if you join us:

• You will learn how to properly configure YTSaurus dynamic tables to withstand any load * You will learn to write multithreaded code in C++ and learn the difference between fibers and threads * You will understand how changing an allocator can speed up a system by 30% — or slow it down * You will figure out how to build truly exactly-once systems

What tasks await you

Fast Advertising Data For business, it is extremely important that any event — be it a change in product price or a user click — is accounted for in the final ranking as quickly as possible. Our task is to reduce these delays to minutes and seconds. We have already made fast profiles of all major advertising entities. Now we are working on incremental updates of all advertising databases and indexes.

Real-Time Machine Learning Apart from simply delivering updated profiles to the runtime, there is a separate task of retraining neural network models on fresh data. We are taking the first steps towards moving dataset construction from MapReduce to RT. We need to build a system capable of processing over 10 GB/s of input data with a minute delay, performing a windowed join of several logs simultaneously. For this, we are actively developing our own streaming data processing framework.

Yandex Advertising Unified Database The database with advertising data must simultaneously possess properties that at first glance seem conflicting: the ability to store tens of TB of data, provide high-load synchronous access, and support regular massive asynchronous non-blocking modifications. Therefore, we are developing our own relational database on top of YTSaurus dynamic tables.

We expect you to

• Have experience in C++ development or a desire to learn it

Will be a plus

• Familiar with YTSaurus, Hadoop, Spark, Flink, Kafka