C++ Developer for the YQL over YT development group

YTsaurus is one of Yandex's key infrastructure systems, used for storing and processing data.

Yandex works with exabytes of data that need to be efficiently stored, processed, and transferred between applications. For this purpose, we created YTsaurus — a distributed platform that combines a unified storage with a range of tools for processing data for various scenarios. Almost every service at Yandex uses our platform to solve its tasks. The platform is actively developed, and its source code is open-source.

Inside YTsaurus, there is support for a whole range of SQL dialects that allow accessing data. One such dialect is YQL. It transforms SQL queries into a sequence of MapReduce operations and/or executes them in the DQ engine, which you will be developing.

The DQ engine, unlike MapReduce, allows parts of a query to be executed entirely in memory without intermediate materialization to disk. This significantly reduces overhead and the time spent executing the entire query.

What tasks await you

Development of a hybrid query execution mode During query execution, the data volume often decreases at each stage — due to filtering, intersections, and aggregations. At a certain stage, using MapReduce may become excessive. In such cases, it is advantageous to switch to executing the remaining operations in the DQ engine.

Custom DQ cluster User workloads can be so diverse that it is difficult to find universal parameters for the engine's operation. We want to implement the ability to configure a custom DQ cluster for specific scenarios and load profiles.

Accelerating YQL over YT on benchmarks YQL over YT at Yandex runs on tens of thousands of machines and hundreds of thousands of cores. It is important for us to constantly search for points to optimize query execution. We use popular benchmarks to identify opportunities for performance improvement.

Publications and product contributions Your results can become the basis for publications, presentations, and speeches at major conferences, and your commits will go into the shared YTsaurus codebase.

Learn more about backend at Yandex — in the channel Yandex for Backend

We expect you to

• Have confident command of C++ and be ready to develop high-performance systems with it
• Understand how to analyze performance, find bottlenecks, and be ready to propose ways to solve problems
• Know Python at a level sufficient for writing tests

It will be a plus

• Have an understanding of how DBMS are structured and have participated in their development
• Know one of the SQL dialects