Data Engineer in Travel

Company: Yandex
Location: Moscow
Employment Type: Full-time
Qualification: Senior
Specialization: Data Engineer

Our team transforms terabytes of data into understandable and useful insights that are used by analysts and a service that helps millions of people travel. Our product is growing at a tremendous pace, and with it, our tasks are growing, for example, the development of our own DWH.

Our data warehouse is built on YTsaurus data storage systems (a MapReduce cluster of our own development) and ClickHouse. For the development, orchestration, and monitoring of ETL processes, our own framework, created within Yandex, is used. Various data warehouse design methodologies are used to organize the data storage structure at different levels: Lambda architecture, Data Lake, Data Vault.

What tasks await you:

• Development, maintenance, and review of ETL processes for data delivery

You will create pipelines for extracting, transforming, and loading data from various sources, configure their scheduling and monitoring. It will be important to regularly check and improve existing code, promptly fix emerging errors, and thoroughly document all transformations and data flows — both your own and your colleagues', ensuring compliance with quality and performance standards.

• Designing the data storage structure in different layers of the warehouse

You will define optimal data organization schemes for each layer, considering requirements for query speed and storage volume. The task includes developing data models, configuring partitioning, indexing, and compression to improve performance, as well as coordinating structures with analytics and development teams and adapting schemes when business processes change.

• Identifying and optimizing inefficient parts in data delivery processes

You will need to analyze the execution time of ETL tasks, profile data to detect anomalies and redundancy, and revise transformation algorithms to reduce system load.

• Immersing yourself in data from applied business areas and becoming an expert in them

You will study business processes and key company metrics in detail, understand the logic of indicator calculations and their formation sources.

We expect you to:

• Have designed DWH (layers, data storage models, ETL processes)
• Have developed ETL processes on Apache Airflow or similar
• Have configured data delivery via API
• Understand SQL at an expert level
• Write in one of the high-level languages (Python, Go, Java, etc.)
• Understand the structure and architecture of relational databases
• Know the main algorithms and data structures, especially those used in big data processing

It will be a plus if you:

• Have used at least one MPP DBMS (Greenplum, ClickHouse, Vertica, Teradata, etc.)
• Understand the principles of distributed data processing technologies (Hadoop, MapReduce, and MPP)

What you will get:

• Scale: working with Yandex's infrastructure and data presents challenges rarely encountered even in large international companies
• Impact: your decisions will directly influence the efficiency of key business processes
• Expertise: the opportunity to share experience with the strongest specialists in the industry and grow professionally
• Resources: access to cutting-edge technologies and tools to implement the boldest ideas
• Career prospects: opportunities for professional and leadership growth within one of Russia's leading IT companies