Hello everyone! We are looking for a talented Data Engineer who is ready to tackle complex challenges in the field of storing and processing big data on a petabyte scale! You will have a unique opportunity to participate in the development of a data platform for working with telemetry of Moscow's autonomous transport - trams and metro, including: developing the DWH architecture, connecting new services, and improving the platform's performance and stability.

What you will do:

• Transform autonomous transport telemetry into data that is convenient for analysis: develop and launch Spark pipelines, think through transformations and quality checks
• Connect new data sources: develop integrations with various systems and ensure reliable data loading into the storage
• Develop the Data Lake / DWH architecture
• Together with the team, form common approaches to development, monitoring, and maintenance of pipelines

What we expect:

• Practical experience in developing and supporting data pipelines
• Experience with PySpark, Python, and SQL
• Experience with Data Lake / DWH and understanding the principles of building ETL/ELT processes
• Understanding the principles of distributed data processing

It will be a plus:

• Experience with Trino+S3
• Experience in orchestrating pipelines using Airflow
• Familiarity with Data Vault 2.0
• Knowledge of Scala and experience using it for data processing
• Experience in developing a data platform in its early stages: from selecting solutions to their implementation and support

Contact person: recruiter contact