Data Engineer (Scala)
We are looking for a Data Engineer to participate in the development of a system that processes sales information across Russia. Our team is involved in creating ETL processes and aggregates for building data marts and reports for BI and end-users. The volume of source data in the project is enormous – several petabytes, hundreds of ETL processes work on building data marts, and tens to hundreds of terabytes of information are processed daily. If you are ready to take on the challenge of working with such data scales, join us!
Technology Stack: Scala, Apache Spark, ClickHouse, HDFS, Apache Airflow, PostgreSQL, Apache Kafka, Apache Hive, Apache Iceberg
Responsibilities:
- Building and supporting ETL processes and solving various business tasks for processing large data volumes using Spark+Scala
- Optimizing data processing speed and system resource consumption
- Identifying and eliminating the causes of errors and anomalies in the resulting data
- Creating optimal medium-complexity SQL queries for analyzing large data volumes according to business requirements
Candidate Profile:
- Proficiency in Scala, Java, or willingness to quickly learn the basics of Scala
- Basic Linux command-line skills
- Experience with Spark, confident understanding of its operating principles and potential issues
- Knowledge of database fundamentals, confident SQL skills
Will be a plus:
- Experience with Scala Dataframe/Dataset API
- Experience in building and orchestrating ETL processes for Big Data processing
- Experience in optimizing Spark queries and configuring consumed resources
- In-depth understanding of Spark operating principles and configuration parameters
- Experience with Zeppelin or Jupyter
- Experience with ClickHouse or other NoSQL databases
- Experience with Apache Airflow
- Knowledge of Hadoop/HDFS, working with Parquet files, working with Hive
- Ability to work with GitLab CI
- Sufficient Bash knowledge to make script corrections
- Minimal Python knowledge
- English at B1 level or higher
We offer:
- Work in a dynamic international team
- Participation in international and Russian projects
- Employment according to the Labor Code of the Russian Federation, 100% payment of sick leave and vacation
- Voluntary medical insurance with dental coverage
- Necessary equipment for work
- Corporate training programs
- Broad opportunities for self-realization, professional and career growth
- Democratic approach to processes and flexible start of the working day.
