We are looking for a Data Engineer to participate in the development of a system that processes sales information across Russia. Our team is involved in creating ETL processes and aggregates for data marts and reports for BI and end-users. The volume of source data in the project is huge - several petabytes, hundreds of ETL processes are working on building data marts, and tens to hundreds of terabytes of information are processed daily. If you are ready to take on the challenge of working with such data scales, join us!
Technology Stack: Scala, Apache Spark, ClickHouse, HDFS, Apache Airflow, PostgreSQL, Apache Kafka, Apache Hive, Apache Iceberg
What you will be doing:
- Building and supporting ETL processes and solving various business tasks for processing large volumes of data using Spark+Scala
- Optimizing data processing speed and system resource consumption
- Identifying and resolving the causes of errors and anomalies in resulting data
- Creating optimal medium-complexity SQL queries for analyzing large data volumes according to business requirements
What we expect from the candidate:
- Proficiency in Scala, Java, or readiness to quickly master Scala basics
- Basic Linux command-line skills
- Experience with Spark, a solid understanding of its operating principles and potential issues
- Knowledge of database fundamentals, strong SQL skills
Will be a plus:
- Experience with Scala Dataframe/Dataset API
- Experience in building and orchestrating ETL processes for Big Data processing
- Experience in optimizing Spark queries and configuring consumed resources
- In-depth understanding of Spark operating principles and configuration parameters
- Experience with Zeppelin or Jupyter
- Experience with ClickHouse or other NoSQL databases
- Experience with Apache Airflow
- Knowledge of Hadoop/HDFS, working with Parquet files, working with Hive
- Ability to work with GitLab CI
- Sufficient Bash knowledge to make corrections to scripts
- Basic Python knowledge
- English at B1 level or higher
We offer the following conditions:
- Work in a dynamic international team
- Participation in foreign and Russian projects
- Employment according to the Labor Code of the Russian Federation, 100% payment of sick leave and vacation
- Voluntary medical insurance with dental coverage
- Necessary equipment for work
- Corporate training programs
- Broad opportunities for self-realization, professional and career growth
- Democratic approach to processes and flexible start of the workday.
In your cover letter, please indicate:
- Are you ready to develop using Spark+Scala?
- Approximately how many ETLs have you implemented and what was their complexity?
- What was the maximum volume of data processed within a single ETL?
- Do you have experience optimizing resource consumption by Spark applications, especially on a YARN cluster?
- What is your current geographical location?
