👋Hello! My name is Irina. I represent the Top Selection group of companies.
🔥We are currently looking for a Data engineer 🔥
🛠 Requirements:
- TeamLead experience (team management) of at least 4 years;
- Experience in developing ETL/ELT processes where a significant part of the logic is implemented on the DBMS side;
- Experience in developing and maintaining stored procedures/functions in DBMS;
- Practical experience with SQL/PLSQL or similar SQL procedural extensions for data loading and transformation;
- Experience participating in ETL system migration projects (Apache stack);
- Development experience in Python (writing ETL scripts, working with pandas, API integration);
- Experience writing and debugging DAGs in Apache Airflow (not just design, but development);
- Experience working with PySpark (transformations, optimization, partitioning);
- Experience migrating ETL processes from Informatica PowerCenter to an open stack (Airflow + Greenplum);
- Deep practical knowledge of NiFi and Airflow: designing flows, connecting sources, error handling, monitoring, debugging;
- Experience with data quality: developing validation rules, analyzing impact on business processes;
- Understanding of architectural approaches to data warehouse design, differences between ETL and ELT;
- Experience working with Hive Metastore (structure, storage on PostgreSQL) and file formats (Parquet, ORC);
- Experience working with Oracle;
- Experience applying complex SQL queries (window functions, joins);
- Ability to optimize queries for a highload environment;
- Experience designing integrations of the Apache stack with Kafka and BI systems;
- Experience setting up streaming data processing;
- Experience working with Hadoop and Hive (knowledge of versions, data sources);
- Experience working with PostgreSQL, Greenplum, Clickhouse;
- Experience working with Grafana/Prometheus for monitoring ETL processes;
- Experience working with billing;
- Experience with automatic validation and data quality control after migration (result comparison, metrics);
📌 Responsibilities:
- Migration of ETL processes from Informatica PowerCenter to the target stack: NiFi + Airflow + Greenplum + SQL/stored procedures.
- Preparation of AS IS and TO BE documentation;
- Writing SQL code;
- Tracking metrics.
