👋Hi! My name is Irina. I represent the Top Selection group of companies.
🔥We are currently looking for a Data Engineer 🔥
🛠 Requirements:
- TeamLead experience (team management) of at least 4 years;
- Experience in developing ETL/ELT processes, where a significant part of the logic is implemented on the DBMS side;
- Experience in developing and maintaining stored procedures/functions in DBMS;
- Practical experience using SQL/PLSQL or similar SQL procedural extensions for data loading and transformations;
- Experience participating in ETL system migration projects (Apache stack);
- Development experience in Python (writing ETL scripts, working with pandas, API integration);
- Experience writing and debugging DAGs in Apache Airflow (not just design, but development);
- Experience working with PySpark (transformations, optimization, partitioning);
- Experience migrating ETL processes from Informatica PowerCenter to an open stack (Airflow + Greenplum);
- Deep practical knowledge of NiFi and Airflow: flow design, source connection, error handling, monitoring, debugging;
- Experience working with data quality: developing validation rules, analyzing impact on business processes;
- Understanding of architectural approaches to designing data warehouses, differences between ETL and ELT;
- Experience working with Hive metastore (structure, storage on PostgreSQL) and file formats (Parquet, ORC);
- Experience working with Oracle;
- Experience applying complex SQL queries (window functions, joins);
- Ability to optimize queries for a highload environment;
- Experience designing integrations of the Apache stack with Kafka and BI systems;
- Experience in setting up real-time data processing;
- Experience working with Hadoop and Hive (knowledge of versions, data sources);
- Experience working with PostgreSQL, Greenplum, Clickhouse;
- Experience working with Grafana/Prometheus for monitoring ETL processes;
- Experience working with billing;
- Experience in automatic validation and control of data quality after migration (comparison of results, metrics);
📌Responsibilities:
- Migration of ETL processes from Informatica PowerCenter to the target stack: NiFi + Airflow + Greenplum + SQL/stored procedures.
- Preparation of AS IS and TO BE documentation;
- Writing SQL code;
- Tracking metrics.
