☁️Position: Data Engineer / Technical Lead (Senior / Lead)
🏙Company: «HuntTech»
💪Experience: 5+ years
💰Salary: Range: 250 - 310k RUB
📅Work Format: Remote
📍Location/Citizenship: RF
📅Employment: Full-time
☎️Contact: @pavel_korab
📌Tasks:
- Migration of ETL processes from Informatica PowerCenter to the target stack: NiFi + Airflow + Greenplum + SQL/stored procedures.
- Preparation of AS IS and TO BE documentation;
- Writing SQL code;
- Tracking metrics.
🔺Requirements:
- Experience in developing ETL/ELT processes where a significant part of the logic is implemented on the DBMS side;
- Experience in developing and supporting stored procedures / functions in DBMS;
- Practical experience using SQL/PLSQL or similar SQL procedural extensions for implementing data loading and transformations;
- Experience participating in ETL system migration projects (Apache stack);
- Development experience in Python (writing ETL scripts, working with pandas, API integration);
- Experience writing and debugging DAGs in Apache Airflow (not just design, but development);
- Experience working with PySpark (transformations, optimization, partitioning);
- Experience migrating ETL processes from Informatica PowerCenter to an open stack (Airflow + Greenplum);
- Deep practical knowledge of NiFi and Airflow: flow design, source connection, error handling, monitoring, debugging;
- Experience working with data quality: developing validation rules, analyzing impact on business processes;
- Understanding of architectural approaches to designing data warehouses, distinction between ETL and ELT;
- Experience working with Hive metastore (structure, storage in PostgreSQL) and file formats (Parquet, ORC);
- Experience working with Oracle;
- Experience applying complex SQL queries (window functions, joins);
- Ability to optimize queries for a high-load environment;
- Experience designing integrations of the Apache stack with Kafka and BI systems;
- Experience in setting up stream data processing;
- Experience working with Hadoop and Hive (knowledge of versions, data sources);
- Experience working with PostgreSQL, Greenplum, ClickHouse;
- Experience working with Grafana/Prometheus for tracking ETL processes;
- Experience with billing;
- Experience in automatic validation and control of data quality after migration (comparison of results, metrics);
➕Will be a plus:
- Experience in ETL system analysis;
- Team Lead experience (team management) of at least 4 years;
- Ability to translate business requirements into technical specifications;
- Experience preparing documentation for users and developers;
- Experience creating ER diagrams, instructions, test cases.
