We are looking for Data Engineers to participate in a large-scale project of migrating a telecom operator's ETL system from Oracle/Informatica to the Apache stack.
🏢 Industry: Telecommunications
👤 Grade: Senior
📍 Location and citizenship: Russian Federation
🏠 Format: remote
📄 Employment type: only for individual entrepreneurs
🕘 Employment: Full-time
⏰ Schedule: MSK ±2 hours
🛠 Key stack: SQL, Oracle (PL/SQL), PostgreSQL, Greenplum / ClickHouse, Apache Spark (PySpark), Apache NiFi, Apache Airflow, Hadoop (HDFS), Hive, Kafka, S3, Grafana, Prometheus / VictoriaMetrics.
📌 Tasks:
- Migration of the ETL system from Oracle/Informatica to the Apache stack.
- Transfer of business logic from Informatica to high-performance SQL.
- Development and optimization of SQL queries for data marts and ETL processes.
- Development of DAGs in Airflow and configuration of pipelines in NiFi.
- Development of Spark jobs (PySpark) for processing large volumes of data.
- Integration with Kafka for real-time data streaming.
- Setting up monitoring of ETL processes and tracking metrics.
- Analysis of incidents related to data quality and loading errors.
- Conducting Code Reviews.
✅ Mandatory requirements:
- Experience in migrating ETL systems, preferably from Oracle/Informatica to the Apache stack.
- Deep knowledge of SQL: window functions, complex JOINs, hierarchical queries.
- Experience in optimizing SQL for Highload: partitioning, indexes, Explain Plans.
- Confident command of Oracle (PL/SQL) and PostgreSQL.
- Experience with Greenplum and/or ClickHouse.
- Experience in designing solutions on Apache Spark (PySpark), NiFi, Airflow. Knowledge of Hadoop (HDFS) and Hive.
- Experience in monitoring ETL via Grafana/Prometheus or VictoriaMetrics.
- Experience with billing systems (telecom domain will be an advantage).
⭐ Will be a plus:
- Kafka Streams / Spark Streaming. Informatica PowerCenter.
- Binary data storage formats in Hadoop.
- Understanding the differences between Hive / Impala, Greenplum / PostgreSQL, ClickHouse / PostgreSQL.
- Patroni for PostgreSQL/Greenplum clustering.
- Experience with S3 storage.
📩 Interested in the vacancy? Send your CV to Telegram recruiter contact
