We are looking for a Data Engineer to help develop the data platform: from streaming event loading and model building to cold storage and data quality control.
Responsibilities
- Develop the streaming of events from RabbitMQ to ClickHouse: connect new streams, set up deduplication and backfill of historical data;
- Build a cold storage on Apache Iceberg: unload the archive from ClickHouse via Spark, transfer historical data, develop the prepared layer and history of directories;
- Configure incremental data loading from 1C and PostgreSQL replication, and in the future, implement CDC;
- Develop data marts and data models on dbt and ClickHouse materialized views;
- Daily reconcile data completeness and sums with sources, set up data quality tests, monitor freshness and volume of streams;
- Orchestrate processes in Airflow, develop CI/CD for models and DAGs, maintain runbooks for streams;
- Participate in architectural decisions and code reviews, develop the data catalog.
Requirements
- 4–5 years of experience in Data Engineering;
- Industrial experience with ClickHouse: MergeTree engines, materialized views, replication, and query optimization;
- Solid knowledge of SQL and Python;
- Production experience with dbt: models, tests, and CI;
- Experience developing and operating DAGs in Apache Airflow;
- Experience with Spark and table formats for lakehouse — Iceberg, Delta, or Hudi; understanding of S3-compatible storage;
- Experience with streaming data loading from message brokers — RabbitMQ or similar; understanding of at-least-once and idempotency;
- Understanding of medallion architecture, SCD2, and incremental loading;
- User-level knowledge of Docker and Kubernetes, experience with Git.
Bonus Points
- Experience loading data from MS SQL or 1C;
- Knowledge of Debezium and CDC;
- Experience with dlt and OpenMetadata;
- Familiarity with payment and personal data;
- Experience with on-prem infrastructure.
Conditions
- Competitive salary — to be discussed after the interview;
- Remote work format;
- 5/2 schedule: from 10:00 to 19:00 or from 11:00 to 20:00;
- Powerful work laptop;
- 100% reimbursement of sick leave;
- Corporate discount on English language learning at Skyeng.
