Middle+ / Senior Data Engineer (Python / Spark / Airflow / dbt / Data Vault 2.0)

Vacancy ID: 92465_SS Rate: To be discussed. Grade: Middle+ / Senior. Workload: Full-time. Engagement Term: 24 months with the possibility of extension. Format: Remote. Time Zone: UTC+3 (Moscow time). Location/Citizenship: Russian Federation / Russian Federation. ⏳ 🚨 URGENT: Vacancy is active until 27.10 inclusive until 17:00! 🚨 📌 Important: candidates with a relevance of experience of 85% or more are considered (outsourcing position). 🏢 About the project: a large retail chain. Building, developing, and optimizing advanced ETL/ELT pipelines, implementing Data Vault 2.0 methodology and modern Data Engineering practices (Data Contracts, Semantic Layer, DQ frameworks). 🧩 About the role: we are looking for a mature Data Engineer who can not only write high-quality code but also design data storage architecture, deeply optimize Spark tasks, build reliable CI/CD processes, and implement cutting-edge engineering practices on the scale of a large retail business. 🚀 Key Tasks: — Creating and developing Spark pipelines, including work with Structured Streaming. — Designing modular and dynamic DAGs in Airflow: developing custom operators, sensors, and hooks. — Building a layered model in dbt (staging, intermediate, marts): implementing incremental models, snapshots, macros, configuring Slim CI. — Designing Data Vault 2.0 structures (Hub, Link, Satellite) for new data sources. — Selecting data storage architecture and formats (Parquet, Iceberg), designing schemas considering expected load. — Configuring CI/CD processes in GitLab (dev, stage, prod environments; working with runners and the approvals mechanism). — Implementing DQ frameworks (e.g., Great Expectations), configuring data quality metrics and alerts. — Developing Semantic Layer (Cube, MetricFlow) and building lineage in DataHub. — Compiling and controlling data contracts. 🎯 Mandatory Requirements (Hard Skills): Experience and Languages (CRITICAL!): — 4+ years of experience in data engineering. — Proficient in Python: OOP, asynchronous programming, multithreading, typing, ability to write tests and perform code profiling. Big Data and Databases (CRITICAL!): — Advanced Spark skills: task optimization and tuning, analysis of execution plans and logs. — Deep understanding of SQL: working with EXPLAIN and query execution plans, index tuning, knowledge of specific RDBMS features. — RDBMS administration skills: partitioning, sharding, replication, backup and migration configuration, understanding locking mechanisms and transaction isolation levels. Tools and Orchestration (CRITICAL!): — Practical experience with Airflow: using XCom, configuring pools and SLAs, working with KubernetesPodOperator. — High level of proficiency with dbt. — Practical experience with Kubernetes and Docker: ability to optimize container images. — Working with S3: configuring lifecycle rules and managing object versioning. Processes and Monitoring: — Effective Git skills: proficiency in rebase and cherry-pick commands, organizing release versioning. — Ability to create dashboards and configure alerts in Grafana or similar tools. 💡 Will be a significant plus: — Experience in implementing and supporting Data Vault 2.0 in practice. — Experience with next-generation data formats (Iceberg, Delta Lake). — Understanding of the specifics of the retail domain and working with large volumes of transactional data.

📲 Contact for application: recruiter contact (⚠️ DEADLINE: 27.10 by 17:00! When applying, please immediately send your updated resume with an emphasis on Spark optimization experience, advanced dbt/Airflow proficiency, and Data Vault understanding. Please include the vacancy ID: 92465_SS)

📢 Source: