Vacancy Data Engineer
π Location: Russian Federation or Republic of Belarus
π Work format: remote.
π Employment: Sole Proprietor / Civil Law Contract (SMZ)
π΅ Salary: from 262,000 rubles to 328,000 rubles
βΌοΈAbout the project:
We are looking for a data engineer with strong data and database architect competencies, who can take responsibility for planning and implementing data storage and processing tools. It is important to have confirmed experience with the project's technology stack (Postgres, MSSQL, Kafka, Flink, Airflow, Iceberg, Trino/StarRocks), be able to design target storage architecture, analyze data flows, and select relevant tools. Skills in working with modern data tools, experience in designing pipelines, ensuring data quality, and confident Python proficiency for writing DAGs, custom operators, and integrations are critical.
β
Tasks:
- Data Pipelines Development: Designing and implementing ETL/ELT pipelines: building DAGs in Apache Airflow, writing custom operators and sensors in Python, configuring retries, alerting, and dependencies.
- Streaming and Batch Processing: Developing streaming jobs on Apache Flink and batch processing on Apache Spark.
- Performance Optimization: Practical experience in identifying and resolving bottlenecks, optimizing slow SQL queries (analyzing execution plans, indexes, partitioning), tuning processing engine (Flink, Spark) and database configurations to minimize resource consumption (CPU, RAM, I/O).
- Data Quality Assurance: Organizing testing of data transformations, implementing mechanisms for checking data integrity and relevance, realizing normalization and deduplication processes.
- Integrations: Developing and supporting integrations between systems: REST/gRPC API, databases, message brokers (Kafka, MQTT), file storage (S3), Timeseries DB.
- Interaction with developers and testers to clarify technical limitations and requirements.
πRequirements:
- Experience as a Data Engineer for at least 4 years in software development.
- Experience with Postgres and MSSQL; experience in configuring and optimizing OLAP engines (Trino, StarRocks) and modern storage (Cassandra, Redis).
- Practical experience with Apache Airflow (writing DAGs, custom operators); experience using Apache Kafka (producer/consumer) and Apache Flink (developing streaming jobs, state, checkpoints).
- Experience with Apache Iceberg, understanding of lakehouse architecture, using Lakekeeper for metadata management, working with S3 object storage.
- Experience in implementing Change Data Capture patterns for synchronizing data between systems.
- Knowledge in data and database architecture: understanding principles of normalization and denormalization, designing relational and columnar models, data management strategies (lifecycle, retention, versioning).
- Proficiency in Python: writing scripts, modules, working with asynchronicity when needed; experience writing DAGs in Apache Airflow, including custom operators, hooks, sensors.
- Ability to profile processes, find bottlenecks, optimize SQL queries, and configure cluster resources for load.
- Experience in organizing data storage for subsequent OLAP / OLTP tasks considering requirements (consistency / integrity).
- Experience in organizing data processing within OLAP / OLTP tasks using various patterns (batch processing, CDC).
- Understanding of key features and application scenarios of various integration flow organization patterns (batch processing, CDC) considering their characteristics (3V+).
- Ability to structure information, work with requirements, document technical solutions.
πWill be a plus:
- Experience with Timeseries databases (InfluxDB, TimescaleDB, TDengine, etc.).
- Deep experience in operating clusters (Hadoop, YARN, Kubernetes) in a production environment.
- Advanced SQL knowledge (optimizing complex queries, window functions, CTE).
- Experience with CI/CD for data pipelines.
- Understanding of DDD, event-driven architecture principles.
- Experience with Apache NiFi.
