Backend Developer for DWH Team in Fintech

We are building a large-scale and secure analytical platform for working with big data.

We need a backend developer who will help build the DWH and accelerate data delivery by 2–3 times within the first 6 months.

What tasks await you

Development, scaling, and migration of the data platform You will design and develop a platform based on S3, Greenplum, Trino, Apache Airflow, Apache Spark, Yandex Cloud, prepare it for multiple growth in data volumes, and ensure migration from Greenplum to Trino/S3 to reduce costs and accelerate data access.

Integration of sources and data load management You will build extraction processes from relational DBMSs, NoSQL, file storage, and Kafka, implement load strategies (full or incremental with CDC, batch or stream processing), introduce Spark Streaming for near-real-time data updates, work with snapshots, increments, audit tables, and CDC.

Ensuring platform reliability and efficiency You will ensure the fault tolerance, security, performance, and scalability of services and data pipelines in 24/7 mode, optimize data processing costs and resource usage, deploy infrastructure and monitor it through CI/CD pipelines, reduce manual operations, and create new integrations.

Observability, quality control, and tool development You will create an observability platform for the DWH, which will help ensure build status monitoring, data quality, Data lineage analysis, and logging. You will be involved in data versioning, testing pipelines, developing internal frameworks and low-code solutions for data loading and transformation.

Security and development organization You will need to ensure automated granting and control of access rights to infrastructure and data, conduct user action audits, create isolated development, testing, and pre-production environments with automated testing, build a secure development environment, and increase solution reliability.

More about backend at Yandex — in the channel Yandex for Backend

We expect you to have

• At least four years of commercial development experience
• Built processes listed in the key tasks
• Optimized complex analytical queries, worked with execution plans
• Designed data schemas, handled partitioning, sharding, created effective indexes
• Worked with various types of DBMS: relational (PostgreSQL, Oracle), NoSQL (MongoDB), MPP systems (Greenplum)
• Managed data in S3-compatible storage
• Developed pipelines in Apache Airflow
• Know how to perform distributed processing with Apache Spark (Spark SQL, DataFrame)
• Can work with data through Trino
• Understand CDC (Change Data Capture) principles and have worked with relevant tools or implemented CDC patterns with your own means