Streaming Systems Developer at YDB

YDB is an open-source, distributed, fault-tolerant relational database that combines high availability and scalability with strong consistency and ACID transactions. Together with the serverless technology stack, YDB enables the organization of a data storage and processing system for the most complex and demanding applications. It supports concurrent transactional (OLTP), analytical (OLAP), and streaming workloads.

YDB Topics is part of the YDB platform, an analog of Apache Kafka. We are continuously developing YDB Topics to respond to the growing needs of the company and external clients. Our topics are accessible via globally recognized standard protocols: Apache Kafka and Amazon Kinesis. We are developing integrations with Apache Spark, Apache Flink, Apache NiFi, Confluent Schema Registry.

Message delivery guarantees in YDB Topics

YDB Topics: A history of relations with Kafka

YDB — parallel asynchronous replication in a distributed database

What tasks await you

Developing new functionality We have many complex and interesting tasks: we have implemented and are developing the world's first system providing transactions between topics and tables, we ensure auto-scaling of topics based on load, integration of the platform with external systems, including data delivery to analytical tables.

Optimizing a high-load distributed system You will search for bottlenecks in the system and optimize them to work with hundreds of thousands of partitions on top of thousands of servers.

Open-source development We conduct most of our development on GitHub, so you can always show the world the results of your work. Furthermore, we frequently speak at various conferences in Russia and abroad.

More about backend at Yandex — on the channel Yandex for Backend

We expect you

• Have strong proficiency in one of the languages: C++, C#, or Java
• Are ready to develop in C++
• Understand key aspects of building fault-tolerant distributed systems
• Know classical algorithms and data structures
• Have experience developing multi-threaded applications

Will be a plus

• Experience working with Apache Kafka, Apache Pulsar, or other message brokers
• Experience developing high-load distributed systems
• Capable of creating new algorithms and protocols
• Have worked on optimization tasks