Reach out directly about this role
Go Developer for the Detection & Response team, Yandex Cloud Security
We are looking for Go developers to work on services and tools for the Detection & Response direction within the Yandex Cloud Security Team. We are only at the very beginning of the journey—so we haven't yet had time to accumulate technical legacy. We have two similar but slightly different directions:
Accordingly, the focal points of effort differ slightly. In the first case, we focus on large volumes of data and complex data processing workflows, while in the second we are concerned with architectural scaling.
Information security services for Yandex Cloud process tens and hundreds of thousands of objects per second, store petabytes of data, and perform search on them. These services are used by both Yandex Cloud's cybersecurity defenders and its clients. We create systems whose application content is developed by experts protecting large-scale installations. To allow experts to populate systems with knowledge independently of programmers, we provide them with a set of DSLs for solving specific security tasks. Our systems are essentially an engine controlled by scripts in several languages.
What we offer:
Follow our telegram channel Inside Yandex Cloud to learn more about our team and technologies!
Creating compilers for narrowly focused DSLs for various data processing stages Our services are built around continuous (and infinite) processes of collecting and analyzing large data streams. Processing this data goes through several stages, and at each stage, new formats, new types, and new combinations of interest to us constantly appear. Since data changes more often than the component processing it, we create a language for each processing stage. Each stage has its own specifics and must scale well; accordingly, the DSLs for all processing stages are different. We have three in-house languages and the popular KQL.
Developing architecture for distributed, scalable data processing clusters We need to process large data streams while maintaining isolation of all clients' data from each other and supporting automatic scaling as the number of Yandex Cloud clients grows. Such a service must consume a predictable and limitable amount of resources.
Here, the ability to design compact, clear gRPC APIs that conform to organizational conventions and will require minimal evolution in the future years of the system's development will come in handy.
Research to select optimal algorithms and data structures and their application in development To collect a data stream into a correct, time-ordered data structure, you need to choose it correctly and efficiently build a picture with a timeline from the incoming stream. It is highly unlikely that a "brute force" solution will be effective. Before starting to write code, you will need to conduct several experiments, perform measurements under different loads, and choose the most efficient solution.
Large-scale data stream processing tasks Our services must perform efficient search across petabytes of data, scale on the fly without restarting, automatically increase and decrease the number of servers in the processing cluster depending on the load, and operate 24/7 without downtime.
You will need to learn to design services in such a way that they do not consume all the computational resources of the universe for a query on unreasonable volumes of data with an unreasonable filter over an unreasonable time interval. Example task: create a way to simultaneously read hundreds of TiB of data from a single source at a time by several parallel queries with different priorities, enriching them on the fly and delivering them within a predictable timeframe.
5 years
Experience
Full-time
Employment
Hybrid, Onsite
Work Format
Middle
Grade
Backend
Specialization
IT & Tech
Industry
Corporation
Company Type
By city
Backend
Specialization
IT & Tech
Industry
Corporation
Company Type