Go Developer for the Detection & Response team, Yandex Cloud Security

We are looking for Go developers to work on services and tools for the Detection & Response direction within the Yandex Cloud Security Team. We are only at the very beginning of the journey—so we haven't yet had time to accumulate technical legacy. We have two similar but slightly different directions:

• Developing tools for the internal security needs of Yandex Cloud and its major clients.
• A set of services that will be available to all Yandex Cloud clients.

Accordingly, the focal points of effort differ slightly. In the first case, we focus on large volumes of data and complex data processing workflows, while in the second we are concerned with architectural scaling.

Information security services for Yandex Cloud process tens and hundreds of thousands of objects per second, store petabytes of data, and perform search on them. These services are used by both Yandex Cloud's cybersecurity defenders and its clients. We create systems whose application content is developed by experts protecting large-scale installations. To allow experts to populate systems with knowledge independently of programmers, we provide them with a set of DSLs for solving specific security tasks. Our systems are essentially an engine controlled by scripts in several languages.

What we offer:

• New projects, which haven't yet accumulated the technical legacy of previous generations
• Work in a close-knit, though geographically distributed team
• Non-trivial services from a product standpoint
• A stream of interesting and complex tasks, discussed collectively with a truly collegial decision-making process

About the team

Follow our telegram channel Inside Yandex Cloud to learn more about our team and technologies!

What tasks await you

Creating compilers for narrowly focused DSLs for various data processing stages Our services are built around continuous (and infinite) processes of collecting and analyzing large data streams. Processing this data goes through several stages, and at each stage, new formats, new types, and new combinations of interest to us constantly appear. Since data changes more often than the component processing it, we create a language for each processing stage. Each stage has its own specifics and must scale well; accordingly, the DSLs for all processing stages are different. We have three in-house languages and the popular KQL.

Developing architecture for distributed, scalable data processing clusters We need to process large data streams while maintaining isolation of all clients' data from each other and supporting automatic scaling as the number of Yandex Cloud clients grows. Such a service must consume a predictable and limitable amount of resources.

Here, the ability to design compact, clear gRPC APIs that conform to organizational conventions and will require minimal evolution in the future years of the system's development will come in handy.

Research to select optimal algorithms and data structures and their application in development To collect a data stream into a correct, time-ordered data structure, you need to choose it correctly and efficiently build a picture with a timeline from the incoming stream. It is highly unlikely that a "brute force" solution will be effective. Before starting to write code, you will need to conduct several experiments, perform measurements under different loads, and choose the most efficient solution.

Large-scale data stream processing tasks Our services must perform efficient search across petabytes of data, scale on the fly without restarting, automatically increase and decrease the number of servers in the processing cluster depending on the load, and operate 24/7 without downtime.

You will need to learn to design services in such a way that they do not consume all the computational resources of the universe for a query on unreasonable volumes of data with an unreasonable filter over an unreasonable time interval. Example task: create a way to simultaneously read hundreds of TiB of data from a single source at a time by several parallel queries with different priorities, enriching them on the fly and delivering them within a predictable timeframe.

We expect that you

• Are capable of creating scalable architectures
• Have experience with high loads and complex data structures
• Know how and want to write clean and optimized code in Go and Java
• Want and are able to learn and apply new things in daily work
• Strive for independence, are ready to make technical decisions
• Understand information security (this is not mandatory, but experience and knowledge in the field of IS will be useful in your work)