Technical Project Manager at YTsaurus ML Platform

Yandex is one of the few companies in the world handling exabytes of data that need to be efficiently stored, processed, and transferred between various systems.

To solve these tasks, we have created and are developing a distributed data storage and processing platform called YTsaurus.

Among other things, YTsaurus manages Yandex's supercomputers, on which ML engineers solve machine learning tasks. Recently, we started a new project to simplify the lives of our colleagues. The users of this new service will be the majority of Yandex's ML developers — hundreds of teams that train models for Alisa, Neuro, and SHeidrum.

Key components of the new service:

• Managed DevCluster for remote development and test runs of tasks on GPU
• A tool for launching training processes that is simple from a UX perspective and efficient on the giant YTsaurus cluster with thousands of GPUs
• An orchestrator that will allow describing data preparation graphs in Python and SQL
• A solution for tracking ML experiments (similar in concept to Weights & Biases)
• An MLOps system for storing and managing the lifecycle of models

We are looking for a person who will help us establish development processes and improve interaction between ML engineers and infrastructure.

What tasks await you

Interaction with users You will have extensive communication with internal and external users, participate in shaping best practices for using ML infrastructure (for example, monitoring the efficient use of GPU resources). Additionally, you will collect and analyze the needs of ML engineers and communicate suggestions for improvements across the entire stack.

Technical challenges You will need to understand how the ML model development process is organized at Yandex, lead internal infrastructure projects, and participate in integration projects together with several development, operations, and testing teams.

Building workflows You will collaborate with developers of other infrastructure services, analyze processes, identify pain points, propose and implement improvements. Furthermore, it will be necessary to manage the development process and ensure its alignment with plans and requirements.

More about product creation at Yandex — in the channel Yandex for Products

We expect that you

• Have a higher technical or mathematical education
• Have worked in a similar position for the last year
• Have managed projects and products
• Have worked with a development team, will find a common language with them
• Have directly developed software
• Have written FAQs and user documentation
• Are capable of managing discussions, capturing results, setting priorities, and solving tasks on time
• Are communicative, speak and write accurately and clearly

Will be a plus

• Are familiar with infrastructure systems (Docker, Kubernetes, Ray, Slurm)
• Have participated in the creation and implementation of infrastructure solutions in large companies
• Write in Python or any other programming language