ML Developer for the VLM Foundations Team

Autonomous technologies make our lives easier and better, but they need constant development to enhance comfort, safety, and accessibility.

Yandex aims to bring autonomous technologies to a new level and implements cutting-edge ML developments. We create technologies that enhance safety and care for the comfort of passengers and other road users.

We are experimenting with VLMs — they have great potential, and we want to utilize it for our tasks.

What tasks await you

Fine-tuning VLMs for driving You will fine-tune a vision-language model to predict trajectories, add reasoning to improve action interpretability, and expand knowledge in the field of driving and understanding of the road situation, road markings, signs, traffic lights, and the actions of other cars and pedestrians. You will also make architectural changes: add encoders for new modalities (e.g., maps and lidars) and implement methods to speed up model inference — from action modality injection and speculative decoding to flow matching head.

Developing RL approaches for VLMs You will develop and implement advanced RL methods to teach VLMs to make safer, more efficient, and more human-understandable decisions. You will tackle non-trivial tasks: creating robust reward functions, combating reward hacking, and building a scalable training system in simulation.

Implementing SOTA solutions You will keep track of modern approaches, study relevant papers, and implement SOTA solutions, as well as quickly test hypotheses.

More about ML at Yandex — in the channel Yandex for ML

We expect you to

• Understand the principles of machine learning and the workings of NLP models
• Understand classical algorithms and data structures
• Have trained and deployed deep learning models into production
• Have worked with large volumes of data
• Know Python
• Keep up with the development and latest achievements in the field of machine learning, read papers from top conferences

Will be a plus

• Experience with VLMs
• Experience in distributed training