Description
Project: We are creating a search service to answer user queries in natural language. We are breaking down the barrier between the static knowledge of a language model and the constantly changing world. We provide GigaChat with access to up-to-date information so users get accurate answers to any questions, including questions about the latest news and events.
Responsibilities
- develop and configure mechanisms for automated data collection, ensure the correctness and completeness of collection, optimize processes so that everything works faster and without manual intervention
- develop pipelines for data preprocessing and transform it into a format optimal for further storage, processing, and use in search tasks
- design and implement storage systems that would allow efficient solving of search tasks
- apply machine learning and artificial intelligence to improve system performance, maintain proper system operation – monitoring, diagnostics and troubleshooting, fixing old bugs and creating new ones.
Requirements
- experience with engines for distributed data processing (Spark, Trino), orchestrators Airflow
- ability to design DWH, Data Lake, Data Management Platform
- experience in building and scaling high-load systems
- experience in developing and optimizing (batch, streaming) pipelines for processing large volumes of data (100TB - 1PB+)
- advanced level of Python and SQL proficiency
WILL BE A PLUS:
- experience with Iceberg format tables
- experience with ElasticSearch/OpenSearch indexes
- experience with GPU (model inference).
Conditions
- comfortable modern office - Kutuzovskaya metro station
- annual salary review, annual bonus
- corporate gym and relaxation areas
- more than 400 educational programs from SberUniversity for professional and career development
- extended voluntary health insurance, preferential insurance for family and corporate pension program
- flexible discount on a mortgage loan, equal to 1/3 of the Central Bank key rate
- free SberPrime+ subscription, discounts on products from partner companies
- referral bonus for recommending friends to the Sber team
- corporate pension program.
