REMOTE RUSSIA | VK
The Trust&Safety team is developing a platform for building data processing pipelines in a self-service format.
This platform is used to build high-load data processing pipelines that process millions of texts and videos daily. The platform allows applying ML models to content, aggregating data from various sources, and delivering it to consumers with minimal latency.
Tasks
- Development of data processing pipelines: creation and deployment of new processing processes for search, moderation, and recommendations
- Integration with ML: implementation of ready-made ML models into data processing pipelines
- Performance optimization: ensuring minimal latency in content delivery to distribution services
- Working with Big Data: processing large volumes of data in batch and streaming modes
- Architecture: building functionality in a new platform, considering strict non-functional requirements
Requirements
- 5+ years of industrial development experience with Java
- Experience with distributed systems and databases (Postgres, YTsaurus will be a plus)
- Understanding of the limitations and capabilities of distributed and high-load systems, making decisions considering this context
- Practical experience with application metrics — sending metrics, building dashboards and alerts
- Proactiveness, ability to work in a team and solve complex architectural problems, close tasks end-to-end
Bonus Points
Big Data skills: experience processing large volumes of data in batch and streaming modes.
