Senior Data Engineer (f/m/d)
What You Will Do
- Scale the Feature Store: Leverage Apache Flink to translate Data Science requirements into high-performance, real-time streams. Enable Data Scientists to contribute new features and implement them yourself when needed.
- Ensure Data Quality, Observability & Integrity: Implement data validation, monitoring, and governance processes to maintain accuracy, consistency, and reliability across all datasets and features in Feature Store.
- Optimize Pipelines Performance: Identify and eliminate bottlenecks in complex ETL jobs, transforming long-running processes into streamlined, rapid-iteration cycles.
- Bridge Data Science & Backend Teams: Act as the key link between data science and backend engineering, ensuring seamless data integration and usage across the organization.
- Explore new Data Sources: Partner with Data Scientists to build custom ingestion logic for unstructured or non-typical data sources, handling the heavy preprocessing needed for experimental research.
- Evolve the Data Architecture: Maintain and optimize our Data Lake. You’ll help us decide on the future of our storage (e.g., moving toward a Data Lakehouse model) and implement best practices.
- Work in an International Environment: Join an international, English-speaking team focused on scaling our adtech platform to new heights.
Who you are
- You have 5+ years of software development experience, working on modern data engineering stack.
- You have proven experience with Apache Flink for stateful stream processing and real-time feature computation.
- You have worked extensively with real-time data streaming systems such as Kafka, Kinesis, or Pub/Sub.
- You have experience with systems handling several TB of data per day and multiple thousand events per second.
- You know how to identify bottlenecks in data pipelines and you have experience in optimizing them and scaling them up.
- You have strong Java knowledge, Golang/Python knowledge would be a plus.
- You have worked closely with Data Scientists on online ML systems with low latency.
- You know how to move beyond “raw data” to design robust, multi-layered data architectures. You have hands-on experience using dbt to build these layers and can guide us on the best tools and formats to manage them at scale.
- You know scheduling frameworks such as Airflow / Kubeflow.
- You know the concepts of data quality and how to apply them in production.
- You are familiar with relational and NoSQL databases.
- You are open to relocating to Hamburg, Germany
- You have strong problem-solving skills and ability to tackle complex technical challenges.
- Plus: You have hands-on experience in working with AWS, Terraform and Kubernetes.
- Plus: You are familiar with the Medallion Architecture and have experience building Semantic Layers for downstream data consumption.
Tech stack & team details
Our Tech Stack: self-hosted open source technologies backed by AWS. Apache Kafka, Apache Flink (Java), Go, Kubernetes, Terraform, S3, Druid, DynamoDB, Redis, TensorFlow, PyTorch, TensorFlow Serving, Triton, Prometheus, Grafana, ELK, OpenObserve, Airflow. Scale at a glance: thousands of requests per second, p99 latency of 100ms for ML models, 100k+ ML predictions per second, 2TB real-time data ingested daily, 100+ Airflow jobs.
Benefits & culture (highlights)
- Invest in your future with development programs and growth opportunities.
- Relocation support to Hamburg, visa assistance, language learning support, relocation bonus.
- Hybrid work with 3 core office days, flexible hours, 30 vacation days, remote work per year.
- In-house gym access, mental health support via EAP, Alster lake view office, snacks and drinks.
- Regular team and company events, hackathons, social gatherings.
