Company: DIENEY (tobacco company project)
Format: Remote (IP/SZ registration)
Salary range: 250,000-270,000
DevOps Engineer (Data Platform & OT Integration)
Project: Building the Data Ocean platform (Lakehouse/Streamhouse) and integration with industrial OT systems.
The project involves deploying infrastructure in isolated environments (no internet access) and integrating with industrial systems (OPC UA, MQTT). We need an engineer who can not only support the infrastructure but also build a reliable, automated, and secure foundation for Data Engineering and OT Integration.
Key Tasks:
- Building the Data Platform: Deploying and administering Lakehouse/Streamhouse components: Apache Iceberg, Apache Paimon, MinIO (S3), Apache Spark, Apache Flink, StarRocks/Trino.
- Integration with OT Systems: Configuring gateways for data collection from industrial sensors and controllers (OPC UA, MQTT), ensuring reliable data delivery to Kafka.
- CDC and ETL: Setting up Change Data Capture processes (Debezium/Kafka Connect) for synchronizing data from ERP/CRM systems.
- Kubernetes & GitOps: Managing Kubernetes clusters (including Edge solutions if needed), developing Helm charts, configuring GitOps processes.
- CI/CD for Data Infrastructure: Building pipelines for building and deploying Data services, automating configuration testing.
- Working in Isolated Environments: Organizing local repositories (Nexus), configuring proxy servers, automating deployment in environments without internet access.
- Security and Monitoring: Integrating with HashiCorp Vault for secrets management, configuring monitoring (Prometheus/Grafana) for Big Data components (Spark/Flink metrics, Kafka Lag, Iceberg status).
Requirements:
- Experience: 5+ years in a DevOps SRE role.
- Kubernetes: Deep understanding of K8s architecture (Pods, Deployments, StatefulSets, Probes, Network Policies). Experience running Big Data tasks (Spark/Flink) in Kubernetes via Operators.
- Data Stack: Hands-on experience with Apache Spark, Apache Flink, Apache Kafka. Understanding of Apache Iceberg or Apache Paimon principles. Experience with S3-compatible storage (MinIO).
- CI/CD & IaC: Experience building pipelines in Jenkins. Knowledge of Terraform and Ansible.
- Linux & Networking: Solid knowledge of Linux (administration, diagnostics, scripting in Bash/Python). Understanding of network protocols (TCP/IP, HTTP, TLS).
- Monitoring: Experience setting up monitoring for distributed systems (Prometheus, Grafana, Loki/ELK).
- Knowledge of Git, experience with GitLab/Jira/Confluence.
- English: Technical documentation reading level (B1+).
Will be a big plus:
- Experience with Apache StarRocks or Trino.
- Experience setting up CDC (Debezium) and working with Kafka Connect.
- Experience working in isolated (Air-Gapped) environments.
- Knowledge of industrial automation protocols (OPC UA, MQTT).
- Experience with Apache Airflow.
