Python developer (Airflow / dbt)

Company: Arenadata ☑️Responsibilities: -Development of product functionality based on Apache Airflow: development of custom operators, hooks, and providers; -Development of dbt modules: creation of adapters for new data sources, development of custom plugins; -Design of module and connector architecture; -Interaction with architects and the product team to define the data platform development strategy; -Development and support of testing infrastructure: unit tests, integration tests; -Preparation of architectural documents and technical notes on adopted solutions; -Defense and justification of architectural and technical solutions; -Participation in code review; -Review of user and technical documentation for products;

☑️Experience: -From 5 years of experience in Python development, a significant part of which is product development of data tools; -Practical experience with dbt Core: understanding of architecture, experience in developing or modifying adapters; -Knowledge of the Big Data stack: Apache Hadoop ecosystem (HDFS, YARN, MapReduce), Apache Spark / PySpark, Apache Ozone, Trino, Apache Hive; -Understanding of distributed storage and computation principles; -Experience with various DBMS for 3+ years;

☑️Professional knowledge, skills, and abilities: -Knowledge of the internal structure of Apache Airflow: architecture, scheduler, executors (Celery, Kubernetes), plugin system, provider API; -Understanding of dbt Core architecture: execution lifecycle, adapter system, data source interaction protocol, Jinja compilation, manifest, and artifacts; -Proficient in Python 3; -Experience in developing connectors/drivers for DBMS: knowledge of DB-API 2.0, SQLAlchemy, ODBC/JDBC; -Knowledge of the principles of design and functioning of Linux family operating systems; -Experience with Git; -Understanding of continuous integration and continuous delivery (CI/CD) principles; -Containerization experience: Docker, Docker Compose; understanding of Kubernetes; -Decomposition of development tasks and ability to estimate effort;

☑️Will be a plus -Experience with open source contributions (Apache Airflow, dbt, dbt adapters); -Experience in developing custom dbt adapters for non-standard sources. **Contacts: **career@arenadata.io