Lead II - AWS Databricks Engineer

• Job ID: 64536
• 7 - 9 Years
• 1 Opening
• Bangalore, Hyderabad, Pune, Trivandrum

Role description

Lead Data Engineer (US Healthcare / Medicare Advantage)

Position Overview

We are seeking a Lead Data Engineer to architect, build, and optimize scalable, cloud-native data systems. In this role, you will lead the development of enterprise data platforms using Databricks and AWS. You must possess deep domain expertise in US Healthcare, with a strict requirement for hands-on experience handling Medicare Advantage data.

Key Responsibilities

• Lead the design, implementation, and maintenance of robust data pipelines on AWS using Databricks.
• Architect scalable lakehouse structures leveraging Apache Iceberg to ingest, process, and store complex healthcare data types.
• Establish comprehensive data governance and data lineage frameworks using Databricks Unity Catalog.
• Implement secure access controls, data masking, and row-level security through Role-Based Access Control (RBAC) to ensure data security rules are enforced.
• Architect and scale ingestion frameworks capable of processing HL7 and FHIR data feeds natively into the lakehouse ecosystem.
• Oversee the end-to-end processing of Medicare Advantage data assets, ensuring accurate risk adjustment and reporting workflows.
• Ensure strict adherence to healthcare regulations, including HIPAA compliance and robust data governance frameworks.
• Mentor junior engineers, conduct code reviews, and drive technical best practices across the data team.

Required Skills & Qualifications

• 7+ years of data engineering experience, with at least 2 years in a technical lead role.
• Production experience building and optimizing large-scale pipelines within Databricks.
• Production experience deploying data infrastructure on AWS (specifically S3, EMR, Lambda, IAM, and Glue).
• Comprehensive understanding of US Healthcare data, with explicit, hands-on experience processing Medicare Advantage data (e.g., CMS enrollment, RAPS/EDPS submissions, MMR/MOR files, or HCC risk adjustment models).
• Proven experience working with healthcare interoperability standards, specifically FHIR and HL7 models.
• Deep understanding and practical experience working with open table formats, specifically Apache Iceberg.
• Expert-level knowledge of Databricks Unity Catalog for centralized data governance.
• Proven experience implementing enterprise-level Role-Based Access Control (RBAC) and row-level security for protected health information (PHI).
• Mastery of Python, PySpark, and advanced SQL for complex distributed computing tasks.

Good-to-Have Skills

• Databricks Certified Data Engineer Professional or AWS Certified Data Analytics certification.
• Familiarity with orchestration tools like Apache Airflow or AWS Step Functions.

Skills

Databricks, AWS, FHIR, Python, Apache Airflow

About UST

UST is a global digital transformation solutions provider. For more than 20 years, UST has worked side by side with the world’s best companies to make a real impact through transformation. Powered by technology, inspired by people and led by purpose, UST partners with their clients from design to operation. With deep domain expertise and a future-proof philosophy, UST embeds innovation and agility into their clients’ organizations. With over 30,000 employees in 30 countries, UST builds for boundless impact—touching billions of lives in the process.