Job Overview

We are currently looking for highly accomplished Senior Data Engineers (10+ years of experience) with deep expertise in Databricks and PySpark to play a key role in an ongoing, largescale data transformation initiative. The ideal candidates will bring extensive handson experience, strong architectural understanding, and the ability to lead and influence data engineering practices across teams.

Key Responsibilities

• Extensive experience in analyzing, interpreting, and modernizing complex data transformation logic written in TSQL, and implementing optimized, scalable equivalents within Databricks using PySpark.
• Proven expertise in architecting and implementing endtoend data ingestion frameworks using Azure Data Factory, managing data flows from multiple source systems through the RAW and subsequent data layers, with a strong focus on reliability and scalability.
• Strong experience in defining, capturing, and analyzing performance metrics, enabling proactive monitoring, bottleneck identification, and continuous optimization of data ingestion pipelines.
• Demonstrated ability to perform advanced performance tuning and optimization of Databricks read/write operations, including partitioning strategies, caching, file formats, and query optimization techniques.
• Capability to provide technical leadership and mentorship to data engineering teams, establish best practices, conduct design reviews, and drive engineering excellence.
• Strong problemsolving skills, a resultsoriented mindset, and the ability to collaborate effectively with crossfunctional teams, including architects, analysts, and stakeholders.

Required Skills

• Proficiency in interpreting data transformation logic written in TSQL and implementing equivalent processes within Databricks.
• Ability to design and implement data ingestion pipelines using Azure Data Factory (from source to RAW layer).
• Basic knowledge of C and SQL (at least read the coding, no need to write).
• Experience in collecting and analyzing performance metrics to optimize data ingestion pipelines.
• Competence in performing performance optimizations for Databricks read/write queries as needed.
• Strong motivation and the ability to provide guidance to other data engineers within the team.

Qualifications

• Bachelor's degree or Computer science (technical background).

Other Details

• Shift: 10.00am 7.00pm
• Experience: 10+ Years
• Position: 2
• Duration: 6 Months