Key Responsibilities

• Design and implement data pipelines and ETL processes using Databricks and PySpark.
• Develop and optimize scalable data solutions for large datasets.
• Integrate Databricks with AWS and Azure services for data storage, processing, and analytics.
• Collaborate with data scientists, analysts, and business stakeholders to deliver high-quality data solutions.
• Ensure data security, compliance, and performance tuning across platforms.
• Troubleshoot and resolve issues related to data processing and infrastructure.

Required Skills

• Programming: Strong proficiency in Python and PySpark.
• Databricks: Hands-on experience with Databricks platform for big data processing.
• Cloud Platforms: Expertise in AWS (S3, EMR, Glue, Lambda) and Azure (Data Lake, Synapse, Functions).
• Data Engineering: Experience in building and managing ETL pipelines and workflows.
• Performance Optimization: Ability to optimize Spark jobs and data pipelines for efficiency.
• Version Control & CI/CD: Familiarity with Git and DevOps practices.