Experienced Databricks & ETL Expert Needed for Medallion Architecture Setup

---

---

We are looking for an experienced Databricks expert to help us quickly set up a Medallion architecture for our data pipeline. Our team is building a robust ETL framework and needs someone who can immediately assist with configuring Databricks, optimizing performance, and structuring our data layers (Bronze, Silver, Gold). Key Responsibilities: - Set up and configure Databricks for scalable data processing - Design and implement a Medallion architecture (Bronze, Silver, Gold layers) - Develop efficient ETL pipelines for data ingestion, transformation, and storage - Optimize Delta Lake performance and data governance best practices - Ensure data integrity, security, and compliance - Provide guidance and documentation for internal teams Requirements: - Extensive experience with Databricks, including cluster setup, notebooks, and SQL/PySpark - Deep knowledge of Delta Lake and best practices for managing large datasets - Strong ETL expertise, including data ingestion, transformation, and pipeline orchestration - Experience with Medallion architecture implementation - Familiarity with AWS cloud storage integrations - Strong SQL and Python (PySpark) skills - Ability to start immediately and work efficiently in a fast-paced environment Preferred Qualifications: Experience with Apache Spark tuning and optimization Knowledge of MLflow and data cataloging best practices Prior work with enterprise-scale data engineering projects

Keyword: cloud

Contractor Tier: Hourly: $80.00 - $250.00

Price: $165.0