Senior Data Engineer - Analytics Experience and CPG Industry a Plus

---

We are looking for a highly skilled Data Engineer with proven experience in the CPG (Consumer Packaged Goods) industry, specifically with market data and analytical models for the US and Europe regions. The ideal candidate will have hands-on expertise in Azure Databricks, Data Factory, ADLS, and PySpark, and a solid understanding of data engineering best practices and data structures. You will be responsible for building robust and scalable ETL pipelines, working closely with business analysts, data scientists, and product teams to transform raw data into valuable business insights. Key Responsibilities: · Design, develop, and maintain scalable and high-performance ETL/ELT pipelines using Azure Data Factory, Databricks, PySpark, and ADLS Gen2. · Collaborate with business and analytics teams to understand data requirements and deliver high-quality data solutions tailored to CPG workflows such as shipment tracking, POS integration, retail analytics, and trade promotions. · Work on ingestion, transformation, quality checks, and enrichment of large-scale datasets from disparate sources. · Implement and manage data lakehouse architectures, applying Delta Lake principles for unified batch and streaming ingestion. · Ensure data quality, integrity, security, and compliance with GDPR and US data protection norms. · Use performance tuning techniques to optimize complex PySpark and SQL queries in Databricks. · Contribute to the CI/CD process for data pipeline deployment and version control (Git-based workflows). · Collaborate with cross-functional teams (Data Scientists, BI Engineers, DevOps) and mentor junior engineers when needed. · Prepare documentation, data dictionaries, and technical guides as part of the delivery process. · Participate in code reviews, sprint planning, and agile development practices. Required Qualifications: · 5+ years of hands-on experience in data engineering roles. · Domain experience working with CPG data for US and/or European markets. · Strong expertise in: Azure Databricks (including Delta Lake), Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), PySpark and Spark SQL. · Strong understanding of data engineering concepts, including: Data modeling, partitioning, and file formats (Parquet, Delta, Avro); Data structures and algorithms; Performance tuning and memory optimization in Spark. · Solid command of SQL, with ability to write optimized queries for large-scale datasets. · Experience handling structured and semi-structured data (JSON, XML, CSV). · Excellent communication and documentation skills. · Familiarity with CI/CD pipelines, Git, and agile methodologies. Preferred Qualifications: · Experience with CI/CD tools like Azure DevOps or GitHub Actions. · Experience with data cataloging tools like Azure Purview. · Knowledge of Trino, Power BI, or data visualization tools is a plus. · Understanding of CPG-specific KPIs such as Nielsen/Circana data mapping, baseline sales, trade uplift, distribution tracking, etc. · Exposure to machine learning pipelines or feature engineering in collaboration with data science teams is an advantage.

Keyword: Data Analysis

Data Engineering Data Transformation Data Analytics Framework Database Design Databricks Platform ETL Pipeline Database Architecture PySpark