Azure Cloud or DevOps Engineer for Prefect ETL Infrastructure

---

Description:

We are seeking an experienced Azure Cloud or DevOps Engineer to provide development and technical support for a cloud infrastructure modernization initiative. The qualified engineer will assist in developing, implementing, and testing an Infrastructure as Code (IaC) deployment to establish a more permanent ETL pipeline and workflow orchestration system hosted on our Microsoft Azure instance. The key component of the new infrastructure will be a self-hosted instance of Prefect Server (https://www.prefect.io/) that will primarily be used by our data engineering and analytics teams to support an existing data warehouse. The project will also include the configuration of supporting cloud resources, including a postgresql flexible server to store Prefect Server’s metadata, configuration of worker pools and execution environments, and separation of development and production environments. The qualified candidate will be comfortable with deploying and supporting Azure cloud infrastructure using Azure DevOps pipelines. This project is intended to enhance our ETL pipeline reliability, improve observability, and increase machine learning scoring capabilities through the deployment of more streamlined and automated infrastructure. This is a 1 to 2 month project focused on: 1. Prefect Implementation: • Setting up a workflow orchestration platform using Prefect in Azure via IaC • Configuring Prefect for ETL pipeline management and observability • Integrating Prefect with our existing Azure SQL Database 2. Azure Environment Configuration: • Creating separate development and production environments • Integrating into our existing security and access controls • Setting up infrastructure as code practices 3. CI/CD Pipeline Establishment: • Developing automated deployment workflows • Implementing automated testing for infrastructure deployment • Creating documentation for infrastructure deployment and maintenance 4. Additional improvements and best-practice recommendations as needed Qualifications • 5+ years of experience with Azure cloud services, DevOps, and cloud engineering • Strong knowledge of Prefect, dbt, or similar ETL orchestration tools • Extensive experience with data warehousing and ETL systems in Microsoft Azure • Experience with CI/CD pipelines (Azure DevOps, GitHub Actions, etc.) • Proficiency in infrastructure as code (Azure ARM templates preferred) • Background in data engineering and ETL pipeline development preferred • Understanding of security best practices for cloud environments (we are a SOC2 environment hosting sensitive data) How to apply Please include in your proposal: 1. A specific list of information you need from us about our existing infrastructure 2. A brief summary of your experience with similar Azure-based infrastructure projects 3. Your experience with Prefect, dbt, and IaC deployments for similar infrastructure 4. Your experience with creating separate development/production environments for ETL systems 5. Your experience with supporting Azure infrastructure that hosts sensitive data 6. Availability to start and hours available to commit during project timeline 7. References or examples of previous work (if available)

Tags: Git, Health & Wellness, Engineering & Architecture, Nonprofit Organization, DevOps, Microsoft Azure, CI/CD, GitHub

Keyword: Machine Learning

Job Type: Hourly: $75.00 - $120.00

Price: $75.0