Data Engineer (x4)

- Create reliable, maintainable, and high-performance data lakes and data warehouses tailored to organizational analysis and reporting needs - Design and implement scalable ELT/ETL pipelines and data infrastructure, utilizing diverse data sources - Partner with Data Modelers and Architects to establish curated, business-centric data models that act as a unified source of truth for reporting purposes - Collaborate with infrastructure and cybersecurity teams to guarantee data security both in transit and at rest - Clean, prepare, and optimize datasets for enhanced performance, applying lineage and quality controls throughout the integration process - Support Business Intelligence Analysts with dimensional modeling and aggregation optimization for effective visualization and reporting - Diagnose and resolve ingestion, transformation, pipeline performance, and data integrity challenges - Collaborate with analysts, scientists, engineers, and architects to create pipelines for enterprise data sharing - Identify and execute on automation opportunities, scalability improvements, and infrastructure enhancements - Utilize Microsoft technologies such as Azure Data Factory, Azure Data Lake, Azure SQL Databases, Synapse Analytics, Databricks, Microsoft Purview, and Power BI - Engage in SCRUM and Kanban methodologies, assisting with backlog management and iterative delivery - Aid in the establishment and upkeep of enterprise data catalogs and metadata repositories - Develop efficient pipelines and models using Python, Spark, and SQL across various data formats (XML, CSV, JSON, REST APIs) - Maintain thorough documentation of pipelines and products, ensuring sustainability through version control practices - Orchestrate pipeline execution to meet latency benchmarks and manage dependencies - Collaborate on tools aimed at minimizing manual tasks and boosting operational efficiency - Assist in the support of CI/CD pipelines for automating infrastructure, code deployment, and release processes - Oversee production solutions, troubleshoot issues, and provide Tier 2 support for datasets - Implement role-based access controls to data products in alignment with governance standards - Design and conduct automated unit and regression tests, support user acceptance and integration testing, and contribute to test case formulation - Participate in peer code reviews to uphold quality and maintainability standards