Senior Data Engineer — Sydney, Sydney Region

New

Senior Data Engineer - Azure - Sydney Location: Sydney, Australia (3 Days in office) Our organisation is a prominent player in the renewable energy sector, focused on delivering innovative and sustainable energy solutions. We are on an exciting journey to expand our in-house data capabilities, transitioning from outsourced functions to a dedicated internal team. Why Join This Business? Be a key player in growing and modernising our data function. Influence technical direction and strategy in an evolving environment. Experience a unique blend of start-up agility and corporate stability. Work in a business driving sustainability, work with teams across renewable assets and electricity trading. Enjoy a dynamic, collaborative culture with regular team events and opportunities for personal growth. Key Responsibilities: Design, build, and maintain data pipelines using Databricks, Python, and PySpark. Develop and manage CI/CD pipelines for Databricks and Azure. Utilise Azure Data Factory, Logic Apps, and Service Principals for data orchestration. Implement DevOps best practices using Git, CI/CD, and Terraform. Identify and resolve data quality issues to ensure accurate reporting and insights. Support and maintain the existing data platform while contributing to new technology initiatives. Work with internal teams to ingest new data streams (e.g., from SFTP, emails, and SQL servers). Build and optimise Bronze, Silver, and Gold data layers for improved real-time monitoring and historical data backfilling. Essential Skills & Experience: Strong skills in Python (and PySpark), and SQL. Must have experience of deploying a data platform in a production environment. Proficient in Azure cloud services, including Azure Data Factory, Logic Apps, Function Apps, and Service Principals. Experience with DevOps practices, including Git, CI/CD, and Terraform. Proven ability to identify and resolve data quality issues. Ability to balance strategic planning with hands-on technical execution. Experience working in an in-house data function managing infrastructure responsibilities. Ability to work independently and drive the data function’s direction. Team & Culture You will join a tight-knit team of approximately 10–12 professionals in our Sydney office, with plans for further growth. Enjoy a diverse and vibrant work culture, characterised by collaboration and regular team events. We foster an inclusive environment and actively encourage diversity within our team. Please note: This role can only take applicants who are based in Sydney with the right to work in Australia. This business will not be able to sponsor any candidates.