Data Engineer — North Ryde, Ryde Area

Expired

Due to business growth, we are now looking for an Azure Data Engineer to join the business. The role of the Data Engineer is responsible for designing, building, and maintaining the infrastructure that supports data storage, processing and retrieval. You will require proven experience working with large data sets and the ability to develop data pipelines that move data from source systems to data warehouses, data lakes and other data storage and processing systems. Skill set / Experience needed - Bachelor's or Master's degree in Computer Science, Engineering, or a related field. 8 years of experience as a Data Engineer or in a related role, with a strong focus on real-time data streaming. · Proven experience with Microsoft Stream Analytics, Power BI, Azure Data Lake, Azure Data Factory, Azure Data Warehouse, Microsoft Fabric, Azure DevOps, SQL Server, and Python. Real-time Data Processing experience: Implement real-time data processing solutions using stream processing frameworks such as Kafka Streams, Apache Flink, or Apache Spark Streaming. Familiarity with CI/CD pipelines and distributed systems and a solid understanding of data architecture principles. Familiarity with event-driven architectures and microservices. Design and Develop Data Pipelines: Architect, develop, and maintain scalable and reliable data pipelines for real-time data processing using Microsoft Stream Analytics. Data Integration: Integrate various data sources (databases, APIs, file systems, etc.) with Kafka and downstream systems to ensure seamless data flow. Monitoring and Troubleshooting: Continuously monitor Azure data pipelines, identify issues, and troubleshoot to ensure smooth data flow and minimal downtime. Performance Optimization: Optimize the performance of Azure data pipelines to manage increasing data loads and maintain low latency. Documentation: Create and maintain comprehensive documentation for Kafka infrastructure, data pipelines, and associated processes. You will require full working rights for Australia. The client offers a hybrid working model. By clicking 'apply', you give your express consent that Robert Half may use your personal information to process your job application and to contact you from time to time for future employment opportunities. For further information on how Robert Half processes your personal information and how to access and correct your information, please read the Robert Half privacy notice: https://www.roberthalf.com/au/en/privacy. Please do not submit any sensitive personal data to us in your resume (such as government ID numbers, ethnicity, gender, religion, marital status or trade union membership) as we do not collect your sensitive personal data at this time.