Data Engineer — North Ryde, Ryde Area

Expired

Data Engineer Location – Sydney preferred • Pivotal role in a team building a data warehouse for a new loan product. • Join an ambitious firm which has exciting growth plans in place. • Allied Credit has a hybrid work environment, we understand the value of a work-life balance. Company Description Allied Credit is a leading and well-established finance partner in the vehicle and leisure asset industry, with strategic alliances incorporating well-known car and motorcycle brands. We offer a range of finance solutions including white label, joint venture arrangements and floor plan finance. Our proprietary loan origination system supports our dealers and distributors to facilitate seamless, convenient and customer focused finance solutions for consumers and businesses. Allied Credit are seeking a Data Engineer, you would be responsible for designing, building and maintaining the infrastructure that supports data storage, processing and retrieval. You will work with large data sets and develop data pipelines that move data from source systems to data warehouses, data lakes and other data storage and processing systems. Responsibilities • Design and Develop Data Pipelines: Architect, develop, and maintain scalable and reliable data pipelines for real-time data processing using Microsoft Stream Analytics. • Data Integration: Integrate various data sources (databases, APIs, file systems, etc.) with Kafka and downstream systems to ensure seamless data flow. • Real-time Data Processing: Implement real-time data processing solutions using stream processing frameworks such as Kafka Streams, Apache Flink, or Apache Spark Streaming. • Monitoring and Troubleshooting: Continuously monitor Azure data pipelines, identify issues, and troubleshoot to ensure smooth data flow and minimal downtime. • Data Quality and Governance: Enforce data quality checks and compliance with data governance standards, while maintaining data security and privacy. • Collaboration: Partner with Data and Project teams to build and sustain the data infrastructure. Provide technical guidance and mentorship to junior data engineers. • Performance Optimization: Optimize the performance of Azure data pipelines to manage increasing data loads and maintain low latency. • Documentation: Create and maintain comprehensive documentation for Kafka infrastructure, data pipelines, and associated processes. Your background • Bachelor’s or Master's degree in Computer Science, Engineering, or a related field. • 8 years of experience as a Data Engineer or in a related role, with a strong focus on real-time data streaming. • Proven experience with Microsoft Stream Analytics, Power BI, Azure Data Lake, Azure Data Factory, Azure Data Warehouse, Microsoft Fabric, Azure DevOps, SQL Server, and Python. • Familiarity with CI/CD pipelines and distributed systems and a solid understanding of data architecture principles. • Experience with machine learning pipelines and integrating real-time data streams into ML models. • Familiarity with event-driven architectures and microservices. • Azure Data and Power BI Engineering certification would be beneficial. • Excellent problem-solving abilities in a fast-paced environment with strong communication and collaboration skills. Benefits of working at Allied Credit • Hybrid working arrangements, we are always willing to discuss flexible working options. • Access to our exclusive corporate superannuation fund. • Salary Continuance Insurance (covering TPD insurance). • Discount offers on selected retailers, car loans and novated leasing options. • Ongoing extensive in-house and external learning & development opportunities. • Wellness day with ongoing wellbeing, diversity program & fund-raising events. • 16 weeks’ paid parental leave. We pledge to treat all candidates with the utmost respect and professionalism throughout the recruitment process. Our interactions will be characterized by courtesy, honesty, and integrity. We promise to conduct a fair, transparent, and inclusive hiring process. We are committed to ensuring that all candidates are evaluated based on their skills, experience, and potential, regardless of race, gender, age, disability, sexual orientation, or any other characteristic protected by law. For further information on Allied Credit, please visit our website at https://alliedcredit.com.au/. Please contact Alex Kelly on 0484 109 354 for a confidential discussion or email hralliedcredit.net.au for further information.