Data Engineer — Braddon, North Canberra

Expired

Set up & build Databricks pipelines for a key APS dataset. Support this data hub of data analysts transition from using CSVs > Databricks database This role is at a Federal Government line agency within a data hub team of 10 experienced analysts who provide analysis across the department’s portfolio responsibilities. This team plays a central role in the department’s large general-purpose PLIDA project, used by around 70 analysts. The Opportunity Working with and on a very large integrated data set (PLIDA) Career/technical development from Data Analyst -> Data Engineer or continuing to advance your Data Engineer skills and experience Newly created role with budget approved Potential to interview and start early 2025 The Role This team of analysts is looking for someone like yourself with expertise in Databricks to set up pipelines to a dataset that gets refreshed annually in a straight forward and replicable way to allow the team to slice, dice and analyse the dataset by cohorts etc You will do this by Proving technical expertise in transitioning an analysis team from using flat files (CSVs) on a shared drive to a database on the Databricks platform Working in PLIDA (formerly MADIP) set up, document and run a process for ingesting data into a Databricks environment and then performing transforms and merges to deliver a high-quality analysis-ready database tier The team currently primarily uses R, including DuckDB and Arrow packages for larger data sets and is wanting to minimise the use of Python to avoid analysts needing expertise across multiple languages – SQL-based scripting would be preferred, or R-related packages such as sparklyr. The Ideal Person This role will suit someone who enjoys the challenge of looking at a problem (data engineer/ pipeline and coming up with a straight forward approach to solving using Databricks (essential) as the technological tool. It would be highly desirable (but not mandatory) if you already have experience in PLIDA (formerly MADIP). Noting that being able to get access is a requirement (existing Datalab accreditation would greatly smooth the onboarding process). Australian citizenship is mandatory with a Baseline security clearance desirable. Salary / Rate This role is being offered as labour hire contract ASAP- 30/6/25 at the APS6/ EL1 equivalent. There is no fixed rate for this role. Rates will align to market rates for this role’s level of requirements and your skills and experience. I am happy to work with you to find a competitive and suitable rate for you