Barnardo’s · data / azure
Turning source data into useful information
Data-pipeline architecture and implementation using Azure Data Factory, bringing source data through ingestion and transformation.
Data pipeline engineering
Data engineering
Engineering & systems
My data-pipeline architecture and implementation contribution to the Barnardo’s engagement.
- Source ingestion
- Data transformation
- Azure Data Factory
The situation
Data needed to be ingested and transformed before it could support downstream use. The work sat in the processing pipeline rather than in a customer-facing interface.
My contribution
My work covered data-pipeline architecture and implementation, using Azure Data Factory and medallion-style processing.
The approach
Worked on the architecture and implementation of the ingestion and transformation stages using Azure Data Factory. A medallion-style processing approach separated the source data from the information prepared for downstream use.
Keep the stages of processing clear
The medallion-style approach separated source ingestion from later transformation and preparation. That structure connected the data architecture with the implementation in Azure Data Factory. The focus of the work was the processing pipeline rather than a customer-facing interface.
Where it landed
Source data could be ingested and transformed through Azure Data Factory pipelines, with stages for processing and refinement. My contribution covered both the architecture and its implementation.
