Project Details

This architecture diagram shows a pipeline for data processing from beginning to end. Four stages make up this pipeline type: extract, transform, load, and report. In this example design, the pipeline pulls data from one organization, joins records from their web application, enriches the data in a staging database and then stores it on redshift.  The outcomes are then displayed on a report built on Quicksight.

The big data stack used in this data engineering project is as follows:

1. AWS EC2 for storage and running IICS.

2. Informatica ELT to orchestrate data ingestion and staging of the data in AWS Aurora.

3. AWS Redshit as a Datawarehouse.

4. AWS Quicksight dashboard to display reports.