Big Data Stack
This architecture diagram shows a pipeline for data processing from beginning to end. Four stages make up this pipeline type: extract, transform, load, and report. In this example design, the pipeline pulls data from one organization, joins records from their web application, enriches the data in a staging database and then stores it on redshift. The outcomes are then displayed on a report built on Quicksight.
The big data stack used in this data engineering project is as follows:
1. AWS EC2 for storage and running IICS.
2. Informatica ELT to orchestrate data ingestion and staging of the data in AWS Aurora.
3. AWS Redshit as a Datawarehouse.
4. AWS Quicksight dashboard to display reports.