Welcome!
Before we begin, I recommend you to read my previous article where I covered data pipelines following the traditional data workflow of Extract, Transform and Load or ETL.
In the modern age, with the advent of technology, compute and storage resources are much cheaper than what they used to