> A data warehouse on the other hand is an OLAP database and is optimized to work on columns
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
It's a good all-round primer, well written.
Would love to hear more about larger-than-memory tasks and running local Dask clusters. I processed many-a-dataset that way that would normally make pandas choke.
Now I'll be thinking of "L" in ETL as "Land" and not "Load".
Although the article doesn't propose that but uses a lot of "Land" terminology.
"Load" => "load where? or FROM where?" - ambiguous
"Land" => "land where?" - clear
A bit of a pedantic nit here: a data warehouse is a usage pattern. It’s not necessarily tied to any specific technology, however it is commonly implemented with OLAP systems like Snowflake, BigQuery, etc. But there’s nothing stopping you from building out your data warehouse in Postgres or MySQL. If you’re stitching together disparate datasets to build a unified model for analytics, you’ve got yourself a data warehouse no matter what system it lives on.
Update: Huh, TIL https://avro.apache.org/docs/%2B%2Bversion%2B%2B/specificati...