A short introduction to the categories of automation testing for Databases, Data Warehouses, and Data Lakes — Testing data systems is painful. Invariably, it turns out to be an overly complicated exercise where you not only have to test the behavior of the SQL queries, ETL scripts, orchestrators, and so on, but you also have to test the data itself. Although testing the data introduces quite a…

A short introduction to open-source data quality & testing tools — dbt, Deequ, and Great Expectations — Background Data quality has always been an important theme while building data systems. Still, data engineering teams of the past have had bigger fish to fry, making sure the data lands up from one place to another (in whatever way or form). Data quality often surfaces as a problem when the…

A peek into the different query optimizers and how they work — SQL is a simple language with very few rules, which is what makes it so popular. It is also enriched with a great number of keywords and features that allow you to interact with your data in all kinds of ways. …

Cutting through the noise around data warehousing — Signal vs. Noise With the appropriation of the term data warehousing to mean a whole lot of things it shouldn’t, the general understanding of what a data warehouse is has become a nomenclature problem more than anything else. …

In Elon Musk's lingo, reading your article added to my unregrettable Medium minutes. Clearly explained and consice. Having spent most of my time doing data engineering, I hadn't thought of the backend as having these two facets BSF and BFD. And, on a separate note, I think SurrealDB is out of beta now. I'm going to give it a shot one of these weekends and will post about it.

The swamp and bedrock analogy is near perfect. I'm glad to see how FOSS and commercial software converge beautifully in your data stack. Also, for those who doubt there isn't enough space for Snowflake and Databricks in the same data stack, here's a brilliant example that proves you wrong. @Nate - are there any plans to open-source DDC anytime soon?

I've said it before, and I'll say it again - this is why I love Cantor's Paradise. It attracts such amazing articles. Beautifully explained!

Going multi-region without leaving Australia — Earlier this week, I attended the AWS Cloud Day in Melbourne — a first after the AWS Melbourne region was launched earlier this year. The event was much better than what I had expected — higher turnout, superb conversations, and great energy. Throughout the day, which started way too early…

Sayeeduddin Dagar’s beautiful musical interpretation of the flow of Ganga and Yamuna, 1999 — Wim De Winter has been uploading some gems on YouTube from his collection of Sayeeduddin Dagar’s recordings. Some of them go back over two decades. I heard this five-minute recording just a couple of weeks ago, and I was awestruck by its simplicity and beauty. Sayeeduddin Dagar demonstrates a composition…