DATA ON AWS

In a recent post in this series, I talked about different options for caching in AWS. I talked about ElastiCache for Redis, as it is one of the most popular caches out there, especially because of its advanced data types like sorted sets which make a developer’s life way easier…

DATA ON AWS

Following from my previous posts on Timeseries databases, where I argued for the case for using timeseries databases along with some hands-on tutorials, in this post, I’ll talk about different timeseries databases and how you can run these databases on AWS.

Running Timeseries Databases on AWS 🏃‍

Hosting and running any open source or enterprise database…

I get where you are coming from, but databases like PostgreSQL and MySQL are extremely good with complex queries. I think a better way to put it would be that relational databases aren't good with handling queries to serve the needs of reporting and business intelligence.

In other words, read scaling is not a problem with relational databases if you're sending the right kind of reads to the database.

Deletes in lakeFS

This feature is super-important. GDPR-compliance in Europe, CDR-compliance in ANZ, and similar privacy regulation elsewhere mandate it. Reading through the documentation, I found that objects can be hard deleted now:

"In order for an object to be hard-deleted, it must be deleted from all branches. You should remove stale branches to prevent them from retaining old objects. For example, consider a branch that has been merged to main and has become stale. An object which is later deleted from main will always be present in the stale branch, preventing it from being hard-deleted."

@Anom - Wouldn't AWS SCT be a good use-case here? Use SCT to convert your MySQL schema to a PostgreSQL schema in a completely lossless fashion. Create the schema in your target PostgreSQL database first and create DMS migration tasks. I have nothing against pgLoader, though :)

Great post!

Here's a nice post on Airflow by @Vinícius Mello! If you're new to Airflow, start here!

Here's a real example of how to use Airflow (with some caveats and suggestions) by @Rachel.

It's unclear if the temporary tables are created in the source database (MySQL) in the pre_mysql step. If that's the case, it might lead us to possible disk space issues. …

DATA ON AWS

Repeat read access patterns have mandated the implementation of a caching layer in most modern applications. These access patterns might be for different use cases, such as configuration files, images, any file on a CDN. …