This article talks about some of the major challenges that data engineers face while creating and maintaining data lakes.

I have dealt with five out of the six challenges that have been discussed here, not necessarily using Spark though.

Must read for data engineers!