Data Lakes: Ending the Confusion

Data lakes evoke technicalites. And tech decision makers don’t know whether to go for the lake, warehouse, or lakehouse. But maybe the question is more personal.

There’s a line in the movie Tron Legacy that perfectly encapsulates data. The line goes: –

“The Grid. A digital frontier. I tried to picture clusters of information as they moved through the computer. What did they look like? Ships? Motorcycles? Were the circuits like freeways? I kept dreaming of a world I thought I’d never see.” – Kevin Flynn, Tron Legacy, Disney, 2010.

Data is this fascinating thing, moving clusters of information that describe behavior, inform choices, move commerce, and shape economies. It is, perhaps, the essence of human knowledge.

And now, data creates the next revolution: AI. But there is so much about data and its storage that is unknown. Engineers can’t agree on definitions and methods.

It is all a bit vague, and the definitions devolve into tech-heavy or too non-technical terms.

With tech, the explanations should be simple and not reductive. Let’s attempt to do that.

What businesses need to understand about data lakes

As a business, you’re probably asking yourself if you need a data lake. So before you or your managers make a decision. You must realize that the data lake is actually a concept- not a storage type.

It is a large bucket of storage that can be scaled as per the needs of the organization. But does every business need it?

Wish there was a clear answer for that, but there isn’t. Ask your devs, your CTOs, and everyone else, you’re not going to come up with a straight answer.

You won’t even get a straight answer on whether you need this technology.

It’s the same answer echoing: it depends on the use cases.

And the main use case for businesses to adopt a data lake is if their databases start crossing a certain threshold.

But what is that threshold? Well, there is a solid answer for that. But it’s a big one to wrap your head around, especially for non-technical people.

The answer is: it depends on the use cases.

Yes, that is not a joke.

Data lakes are more complicated than anyone might think. And see if ChatGPT can give you a straight answer on it. It’ll confuse you even more.

So, do businesses need a data lake?

This is the million-dollar question. Let’s try to answer this question.

There are some things that are pretty common across the board: –

1. You need data lakes if you’re storing large amounts of data.
2. But it depends on the structure of your data.

These are two contradictions that plague decision-making regarding buying data. But we can assume that there will come a time when you know the data lake will be more efficient than the warehouse.

But there’s more to this: there’s also the lake-house. It is a middle ground between the warehouse and the lake, offering a more flexible option between the two.

So you think to yourself: Ah, that means well, just go for the lake-house. Well, tough luck, because no one can really make heads or tails of that either. Amazon S3 is both a lake and a lake-house depending on how YOU use it.

Yet, this is a non-answer. There needs to be a method. One that helps CISOs, CIOs, and every other decision-maker make sense of spend. If it’s needed, and how to identify that need.

Let’s give it a shot.

How can business leaders identify if they need a data lake, a warehouse, or a lake-house?

There are a few assumptions we must make here:

1. Under no circumstances do we want to create a data swamp.
2. It should not increase budget and overhead costs without adjusting the customer lifetime value.
3. It should be manageable.

Then, here are some facts that will help you understand the difference between the three.

What data storage architecture matches your needs?

1. Data Warehouses
   1. Used for structured data, quick insights, and pre-processed data. Perfect for teams not on a budget. But it cannot store unstructured or raw data.
2. Data Lakes
   1. Used for unstructured, semi-structured, and structured data. It is low-cost and flexible.
3. Data Lake-houses
   1. A combination of the two. It is lower cost than warehouses, more structured and analytical than lakes. Flexible.

Yes, lakehouses seem like the perfect fit. But not everyone needs them. Sometimes, a warehouse would do. Or, if the data you have is fairly exceeding the bounds, then you should opt for the data lake.

The pro of the data lake and lake house is that they are highly scalable. Warehouses, because of their structured approach, can be tricky.

The Diagnosis

So now that we have enough information. We can map a diagnosis structure onto this:-

The first question leaders should ask themselves is:

1. What does my team think about this?

If they think you need it, what is the reasoning, and how many of them believe it is necessary? There is a good chance you will find that the teams are divided.

The second question then is:

1. What are the clear advantages of having either one of these?

Then: –

1. What are the constraints that each of them offers, and can the hybrid lake house be the fix for it?

Followed by: –

1. Does our budget allow for such changes, and is the trade-off worth the migration and other hassles that come with the decision?

And finally,

1. Will this mitigate any future or present problems?

You may notice that the diagnosis is based purely on a strategic, human-first approach. Because that’s how decisions are made. In the research for the blog, we found that 86% of tech decision makers feel analysis paralysis.

That’s a lot. Although malicious actors and tight budgets have made this easier. Analysis paralysis is the reason why buying committees take time to make decisions.

The ripples of the decisions are too much. Add information from AI and other sources, and leaders’ confusion only grows. But the reason behind it is much simpler and driven by human psychology: the inability to learn from ground-level staff.

The Pigeonhole Problem

Leaders are good at doing their job- managing people and solving problems. This causes decision-fatigue to build up. There’s a reason why you feel hung out to dry, because your nervous system is actually tired from all the mental hard work you do.

It is not a joke. And neither is it disconnected from this conversation. So what happens?

Your vision narrows down, and the sight of what is happening on the ground becomes blurry. You have to manage stakeholders and user expectations- now this?

So the easiest part is to understand your own engineer’s perspective. And then use that to make a decision, using your honed decision-making instinct.

The pigeonhole problem is that you narrow down to a result and forget the process. And add to that your buying committee, which becomes an echo chamber.

Remember, decisions are people-first.

So, what do you do?

The tech community is facing a huge problem. Everyone thinks it’s run by logic. But it is run by experimentation, mistakes, and a whole lot of frustration.

Why does this go unacknowledged?

Think of data architecture, won’t it be personal to your context? Yes, you’re googling or LLMing whether to buy a data lake.

But what do your engineers, devs, and product teams think? And is your business ready for this decision?

Of course, you can hop on the trend and just do it. The lake house is perfect for it. That’s the answer right there. But that does not mean it will eliminate your problems. There is a chance it could add to it.

And don’t you forget the other layer- these are all concepts. They are not actually a thing. When you buy the S3 or Snowflake, you get the option to choose between structured, unstructured, semi-structured, and everything else in between.

The reality is whether you can afford it. But that’s a difficult question to answer. It is a tough decision to make because if you miss a trend or an opportunity, you might fall behind. Isn’t that why you decided to invest in AI?

Data lakes aren’t the problem. It’s not understanding what a business needs.

Your business needs are unique. The reason GPTs and Reddit return the answer, it depends, is because data is molecularly contextual.

And that’s actually the magic of it. Your customer segments, even though the same across your competitors, will show different behavior. Your data point, pulled from the same data pool, will vary across silos.

Customer success, the AI/ML division, marketing, sales, and every other data will point towards a radically different idea. It will confuse you. But the way out of this confusion is understanding where the clarity lies.

Data lakes bring an end to siloed visions, but they can become a swamp- so what do you do?

You create architecture that doesn’t overwhelm your teams. The answer is never in managing complexity but in making complexity easier to understand and translate. In tech, you can’t outrun entropy, but you can make it work for you.