My yap
Essay

Reflection piece on the MapReduce debate, 16 years on

This is a reflection piece written as part of my IMDA mentorship programme. I was tasked to read 2 articles on the roots of the big data revolution, one of which is this piece, and the other is this one.

I’ll give my thesis right away: Structured data is for asking questions. Raw data is great as a ground truth. We should keep both, and rebuild the first from the second.

Context from both articles

Relational databases (think SQL) have been around for decades. They are very good at answering questions about structured data. For example, “What is the total number of items customer A bought this week?” Structured databases are also one of the first things I learnt in my own Databases course when I did my Part IA last year in Cambridge.

But, as Stonebraker claimed (and I would choose to believe him since he does know his stuff, and he did use this in 2010. What was I doing in 2010?), they are also hard to run. Installing, configuring and tuning a parallel database was specialist work, and the licences were expensive.

In 2004, Google published a paper on MapReduce, the system it used to build its search index. The model is simple. You write a map function and a reduce function, and the framework spreads the work across thousands of cheap machines. It handles partitioning, scheduling and machine failures for you. Hadoop, an open-source version built at Yahoo, brought the idea to everyone else.

Engineers and data scientists loved it. It was free. It needed no schema. It could chew through petabytes of messy data. Facebook built a data warehouse on it. People started asking whether databases were legacy technology.

In January 2010, Communications of the ACM ran two articles side by side on exactly this question.

The first is by Michael Stonebraker and six co-authors, all database researchers. Their title asks whether MapReduce and parallel databases are friends or foes. Their answer is friends, but on the database’s terms. They benchmarked Hadoop against two commercial parallel databases on a 100-node cluster. Once the data was loaded, the databases won all three tasks in the article. On a join, one database took 32 seconds. Hadoop took 1,158. They argue MapReduce is really an extract-transform-load (ETL) tool. It should sit upstream, cleaning raw data before it goes into a database.

The second is by Jeffrey Dean and Sanjay Ghemawat, who built MapReduce at Google. They say the benchmark was unfair. It used slow text formats, ignored indexes, and timed steps nobody needs in practice. They point out that the databases took 5 to 50 times longer to load the data than Hadoop took to analyse it. And they argue MapReduce has three lasting advantages. It works with data wherever it is stored. It runs code too complex for SQL. It survives failures in the middle of long jobs.

Reading them back to back, I noticed two things.

First, they agree on more than they admit. Both say the performance gaps come from how the systems were built, not from the ideas behind them. Both say text formats are bad. Both see a role for MapReduce as a loader for databases.

Second, they picture different data. Stonebraker pictures a sales warehouse that analysts query every day. Dean pictures a web crawl that gets processed once.

A made-up company

This came about as I used Claude as a conversational partner/critic as I navigated my way around the 2 articles. I found this scenario helpful in synthesising my views, and here it is in summary.

Imagine an online retailer in 2009.

  • Orders and customers: about 500GB in Oracle. Clean and structured.
  • Clickstream logs: about 50GB a day from several web servers. Semi-structured. The format changes whenever the front-end team ships something.
  • Users: 15 business analysts who work in SQL and Excel, and 2 data scientists who want to build sessions and a recommendation model.
  • Team: 3 data engineers. No database specialist. A tight budget.

I was made to choose between Hadoop and a parallel DBMS to implement.

My first instinct was the hybrid. Hadoop cleans the logs, and a parallel database handles everything else. That is Stonebraker’s answer, and Dean half-agrees.

My second instinct was to pick Hadoop because it is free, considering budget constraints.

That turned out to be a bit shallow.

Who pays, and when

Free software is not cheap software. A database licence is a line item. What I didn’t consider was the team’s time: how much time would the analysts have to spend to learn Hadoop? How much time would the data scientists have to spend to tune Hadoop? In Stonebraker’s tests, he and his team (being experts) took more time tuning Hadoop than both DBMSes combined.

So the question is not which system costs less. It is who pays, and when.

Considering data analysts

A parallel database is expensive up front. Someone has to install it, tune it, design a schema and load the data. Stonebraker’s own team admits one of their databases only ran fast after help from the vendor’s experts. But that cost is paid by a few people, mostly once. After that, every query benefits. The data is already parsed and laid out, so the daily dashboards are fast.

Hadoop is cheap to stand up. Stonebraker admits that too: Hadoop had the best out-of-the-box experience. But every run re-reads and re-parses the raw data. And every new question needs a new program, which was the pushback I got when talking this through with Claude.

And yes, I used AI to discuss ideas. I do this quite often, and I find it quite useful. I think it’s more important to be transparent about AI use, which is why I declared it.

While iterating with Claude, I actually unknowingly assumed that DBMS was more suitable as setup cost is incurred once, and the same workflows would be more efficient. This is technically true if the assumption holds, but analysts do not run the same dashboard forever.

To drive this point home for me, Claude came up with another scenario:

Say the head of marketing asks how the new checkout page affected repeat buyers in Singapore.

In SQL, an analyst writes something like this in ten minutes:

SELECT COUNT(DISTINCT o.customer_id)
FROM orders o JOIN sessions s ON o.session_id = s.session_id
WHERE s.page_version = 'new_checkout'
  AND o.country = 'SG'
  AND o.customer_id IN (SELECT customer_id FROM orders
                        GROUP BY customer_id HAVING COUNT(*) > 1);

My first instinct when I saw this was to focus on GROUP BY and COUNT. These are aggregation functions, and Stonebraker et al. argued that DBMS is better at this than MapReduce.

Then if we consider 2009 Hadoop, that is a join, a filter and a grouped count. It is likely two chained Java jobs, each compiled, submitted and debugged. And almost every business question looks like this: filter, join, aggregate. Even Dean concedes that selection and aggregation are SQL’s strength.

So for the analysts, DBMS is a more sensible choice. Choosing DBMS meant we had to incur painful setup and tuning. That requires a (few) data wizard(s) paid up front. But compared to choosing Hadoop, having to first send the analysts for “Java 101” classes and having them write a program for every query, maybe the upfront payment for the wizards (not the Hogwarts ones) doesn’t seem so bad.

Loud failures and silent ones

So far so good! Sounds like DBMS is the way to go? Until it isn’t. Well, more specifically, when we consider upstream: how do we get the nice, structured data the analysts work with? That table (or equivalent form) does not exist yet. Someone has to do the dirty work and build it from 50GB a day of messy logs. And the logs change whenever the front-end team changes anything. Yikes.

Consider this scenario (again, helpfully generated by Claude. Oops.)

Say the front-end team renames a field on Monday. checkout_ver becomes checkout_version. What would happen?

If the logs are loaded straight into a database table, nothing crashes: the old column is simply empty for every new row. Looks good on paper, but a silent error happens.

The monthly report still reads checkout_ver instead of checkout_version, and from that day on, that column will not have data populated. Nothing errors.

The problem? Using NULL when fitting unstructured/semi-structured data into DBMS. In unstructured data, not all records will have all fields: some will have fields that belong to them and only them, some will have some overlap, but most of the time fields come in all shapes and forms.

By using NULL, we can represent fields that do not exist under the record so every record can exist in the same table, with columns denoting the possible fields that exist.

This sounded great on paper, even to me! Until further back-and-forths (with Claude, again) helped me understand that the above scenario actually would fail here. Why? checkout_version is unrecognised, so it would not be loaded into the DBMS, and checkout_ver does not exist as a field, so when loading records, the value for that column will become NULL.

So a NULL can serve as a marker for fields that do not exist. But, it can also mean “this was recorded under a different name”. The system cannot tell the difference. A column store can hold that NULL very efficiently, but something is still missing.

That is the difference between “failing loudly” and “failing silently”. If the data is checked against a schema when it is loaded, the rename fails loudly. An engineer fixes the mapping, and the analysts never see bad numbers. If structure is only applied when someone queries, the failure is silent, and someone makes a decision on it.

If you were to ask me, I would prefer “loud failures” over “silent ones”: easier to debug, more timely interventions. Failing loudly has its own cost, though. While the load is broken, Monday’s data is not in the warehouse at all.

How do we solve that?

Keeping raw data

In the above scenario, a concern is that after loading in Monday’s data and realising there is an error, we would lose the original data and therefore lose the opportunity to load the data in correctly.

A simple workaround is to retain a copy of the raw data and use that as a “ground truth”, while other copies can be manipulated. We therefore then can run a job on one copy of the raw data that cooks them into an intermediate sessions table (which essentially is the process of loading raw data into a structured table). The job checks every record against the fields it expects. When it sees an unknown field, it raises an alarm but keeps writing everything else. This can be achieved using MapReduce, which Dean claimed has fault tolerance, so jobs do not need to restart from the beginning.

If we go back to our small made-up world: On Tuesday, someone fixes the checkout_ rule. And since we kept the raw data, Monday’s records are not lost. We can rerun the fixed job over Monday’s raw files and rebuild the correct rows, and Monday is repaired as if nothing happened. If the raw files had been thrown away after loading, Monday’s rows would stay empty forever.

The design decision that makes this possible is simple. Keep the raw data, untouched, next to the cleaned tables. The raw copy is the ground truth. The tables are just today’s best interpretation of it. When the interpretation is wrong, rebuild it.

Today this is how many data lakes are organised. Databricks popularised names for the layers:

  • Bronze: raw data, exactly as it arrived.
  • Silver: cleaned, validated and joined. The sessions table lives here.
  • Gold: small tables shaped for one business use, like daily revenue by country. The dashboards read these.

But with every solution, there is a cost: we store everything at least twice, pay for compute every time a job is rerun, and a raw data layer nobody looks after turns into a “data swamp”. But at least, this turns a silent failure into a recoverable one.

The assumption both authors shared

This is where I think both articles miss something.

Look at what each one assumes happens to raw data after it is processed. Stonebraker calls MapReduce’s best use case “read once” data sets. Dean says data sets are often “processed once or twice, and then discarded”. The two sides disagree about almost everything else. But they agree that raw data is temporary. The processed result is what matters. The raw copy goes in the bin.

On the contrary, I think raw data is the permanent asset. Processed tables are temporary interpretations that can always be rebuilt. Every fixed bug, every new question and every new model starts from the raw history.

Why it made sense in 2009, and what changed

To be fair to both of them, discarding raw data was a reasonable assumption at the time.

Storing it was not impossible. Google already had petabyte-scale data sets. The problem was that storage was expensive in a particular way. In a Hadoop cluster, storage and compute lived on the same machines. Keeping more data meant buying more servers, even if those servers mostly sat idle.

Two things changed since then:

  1. Storage split off from compute. Cheap object storage like Amazon S3 lets you keep everything for very little, and rent compute only when you need to reprocess. Open file formats like Parquet let many different tools read the same files.
  2. Old raw data became valuable. Large language models are trained on years of raw web crawl that someone chose not to throw away.

So two forces moved in opposite directions. Keeping raw data got cheaper just as it got more valuable.

The data scientists

So far the database has won the analysts’ side of the debate. But the data scientists are where Dean’s argument is strongest.

In our make-believe world, they have two jobs:

First, sessionisation. Take each user’s clicks, sort them by time, and start a new session whenever there is a gap of more than 30 minutes. This is called a session window. Each click only makes sense relative to the previous click by the same user. So you have to group, sort and walk through them in order.

That was awkward in 2009 SQL. It fits MapReduce naturally. Map emits each click keyed by user. The shuffle groups clicks by user. Each reducer sorts one user’s clicks and walks through them. The input is messy raw logs, which a database could not load cleanly anyway. If we adopt Databricks’ layering terminology, this job “reads bronze” and “writes silver”.

To be fair to SQL, modern SQL handles this with window functions like LAG.

Second, recommendations. A typical algorithm makes many passes over the data, and each pass refines the last. Stonebraker himself gives this kind of work to MapReduce. He says multi-pass analytics cannot be written as a single SQL aggregate query.

But MapReduce is slow at it too. Each pass is a full job. It reads everything from disk and writes everything back to disk before the next pass starts. Fifty passes means fifty round trips. And unlike the analysts, the data scientists pay this cost every week, since they have to run this every week.

The fix was to keep the working data in memory between passes. Load and parse once, hold the data across the cluster, and run every pass against it. That is what Spark did. It came out of Berkeley around 2010, motivated by exactly this kind of iterative job.

But with that, a new problem: memory is lost when a machine fails. Spark needed a new safety net. It remembers how each piece of data was computed. If a piece is lost, it recomputes only that piece from its inputs.

That only works if the inputs still exist.

And surprise surprise, keeping the raw data is what makes this kind of recovery possible.

The verdict

Dean wins here, but not with MapReduce. Three of his ideas won:

  1. Process data where it sits, which led to data lakes.
  2. Survive failures in the middle of big jobs, which every large system now does. Meta’s Llama 3 pre-training saw 466 interruptions in a 54-day period and still kept over 90% effective training time.
  3. And real work needs more than SQL.

But Stonebraker won too, more quietly. MapReduce itself did not last. Within a year of these articles, Google moved its search index off MapReduce, because it needed the index updated continuously rather than in big batches. In 2014, it said it no longer used MapReduce at all. The systems that replaced it, like Spark and Flink, started as better versions of MapReduce. Then they added SQL.

Closing

In 2010, these articles were framed as a fight. Sixteen years later, it reads more like a negotiation. Each side was right about its own users. The database people were right about the analysts. The MapReduce people were right about messy data and long jobs. The systems we use now took the best of both.

References

  • Stonebraker, M. et al. “MapReduce and Parallel DBMSs: Friends or Foes?” Communications of the ACM 53(1), 2010.
  • Dean, J. and Ghemawat, S. “MapReduce: A Flexible Data Processing Tool.” Communications of the ACM 53(1), 2010.
  • Pavlo, A. et al. “A Comparison of Approaches to Large-Scale Data Analysis.” SIGMOD 2009.
  • Stonebraker, M. and Pavlo, A. “What Goes Around Comes Around… And Around…” SIGMOD Record 53(2), 2024.
  • Llama Team, Meta. “The Llama 3 Herd of Models.” 2024.
  • Zaharia, M. et al. “Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing.” NSDI 2012.

My yap