Big data: how big is it? And does it really matter?

Big data is a phrase that gets bandied around so often that it’s easy to become blasé about what it actually means.
But I saw an interesting infographic the other day that helped contextualise the sheer size of the big data challenge.
It goes something like this.

If one byte of data equals one grain of rice, then:

1 kilobyte = 1 cup of rice

1 megabyte = 8 sacks of rice

1 gigabyte = 3 container trucks

1 terabyte = 2 container ships

1 petabyte = covers Manhattan

1 exabyte = covers the US West Coast

1 zettabyte = fills the Pacific Ocean

Yikes. According to the experts, we filled the Pacific years ago. IDC says that in 2011 we created 1.8 zettabytes (ZB) of information. In 2012 it reached 2.8ZB. And, assuming that volumes continue to double every two-and-a-half years, we will generate 40ZB by 2020.

Let’s take a minute for that to sink in.

Back in 1998, my mid-range iMac G3 had a hard-drive of just 4GB – pitifully small by today’s standards. But even so, using the above list, that still equated to 12 container lorries worth of storage. That’s a lot of rice. Plenty for one person, you would think – certainly back in the days before iTunes, and when digital cameras were still in their infancy.

What has happened since is an unprecedented explosion in the volumes of data generated and stored. According to Cisco, 80 things are connecting to the internet per second, and by 2020 there will be 50 billion connected devices – everything from smartphones to cars to refrigerators. In the age of the internet of things, sensors will be embedded in everything.

“By 2020 there will be 50 billion connected devices”

Take the example of a jet engine. According to National Instruments, for every 30 minutes that a Boeing jet engine runs, the system creates 10 terabytes of operations information. That means for a single journey across the Atlantic Ocean, a four-engine jumbo jet can create 640 terabytes of data. Multiply that by the more than 25,000 flights flown daily, and you get an understanding of the enormous amount of data that’s generated each day.

Similarly, it’s now estimated that for every 30,000 miles of natural gas pipeline, we generate 17 terabytes of data each day – more data than the entire printed collection of the Library of Congress. There are more than 305,000 miles of natural gas pipelines in the United States alone.

No wonder executives are freaking out about how to cope. I’ve moderated countless roundtables on the big data topic, and the refrain is always the same: how do we cut through the noise and get to the information that really matters?

“There is so much data available, and we’re only using a portion of that; it’s overwhelming. We need to prioritise,” complained one executive, head of upstream operations for a major global oil and gas company. “Big data represents a huge asset, but also a challenge in terms of how we store, analyse and use it,” said another, head of data for a large British utility firm.

The scale of the challenge is significant, with the latest research suggesting that petroleum engineers, for example, now typically spend up to 60 percent of their time on data mining.

But focusing on size alone is something of a red herring. Because here’s the thing about all this data: sure the scale is massive – way beyond the scope of anything we’ve ever seen before. But the challenge still remains the same as it always has done. How do we get to the insight? How do we mine our data for useful information that can make a meaningful difference?

“The answer is in the data, and we need to grow our skills”

If you start with this mindset, then size becomes irrelevant. The key, according to those who have already begun the journey, is to focus on the outcomes you are looking for. “It’s about getting business users together and getting them to talk about objectives,” says one, head of analytics at a major banking group. “You need to have a business strategy around data,” says another, head of digital at a major investment house. “People have to be able to see something tangible in order to see value.”

“Technology won’t tell us the answer,” concludes the head of big data at one of the world’s largest financial services businesses. “The answer is in the data, and we need to grow our skills to allow the data to tell us that.”

It’s all about unlocking the value. Going back to our rice analogy, big data is like counting grains in front of a hungry man. He doesn’t care about the number of grains. He just wants something to eat.
Likewise with business: it’s not the size of the data that’s important, but the value it holds within.

Arrange a Conversation