Making the most of big data

With enterprises generating ever greater piles of data, and the costs of storing the information falling ever cheaper, more businesses than ever are recognizing the power of big data in resource and strategy planning.

But a report from the Economist Intelligence Unit shows that big data is next to worthless without the capacity to make effective use of it.
The SAS-sponsored report, based on a survey of 586 senior executives, found that there is a growing disparity between enterprises able to make effective use of big data, and those collecting data despite lacking the resources or processes to properly analyze it.
The report urges businesses without an effective big data strategy to rethink their entire approach to data management, strategically, operationally and culturally. It asserts that the era of big data has only just begun.
Strategic data management
The survey shows that big data is an increasingly valuable resource for enterprises, and data retention is swelling as a result. Nearly three-quarters of respondents agreed that their organizations' data collection practices have increased in the last year.
The growing importance of big data is already stimulating a significant change in the way businesses approach data management. 
While historically, decisions on data management would typically fall to a mid-level IT manager, perhaps with specialized database skills, it is today becoming more common for these decisions to be set by the CEO or another C-level executive.
Around 44% of respondents to EIUs survey state that big data decisions are made by the CEO or another senior executive, while another 42% say the duties fall on the CIO or another senior-level IT manager. Just 7% of businesses still leave this task to mid-level IT.
Companies are divided on their purposes for collecting data. While a majority of respondents hope to improve operational efficiency from the data, other motivations vary wildly. 
Some want to use big data to inform their strategic direction, others are aiming to use it as a guide for developing new products or services, or for improving the customer experience.
Other motivations include identifying new markets for existing products or services, expediting time-to-market or complying with regulations.
But there is a growing gulf between companies able to effectively utilize big data sets to grow their business, and those struggling to make effective use of the resource.
The EIU's report concludes that companies fall into one of four categories -- or stages of evolution -- of data management.
At the top are the “strategic data managers,” which have well-defined data management strategies in place, focusing resources on analyzing the most valuable data. Below this are “aspiring data managers,” which are putting in place the resources to graduate to the strategic category.
“Data collectors” collect large amounts of data but do not consistently extract optimal value, while at the bottom, “data wasters” gather data but then severely underuse the resource.
Executives from companies that use big data effectively clearly benefit financially. The survey shows that 53% of strategic data managers say they outperformed their peers last financial year, with 44% stating they are on par and just 1% stating they are behind.
By contrast, just 36% of data collectors and 24% of data wasters outmatched their competitors financially – and nearly a third of data wasters reported lagging their rivals.
Big data, big challenges
Despite the benefits big data analytics can bring, the survey results show that many companies are struggling to make effective use of the resource.
Many enterprises are struggling under the weight of the sheer volume of data. According to the report, most businesses are still of the mindset that it is better to collect as much data as possible now, and figure out what to do with it later.
But businesses need to take a step back and realize that data by itself is meaningless without the mechanisms to properly analyze it. 
Nearly one in four respondents lamented that the vast majority of their organization’s data remains untapped, while another 53% reported that they only use around half their important data. Only 22% said they put the majority of their most valuable data to good use.
In contrast to the “strategic” data managers, many companies are still struggling with the most basic aspect of data management and governance. Nearly a third of respondents admitted to having no formal process around data management, but were scared to stop gathering data.
Some lack the workforce skills required to analyze big data or have inefficient processes for verifying information or reconciling it with other data sets. Many are also unable to ensure that the data falls into the right hands at the right time.
When the survey participants were asked about their major big data problems and challenges, the most popular responses were reconciling different data sets, the quality and accuracy of the information and the lack of an organizational view into big data.  
The report concludes that businesses that fail to surmount these challenges and graduate to the ranks of strategic data managers will risk getting left behind in the big data era.