Next Shift: From Big Data to Deep Data

It is no a secret that today’s consumer looks and acts differently than the consumer of fifteen years ago.

While in the past, consumers may have been satisfied with standardized services and solutions that matched those of their neighbours, friends or acquaintances, people now expect products and services that best fit their individual needs and interests.

We have already seen this play out in multiple industries, perhaps most visibly in the telecom world. Thanks to smartphones and data networks, that industry has moved from a system of telephone poles and wires that delivered the same services to everyone, to one defined by personalized service and application delivery which allows consumers to use their phones in ways that best fit their lifestyles. Telecom companies actively leverage their consumers’ behavioural, transactional and demographic data to personalize their services and engage customers more deeply than they did just a few years ago. Today, these data-centric approaches to customer engagement are redefining other industries like energy, healthcare and retail, which all rely on good data to stay relevant to their customers.

With the growing presence of big data, businesses are becoming more equipped to handle the tsunami of data about their customers. However, because of a lack of precise knowledge of the value embedded within this huge crush of data, many businesses have been stuck in the “data for data’s sake” trap, capturing every piece of available information.

Big data to deep dataWith this approach, it is easy to become overwhelmed by the amount of data available and get stuck focusing on the challenges to collect and store terabytes of information, rather than effectively leverage the information to deliver on the promise of personalized relationships and to add meaningful business value.

As big data moves beyond hype to realized value, things are beginning to change. As we enter 2015, companies will move toward the “Deep Data” framework, an approach based on the premise that a small number of information, rich data streams, leveraged properly, can yield more value than masses of captured data. By shifting to a deep, rather than big, data approach, businesses are able to better understand their customers and offer actionable, scalable and customized insights while crucially enhancing the value of the economic investment in data to their businesses.

As consumers become “prosumers”, a mix of producer and consumer, who create new energy options, often at their own sites, utilities must find new, innovative ways to deliver value to customers who want the most cost-effective, energy efficient options.

On the technical front, a successful deep data company would meld and leverage three core elements: domain expertise (expert knowledge of the specifics of the business), data science (a set of specialized computational and mathematical skills that enable scaling of data-centric insight), and the right IT infrastructure. Companies typically have in-house experts available, but do not always leverage their deep expertise fully by abstracting their specific insights to build models of consumer behaviour. Getting these resources together with the technical folks is a key element.

big data deep dataThe second resource, data scientists, is an emerging lineage of specialists trained to leverage modern computing and mathematical knowledge in the context of specific business problems.

Regardless of industry, deep data is poised to disrupt the way organizations engage customers. Those that embrace the deep data model will see success in sales and customer engagement, while those that shy away from analytics will fail to meet consumer demand – ultimately failing to compete in the fast paced and fast changing world. As data-savvy organizations rise to the top, we can expect more options and opportunities to make well-informed spending decisions in 2015.