The top marketing executive at a sizable US retailer recently found herself perplexed by the sales reports she was getting. A major competitor was steadily gaining market share across a range of profitable segments. Despite a counterpunch that combined online promotions with merchandizing improvements, her company kept losing ground.
When the executive convened a group of senior leaders to dig into the competitor’s practices, they found that the challenge ran deeper than they had imagined. The competitor had made massive investments in its ability to collect, integrate, and analyze data from each store and every sales unit and had used this ability to run myriad real-world experiments. At the same time, it had linked this information to suppliers’ databases, making it possible to adjust prices in real time, to reorder hot-selling items automatically, and to shift items from store to store easily. By constantly testing, bundling, synthesizing, and making information instantly available across the organization—from the store floor to the CFO’s office—the rival company had become a different, far nimbler type of business.
What this executive team had witnessed first hand was the game-changing effects of big data. Of course, data characterized the information age from the start. It underpins processes that manage employees; it helps to track purchases and sales; and it offers clues about how customers will behave.
But over the last few years, the volume of data has exploded. In 15 of the US economy’s 17 sectors, companies with more than 1,000 employees store, on average, over 235 terabytes of data—more data than is contained in the US Library of Congress. Reams of data still flow from financial transactions and customer interactions but also cascade in at unparalleled rates from new devices and multiple points along the value chain. Just think about what could be happening at your own company right now: sensors embedded in process machinery may be collecting operations data, while marketers scan social media or use location data from smartphones to understand teens’ buying quirks. Data exchanges may be networking your supply chain partners, and employees could be swapping best practices on corporate wikis.
All of this new information is laden with implications for leaders and their enterprises.1 Emerging academic research suggests that companies that use data and business analytics to guide decision making are more productive and experience higher returns on equity than competitors that don’t.2 That’s consistent with research we’ve conducted showing that “networked organizations” can gain an edge by opening information conduits internally and by engaging customers and suppliers strategically through Web-based exchanges of information.3
Over time, we believe big data may well become a new type of corporate asset that will cut across business units and function much as a powerful brand does, representing a key basis for competition. If that’s right, companies need to start thinking in earnest about whether they are organized to exploit big data’s potential and to manage the threats it can pose. Success will demand not only new skills but also new perspectives on how the era of big data could evolve—the widening circle of management practices it may affect and the foundation it represents for new, potentially disruptive business models.
Five big questions about big data
In the remainder of this article, we outline important ways big data could change competition: by transforming processes, altering corporate ecosystems, and facilitating innovation. We’ve organized the discussion around five questions we think all senior executives should be asking themselves today.
At the outset, we’ll acknowledge that these are still early days for big data, which is evolving as a business concept in tandem with the underlying technologies. Nonetheless, we can identify big data’s key elements. First, companies can now collect data across business units and, increasingly, even from partners and customers (some of this is truly big, some more granular and complex). Second, a flexible infrastructure can integrate information and scale up effectively to meet the surge. Finally, experiments, algorithms, and analytics can make sense of all this information. We also can identify organizations that are making data a core element of strategy. In the discussion that follows, we have assembled case studies of early movers in the big data realm (see “Seizing the potential of ‘big data’” and the accompanying sidebar, “AstraZeneca’s ‘big data’ partnership.”)
Parsing the benefits: Not all industries are created equal
Even as big data changes the game for virtually every sector, it also tilts the playing field, favoring some companies and industries, particularly in the early stages of adoption. To understand those dynamics, we examined 20 sectors in the US economy, sized their contributions to GDP, and developed two indexes that estimate each sector’s potential for value creation using big data, as well as the ease of capturing that value.1
As the accompanying sector map shows (exhibit), financial players get the highest marks for value creation opportunities. Many of these companies have invested deeply in IT and have large data pools to exploit. Information industries, not surprisingly, are also in this league. They are data intensive by nature, and they use that data innovatively to compete by adopting sophisticated analytic techniques.
The ease of capturing big data’s value, and the magnitude of its potential, vary across sectors.
The public sector is the most fertile terrain for change. Governments collect huge amounts of data, transact business with millions of citizens, and, more often than not, suffer from highly variable performance. While potential benefits are large, governments face steep barriers to making use of this trove: few managers are pushed to exploit the data they have, and government departments often keep data in siloes.
Fragmented industry structures complicate the value creation potential of sectors such as health care, manufacturing, and retailing. The average company in them is relatively small and can access only limited amounts of data. Larger players, however, usually swim in bigger pools of data, which they can more readily use to create value.
The US health care sector, for example, is dotted by many small companies and individual physicians’ practices. Large hospital chains, national insurers, and drug manufacturers, by contrast, stand to gain substantially through the pooling and more effective analysis of data. We expect this trend to intensify with changing regulatory and market conditions. In manufacturing, too, larger companies with access to much internal and market data will be able to mine new reservoirs of value. Smaller players are likely to benefit only if they discover innovative ways to share data or grow through industry consolidation. The same goes for retailing, where—despite a healthy strata of data-rich chains and big-box stores on the cutting edge of big data—most players are smaller, local businesses with a limited ability to gather and analyze information.
A final note: this analysis is a snapshot in time for one large country. As companies and organizations sharpen their data skills, even low-ranking sectors (by our gauges of value potential and data capture), such as construction and education, could see their fortunes change.
In addition, we’d suggest that executives look to history for clues about what’s coming next. Earlier waves of technology adoption, for example, show that productivity surges not only because companies adopt new technologies but also, more critically, because they can adapt their management practices and change their organizations to maximize the potential. We examined the possible impact of big data across a number of industries and found that while it will be important in every sector and function, some industries will realize benefits sooner because they are more ready to capitalize on data or have strong market incentives to do so (see sidebar, “Parsing the benefits: Not all industries are created equal”).
The era of big data also could yield new management principles. In the early days of professionalized corporate management, leaders discovered that minimum efficient scale was a key determinant of competitive success. Likewise, future competitive benefits may accrue to companies that can not only capture more and better data but also use that data effectively at scale. We hope that by reflecting on such issues and the five questions that follow, executives will be better able to recognize how big data could upend assumptions behind their strategies, as well as the speed and scope of the change that’s now under way.
1. What happens in a world of radical transparency, with data widely available?
As information becomes more readily accessible across sectors, it can threaten companies that have relied on proprietary data as a competitive asset. The real-estate industry, for example, trades on information asymmetries such as privileged access to transaction data and tightly held knowledge of the bid and ask behavior of buyers. Both require significant expense and effort to acquire. In recent years, however, online specialists in real-estate data and analytics have started to bypass agents, permitting buyers and sellers to exchange perspectives on the value of properties and creating parallel sources for real-estate data.
Beyond real estate, cost and pricing data are becoming more accessible across a spectrum of industries. Another swipe at proprietary information is the assembly by some companies of readily available satellite imagery that, when processed and analyzed, contains clues about competitors’ physical facilities. These satellite sleuths glean insights into expansion plans or business constraints as revealed by facility capacity, shipping movements, and the like.
One big challenge is the fact that the mountains of data many companies are amassing often lurk in departmental “silos,” such as R&D, engineering, manufacturing, or service operations—impeding timely exploitation. Information hoarding within business units also can be a problem: many financial institutions, for example, suffer from their own failure to share data among diverse lines of business, such as financial markets, money management, and lending. Often, that prevents these companies from forming a coherent view of individual customers or understanding links among financial markets.
Some manufacturers are attempting to pry open these departmental enclaves: they are integrating data from multiple systems, inviting collaboration among formerly walled-off functional units, and even seeking information from external suppliers and customers to cocreate products. In advanced-manufacturing sectors such as automotive, for example, suppliers from around the world make thousands of components. More integrated data platforms now allow companies and their supply chain partners to collaborate during the design phase—a crucial determinant of final manufacturing costs.
2. If you could test all of your decisions, how would that change the way you compete?
Big data ushers in the possibility of a fundamentally different type of decision making. Using controlled experiments, companies can test hypotheses and analyze results to guide investment decisions and operational changes. In effect, experimentation can help managers distinguish causation from mere correlation, thus reducing the variability of outcomes while improving financial and product performance.
Robust experimentation can take many forms. Leading online companies, for example, are continuous testers. In some cases, they allocate a set portion of their Web page views to conduct experiments that reveal what factors drive higher user engagement or promote sales. Companies selling physical goods also use experiments to aid decisions, but big data can push this approach to a new level. McDonald’s, for example, has equipped some stores with devices that gather operational data as they track customer interactions, traffic in stores, and ordering patterns. Researchers can model the impact of variations in menus, restaurant designs, and training, among other things, on productivity and sales.
Where such controlled experiments aren’t feasible, companies can use “natural” experiments to identify the sources of variability in performance. One government organization, for instance, collected data on multiple groups of employees doing similar work at different sites. Simply making the data available spurred lagging workers to improve their performance.
Leading retailers, meanwhile, are monitoring the in-store movements of customers, as well as how they interact with products. These retailers combine such rich data feeds with transaction records and conduct experiments to guide choices about which products to carry, where to place them, and how and when to adjust prices. Methods such as these helped one leading retailer to reduce the number of items it stocked by 17 percent, while raising the mix of higher-margin private-label goods—with no loss of market share.
3. How would your business change if you used big data for widespread, real-time customization?
Customer-facing companies have long used data to segment and target customers. Big data permits a major step beyond what until recently was considered state of the art, by making real-time personalization possible. A next-generation retailer will be able to track the behavior of individual customers from Internet click streams, update their preferences, and model their likely behavior in real time. They will then be able to recognize when customers are nearing a purchase decision and nudge the transaction to completion by bundling preferred products, offered with reward program savings. This real-time targeting, which would also leverage data from the retailer’s multitier membership rewards program, will increase purchases of higher-margin products by its most valuable customers.
Retailing is an obvious place for data-driven customization because the volume and quality of data available from Internet purchases, social-network conversations, and, more recently, location-specific smartphone interactions have mushroomed. But other sectors, too, can benefit from new applications of data, along with the growing sophistication of analytical tools for dividing customers into more revealing microsegments.
One personal-line insurer, for example, tailors insurance policies for each customer, using fine-grained, constantly updated profiles of customer risk, changes in wealth, home asset value, and other data inputs. Utilities that harvest and analyze data on customer segments can markedly change patterns of power usage. Finally, HR departments that more finely segment employees by task and performance are beginning to change work conditions and implement incentives that improve both satisfaction and productivity.4
4. How can big data augment or even replace management?
Big data expands the operational space for algorithms and machine-mediated analysis. At some manufacturers, for example, algorithms analyze sensor data from production lines, creating self-regulating processes that cut waste, avoid costly (and sometimes dangerous) human interventions, and ultimately lift output. In advanced, “digital” oil fields, instruments constantly read data on wellhead conditions, pipelines, and mechanical systems. That information is analyzed by clusters of computers, which feed their results to real-time operations centers that adjust oil flows to optimize production and minimize downtimes. One major oil company has cut operating and staffing costs by 10 to 25 percent while increasing production by 5 percent.
Products ranging from copiers to jet engines can now generate data streams that track their usage. Manufacturers can analyze the incoming data and, in some cases, automatically remedy software glitches or dispatch service representatives for repairs. Some enterprise computer hardware vendors are gathering and analyzing such data to schedule preemptive repairs before failures disrupt customers’ operations. The data can also be used to implement product changes that prevent future problems or to provide customer use inputs that inform next-generation offerings.
Some retailers are also at the forefront of using automated big data analysis: they use “sentiment analysis” techniques to mine the huge streams of data now generated by consumers using various types of social media, gauge responses to new marketing campaigns in real time, and adjust strategies accordingly. Sometimes these methods cut weeks from the normal feedback and modification cycle.
But retailers aren’t alone. One global beverage company integrates daily weather forecast data from an outside partner into its demand and inventory-planning processes. By analyzing three data points—temperatures, rainfall levels, and the number of hours of sunshine on a given day—the company cut its inventory levels while improving its forecasting accuracy by about 5 percent in a key European market.
The bottom line is improved performance, better risk management, and the ability to unearth insights that would otherwise remain hidden. As the price of sensors, communications devices, and analytic software continues to fall, more and more companies will be joining this managerial revolution.
5. Could you create a new business model based on data?
Big data is spawning new categories of companies that embrace information-driven business models. Many of these businesses play intermediary roles in value chains where they find themselves generating valuable “exhaust data” produced by business transactions. One transport company, for example, recognized that in the course of doing business, it was collecting vast amounts of information on global product shipments. Sensing opportunity, it created a unit that sells the data to supplement business and economic forecasts.
Another global company learned so much from analyzing its own data as part of a manufacturing turnaround that it decided to create a business to do similar work for other firms. Now the company aggregates shop floor and supply chain data for a number of manufacturing customers and sells software tools to improve their performance. This service business now outperforms the company’s manufacturing one.
Big data also is turbocharging the ranks of data aggregators, which combine and analyze information from multiple sources to generate insights for clients. In health care, for example, a number of new entrants are integrating clinical, payment, public-health, and behavioral data to develop more robust illness profiles that help clients manage costs and improve treatments.
And with pricing data proliferating on the Web and elsewhere, entrepreneurs are offering price comparison services that automatically compile information across millions of products. Such comparisons can be a disruptive force from a retailer’s perspective but have created substantial value for consumers. Studies show that those who use the services save an average of 10 percent—a sizable shift in value.
Confronting complications
Up to this point, we have emphasized the strategic opportunities big data presents, but leaders must also consider a set of complications. Talent is one of them. In the United States alone, our research shows, the demand for people with the deep analytical skills in big data (including machine learning and advanced statistical analysis) could outstrip current projections of supply by 50 to 60 percent. By 2018, as many as 140,000 to 190,000 additional specialists may be required. Also needed: an additional 1.5 million managers and analysts with a sharp understanding of how big data can be applied. Companies must step up their recruitment and retention programs, while making substantial investments in the education and training of key data personnel.
The greater access to personal information that big data often demands will place a spotlight on another tension, between privacy and convenience. Our research, for example, shows that consumers capture a large part of the economic surplus that big data generates: lower prices, a better alignment of products with consumer needs, and lifestyle improvements that range from better health to more fluid social interactions.5 As a larger amount of data on the buying preferences, health, and finances of individuals is collected, however, privacy concerns will grow.
That’s true for data security as well. The trends we’ve described often go hand in hand with more open access to information, new devices for gathering it, and cloud computing to support big data’s weighty storage and analytical needs. The implication is that IT architectures will become more integrated and outward facing and will pose greater risks to data security and intellectual property. For some ideas on how leaders should respond, see “Meeting the cybersecurity challenge.”
Although corporate leaders will focus most of their attention on big data’s implications for their own organizations, the mosaic of company-level opportunities we have surveyed also has broader economic implications. In health care, government services, retailing, and manufacturing, our research suggests, big data could improve productivity by 0.5 to 1 percent annually. In these sectors globally, it could produce hundreds of billions of dollars and euros in new value.
In fact, big data may ultimately be a key factor in how nations, not just companies, compete and prosper. Certainly, these techniques offer glimmers of hope to a global economy struggling to find a path toward more rapid growth. Through investments and forward-looking policies, company leaders and their counterparts in government can capitalize on big data instead of being blindsided by it.
