Print Page   |   Contact Us   |   Your Cart   |   Sign In   |   Register
Site Search
Event Calendar

7/17/2017 » 7/18/2017
The Product Bootcamp: Discovery, Visioning, and Roadmap Prioritization

BPMA and BWP Summer Party!

The Art of Writing High-Fidelity User Stories: Product Usability Starts Here!

Big data - the secret sauce for building great products?
Share |

By Karthik Yajurvedi

Published December 25, 2013

Big Data heralds a new area of computing - an inflection point where data in any format may be explored and harnessed for ground-breaking insights. This article defines Big Data and Big Data Analytics and how the myriad sources of big data can be utilized like never before to understand customer behavior, sentiment and unmet needs to build truly great products.

Category: Technologies and Trends


Are you delivering what your customers really need? Are your products delivering business value? How satisfied are your customers with your service? Can you have insight into what else they might need and buy? Enter Big data and Big Data Analytics (BDA) to help provide deep knowledge, discover hidden patterns, unknown correlations of current and future market behaviors.

The promise

May not be the magic bullet to solve all problems, but Big Data analytics has been emerging beyond its hype, with the potential to transform the way organizations do business. According to Gartner's 2012 and 2013 studies and survey, business cases around customer experience dominate big data wish lists and 55 percent of organizations said that they are currently addressing enhanced customer experience using Big Data. Cutter IT Journal[i] states that companies that fully realize this discipline can reap great rewards in terms of increased profitability, potentially by as much as 20%[ii]. There is promise that products can use big data and real-time analytics to study the impact of marketing campaigns to enhance customer experience and user behavior. Product innovation, being a costly, time consuming effort resulting in majority of new products failing as they enter the market due to the trial and error approach, companies are now looking into platforms like Hadoop to integrate  big data into the front end of innovation pipeline to shorten time to market, reduce costs, improve product adoption and create innovative post-sales service offerings. The hard part is for companies to know where to start and how to connect the dots. Before we address BDA's "why" and "how" specifically related to marketing and product development, let us look under the hood to understand "what" is Big Data and BDA.

Big Data and Big Data Analytics definitions

Big Data includes data from social networks, Internet text and searches, blogs, call details records, medical records, e-commerce records, data used by sciences, photo and video archives, purchase transactions, smartphone applications, sensors etc. With storage and computing costs decreasing daily, it has been possible to store these large data sets even with the ever increasing sources of data. Big Data is also defined by Doug Laney's[iii] "3 Vs" - volume, velocity and variety to describe the large amounts of data generated by the myriad data sources at a rapid speed. BDA, as the name implies is a combination of Big Data and analytics. The term analytics from a business perspective is to study, dissect and extract valuable insights from data to identify patterns, and assess performance. It is also the analysis of decisions and events to quantify their impacts on business; improved ability to profile an individual or a demographic group of customers to better understand and satisfy their purchasing behaviors, which can then be reflected in the end product.  Big Data by itself is not that useful , but when used in conjunction with analytics, is powerful enough to generate true business value.

Opportunity knocks for Marketing and Product organizations 

What can BDA bring to the table for product development and marketing? By turning the concept of Big Data inside out, companies can make their products and product development better. Volumes of data are generated internally from product research activities, sensors, logs and usage statistics that creates huge opportunities to understand how customers use the products and feed that information back into the design process to help optimize the user's experience in the future. Marketing organizations can harness the power of Big Data as well. While market research has leaned on well established processes such as periodic surveys, focus groups, behavior observation and ethnographic studies, it no longer can scale to the requirements of global markets or mass customization. Even the newer techniques like crowdsourcing that enable product development to be much more precisely focused and organic by brainstorming with a potential market in the conceptual stage relies heavily on picking the right segment and asking the right question. The latest strategy is to accumulate data from sources closest to the customer such as retail, insurance, transportation, telecom, healthcare providers, utilities and banks, coupled with data from social media, location-based services and sensors. By applying advanced analytics in the categories of text, voice, video and social media, it is possible to glean deep insights about the voice of the customer, sentiment analysis, opinion mining and intelligence that can be channeled towards adjustments to brand management, messaging and product development.      

Putting BDA into action

• Assemble a cross-functional "A" team comprising of business, IT, management and human resources. It is vital to get complete buy-in and focus from all parties to implement BDA and reap true business value.

• IT to identify the correct technology platform; management to devise corporate strategy for Big Data and human resources to upgrade the organizational skill set.

• Explore and clearly define the business case. It is critical to search deeply and identify the data in the context of its usage, fully mindful of the business value.  

• Initiate a "proof-of concept" to evaluate if BDA can indeed tackle the identified problem. Business and IT to work closely together to make sure business architecture and data architecture are in synch.

• On successfully proving the concept, final implementation needs the whole organization to embrace the changes to truly harness value from Big Data. Training and education to unlock precious data out of any silos must be fostered. Successful, sustaining BDA capabilities should look like a well-oiled "data supply chain" - one that is living and dynamic with the lifecycle of data driving product and business value.  

Pitfalls to watch out for 

• BDA effort should be "requirements driven" with clear business value. It should not be a pure IT initiative on infrastructure or storage just to bring Big Data capabilities into the enterprise without a valid business case.

• Doing something different but not better.

• Go against Big "smart" (but not bad) Data to make sure analysis is done on data relevant to question at hand.


Big Data and BDA are making big waves in enabling organizations to step into a new era of advanced analytics. As the volume, velocity, variety of unstructured data from social media, web search, blogs, media archives, sensors and monitors continue to expand, Hadoop platform and its related technologies is helping analyze and mine customer needs and desires that is translating into building products that bring true value to the customer. Marketing is leveraging BDA capabilities to augment data from traditional market research channels with rich data obtained from sources closest to the customer. Finally, a concerted effort by business, IT, management and human resources is needed to deploy a successful BDA solution; one that can continuously analyze moving data harnessing deep insights in the process to provide companies the competitive edge to increase profitability, revenues and market share.


[i] Walsh, Richard and O'Callaghan, Richard and Yoffou, Sabine, Big Data is a Solution - So Where's the Problem?

[ii] Columbus, Louis. "Hype Cycle for Cloud Computing Shows Enterprises Finding Value in Big Data, Virtualization." Forbes, 4 August 2012.

[iii] Laney, Doug. "3D Data Management: Controlling Data Volume, Velocity, and Variety." Meta Group, 2001.

We Would Like to Thank Our Sponsors