2015 Big Data Survey: Current Implementation Challenges


The amount of data in both the private and public domain is experiencing exponential growth. Mobile devices, sensors, audio and video feeds, social media, and what has become known as “The Internet of Things” are all contributing to this increase in information variety, volume and velocity. The significant increase in data in recent years, coupled with the development of new techniques and technologies to analyze it (“Big Data”), enables disruptive business models to flourish and is now spreading into the more traditional corporate models and activities.

To better understand the challenges faced by organizations trying to leverage Big Data, Knowledgent recently conducted a survey designed to gauge the levels of difficultly experienced in key areas that, in Knowledgent’s perspective, are potential pain points. In this survey, we asked questions relative to the status of Big Data initiatives and projects and the value being received by these efforts.

The survey found that:

  • Big Data continues to grow in importance despite significant obstacles.
  • The combination of traditional and more unstructured data sources, combined with advanced analytics, are contributing to the development of new business insights.
  • Big Data initiatives are transitioning from Proofs-of-Concept to production.

Over 60% of respondents indicated that Big Data initiatives were either very or extremely important to their organizations. However, even with Big Data’s growth and benefits, there are still significant challenges to organizational adoption:

  • Resources, both human and other, continue to be a major constraint.
  • Putting together an overall “production grade” program, particularly those aspects related to standardizing process, is a notable challenge.
  • The “Data Lake” architecture needs to evolve and mature to better support end users.

Implementation Status

Based on the survey results, it is clear that Big Data is moving out of the experimental stage. The results indicate that by the end of 2015, the majority of respondents expect to be utilizing Big Data in a production environment.

  • Over 60% of respondents indicated that Big Data initiatives were either very or extremely important to their organizations.
  • On an industry basis, the Financial Services and Healthcare sectors attributed more importance to Big Data than other sectors, while Insurance trailed.
  • 25% of respondents reported having already implemented a Big Data solution, while most other respondents indicated they were within six months of doing so.
  • Over 75% of respondents view Big Data as having gone from proof-of-concept to production.

Priority of Big Data Initiatives

Figure 1: Priority of Big Data initiatives

big data time frame

Figure 2: Big Data Implementation Timeline

big data implementation status

Figure 3: Implementation Status of Big Data Technology

What is the Value Being Realized?

In the early stages of Big Data evolution, there was an emphasis on the cost savings to be realized by open-source software and commodity hardware. However, it has always been Knowledgent’s observation that the biggest value comes from gaining new analytical insights, particularly those gained by the combination of traditional, structured data and newer, non-tabular data formats.

  • Most respondents agreed that Big Data is effectively enabling the combination of structured and unstructured data.
  • Most respondents agreed that Big Data is driving the use of advanced analytics and leading to new analytical insights.
  • There was a split opinion on the value of Big Data as a data processing hardware/software cost-reduction strategy.

big data value

Figure 4: Realization of Big Data Value

What are the Challenges?

The survey showed that there is very broad agreement that many aspects of implementing a Big Data solution remain at least somewhat challenging. The biggest task continues to be in finding experienced resources.

  • Over 55% of respondents identified finding resources with the required Big Data skills as either very or extremely challenging.
  • Gaining buy-in from business stakeholders scored as the least challenging in this category.

big data challenges

Figure 5: Big Data Challenges

Big Data – IT Management Perspective

Respondents, perhaps unsurprisingly given the maturity of the domain, are finding the greatest challenges in the overall program development and management of Big Data initiatives. Under this umbrella, it seems that standard processes for data ingestion and transformation are still evolving.

  • On average, at least 75% of respondents noted that many aspects of managing and operating a Big Data environment still remain at least somewhat challenging.
  • The most challenging aspect noted was in developing the overall program.
  • The least challenging was in controlling access and privileges.

big data it management

Figure 6: Challenges Faced by IT Managers

Big Data – End-User Support Perspective

Knowledgent crafted the questions in this section based on field observations across multiple projects, particularly those with some flavor of the “Data Lake” architecture. We have noted end-user challenges with locating data, understanding data, and requisitioning data for analytical use.  We wanted to gauge if our observations were being more broadly experienced.

  • In this category, all aspects questioned remain at least somewhat challenging for at least 75% of respondents.
  • This is entirely consistent with Knowledgent’s experience and one of the reasons we developed Kariba, our data and analytics platform.

big data end user support

Figure 7: End User Support Challenges

Survey Method and Demographics

The Big Data survey was a one-time survey conducted from March 12 to April 9, 2015 by Knowledgent. The survey was targeted at IT practitioners with some exposure to Big Data technology. Survey candidates were asked to complete an online questionnaire of 27 questions hosted on Knowledgent’s website. The questions were closed ended with answer options along a Likert scale.

A broad range of respondents took the survey. They represented many industry sectors and sizes of organization.

  • Almost 100 people responded to the survey. Over one-third of the respondents came from the Financial Services sector. The Healthcare, Insurance, and Life Sciences sector made up a further 30%.
  • More than 50% of those taking the survey came from companies with a $1B or more in revenue.

Over 50% of respondents hold a position of manager or above.


Kariba is Knowledgent’s innovative software product that provides a Data and Analytic Self-Service platform. Kariba’s powerful keyword, faceted, and semantic search capabilities revolutionize users’ ability to quickly and accurately find not only structured data like files and tables but also unstructured data like social media posts, chat logs, and web logs. In addition, users learn to trust and understand the data by viewing the quality and lineage of the data and through additional information obtained from the broader user community, such as reviews and common use cases.

From an IT management point-of-view, Kariba provides utilities to rapidly ingest new data into the Data Lake. Typically, the development of a new ingestion process can be a lengthy, customized development task. Kariba’s configuration-driven ingestion mechanism provides a high level of automation and reuse, enabling easy and secure content ingestion from new sources.

For more on Kariba visit: http://knowledgent.com/kariba/

Download White Paper

Never miss an update:

Subscribe to our newsletter!

Newsletter Sign Up Form

  • This field is for validation purposes and should be left unchanged.

New York, NY • Warren, NJ • Boston, MA • Toronto, Canada
©2018 Knowledgent Group Inc. All rights reserved.