Data Access in a Big Data World

Organizations are collecting tons of information. And the volume is getting larger every year.

As data volumes expand, organizations have started to explore what they can do with all of that information. Thus, there's an intense interest in big data technology.

But, before investing in new solutions, you need to understand:

  • The scope of information growth
  • The challenges associated with big data access and analysis
  • How to overcome those issues

In 2005, the International Data Corporation estimated the total volume of enterprise data at 130 exabytes. That number is on pace to double every two years, reaching 40,000 exabytes by 2020. Because traditional databases were not equipped to handle that volume, big data emerged as both a problem and a solution.

On one hand, pulling together large volumes of information from numerous platforms and systems can offer more comprehensive insight not afforded by traditional analytics. On the other, big data, by definition, has such a large volume that legacy databases and software don’t have the necessary functionality to deliver data quickly and effectively.

‘Data at scale’ is a popular data management practice for growing volumes on IBM i. Unfortunately, it creates a partitioning paradox. IBM i operators need to use table (or range) partitioning to deal with increasing data. This allows IT to break files into multiple components.

For example, a partition could be created to organize employee data by identification number or name. Up to 32,767 data partitions can be created on IBM i, so this is a viable option for breaking through capacity constraints.

However, legacy software such as Query/400 only allows access to one partition member at a time. That simply isn't enough for most organizations. 

In a big data environment, IT departments likely need to draw on information from several members, which would require several queries. The issue of volume is not isolated to IBM i. It can also create demand for new storage hardware and make processing tasks more resource intensive.

Today, solutions are available for the sole purpose of managing big data. From marketing departments to IT administrators, employees throughout many organizations are delving into and making conclusions from the vast quantities of data now available to them.

However, all big data initiatives need to address the core issues associated with the technology:

  • Volume of information
  • Wide range of data types and sources
  • Speed at which it must be accessed

Big Interest in Big Data Analytics

While big data creates challenges for IT professionals and business users alike, many decision makers believe it’s worth it.

IDC’s “Big Data, Analytics, and Cloud Drive Enterprise Software Growth in 2012” report attributed much of the revenue growth of the application development and deployment (AD&D) market to enterprise interest in analytics. The big data sector has definitely benefited from healthy investment. AD&D accounted for 24 percent of all software revenue in 2012.Yo

u rorganization can gain significant value from overcoming the obstacles of using big data technology effectively. That's why it’s important to adopt business processes and technology solutions that streamline collection and data management.

Both IT users and business users alike are pushing for these investments. But Gartner revealed that much of the pressure to adopt analytics solutions comes from the data center. 42 percent of IT leaders already benefiting from big data deployments or planning to invest in the technology within a year’s time. According to Gartner researcher Frank Buytendijk, one of the most prominent concerns among adopters is that their deployments have come too late, putting them behind the competition.

Big Data Challenges: The Three Vs

The defining attributes of big data—volume, variety, and velocity—make it challenging to deal with. The amount of data entering business technology ecosystems is more than what traditional technology solutions were meant to deal with, and it is only getting larger.

1. Volume

In InformationWeek’s “2013 State of Storage Survey,” the percentage of respondents managing 100 terabytes of data or more nearly doubled to 42 percent between 2009 and 2012.

2. Variety

The exploding data volume issue is further complicated by the fact that information is coming in from a greater variety of sources. For IT managers, this means data must be collected from numerous systems that may be running different platforms.

3. Velocity

There's another layer of complexity to the big data problem. When it comes to data variety, a large portion of data in enterprise IT systems today is considered unstructured. This type of information does not have a defined data model and does not fit well into traditional relational databases. Considering that researchers from Dataversity estimate between 40 and 80 percent of enterprise information is unstructured, it is essential to ensure that this information can be effectively categorized and searched.

The final ‘V” of big data—velocity—means that companies must be able to find and use critical information quickly.

The value of any given data diminishes over time. An IT employee looking at the number of scheduled jobs that were completed on time would benefit from knowing which ones failed sooner rather than later—the same concept applies to data analytics initiatives whether they deal with large or small amounts of information.

The problem in dealing with the three Vs is that many businesses lack the proper tools and expertise to leverage big data environments.

GigaOM’s “Deploying Big Data 2012” survey revealed a lack of in-house expertise as the most prominent challenge for an estimated 50 percent of organizations. Analysts also noted that IT departments typically spend more time managing their infrastructure and architecture resources that store their information than what they spend actually gleaning insight from it.

While some organizations have migrated their big data initiatives to the cloud, GigaOM’s researchers said this is not an acceptable option for many businesses due to security and compliance concerns, particularly when considering the type of information that big data applications tend to touch:

  • Financial and customer records
  • Intellectual property
  • Sensitive employee information

Your organization may benefit more from purchasing tools to improve big data access and analysis practices—rather than put your enterprise information at risk.

Is Your IBM i Data Access Tool Up to the Challenge? Find Out >

Overcome Big Data Challenges with a Successful Strategy

Big data should be tackled like any other business operation. In other words, stakeholders must think of data as an investment—shifting attitudes and making strategic technological purchases to facilitate the transformation to a data-driven culture. It may be helpful to consider big data from two primary angles:

  1. Operational changes needed to most effectively leverage analytics
  2. Technology required to access and process large volumes of information

Forrester analyst Michele Goetz explored how strategies can impact the overall success of big data initiatives in March 2013. She highlighted three qualities in particular that can determine a successful business outcome:

  • Attitude, courage, and discipline
  • Collect, invest, and innovate
  • Hub-and-spoke architecture

A successful big data strategy builds a business culture around analytics, tempers idealism with valuable use cases, and uses data to innovate business processes. This is easier said than done. Goetz noted that most big data projects never make it past the pilot stage. However, if you focus first on improving how your organization manages data and gives end users access to critical information, you'll stand to gain a higher return on investment.

IT departments tasked with launching big data initiatives can benefit from first taking stock of existing data management practices. If you're still relying on manual tasks to ensure data integrity or lack data cleansing tools entirely, your business will need to adjust its practices.

A study from Tata Consulting revealed that leaders in analytics often formed separate big data departments. This may not be financially feasible for every organization, but your IT team can use many of the same strategies.

One of the essential components to success is collaboration between technical and business employees. Technical employees understand the infrastructure and architecture that forms the backbone of these initiatives. So, they are best positioned to determine what software is needed to achieve business goals. Business employees will be the ones using the majority of the data, so their input is also essential for identifying objectives.

Big Data Solutions: Features to Look For

Once big data initiatives have been defined, it is important to address the technical issues that often emerge in big data deployments. Although current trends in analytics focus on making things easy for business users, IT departments still play a significant role in delivering this insight.


Many organizations launching IBM i analytics initiatives rely on manual processes to bring together information from different partitions. This is time consuming for IT professionals and frustrating for business units that do not get their data quickly enough. A March 2013 report from Oracle focused on the velocity component of big data, noting that speed is particularly important for realizing several key advantages:

  • Building new services and improvising existing ones
  • Creating better customer experiences
  • Increasing operational efficiency
  • Improving operations quality

Success depends on the ability to bring together and analyze data in real time. Manual collection and analysis processes could extend the delivery time to days, making the information less valuable. Consider investing in solutions that automatically migrate data between different partitions or even different platforms.


The value of any big data investment depends heavily on how usable the data becomes. This means it’s important to consider visualization tools and how information is displayed. A CFO delving into a spreadsheet of financial records would likely find it difficult to see areas of growth at a glance. However, the ability to create charts and other visual features is beneficial for both IT and non-technical users. Analytics is about seeing relationships between different data sets or finding trends within large volumes of information.

Looking at a chart instead of a spreadsheet filled with numerical values is much better for processing information. Visualizations also enhance the quality of the analysis by providing more information to draw from. As a May 2013 report from Aberdeen Group revealed, organizations employing visualization tools in their big data initiatives were able to leverage an average of 42 unique data sources, compared with 17 for analytics initiatives that lacked visualizations.

The report identified several other advantages:

  • 21% shorter time-to-information—1.9 times greater than analytics initiatives without visualizations 
  • 22% improved accuracy of business decisions—1.8 times greater
  • 22% higher quality analysis—11 times greater
  • 27% improvement in the visibility over business data—4.5 times greater

It is also important to keep in mind that a one-size-fits-all solution will not likely accommodate users’ diverse needs.

To account for different needs and levels of expertise, accessibility features should be customized. For example, one user may require an intuitive graphical interface; while an employee with technical expertise would be able to generate his or her own queries and reports.


In the rush to ensure usability and solve security challenges it is important to remember the basics, such as how users can access reports and insight.

Today’s employees are increasingly mobile, as reflected by enterprise hardware purchasing trends. Gartner estimated that worldwide tablet purchases by businesses will more than triple by 2016.

Enterprises interested in employees bringing in their own devices further complicate the issue of access. This means it is important to account for different screen sizes, as well as how end users will interact with their reports. For example, while drag-and-drop functionality is convenient on a desktop, it's a must for mobile users who rely on a touchscreen.

As platform diversity and enterprise mobility continue to grow, it becomes critical to facilitate data access for all key employees. Solutions that offer secure, platform-agnostic remote access are best equipped to handle the ever-evolving hardware ecosystem now present in many modern companies.


Just as employees require access to data, they must also have tools that enable them to use it quickly and effectively. Technology with a high learning curve for non-technical users can raise the related costs of adoption by making IT spend more time offering user training sessions.

The issue is further complicated by the fact that accessibility can mean different things to different users. For example, employees may simply want the ability to focus on the data that is most relevant to them. This necessitates some customization in reporting tools so that analysis can be adjusted to specific needs and preferences.

Deloitte Analytics senior advisor Tom Davenport recently explored the issue of accessibility in an interview with TechTarget. Speaking to the role of visualizations, he emphasized the importance of displaying information in multiple ways.

“I think visual presentation is powerful for a lot of people,” Davenport explained. “But some people interact better with text and even numbers… I’d rather see a narrative-based description. In many cases we benefit from multiple ways of telling a story visually and in terms of narrative, so you want to have as many going on as possible in order to find one that engages the consumer.”

Selecting a Big Data Solution for IBM i

While many of the challenges associated with big data are related to business strategy, they also require technology to solve. Selecting the right big data solution will have a dramatic impact on the success of any big data initiative. An effective solution should facilitate access as well as make it easier to interpret results, with features such as:

  • Access and accessibility for all levels of users
  • Web interface for anywhere access
  • Custom visualization tools to display objects in charts and graphs
  • Support for multi-platform databases
  • Unstructured and structured data management

Most business decision makers recognize the potential value of big data as numerous success stories emerge, but are hesitant to adopt solutions due to fears over data privacy, security, and the time-consuming tasks traditionally involved in managing large volumes of information. However, selecting the right big data tool can satisfy the core technical challenges, as well as empower users with secure, reliable access.

Get Help with Your Big Data
When it comes to managing big data on IBM i, you need all the help you can get. Is your IBM i data access tool cutting it? Find out. Use our comparison checklist to define your must-haves and figure out if it's time to upgrade.

Related Products

Related Solutions