May 15, 2014
By Maria Minsker, contributing writer

DBTA Data Summit Day 2: Big Data Strategy Means Finding the Right Tools for the Right Job

NEW YORK—Big data is changing the way people conduct business, but the relationship is not a one-way street; the way people conduct business is having a major impact on big data as well, Anne Buff, thought leader at SAS Institute, said during her keynote on day two of the DBTA Data Summit.

Growing expectations associated with the role of the data scientist is making it more difficult for organizations to find the level of talent needed to fill unoccupied positions. Between 140,000 and 190,000 are currently unfilled, Buff said, highlighting the urgency companies are feeling when it comes to hiring new employees. But besides the physical shortage, there's also a gap in the understanding of big data culture. "There is a major generational difference between those in Gen X and Millennials, and companies need to designate individuals to conduct a generational study and determine how consumers understand data. In some cases, part of the branding has to involve explaining how customer data is being used and how privacy is being protected," Buff said.

Big data processes are dictating the role that that data plays within organizations as well. Those tasked with decision management, for example, have to let go of the notion that data belongs to them alone, and view the data from the perspective of the business that's relying on it. "You can't focus on making data look the way that data professionals in your company want it to look. You have to make it look the way your customer needs it to look to deliver the most valuable insight," she explained.

Buff also noted that the technology that businesses select to meet their big data needs is critically important, and debunked several common big data misconceptions. For instance, while volume and velocity of processing are "great," there's a difference between real time and just in time, Buff explained. "People assume that everything needs to be in real time now, and that's not true, because often you're sacrificing something else for that speed. Sometimes what you should be looking for is 'just-in-time' analytics," she asserted.

Furthermore, Buff cautioned attendees against relying on one technology to solve all their problems. "Please don't walk away from this conference thinking that Hadoop equals big data, because there's much more to it than that," Buff urged. While Hadoop may work for some, company executives should be prudent about assessing which tools will work for their specific businesses, selecting technologies that have the capability to process new types of unstructured data, including video and images.

Dan Burke, vice president of e-business at HP Autonomy, supported Buff's assessment of the various factors affecting big data, adding that the idiosyncrasy of consumer data is also a key element in the shift. By 2015, Burke shared, roughly 90 percent of digital content will be unstructured human data, which is difficult to process for a variety of reasons.

"Ideas don't exactly match like structured data does; they have distance," he said, explaining that training technology to process concepts like sarcasm, irony, and idioms is tough. "Human information is also not static—it's dynamic and lives everywhere, and comes in very diverse forms," he added.