Society is creating digital data at a rapid pace, exceeding previous estimates by at least 10 percent, a new study by industry analyst IDC says. The report, called The Diverse and Exploding Digital Universe, estimates the size of the digital universe at 281 exabytes, or 281 billion gigabytes. The increased estimate is due to faster growth in digital cameras, digital-TV shipments and a better understanding of information replication.
The study also estimates that by 2011, the digital universe will grow an order of magnitude -- 10 times -- from its size in 2006.
Imperatives for IT
IDC said IT organizations will face three basic imperatives to deal with this explosive growth of data.
First, they will need to "transform their existing relationships with the business units," IDC said. "It will take all competent hands in an organization to deal with information creation, storage, management, security, retention and disposal." The digital universe is a business problem, "not a technical problem," IDC said.
Next, organizations must develop "organization-wide policies for information governance: information security, information retention, data access, and compliance," the analysts said.
Finally, IT must "rush new tools and standards into the organization," including storage optimization, unstructured data searches, database analytics, virtualization, and management and security tools. "All will be required to make the information infrastructure as flexible, adaptable, and scalable as possible. We have many of the tools in place -- from Web 2.0 technologies and terabyte drives to unstructured data-search software and the Semantic Web -- to tame the digital universe. Done right, we can turn information growth into economic growth."
Fast Growth in Images and the Cloud
The fastest-growing areas are related to digital television, surveillance cameras, Internet access in emerging countries, sensor-based applications, cloud computing, and social networks, the study found.
One intriguing development is the growth of packets over stored data. IDC researchers said increased use of RFID tags, Voice over IP and sensors means that "information containers" such as files, images and packets are growing 50 percent faster than the number of gigabytes. By 2011, we will generate 20 quadrillion containers, the report said.
"What's interesting about the unfamiliar and emerging data sources is how difficult they will be to manage and leverage successfully," noted Charles King, principal analyst for Pund-IT, in an e-mail. "The sheer volume of digital surveillance video makes it impractical or impossible for humans to effectively search. Similarly, massive numbers of sensors and related data make some kind of automated management imperative."
For individuals, most of the data being created is from user activities like taking pictures or using cell phones. From "shadow" activities, it's surveillance photos, mailing lists, records of web searches and financial transactions. The shadow encompasses everything from what Amazon knows about book-buying habits, to what Google knows about Web and ad clicks, to what public video cameras record about citizens' activities.
"The idea of a digital shadow goes from curious or irritating to scary when you factor in the risk of identity theft," the report said.
A Brave New Data World
On the enterprise side, some industries have digital footprints far exceeding their IT spending or contribution to gross domestic product. The media, entertainment and communications industries represent 10 times as much digital data as economic output, the study said, while the financial sector, which accounts for 20 percent of IT spending, generates only six percent of the digital universe.
Some 70 percent of the digital universe is created by individuals, but enterprises are
responsible for the security, privacy, reliability and compliance of 85 percent.
"Few IT professionals are ready to embrace the new data types into their information-management domain; few understand the potential impact on computing and information architecture," the study said.