I will be providing the keynote address on a half-day seminar hosted by Sita Corp, SAP, and HP at New York Athletic Club, on October 15, 2013 from 8:30-10:30 am.
I am going to be talking about the challenges of Information Governance in a Big Data world.
Register now at: http://ow.ly/po2mm
I recently provide a briefing to a group of e-discovery professionals about Big Data and why it matters to them, and I thought there might be some value in sharing my notes.
1. What is Big Data?
Gartner: Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision-making.
McKinsey: ‘Big data’ refers to datasets whose size is beyond the ability of typical database software tools to capture, store, manage, and analyze. This definition is intentionally subjective . . .
It is subjective, but has definable elements
The data itself: large, unstructured information
The infrastructure: “Internet scale” in the enterprise
The analysis: Asking questions using very large data sets
2. Why Does Big Data Matter to E-Discovery Professionals?
Data scientists and technologists do not understand the risk side of information
You need to be at the table to educate them on:
The legal and business value of deleting information
The privacy requirements and implications
E-Discovery implications of too much data
The technologies of Big Data may process and manipulate information in a way that affects their accessibility and evidentiary value – you need to be aware of this and guide your clients appropriately
3. Does Big Data offer value to the legal community?
Performing sophisticated analysis on large pools of data is not exclusive to any particular industry – there is no reason it could not be applied to the legal community (and already is being used in some limited ways)
Relatively speaking, most law firms do not generate massive amounts of data in their day-to-day operations
In e-discovery, the technology innovations of Big Data could be helpful in very large cases to help with storage and processing tasks
4. What are some examples of Big Data in action?
President Obama’s data-driven election campaign.
An online travel company showing more expensive travel options to those who used higher-prices Macintosh computers to access their website.
Fraud Detection: Targeting $3.5 trillion in fraud from banking, healthcare utilities, and government.
The City of New York finding those responsible for dumping cooking oil and grease into the sewers by analysing data from the Business Integrity Commission, a city agency that certifies that all local restaurants have a carting service to haul away their grease. With a few quick calculations, comparing restaurants that did not have a carter with geo-spatial data on the sewers, they generate a list of statistically likely suspects to track down dumpers with a 95% success rate.
5. What professional and career opportunities does Big Data represent for e-discovery professionals?
Organizations need people who understand the risk side of the equation and who can provide practical guidance
Your clients may have Big Data projects that right now, today, are creating unmonitored, unmitigated risk; you need to be able to help them identify and manage that risk
Big Data focuses on unstructured information, i.e., the documents, email messages and other information that the e-discovery community knows well. These same skills and techniques can be very useful to business-led Big Data projects.
Here is the fifth and final (except for a bonus video coming soon) in our five-part video series where I asked 30 Information Governance the same 5 questions. This video is the longest of the five, as I ask our interviewees to tell us their favorite story about IG – something that illustrates what it is, why it is hard, challenges they have faced and so on. There are some great stories, so get yourself a fresh cup of coffee and a snack and enjoy.
Here is the fourth video in our five-part series where I asked 30 Information Governance experts the same question, then produced a 5 minute video of their responses. As you watch the series, it is very interesting to see the common threads that weave through the answers, depending on the role and the type of organization the interviewee comes from.
This is the second video in our series, “5 Questions about Information Governance in 5 Minutes.” In this video IG experts answer the tricky question, “Who Should Own Information Governance?”
A few weeks ago, I mentioned that I was working on new feature article for Law Technology News about about how making more and more data “easily accessible” is both essential for Big Data to fulfill its promise and also a huge risk to privacy, intellectual property, and so on.
The promise of Big Data is based on a central assumption: that information will be easily, quickly, and cheaply available, on a grand scale. The plumbing of Big Data — the technology infrastructure — is designed to bring internet scale to enterprise data. Some of the surprising insights that data scientists hope to gain from Big Data analytics come from correlating information from disparate sources, in a context that was never imagined when the information was first created — such as correlating the type of computer used to book a trip with how much a traveler is willing to pay for a hotel room. Or using prescription drug history to screen health insurance applicants.
The problem of protecting privacy, intellectual property, and other rights will only grow more complex as our ability to access and process information becomes more sophisticated.
I also write about how these issues came to the forefront in the wake of the shooting tragedy at Sandy Hook Elementary school in Newton, CT. I also explore emerging technology that allows electronic content to “self-destruct.”
The article has now been published, and you can read it here (free registration required).
I was also interviewed about the article by Monica Bay, Editor-In-Chief of LTN, on Law Technology Now. You can listen to our discussion on the embedded podcast below.
Author: Barclay T. Blair
Early this year I was lucky enough (thanks to a great sponsor) to carve out some significant research and writing time to answer a complicated (and maybe even complex) set of questions: what does unstructured information really cost? How do we answer this question? Which kinds of costs should be included in the answer? Can we use this answer to drive desirable Information Governance behaviors?
I looked at existing models for structured data, studied the emerging Big Data market, talked to clients and experts, and developed some answers to these questions that I think are actually pretty novel. You can download the entire paper here now (at the website of Nuix, the sponsor), and you can also follow along here as I discuss out some of the key ideas and findings over the next few weeks.
A PowerPoint slide (with notes) is available for download here: IG PowerPoint Slide of the Day from Barclay T Blair-10 Factors Driving Unstructured Information Cost. If you do use it, I would appreciate you letting me know how and where.
Unstructured information is ubiquitous. It is typically not the product of a single-purpose business application. It often has no clearly defined owner. It is endlessly duplicated and transmitted across the organization. Determining where and how unstructured information generates cost is difficult.
However, it is possible. Our research shows that there are at least ten key factors that drive the total cost of owning unstructured information. These ten factors identify where organizations typically spend money throughout the lifecycle of managing unstructured information. These factors are listed in Figure 1, along with examples of elements that typically increase cost (“Cost Drivers,” on the left side) and elements that typically reduce costs (“Cost Reducers,” on the right hand side).
- E-Discovery. Finding, processing, and producing information to support lawsuits, investigations and audits. Unstructured information is typically the most common target in e-discovery, and a poorly managed information environment can add millions of dollars in cost to large lawsuits. Simply reviewing a gigabyte of information for litigation can cost $14,000.[i]
- Disposition. Getting rid of information that no longer has value because it is duplicate, out of date, or has no value to the business. In poorly managed information environments, just “separating the wheat from the chaff” can cost large organizations millions of dollars. For enterprises with frequent litigation, the risk of throwing away the wrong piece of information only increases risk and cost. Better management and smart information governance tools drive costs down.
- Classification and Organization. Keeping unstructured information organized so that employees can us it. Also necessary so management rules supporting privacy, privilege, confidentiality, retention, and other requirements can be applied.
- Digitization and Automation. Many business processes continue to be a combination of digital, automated steps and paper-based, manual steps. Automating and digitizing these processes requires investment, but also can drive significant returns. For example, studies have shown that automating Accounts Payable “can reduce invoice processing costs by 90 percent.”[ii]
- Storage and Network Infrastructure. The cost of the devices, networks, software, and labor required to store unstructured information. Although the cost of the baseline commodity (i.e., a gigabyte of storage space) continues to fall, for most organizations overall volume growth and complexity means that storage budgets go up each year. For example, between 2000 and 2010, organization more than doubled the amount they spent on storage-related software even though the cost of raw hard drive space dropped by almost 100 times.[iii]
- Information Search, Access, and Collaboration. The cost of hardware, software, and services designed to ensure that information is available to those who need it, when they need it. This typically includes enterprise content management systems, enterprise search, case management, and the infrastructure necessary to support employee access and use of these systems.
- Migration. The cost of moving unstructured information from outdated systems to current systems. In poorly-managed information environments, the cost of migration can be very high – so high that some organizations maintain legacy systems long after they are no longer supported by the vendor just to avoid (more likely, to simply defer) the migration cost and complexity.
- Policy Management and Compliance. The cost of developing, implementing, enforcing, and maintaining information governance policies on unstructured information. Good policies, consistently enforced will drive down the total cost of owning unstructured information.
- Discovering and Structuring Business Processes. The cost of identifying, improving, and routinizing business processes that are currently ad hoc and disorganized. Typical examples include contract management and accounts receivable as well as revenue-related activities such as sales and customer support. Moving from informal, email and document-based processes to fixed workflows drives down cost.
- Knowledge Capture and Transfer. The cost of capturing critical business knowledge held at the department and employee level and putting that information in a form that enables other employees and part of the organization to benefit from it. Examples include intranets and their more contemporary cousins such as wikis, blogs, and enterprise social media platforms.
[i] Nicholas M. Pace, Laura Zakaras, “Where the Money Goes: Understanding Litigant Expenditures for Producing Electronic Discovery,” RAND Institute for Civil Justice, 2012. Online at, http://www.rand.org/content/dam/rand/pubs/monographs/2012/RAND_MG1208.pdf
[ii] “A Detailed Guide to Imaging and Workflow ROI,” The Accounts Payable Network, 2010