Trust the Evidence, Not Our Instincts
On September 4, 2011, an article with nearly the same title as this blog was published in the New York Times. Jeffrey Pfeffer and Robert Sutton, both leaders in evidence-based management (or EBM), present a lucid and efficient case for how EBM can improve medical care and business management. But the title intrigued me: What can EBM theory and experience teach us, if anything, regarding how we present, and rely upon, digital information as evidence of the truth? How do we trust digital evidence, and not our instincts, in the courtroom? How do we enable judges, juries, regulators, investigators (and even trading partners in business) to do so?
To those trained in law, EBM is not too hard to grasp. Lawyers learn to evaluate the facts of a case and, in giving legal guidance on how to structure a commercial transaction, making a case for civil liability, or advocating the legal innocence of a criminal defendant, research published case decisions that are comparable on their facts. They seek cases that align best to their facts, try to find as many points of alignment as possible (and minimize where they do not align) and then offer a compelling analysis to support the outcome desired by the lawyer (and their client!).
In a given clinical situation, a doctor is encouraged to not merely rely on their clinical instincts but to define the diagnostic challenge in terms that allow the doctor to conduct research and identify scientific research, studies, and analysis that can be evidence of the most likely treatment plan. The doctor evaluates the published research in terms of the congruity to a patient’s condition, research design and execution, etc. and integrates that analysis into treatment decisions.
In social and behavioral sciences, evidence-based programs (EBP) are those treatment plans or programs that have been objectively measured in objective studies to align outcomes with the design. The studies themselves should be peer-reviewed to assure their integrity as measures of effectiveness and reliability. In effect, the evidence-based programs are supported by scientific evidence that the programs work at achieving the intended result. Funding decisions, program resources, and treatment plans are weighted to favor EBP. Increasingly, treatment programs that lack evidence-based support struggle to recruit advocates and sponsorship, even if experienced clinicians and counselors instinctively believe those programs will make a difference.
So, in the first instance, EBM and EBP seem almost quaint in suggesting that precedents, expressed in the published research (for EBM) or validation studies (for EBP), help reinforce the decision process. That concept is one of the first lessons in law school and a foundational concept in legal reasoning. It is almost unsettling to consider that medicine, social services and behavioral sciences are only now getting to the point that diagnosis and treatment strategies might rely on more than the instincts and good sense of the person asked to make a decision on the right course. Law is so much further along . . . or is it?
In our digital world today, over 97% of all business information is created by and with computers. The machines function with astounding efficiency, applying complex algorithms faster than eyes blink, using astounding computational functionality to assure the accuracy, integrity, consistency, security, and reliability of the information inputted, as well as the processed products—nearly 295 exabytes of information globally stored by 2007, as reported by the BBC (enough to fill 1.2 billion average computer hard drives!). To even exist, electronically stored information (ESI) is the product of systemic efficiency, multiple automated controls, and structured tracking, reporting, and management. Metadata, the data about data that computers and applications automatically create to manage the files, records, and histories of their own operation, has become the documentation of the computer as witness—objective, intensive, accurate, and usually immune to human intervention or alteration without special tools, access, and skill.
But, and this is a critical “but”, when we turn to the rules of evidence which govern American courts, as well as those in most other nation states with judicial systems, the admissibility of digital information as evidence is left entirely to the unrestrained, and immeasurable, discretion of the human judge who presides. Evidence in any form—oral testimony, paper documents, tangible objects, or digital records—is considered as evidence only if the judge determines, subjectively, that the evidence is what it purports to be (authenticity). Once admitted, however, whether the evidence is to be believed, credible, or persuasive (collectively the probative value of the evidence) is left entirely to the subjective instincts of the judge or the jury. Indeed, many jurisdictions expressly prohibit a judge from instructing the jury on the weight to be given to evidence or particular testimony. In a case tried without a jury, the judge may make his or her own determinations of weight.
So, in a digital world, with astounding documentation available on the origin, history, and calculation of virtually any character or pixel in any ESI, a world in which other professions are relying on evidence-based analytical methods to improve their diagnoses, treatment strategies, and programs, we have the extreme irony that the most evidence-based process in our social infrastructure—the finding of truth in the courtroom—relies, in fact, on the unmeasured, human instincts of the judge or jury!
When evidence is presented to the judge for a determination whether the evidence is to be admitted, the party offering the evidence must lay the foundation. Oral testimony (or, in rare cases, sworn written affidavits) must be offered from suitable witnesses to establish the authenticity of the evidence. For ESI, the oral testimony may itself be supported by computer records (such as system operating logs, access activity logs, file revision histories, etc.) to substantiate the authenticity of the primary evidence.
Yet, to this day, no one has published an objective, authoritative set of criteria—perhaps based on best practices in information management, information security, and digital storage and recovery services—with which to enable the judges to make evidence-based decisions as to authenticity. Even further, no such resource exists to enable the juries to evaluate the probative value of digital information as evidence. Instead, if the weight of ESI is vulnerable to challenge, the more likely scenario is for both sides to assemble, rehearse, and present multiple computer geeks to provide—you guessed it—oral testimony to undermine or reinforce the weight to be given to the digital material. The message to the jury is “Sure, the judge let you see this drivel, but we don’t know what she was thinking. This stuff is worth less as evidence than the paper on which it is printed.” It is no wonder juries fall asleep during these dances on the pin by the computer angels.
Our legal system of justice has an institutional bias toward the role of oral, human testimony. Indeed, at the starting point of determining whether to admit digital records to prove the truth, the records are considered as “hearsay” and not admissible unless exceptions specified in the Federal Rules of Evidence are satisfied. To this day, one of the most challenging chapters in law school is learning how to navigate the 23 express exceptions under which written or computer records can be admitted as hearsay, even if a human witness is available to testify.
It is somewhat remarkable that the Rules of Evidence have not been amended to expressly carve a more suitable basis on which digital business records can be admitted into evidence. Even under the existing rules, counsel will fail to understand how to demonstrate the authenticity of digital records. In one landmark case, in which both parties submitted printed copies of electronic mail to support their motions, the court rejected both sides on the basis they never established the authenticity of the electronic mail. In another, American Express failed to obtain a judgment against a non-appearing bankrupt consumer because the court ruled American Express had not shown the reliability of their systems from which the consumer’s statements had been printed.
The Federal Rules express one further exception allowing the admission of hearsay that is worth mentioning. Formally titled the “Residual Exception” of Rule 807, this exception enables the judge to admit specific records despite their hearsay character and despite the fact 23 other exceptions do not apply. Under this rule, even if there is no other basis for admitting “hearsay”, a court may determine to admit a record if four conditions are met:
(a) the statement (i.e, the record) has “equivalent circumstantial guarantees of trustworthiness”;
(b) the statement is offered as evidence of a “material fact”;
(c) the statement is “more probative on the point for which it is offered than any other evidence which the proponent can procure through reasonable efforts”; and
(d) the general purposes of the rules of evidence and the interests of justice will best be served by admitting the statement into evidence.
Rule 807 illustrates how, when it comes to evidence and the evaluation of digital records as evidence, our system perpetuates a decision process that is the antithesis of evidence-based anything. Under five different criteria (“equivalent guarantees”, “material”, “more probative . . . than any other evidence”, “reasonable efforts”, “best be served”), the application of which is entirely incapable of objective review (such as an appellate court may provide on other evidential matters), a court can bar digital information from its chambers.
So, is there a means for us to institute into our legal system a means to “trust the evidence, not our instincts”? Can we develop a methodology by which judges asked to serve the interests of justice can rely upon objective, researched, documented criteria by which to measure the authenticity of digital information? Can judges and juries be provided a template—a scorecard perhaps—by which to objectively measure the weight of digital information? Can counsel, judges, and juries be equipped with the evidence-based means of determining when a computer record is more probative (even conclusive) than oral testimony (e.g., a computerized traffic camera vs. a driver’s denial he was “on that road at 2:00 am”) or when a computer record is to be entirely viewed as fraudulent (such as the false trading reports generated by Bernie Madoff’s software programs)?
Imagine, as an alternative, that before counsel even considered offering digital information as evidence, the lawyer first completed an inquiry into the provenance and pedigree of a digital record by evaluating:
In each case, detailed checklists would exist, to be completed by the systems themselves, and capable of being audited by independent experts if required. The elements on each checklist are evidence-based—that is, each functional feature or characteristic is included because it is scientifically supported by objective evidence to be a valued characteristic for digital information to be trustworthy as evidence of the truth. The performance reported for each element would be objective—measured against known metrics and computationally scored, independent of human subjectivity or discretion.
On these checklists, features can be weighted or optional, based on the quality and nature of the data or report. But the presence or absence of any of the listed features is always disclosed. These checklists report compliance, as well as deviations, from the criteria. But the checklists enable the lawyer, even before appearing in the court, to know whether the digital ESI, as evidence, is worth the paper on which it is printed.
Together with the ESI, such reports are submitted to the court. Now, at this point, the courts can eliminate reliance on their “instincts” and can make far more objective determinations of authenticity. And, in the event competing versions of the same evidence exist (such as two versions of the same electronic mail, with one conspicuously missing the metaphors comparing an employee’s size to a farm animal), these reports also enable the weight of the competing versions to be objectively evaluated by judge, and jury.
Unrealistic and fanciful? Not at all. What is unrealistic is that our justice system perpetuates making decisions on the authenticity and integrity of electronic information on “instinct” rather than the available objective evidence. What is fanciful is that the electronic discovery collar that currently enslaves judicial efficiency has absolutely no criteria whatsoever with which to disqualify ESI from serious consideration because the systems, devices, and records associated cannot demonstrate in the first instance they meet an evidence-based threshold of reliability as potential evidence.
Every day, as data moves with more and more volume and greater computational efficiency, millions of transactions, communications, wealth transfers, knowledge transfers, reports, and records are objectively measured against published, visible, electronically expressed criteria to determine their continued transit through additional processes, applications, devices, systems, and networks. A single deviating byte and an entire record may be suspended, re-processed, or rejected. Indeed, the entire function of the Internet demands protocol compliance as a condition to transfer.
That type of brutally efficient, objective, and ruthless calculation is what computers do to protect themselves from false identities, fictitious data, improperly modified reports, and other products of malicious intent or negligent competency. Yet time and again we continue to invest billions of dollars annually (software, hardware, professional services, lost production time, etc.) in preserving, processing, filtering, and reviewing digital information as evidence without ever asking the basic questions: does this ESI meet any objective, measurable standard of integrity?
If we fail to come to the cause of justice by developing evidence-based means to trust digital information as evidence, and relieve judges and juries from the burdens of subjective decisions based on “instinct”, I am concerned for the future of justice in a digital age.
Companies that do not invest in executing for their ESI, as evidence, the same quality controls for integrity and reliability that their machines demand in daily processing should be handicapped in being able to rely on such records, or to have their deficiencies exposed more efficiently than through conflicting oral testimonies of dancing geeks on the witness stand.
Counsel that do not develop the skills to be able to ask whether digital evidence measures up are not serving their client’s interests, whether evaluating the evidence tendered by an adversary or the evidence available from their own client.
Insurance companies, as underwriters, should not fund the defense of companies that fail to install and operate systems for assuring the integrity and availability of their ESI. (Even today, carriers are imposing information security standards, demanding performance reporting, and conducting audits in issuing business interruption and other coverages. The same principle as a good driver discount should apply as an incentive for improved records management.)
Judges and juries should not be imposed upon to judge the merits of the case without assurances that any digital records offered as evidence meet certain objective “evidence-based” criteria for trustworthiness. The weight to be given to those records can be objectively expressed, and not left to the judge or jury to establish by human emotion the “equivalent circumstantial guarantees of trustworthiness”.
It is time to move forward and install evidence-based management in the administration of legal justice. We are soon approaching a day when the computer record will transcend the reliability and integrity of oral testimony of any person on nearly any matter of legal consequence except, perhaps, acts of physical violence that occur outside of the recording capability of any cameras, video devices or microphones. But only certain computer records will deserve to have that credibility and weight—now is the time to begin to figure out the differences among them.
* * * *
The role of automated evidence-based means to evaluate evidence is just one of 10 trends I examine in my keynote presentation on “Digital Justice in the Year 2058”. If you would like to hear the other trends, and explore how technology is transforming who will win—and lose—in how law and justice will be administered, pick up the phone and give me a call to discuss scheduling a presentation to your conference, your management team, your partners, or just a coaching session with yourself.
