As we begin our discussion of laboratory validation, it is important to define several words that will be frequently used throughout the chapter. These words
D.N.A. Box 16.1 'Houston, we have a problem... ': the importance of quality DNA results
An alarming audit of the Houston Police Department (HPD) Crime Laboratory in December 2002 found that a number of problems abounded in this unaccredited laboratory. In a poorly funded and managed environment, laboratory personnel were not adequately trained, evidence was often consumed, and even the roof in the evidence storage area leaked from rain damage. The city of Houston shut down operations in the HPD laboratory and outsourced hundreds of cases for review to a private Houston-based laboratory named Identigene. While most of the retesting supported the original conclusions that the suspect in a case could be included in contributing the crime sample, unfortunately errors in data interpretation by the HPD laboratory led to the false conviction and incarceration of a young man accused of a 1998 rape.
In March 2003 Josiah Sutton's case made national headlines when it was revealed that DNA tests performed by Identigene found that he could not have committed the crime for which he had been incarcerated for more than four years based on DNA evidence originally analyzed by HPD. With this news also came the stigma that DNA testing was fallible.
It is important to point out that many of the problematic tests performed by the HPD laboratory involved DNA mixtures and the use of an earlier, low-resolution PCR-based test known as HLA-DQA1 rather than the current and more precise method of STR typing. Since only six alleles are possible with DQA1 typing, it is inherently poor at separating mixture components.
As of mid-2004, over $4.6 million has been allocated for retesting of samples from almost 400 cases originally handled by the HPD laboratory. Thus, failure to achieve laboratory accreditation, properly train personnel, maintain adequate facilities, and follow guidelines for data interpretation can cost significantly more than just a laboratory's reputation. In May 2004, the HPD crime laboratory finally applied for accreditation with ASCLD/LAB. Hopefully in the coming months this laboratory will join the ranks of the careful laboratories around the world conducting quality forensic DNA testing.
The FBI DNA Laboratory has also come under fire in recent years largely due to the deceitful actions of a forensic biologist named Jacqueline Blake. Ms. Blake apparently ran over 100 cases in the FBI's DNA Analysis Unit I without performing testing of her negative control samples - and then falsified documents to make it appear as though she had followed the standard operating procedure. The Department of Justice's Office of the Inspector General issued a report in May 2004 reviewing the protocol and practice vulnerabilities of the FBI DNA Laboratory so that this type of failure is not observed again.
It is important to keep in mind that these two cases represent the rare exception rather than the rule as the vast majority of forensic laboratories work hard to be accredited, maintain analyst training and proficiency, carefully validate methods, and follow standard operating procedures. The science itself is sound and reliable when performed correctly. These situations simply illustrate the need for consistent internal quality assurance and external oversight to ensure procedural accuracy within a laboratory.
http://www.chron.com/content/chronicle/special/03/crimelab/index.html (Houston Chronicle); 'The FBI DNA Laboratory: A Review of Protocol and Practice Vulnerabilities' (Dept of Justice Office of the Inspector General, May 2004), see http://www.usdoj.gov/oig/special/0405/final.pdf
include validation, proficiency testing, laboratory accreditation, and the terms robust, reliable, and reproducible.
Validation refers to the process of demonstrating that a laboratory procedure is robust, reliable, and reproducible in the hands of the personnel performing the test in that laboratory. A robust method is one in which successful results are obtained a high percentage of the time and few, if any, samples need to be repeated. A reliable method refers to one in which the obtained results are accurate and correctly reflect the sample being tested. A reproducible method means that the same or very similar results are obtained each time a sample is tested. All three types of methods are important for techniques performed in forensic laboratories.
A proficiency test, as it relates to the DNA typing field, is an evaluation of a laboratory's performance in conducting DNA analysis procedures. These tests are performed periodically, usually on a semi-annual basis, for each DNA analyst or examiner. In fact, the DNA Advisory Board Standard 13.1 requires that each DNA analyst undergo an external proficiency test at regular intervals not exceeding 180 days (see Appendix IV). Biological specimens with a previously determined DNA profile are submitted to the laboratory personnel being tested. The purpose of the test is to evaluate their ability to obtain a concordant result using the laboratory's approved standard operating protocols (SOPs).
The tests may be administered by someone else in the laboratory (internal proficiency test) or by an external organization (external proficiency test). If the test administered by an external organization is performed such that the laboratory personnel do not know that a test is being conducted, then it is termed a blind external proficiency test. A blind external proficiency test is generally considered the most effective at monitoring a laboratory's abilities but can be rather expensive and time-consuming to arrange and conduct (Peterson et al. 2003a, 2003b). Participation in a proficiency-testing program is an essential part of a successful laboratory's quality assurance effort. Forensic laboratories develop their own proficiency-testing program or establish one in cooperation with other laboratories (see Rand et al. 2002, 2004). The German DNA profiling group (GEDNAP) has established a successful blind proficiency-testing program (D.N.A. Box 16.2).
A laboratory audit evaluates the entire operation of a laboratory. It is a systematic examination that may be conducted by the laboratory management or by an independent organization according to pre-established guidelines. A laboratory must possess standard operating protocols and adhere to them. Likewise, instruments and other equipment vital to the successful completion of a forensic DNA case must be maintained properly and personnel must be appropriately trained to perform their jobs. Records of an audit are maintained and serve to describe the findings of the audit and a course of action that may be taken to resolve any existing problems.
The purpose of proficiency testing is to evaluate the performance of an analyst using a sample or set of samples that is unknown to the analyst but known to the test provider. Recommendation 3.2 of NRCII (see Appendix VI) states that: 'Laboratories should participate regularly in proficiency tests, and the results should be available for court proceedings.' Successful completion of this examination permits a degree of confidence to exist in how an analyst might perform on a real forensic case sample. Unfortunately, if analysts are aware that they are being tested, they might be more careful than they would when normally processing routine samples on a daily basis. Thus, the concept of blind proficiency has often been discussed in order to have a true test of the entire system because the analysts would not know that they were being tested. However, a number of challenges and costs are associated with blind proficiency tests.
Four models exist for blind proficiency testing (Peterson et al. 2003a):
(1) Blind/Law Enforcement, where a law enforcement agency fully disguises the test as a routine case and participates in the deception of the target laboratory;
(2) Blind/Conduit Lab, where another laboratory perhaps part of a multi-laboratory state system submits appropriate specimens for a case and is part of the deception of the target laboratory; (3) Blind Analyst, where only the DNA analysts are in the dark about the test while laboratory QA coordinators administer the test, and (4) Random Audit/Re-analysis, where a re-examination of a case is performed by another analyst or an auditor external to the laboratory to review and even re-analyze the samples. Cost estimates for these various forms of blind proficiency range from $1400-$10000 per test with the blind analyst approach (number 3) being the least expensive and easiest to implement. Thus, running a program that tests say 150-200 DNA laboratories in the United States with two tests per year would become rather expensive.
Another challenge besides cost is the fact that deception of the other party is necessary in an effort to create a 'real' case situation. When laboratories and law enforcement agencies are trying to build trust, deception for the purpose of quality assurance may seem a bit extreme. In addition, protection of specimen donors is important and if a 'case' is entered into CODIS, can the profiles for the innocent donors be purged? Likewise, if only a few donors are used in the proficiency test, then DNA laboratories might figure out in a short period of time which profiles are part of the blind proficiency test. Based upon these and other considerations, implementation of a large national blind proficiency-testing program was not recommended in the U.S. (Peterson et al. 2003b).
In Europe, the German DNA profiling group (GEDNAP) has developed a blind trial concept, which is really a 'graded' inter-laboratory test. The primary requirement of this blind trial is that all participants receive exactly the same material to be tested enabling a direct comparison with the known standard as well as an inter-laboratory comparison to be carried out (Rand et al. 2002). Samples are prepared to be as close to real casework situations as possible.
The GEDNAP trials have four purposes: (1) standardization of methods and procedures; (2) standardization of nomenclature; (3) evaluation of the competence of a laboratory to obtain the correct result, and (4) elimination of errors in typing. GEDNAP trial 22 and 23 conducted in 2001 had 122 participating laboratories from 28 European countries while participation grew to 160 laboratories for trial 26 and 27 held in 2003 (Rand et al. 2004).
Laboratories are each assigned a code number that enables anonymity throughout the inter-laboratory process. Typically seven samples are provided with each GEDNAP trial. When results are returned to the organizing laboratory in Münster, each allele call is classified into one of four categories: (1) no errors; (2) mixture not detected; (3) error in typing but would not be reported; and (4) error in typing which would be reported. Finally, a certificate is issued by the organizing laboratory, which states that the laboratory in question has successfully completed the blind trial for the particular loci examined (Rand et al. 2002).
The types of errors observed in the GEDNAP trials show that human carelessness is the predominant source of error with transposition of samples and transcription errors (Rand et al. 2004). The error rate over the past few years has held relatively constant at 0.4-0.7%.
Peterson, J. et al. (2003) The feasibility of external blind DNA proficiency testing. I. Background and findings. Journal of Forensic Sciences, 48, 21-31. Peterson, J. et al. (2003) The feasibility of external blind DNA proficiency testing. II. Experience with actual blind tests. Journal of Forensic Sciences, 48, 32-40.
Rand, S. et al. (2002) The GEDNAP (German DNA profiling group) blind trial concept. International Journal of Legal Medicine, 116, 199-206. Rand, S. et al. (2004) The GEDNAP blind trial concept part II. Trends and developments. International Journal of Legal Medicine, 118, 83-89.
Laboratory accreditation results from a successful completion of an inspection or audit by an accrediting body. A list of major accrediting organizations that are recognized by the forensic DNA community is contained in the next section. Accreditation requires that the laboratory demonstrates and maintains good lab practices including chain-of-custody and evidence handling procedures.
The accreditation process generally involves several steps such as a laboratory self-evaluation, filing application and supporting documents to initiate the accreditation process, on-site inspection by a team of trained auditors, an inspection report, and an annual accreditation review report. The inspection evaluates the facilities and equipment, the training of the technical staff, the written operating and technical procedures, and the casework reports and supporting documentation of the applicant laboratory.
According to the DNA Advisory Board Standards accepted as the national standards in the United States (see Appendix IV), all examiners who are actively engaged in DNA analysis need to undergo proficiency tests on at least a semi-annual basis (standard 13.1). Likewise, laboratory audits need to be conducted on an annual basis by the laboratory (standard 15.1) and on a bi-annual basis by an outside agency (standard 15.2).
ORGANIZATIONS INVOLVED IN ENSURING QUALITY AND UNIFORMITY OF DNA TESTING
A number of organizations exist around the world that work on a local, national, or international level to ensure that DNA testing is performed properly. The organizations are made up primarily of working scientists who want to coordinate their efforts to benefit the DNA typing community as a whole.
Was this article helpful?
This book discusses the futility of curing stammering by common means. It traces various attempts at curing stammering in the past and how wasteful these attempt were, until he discovered a simple program to cure it. The book presents the life of Benjamin Nathaniel Bogue and his struggles with the handicap. Bogue devotes a great deal of text to explain the handicap of stammering, its effects on the body and psychology of the sufferer, and its cure.