Clinical and Diagnostic Laboratory Immunology, June 2005, p. 685-692, Vol. 12, No. 6
1071-412X/05/$08.00+0 doi:10.1128/CDLI.12.6.685-692.2005
Copyright © 2005, American Society for Microbiology. All Rights Reserved.
Department of Pathobiological Sciences, School of Veterinary Medicine, University of Wisconsin-Madison, 2015 Linden Dr., Madison, Wisconsin 53706-1102,1 Department of Veterinary Population Medicine, College of Veterinary Medicine, University of Minnesota, 1354 Eckles Ave., St. Paul, Minnesota 55108,2 Minnesota Board of Animal Health, 90 W. Plato Blvd., St. Paul, Minnesota 55107,3 University of Minnesota Veterinary Diagnostic Laboratory, 1333 Gortner Ave., St. Paul, Minnesota 55108,4 School of Veterinary Medicine, University of Pennsylvania, New Bolton Center, 382 West Rd., Kennett Square, Pennsylvania 19348-16925
Received 18 February 2005/ Returned for modification 7 March 2005/ Accepted 14 March 2005
|
|
|---|
99.8%. The specificity of the remaining two ELISAs, both done on serum, was 94.9 and 84.7%. Four of the five ELISAs evaluated produced similar sensitivity in detecting fecal culture-positive cattle (27.8 to 28.9%). Serum ELISA "D" had the lowest specificity (84.7%) and the highest sensitivity (44.5%), but if the cutoff value defining a positive test was changed from 125 to 250% (of the positive control) the sensitivity and specificity, 31.8 and 97.5%, respectively, were comparable to those of the other four assays. If the case definition for M. paratuberculosis infection was based on the culture results of a single laboratory instead of the combined results of three laboratories, ELISA sensitivity estimates were 45.7 to 50.0%. With the exception of ELISA D, assay agreement was high (kappa 0.66 to 0.85) for categorical assay interpretations (positive or negative), but linear regression of quantitative results showed low correlation coefficients (r2 = 0.40 to 0.68) due to the fact that ELISA results for some cows were high in one assay but low in another assay. Likelihood ratio analysis showed a direct relationship between the magnitude of ELISA result and the odds of a cow shedding M. paratuberculosis in its feces. If used judiciously and interpreted quantitatively, these ELISAs are useful tools in support of paratuberculosis control programs in dairy herds. |
|
|---|
When within-herd infection rates are high, it is not economically feasible to cull (remove for slaughter) all test-positive cows. The net cost to cull and replace a Holstein dairy cow for a dairy producer is at least US$1,000 at today's cattle prices. Consequently, it is necessary to adopt testing strategies that provide both diagnostic and prognostic information. The owner needs to know which cows are most infectious and are not likely to survive another lactation; these cows need to be removed from the herd. It also would be helpful to know which infected cows are least infectious and are capable of sustaining another lactation and generating farm income.
Veterinary diagnostics for food animals are more strongly affected by end user economics than diagnostics for human diseases: there are no third-party payers, and profit margins in animal agriculture are small. Consequently, the most accurate and informative test results must be provided to the end users at the least cost. The diagnostic technology fulfilling this need is often based on antibody detection using enzyme-linked immunosorbent assay (ELISA) technology because of its low cost and high-throughput potential through automation. Several new ELISA kits for bovine paratuberculosis based on serum antibody detection have become available, and some companies have adopted this technology for milk samples. The purpose of the present study was to evaluate the sensitivity and specificity of five such assays.
|
|
|---|
|
View this table: [in a new window] |
TABLE 1. Study herd characteristics
|
Fecal culture. Fecal samples were tested fresh (not after freezing) for M. paratuberculosis by culture independently by three laboratories using different protocols. The University of Wisconsin laboratory decontaminated samples with 1% hexadecyl cetylpyridinium chloride (HPC) for 24 h, concentrated samples by filtration, and cultured the processed samples in modified BACTEC 12B media as previously described (5). The University of Minnesota laboratory decontaminated samples by using 0.75% HPC for 24 h, concentrated the samples by sedimentation for 72 h, and inoculated the processed samples onto four Herrold's egg yolk (HEY) agar slants containing mycobactin-J and antibiotics (BD Diagnostic Systems, Sparks, MD) as previously described (37). The University of Pennsylvania laboratory used 0.75% HPC for decontamination, double incubation, and centrifugation to concentrate samples and then inoculated one tube of the same commercial HEY medium as used by the Minnesota laboratory and three tubes of HEY without antibiotics but containing mycobactin-J (39). All M. paratuberculosis isolates were confirmed by PCR for IS900 and/or mycobactin dependency testing (2, 36). The fourth aliquot of feces was used for evaluation of a new direct PCR-based assay for M. paratuberculosis (to be reported in a separate publication).
All laboratories provided a score (0 to 4) to categorize the number of M. paratuberculosis organisms per gram of fecal sample tested. The laboratories using HEY converted the average number of M. paratuberculosis colonies seen per slant to scores as follows: 0, none; 1, 1 to 9 colonies; 2, 10 to 49 colonies; 3, 50 to 99 colonies; and 4,
100 colonies. BACTEC readings in weeks to positive were converted to scores as follows: 0, no M. paratuberculosis isolated after 12 weeks incubation; 1, >8 weeks; 2, 6.1 to 8 weeks; 3, 4.1 to 6 weeks; 4,
4 weeks. This system is based on previous studies showing a direct relationship between the time to detection and the number of M. paratuberculosis inoculated into the BACTEC vial (22). The mean score for the three laboratories was used to classify each animal's fecal shedding rate.
Antibody detection-based assays. Four commercial ELISA kits, designated A, B, C and D, were evaluated for M. paratuberculosis serum antibody detection. All kits are self-contained having all necessary reagents, positive and negative control sera, and interpretation criteria. Kits A (HerdCheck M. paratuberculosis ELISA; IDEXX Laboratories, Inc., Westbrook, ME), B (ParaCheck; CSL/Biocor, Omaha, NE), and D (SERELISA ParaTB; Synbiotic Corp., San Diego, CA) are licensed by the U.S. Department of Agriculture (USDA). Kit C is produced in France (ELISA Paratuberculosis; Institut Pourquier, Montpellier, France). All antibody assays, except D, use a serum absorption step to remove antibodies that cross-react with Mycobacterium phlei a procedure that Yokomizo et al. demonstrated to increase M. paratuberculosis ELISA specificity (41, 42). All serum antibody assays were performed by University of Wisconsin laboratories according to manufacturers' instructions and interpreted as prescribed by the kit insert. The testing laboratories were blinded to fecal culture results.
A fifth ELISA (E) was based on detection of M. paratuberculosis antibody in milk samples. This in-house assay is not sold as a diagnostic kit but rather is offered as a service by a Michigan laboratory (Antel BioSystems, Inc., Lansing, MI). Milk ELISA results were provided by the testing laboratory without knowledge of the animal or herd M. paratuberculosis-infection status. Fewer cows were tested by ELISA on milk than on serum because not all cows in herds were lactating on the day of sampling.
Data analysis. The case definition of a noninfected cow was any cow resident in any one of the seven Johne's disease program level 4 herds (35). The absence of M. paratuberculosis infection in these herds was confirmed by fecal culture of all adult cattle in each of the seven herds by all three participating laboratories. Assay specificity estimates were based on tests done on these cattle, i.e., all positive ELISA tests on cattle in these herds were considered to be false-positive tests.
The case definition of an infected cow was isolation of M. paratuberculosis from a fecal sample by any one of the three laboratories. The antibody detection-based ELISAs were compared for their ability to detect the cattle that were fecal culture positive at the time of sample collection. This is not a measure of true assay sensitivity based on comparison to an accepted "gold standard" because infected cattle shedding M. paratuberculosis in their feces when sampled (or shedding but not detected by any of the three cultures) obviously were not counted as cases. For simplicity, however, although the sensitivity estimates reported in the present study are "relative sensitivity" estimates, i.e., relative to fecal culture, we use the term "sensitivity" throughout this study. No prior studies have used M. paratuberculosis culture results for the same sample from multiple independent laboratories to increase the chances of identifying cases of infection. Consequently, to gauge the effect this more rigorous case definition had on assay sensitivity estimates, we also report estimates based on a single laboratory's fecal culture results, namely, those from the University of Minnesota. This laboratory was chosen because it used the culture method most commonly cited in prior publications in which fecal culture was used as the reference test for serological test evaluation and because this method had a M. paratuberculosis detection rate in between those of the other two laboratories that performed fecal culture. The rate of positive ELISAs for culture-positive cows was also calculated by the level of M. paratuberculosis shedding by using the mean fecal shedding score.
Sensitivity and specificity differences among herds were determined by using the Z-statistic for comparison of independent proportions (25). Agreement between pairs of assays was evaluated by McNemar's chi-square analysis and kappa calculation (25, 28). In similar pairwise fashion, linear regression was performed, and r2 values were reported as measures of the correlation in magnitude of quantitative test results (GraphPad InStat, version 3.00; GraphPad Software, San Diego, CA). Differences were considered significant at P < 0.05.
The likelihood ratio (LR) was determined for each of five ELISA result interpretation ranges for each of the five assays (30). The test manufacturer's cutoff for a positive test was used as the lower limit for one of the five test interpretation ranges. The LR was calculated as follows: LR = (percent infected cows with results in the range)/(percentage of noninfected cows with results in the range). Post-test M. paratuberculosis infection probabilities were calculated for each of five assay interpretation levels and six within-herd infection rates ranging from 1 to 30% by standard methods (30).
|
|
|---|
Diagnostic specificity among the five ELISAs evaluated ranged from 84.7% (95% CI: 82.9 to 86.5%) for assay D to 100% for assay C (Table 2). The three highest specificity assaysB, C, and Ehad zero to two false-positive tests among the cattle tested, and assay specificities were not significantly different. The specificity of assays A and D varied significantly among herds ranging from 84.0 to 100% for assay A and 62.0 to 94.6% for assay D.
|
View this table: [in a new window] |
TABLE 2. Specificity of antibody detection tests
|
![]() View larger version (14K): [in a new window] |
FIG. 1. Number and percentage of cows at each level of fecal shedding based on their average fecal shedding score for 443 M. paratuberculosis-infected cows among seven infected herds.
|
|
View this table: [in a new window] |
TABLE 3. Percentage of cattle testing antibody positive compared to fecal culture results among the seven M. paratuberculosis infected dairy herds
|
With the exception of assay D, categorical result interpretation agreement among the assays was generally high: >86% of assays were in agreement when the sample was classified as positive or negative, no differences were noted by
2 analysis, and kappa values were 0.66 to 0.85 (Table 4). Assay D was different from all other assays due to its higher sensitivity (44.5%) and lower specificity (84.7%), resulting in kappa values of 0.47 to 0.59 in pairwise analysis between ELISA D and each of the other four assays.
|
View this table: [in a new window] |
TABLE 4. Pairwise comparison of antibody detection-based tests for paratuberculosis on fecal culture-positive cows
|
![]() View larger version (27K): [in a new window] |
FIG. 2. Scatter plots with linear regression lines comparing quantitative results for four pairs of ELISAs selected to show a range of correlations. (A) Kit A and kit B; (B) kit A and kit C; (C) kit D and kit E; (D) kit B and kit E.
|
|
View this table: [in a new window] |
TABLE 5. LRs for five result ranges for each of the five assays evaluated
|
0.100, the post-test probability of M. paratuberculosis infection was >90% when the within-herd prevalence was >10%. At the highest level of ELISA B result (i.e.,
0.500) the post-test probability of infection was >90% when the within-herd prevalence was >5%. |
View this table: [in a new window] |
TABLE 6. Post-test probabilities of M. paratuberculosis infection for ELISA B at five levels of assay result and within-herd M. paratuberculosis infection prevalence ranging from 1 to 30%a
|
|
|
|---|
For paratuberculosis diagnostics it can be challenging to evaluate an assay's specificity. In dairy cattle the time from infection, usually as a calf, to clinical disease varies from as short as 1 year to as much as 12 years, the upper limit of life span for dairy cattle on most farms (17). Prolonged disease latency makes it impossible to rely on a negative fecal culture as a "gold standard" to define absence of M. paratuberculosis infection for animals residing in known infected herds. Instead, the best standard for absence of M. paratuberculosis infection for an individual animal must be based on absence of infection from the entire herd in which the animal was raised. Moreover, absence of infection from a herd must be based on multiple annual negative tests of all adult cattle, as well as strict adherence to recommended management practices that prevent introduction of M. paratuberculosis-infected cattle into the herd. These standards are outlined in the Voluntary Bovine Johne's Disease Control Program (35) and served as the gold standard to qualify the adult cattle in seven dairy herds for ELISA specificity estimation in the present study.
The diagnostic specificity for three of the five ELISAs evaluated was
99.8% and not statistically different. This corroborates the findings of similar studies on the same ELISA kits (27). ELISAs A and D had significantly lower specificities, 94.9 and 84.7%, respectively (Table 2). The specificity rates of these two assays also varied among herds, whereas the other three ELISAs' specificity rates did not. Reasons for assay specificity differences among herds are not known but likely relate to qualitative or quantitative differences in the microbial flora the animals in those herds are exposed to, notably the presence of other mycobacteria, as well as the composition of antigens used as solid-phase antigens on ELISA plates and/or antigens contained in serum or milk absorption reagents. Herd-to-herd assay specificity variation is an important consideration in judging assay performance; it may even be necessary to include herds from different geographic areas (and therefore likely varied microbial flora) to obtain a full sense of the range of specificities displayed by the assay. Herd screening for possible M. paratuberculosis infection by ELISA demands use of high specificity assays to limit false-positive herd classifications and the time and effort spent trying to confirm the diagnosis (7, 16).
Among the fecal culture-negative cattle in the infected herds, <2.9% tested positive for serum antibody and <3.7% tested positive for antibody in milk for the three highest-specificity ELISAs. Arguably, these apparently ELISA false-positive cattle were truly infected with M. paratuberculosis, but fecal culture failed to detect the organism because the animal was not shedding it in feces on the day of sampling or was shedding at a level below the culture detection limits. M. paratuberculosis exposure, triggering antibody production without progressive infection, e.g., exposure as an adult when more infection resistant, is also a plausible explanation for ELISA positive findings in fecal culture-negative cattle. Regardless, this was a low-frequency occurrence for the high-specificity assays and of limited concern should these assays be used in a paratuberculosis control program.
The sensitivity of M. paratuberculosis fecal culture-positive dairy cattle detection by ELISA was the same for four of five assays and was related to the rigor of the case definition. If the sensitivity of these same assays were judged in comparison to the fecal culture results of a single laboratory (Minnesota Veterinary Diagnostic Laboratory), then the estimated sensitivities would be much higher, i.e., 45.7 to 50.0%, and in agreement with previous reports (6, 15, 21, 29, 31, 32, 39, 40). It is noteworthy that the diagnostic sensitivity of a new direct high-throughput PCR assay applied to fecal samples from cattle used in the present study, using the same single case definition, was also in this range (data not shown). Clearly, the fecal culture methodology and number of samples evaluated to establish a case definition affect the estimates of the sensitivity of other assays being evaluated and are a major study design issue to be considered when evaluating diagnostic tests for paratuberculosis. Given that antibody-based diagnostics detected fewer than one-third of all M. paratuberculosis-infected and excreting dairy cattle in a herd, the percentage of cattle testing positive could be multiplied by at least a factor of three to get a rough estimate of true within herd prevalence of infection. This is only applicable, of course, for herds that have not previously been testing for paratuberculosis and culling cattle based on test results.
The number of CFU of M. paratuberculosis in bovine fecal samples is considered a measure of the stage of infection in the animal. In the present study, the ability of an ELISA to detect an infected animal was directly related to the level of M. paratuberculosis shedding (Table 3), a finding consistent with previous reports (6, 32). Cattle with large numbers of M. paratuberculosis per gram of feces, so-called "heavy shedders," were detected by the evaluated ELISAs >72% of the time. This is significant in that these animals logically represent the greatest risk for environmental contamination and thus infection transmission on a dairy farm. Evidence from field studies to support the contention that detection and management of the heavy shedders is critical to control of bovine paratuberculosis are lacking. However, both modeling and field data on human tuberculosis verify that focusing treatment and control efforts on the most infectious subset of Mycobacterium tuberculosis-infected persons can cause dramatic reductions in the prevalence of the infection in the general population (10). Whether the cattle shedding large numbers of M. paratuberculosis but missed by ELISAs quickly progress to clinical paratuberculosis and are therefore culled from the herd or remain in the herd and continue to spread infection is not known. Only well-designed and controlled longitudinal field studies will be able to answer these questions.
When ELISA results were judged dichotomously (positive or negative), the assays showed comparable accuracy and good agreement. ELISAs A, B, C, and E did not differ in classification of culture-positive cattle as test positive or negative (
2, P > 0.05), and the kappa statistic indicated a high level of agreement (kappa
0.66; Table 4). Of these assays, B, C, and E were not significantly different in either sensitivity or specificity. ELISA D would have had comparable sensitivity and specificity estimates, i.e., 31.8 and 97.5%, respectively, if an assay cutoff of 250% (sample OD as a percentage of kit positive control) were used instead of 125% as recommended by the kit manufacturer (Table 5).
Scatter plots illustrate that serum or milk from individual cattle can respond very strongly when tested by one ELISA but not another. Representative plots are shown in Fig. 2, and linear regression correlation coefficients for all pairwise assay comparisons are listed in Table 4. Clearly, cattle can respond to M. paratuberculosis infection with serum antibodies detected by one ELISA kit but not in another. Since the formulation of the M. paratuberculosis solid-phase antigen and M. phlei serum absorption antigens used in these kits is proprietary, the nature of putative antigenic differences in kit components is not known. Identification of the diversity of M. paratuberculosis antigens or epitopes that induce antibody responses in cattle could lead to improved diagnostic kits. Purified single antigen assays will likely suffer from low diagnostic sensitivity.
LR analysis confirmed the earlier report that multilevel paratuberculosis ELISA interpretation is useful (4, 26). All five ELISAs evaluated showed a direct relationship between the magnitude of ELISA result and the likelihood the tested cattle were fecal culture-positive for M. paratuberculosis. LRs in combination with estimated within herd prevalence, i.e., the pretest probability of M. paratuberculosis infection, can be used to calculate the post-test probability of infection (shown for ELISA B in Table 6). For ELISAs A, B, C, and E, cattle with results in the highest of the five ELISA result ranges, so called "strong-positive" results, had a post-test probability of concurrent M. paratuberculosis fecal shedding of >90% if herds had a within herd prevalence >10%. For ELISAs B and C, a high (>90%) post-test probability of M. paratuberculosis infection was also possible for the next-lower level of assay result when herds had a >15% within-herd infection prevalence. For most M. paratuberculosis-infected commercial dairy herds, confirmatory testing of such animals by fecal culture would be neither necessary nor cost-effective. Post-test infection probabilities can be used in decision analysis models to optimize economic outcomes from decisions based on ELISA results. However, this requires estimation of the infection transmission risks for animals at each level of M. paratuberculosis shedding, a factor affected by on-farm management systems. Measurement of infection transmission risks for cows by stage of infection under different farm management conditions by well-controlled epidemiological field studies is an important next step in optimization of paratuberculosis control programs in dairy herds.
An important first step in a national paratuberculosis control program is segregation of cattle herds into infected and not infected categories (1, 18, 33). To halt the spread of M. paratuberculosis infection, the next step is to apply voluntary or mandatory biosecurity regulations to prevent movement of cattle from infected (or untested) herds into noninfected or at least test-negative herds. Rapid, low-cost, high-throughput assays are necessary if even a modest proportion of the 95.8 million cattle in the United States are to be tested (34). The use of high-specificity ELISAs avoids unnecessary follow-up testing of herds due to false-positive tests (7). Higher assay sensitivity than that available with current ELISAs for detection of infected individual animals would be desirable, but herd-level sensitivity is the more critical factor, and this can be improved by increasing the number of animals tested per herd and by focusing testing on the older animals since they have had sufficient time for a M. paratuberculosis infection to progress and induce antibody production (15).
ELISA E performed as well or better on milk samples than the ELISAs that used sera and are in agreement with a recent report on evaluation of this same commercial test (14). Assays that use milk samples are ideally suited for paratuberculosis control in dairy herds but impractical for use on beef cattle. Milk samples are normally collected by testing laboratories from all lactating cows in a herd on a monthly basis for measurement of somatic cell counts (a measure of udder health), plus milk fat and protein content. Milk testing laboratories are equipped with automated sample handing equipment that allows thousands of samples to be analyzed daily. In addition, electronic transmission of results downloaded to on-farm computers is common. These data are used on a daily basis for herd management. Milk testing laboratories could thus adapt paratuberculosis ELISA technology to expand the services they provide to dairy producers and more cost-effectively test dairy cattle. Unfortunately, at present there are no USDA-licensed kits validated for testing milk samples for antibodies to M. paratuberculosis.
Organism detection tests for paratuberculosis will remain a mainstay for the definitive diagnosis of paratuberculosis. Isolation of M. paratuberculosis in liquid or on solid medium not only provides the most sensitive and specific diagnostic method available but also offers the ability to complete molecular studies that increase our understanding of the epidemiology and microbial ecology of this pathogen (3, 39). Genetics-based diagnostic tests have not yet matched the accuracy of culture-based diagnostics for paratuberculosis and fail to provide viable organisms for further study (9, 11, 20, 24). If M. paratuberculosis becomes classified as a zoonotic pathogen and for adequate food safety it is decided that dairy and beef products only originate from paratuberculosis test-negative herds, the need for all types of paratuberculosis diagnostics on multiple types of animals and possibly humans will be enormous.
The dedicated efforts of the technical staff of the Johne's Testing Center, Vic Eggleston, Gail Thomas, Heather Cushing, Kelly Anklam, Brenna Kunkel, Melissa Hampton, Johnnie Ambrosy, the R. D. Schultz laboratory, and Terry Fyock at the University of Pennsylvania are sincerely appreciated. Also, Lindsey Aipperspach and the field veterinarians of the Minnesota Board of Animal health and the technicians of the various Dairy Herd Improvement associations who helped collect samples from dairy herds are gratefully acknowledged. We are especially indebted to the dairy herd owners who not only made their herds available for the present study but also assisted with sample collection and animal handling. We thank Becton Dickinson for the donation of a portion of the culture media used in the study and Philippe Pourquier of Institut Pourquier and Synbiotics for the donation of ELISA kits.
|
|
|---|
This article has been cited by other articles:
| |||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Copyright © 2009 by the American Society for Microbiology. For an alternate route to Journals.ASM.org, visit: http://intl-journals.asm.org | More Info»