Validation of Immune-Complex Enzyme Immunoassays for Diagnosis of Pneumococcal Pneumonia among Adults in Kenya

RESULTS

The mean ages of the pneumococcal pneumonia patients and controls were 32 and 35 years, and the percentages of males in each group were 67 and 51%, respectively. Half of the pneumococcal pneumonia group and 31% of the control group were human immunodeficiency virus (HIV) positive. Each control provided one serum sample. Each of the 129 pneumococcal pneumonia patients was sampled on admission, a median of 6 days after the onset of illness. Most pneumococcal pneumonia patients were also sampled at discharge (n = 89), a median of 5 days after admission, and at a follow-up appointment (n = 102), a median of 27 days after admission. Every sample was assayed, but only the highest value for each subject was used in the analysis. For the pneumolysin IC assay the maximum value was obtained from the acute-phase serum in 41 patients (32%), from the discharge serum in 47 patients (36%), and from the follow-up serum in 41 patients (32%). The optical density (OD) result for each serum was standardized from plate to plate by using the formula OD =a × m/c, where a is the mean OD of the test serum duplicates, m is the mean OD for the high-reactivity control throughout the whole assay, and c is the OD for the high-reactivity control on the plate being read. For each assay, the distribution of these standardized ODs was log normal. The coefficient of variation of the pneumolysin IC-EIA, estimated with one sample run 19 times, was 9.2%.

Receiver operating characteristic (ROC) curves were constructed from estimates at each 0.1-U increase in log OD. The curve for the pneumolysin IC-EIA is shown in Fig. 1. In addition, sensitivity estimates were maximized by selecting the lowest OD threshold that would yield specificity at a fixed level. At a specificity of 0.99, the sensitivity estimates for the IC-EIAs for pneumolysin, C-polysaccharide, and mixed capsular polysaccharide antigens were 0.17, 0.14, and 0.12, respectively; the results at specificity criteria of 0.95 and 0.90 are shown in Table1. Restricting the sensitivity analysis to 74 patients who had positive blood or lung aspirate cultures did not alter the performance of the three tests (Table 1).

• Open in new tab
• Download powerpoint

Fig. 1.

ROC curve for the pneumolysin IC-EIA.

To examine whether the poor detection characteristics of the pneumolysin IC-EIA were restricted to B. subtilis-derived antigen, the assay was repeated with E. coli-derived antigen for 95 of the controls and 126 of the cases of pneumococcal pneumonia. The correlation between the two assays was good (r = 0.85), and the detection characteristics with E. colipneumolysin were indistinguishable from those with B. subtilis pneumolysin (Table 1).

The pneumolysin IC-EIA results varied significantly with HIV infection status. Among pneumococcal pneumonia patients, the geometric mean OD was 639 for HIV-positive patients and 321 for HIV-negative patients (P < 0.0001); for the negative controls, the results were 356 and 129 respectively (P < 0.0001). However, stratification of the analysis by HIV infection status did not significantly alter the sensitivity estimates obtained at the fixed specificity thresholds (Table 1). Males were overrepresented in the pneumococcal pneumonia patient group, but restriction to male sex did not alter the sensitivity either (Table 1).

To explore whether a subgroup of the control population was responsible for the poor detection characteristics of the pneumolysin IC-EIA, the subjects were grouped by clinical diagnosis. The effect upon specificity of eliminating any 1 of 14 diagnostic groups from the analysis was estimated for each group in turn at a fixed sensitivity of 0.50 (Table 2). The specificity estimates varied within a narrow range (0.80 to 0.86), being greatest after elimination of patients with gastroenteritis, among whom HIV-seropositive patients were strongly represented (13 of 17).

The ROC curves for the C-polysaccharide IC-EIA and the capsular polysaccharide IC-EIA were similar to those for the pneumolysin IC-EIA, as were the maximized sensitivity estimates at fixed specificity thresholds (Table 1).

DISCUSSION

Epidemiological investigations, and particularly vaccine efficacy studies, require a diagnostic case definition with an extremely high specificity, in the region of 0.98 to 1.00. Misclassification of true negatives as positive cases dilutes the association between unvaccinated status and disease, leading to an artificially low point estimate of vaccine efficacy with a broader variance estimate. With only a modest reduction in specificity it is possible to observe a null result in the study of a highly efficacious vaccine. It is critically important to evaluate the specificity of the case-defining techniques prior to their inclusion in a study. We have evaluated IC-EIAs by using pneumolysin, C-polysaccharide, and capsular polysaccharide antigens and found that the OD thresholds required to exceed the sensitivity of blood cultures in the diagnosis of pneumococcal pneumonia yielded specificity estimates of <0.90.

In previous studies of the pneumolysin IC-EIA, OD thresholds have been determined by sampling normal healthy children to set a specificity of 0.93 to 1.00, and this has produced a useful number of diagnoses in studies of CAP (11, 12; Lankinen et al., 1st Int. Symp. Pneumococci Pneumococcal Dis.). For adults, published specificity estimates have also been derived only from populations of healthy subjects (12, 13). In choosing a negative control group, we must strike a balance between two competing requirements: the need to stress the assay with sera from sick individuals and the need to exclude all cases of pneumococcal disease. Sera from acutely ill patients can give rise to nonspecific positivity in serological assays. For example, in 1968 Sutliff and Zoffuto (18) set a threshold for their pneumolysin antibody neutralization test that excluded 24 of 25 normal healthy controls (i.e., specificity = 0.96). This correctly diagnosed half of 48 pneumococcal pneumonia patients, but when applied to a control population of 62 medical admissions without fever or pneumonia, it produced a specificity of only 0.73 (18). As the test will be applied only to sick individuals, the selection of normal healthy controls tends to overestimate specificity. If we select controls who are not healthy, however, it is important to ensure that their illnesses are not due to pneumococcal disease; otherwise, the specificity of the assay will be underestimated. In practice, it is impossible to exclude cases of pneumococcal disease by using laboratory diagnostic tests because they have such low sensitivities. Blood cultures fail to diagnose 70 to 80% of patients with pneumococcal pneumonia, and the urine antigen assay used here misses more than half of all cases (17). For these reasons we recruited sick individuals to our negative control group but attempted to exclude patients with pneumococcal disease on broad clinical criteria.

The prevalence of HIV in our controls was high, and as pneumococcal disease may have unusual manifestations in HIV-positive subjects (16), there is a possibility that the poor specificity we observed could be due to inadvertent inclusion of atypical pneumococcal cases in the negative control group. However, if the true sensitivity and specificity of the pneumolysin IC-EIA were 0.50 and 0.98, respectively, we would have to postulate that as many as 17% of the controls were occult pneumococcal cases to obtain concordance with the observed results, and this seems most improbable.

The serological differentiation of populations with and without pneumococcal disease may be hampered by cross-reactivity in human sera, although the identity of potential cross-reactive antigens is not immediately obvious. Pneumolysin resembles other cytolytic, gram-positive proteins, including streptolysin O, perfringolysin, tetanolysin, cereolysin, and listeriolysin (6, 20), but previous studies of anti-streptolysin O and antipneumolysin antibodies have revealed little correlation (6). Recombinant pneumolysin might also contain natural antigen from the vector organisms, although again there is little evidence indicating a cross-reactive effect; antibody EIA results with B. subtilispneumolysin have been shown to be comparable to those obtained with natural antigen in Finnish subjects (4), and the use ofE. coli, to which human antibodies are widespread (3), as a vector did not produce a significant decrease in the detection characteristics of the assay in our hands.

An alternative explanation of the disappointing results reported here is that transient, intermittent, subclinical pneumococcal bacteremia may lead to detectable concentrations of immune complexes in serum. This idea receives support from the demonstration of pneumolysin gene fragments in the blood of normal healthy children in Israel by PCR (2). If adults also experience transient pneumococcal bacteremia, this could give rise to false-positive results with the IC-EIAs, particularly in the tropics, where exposure to S. pneumoniae is thought to be higher than it is elsewhere.

The insensitivity of individual serological tests might be overcome by combining a number of tests into a single protocol (10), but it is important to remember to apply the same protocol to the negative control group as well. Using the OD thresholds that yielded a specificity of 0.90 in each of the three different antigen assays, we assessed the detection characteristics of two combination case definitions. Protocol A was defined by a single positive result on any one of the IC-EIAs, i.e., that for pneumolysin, C-polysaccharide, or capsular polysaccharides. Protocol B was defined by positivity on all three of the different IC-EIAs. Sensitivity was marginally improved by protocol A (0.45) but with significant loss in overall specificity (0.82). Specificity was improved by protocol B (0.96) but with significant loss in sensitivity (0.16). Combining diagnostic tests is likely to improve diagnosis only if the specificities of the individual tests are very high and the results of the different assays are not highly correlated. Similarly, repetition of the serological tests with several sera per patient, as undertaken in our study, should ideally be reproduced in the control group to measure the accumulation of false-positive results with each repetition. As we did not do this, our results may actually overestimate the performance of these tests.

The assay of IC to common pneumococcal antigens is an innovative response to a difficult diagnostic problem, but unfortunately it has not differentiated cases from controls sufficiently well to prove clinically useful in Kenya. Variations in the methodology, e.g., assaying sera at several dilutions or including a dissociation step, might improve the detection characteristics, although they would detract from the simplicity of the assay described. In a temperature climate, where the prevalence of carriage and the incidence of transient pneumococcal invasion are thought to be lower than they are in Kenya, the tests may perform significantly better than reported here, but they have yet to be validated with an appropriate control group. For Kenyan adults, the IC-EIAs could surpass the sensitivity of blood cultures only at levels of specificity that were insufficient for the performance of analytical epidemiological investigations or vaccine efficacy studies.