Performance evaluation of a Sars-CoV-2 rapid test and two automated immunoassays

Introduction: Due to urgency and demand of a response to the Covid-19 pandemic, numerous Sars-CoV-2 immunoassays have been rapidly developed. Objective: This study aimed at assessing the performance of rapid Sars-CoV-2 antibody test in comparison to high-throughput serological assays. Methods: A total of 86 serum samples were evaluated in the three assays: a lateral flow immunoassay – Wondfo Sars-CoV-2 Antibody Test (WRT) – and two chemiluminescence immunoassays: Elecsys Anti-Sars-CoV-2 (ECLIA), and Sars-CoV-2 IgG (CMIA-IgG). Results: The estimated diagnostic sensitivities of serological tests in the evaluation of serum samples from the epidemiological survey were: WRT 59% [95% confidence interval (CI) 43.4%-72.9%], ECLIA 66.7% (51%-79.4%), and CMIA-IgG 61.5% (47.1%-73%). Meanwhile, the estimated diagnostic specificity was for WRT 78.7% (95% CI 65.1%-88%), ECLIA 72.3% (58.2%-83.1%), and CMIA-IgG 76.6% (74%-95.5%). The sensitivity and specificity values were lower than manufacturers’ claimed. Although 16.2% (14/86) of serological results were discordant among the three Sars-CoV-2 serological assays, the degree of agreement by the kappa index was adequate: WRT/CMIA-IgG [0.757 (95% CI 0.615-0.899)], WRT/ECLIA [0.715 (0.565-0.864)], and ECLIA/CMIA-IgG [0.858 (0.748-0.968)]. Conclusion: The serological testing may be a useful diagnostic tool, which reinforces its careful evaluation, and, as well as the correct time to use it.


INTRODUCTION
Laboratory diagnosis has played a fundamental role in rapid diagnostic of coronavirus disease 2019 (Covid- 19) pandemic (1) . Some different methods are used routinely to confirm the diagnostic or in epidemiological survey and post-vaccination seroconversion.
Among nucleic acid tests, the reverse-transcriptase polymerase chain reaction (RT-PCR) method is considered as the 'gold standard' for the detection of severe acute respiratory syndrome coronavirus 2 (Sars-CoV-2) because of its benefits as specificity and for representing a simple qualitative assay (2) . However, in some situations, the sensitivity of RT-PCR tests has been sub-optimal due to different issues as: low viral loads at the time of infection, improper sample collection, conservation and transport of sample, and characteristics of gene targets (2)(3)(4)(5) .
In cases with high clinical suspicion of infection with negative PCR results, antibody detection can be a useful tool to establish exposure, infection, and immunity to Sars-CoV-2, as well as to perform epidemiological studies (5,6) . Compared to PCR, serological test is advantageous because of its faster turnaround time, highthroughput, less workload, and affordability of samples (3,7) .
Serology plays a key role in contact tracing, epidemiological survey, identification of convalescent plasma donors (which may serve as a possible treatment for those who are seriously ill from , study of the efficacy and pattern of immune response to vaccines, and in the estimation of seroprevalence (4,8,9) . While RESUMEN Introducción: Debido a la urgencia y la demanda de una respuesta a la pandemia de Covid-19, se han desarrollado rápidamente numerosos inmunoensayos del Sars-CoV-2. Objetivo: Este estudio tuvo como objetivo evaluar el rendimiento de la prueba rápida de anticuerpos contra el Sars-CoV-2 en comparación con los ensayos serológicos de alto rendimiento. Métodos: Se evaluaron un total de 86 muestras de suero en los tres ensayos: un inmunoensayo de flujo lateral -Wondfo Sars-CoV-2 Antibody Test (TRW)y dos inmunoensayos de quimioluminiscencia: Elecsys Anti-Sars-CoV-2 (ECLIA) y Sars-CoV-2 IgG (CMIA-IgG). Resultados: Las sensibilidades diagnósticas estimadas de las pruebas serológicas en la evaluación de muestras de suero de la encuesta epidemiológica fueron: WRT 59% [ further evaluation is needed, due to urgency and demand of pandemic response, numerous Sars-CoV-2 immunoassays were rapidly being developed and placed on the market with limited validation on clinical samples (1,8) . However, to properly use serology tests, it is important to understand their performance characteristics and limitations (10) .
That way, validation of serological test is required for accurate diagnosis of Sars-CoV-2 infection (8) . However, there is much concern about lateral flow immunochromatographic assays, which has widespread due to their easy and fast performance but many of these tests lack adequate sensitivity and specificity evaluation (4) .
Different types of assays can be used to determine different aspects of immune response and functionality of antibodies. The tests can be broadly classified to detect either binding or neutralizing antibodies. The humoral immune response produces antibodies against different viral antigens including the nucleocapsid (N) protein and spike (S) protein; S protein is present on the viral surface and is essential for virus entry and is present on the viral surface, N protein is the most abundantly expressed immunodominant protein that interacts with ribonucleic acid (RNA) (11) .
Although current utility of serological testing is mostly for epidemiological studies, proper evaluation of the clinical sensitivity and specificity of these tests within the tested population is very important. The test specificity directly affects its positive predictive values (PPV) and thus the reliability of a positive result. Although high specificity is reported for many commercial Sars-CoV-2 serologic assays, not all of them consistently meet this specificity threshold (12) . In this context, identification of individuals showing a serological response to Sars-CoV-2 provides important complementary information by giving an evaluation of the fraction of individuals who have previously been infected (13) . In this study was aimed at assessing the performance of rapid test used in an epidemiological survey for Sars-CoV-2 compared to other serological assays, taking into account the specificity and sensitivity of the methods.
In addition to these assays, a CMIA [Sars-CoV-IgM (Abbott Diagnostics, Ireland)], test for specific IgM antibodies to the Sars-CoV-2 (CMIA-IgM) binding domain of the spike protein (S) was used only in the control group samples.
All samples were evaluated in the three Sars-CoV-2 serological tests.

Serum sample
The flowchart ( Figure) shows the samples evaluated in this study.

Epidemiological survey
Eighty six respiratory and serum samples were collected from July 5 th to July 26 th , 2020, as part of the epidemiological survey samples carried out at the Instituto Adolfo Lutz (IAL) to assess the exposure of its employees to Sars-CoV-2. All respiratory samples were evaluated for Sars-CoV-2 RT-PCR, 39 (45.3%) with detectable viral RNA and 47 (54.7%) with undetectable Sars-CoV-2 RNA (12 with Covid-19-compatible symptoms, and 35 without symptoms), and all serum samples by the three serological tests WRT, Roche ECLIA and Abbott CMIA-IgG.
The eighty six participants were grouped according to the time interval between the onset of symptoms and the blood collection (stage of the disease) for analysis in serological tests -initial phase (seven to 13 days after the symptoms of the disease), intermediate (14 to 20 days after the symptoms of the disease), and late phase (≥ 21 days after the symptoms of the disease) (8) .
In this study, the association of signs and symptoms to define the diagnosis of Covid-19 was according to the criteria established by the Ministry of Health of Brazil (14) .

Control group
Eighty four control serum samples were included to assess the performance of the assays used in this study, which were distributed in two groups: • positive control group -38 (45.2%) serum samples were obtained from 22 hospitalized patients with Covid-19 to assess the sensitivity of assays. All enrolled cases were confirmed to be infected with Sars-CoV-2 by means of RT-PCR test in samples from the respiratory tract. Serum samples were obtained in different stages of Covid-19 -15 (39.5%) samples in the initial phase, 12 (31.6%) in the intermediate and 11 (28.9%) in the late phase. The classification of the stage of the disease was the same used in the epidemiological survey samples (8) . The interval between the respiratory sample collection (RT-PCR) and the serum sample collection varied from -1 to +45 days (mean = 13.6 ± 11.6 days; median = 10 days); • negative control group -46 (54.8%) remnant serum samples from diagnostic routine of infectious diseases stored at the IAL from December 2017 to March 2018 were used to assess the specificity of assays and were characterized as follows: forty (87%) sera with positive results for one or more markers in serological tests of human immunodeficiency virus (HIV), syphilis, hepatitis B virus (HBV), hepatitis C virus (HCV), Mycoplasma pneumoniae (MP), dengue virus (DENV), Chikungunya virus (CHIKV), yellow fever virus (YFV), and six (13%) samples with negative results for serological markers in blood banks. All negative samples used were collected prior to 2020, before the Sars-CoV-2 is known to have circulated in Brazil.

STATISTICAL ANALYSIS
The diagnostic sensitivity of antibody test was defined as the percentage of positive results for antibodies against Sars-CoV-2 in the serum samples from confirmed patients with Covid-19 (with detectable Sars-CoV-2 RNA by RT-PCR), whereas the diagnostic specificity was defined as the percentage of negative results for antibodies against Sars-CoV-2 in the serum samples from individuals without Covid-19 (without detectable Sars-CoV-2 RNA by RT-PCR or negative control group samples). For the calculation of diagnostic sensitivity, diagnostic specificity, positive predictive value, negative predictive value, accuracy, and seropositivity the Microsoft Office Excel Program was used.
The measurement agreements between different antibody tests were interpreted by means of the Kappa (k) index, as proposed by Altman (1999) (15) and adapted from Landis and Koch (1977).  In the evaluation of the control group samples, WRT and CMIA-IgM presented the maximum sensitivity and specificity values (100%), while ECLIA showed the sensitivity and specificity values of 92.1% and 97.8%, respectively. The global sensitivity value of CMIA-IgG was 94.7%, with two samples of positive group control yielding negative results. On the other hand, these two samples showed positive results in CMIA-IgM and negative results in ECLIA, even though the later one is able to detect both G and M specific immunoglobulins. For the negative control group samples, which have been collected and stored before the appearance of Sars-CoV-2, we observed one negative sample for serological markers of blood banks with positive result by ECLIA (4.39 index: reference value < 1.00).

Control group samples
In the assessment of the 84 control group sera, the degree of agreement between serological tests is described in the Table 2.
The agreement between the two automated immunoassays and one rapid test for Sars-CoV-2 specific antibodies detection varied from 95.2% (k = 0.903) to 100% (k = 1.00). Although percent agreement between ECLIA and CMIA-IgG has been very good 97.6% (k = 0.951), both showed two false-negative results in samples of positive control group, classified in the initial stage of the disease.

Epidemiological survey samples
The samples were separated into four distinct groups according to the results of the RT-PCR and reports of symptoms: (i) detectable Sars-CoV-2 RNA with symptoms; (ii) detectable Sars-CoV-2 RNA without symptoms; (iii) undetectable Sars-CoV-2 RNA with symptoms; and (iv) undetectable Sars-CoV-2 RNA without symptoms. These results and those obtained from serological tests are detailed in the Table 3. The occurrence of an antibody positive result (any assay) among those with Covid-19 symptoms but with negative RNA test was significantly higher than among those RNA negative without symptoms (9/12 vs 5/35, p = 0.0002). Even if we consider as positive only those with reactivity in all three tests, the difference is still significant (p = 0.02), Table 3.

As shown in
The evaluation of the 86 samples showed eight cases with undetectable Sars-CoV-2 RNA and positive result in the three serological tests.
Although 16.3% (14/86) of serological results of survey showed some discordant result, the agreement between serological tests was adequate according to interpretation of Kappa index; the data are summarized in Table 4.
The estimated sensitivity and specificity values of the serological tests in the serum samples from the survey participants were calculated in relation to the Sars-CoV-2 RT-PCR results (presence of RNA/absence of RNA).

DISCUSSION
In this study we evaluated three serological assays with different designs to assess their performance in serum samples from documented Sars-CoV-2 RNA positive cases, hospitalized patients and participants of an epidemiological survey that included asymptomatic individuals and cases with symptoms compatible to Covid-19. RNA testing was performed at least once by all participants. As also samples collected before the pandemic, with or without other prevalent infections, were tested to estimate the specificity of antibodies to Sars-CoV-2. Although Sars-CoV-2 specific antibodies detection cannot be assumed as an immunity correlate to the virus, it is a useful tool for public health tasks especially in epidemiological surveys.
We compared the results of Sars-CoV-2 IgG/IgM WRT, performed on serum samples, and two high-throughput automated serological tests, all according to respective manufacturer's instructions. The use of point of care (POC) serological rapid tests facilitates access to diagnosing and allows for their use outside traditional health system settings. However, use of an antibodydetection POC test with serum, instead of whole blood, might improve its sensitivity, in addition to allowing direct comparisons with other serological assays that use serum samples. The results in this work refer to laboratory test of serum and extrapolation to field POC testing must consider this.
Sensitivity and specificity estimates shown may not be indicative of the real world performance of the tests (10) . The predictive value of a test may be different when it is applied in a general population rather than in the study sample in which it was first developed (16) . There was an increase in the sensitivity and specificity values of WRT when the samples of the control groups were assessed (Table 1). However, some differences may exist due to the different clinical conditions, period of onset of symptoms ("time of serological turning point"), epidemiological data, among others (17) . The positive control group consisted of samples from hospitalized patients who had moderate to severe symptoms of Covid-19, while survey participants reported only mild symptoms of the disease. The WRT sensitivity was increased with clinical severity and days of symptoms, corroborating data in the literature (18) . The negative control group was composed of serum samples collected long before the first case of Sars-CoV-2 reported in Brazil, unlike the survey participants samples with negative RNA Sars-CoV-2 results by RT-PCR, who may have acquired the infection previously or even other infections that may cause crossreactive antibody binding reactions. The agreement between ECLIA and CMIA-IgG was very good, but both showed disagreement in three samples from the control group -one from the negative control group and two from the positive control group. The two samples of positive control group, when evaluated in CMIA-IgM and WRT (IgM/IgG) were positive, however ECLIA, for total antibodies, did not detect the presence of the immunoglobulins in these samples in the initial stage of the disease. In patients with Covid-19, the production of IgM and IgG can be simultaneous and reach in some patients a plateau level after six days (initial stage of the disease) (19) . And about the negative control group sample, there was the presence of cross-reactive antibodies in the ECLIA.
In eight cases of the epidemiological survey with positive results in the three serological tests but without detected Sars-CoV-2 RNA, four of which did not report symptoms, the infection may have occurred previously and was not documented by the RT-PCR. In other four symptomatic cases it is possible that e4252021 the diagnostic sensitivity of RT-PCR has been influenced by the timing of specimen collection and period of the disease development (2,5) . Even though we were not able to rule out the hypothesis that this occurred due to the collection of the PCR sample after the oro/nasopharyngeal viremia period, for the purpose of the present study in the absence of Sars-CoV-2 RNA the serological reagent results were considered false-positive. The interpretation of our rates has to consider the fact that in most cases only one RT-PCR was obtained.
As observed in the study by Costa et al. (2020) (20) , the performance of the serological test may be superior to that of RT-PCR. Discordant results must be cautiously interpreted. In the case of RT-PCR negative result with clinical features suspicion for Covid-19, the sampling timing and the stage of the disease development play an important role in assay results (2) . Although the nucleic acid test or genetic sequencing serves as the gold standard method for confirmation of infection, yet several recent studies have reported Sars-CoV-2 false-negative results by RT-PCR (21) . One study indicates that 2/10 negative cases by the RT-PCR were confirmed as positive for Covid-19, resulting in a false-negative rate of approximately 20% (22) .
In the same way, the negative results of the serological tests in samples with presence of Sars-CoV-2 RNA by RT-PCR were considered false-negative. These samples included six participants in the late stage of the disease (29-50 days of onset symptoms), with sufficient time for seroconversion. False-negative results can make it difficult to prevent and control the Covid-19 pandemic, in the same way that false-positive results can lead to unnecessary stress for patient, contacts and the health system. To improve accuracy of diagnosis, it is necessary a rational approach based on the usefulness of each type of test, the how to collect the sample from the patient, and the right time to perform the sample collection (22) .
The serological surveys can aid investigation of an ongoing outbreak and retrospective assessment of the attack rate or extent of an outbreak. In cases where molecular assays are negative and there is a strong epidemiological link to Covid-19 infection, paired serum samples (in the acute and convalescent phase) could support diagnosis with available validated serology tests (23) . Testing algorithms with more than one test may be necessary to rule out false-positives by initial tests, such that has been the rule for HIV and hepatitis. The potential of some assays to have high specificities depending on their target isotypes, the antigens used, and establishment of cut-off values (24) .
In this context, the clinical data of individuals based on signal and symptoms of Covid-19 may be useful for the evaluation of serological test results. Serology is suited for supporting the e4252021 Performance evaluation of a Sars-CoV-2 rapid test and two automated immunoassays diagnosis of Covid-19 infection in RNA-negative symptomatic patients, and establishing seroprevalence in population studies (25) . Twelve survey participants with Sars-CoV-2 RNA not detected had reported the presence of symptoms of Covid-19, however these clinical indicators were not used to confirm the presence of disease for the purposes of this evaluation, because inevitably this can introduce some incorporation bias, it is expected to find more positive results in the tests used, and this tends to inflate the measured sensitivity of these tests. In this study, the result of RT-PCR was used as a parameter to define the infection by Sars-CoV-2. It's worth mentioning that, serological assays do not typically replace direct detection methods as the primary tool for diagnosing an active Sars-CoV-2 infection, but they do have several important applications in monitoring and responding to the Covid-19 pandemic (11) .
Eleven survey samples with detectable Sars-CoV-2 RNA and negative results in the three serological tests (eight cases with symptoms and three without symptoms), mainly in the six cases of late stage of the disease, possibly are false-negative serological results or factors related to the immune system of individuals. It is also important to note that some individuals do not develop detectable IgG or IgM antibodies following infection (absence of humoral immune response to Sars-CoV-2) or the antibodies may not be long-lasting. Thus, the absence of detectable antibodies does not necessarily rule out that they could have previously been infected (5,11,17) . In this context, although it's in the early stages of critical evaluation, can be included the relationship between disease severity and antibody titer (1) . All participants in this epidemiological survey who reported symptoms were mild, that reinforces this theory. In three (3/11) asymptomatic cases they may show lower antibody responses (26) . In the other two cases (2/11), serology for Sars-CoV-2 was performed less than seven days after the onset of symptoms. In the early days of an infection when the body's immune response is still building, antibodies may not be detected (low antibody titer). Antibodies most commonly become detectable 1-3 weeks after symptom onset, at which time evidence suggests that infectiousness likely is greatly decreased and that some degree of immunity from future infection has developed (11) . In the initial stage of illness, molecular detection of the virus is the primary tool for early and accurate diagnosis of disease, as antibody production is usually delayed or absent in this phase (27) .
Of the 39 participants with Covid-19 (30 cases with symptoms and nine without symptoms) confirmed by the presence of RNA Sars-CoV-2, for the three serological tests used, the lowest estimated sensitivity rate was 59% for the WRT (43.4%-72.9%), and the highest was 66.7% for the ECLIA (51%-79.4%) followed by 61.5% (47.1%-75%) for the CMIA-IgG. Even though different tests show comparable results in detecting anti-Sars-CoV-2 antibodies, factors such as test-specific methodology, the type of antigen used, cut-off value determination, the temporal dynamics of virus-specific IgM and IgG immune responses, and the presence of cross-reactive antibodies remain important and affect the overall performance of serological tests (5,17) .
A better sensitivity for serum samples with WRT was reported by Cota et al. (2020) (18) in a Brazilian study accessing the performance of 12 serological tests for Covid-19 diagnosis, that described an overall sensitivity for Wondfo at 71.7% (64.3%-78.2%). In another evaluation of the Wondfo, Conte et al. (2021) (19) have shown a sensibility of 47.6% (37.2%-58.1%) in capillary whole blood. However, it is important to note that the WRT may have a lower sensitivity when used with capillary blood. The antibody detection is more effective in plasma or serum samples than in whole blood (16,28) .
In the pairwise comparison of results from epidemiological survey participants testing the highest percent agreement (93%) [k = 0.858 (very good)] was observed between the ECLIA (total antibodies) and CMIA-IgG, both assays employing the Sars-CoV-2 nucleocapsid protein as antigen. Followed by WRT (antibody assay targeting the Sars-CoV-2 spike protein) and CMIA-IgG with 88.4% [k = 0.757 (good)], and WRT and ECLIA with 86% [k = 0.715 (good)]. Previous study showed that antibody to the nucleocapsid protein of Sars-CoV-2 is more sensitive than spike protein antibody for detecting early infection (26) . It is important to highlight that the results presented here cannot directly support a correlation between detectable Sars-CoV-2 antibodies and immunity to Covid-19.
Regarding the WRT used in this epidemiological survey, the sensitivity, specificity, and accuracy values reported in the Table 5 are lower than the manufacturers' claimed [sensitivity 86.43% [95% confidence interval (CI) (82.51%-89.58%); specificity 99.57% (95% CI 97.63%-99.92%); accuracy 91.61% (95% CI 89.10%-93.58%)]. Specificities of at least 99.5% are required to achieve a high positive predictive value in low-prevalence populations (11) . About the results of serological tests, in most of the country, including areas that have been heavily impacted by Covid-19, the prevalence of Sars-CoV-2 antibody is expected to be low, ranging from < 5% to 25%, so that testing at this point might result in relatively more false-positive results and fewer false-negative results (11) . In this study, the seropositivity (45.3%) is higher due to the fact that many participants present a positive result on RT-PCR, and the WRT showed slightly more false-negative (16/86) than false-positive (10/86) results, with similar positive (69.7%) and negative (69.8%) predictive results.
It should be noted that there are some limitations of this study as the number of serum samples evaluated, which was based on the availability of chemiluminescence assays, at that time. It's essential for refinement of the sensitivity values of assays a larger number of sera be tested, especially in samples from patients at different stages of the disease, because the sensitivity values of antibody tests are highly relevant for calculating the frequency of Sars-CoV-2 exposure in seroprevalence studies (29) . Samples from individuals at an initial stage of the disease are very important in assessing the performance of a serological test to check their sensitivity, in addition monitoring the seroconversion process. The Sars-CoV-2specific antibodies can be detected in the serum of approximately 40% of Covid-19 patients as early as seven days after the onset of symptoms, with seroconversion rates rapidly increasing to > 90% by day 14 (8,30) . As well as the evaluation of Sars-CoV-2 negative samples containing antibodies to other different diseases that have not been evaluated, such as cytomegalovirus (CMV), Epstein-Barr (EBV), other coronavirus, and autoantibodies to rule out crossreactivity (false-positive result) (5) . Another limitation of the study was a small number of samples in the initial and intermediate stages, which made it impossible to assess sensitivity and specificity in groups by stages of the disease, and be able to compare with other studies. Another caveat is that the results refer only to the batches of serological tests used, so it is important further studies for evaluating the clinical performance in other different batches to ensure accuracy.

CONCLUSION
In addition to the RT-PCR for Sars-CoV-2, the serological testing may be a useful diagnostic tool as long as a correct timing regarding antibody production is considered. Serologic tests can be used in epidemiological survey, but should be assessed before implementation to ensure analytical performance. On the other hand, a proportion of persons who are infected with Sars-CoV-2 may not develop measurable antibodies, therefore limiting the sensitivity of any antibody test.