Adherence to Emergency Department Referral Criteria in a Direct-to-Consumer Telemedicine Center

Introduction

Boosted by the COVID-19 pandemic, patients often seek direct-to-consumer telemedicine (TM) encounters as their first contact with the health care system, especially in acute situations.¹ The initial TM assessment comes along with a reduction of face-to-face evaluations.² The quality and safety of TM-discharged patients are closely related to guideline adherence.³ Virtual providers must discern red flags in each presented condition, whose presence implies immediate referral to the emergency department (ED).⁴

ED and TM centers usually have decision support algorithms that are easily available. The goal of clinical practice guidelines is to synthesize the best available evidence to support clinical decision making, which improves the quality of care, improves patient outcomes, and provides the most cost-effective care.⁵ However, the publication of guidelines does not guarantee their implementation or adherence to them by the clinician, and for this reason, protocol stewardship is critical.⁶

Increasing evidence shows that TM assessment is associated with high prescription adherence to current protocols. A recent study by our group found a high association between prescribing antimicrobials through TM and the current recommendations in the guidelines.⁷ To our knowledge, no previous studies have evaluated discharge or referral criteria after TM medical consultations.

We hypothesized that medical doctors follow current guidelines in referring their TM patients for face-to-face consultations if their center adopts stewardship protocols. This study aimed to analyze the guideline adherence of video TM encounters, defined as appropriate referral to the ED for face-to-face evaluation of acutely ill patients who spontaneously sought evaluation.

Methods

STUDY DESIGN AND PARTICIPANTS

This unicentric retrospective study was performed at the Telemedicine Center of Hospital Israelita Albert Einstein (São Paulo, Brazil). The trial protocol was approved by the institutional ethics board with registration number CAAE 58048422.9.0000.0071 and was named the Tele Red Flag study. All data can be accessed in the institutional digital records. Data were gathered and confidentially stored by TM physicians. All authors wrote the article’s initial draft, had full access to all data, and reviewed the article.

No one who was not an author contributed to the writing of the article. All analyses were performed by the TM Center coordinating the study. All authors decided to submit the article for publication and attest to the data’s integrity and accuracy and the trial’s fidelity to the preset protocol.

The study population included patients >16 years old who spontaneously sought care at the virtual ED from March 2020 to March 2022. All patients presenting with any situation were included. Exclusion criteria comprised only connection problems that did not allow medical records to be created, as these participants did not complete a medical assessment, and there are no records of their visits in the data bank.

TM CONSULTATION

The TM consultation was performed using the internet, proprietary videoconference software, and electronic health records. All remote physicians were board certified and had TM and emergency medicine training. The TM Center provided easy access to current guidelines, did periodical analyses of consultation results, and gave semiannual feedback to all medical staff in a multiparameter stewardship manner. Medical notes were recorded into the electronic medical documentation, which had a specific field to fill in the International Classification of Diseases, 10th Revision main hypotheses and referral reasons.

RED FLAGS

Twenty-six guidelines were available for immediate consultation and were structured with highlighted red flags. Physicians were trained to refer the patient to the ED if at least one red flag was identified in the encounter.

DATA EXTRACTION

The percentage of patients referred to the ED was analyzed from a specific field in the electronic medical record. A software algorithm was used to identify whether the criteria entered in the appropriate field of the electronic medical record justifying referral were present in the institutional guidelines.

STATISTICAL ANALYSIS

We used IBM-SPSS for Windows version 22.0 software for statistical calculations. The statistic was only descriptive. Categorical variables are presented as counts and percentages. There were no missing data.

Results

From a total of 232,197 patients available, 14,051 (6.05%) patients were referred to the ED. The top 50 International Classification of Diseases diagnoses are shown in Figure 1, of these, those with a specific recommendation for referral are listed in Table 1. A total of 8,829 (68.4%) referrals were backed by specific guidelines according to the ICD hypothesis listed, and 8,708 (98.6%) referrals were justified according to guideline guidance (Fig. 2).

Table 1. Pooled International Classification of Diseases, 10th Revision and Corresponding Red Flags

ICD-10	FINAL DIAGNOSIS	RED FLAGS
K12	Recurrent oral aphthae	Fever >37.8°C for >3 days • Lesions >1 cm in diameter • Lesions recurring or lasting >10 days • Presence of decompensated comorbidities, immunosuppression, and autoimmune diseases
R51, G43, G44, G44.2	Headache	Abrupt-onset headache • Worst headache of life • New headache • Headache that causes nocturnal awakenings • Severe comorbidities, immunosuppression, neurological disease or history • Neurological symptoms or neurological physical examination changes • Headache triggered by physical exertion or sexual intercourse
N30	Cystitis	Comorbidities • Age >65 years • Symptoms of systemic involvement, fever, or significant low-back pain • Pregnant women • Massive hematuria • Recurrence in <30 days
S20, S30, S40, S50, S60, S70, S80, S90, T14	Contusions	Cut and blunt wound >1 cm • Joint hypomobility • Disabling pain • Extensive hematoma • Age >65 years
I10, I15, O10, O12, O13, O14, O16, R03	Hypertension	Decompensated cardiovascular comorbidities • Neurological symptoms or visual changes • BP >180/110 mmHg measured under appropriate conditions • Cardiovascular or renal symptoms, including lower limb edema • Urinary changes such as foamy urine, oliguria, or hematuria • Pregnant women
L20, L21, L23, L24, L25, L30, L30.9	Dermatitis	Large affected area • Significant desquamation • Mucosal involvement • Comorbidities
A09, K58, K59	Diarrhea	Severe abdominal pain or disproportionate to diarrhea • Jaundice • Blood or mucus in stool • Signs of dehydration or sepsis • Uncontrollable vomiting • Recent antibiotic use
N94, N94.6	Dysmenorrhea	Menstrual delay >48 h • Pain outside the midline • Pain without improvement with anti-inflammatory drugs • Presence of significant dyspareunia or purulent discharge (risk of inflammatory disease) • New dysmenorrhea with intensity >7 • Severe comorbidities • Allergy to dipyrone + NSAIDs
K30	Dyspepsia	Persistent pain in the upper floor in the presence of high cardiovascular risk • Severe, recurrent, cramping pain in the right upper quadrant region. • Comorbidities • Age >65 years • Fever, jaundice, pallor • Recurrent vomiting • Significant decline in general condition • Melena/fecal acholia • Pregnancy
M08, M16, M17, M18, M23, M24, M25, M25.5, M75, M77, M96, S53, S83, S93, Z96	Articular pain	Significantly limited range of motion • Joint instability • Arthritis or joint effusion • Disabling pain • Pain associated with trauma • Chills, fever, constitutional symptoms.
G71, M41, M62, M79.6, M95, M96, R25.2, R93, S86	Muscle pain	A significant limitation of joint range • Disabling pain • Suspected muscle tear • Pain associated with trauma • Significant swelling • Signs of neural impairment (paresthesias, reduced strength)
B86	Scabies	Comorbidities • Fever • Pervasive affected area • Absence of improvement with treatment already instituted
J02, J03, J06	Pharyngotonsillitis	Age >75 years • Comorbidities • Dyspnea/stridor/signs of airway obstruction • Signs of retropharyngeal abscess such as bulging palate or severe unilateral hypertrophy • Torticollis • Exaggerated prostration—signs of sepsis.
M10	Gout	Fever • First episode • Arthritis of any joint other than First MTP • Pregnancy • NSAID allergy • Age >65 years • Disabling pain • Vomiting
I84, K60.2, K61, K60.3, L05, O22, O87	Hemorrhoids	Presence of fever or systemic symptoms • Comorbidities • Pain refractory to analgesia • Heavy bleeding and presence of clots • Local painful bulging or nodulation
A60, B00, B02	Herpes simplex	Concurrent systemic infection • Lesion extending beyond the perilabral region • Signs of abscess or secondary infection • Comorbidities
B34.2, B97	COVID-19
M51, M54, S32, S33, S34	Back pain	Age >65 years • Fever • Weight loss • Trauma • Comorbidities • Neurological signs in lower limbs • Significant limitation • Urinary symptoms
L55, T20–T31, T95	Burn	More than 5% of total body area affected by second degree burn • Any third degree burn • Affects face, perineum, hand or foot (except minor superficial burns) • Crosses large joints • Circumferential • Suspected inhalation or high-voltage injury
J11	Flu syndrome	The groups with the highest chance of complications and death (1) Children <5 years old (especially <2 years old); (2) elderly from >60 years (especially >65 years); (3) patients with chronic diseases: (a) respiratory: for example, asthma, COPD, interstitial diseases; (b) cardiovascular: for example, heart failure (except isolated SAH); (c) kidneys: for example, chronic kidney dysfunction or failure; (d) liver: for example, cirrhosis and liver failure; (e) hematological: for example, sickle cell anemia; (f) endocrinological: for example, diabetes (especially when using insulin); (g) neurological: epilepsy, previous stroke. (4) low immunity (e.g., by drugs, chemotherapy, HIV/AIDS); (5) pregnant women and postpartum women (up to 2 weeks postpartum); (6) nursing home residents; (7) grade 4 obesity (BMI ≥40); (8) indigenous population (Native Americans). Warning signs: (1) worsening the clinical status after initial improvement (a disease in two phases). Suggestive of bacterial pneumonia. Increased cough, sputum, yellowish discharge, dyspnea, chest pain, and return of fever; (2) persistence of high fever, dyspnea, and cyanosis, suggestive of viral pneumonia. (3) Rare complications: myositis and Guillain–Barré.
J01	Sinusitis	Age >75 years • Comorbidities • Dyspnea • Persistent fever, especially if associated with productive cough • Periorbital edema, facial erythema, proptosis, vision changes
G2.3, M43.6	Torticollis	Cervical trauma • Neurological symptoms (vertigo, gait disturbance, bladder dysfunction) • Paresthesia in shock on cervical flexion • Fever • Comorbidities • Age >65 years • Anterior neck pain • Severe headache
R05	Cough	Age >75 years • Comorbidities • Dyspnea • Presence of fever >37.8°C for >3 days (or >38.5°C once) • Productive cough >5 days (or progressive increase in sputum), wheezing
L50, L56	Urticaria	Respiratory symptoms • Concurrent angioedema • Mucosal lesions • Vomiting, abdominal pain, fainting • Age >65 years • Comorbidities
A88, H81	Vertigo	Age >65 years • Vertical nystagmus • Neurological signs • Gait instability • Any additional neurological changes • Vomiting, important limitation
N77, N77.1, B37.3	Vulvovaginitis	Presence of fever or systemic symptoms • Comorbidities • Risk of pregnancy • Dyspareunia (pain with intercourse) or significant pelvic pain • Overtly purulent discharge

Diarrhea diagnosis resulted in the highest rate of referral adherence, observed in 97.6% of cases, followed by COVID-19 (90%), headache (84.2%), and conjunctivitis (78.8%) (Fig. 3). A total of 5,222 (31.6%) patients were referred for diagnoses who did not have specific institutional guidelines warranting referral, but 5,100 (97.6%) of these were justified according to the doctor’s clinical judgment (Fig. 2).

Discussion

TM has emerged as a vital tool in providing medical care in multiple scenarios and became more popular during the COVID-19 pandemic.⁸ This study collected data from the first pandemic year and two subsequent years when mass immunization was in effect. During the first COVID-19 waves, ∼25% of our ED referrals were of suspected or confirmed COVID-19 patients.² Including all acute illnesses during the study period, our center referred only 6% of patients to the ED. These data suggest our doctors are highly resolute in their TM encounters of low-risk patients and strictly follow the scarce direct scientific evidence.⁹

With the rapid adoption of virtual care, it is crucial to ensure that the same standards of care and guidelines are maintained in TM as in traditional in-person consultations.¹⁰ Guideline adherence refers to how health care providers adhere to established clinical guidelines and protocols in their practice. This helps to mitigate the risk and ensure that the best delivery of care is related to quality and safety.¹¹ The lack of physical examination and limited access to patient information are the main challenges in a TM encounter. This can make it difficult for providers to make an accurate diagnosis and prescribe the appropriate treatment.¹²

In-person consultations frequently do not follow guideline therapy recommendations.¹³ Mounting evidence shows that TM assessment has been characterized by high guideline adherence regarding issues such as correct and safe prescriptions.¹⁴ This success is ensured through complete control of the data produced by the virtual care routine and frequent feedback and staff training. Our center avails itself of 26 internationally adapted guidelines to support assistance with TM consultations.

Although most diseases have a low risk of poor prognosis, some red flags may be present, and their recognition is a cornerstone of safety management.¹⁵ There is no previous study addressing TM quality referral based on red flag identification. A stewardship protocol is an essential element in the provision of TM services. This is a set of guidelines that govern the conduct of health care professionals when providing remote medical consultation.

Beyond prescription, administrators must check other features of the visit, such as correct diagnosis and referral. TM centers should consider developing protocols and guidelines for virtual consultations that consider the limitations of TM and the propensity of TM doctors to make ED referrals. Our center has a predominance of low-complexity patients, and the referral criteria were determined to be very sensitive.

Conclusion

TM assessment of acutely ill patients has high rates of adherence to emergency-referral guidelines. Stewardship protocol adoption provides highly accurate red flag identification, even when referring patients for nonpolicy diseases. TM should be reinforced as a strategy to provide first access to the health system associated with high resolution rates for low-complexity conditions and high adherence to guideline-directed referral to the ED.

Authors’ Contributions

Conception, planning, analysis, and interpretation of data were contributed by T.A.D.A., F.T.M., and A.A.E. The writing of the article or its critical intellectual review was carried out by T.A.D.A., F.T.M., K.F.K., and K.D.A.L. Responsibility for the final approval for publication was on R.A.M. and C.H.S.P.

Disclosure Statement

No competing financial interests exist.

Funding Information

No competing financial interests exist.

REFERENCES

• 1. Hollander JE, Carr BG. Virtually perfect? Telemedicine for Covid-19. N Engl J Med 2020;382(18):1679–1681. Crossref, Medline, Google Scholar
• 2. Accorsi TAD, Amicis K, Brígido ARD, et al. Assessment of patients with acute respiratory symptoms during the COVID-19 pandemic by telemedicine: Clinical features and impact on referral. Einstein (Sao Paulo) 2020;18:eAO6106. Crossref, Medline, Google Scholar
• 3. Hersh W, Helfand M, Wallace J, et al. A systematic review of the efficacy of telemedicine for making diagnostic and management decisions. J Telemed Telecare 2002;8(4):197–209. Crossref, Medline, Google Scholar
• 4. Sperling S, Andretta CRL, Basso J, et al. Telehealth for supporting referrals to specialized care during COVID-19. Telemed J E Health 2022;28(4):544–550. Link, Google Scholar
• 5. Lopes MACQ, Oliveira GMM, Ribeiro ALP, et al. Guideline of the Brazilian Society of Cardiology on Telemedicine in Cardiology—2019. Arq Bras Cardiol 2019;113(5):1006–1056. Medline, Google Scholar
• 6. Fernández Mondéjar E. Considerations on the low adherence to clinical practice guidelines. Med Intensiva 2017;41(5):265–266. Medline, Google Scholar
• 7. Pedrotti CHS, Accorsi TAD, De Amicis Lima K, et al. Antibiotic stewardship in direct-to-consumer telemedicine consultations leads to high adherence to best practice guidelines and a low prescription rate. Int J Infect Dis 2021;105:130–134. Crossref, Medline, Google Scholar
• 8. Kato-Lin YC, Thelen ST. Telemedicine for acute conditions during COVID-19: A Nationwide Survey Using Crowdsourcing. Telemed J E Health 2021;27(7):714–723. Link, Google Scholar
• 9. Sri-Ganeshan M, Mitra B, Soldatos G, et al.; Southeast Region Virtual Emergency Department (SERVED) Research Group. Disposition of patients utilising the virtual emergency department service in southeast region of Melbourne (SERVED-1). Emerg Med Australas 2023;35(4):553–559. Crossref, Medline, Google Scholar
• 10. Grimshaw JM, Russell IT. Effect of clinical guidelines on medical practice: A systematic review of rigorous evaluations. Lancet 1993;342(8883):1317–1322. Crossref, Medline, Google Scholar
• 11. Grol R, Grimshaw J. From best evidence to best practice: Effective implementation of change in patients’ care. Lancet 2003;362(9391):1225–1230. Crossref, Medline, Google Scholar
• 12. Sher L, Semciw A, Jessup RL, et al. Structured evaluation of virtual emergency department triage model of care: A study protocol. Emerg Med Australas 2022;34(6):907–912. Crossref, Medline, Google Scholar
• 13. Cabana MD, Rand CS, Powe NR, et al. Why don’t physicians follow clinical practice guidelines? A framework for improvement. JAMA 1999;282(15):1458–1465. Crossref, Medline, Google Scholar
• 14. Kulchar RJ, Chen K, Moon C, et al. Telemedicine, safe medication stewardship, and COVID-19: Digital transformation during a global pandemic. J Interprof Educ Pract 2022;29:100524. Crossref, Medline, Google Scholar
• 15. Alshahrani M, Katbi F, Bahamdan Y, et al. Frequency, causes, and outcomes of return visits to the emergency department within 72 hours: A Retrospective Observational Study. J Multidiscip Healthc 2020;13:2003–2010. Crossref, Medline, Google Scholar