Development and Validation of an Assessment Tool for Competency in Critical Care Ultrasound

• DOI: 10.4300/jgme-d-14-00613.1
• Publication Date: 2015-10-01T18:42:55Z
• Authors: Paru Patrawalla; Lewis Ari Eisen; Ariel Shiloh; Brijen J. Shah; Oleksandr Savenkov; Wendy Wise; Laura Evans; Paul Mayo; Demian Szyld
• Keywords: Critical Care; Delphi Technique; Point-of-Care Systems; Reproducibility of Results; Videotape Recording; Checklist; 3. Good health; 03 medical and health sciences; 0302 clinical medicine; Education, Medical, Graduate; Humans; Clinical Competence; Educational Measurement; Prospective Studies; Monitoring, Physiologic; Quality of Health Care; Ultrasonography

ABSTRACT

Background Point-of-care ultrasound is an emerging technology in critical care medicine. Despite requirements for critical care medicine fellowship programs to demonstrate knowledge and competency in point-of-care ultrasound, tools to guide competency-based training are lacking.

Objective We describe the development and validity arguments of a competency assessment tool for critical care ultrasound.

Methods A modified Delphi method was used to develop behaviorally anchored checklists for 2 ultrasound applications: “Perform deep venous thrombosis study (DVT)” and “Qualify left ventricular function using parasternal long axis and parasternal short axis views (Echo).” One live rater and 1 video rater evaluated performance of 28 fellows. A second video rater evaluated a subset of 10 fellows. Validity evidence for content, response process, and internal consistency was assessed.

Results An expert panel finalized checklists after 2 rounds of a modified Delphi method. The DVT checklist consisted of 13 items, including 1.00 global rating step (GRS). The Echo checklist consisted of 14 items, and included 1.00 GRS for each of 2 views. Interrater reliability evaluated with a Cohen kappa between the live and video rater was 1.00 for the DVT GRS, 0.44 for the PSLA GRS, and 0.58 for the PSSA GRS. Cronbach α was 0.85 for DVT and 0.92 for Echo.

Conclusions The findings offer preliminary evidence for the validity of competency assessment tools for 2 applications of critical care ultrasound and data on live versus video raters.