Therapist-Observer Concordance in Ratings of EBP Strategy Delivery: Challenges and Targeted Directions in Pursuing Pragmatic Measurement in Children’s Mental Health Services

• DOI: 10.1007/s10488-020-01054-x
• Publication Date: 2020-06-08T04:01:57Z
• Authors: Lauren Brookman-F...; Nicole A. Stadnick; Teresa Lind; Scott Roesch; Laura Terrones; Miya L. Barnett; Jennifer Regan; Catherine A. Kennedy; Ann F. Garland; Anna S. Lau
• Keywords: 360; Pediatric; Evidence-based practice implementation; Psychiatry; Mental Health Services; Pediatric Research Initiative; Attitude of Health Personnel; Clinical Sciences; Community mental health; 3. Good health; Psychotherapy; 03 medical and health sciences; Mental Health; Good Health and Well Being; 0302 clinical medicine; Pragmatic measures; Clinical Research; Evidence-Based Practice; Public Health and Health Services; Psychology; Humans; Mental health; Child

Pragmatic measures of therapist delivery of evidence-based practice (EBP) are critical to assessing the impact of large-scale, multiple EBP implementation efforts. As an initial step in the development of pragmatic measurement, the current study examined the concordance between therapist and observer ratings of items assessing delivery of EBP strategies considered essential for common child EBP targets. Possible EBP-, session-, and therapist-levels factors associated with concordance were also explored. Therapists and independent observers rated the extensiveness of therapist (n = 103) EBP strategy delivery in 680 community psychotherapy sessions in which six EBPs were used. Concordance between therapist- and observer-report of the extensiveness of therapist EBP strategy use was at least fair (ICC ≥ .40) for approximately half of the items. Greater therapist-observer concordance was observed in sessions where a structured EBP was delivered and in sessions where therapists reported being able to carry out planned activities. Findings highlighted conditions that may improve or hinder therapists' ability to report on their own EBP strategy delivery in a way that is consistent with independent observers. These results can help inform the development of pragmatic therapist-report measures of EBP strategy delivery and implementation efforts more broadly.