Assessing the reliability of the CAMS rating scale using a generalizability study

Background: An important consideration when conducting randomized controlled trials is treatment differentiation. Direct observation helps ensure that providers in different treatment groups are delivering distinct interventions. One direct observation method is the use of a measure to rate clinician performance when delivering an intervention. Aims: This generalizability study evaluated the reliability of the CAMS Rating Scale (CRS), a measure used to assess delivery of the Collaborative Assessment and Management of Suicidality (CAMS). Method: Digitally recorded tapes of clinicians delivering either CAMS or Enhanced Care-As-Usual (E-CAU) were coded using the CRS. Sessions (N = 36) were each coded by two raters, and encompassed four clinicians, four time points, and 34 unique patients across two treatment groups. A reliability coefficient (i.e., G coefficient) and the percentages of variance contributed by each component of the measurement model were obtained. Results: The CRS reliably differentiates CAMS from E-CAU, minimizes measurement error relative to expected variance sources, and continues to demonstrate high inter-rater reliability. Limitations: The absence of blind raters, a formal training protocol for the rating team, and ratings from all clinician-patient dyads at all time points was a limitation. Conclusion: The CRS is a reliable treatment differentiation measure that can play an integral role in studies evaluating CAMS.