Type I Error Rates and Power of two Randomization Test Procedures for the Changing Criterion Design

Abstract

Single-case experimental design (SCED) data can be analyzed following diferent approaches. One of the frst historically proposed options is randomizations tests, benefting from the inclusion of randomization in the design: a desirable methodological feature. Randomization tests have become more feasible with the availability of computational resources, and such tests have been proposed for all major types of SCEDs: multiple-baseline, reversal/withdrawal, alternating treatments, and changing criterion designs. The focus of the current text is on the last of these, given that they have not been the subject of any previous simulation study. Specifcally, we estimate type I error rates and statistical power for two diferent randomization procedures applicable to changing criterion designs: the phase change moment randomization and the blocked alternating criterion randomization. We include diferent series lengths, number of phases, levels of autocorrelation, and random variability. The results suggest that type I error rates are generally controlled and that sufcient power can be achieved with as few as 28–30 measurements for independent data, although more measurements are needed in case of positive autocorrelation. The presence of a reversal to a previous criterion level is benefcial. R code is provided for carrying out randomization tests following the two randomization procedures.