Effects of an Online Module to Improve Preschool Teachers’ Progress Monitoring Implementation and Abilities

Design

This study employed a randomized group design across 28 classrooms, in which preschool teachers were randomly assigned to a business-as-usual (BaU) condition or a treatment condition. Randomization was conducted by assigning each teacher a random number using the RANDOM function in Microsoft Excel, and then ordering the numbers from lowest to highest. The 14 teachers with the lowest numbers were assigned to the treatment condition and the teachers with the 14 highest numbers were assigned to the BaU condition. Assignment occurred after the second wave of data collection (discussed further in the Data Collection section). The study also included 55 child participants, referred to as focal children within each classroom. Given that randomization was completed at the teacher level, the participating children were assigned to the BaU or treatment condition based on the assignment of their classroom teacher. Refer to Figure 1 for an overview of the study’s timeline and activities. The total duration of the study was 5.5 months.

Figure 1: Study timeline and activities.

Note. TEBS = Teacher Efficacy Beliefs Scale-Self; PPMM = Preschool Progress Monitoring Measure, B-PPMM = Brief Preschool Progress Monitoring Measure, ICP = Inclusive Classroom Profile, CLASS = Classroom Assessment Scoring System, BIED = Brigance Inventory of Early Development, PMPT = Progress Monitoring for Preschool Teachers.

The independent variable was access to and completion of the online training module, Progress Monitoring for Preschool Teachers (PMPT). Teachers assigned to the treatment condition completed the PMPT after the second wave of data collection. No additional coaching or follow-up support was provided as part of the independent variable. Teachers retained access to the module throughout the remainder of the study. Teachers assigned to the BaU condition did not receive access to the module until after the study was completed.

Setting, Sampling, and Participants

Seven school districts in the southeast United States were initially recruited for participation. These districts were selected given their (a) between-district variation and (b) proximity to the research team. Five of the seven districts ultimately participated in the study. Representatives for one district declined to participate given prior professional development commitments for their teachers, and representatives for one district declined to participate due to perceptions of the study being misaligned with the needs of their teachers at the time. Of the participating districts, one encompassed the second most populous city in the state and was classified as a midsize city, two districts comprised town-sized communities, and two districts were classified as rural (National Center for Education Statistics, 2021).

The research team presented information about the study to each district, and then teachers volunteered to participate. Eligible teachers needed to be employed by a school district as a certified preschool teacher and serve children 3 to 5 years old in a classroom setting. Thirty teachers were initially recruited, but two teachers decommitted prior to submitting their consent forms, and there was insufficient time for the research team to recruit replacement teachers before the start of data collection. The final sample of 28 participating teachers was comprised of 50% from the midsize city and 50% from the town and rural districts. These percentages reflect relative oversampling from the town and rural communities given that the midsize city comprised two-thirds of the preschool teachers eligible for study participation across all participating districts.

After teachers consented to participate, consent forms were sent home with two children in each classroom. These children were initially selected based on teacher nomination. After a child’s consent form was returned, the research team confirmed through a teacher-completed questionnaire that the child met the study’s inclusion criteria. Participating children needed to (a) demonstrate educational needs focused on pre-academic literacy or math skills, (b) be able to follow simple one-step directions (e.g., stand up, clap hands), and (c) be able to imitate speech and motor movements. Children who did not meet the inclusion criteria were dropped from the study, and a teacher nominated a new child for participation. Given stipulations of the Institutional Review Board approving the study, the research team was unable to determine child eligibility prior to obtaining consent to discuss the child’s educational needs with the teacher. A total of 56 children meeting the eligibility criteria were initially recruited for participation (i.e., two children in each classroom); however, one child moved schools prior to the first wave of data collection, resulting in a final sample size of 55 focal children. Based on information reported in the teacher-completed questionnaires, the research team assigned one focal child in their classroom as a math child and one focal child as a literacy child. The research team then provided the teachers with a list of literacy and math skills to select for each child, for which the teacher would monitor the child’s progress throughout the course of the study. Refer to Table 1 for a list of the literacy and math skills from which teachers could select. The skills were identified by the research team to ensure a level of standardization was present across the teachers when engaging in progress monitoring.

Experimental Conditions

Dependent Measures

Two types of dependent measures were utilized in this study, confirmatory and exploratory. Confirmatory measures were those for which the research team was seeking to identify causal relations, which informed the study’s a priori power analysis and resultant sample size. Exploratory measures were included to inform future confirmatory research studies.

Data Collection

Four waves of data collection occurred throughout the study. Wave 1 occurred immediately after eligible teachers and children consented to participate. During this wave, teachers completed a series of surveys about their background, classroom, work experience, and professional development history; and the focal children in their classroom. Surveys were distributed via Qualtrics. In addition, teachers completed the TEBS, which was also distributed via Qualtrics. Completion of the surveys and the TEBS required an approximated average of 30 min for each teacher.

Wave 2 occurred 4 to 6 weeks after teachers completed Wave 1 and prior to the assignment of teachers to conditions. During this wave, the research team scheduled observations with each participating teacher to complete the CLASS, ICP, and BIED. Observations were scheduled during center times to ensure that two cycles of the CLASS could be completed across the same activity for each teacher, as this aligned with recommendations for using the latest edition of the CLASS for program level evaluation purposes (Teachstone, 2022b). The ICP was completed at a time during the observation when a teacher could step away from their teaching responsibilities to answer the interview questions and provide the required documentation. The research team completed the Mathematics portion of the BIED with the math focal child and the Literacy portion with the literacy focal child. The BIED was completed in an area suggested by the teacher, often at a table that was not being used for center activities. If preferred by the child or recommended by the teacher, the research team administered the BIED on the floor, rather than a table. Teachers also uploaded content for the PPMM and completed the B-PPMM. Access to the B-PPMM was provided to the teachers through a password-protected web address. Videos submitted by the teachers for the PPMM were used by the research team to extract the duration of each teacher’s data collection sessions. Lastly, teachers completed a brief questionnaire asking them to report the estimated amount of time that they spent collecting and reviewing data on their focal children’s selected skill.

During Wave 3, teachers completed the B-PPMM. This occurred after teachers were assigned to conditions and the teachers in the treatment condition had completed the PMPT. No additional data were collected during this wave.

Wave 4 occurred four to six weeks after teachers in the treatment condition completed the PMPT. Teachers again uploaded content for the PPMM, completed the B-PPMM, and responded to questions about the amount of time they spent collecting and reviewing data. Teachers also completed a brief survey about any changes to their classroom during the course of the study and the amount of professional development they received throughout the study. The research team scheduled observations and readministered the CLASS, ICP, and relevant portions of the BIED. Teachers in the treatment condition also completed a researcher-created survey containing open-ended questions about the usability of the module. Lastly, teachers completed the TEBS again, but the administration was different. During this administration of the TEBS, teachers were asked to indicate both (a) their perception of how confident they felt in their teaching abilities at the start of the study and (b) at the present time. This change to the administration was done to control for potential self-enhancement bias (Deffuant et al., 2024) and response-shift bias (Ortega-Gómez et a., 2022). These biases occur when an individual completes a self-evaluation and rates themselves as being better than they actually are, which can cause a problem in evaluation studies. If an individual thinks they know a lot about a topic at the start of a study, then they will rate themselves favorably on the topic; but then as they progress through the study, they may come to realize how much they actually do not know, and so then at the end of the study they rate themselves poorly on the topic despite actually learning some things. By asking participants at the end of a study to reflect on their abilities at the start of a study, this can help control for these biases (Sajobi et al., 2018).

Analysis

Data on the pre- and post-treatment characteristics of the teachers, classrooms, and focal children were analyzed to identify differences between participants in the treatment and BaU groups. For continuous variables, and categorical variables with only two categories reported (e.g., all teachers reported identifying as either male or female), differences were assessed by entering each variable as the outcome within an OLS regression model and including a dichotomous assignment variable (0 = BaU, 1 = treatment) as the independent variable. The p-value for the assignment variable was used to gauge the significance of a difference between the groups. For categorical variables with more than two categories reported, only descriptive statistics are provided given that inferential statistics would likely yield unreliable estimates due to the low sample size of teachers and children (n ≤ 28).

Dependent variables were analyzed to evaluate the impact of the online module on confirmatory measures and to examine associations between the online module and exploratory measures. Both confirmatory and exploratory outcomes were analyzed within OLS regression models with robust standard errors applied and Benjamini-Hochberg corrections assessed for p-values to account for multiple significance tests. Post-treatment scores were entered as the dependent variable (Y) with assignment to treatment (1) or the BaU condition (0) as the primary effect of interest (β₁). In alignment with the preregistered analysis plan, pre-treatment scores on the corresponding dependent variable were entered as a covariate (β₂). We report outcomes across models that exclude (Model 1) and include (Model 2) the covariate for pre-treatment scores, as output from Model 1 provides context for interpreting the output from our primary analytical model, Model 2. The formula for Model 2, with subscript i referring to each teacher (or child depending upon the dependent variable) and ɛ as the error term, is:

$\mathit{\text{PostTx}}\_{\mathit{\text{Score}}}_i = {\beta }_0+{\beta }_1{\mathit{\text{Assignment}}}_i+ {\beta }_2\mathit{\text{PreTx}}\_{\mathit{\text{Score}}}_i+{\epsilon }_i$

Confirmatory

For teachers who completed the PMPT, they scored 29.55 percentage points (SE = 3.30, 95% CI [22.74, 36.35]) higher on the PPMM, on average, than teachers in the BaU group when working with a literacy focal child. This finding suggests that if a teacher in the BaU group scored 40.00 percentage points on the PPMM when working with a child on a literacy skill, a teacher who completed the online module would score 69.55 percentage points on the PPMM. When working with a math focal child, teachers in the treatment group scored 31.35 percentage points (SE = 4.58, 95% CI [21.94, 40.75]) higher on the PPMM, on average, than teachers in the BaU group who relied solely on their district provided professional development. When controlling for pre-treatment scores in Model 2, both the magnitude and precision of the effects were relatively unchanged for the PPMM scores pertaining to literacy and math focal children. Treatment estimates of the PMPT’s impact on teachers’ PPMM scores retained statistical significance at the 5 percent level when applying Benjamini-Hochberg corrections (p = .003).

One week after teachers completed the PMPT, they scored an average of 19.71 percentage points (SE = 3.76, 95% CI [11.99, 27.43]) higher on the B-PPMM than teachers in the BaU group. This finding suggests that if an average teacher in the BaU group scored 45.00 percentage points on the B-PPMM, then an average teacher who completed the PMPT would score 64.71 percentage points on the B-PPMM. When controlling for pre-treatment scores on the B-PPMM in Model 2, both the magnitude of the effect (B = 19.19) and its precision (SE = 3.63) remained relatively unchanged from Model 1. One month after the initial completion of the PMPT, teachers scored an average of 20.69 percentage points (SE = 4.29, 95% CI [11.87, 29.51]) higher on the B-PPMM than teachers who relied solely on their district provided professional development. Again, the magnitude and precision of the effect in Model 2 did not indicate meaningful differences from Model 1. All treatment estimates of the PMPT’s impact on teachers’ B-PPMM scores retained statistical significance at the 5 percent level when applying Benjamini-Hochberg corrections (p = .003).

Exploratory

No meaningful associations were observed across the CLASS or ICP between teachers who did and did not complete the PMPT (p > .090). Completion of the PMPT was associated with a 0.43 point (SE = .13, 95% CI [0.16, 0.70]) increase in self-efficacy as measured by TEBS ratings relative to teachers who did not complete the module. This finding suggests that if an average teacher in the BaU group reported a self-efficacy rating on the TEBS of 3.00, an average teacher who completed the PMPT would report a rating of 3.43. The association between PMPT completion and TEBS ratings retained statistical significance at the 5 percent level when applying Benjamini-Hochberg corrections (p = .013). When including a covariate for pre-treatment TEBS scores based on either (a) Wave 1 TEBS data or (b) Wave 4 TEBS data in which teachers reflected on their abilities at the start of the study, coefficient estimates were consistent (B = .39, B = .46) and retained statistical significance when applying Benjamini-Hochberg corrections (p < .008).

Descriptive differences were observed across teachers’ reporting of how much time they estimated spending on data collection and review, with teachers who completed the module reporting that they spend less time on both data collection (B = –20.79) and review (B = –22.42) than teachers who did not complete the module; however, these estimates did not achieve statistical significance (p > .088). The observational measure completed by calculating the amount of time teachers spent engaging in data collection based on video recordings, did achieve statistical significance (p = .046), with completion of the module being associated with a 90 s (SE = 43 s, 95% CI [–180 s, –2 s]) average decrease in the duration of a teachers’ data collection sessions. When controlling for pre-treatment durations, the magnitude of the association further decreased by an average of 6 s, yielding a 25% decrease in data collection duration for the treatment group relative to the BaU group. This finding did not retain statistical significance at the 5 percent level when applying Benjamini-Hochberg corrections (p = .099).

When gathering data on child outcomes, assessments could not be completed for three literacy children in the treatment group, three literacy children in the BaU group, two math children in the treatment group, and two math children in the BaU group. The assessments could not be completed due to challenges in securing child attention throughout administration of the assessments or children vocally refusing to participate which the research team considered a form of dissent. Overall attrition for the BIED Literacy measure was 25.93% and differential attrition was 9.34%. Overall attrition for the BIED Mathematics measure was 14.29% and differential attrition was 0.00%. Anecdotally, it was reported by multiple teachers that these children did not have experience with performance-based standardized assessments, such as the BIED. Authentic assessments (e.g., Teaching Strategies GOLD; Assessment, Evaluation, and Programming System) were typically used for program planning purposes in the classrooms and to support eligibility decisions along with the use of indirect assessments (e.g., teacher- and parent-rating scales). Exploratory analyses do not indicate a meaningful association between the completion of the PMPT and children’s scores on the BIED Literacy subdomain nor the BIED Mathematics subdomain (p > .200).