We invited faculty, staff, and administrators from participating districts, including each school’s adult Ci3T LT members (see introduction for description of team makeup) following IRB-approved procedures (e.g., informational videos). We provided informational letters for faculty and staff and obtained active consent from Ci3T LT members.

Participating Ci3T LT members agreed to (a) attend PL offerings with other team members (5–6 sessions per year, 2–3 hours each), with the option to provide feedback at the end of each session, (b) communicate regularly with their faculty and staff during scheduled meetings or through other mechanisms (e.g., email communication), (c) complete a confidential survey of their perceived and actual knowledge of Ci3T concepts and practices as well as their perceptions of confidence and usefulness of these same concepts and practices (Ci3T KCU; measures described below), (d) share their view about feasibility and usability of the PL experiences and resources by completing the Ci3T Usage Rating Profile (Ci3T URP) and possibly participating in a semi-structured interview using remote technology, (e) complete a demographic from, and (f) complete two brief surveys about their own professional well-being (analyzed in a separate study). We invited consented Ci3T LT members to attend Ci3T Implementation PL Series. Those who did not consent continued serving on their school’s team.

For qualitative inquiry, we invited participants who (a) served as a school-site Ci3T LT member, district leader, or implementing faculty and staff member, and (b) completed the Ci3T Usage Rating Profile (Ci3T-URP) measure to provide feedback on their PL experiences. From this pool, we used purposive sampling to invite 1–3 Leadership Team members per school by reviewing all Ci3T-URP responses to identify participants representing a range of scores on the usage subscales (i.e., feasibility, system climate, system support, understanding, acceptability). Ci3T researchers contacted each potential participant via email to assess their interest. Then we scheduled semi-structured interviews with interested participants, who consented to participate. This process resulted in six participants completing interviews in summer 2023 following Year 1, with three of these individuals participating again in summer 2024 at the close Year 2. We provided participants with $25 in appreciation for their time.

As noted in the introduction, the E-Ci3T ISD is a comprehensive PL journey featuring three core facets: (a) foundational Ci3T implementation, (b) data collection and data-informed decision-making, and (c) Ci3T district decision makers and school principals. Foundational Ci3T implementation involves communication and meeting structures (e.g., Ci3T LT meetings at each school-site, school-based collaboration and information sharing, PL communities), as well as PL (e.g., multiple parallel series led by Ci3T researchers as well as district-provided offerings). Series include the Ci3T Implementation PL Series for Ci3T LT members (a focus of this paper), Ci3T EMPOWER sessions, stand-alone topical, 2-hour sessions, for all faculty and staff and community members, and Ci3T Trainers and Coaches Calls, 1-hour monthly, for system leaders (e.g., Ci3T Coaches). Districts incorporated other PL activities such as training opportunities for district-selected Tier 2 and Tier 3 interventions. The remaining two facets of the E-Ci3T ISD support the aforementioned structures. Data collection and data-informed decision-making involves ongoing data collection adhering to district assessment schedules. This involves conducting academic and behavioral screening three times per year, collecting data from faculty and staff regarding treatment integrity and social validity of Ci3T practices two times per year, and ongoing analysis of these and other data sources to inform instruction and subsequent PL (i.e., data-informed PL). Lastly, the final facet focuses on supporting district and school administrators to build expertise and maintain alignment of district strategic goals and initiatives with Ci3T. These activities are supported through use of specific Enhancing Ci3T Modules (descriptions to follow; see OSF).

Ci3T Implementation PL (PL) Series. In this study, we focus primarily on Ci3T LT members’ experiences and learning outcomes in the context of the E-Ci3T ISD, with a particular focus on the Ci3T Implementation PL Series. This series is the component of the E-Ci3T ISD most closely aligned to experiences of Ci3T LT members, though we did not limit data collection to only the Ci3T Implementation PL Series as these school leaders’ experiences span multiple other activities (e.g., participating in monthly school-site Ci3T LT meetings, leading school-based PL). The Ci3T Implementation PL Series consists of five, 2-hr sessions hosted over the school year, with an optional summer symposium offered to assist Ci3T LT in preparing for the next year. We scheduled sessions to align with key milestones and data collection activities. For example, sessions three and five provide time to create action plans based on their recently collected treatment integrity and social validity data. Sessions were led by the Ci3T Research Team on Zoom. We hosted each Ci3T Implementation Series session twice, with one series for Kansas districts and another series for the Vermont district. Sessions were identical across the two series, both of which took place in the late afternoon after school. Districts requested this timing due to sub shortages, creating challenges with PL held during the school day.

Session format involved teams joining the Zoom meeting either individually or in a shared meeting location (e.g., school conference room). Sessions typically involve a whole-group presentation format covering timely topics and connecting participants to resources (≈ 60% of session duration) and breakout rooms in which teams review their school’s data and collaborate on action plans (≈ 40% session duration). Ci3T Research Team members coach during breakout rooms periods, rotating through rooms offering guidance and answering questions. We monitored procedural integrity of the Ci3T Implementation PL Series using detailed agendas with action items, session pacing sheets (i.e., detailed timelines of content & activities), and communication (i.e., monitoring of week-before, day-before, and follow-up communications).

Enhancing Ci3T Modules. Enhancing Ci3T Modules served as a frequent supplement to PL activities engaged in by Ci3T LT members. Some activities in the Ci3T Implementation PL Series involved reviewing content from specific modules covering topics tied into session content (e.g., low-intensity strategies to support student engagement, integrating behavioral, and social skill instruction into academic lessons). Session presenters and Ci3T Coaches encouraged Ci3T LT members to utilize these modules with their faculty and staff, both by sharing specific resources contained therein (e.g., infographics, videos) or by supporting their colleagues in reviewing the module in full. See Buckman, Lane et al. (2024, 2025) for more comprehensive descriptions of the design and content of Enhancing Ci3T Modules, and OSF Supplemental Materials (Lane et al., 2022) for a complete listing of modules.

The Ci3T-Usage Rating Profile: Ci3T LT (Ci3T-URP: LT; Briesch & Chafouleas, 2022) is a measure adapted from the Usage Rating Profile-Intervention, Revised (URP-IR; Chafouleas et al., 2011). The URP-IR was designed to measure factors related to intervention usage, and included factors across the following domains: Acceptability, Feasibility, Understanding, System Climate, System Support, and Family-School Collaboration (α = 0.67–0.95; Briesch et al., 2013). The Ci3T-URP: LT adapts items from the URP-IR to assess usage factors related to Ci3T PL experiences from the perspective of school and district leadership team members. The rating scale is consistent with the original URP-IR (i.e., 1 = strongly disagree–6 = strongly agree). Authors abridged the measure to limit completion time (~15 min) to encourage participation in fall and spring during this two-year study. The Ci3T-URP: LT included 17 items: (a) nine adapted from the URP-IR’s Acceptability subscale (summed for an overall Acceptability score; range 6–54), (b) one adapted global item for each construct, with the System Climate construct divided into two items (climate related to administrators; climate related to non-administrative faculty and staff), and (c) one additional Opportunities for Use item to determine how educators viewed opportunities to access the module in their school context. For the Acceptability score, higher scores indicate higher acceptability (α = 0.84–0.94).

Directions to the recipient provided context for raters, with variation in wording for measures delivered at the onset and end of an academic year. For example, directions at the onset of the academic year were as follows: When answering these questions, please consider the usability of the Ci3T Implementation Series and Delivery that you have received this year – which includes the 6-part implementation series, monthly meetings, time with coach, and access to online materials. Select the answer that best agrees with each statement, using the scale provided. The measure also included an open-ended item about usability factors (perceived facilitators and barriers). Specifically, directions provided included: Thinking about your setting, provide a brief narrative of the usability factors that may influence success or present a challenge to your engaging in the Ci3T implementation series and delivery. Usability factors might include personal acceptability, feasibility of the strategy, system climate in terms of support from administrators or colleagues, coaching supports, or family-school collaboration.

The Ci3T-URP: LT required ~15 min to complete, including the open-ended item. To reduce burden for Ci3T LTs, we did not administer this measure in fall 2022. This measure was completed by team members three times: spring 2023, fall 2023, and spring 2024.

To assess engagement in the E-Ci3T ISD, project staff recorded Ci3T LT members’ attendance across all PL series. For example, during Zoom-hosted Ci3T Implementation PL Series sessions, a project staff checked-in participants on an Excel spreadsheet and another staff member checked reliability during the session. This process was replicated across other series as well (i.e., Ci3T EMPOWER, Ci3T Trainers and Coaches Calls).

To track participant use of E-Ci3T ISD materials, project staff monitored participants’ access of Enhancing Ci3T Modules. Prior to accessing any modules, participants completed a one-time registration process on ci3t.org/enhance by entering their district email. Every login attempt was recorded via Qualtrics as a unique observation and was retained in this data set. Data for this study are limited to July 1, 2022 (beginning of academic Year 2022–2023) through June 20, 2024, when we exported access data from Qualtrics. For this study, we limited the sample to Ci3T LT members. We examined the number of total users and unique users for each module in this time frame. Then, we calculated an average monthly rate of total access and unique users using the number of days the module was available divided by 30.42, the average number of days in a month.

The Ci3T Knowledge, Confidence, and Use (Ci3T KCU v2.0; Lane & Oakes, 2010) survey, adapted from Borthwick-Duffy et al. (2002), examined participants’ learning throughout the E-Ci3T ISD across two academic years. Participants independently completed the Ci3T KCU survey at the onset and end of the Ci3T Implementation PL Series each academic year to rate their perceived knowledge, perceived confidence, perceived usefulness, and actual knowledge of key concepts and strategies constituting the Ci3T model of prevention. The Ci3T KCU survey consisted of a rating scale component and open-ended responses. First, participants rated their knowledge of, confidence in, and perceived usefulness of 16 core components of Ci3T using a 4-point rating scale ranging from 0 (I have no knowledge of this concept or strategy; I am not confident in my ability to use or implement this concept or strategy; I do not view this concept or strategy as useful and/or relevant in my teaching) to 3 (I have a substantial amount of knowledge about this concept or strategy; I am very confident in my ability to use or implement this concept or strategy; I view this concept or strategy as highly useful and/or relevant in my teacher). For each construct, we summed all items to create a total score. Scores ranged from 0–48, with higher numbers reflecting greater levels of perceived knowledge, confidence, or usefulness of each Ci3T concept or strategy.

Next, participants responded to the prompt: Tell us what you know about these topics in the space provided, for each of the 16 core components of Ci3T, followed by a listing of terms matched to the previously described rating scales. Each open-ended response was rated on a 4-point rating scale (see subsequent description) to gauge actual knowledge. After inspecting the data we observed many participants completed the rating scale portion of the measure but answered no open-ended items. This response pattern, which was not as prominent in previous KCU studies, created artificially low sum scores for these participants, and substantially influenced means. Rather than interpreting non-response as equivalent to having no knowledge of an item, we elected to score the actual knowledge portion for participants with ≥75% of items completed (i.e., 12 of 16). We maintained the scoring approach of previous inquiry of summing all items to create an actual knowledge composite score. See Design and Analysis for further details, and Discussions for limitations related to this change in procedure.

We note the Ci3T KCU survey used in this study (v2.0) was an updated form, adding two additional items to the version used in previous studies (e.g., Lane, Oakes et al., 2020). Additionally, in the previous version of the Ci3T KCU, participants’ actual knowledge was assessed for only a subset (n = 8) of items to limit completion time, whereas v2.0 assessed actual knowledge for all 16 items. The Ci3T KCU v2.0 required ~15 min to complete.

Coder Training. Three research team members—a senior scholar and developer of the original KCU survey, an associate professor of special education, and a doctoral student in research, measurement and evaluation—coded each open-ended response for all four time points. At the end of each academic year (Time 0 and 1 for 2022–2023; Time 3 and 4 for 2023–2024), the project director randomized open-ended responses across time points before coding. To become reliable, three coders reviewed the scoring rubric for each item, clarifying language and examples, and practiced scoring. Coders independently rated approximately five responses for each item, discussing and reconciling differences, with the senior scholar providing summative decisions. This process continued for each item until all three coders reached 90% agreement.

Interrater Reliability. Following this calibration process for each individual item, two coders independently scored all responses (100%) in the data set to check reliability. We computed Pearson correlation coefficients in SAS 9.4 for open-ended responses for each item at each time point. Correlations ranged from r = .94–1.00 at Time 0 (Fall, 2022), r = .94–1.00 at Time 1 (Spring, 2023), r = .95–.99 at Time 2 (Fall, 2023), and r = .92–.99 at Time 3 (Spring, 2024).

Participants completed a confidential Ci3T Demographic form at least once per year. Demographic items included sex, age, ethnicity and race, experience, role, certification, educational attainment, and PL history (e.g., familiarity with Ci3T-related PL). Each measure included the demographic items at the end to be able to describe the sample for each measure.

Two Project ENHANCE primary investigators developed a semi-structured interview protocol to explore Ci3T LT members’ perceptions of the usability of the key components of the E-Ci3T ISD. We used semi-structured interviewing techniques to balance covering pre-determined topics and eliciting information of particular concern or interest to the participants. Participants reflected on their experiences participating in the Ci3T Implementation Series and Delivery in the spring of each academic year. The interview protocol included seven questions asking participants about their overall impressions, what they perceived to be the most effective features, what components they accessed most often, what components they found most useful, and what suggestions they had for enhancing usefulness of the Ci3T Implementation Series and Delivery. Interviews lasted an average of 13 min (Range = 8–21) in 2023 and 17 min (Range = 8–21) in 2024. Research staff who did not lead PL series offerings conducted semi-structured interviews to reduce the likelihood of social desirability effects.

We conducted a descriptive study, more specifically a single group longitudinal study involving a sample consisting of data from Ci3T LT members who participated over both project years (2022–2024). We examined their (a) views (quantitative and qualitative data), (b) engagement, and (c) learning outcomes.

To examine views of the Ci3T Implementation Series and Delivery journey, we analyzed Ci3T-URP: LT scores over time: Spring 2023, Fall 2023, and Spring 2024. We computed descriptive statistics (e.g., M, SD) to examine their views of feasibility and usability by examining the overall acceptability composite score (sum of eight items) as well as single item measures of Use, Acceptability, Feasibility, Understanding, System Climate: Administrator, System Climate: Colleagues, System Support: Coaching, and Family-School Collaboration. We computed Hedges’ g effect sizes using the pooled standard deviation in the denominator to address unequal sample sizes. We interpreted effect sizes as follows: 0.2 = small, 0.5 = medium, and 0.8 = large effects (Cohen, 1988). We also simultaneously collected and analyzed qualitative data collected from semi-structured interviews to further examine Ci3T LTs’ views on usability. Subsequent to transcription, we integrated all participant responses to each question into a single document. Research team members then conducted reflexive thematic analysis using techniques recommended by Braun and Clarke (2021). That is, after reading through the transcripts to become familiar with underlying content, we sought to identify pieces of text within the deidentified transcripts we believed to be relevant to the concept of usability (e.g., acceptability, feasibility, understanding). We applied a label (i.e. code) to each piece of text to describe the underlying idea. Once we coded all transcripts, both researchers independently and inductively grouped codes with similar meanings into broader categories (i.e., initial themes) to answer the question of how Ci3T LT members viewed the usability of different components of the E-Ci3T ISD. After both coders reviewed and confirmed themes, they met to cross-check the identified themes, discuss any discrepancies, and refine and finalize themes. As a final step, researchers revisited the original transcripts to verify that the identified themes accurately reflected the underlying data.

For Ci3T LT member engagement in the E-Ci3T ISD, we assessed attendance of team members across PL offered. We anticipated attendance of team members would be concentrated in the Ci3T Implementation PL Series, as this event is specifically designed with these users in mind. We also monitored whether members joined other events, including Ci3T EMPOWER and Ci3T Trainer and Coaches Calls to determine whether these were accessed as a supplemental resource. We also assessed the degree to which Ci3T LT members accessed Enhancing Ci3T Modules. Use of these modules was sometimes included in Ci3T Implementation PL Sessions, and Ci3T coaches encouraged team members to use these resources throughout the year to address specific implementation needs and share embedded content (e.g., videos, infographics) with colleagues. Consistent with previous design inquiry (see Buckman, Lane et al., 2024) we calculated descriptive statistics including frequency counts, means, standard deviations, minimum and maximum values for unique users and total users per module to examine frequency and usage patterns of module access. We reported the number of unique users as well as total module access numbers, to account for repeated module use by Ci3T LTs. Given modules released at different times, we reported rate of module access (i.e., total access frequency divided by the number of months the module was available).

For each Ci3T KCU construct, we began by descriptively analyzing Ci3T LT members’ learning outcomes from four time points (Fall, 2022; Spring, 2023; Fall, 2023; & Spring, 2024) during their two years participating in the E-Ci3T ISD, including the Ci3T Implementation PL Series. We analyzed learning outcomes within a multilevel modeling framework using SAS PROC MIXED (version 9.4) for univariate analyses. In this design, for Ci3T LT member outcomes—which included perceived knowledge, perceived confidence, perceived usefulness, and actual knowledge—we nested observations (level 1 units) within Ci3T LT members (level 2 units) who were nested within schools (level 3 units). While schools were nested within districts, given the participation of only four districts, we modeled this dependency by including a fixed effect for district resulting in models with appropriate standard errors for the clustered sample. We selected District 2 (Midwest; see OSF Supplemental Materials in Lane et al., 2022 for additional details) to serve as the reference group for the district fixed effect. The basic model equation for Ci3T LT outcomes with all possible random effects was:

When interpreting the model, Y_tfs is the outcome at time t for Ci3T LT adult members (referred to as f) in school s, Time_tfs takes on value of ‘0’ for fall and then positive integers for subsequent observations indicating time point (0, 1, 2, 3), β₀₀₀ is the average fall outcome for teams, β₁₀₀ is the typical change in outcome at adjacent time points, e_tfs is a time-specific residual, u_0fs, u_1fs, are random Ci3T LT members’ effects in fall and growth in outcomes over time, η_0s is the random school effect.

We began by analyzing unconditional models with no fixed effects to determine which random effects were needed, comparing models with random Ci3T Leadership Team members intercepts only to models with random school intercepts and/or random Ci3T Leadership Team members’ slopes. After adding fixed effects for district and a linear time slope, we re-examined the random effects to determine if the addition of random slopes would improve the model which included the fixed effects. We modeled time as 0 (Fall, 2022), 1 (Spring, 2023), 2 (Fall, 2023), and 3 (Spring, 2023). We initially preregistered the model to have Time modeled in months since baseline assessment and centered on 18 months (~ two academic years) such that intercepts will reflect outcomes at the end of the trial. Yet, in collaboration with our senior statistician we shifted to Time (0, 1, 2, 3) given the relatively equal intervals between assessments, similarity in outcomes using both approaches, and greater ease of interpretation for readers. We were most interested in determining whether there is a significant positive linear slope indicative of increases in perceptions of knowledge, confidence, and usefulness over time. For each KCU construct we computed effect sizes between Time 0 and Time 3 using Hedges’ g by dividing the difference in model-estimated mean scores by the standard deviation of pre-training raw scores. We interpreted effect sizes as follows: d ≥ .20 = small; ≥ .50 = medium; ≥ .80 = large (Cohen, 1988).

Of available E-Ci3T ISD activities, Ci3T LTs mostly engaged with Ci3T Implementation PL Sessions. In Year 1 (2022–2023), attendance was most dense at the beginning of the year, with a decelerating trend in attendance as the year proceeded. Within each district, Districts 3 and 4 had relatively high attendance throughout the year, whereas District 2 had relatively low attendance. We note attendance from District 1 dropped from Session 3 to Session 4 as two schools’ teams elected to withdraw from the study between those sessions (see Method for information on withdrawals), reducing the number of participants available to attend. In some instances, specific dates were untenable for members from some districts to attend (e.g., re-scheduled district meetings for District 3 prevented a block of members from attending Session 5). In Year 2, attendance was more consistent across sessions throughout the course of the year. District 1’s number of members again dropped due to the withdrawal of another school following a change in principal, whereas, Districts 3 and 4 maintained a relatively high level of participation. Outside of team-focused Ci3T Implementation PL Sessions, team members rarely attended peripheral PL sessions (e.g., EMPOWER, Ci3T Trainers and Coaches Calls; see Table 2).

In terms of engaging with other activities as part of the E-Ci3T ISD, Ci3T LT members accessed Enhancing Ci3T Modules a total of 1,547 times between July 1, 2022–June 20, 2024. By module, average monthly access rates for individual modules ranged from 0.38 total users and 0.34 unique users (Aligning Strategic Goals and Initiatives with Ci3T) to 5.86 total and 2.70 unique users (6-step Process for Responding to Challenging Behavior). In all four metrics evaluated, four modules were the most popular: 6-step Instructional Approach, Managing Acting-Out Behavior at Tier 3, Behavior-Specific Praise, and Tier 1: Practical Strategies (Table 3). Each of these modules was accessed over 100 times by leadership team members and had 1.90 to 2.70 unique users per month on average. These four modules, along with four others—Using Data to Set Goals and Monitor Implementation; Ci3T as a Structure to Create Positive, Productive, Safe Learning Environments; Principles of Behavior Science; Universal Reinforcement System—saw the greatest repeated use with a ratio of at least two to one total access points to unique users. The most accessed modules tended to be behaviorally focused, particularly at Tier 1. Notably, modules in the Serving as a Ci3T Leader series (e.g., Effective Team Meetings, Roles and Responsibilities, Enhancing Ci3T Overview) were minimally accessed in this sample of Ci3T LT members except for Effective Onboarding Processes.

Neither the addition of a random intercept for school, –2ΔLL(1) = 2.40, p = .12, nor addition of a random slope for Ci3T LT members, –2ΔLL(1) = 0.5, p = .48 resulted in a significant improvement in model fit. Therefore, we modeled only a random intercept for participants. The two-level empty means model indicated 66% of variance between participants. The linear slope for time indicated a significant increase in perceived knowledge across the school years, F(1,189) = 57.42, p < .01. The fixed effect for district was not statistically significant, F(3,189) = 2.31, p = .08, but was retained to model variability across districts. Table 6 provides parameter estimates for the 1.82 increase in perceived knowledge across each time point and the difference between District 2 and each of the other districts at pretest. Comparison of model-estimated mean perceived knowledge scores indicated a medium magnitude difference (g = .69) from Time 0 to Time 3 (Table 5).