- Research
- Open access
- Published:
Implementation fidelity of a Brazilian drug use prevention program and its effect among adolescents: a mixed-methods study
Substance Abuse Treatment, Prevention, and Policy volume 17, Article number: 71 (2022)
Abstract
Background
Based on the US DARE-kiR, a version of the Keepin’ it REAL program, the Drug and Violence Resistance Educational Program (PROERD) is the most widely implemented Brazilian prevention program. It originates from the translation of the DARE-kiR, a version of the Keepin’ it REAL program. Previous results suggest its inefficiency in preventing drug use among Brazilian adolescents. Since kiR fidelity can impact program outcomes, this mixed-methods study evaluates the PROERD implementation fidelity and its effects on preventing drug use among adolescents.
Methods
Data from two cluster randomized controlled trials (cRCTs) with 4,030 students from 30 public schools in São Paulo (1,727 fifth graders and 2,303 seventh graders), assessed at two-time points, were analyzed quantitatively. After implementing each lesson during the cRCT, 19 PROERD instructors answered fidelity forms. The effect of PROERD fidelity on alcohol, cigarettes, marijuana, inhalant, and cocaine use (the last two only among seventh graders) in the six months prior to follow-up assessment was analyzed by logistic regressions for fifth grade and mixed effect models for seventh graders. For qualitative analysis, semi-structured interviews were conducted with PROERD instructors and investigated by thematic analysis.
Results
Quantitative analysis showed that PROERD implementation fidelity had no impact on drug use among fifth and seventh graders. Conversely, the qualitative analysis revealed important aspects that may influence implementation fidelity and consequently program effectiveness, such as adaptations made by instructors, school infrastructure, among others, besides program application.
Conclusion
PROERD requires cultural adaptation to improve its implementation in Brazilian public schools.
Introduction
Adolescence is the typical period during which initial alcohol and drug use occurs [1], impacting mental health [2] and facilitating the development of drug-related issues [3], as well as drug dependence in adulthood [4]. Hence, alcohol use in early adolescence is an important public health concern [5], contributing to the global disease burden [6].
Among Brazilian adolescents between 13 and 15 years of age, 55.5% reported alcohol consumption and 9.0% had used illicit drugs at least once in their lives, whereas 22% reported episodes of drunkenness [7]. Despite being a major public health concern, few evidence-based programs for drug use prevention have been implemented and proven efficacious in Brazilian schools [8]. The Drug and Violence Resistance Educational Program (Programa Educacional de Resistência às Drogas e à Violência – PROERD) is the most widely implemented school-based prevention curricula, reaching approximately 40% of schools [8]. A recent study showed that PROERD failed to achieve better preventive drug outcomes in the intervention group compared to the control group for both 5th and 7th grades [9]. Moreover, the program appears to negatively impact secondary outcomes, as the seventh grade curriculum seems to increase the intention to use cigarettes in the future and chances of accepting marijuana, whereas the fifth grade curriculum slightly reduces decision-making skills [10].
Its current curriculum is a Brazilian Portuguese translation of the US DARE-Keepin’it REAL (DARE-kiR) program [11], renamed as “PROERD-Caindo na Real,” and implemented by military police officers in all Brazilian states. Of wide implementation in US schools, DARE-kiR is also enforced by the military police. However, no studies have reported its impact on drug use prevention [12]. The only published paper reporting DARE-kiR findings is a quasi-experimental matched group study that looked at its impact on secondary outcomes [11]. Prior to adaptation by DARE, kiR was originally designed and developed to be implemented in 7th grade classrooms by their teachers. The original program was tested with 7th graders in Arizona-USA [13], and its culturally adapted versions were also evaluated by multiple RCTs in the US and in other countries [14,15,16,17], showing consistently positive results. As the kiR program for fifth graders proved to be ineffective, its developers recommended continued intervention only for 7th grades [18, 19]. Randomized control trial (RCT) evaluations of the kiR seventh grade curricula have shown consistently favorable results on drug use prevention in the US [13], Guatemala [14], Mexico [15], and Spain [17]. RCT evaluations of the kiR seventh grade curriculum have reported positive outcomes for discontinuing alcohol use [20] and intoxication episodes [17], as well as preventing cigarette [16, 21], marijuana [14, 16, 22] and other illicit drugs [15, 22] use. Conversely, kiR fifth grade curricula remain poorly investigated, and the only RCT reported a significantly increase in the prevalence of substance use over the 3-year period [19]. Despite the scarcity of reports on DARE-kiR implementation, some studies measured and reported findings on kiR fidelity. Marsiglia et al [23, 24] reported findings on fidelity in the context of cultural adaptation of the program to the Mexican population, as well as Cutrin to the Spanish population [17]. The only study that we are aware of that has investigated the impact of kiR implementation fidelity on programs outcomes found that the program’s impact on positive outcomes can increase based on delivery quality [25]. PROERD and DARE-kiR are a translated version of the same program, share the same theoretical model of their original version (kiR program). The main difference between the programs is that kiR is implemented by teachers, and DARE-kiR and PROERD are implemented by police officers.
Considering these unexpected results from the Brazilian version of the DARE-kiR program, we must investigate which factors might be affecting the outcomes. One factor that reportedly impacts a prevention program’s effectiveness is the excellence (or lack thereof) of its implementation [26], which can be determined by implementation fidelity, that is, the degree to which an intervention and its core components are delivered as intended by the program developers [27, 28]. A crucial aspect of implementation, that provides important information for measuring fidelity, especially when the program is not implemented by researchers, is the dosage, that is, the amount of program delivered [29]. In a review, Hill and Erickson [30] found that implementation fidelity plays an important role in program outcomes, as programs delivered with high or moderate levels of fidelity show more than double the potential to achieve positive results, contradicting the null outcome seen in our study. Despite the scarcity of reports on DARE-kiR implementation, a study investigating kiR implementation fidelity found that its impact can increase based on the quality of delivery [25].
According to Pettigrew et al. [31], teachers may adapt the program to their teaching characteristics. Such adaptation of evidence-based programs, that is, modifying the design or delivery of an intervention to address cultural and contextual specificities, can impact their results [32]. However, the relation between fidelity and adaptation remains controversial, with some researchers arguing that practices implemented with high fidelity result in better outcomes [33], while others highlight the importance of balancing between fidelity and flexibility for a successful interventions [34]. These findings point to the importance of identifying aspects that may require adaptation, as previously done among populations in Guatemala [35] and Mexico [23], as well as in rural communities in Pennsylvania and Ohio [25].
Given the relevance of implementation fidelity to better program outcomes, we hypothesized that PROERD delivered with high fidelity would have a better effect among students. Hence, this study sought to evaluate, by quantitative and qualitative methods, the PROERD implementation fidelity and its effect on drug use prevention.
Methods
Of a mixed-methods design, the study obtained quantitative data from two cluster randomized controlled trials (cRCTs) and collected qualitative data using (1) questionnaires applied to cRCTs participants, (2) fidelity forms answered by instructors (police officers) after each cRCT lesson, and (3) semi-structured interviews conducted with the instructors who delivered lessons. The interviews and qualitative analysis allowed us to answer research questions and clarify cRCT findings and quantitative results [36].
Intervention
The school-based “PROERD-Caindo na Real” program consists of 10 weekly classes (50 min each) delivered by trained police officers in the classroom environment, using student and teacher manuals. The police officer responsible for teaching the class uses the teacher’s manual, which provides information on procedures, objectives, materials needed, and tips for each lesson, including 1–3 activities. All participating officers underwent 80 h of training offered by the Military Police under the guidance of DARE America. Fifth and seventh graders were taught by the same instructor. All curricula were developed based on theories of narrative engagement [37], the principle of cultural grounding [38], social and emotional learning [39], and normative beliefs on drug use [40]. In Brazil, the program is implemented by the Military Police. Despite the lack of information about the program’s cultural adaptation process, a comparative reading of the DARE-kiR and PROERD manuals suggests that the latter is simply a translation into Brazilian Portuguese of the DARE material, lacking cultural and socio-environmental elements specific to the Brazilian context. Thus, its effectiveness needs to be assessed considering this factor.
Quantitative methods
cRCT study design
Two parallel two-arm cRCTs were conducted with 4,030 fifth and seventh graders from 30 public schools in São Paulo in 2019 to evaluate the PROERD curricula for drug use prevention. Of these, 1,727 were fifth graders enrolled in 72 classes at 28 schools, and 2,303 were seventh graders enrolled in 90 classes at 30 schools. Both intervention groups attended 10 classes taught by 19 trained police officers; the control group received no intervention. State schools in the municipality of São Paulo that offered 5th and 7th grade and had not received PROERD in the last three years were included in the randomization. The first 30 schools listed were considered the study sample and the following 29 schools were included as backup in case of refusal. In the schools that agreed to participate in the study, all 5th and 7th graders participated in the cRCT. Sample size calculation, school selection, and the randomization process were performed according to Valente & Sanchez [10].
First data source: questionnaires applied to cRCT participants
Data were collected at two-time points. Baseline assessment was conducted prior to program implementation between February and March 2019. As the Brazilian academic year usually runs from February to December, follow-up data were collected 9 months after baseline in November and December 2019. Control and intervention data were collected simultaneously. An anonymous, self-administered audio-guided questionnaire was applied to students using smartphones by researchers in the classroom without a teacher. This instrument has been employed in previous studies to evaluate school-based drug prevention programs in Brazil [41, 42]. It was designed based on the European Drug Addiction Prevention Trial (EU-DAP) questionnaire [43], translated and adapted into Brazilian Portuguese. We added a few questions from the World Health Organization (WHO) questionnaire, used in the VI Brazilian Survey on Drug Use among Students [44], and the National Survey of School Health (PENSE) questionnaire, from the Ministry of Health [45].
The outcome analyzed was the prevalence of drug use among fifth and seventh graders in the past 6 months before the follow-up assessment (yes or no), including alcohol, tobacco, marijuana, binge drinking, inhalants, and cocaine (the last two assessed only among seventh graders). During follow-up, the adolescents answered questions such as “Have you drunk alcoholic beverages in the past six months?” Binge drinking was considered as the consumption of five or more alcoholic beverages on a single occasion.
Control variables consisted of sex (male/female), age, and socioeconomic status (SES). SES was assessed by the Brazilian Association of Research Companies (ABEP) scale, which considers the schooling level of the head of the household and the goods and services used. ABEP score ranges from 1 to 100 points, graded from A (highest) to D/E (lowest) according to established cutoff points: A (45–100 points), B (29-44 points), C (17-28 points), and D/E (0–16) [46].
In each assessment, students provided a code generated from letters and numbers of their personal information, which allowed to match pre- and post-tests, ensuring anonymity and confidentiality, as used previously in drug prevention program evaluations [47]. Since some students may overreport their drug use, we included questions related to fictional drugs called “holoten” and “carpinol.” Questionnaires positive for lifetime use of these drugs were excluded from the analysis (14 and 12 questionnaires at baseline and 11 and 8 questionnaires at follow-up for fifth and seventh graders, respectively).
Second data source: fidelity forms answered by instructors
Data on implementation fidelity were collected using self-administered online questionnaires completed by the instructors after each lesson, reporting whether the scheduled activities were delivered (yes or no) and whether any activities were altered by the instructors (yes or no). The fidelity forms listed all activities planned for each program lesson, based on the teacher’s manual. The first author (J.D.G.) trained the police officers on how to fill the form. Each item had data on the percentage of activity completeness (the numerator was the number of activities delivered, and the denominator was the total number of activities planned), and percentage of alterations (the numerator was the activities instructors reported changing, and the denominator was the number of activities planned). A fidelity variable for each class was calculated as follows: fidelity = % completeness× (1-% alteration). Classes were then divided into two groups according to the level of fidelity: those that received ≥ 80% of the proposed PROERD activities were considered to have completed the program, whereas those that received ˂80% of the activities were considered to have incomplete implementation.
Since we evaluated 35 fifth grades and 47 seventh grades, and awaited 10 fidelity forms from each class (1 for each of the 10 lessons delivered), we expected a total of 350 and 470 forms for fifth and seventh grade classes, respectively. However, some instructors failed to return the forms, resulting in 47 (13%) and 83 (17,66%) fifth and seventh grade forms not delivered and considered missing. Online completion allowed researchers to mark all responses as “mandatory,” thus avoiding missing answers. Completeness and alteration calculations were proportional to the information available without missing data.
Qualitative methods
After the intervention, data were collected by semi-structured interviews conducted with all 19 PROERD instructors involved in the cRCT (19 instructors delivered the program in 30 intervention schools).
Third data source: semi-structured interviews with instructors
Qualitative data were collected by semi-structured in-depth interviews [48], with a set of previously defined questions, to which the interviewer was free to add new questions if necessary. To reduce interviewer interference and facilitate data organization, comparison, and analysis, all interviewees were asked the same basic set of questions [49].
Interviews lasted around 45 min and touched on the following topics: (i) how and why the participants became a PROERD instructor; (ii) how they perceive the effects (if any) of the program; (iii) their relationship with the school counselor and how it could affect the program’s impact; (iv) whether they considered the program training sufficient and how it influenced their work in PROERD; (vi) their opinions on the material and its content; (vii) what planned activities were well and poorly received; (viii) differences in applying the curricula for fifth and seventh graders; (ix) how PROERD demands affected other officer duties and vice versa; (x) what they would change in the program; (xi) what could be done to improve PROERD implementation. All interviews were recorded with prior consent from the interviewees. The resulting data were anonymized, transcribed verbatim, and identified by an alphanumeric code combining the letter (P), for police officer, and a random number assigned according to the order of the interviews (01, 02, 03…).
Data analysis
Quantitative analysis
Sociodemographic characteristics and drug use data underwent descriptive analysis, with categorical variables expressed as numbers and percentages, and numerical variables expressed as means and standard deviations. We then performed inferential analysis to assess the impact of PROERD implementation fidelity on reducing adolescent drug use. For seventh graders, the impact of PROERD implementation fidelity on alcohol, cigarette, marijuana, inhalants, and cocaine use and binge drinking in the past 6 months was examined by a mixed-effects linear model. This model considers variability within individuals (from baseline to follow-up) and between individuals (children nested in schools), highlighting the relations between the observed response and explanatory covariates [50, 51]. Given the extremely low prevalence of drug use among fifth graders, this impact was assessed using logistic regressions. All analyses considered 0 as control group, 1 as low fidelity, and 2 as high fidelity. Analysis was performed using STATA software version 17.0 and adjusted for sex, age, and SES, considering non-independence of the sample (children nested in school).
Data obtained from the fidelity forms on the completeness and alterations of PROERD lessons were underwent descriptive analysis, summarized as numbers and percentages.
Qualitative analysis
Data underwent thematic analysis [52] using axial coding, in which a priori analytical categories were generated based on interview guide topics, subsequently linked to other subcategories along the lines of their properties and dimensions [53]. Of the 25 codes identified, we chose to analyze 8 codes based on their possible influence on PROERD effectiveness (see Figure S1), organized into 3 topics: (1) adaptations, (2) accumulation of functions, and (3) school infrastructure. Qualitative analysis was performed using ATLAS.ti © version 7.5.4.
After initial coding performed by the first author (J.D.G.), a PhD candidate in public health with training and experience in qualitative analysis, data underwent interpretive triangulation by the second author (R.G.C.), an anthropologist and PhD candidate in collective health, who analyzed the data in parallel. Disagreements were resolved by discussions, and a second review of the interview transcripts. Topics reported and discussed in this manuscript resulted from a consensus among researchers.
Results
Quantitative analysis
Among fifth graders, 1,727 students answered the baseline questionnaire and 1,334 the follow-up questionnaire (77.24%). Among seventh graders, 2,303 students answered the baseline questionnaire and 1,739 the follow-up questionnaire (75.51%) (Fig. 1).
Table 1 summarizes the characteristics of the fifth and seventh graders who participated in the cRCT baseline assessment. At baseline, the intervention and control groups were homogenous in terms of sex, age, and SES (see Table S1 and Table S2 in the Supplementary File). The attrition analysis found no significant difference between groups and between sex. As expected, however, students who missed the 9-month follow-up showed a significantly higher prevalence of substance use at baseline, especially among 7th graders [42]. Table 2 presents the descriptive results of drug use at follow-up according to group and level of implementation fidelity. Alcohol was the most commonly used drug in both grades and groups.
As for program implementation fidelity, 37.1% of fifth grade classes received the program with high fidelity, with lesson seven (effective communication) being the most incomplete (20% completeness) and lesson three (making choices) the most altered (17.9% alteration). Similarly, 61.7% of seventh grade classes received the program with high fidelity, with lesson ten (eco map) being the most incomplete (67.5% completeness) and lesson four (assertive refusal) the most altered (21.6% alteration) (Table 3). We observed no effect of implementation fidelity on reducing drug use for either grade (see Tables 4 and 5).
Qualitative analysis
Table S3 (Supplementary File) summarizes the interviewees’ characteristics, and Annex S1 (Supplementary File) presents the semi-structured interview script applied to police officers.
Findings were classified into three dimensions—adaptations, accumulation of functions, and school infrastructure—, corresponding to the eight codes. Table 6 summarizes the respective golden quotes.
Adaptations: Of the 19 interviewed instructors, 15 reported adapting the program, usually spontaneously and based on individual perception of student’s needs (without any systematics), such as reading and writing difficulties. Instructors also chose to adapt the program when lessons were thought to flow better if applied differently from the established program. We identified four types of adaptations to PROERD curricula: alterations according to student literacy, addition of content and/or activities, changes in curricula, and exclusion of content and/or activities.
Accumulation of functions: Accumulation of two functions strongly influenced the instructor’s work. Police officers responsible for implementing PROERD do not necessarily leave their regular policing duties, which hinders the planned 10-week application, as some officers may need to report for duty during class hours or because their shifts interfere with class preparation, thus impacting the intended program delivery. Only four instructors reported being rarely assigned to other activities besides the program, which facilitated their work as PROERD instructors. Of the remaining interviewees, 11 stated being assigned to other activities or experiencing a lack of support from their superiors, as they eventually missed classes due to other duties.
School infrastructure: According to the instructors, the reality of public schools in São Paulo makes it impossible to show videos included in the program’s material, resulting in adaptations to the curriculum.
Discussion
This study analyzed the impact of PROERD implementation fidelity based on quantitative and qualitative methods. According to quantitative results, the level of implementation fidelity had no influence on the program’s ability to reduce adolescent drug use. Conversely, the qualitative analysis revealed important implementation aspects that must be considered when examining the program outcomes, such as adaptations made by the instructors, accumulation of functions (police officers who are also PROERD instructors), and school infrastructure.
Contrary to previous literature, which suggested that implementation fidelity plays an important role in program outcomes [25, 30], the present study found no influence of implementation fidelity on PROERD results. Previous studies have addressed the importance of examining the program design and contextual factors to better understand null results [54, 55]. Our null findings may be explained by the lack of cultural adaptation of the program to the Brazilian context, which is corroborated by the interviews. Moreover, compared with students in 35 other member countries of the Organization for Economic Cooperation and Development, Brazilian students’ reading performance is below average (OECD, 2019; PISA, 2016), requiring a series of adaptations to the program curricula to ensure global understanding of the proposed lessons. As PROERD appears to be simply a translation of the DARE-kiR into Brazilian Portuguese, these findings highlight the importance of a reexamination by the Military Police of São Paulo, focusing on appropriate and structured cultural adaptation [56]. In the absence of a well-designed, evidence-based cultural adaptation, instructors adapt program curricula according to their judgment, making it difficult to target core elements and consequently achieve the expected results [33].
The higher rates of low implementation fidelity suggest that instructors, when implementing evidence-based programs in schools, often encounter unpredictable situations that lead to changes in activities, such as lack of school infrastructure, affecting the level of implementation fidelity [31, 33]. Differences in implementation fidelity between fifth and seventh grades may stem from the fact that, in the state of São Paulo, PROERD is delivered mainly to fifth graders, which could lead to a greater adaptation, that is, the greater the knowledge about the curricula, the more instructors feel comfortable to adapt according to their previous experience [57]. The challenge for researchers and developers of prevention programs is therefore to define core elements that are mandatory to achieve the expected outcomes [23, 56], while allowing for some flexibility so the program can adapt its theoretical model to the local context and culture [34, 56]. As the fidelity forms used quantified only the lessons taught and ignored teaching quality [58], the quantitative analysis failed to assess important aspects of implementation. Hence, a mixed-methods approach allows one to evaluate details that would be lost when only one method is used [59].
Qualitative data revealed that instructors adapted the program curricula according to aspects such as student literacy, school infrastructure, and previous teaching experience, thus compromising implementation fidelity. Previous studies [60, 61] have also found that teachers tend to adapt curricula according to their students’ needs and vulnerabilities, such as reading proficiency and violence perpetration, which are prevalent issues in Brazilian public schools. In a study on teachers’ adaptations of kiR curricula, Miller-Day et al. [61] reinforced that, as much as such adaptations are expected, training must instruct teachers on how to adapt without changing the core aspects of each lesson.
According to the Survey on the Use of Information and Communication Technologies in Brazilian Schools [62], most public schools lack computer labs and technological infrastructure, such as laptops or tablets. As the PROERD curricula for fifth and seventh grade require audiovisual resources, school infrastructure is essential for program implementation as developed by kiR creators. Warren et al. [63] state that kiR videos are essential for interventions to achieve positive outcomes. As such, PROERD implementation fidelity cannot be expected to play an important role, since core elements such as video materials are not delivered as planned.
According to Medeiros et al. [64], teachers who deliver prevention programs feel overburdened by program activities, which require planning and classroom preparation which consequently affects their regular curricular activities. Thus, implementation of a school-based program by outside individuals is likely to be a positive experience for teachers and for the school itself. Although PROERD is delivered by police officers, the interviews show that these instructors are equally burdened by this role, as they continue to perform other policing functions. Given the importance of implementation quality [33], developers must focus on program viability to ensure that the required activities do not overburden instructors (from inside or outside the school setting) and compromise its quality [64].
In analyzing the interviews, it became evident that the null effect of implementation fidelity lies on the lack of cultural adaptation, as officers reported having to adapt the program to make implementation feasible due to student difficulties and school infrastructure.
Limitations
A first limitation to consider is that the schools selected to participate in the RCTs were located in the low-income regions of São Paulo, which experience high exposure to drug use [65]; hence, instructors working in different areas could have distinct experiences and perceptions regarding the program. Since our sample consisted of schools that did not receive any intervention in the three years prior to the study to ensure non-contamination, the data cannot be generalized to all schools in São Paulo. Another limitation concerns the fidelity forms, which were subject to information bias due to self-reporting [29]. Moreover, both the questionnaire and fidelity forms were measured by dichotomous answers, limiting the analysis. One final limitation is that we didn’t use school achievement and other important predictors of drug use, such as mental health, family environment and drug access as control variables, which have strong predictive power for substance use.
Conclusion
This study is the first to evaluate PROERD implementation fidelity in Brazil, highlighting important considerations to improve is effectiveness and sustainability. Future studies examining these parameters should employ better reliable instruments to measure implementation fidelity (dosage and quality), such as class observations. Moreover, future PROERD investigations must define which activity touches upon the core element of a lesson [61]. PROERD instructors with a clear understanding of the essential elements to be taught is key, allowing them to adapt less important parts in a structured manner according to the audience. Given the country’s territorial extent and its different realities, PROERD curricula must undergo a process of cultural adaptation. Moreover, the Military Police should incorporate the scientific findings regarding PROERD implementation in São Paulo and continue to investigate its effects not only in the city, but throughout the country.
Data Availability
Data and materials are available upon request.
Abbreviations
- ABEP:
-
Brazilian Association of Research Companies.
- cRCT:
-
Cluster Randomized Controlled Trial.
- DARE-kiR:
-
DARE-Keepin’it REAL.
- EU-DAP:
-
European Drug Addiction Prevention Trial.
- kiR:
-
Keepin’ it REAL.
- OECD:
-
Organization for Economic Cooperation and Development.
- PENSE:
-
National Survey of School Health.
- PROERD:
-
Drug and Violence Resistance Educational Program (Programa Educacional de Resistência às Drogas e à Violência).
- RCT:
-
Randomized Control Trial.
- SES:
-
Socioeconomic Status.
- WHO:
-
World Health Organization.
References
Bates MLS, Trujillo KA. Use and abuse of dissociative and psychedelic drugs in adolescence. Pharmacol Biochem Behav [Internet]. 2021;203:173129. Available from: https://doi.org/10.1016/j.pbb.2021.173129.
Wittchen HU, Fröhlich C, Behrendt S, Günther A, Rehm J, Zimmermann P, et al. Cannabis use and cannabis use disorders and their relationship to mental disorders: A 10-year prospective-longitudinal community study in adolescents. Drug Alcohol Depend. 2007;88(SUPPL.1):60–70.
Kim MJ, Mason WA, Herrenkohl TI, Catalano RF, Toumbourou JW, Hemphill SA. Influence of Early Onset of Alcohol Use on the Development of Adolescent Alcohol Problems: a Longitudinal Binational Study. Prev Sci [Internet]. 2017;18(1). Available from: https://doi.org/10.1007/s11121-016-0710-z.
Hingson RW, Heeren T, Winter MR. Age at drinking onset and alcohol dependence: Age at onset, duration, and severity. Arch Pediatr Adolesc Med. 2006;160(7):739–46.
Hall WD, Patton G, Stockings E, Weier M, Lynskey M, Morley KI, et al. Why young people’s substance use matters for global health. The Lancet Psychiatry [Internet]. 2016;3(3):265–79. Available from: https://doi.org/10.1016/S2215-0366(16)00013-4.
Whiteford HA, Degenhardt L, Rehm J, Baxter AJ, Ferrari AJ, Erskine HE, et al. Global burden of disease attributable to mental and substance use disorders: findings from the Global Burden of Disease Study 2010. Lancet. 2013 Nov;382(9904):1575–86.
IBGE - Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar 2015. Rio de Janeiro; 2016.
Pereira APD, Sanchez ZM. Characteristics of school-based drug prevention programs in Brazil. Cienc e Saude Coletiva. 2020;25(8):3131–42.
Sanchez ZM, Valente JY, Gusmões JDP, Ferreira-Junior V, Caetano SC, Cogo-Moreira H, et al. Effectiveness of a school-based substance use prevention program taught by police officers in Brazil: Two cluster randomized controlled trials of the PROERD. Int J Drug Policy [Internet]. 2021;98:103413. Available from: https://doi.org/10.1016/j.drugpo.2021.103413.
Valente JY, Sanchez ZM. Short – Term Secondary Effects of a School – Based Drug Prevention Program: Cluster – Randomized Controlled Trial of the Brazilian Version of DARE ’ s Keepin ’ it REAL. Prev Sci [Internet]. 2021;(0123456789). Available from: https://doi.org/10.1007/s11121-021-01277-w.
Day LE, Miller-Day M, Hecht ML, Fehmie D. Coming to the new D.A.R.E.: A preliminary test of the officer-taught elementary keepin’ it REAL curriculum. Addict Behav [Internet]. 2017;74(November 2016):67–73. Available from: https://doi.org/10.1016/j.addbeh.2017.05.025.
Caputi TL, Thomas McLellan A, Truth and D.A.R.E.: Is D.A.R.E.’s new Keepin’ it REAL curriculum suitable for American nationwide implementation? Drugs Educ Prev Policy [Internet]. 2017;24(1):49–57. Available from: https://doi.org/10.1080/09687637.2016.1208731.
Hecht ML, Marsiglia FF, Elek E, Wagstaff DA, Kulis S, Dustman P, et al. Culturally Grounded Substance Use Prevention: An Evaluation of the keepin’ it R.E.A.L. Curriculum Michael. Prev Sci. 2003;4(4):233–48.
Kulis SS, Marsiglia FF, Porta M, Arévalo Avalos MR, Ayers SL. Testing the keepin’ it REAL Substance Use Prevention Curriculum Among Early Adolescents in Guatemala City. Prev Sci. 2019 May;20(4):532–43.
Kulis SS, Marsiglia FF, Medina-Mora ME, Nuño-Gutiérrez BL, Corona MD, Ayers SL. Keepin’ It REAL—Mantente REAL in Mexico: a Cluster Randomized Controlled Trial of a Culturally Adapted Substance Use Prevention Curriculum for Early Adolescents. Prev Sci [Internet]. 2021;22(5):645–57. Available from: https://doi.org/10.1007/s11121-021-01217-8.
Hecht ML, Shin Y, Pettigrew J, Miller-Day M, Krieger JL. Designed Cultural Adaptation and Delivery Quality in Rural Substance Use Prevention: an Effectiveness Trial for the Keepin’ it REAL Curriculum. Prev Sci. 2018 Nov;19(8):1008–18.
Cutrín O, Kulis S, Maneiro L, MacFadden I, Navas MP, Alarcón D, et al. Effectiveness of the Mantente REAL Program for Preventing Alcohol Use in Spanish Adolescents. Psychosoc Interv [Internet]. 2021;30(3):113–22. Available from: https://doi.org/10.5093/pi2020a19.
Marsiglia FF, Kulis S, Yabiku ST, Nieri TA, Coleman E. When to Intervene: Elementary School, Middle School or Both ? Effects of keepin ’ It REAL on Substance Use Trajectories of Mexican Heritage Youth. 2011;48–62.
Elek E, Wagstaff DA, Hecht ML. Effects of the 5th and 7th Grade Enhanced Versions of the Keepin’ it Real Substance Use Prevention Curriculum. J Drug Educ. 2010;40(1):61–79.
Kulis S, Nieri T, Yabiku S, Stromwall LK, Marsiglia FF. Promoting reduced and discontinued substance use among adolescent substance users: Effectiveness of a universal prevention program. Prev Sci. 2007;8(1):35–49.
Marsiglia FF, Booth JM, Ayers SL, Nuño-Gutierrez BL, Kulis S, Hoffman S. Short-Term Effects on Substance Use of the Keepin’ It REAL Pilot Prevention Program: Linguistically Adapted for Youth in Jalisco, Mexico. Prev Sci. 2014;15(5):694–704.
Kulis SS, Garcia-Perez H, Marsiglia FF, Ayers SL. Testing a Culturally Adapted Youth Substance Use Prevention Program in a Mexican Border City: Mantente REAL. Subst Use Misuse [Internet]. 2020;56(2):245–57. Available from: https://doi.org/10.1080/10826084.2020.1858103.
Marsiglia FF, Medina-Mora ME, Gonzalvez A, Alderson G, Harthun M, Ayers S, et al. Binational Cultural Adaptation of the keepin’ it REAL Substance Use Prevention Program for Adolescents in Mexico. Prev Sci. 2019;20(7):1125–35.
Marsiglia FF, Kulis SS, Cutrín O, Medina-Mora ME, Real T, Nuño-Gutiérrez BL, et al. The feasibility, acceptability, and utility of Mantente REAL: The culturally adapted version of keepin’ it REAL for Mexico. Prev Sci [Internet]. 2022; Available from: https://doi.org/10.1007/s11121-022-01409-w.
Pettigrew J, Graham JW, Miller-Day M, Hecht ML, Krieger JL, Shin YJ. Adherence and delivery: Implementation quality and program outcomes for the 7th grade keepin’ it REAL program. Prev Sci [Internet]. 2015;16(1):90–9. Available from: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3624763/pdf/nihms412728.pdf.
Keller-margulis MA. Fidelity of implementation framework: a critical need for response to intervention models. 2012;49(4):342–52.
McDavid JC, Huse I, Hawthorn LR. Program evaluation and performance measurement: An introduction to practice. Sage Publications; 2018.
Yeaton WH, Sechrest L. Critical Dimensions in the Choice and Maintenance of Successful Treatments: Strength. Integr Eff. 1981;49(2):156–67.
Dusenbury L, Brannigan R, Falco M, Hansen WB. A review of research on fidelity of implementation: implications for drug abuse prevention in school settings. 2003;18(2):237–56.
Hill HC, Erickson A. Using Implementation Fidelity to Aid in Interpreting Program Impacts: A Brief Review. Educ Res. 2019;48(9):590–8.
Pettigrew J, Miller-Day M, Shin YJ, Hecht ML, Krieger JL, Graham JW. Describing Teacher-Student Interactions: A Qualitative Assessment of Teacher Implementation of the 7th Grade keepin’ it REAL Substance Use Intervention. Am J Community Psychol. 2013;51(1–2):43–56.
Stirman SW, Miller CJ, Toder K, Calloway A. Development of a framework and coding system for modifications and adaptations of evidence-based interventions. Implement Sci. 2013;8(1):1–12.
Harn B, Parisi D, Stoolmiller M. Balancing Fidelity With Flexibility and Fit: What DoWe Really Know About Fidelity of Implementation in Schools? 2013;79(2):181–93.
Sanetti LMH, Collier-Meek MA, Fallon LM. Fidelity with flexibility: Treatment acceptability and individualized adaptations of evidence-supported treatments. In: The Oxford handbook of treatment processes and outcomes in psychology: A multidisciplinary, biopsychosocial approach. Oxford University Press; 2016. pp. 289–308.
Kulis SS, Marsiglia FF, Porta M, Arévalo Avalos MR, Ayers SL. Testing the keepin’ it REAL Substance Use Prevention Curriculum Among Early Adolescents in Guatemala City. Prev Sci. 2018;20(4):532–43.
Creswell JW, Clark VLP. Designing and Conducting Mixed Methods Research. Thousand Oaks: Sage; 2006. 520 p.
Miller-Day M, Hecht ML. Narrative Means to Preventative Ends: A Narrative Engagement Framework for Designing Prevention Interventions. Health Commun. 2013;28(7):657–70.
Hecht ML, Krieger JLR. The principle of cultural grounding in school-based substance abuse prevention: The drug resistance strategies project. J Lang Soc Psychol. 2006;25(3):301–19.
Botvin GJ. Substance Abuse Prevention Research. Recent Developments and Future Directions. J Sch Health. 1986;56(9):369–74.
Cialdini RB, Reno RR, Kallgren CA. A focus theory of normative conduct: Recycling the concept of norms to reduce littering in public places. J Pers Soc Psychol. 1990;58(6):1015–26.
Sanchez ZM, Valente JY, Sanudo A, Pereira APD, Cruz JI, Schneider D, et al. The #Tamojunto Drug Prevention Program in Brazilian Schools: a Randomized Controlled Trial. Prev Sci. 2017;18(7):772–82.
Sanchez ZM, Valente JY, Galvão PP, Gubert FA, Melo MHS, Caetano SC, et al. A cluster randomized controlled trial evaluating the effectiveness of the school-based drug prevention program #Tamojunto2.0. Addiction. 2021;116(6):1580–92.
Faggiano F, Vigna-Taglianti F, Burkhart G, Bohrn K, Cuomo L, Gregori D, et al. The effectiveness of a school-based substance abuse prevention program: 18-Month follow-up of the EU-Dap cluster randomized controlled trial. Drug Alcohol Depend. 2010;108(1–2):56–64.
Carlini EL, de A, Noto, Carlini AR, CM de A, Locatelli, Abeid DP, LR, Amato T de C, et al. VI Levantamento Nacional sobre o Consumo de Drogas Psicotrópicas entre Estudantes do Ensino Fundamental e Médio das Redes Publica e Privada de Ensino nas 27 Capitais Brasileiras. 2010.
IBGE -. Instituto Brasileiro de Geografia e Estatística. Pesquisa Nacional de Saúde do Escolar 2015. 2016.
ABEP AB de E de P-. Critério de Classificação Econômica do Brasil [Criteria for Economic Classification in Brazil] [Internet]. 2012. Available from: http://www.abep.org/criterio-brasil.
Valente JY, Cogo-Moreira H, Swardfager W, Sanchez ZM. A latent transition analysis of a cluster randomized controlled trial for drug use prevention. J Consult Clin Psychol [Internet]. 2018 Aug;86(8):657–65. Available from: http://doi.apa.org/getdoi.cfm?doi=10.1037/ccp0000329.
Who - World Health Organization. Qualitative research for health programmes. 1994. p. 99.
Patton MQ. Qualitative research and evaluation methods. 3rd Editio. Thousand Oaks: Sage Publications; 2002.
Beroho M, Briak H, El Halimi R, Ouallali A, Boulahfa I, Mrabet R, et al. Analysis and prediction of climate forecasts in Northern Morocco: application of multilevel linear mixed effects models using R software. Heliyon [Internet]. 2020;6(10):e05094. Available from: https://doi.org/10.1016/j.heliyon.2020.e05094.
Pinheiro JC, Bates DM. Mixed-Effects Models in S and S-PLUS. Springer; 2000.
Fereday J, Muir-Cochrane E. Demonstrating Rigor Using Thematic Analysis: A Hybrid Approach of Inductive and Deductive Coding and Theme Development. Int J Qual Methods. 2006;5(1).
Strauss A, Corbin J. Bases de la investigación cualitativa - técnicas y procedimientos para desarrollar la teoría fundamentada. 2012.
Kim JS. Making Every Study Count: Learning From Replication Failure to Improve Intervention Research. Educ Res. 2019;48(9):599–607.
Jacob RT, Doolittle F, Kemple J, Somers MA. A Framework for Learning From Null Results. Educ Res. 2019;48(9):580–9.
Castro FG, Barrera M, Martinez CR. The Cultural Adaptation of Prevention Interventions: Resolving Tensions Between Fidelity and Fit. Prev Sci. 2004;5(1):41–5.
Parson S, Davis S, Scales R, Williams JB, Kear K. How and Why Teachers Adapt Their Literacy Instruction. English. 2010. p. 235.
Odom SL, Fleming K, Diamond K, Lieber J, Hanson M, Butera G, et al. Examining different forms of implementation and in early childhood curriculum research. Early Child Res Q [Internet]. 2010;25(3):314–28. Available from: https://doi.org/10.1016/j.ecresq.2010.03.001.
Creswell JW. Research Design - Qualitative, Quantitative, and Mixed Methods Approaches. 4th ed.: SAGE; 2014.
Ringwalt CL, Vincus A, Ennett S, Johnson R, Rohrbach LA. Reasons for teachers’ adaptation of substance use prevention curricula in schools with non-white student populations. Prev Sci. 2004;5(1):61–7.
Miller-Day M, Pettigrew J, Hecht ML, Shin YJ, Graham J, Krieger J. How prevention curricula are taught under real-world conditions: Types of and reasons for teacher curriculum adaptations. Health Educ. 2013;113(4):324–44.
Center BNI. TIC EDUCAÇÃO - Pesquisa Sobre o Uso das Tecnologias de Informação e Comunicação nas Escolas Brasileiras 2018 [Internet]. Pesquisa sobre o uso das tecnologias de informação e comunicação na escolas brasileiras: TIC educação 2018. Núcleo de Informação e Coordenação do Ponto BR e Comitê Gestor da Internet no Brasil. 2019. Available from: https://cetic.br/media/docs/publicacoes/216410120191105/tic_edu_2018_livro_eletronico.pdf.
Warren JR, Hecht ML, Wagstaff DA, Elek E, Ndiaye K, Dustman P, et al. Communicating Prevention: The Effects of the keepin’ it REAL Classroom Videotapes and Televised PSAs on Middle-School Students’ Substance Use. J Appl Commun Res. 2006 May;34(2):209–27.
Medeiros PFP, Cruz JI, Schneider D, Sanudo A, Sanchez ZM. Process evaluation of the implementation of the Unplugged Program for drug use prevention in Brazilian schools. Subst Abus Treat Prev Policy [Internet]. 2016;11(1):1–11. Available from: https://doi.org/10.1186/s13011-015-0047-9.
Daniel JZ, Hickman M, Macleod J, Wiles N, Lingford-Hughes A, Farrell M, et al. Is socioeconomic status in early life associated with drug use? A systematic review of the evidence. Drug Alcohol Rev. 2009;28(2):142–53.
Acknowledgements
We thank the Military Police of the State of São Paulo, the police officers, the school directors, teachers, field researchers, and especially the students who participated in this study. We would also like to thank Editage for the English language editing.
Funding
Study funded by Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) under grant number 17-22300-7 (ZMS).
Author information
Authors and Affiliations
Contributions
J.D., R.G., J.Y., I.P., and Z.M. wrote the manuscript. J.D. and R.G. prepared the figures. J.D. and J.Y. prepared the tables. All authors reviewed the manuscript.
Corresponding author
Ethics declarations
Ethics Approval
All procedures in the present study followed the ethical standards established by institutional and/or national research committees and the 1964 Helsinki Declaration and its later amendments or comparable ethical standards. Study registered in the Brazilian Ministry of Health Register of Clinical Trials (REBEC) under protocol no. 6q23nk (https://ensaiosclinicos.gov.br/rg/RBR-6q23nk). Study protocol was approved by the Research Ethics Committee of the Universidade Federal de São Paulo (No.1327/2018).
Consent to Participate
All participants gave informed consent to participate in the study.
Consent for Publication
Not applicable.
Conflict of Interest
None.
Trial Registration
Study registered in the Brazilian Ministry of Health Register of Clinical Trials (REBEC) under protocol no. 6q23nk.
Completeness
N = times when lesson activities were delivered. % = activities completed in each class.
Alteration
N = times when lesson activities were altered. % = activities altered in each class.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated in a credit line to the data.
About this article
Cite this article
Gusmoes, J.D., Garcia-Cerde, R., Valente, J.Y. et al. Implementation fidelity of a Brazilian drug use prevention program and its effect among adolescents: a mixed-methods study. Subst Abuse Treat Prev Policy 17, 71 (2022). https://doi.org/10.1186/s13011-022-00496-w
Received:
Revised:
Accepted:
Published:
DOI: https://doi.org/10.1186/s13011-022-00496-w