I'd like to request further clarification of some of the points raised within this paper.
Readers would benefit from information explaining how the schools were selected and the decision process that assigned them to treatment or control conditions. It would be useful to see further information about the schools to adjudge whether they were comparable in nature and representative of high schools across the U.S. This information would be particularly valuable as there appear to be considerable differences in the groups’ composition with regard to gender and ethnicity.
Within the paper there is a degree of confusion regarding the number of subjects who participated in the research. In Table 2, describing the subjects’ demographics, there is a disagreement between the number of participants when totaled by age group (779) and when totaled by gender (795).
A related matter is that in Table 9, presenting the subjects’ attitude to drug use, results are given for 807 participants at the 6-month assessment. The authors state that 726 participants completed the baseline and 6-month assessments. This would seem to imply that approximately 10% of participants assessed at the 6-month follow-up were not part of the baseline assessment. If this is correct then the question is raised as to why the validity of the comparison was not protected by limiting it to those participants who were assessed at both points.
Additional explanation would be useful as to the reporting of elements of data in Tables 5 and 9, particularly items D7-D9 in Table 5 and item 22 in Table 9.
Given that the aim of the study is to test the ability of Narconon’s program to change drug use behavior it is a pity that no other student substance abuse interventions were assessed. Without this element the reader is unable to assess whether any changes detected in the study are due to the particular nature of the Narconon program or could be achieved simply by drawing students’ attentions to the hazards of drugs by other means. When the authors conclude that Narconon’s program meets a vital need it is unfortunate that we are not able to make a comparison against the efficacy of other programs and therefore assess whether a need is being uniquely met.
An additional benefit of including another student intervention in this activity would have been to enhance the perceived independence of study: given that one of the authors has held a position in Narconon and that program delivery and data collation were conducted by Narconon employees, readers would appreciate the assurance that the study has been conducted without a desired result in mind.
A pivotal element of the study that warrants further expansion is what is included in the authors’ definition of “drugs”. The program asks questions regarding illegal narcotics, but also appears to include prescription medications. In the text the dangers of having a dichotomy between “good” and “bad” drugs are discussed, but it is unclear as to what is the scope of this definition. It would be facilitate understanding of the paper if the authors could make plain as to what substances fall within the scope of their definition.
The authors conclude that the Narconon program has “thorough grounding in theory and substance abuse etiology”. It would be beneficial if readers could be made aware as to the nature of the theory upon which the program is based. The authors state that the program is based on the writings and research of L. Ron Hubbard. If this is the case, then it must be noted that the scientific validity of Mr. Hubbard’s contributions to the fields of toxicology and medicine are by no means universally accepted by specialists in these areas.
I would be grateful for the authors’ response to the above to ensure full understanding of their work.
Yours,
David Catt
Competing interests
None declared
Reply to questions posed by Mr. Catt.
Marie Cecchini, Author
22 May 2008
The topic of drug abuse prevention is not an easy one to solve. It is important to publish findings of each step along the continuum of sound research. This note answers each question raised and shows where to find the information within the publication.
As a point of understanding, we have identified ourselves with our credentials and affiliations as authors, we request that Mr Catt likewise provide his credentials and/or affiliations. We note his declaration of "no competing interests," however; in searching on the internet found critical responses posted by a “David Catt” on at least one other site owned by a group with clearly declared prejudice and competing interests. If he is this individual, these affiliations would need to be properly disclosed as they include competing interests.
This study is a naturalistic evaluation of the Narconon program, in other words we evaluated the program in a real world setting where schools select students for their course rooms. The typical drug education program is delivered to course rooms of students; we matched those conditions and looked for detectable drug use behavior change compared with controls who did not receive this program during the same period of time. As described in the paper, classrooms were nested in the statistical analysis, an important statistical consideration for groups. Any further randomization that would assign youths randomly to treatment or control groups has its own limitations and potential controversy in the prevention field.
The selection of schools is described in the first paragraph of the methods section and additionally described in the Results section, paragraph 2. “…selection of sites for ‘no treatment’ attempted to match the demographic composition at intervention sites with respect to residence state, age, and general economic group...” As schools were recruited, any imbalances between these points were corrected by recruiting schools that would provide balance. As we clearly state, “No provision was made to adjust representation by gender or potentially interesting ethnic or risk groups.” We excluded schools where subjects had previously participated in the Narconon program based on records held at Narconon.
It is not always possible to wind up with exactly matched groups. Therefore, our methods carefully tested whether there were baseline differences between education or control schools that could confound the results. As described in Table 3, even though there were differences in gender between groups, there is no detectable difference in drug use. There is literature showing that males tend to use more drugs; however, we are not aware of literature supporting gender-based differences in efforts to change drug use. Further, as described in the paper, ANCOVA controls for baseline differences.
As to Mr. Catt’s concern for the differences in the number of youths who answer each question at each sampling point, this problem is somewhat typical of all self-report surveys. A given respondent may not answer every question. In fact, consistent with student assent (which is obtained additional to parental consent), it is standard for professional surveyors to indicate that a youth need not answer a question if they are not comfortable doing so. Any review or checking for completion would violate confidentiality. Students were instructed to place their answer sheets in a privacy envelope and seal it when complete with answering the questions. Therefore, numbers of responses vary between test points. As stated in the paper, questions that were not answered were recoded as missing. As the amount is not great, it did not produce a threat to validity.
Further, as described in the paper, ANCOVA takes into account change among only those youths who completed both the baseline assessment and the six-month follow-up, controlling for baseline differences.
We are not sure what is intended by the request to design a study that would directly compare the Narconon program with other prevention programs. Such a study design would not be warranted until after the completion of this exact type of study demonstrating a given program’s ability to change drug use behavior compared with a control that existed under natural influences.
The main conclusion in this study is that the Narconon program changed drug use behavior – we have a lot more to do to understand what is causing that change. In a naturalistic setting, at any given time there exist a myriad of both pro-drug and prevention influences affecting the lives of youths. A change in behavior given these ongoing influences is important. Mr. Catt should be aware that a direct comparison trial where youth’s are selected for “treatment 1” vs. “treatment 2” is not only uncommon in prevention but has ethical issues.
Mr. Catt makes unwarranted implications regarding his concern for “perceived independence of the study.” The methods section describes that the surveys were administered by credentialed individuals and not the prevention facilitators. Further, the methods section clearly describes how “Completed answer forms were placed by each student into a security envelope, sealed, and returned to survey staff for mailing to the Principal Investigator for scanned data entry, data management, and statistical analysis.”
Additionally, Mr. Catt shows concern over what is used as a definition of a drug and the validity of other program information. The 2007 Merck Manual of Diagnosis and Therapy states: “A drug is defined by U.S. law as any substance (other than a food or device) intended for use in the diagnosis, cure, relief, treatment, or prevention of disease or intended to affect the structure or function of the body.”
Please see Table 1 for a detailed description of the program constructs. This table as well as the discussion section provide a generous number of citations; specific references help any reader evaluate for themselves the existence of supporting scientific literature.
I hope that this answers the request for further information and helps Mr. Catt better understand what is being presented.
Sincerely,
Marie Cecchini, MS
Competing interests
None declared
Further request for clarification
David Catt, N/A
11 June 2008
I would like to thank Ms Cecchini for taking the time to post a reply to my remarks about the study. Having read the author’s response I must comment that there are specific issues regarding the treatment and reporting of data that remain unanswered.
In Table 9, item D22 shows that a greater percentage of the control group feel they can easily resist pressures to take drugs than the drug education group (78.8% compared with 74.5%). The text on page 11 of the report states that “students who received the curriculum were more likely to say they could resist pressures to use drugs than those who did not receive the program”. Could I ask the authors to account for this seeming contradiction?
In Table 5, item D5 shows that the drug education group believes there is less risk in cigarette smoking than the control group since their mean score on this item is lower. In items D7-D9 relating to marijuana and alcohol usage the drug education group again have a lower mean score than the control group, but this is reported in the following column as if the opposite had been found.
It has been suggested that this anomaly may be accounted for by the direction of questions D7-D9 being reversed, so that a lower mean score on these questions indicates a higher perception of risk, but this cannot be the case.
As stated on page 3, this study makes use of the Center for Substance Abuse Prevention (CSAP) Participant Outcome Measures for Discretionary Programs (Form OMB No. 0930-0208 Expiration Date 12/31/2005), which can be inspected at the SAMHSA website. Table 5 in this study corresponds to Section G on the CSAP questionnaire. It is clear from the questionnaire that questions 5 – 9 (corresponding to Table 5,D5-D9) have exactly the same format: the respondent is asked how much risk is entailed in engaging in each activity and given options ranging upwards from “No Risk” to “Great Risk”. With this in mind it therefore seems reasonable to expect that mean scores derived from answers to these questions would be compared in the same way for each of these items. Could the authors explain how items D7-D9 are reported in an opposite manner to D5 given that these questions are identical in form?
A similar issue appears to be present in the reporting of item D12, which is opposite to that of items D10, D12 and D13. Once again the format of the questions within this section of the CSAP instrument are identical therefore it is difficult to see how this result was derived.
It would be beneficial if the authors could address these specific points to ensure that readers can be confident that the data has been handled and reported correctly.
Request for clarification
8 May 2008
I'd like to request further clarification of some of the points raised within this paper.
Readers would benefit from information explaining how the schools were selected and the decision process that assigned them to treatment or control conditions. It would be useful to see further information about the schools to adjudge whether they were comparable in nature and representative of high schools across the U.S. This information would be particularly valuable as there appear to be considerable differences in the groups’ composition with regard to gender and ethnicity.
Within the paper there is a degree of confusion regarding the number of subjects who participated in the research. In Table 2, describing the subjects’ demographics, there is a disagreement between the number of participants when totaled by age group (779) and when totaled by gender (795).
A related matter is that in Table 9, presenting the subjects’ attitude to drug use, results are given for 807 participants at the 6-month assessment. The authors state that 726 participants completed the baseline and 6-month assessments. This would seem to imply that approximately 10% of participants assessed at the 6-month follow-up were not part of the baseline assessment. If this is correct then the question is raised as to why the validity of the comparison was not protected by limiting it to those participants who were assessed at both points.
Additional explanation would be useful as to the reporting of elements of data in Tables 5 and 9, particularly items D7-D9 in Table 5 and item 22 in Table 9.
Given that the aim of the study is to test the ability of Narconon’s program to change drug use behavior it is a pity that no other student substance abuse interventions were assessed. Without this element the reader is unable to assess whether any changes detected in the study are due to the particular nature of the Narconon program or could be achieved simply by drawing students’ attentions to the hazards of drugs by other means. When the authors conclude that Narconon’s program meets a vital need it is unfortunate that we are not able to make a comparison against the efficacy of other programs and therefore assess whether a need is being uniquely met.
An additional benefit of including another student intervention in this activity would have been to enhance the perceived independence of study: given that one of the authors has held a position in Narconon and that program delivery and data collation were conducted by Narconon employees, readers would appreciate the assurance that the study has been conducted without a desired result in mind.
A pivotal element of the study that warrants further expansion is what is included in the authors’ definition of “drugs”. The program asks questions regarding illegal narcotics, but also appears to include prescription medications. In the text the dangers of having a dichotomy between “good” and “bad” drugs are discussed, but it is unclear as to what is the scope of this definition. It would be facilitate understanding of the paper if the authors could make plain as to what substances fall within the scope of their definition.
The authors conclude that the Narconon program has “thorough grounding in theory and substance abuse etiology”. It would be beneficial if readers could be made aware as to the nature of the theory upon which the program is based. The authors state that the program is based on the writings and research of L. Ron Hubbard. If this is the case, then it must be noted that the scientific validity of Mr. Hubbard’s contributions to the fields of toxicology and medicine are by no means universally accepted by specialists in these areas.
I would be grateful for the authors’ response to the above to ensure full understanding of their work.
Yours,
David Catt
Competing interests
None declared
Reply to questions posed by Mr. Catt.
22 May 2008
The topic of drug abuse prevention is not an easy one to solve. It is important to publish findings of each step along the continuum of sound research. This note answers each question raised and shows where to find the information within the publication.
As a point of understanding, we have identified ourselves with our credentials and affiliations as authors, we request that Mr Catt likewise provide his credentials and/or affiliations. We note his declaration of "no competing interests," however; in searching on the internet found critical responses posted by a “David Catt” on at least one other site owned by a group with clearly declared prejudice and competing interests. If he is this individual, these affiliations would need to be properly disclosed as they include competing interests.
This study is a naturalistic evaluation of the Narconon program, in other words we evaluated the program in a real world setting where schools select students for their course rooms. The typical drug education program is delivered to course rooms of students; we matched those conditions and looked for detectable drug use behavior change compared with controls who did not receive this program during the same period of time. As described in the paper, classrooms were nested in the statistical analysis, an important statistical consideration for groups. Any further randomization that would assign youths randomly to treatment or control groups has its own limitations and potential controversy in the prevention field.
The selection of schools is described in the first paragraph of the methods section and additionally described in the Results section, paragraph 2. “…selection of sites for ‘no treatment’ attempted to match the demographic composition at intervention sites with respect to residence state, age, and general economic group...” As schools were recruited, any imbalances between these points were corrected by recruiting schools that would provide balance. As we clearly state, “No provision was made to adjust representation by gender or potentially interesting ethnic or risk groups.” We excluded schools where subjects had previously participated in the Narconon program based on records held at Narconon.
It is not always possible to wind up with exactly matched groups. Therefore, our methods carefully tested whether there were baseline differences between education or control schools that could confound the results. As described in Table 3, even though there were differences in gender between groups, there is no detectable difference in drug use. There is literature showing that males tend to use more drugs; however, we are not aware of literature supporting gender-based differences in efforts to change drug use. Further, as described in the paper, ANCOVA controls for baseline differences.
As to Mr. Catt’s concern for the differences in the number of youths who answer each question at each sampling point, this problem is somewhat typical of all self-report surveys. A given respondent may not answer every question. In fact, consistent with student assent (which is obtained additional to parental consent), it is standard for professional surveyors to indicate that a youth need not answer a question if they are not comfortable doing so. Any review or checking for completion would violate confidentiality. Students were instructed to place their answer sheets in a privacy envelope and seal it when complete with answering the questions. Therefore, numbers of responses vary between test points. As stated in the paper, questions that were not answered were recoded as missing. As the amount is not great, it did not produce a threat to validity.
Further, as described in the paper, ANCOVA takes into account change among only those youths who completed both the baseline assessment and the six-month follow-up, controlling for baseline differences.
We are not sure what is intended by the request to design a study that would directly compare the Narconon program with other prevention programs. Such a study design would not be warranted until after the completion of this exact type of study demonstrating a given program’s ability to change drug use behavior compared with a control that existed under natural influences.
The main conclusion in this study is that the Narconon program changed drug use behavior – we have a lot more to do to understand what is causing that change. In a naturalistic setting, at any given time there exist a myriad of both pro-drug and prevention influences affecting the lives of youths. A change in behavior given these ongoing influences is important. Mr. Catt should be aware that a direct comparison trial where youth’s are selected for “treatment 1” vs. “treatment 2” is not only uncommon in prevention but has ethical issues.
Mr. Catt makes unwarranted implications regarding his concern for “perceived independence of the study.” The methods section describes that the surveys were administered by credentialed individuals and not the prevention facilitators. Further, the methods section clearly describes how “Completed answer forms were placed by each student into a security envelope, sealed, and returned to survey staff for mailing to the Principal Investigator for scanned data entry, data management, and statistical analysis.”
Additionally, Mr. Catt shows concern over what is used as a definition of a drug and the validity of other program information. The 2007 Merck Manual of Diagnosis and Therapy states: “A drug is defined by U.S. law as any substance (other than a food or device) intended for use in the diagnosis, cure, relief, treatment, or prevention of disease or intended to affect the structure or function of the body.”
Please see Table 1 for a detailed description of the program constructs. This table as well as the discussion section provide a generous number of citations; specific references help any reader evaluate for themselves the existence of supporting scientific literature.
I hope that this answers the request for further information and helps Mr. Catt better understand what is being presented.
Sincerely,
Marie Cecchini, MS
Competing interests
None declared
Further request for clarification
11 June 2008
I would like to thank Ms Cecchini for taking the time to post a reply to my remarks about the study. Having read the author’s response I must comment that there are specific issues regarding the treatment and reporting of data that remain unanswered.
In Table 9, item D22 shows that a greater percentage of the control group feel they can easily resist pressures to take drugs than the drug education group (78.8% compared with 74.5%). The text on page 11 of the report states that “students who received the curriculum were more likely to say they could resist pressures to use drugs than those who did not receive the program”. Could I ask the authors to account for this seeming contradiction?
In Table 5, item D5 shows that the drug education group believes there is less risk in cigarette smoking than the control group since their mean score on this item is lower. In items D7-D9 relating to marijuana and alcohol usage the drug education group again have a lower mean score than the control group, but this is reported in the following column as if the opposite had been found.
It has been suggested that this anomaly may be accounted for by the direction of questions D7-D9 being reversed, so that a lower mean score on these questions indicates a higher perception of risk, but this cannot be the case.
As stated on page 3, this study makes use of the Center for Substance Abuse Prevention (CSAP) Participant Outcome Measures for Discretionary Programs (Form OMB No. 0930-0208 Expiration Date 12/31/2005), which can be inspected at the SAMHSA website. Table 5 in this study corresponds to Section G on the CSAP questionnaire. It is clear from the questionnaire that questions 5 – 9 (corresponding to Table 5,D5-D9) have exactly the same format: the respondent is asked how much risk is entailed in engaging in each activity and given options ranging upwards from “No Risk” to “Great Risk”. With this in mind it therefore seems reasonable to expect that mean scores derived from answers to these questions would be compared in the same way for each of these items. Could the authors explain how items D7-D9 are reported in an opposite manner to D5 given that these questions are identical in form?
A similar issue appears to be present in the reporting of item D12, which is opposite to that of items D10, D12 and D13. Once again the format of the questions within this section of the CSAP instrument are identical therefore it is difficult to see how this result was derived.
It would be beneficial if the authors could address these specific points to ensure that readers can be confident that the data has been handled and reported correctly.
Yours,
David Catt
Competing interests
None declared