Letter to the editor of Substance Abuse Treatment, Prevention and Policy (re: A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada)
Trevor Arnason, University of Ottawa
10 November 2014
We read the article by Jozaghi et al. (1) with interest in regards to evidence for the differential effects of supervised injection facilities on HIV and/or HCV transmission compared to existing programs that provide needle distribution. We have concerns about the scientific rigour of this study which undermines the authors’ conclusions. Key information for interpreting the results of economic evaluations such as the perspective of the analysis, the time-horizon, discounting methods and the calculation of incremental cost-effectiveness ratios, benefit-cost and cost-benefit ratios are absent in the published study. As no value is placed on health outcomes, such as in quality-adjusted life years (QALYs), it is difficult to determine the potential impact of supervised injection facilities in Ottawa and assess cost-effectiveness against commonly used thresholds.
Furthermore, there is inadequate verification and minimal rationale provided for parameter values used in the cost-benefit/effectiveness models. Notably, it appears that an inaccurate 14% baseline estimate of the needle sharing rate (s or λ) was taken, in error, from a 2012 study (2). That study estimated the rate of needle sharing in Ottawa to be 4.5% (range 3.5% to 5.6%). The authors incorrectly assert that their estimate is conservative or an underestimate. We question estimates of other parameter values for which a clear justification for their use is not provided. This includes the percentage of HCV and HIV infection from a single injection, percentage of needles not cleaned, number of needles in circulation and lifetime cost-savings per HIV and HCV case. For example, the estimate of the percentage of needles not cleaned (d) is taken from two separate studies of needle-exchange programs operating in different contexts in the 1990s. These two studies produced divergent estimates for the same parameter value, but there is no explanation of how this discrepancy was reconciled to arrive at the estimate used in the models.
Although a sensitivity analysis applied to the needle sharing rate demonstrated the utility of this approach, none was conducted for the other parameters. This is a significant omission given the dynamic and hypothetical circumstances being modelled in this analysis. Further sensitivity analyses would provide some assurance that varying parameter estimates will not alter the conclusions about the economic attractiveness of establishing supervised injection facilities in Ottawa.
Given the lack of complete information on methodology and rationale for the chosen parameter estimates, we find it impossible to draw meaningful conclusions from this study. We remain interested in high-quality research on this topic in the Ottawa setting, but studies that do not adhere to standard practices for cost-benefit/effectiveness analyses risk confusing the interpretation of results and misguiding public health policy.
Trevor Arnason, Public Health and Preventive Medicine Resident, University of Ottawa
Beate Sander, Scientist, Public Health Ontario
Doug Sider, Medical director, Communicable Disease Prevention and Control, Public Health Ontario
Vera Etches, Deputy Medical Officer of Health, Ottawa Public Health
References:
1) Jozaghi et al. A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada. Substance Abuse Treatment, Prevention, and Policy 2014 9:31
Rebuttal: Letter to the editor of Substance Abuse Treatment, Prevention and Policy (A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada).
Ehsan Jozaghi, Simon Fraser University
11 November 2014
The establishment of supervised injection facilities (SIFs) has occurred in many cities and countries around the world. Despite the existence of all these SIFs, and the scientific literature supporting their existence [1-13], there are often criticisms that emerge based more on politics than science.
Our most recent journal article regarding the establishment of SIFs in Ottawa [14], has had its scientific rigour put into question [15]. The criticisms are as follows: we did not use “standard practices for cost-benefit/effectiveness analyses”, a lack of clear justification for parameter values, a sensitivity analysis that is not extensive enough, and our choice of parameter value for the needle-sharing rate.
It is true that we did not use a more “standard” method that considers quality-adjusted life years (QALYs). In fact, our methods may be considered simple in comparison, not formally taking into account the dynamics of the social system and a score of parameters. Rather, we employed a model used to evaluate needle exchange programs that has been adapted to the context of SIFs [16-18]. These models have proven to be instructive in other contexts, generating results that have been verified with known HIV occurrence, for example, as well as calculations made by other authors [19].
We do not feel there is much need to comment on the justification for the parameter value used in these models (aside from the needle-sharing rate, discussed below) simply because we used the estimate we considered to be the most reliable or cited in the scientific literature. Similarly, we do not consider any need to discuss our sensitivity analyses; we conducted sensitivity analyses for those parameters we considered to be most critical. One must stop analyzing the data at some point and write up the results.
The most important criticism put forth by [15] is our choice of value for the needle-sharing rate. They state that we should have used 4.5%, not 14%, as identified as the needle-sharing rate by Bayoumi and Strike [20]. However, these authors have not considered the differences in the respective analyses. The 4.5% value used by Bayoumi and Strike [20] is based on needle-sharing within the past month. Our model considers an entire year. As such, the 4.5% value is completely inappropriate for our model. Bayoumi and Strike [20] state the 14% value in three separate instances in their report (pages 7, 23, & 34). We used this value because it is more representative of the time frame necessary for our model. Moreover, other studies have found that the needle sharing rate in Ottawa to range from 19 – 37%, depending on the sub-population [21, 22]. Consequently, we do consider the 14% to be conservative.
We stand by our model, but also recognize that it has its limitations; all models do because they are simplifications of highly complex systems. We encourage these authors, and others, to undertake different forms of analyses to evaluate the potential for the establishment of SIFs in Ottawa and elsewhere. Will more standard/complex mathematical models generate realistic results that will prove to be instructive for public policy? Time will tell. Only through replication, modification, and extension shall we make progress in science in order to better inform public policy.
References
Wood E, Kerr T, Small W, Li K, Marsh DC, Montaner JSG, Tyndall MW. Changes in public order after the opening of a medically supervised safer injection facility for drug users. Can Med Ass J, 2004; 171(7): 731-734.
DeBeck K, Kerr T, Bird L, Zhang R, Marsh D, Tyndall M, Montaner J, Wood E. Injection drug use cessation and use of North America’s first medically supervised safer injecting facility. Drug and Alcohol Dep, 2011; 113: 172 – 176.
Kerr T, Tyndall M, Li K, Montaner J, Wood E. Safer injection facility use and syringe sharing in injection drug users. Lancet, 2005; 366: 316–318.
Kerr T, Wood E, Palepu A, Wilson D, Schechter MT, Tyndall MW. Responding to an explosive HIV epidemic driven by frequent cocaine injection: Is there a role for safe injection facilities? J Drug Issues, 2003; 33: 579-608.
Marshall BDL, Milloy MJ, Wood E, Montaner JSG, Kerr T. Reduction in overdose mortality after the opening of North America’s first medically supervised safer injection facility: A retrospective population-based study. Lancet, 2011; 377 (9775): 1429-1437.
Small W, Wood E, Lioyd-Smith E, Tyndall M, Kerr T. Accessing care for injection-related infections through a medically supervised injection facility: A Qualitative study. Drug and Alcohol Dep, 2008; 98(1-2): 159-162.
Wood E, Tyndall MW, Stoltz J, Small W, Zhang R, O’Connell J. Safer injection education for HIV prevention within a medical supervised safer injection facility. Int J of Drug Policy, 2005; 16(4): 281-284.
Tyndall MW, Kerr T, Zhang R, King E, Montaner JG, Wood E. Attendance, drug use patterns, and referrals made from North America’s first supervised injection facility. Drug and Alcohol Dependence, 2006; 83: 193–198.
Wood E, Kerr, T, Small W, Li K, Marsh DC,Montaner, JSG. et al. (2004). Changes in public order after the opening of a medically supervised safer injection facility for illicit injection drug users. Canadian Medical Association Journal, 2004; 171(7): 731-734.
Wood E, Tyndall MW, Li K, Lioyd-Smith E, Small W, Montaner, JSG. et al. Do supervised injection facilities attract higher-risk injection drug users? American Journal of Preventive Medicine, 2005; 29(2): 126-130.
Wood E, Tyndall MW, Montaner JSG, Kerr T. Summary of findings from the evaluation of a pilot medically supervised safer injecting facility. Canadian Medical Association Journal, 2006; 175(11): 1399-1404.
Kerr T, Tyndall MW, Lai C, Montaner JSG, Wood E. Drug-related overdose within a medically supervised safer injection facility. The International Journal of Drug Policy, 2006; 17(5): 436-441.
Stoltz J-A, Wood E, Small W, Li K, Tyndall MW, Montaner JSG. et al. Changes in injecting practices associated with the use of a medically supervised safer injection facility. Journal of Public Health, 2007; 29(1): 35-39.
Jozaghi E, Reid A, Andresen A, Juneau A. A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada. Substance Abuse Treatment, Prevention, and Policy, 2014; 9 (31): 1-13.
Arnason T., Sander B, Sider D, Etches V. Letter to editor re: A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada. Substance Abuse Treatment, Prevention, and Policy, 2014.
Andresen MA, Boyd NT. A cost–benefit and cost-effectiveness analysis of Vancouver’s supervised injection facility. Int J of Drug Policy, 2010; 21: 70–76.
Andresen MA, Jozaghi E. The point of diminishing returns: An examination of expanding Vancouver’s Insite. Urban Studies, 2012; 49(16): 3531-3544.
Jozaghi E, Reid AA, Andresen MA. A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Montreal, Canada. Sub Abuse Treat, Prev, and Policy, 2013; 8(25).
Des Jarlais DC, Arasteh K, Hagan H. Evaluating Vancouver’s supervised injection facility: Data and dollars, symbols and ethics. Can Med Assoc J., 2008; 179: 1105-1106.
Leonard L, Navarro C, Birkett N: A gendered analysis of injection practices and sexual behaviours associated with high levels of HIV infection among injection drug users in the city of Ottawa 1996–2003 update: Issues for HIV prevention programming and policy development. University of Ottawa: Department of Epidemiology and Community Medicine; 2004.
Leonard L, Navarro C, Birkett N, Remis RS, The POINT Project: Department of Epidemiology and Community Medicine. University of Ottawa: Faculty of Medicine; 2005
Competing interests
Jozaghi E, Reid AA, Andresen AA, Juneau A. (Authors of the study)
Substance Abuse Treatment, Prevention, and Policy
Letter to the editor of Substance Abuse Treatment, Prevention and Policy (re: A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada)
10 November 2014
We read the article by Jozaghi et al. (1) with interest in regards to evidence for the differential effects of supervised injection facilities on HIV and/or HCV transmission compared to existing programs that provide needle distribution. We have concerns about the scientific rigour of this study which undermines the authors’ conclusions. Key information for interpreting the results of economic evaluations such as the perspective of the analysis, the time-horizon, discounting methods and the calculation of incremental cost-effectiveness ratios, benefit-cost and cost-benefit ratios are absent in the published study. As no value is placed on health outcomes, such as in quality-adjusted life years (QALYs), it is difficult to determine the potential impact of supervised injection facilities in Ottawa and assess cost-effectiveness against commonly used thresholds.
Furthermore, there is inadequate verification and minimal rationale provided for parameter values used in the cost-benefit/effectiveness models. Notably, it appears that an inaccurate 14% baseline estimate of the needle sharing rate (s or λ) was taken, in error, from a 2012 study (2). That study estimated the rate of needle sharing in Ottawa to be 4.5% (range 3.5% to 5.6%). The authors incorrectly assert that their estimate is conservative or an underestimate. We question estimates of other parameter values for which a clear justification for their use is not provided. This includes the percentage of HCV and HIV infection from a single injection, percentage of needles not cleaned, number of needles in circulation and lifetime cost-savings per HIV and HCV case. For example, the estimate of the percentage of needles not cleaned (d) is taken from two separate studies of needle-exchange programs operating in different contexts in the 1990s. These two studies produced divergent estimates for the same parameter value, but there is no explanation of how this discrepancy was reconciled to arrive at the estimate used in the models.
Although a sensitivity analysis applied to the needle sharing rate demonstrated the utility of this approach, none was conducted for the other parameters. This is a significant omission given the dynamic and hypothetical circumstances being modelled in this analysis. Further sensitivity analyses would provide some assurance that varying parameter estimates will not alter the conclusions about the economic attractiveness of establishing supervised injection facilities in Ottawa.
Given the lack of complete information on methodology and rationale for the chosen parameter estimates, we find it impossible to draw meaningful conclusions from this study. We remain interested in high-quality research on this topic in the Ottawa setting, but studies that do not adhere to standard practices for cost-benefit/effectiveness analyses risk confusing the interpretation of results and misguiding public health policy.
Trevor Arnason, Public Health and Preventive Medicine Resident, University of Ottawa
Beate Sander, Scientist, Public Health Ontario
Doug Sider, Medical director, Communicable Disease Prevention and Control, Public Health Ontario
Vera Etches, Deputy Medical Officer of Health, Ottawa Public Health
References:
1) Jozaghi et al. A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada. Substance Abuse Treatment, Prevention, and Policy 2014 9:31
2) Bayoumi AM, Strike C. Report of the Toronto and Ottawa supervised consumption assessment study. St. Michael’s Hospital; 2012. Available from: http://www.stmichaelshospital.com/pdf/research/SMH-TOSCA-report.pdf
Competing interests
No competing interests to declare.Rebuttal: Letter to the editor of Substance Abuse Treatment, Prevention and Policy (A cost-benefit/cost-effectiveness analysis of proposed supervised injection facilities in Ottawa, Canada).
11 November 2014
The establishment of supervised injection facilities (SIFs) has occurred in many cities and countries around the world. Despite the existence of all these SIFs, and the scientific literature supporting their existence [1-13], there are often criticisms that emerge based more on politics than science.
Our most recent journal article regarding the establishment of SIFs in Ottawa [14], has had its scientific rigour put into question [15]. The criticisms are as follows: we did not use “standard practices for cost-benefit/effectiveness analyses”, a lack of clear justification for parameter values, a sensitivity analysis that is not extensive enough, and our choice of parameter value for the needle-sharing rate.
It is true that we did not use a more “standard” method that considers quality-adjusted life years (QALYs). In fact, our methods may be considered simple in comparison, not formally taking into account the dynamics of the social system and a score of parameters. Rather, we employed a model used to evaluate needle exchange programs that has been adapted to the context of SIFs [16-18]. These models have proven to be instructive in other contexts, generating results that have been verified with known HIV occurrence, for example, as well as calculations made by other authors [19].
We do not feel there is much need to comment on the justification for the parameter value used in these models (aside from the needle-sharing rate, discussed below) simply because we used the estimate we considered to be the most reliable or cited in the scientific literature. Similarly, we do not consider any need to discuss our sensitivity analyses; we conducted sensitivity analyses for those parameters we considered to be most critical. One must stop analyzing the data at some point and write up the results.
The most important criticism put forth by [15] is our choice of value for the needle-sharing rate. They state that we should have used 4.5%, not 14%, as identified as the needle-sharing rate by Bayoumi and Strike [20]. However, these authors have not considered the differences in the respective analyses. The 4.5% value used by Bayoumi and Strike [20] is based on needle-sharing within the past month. Our model considers an entire year. As such, the 4.5% value is completely inappropriate for our model. Bayoumi and Strike [20] state the 14% value in three separate instances in their report (pages 7, 23, & 34). We used this value because it is more representative of the time frame necessary for our model. Moreover, other studies have found that the needle sharing rate in Ottawa to range from 19 – 37%, depending on the sub-population [21, 22]. Consequently, we do consider the 14% to be conservative.
We stand by our model, but also recognize that it has its limitations; all models do because they are simplifications of highly complex systems. We encourage these authors, and others, to undertake different forms of analyses to evaluate the potential for the establishment of SIFs in Ottawa and elsewhere. Will more standard/complex mathematical models generate realistic results that will prove to be instructive for public policy? Time will tell. Only through replication, modification, and extension shall we make progress in science in order to better inform public policy.
References
Competing interests
Jozaghi E, Reid AA, Andresen AA, Juneau A. (Authors of the study)