Estimating the Number of Self-Harm Events Not Identified by Encounter Diagnoses in Health Records ================================================================================================= * Gregory E Simon * Susan M Shortreed * Jennifer M Boggs * Gregory N Clarke * Rebecca C Rossom * Julie E Richards * Arne Beck ## ABSTRACT **Background** Public health and research assessment of self-harm or suicide attempts must sometimes rely on health system records. Previous research regarding accuracy of encounter or billing diagnoses to identify self-harm or suicide attempt has yielded mixed results. **Methods** Using data from a large pragmatic trial of outreach to prevent suicide attempt, investigators identified ICD-10-CM diagnostic codes in categories where suicide attempt or self-harm might be missed or misclassified: injuries and poisonings coded as having “undetermined intent, injuries and poisonings coded as accidental, and injuries with no coding of intent. For injury and poisoning events with these selected codes, relevant text was extracted from full-text electronic health records and rated by a blinded panel of investigators as indicating or not indicating self-harm **Results** Diagnostic codes selected for review include 43 codes for injuries and poisonings with undetermined intent, 26 codes for accidental injuries and poisonings, and 46 codes for injuries without coding of intent. Among injury and poisoning events in this sample, full-text records were available for review for 60 coded as undetermined intent, 173 coded as accidents, and 288 with no coding of intent. After review of text, raters judged 46 (77%), 152 (88%), and 275 (95%) respectively as having adequate text documentation for confident rating of self-harm intent. Among those with adequate text for confident rating, the proportions of events classified as due to self-harm were 19% for injuries and poisonings coded as undetermined intent, 6% for injuries and poisonings coded as accidents, and 15% for injuries without coding of intent. **Conclusions** In this sample of people at high risk for self-harm or suicide attempt, we low rates of misclassified self-harm events among injuries and poisonings coded as having undetermined intent, accidental intent, or no coding of intent. In this sample, inclusion of all misclassified events identified through this search process would increase the total number of self-harm events identified by approximately 10%. Public health and research assessment of self-harm or suicide attempts must sometimes rely on health system records. Population-level surveillance typically depends on insurance claims or billing records from emergency department and inpatient treatment settings. National suicide prevention efforts recommend use of records data to monitor progress as a core improvement strategy1. Evaluations of clinical strategies or care improvement efforts also rely on health records data as a primary or secondary source to evaluate clinical impact. Previous research regarding accuracy of encounter or billing diagnoses to identify self-harm or suicide attempt has yielded mixed results2. In the ICD-9 diagnostic system in use until 2015, coding of intent (i.e. accident, assault, self-inflicted) required use of a separate cause-of-injury code or E-code. We have previously reported that actual use of those cause-of-injury codes varied widely across health systems and within health systems over time3. Among studies reviewing clinical notes for encounters with recorded diagnoses of self-inflicted injury or poisoning4-6, rates of documentation of self-harm ranged between 36% and 100% (i.e. false positive error rates from 0% to 64%). Assessments of sensitivity or false negative error rates have used variable methods and have reported widely varying estimates of the proportion of self-harm events not captured by ICD-9-CM codes2. These disparate findings regarding accuracy of coding likely reflect both true differences between settings in coding practices and differences in methods used to assess accuracy. We have previously reported that review of clinical notes for injuries and poisonings receiving ICD-9-CM diagnoses of “undetermined intent” often included clear documentation of self-harm7. Not surprisingly, varying findings have led to varying recommendations regarding use of ICD-9-CM diagnostic codes to identify self-harm2, 7. Classification of injuries and poisonings changed significantly with the transition from the ICD-9-CM diagnostic system to ICD-10-CM. Coding of intent (i.e. accident, assault, or self-harm) was integrated into primary coding of all poisonings and some injuries, so that separate cause-of-injury codes are no longer required. The ICD-10-CM system also requires much more detailed specification of injury and poisoning categories. Using records data from several large integrated health systems, we have previously reported that this transition led to a marked decrease in rates of injuries and poisonings coded as having undetermined intent and a corresponding increase in rates of injuries and poisonings coded as self-harm8. We are aware of no research assessing accuracy of self-harm coding (either false positive or false negative errors) following the transition to ICD-10-CM. We describe here a systematic assessment of potential false-negative errors in ICD10-CM diagnoses of self-harm extracted from health system records. Specifically, we use clinical notes to identify injury or poisoning self-harm events not coded as self-harm. We focus on three diagnostic categories in which misclassified self-harm events would most likely occur: injuries and poisonings coded as due to accidents, injuries and poisonings coded as having undetermined intent, and injuries classified as to site and type of injury with no coding of mechanism or intent. Findings regarding the accuracy of ICD-10-CM coding for self-harm should be useful for health systems evaluating the impact of suicide prevention efforts and researchers using records data to identify self-harm events. ## METHODS This work was conducted to support a large pragmatic trial of outreach programs to prevent suicide attempt or other self-harm, conducted in four large health systems (HealthPartners and the Colorado, Northwest, and Washington regions of Kaiser Permanente). Trial methods are described in detail elsewhere9, 10 and are briefly summarized here. Between March 2015 and September 2018, adult outpatients seen in health system primary care or mental health clinics and reporting frequent “thoughts that you would be better off dead, or thoughts of hurting yourself in some way” on routinely administered depression questionnaires11 were automatically enrolled and randomly assigned to one of three intervention groups: one receiving continued usual care, one receiving usual care supplemented by an outreach and care management program, and one receiving usual care supplemented by a coach-supported online program teaching Dialectical Behavior Therapy (DBT)12 skills. For each enrolled participant, health system electronic health record and insurance claims data identified all encounters in any setting (outpatient, emergency department, or inpatient) with any injury or poisoning diagnosis during 18 months after each participant’s date of randomization, including encounters at external facilities identified by insurance claims. Injury and poisoning codes occurring in this sample were sorted into three groups for potential record review. The first group, codes indicating undetermined intent, included all injury codes in the range of Y21 through Y33 as well as codes for poisoning (T36 through T65) and asphyxiation (T71) that included a specifier for undetermined intent. The second group, c odes indicating accidental intent, included codes for poisoning (T36 through T65) and asphyxiation (T71) that included a specifier for accidental intent. The third group, Injury codes without coding of intent, included codes in the range from S00 through T32 that were not accompanied by an external cause code in the range from V00 through Y99. All ICD-10-CM codes occurring in this sample in these three groups were then rated by a panel of five trial investigators (GS, RR, AB, GC, JB) with experience in use of health records data for suicide epidemiology and prevention research. Given our previous finding that codes for undetermined intent were often accompanied by documentation of self-harm, we presumed a relatively high rate of true self-harm events in that group. Consequently, investigators were asked to identify specific codes in the undetermined intent group that were unlikely to represent self-harm so that those could be excluded from chart review. Codes identified as unlikely mechanisms of self-harm by at least three investigators were excluded from further review, with all other codes in that group included. Conversely, we presumed a relatively low rate of true self-harm events among injuries or poisonings coded as accidents or injuries without coding of intent. Consequently, investigators were asked to identify specific codes in those groups indicating common or likely mechanisms of self-harm. Those identified as common mechanisms of self-harm by at least three investigators were selected for further review, with all other codes in those groups excluded. The resulting code lists were then used to select specific injury or poisoning events for review of full-text clinical notes. Reviewable events were defined by occurrence of at least one reviewable diagnosis code (as described above) if there was no diagnosis code indicating definite self-harm occurring on the same day or any previous day (e.g. an injury or poisoning event with diagnostic codes for both self-harm and undetermined intent on the same day was not reviewed). This scheme was consistent with the study aim of measuring time to first self-harm event after randomization. A single injury or poisoning event could be selected based on diagnosis codes in two different groups (e.g. a code for poisoning of undetermined intent and a code for injury without coding of intent occurring on the same day). To preserve blinding regarding intervention group assignment, this review process was separated into two steps: extraction of relevant text by chart abstractors (not involved in intervention delivery) at each site followed by classification of that extracted and de-identified text by a separate panel of raters blinded to intervention group assignment and study site. Chart abstractors were instructed to consider records of any encounters (outpatient, emergency department, inpatient, and telephone encounters) within 14 days before or after the date of the selected diagnosis. Abstractors identified and copied any text during that time period relevant to the intent of the injury or poisoning selected for review, including nursing notes, treating clinicians’ notes, and direct quotes from patients. Abstractors were advised to specifically identify text that would clarify presence or absence of self-harm intent, including both suicidal intent and intentional self-harm not necessarily accompanied by intent to die (i.e. non-suicidal self-injury). Abstraction typically began with any encounter on the diagnosis data, extending to encounters before and after that date until clear documentation of intent was identified or until all encounters during the interval were reviewed. Abstractors redacted any information that might reveal study intervention group assignment, any information that might allow re-identification of individual patients, and any information that might identify healthcare providers or facilities. All relevant text for each event were then presented to a panel of six study investigators, with each event considered by three raters and each rater considering approximately half of all events. Raters were instructed to rate each injury/poisoning event as self-harm (a forced-choice rating of yes or no) and a separate three-level (high/medium/low) rating of confidence in that self-harm classification. Descriptive analyses of ratings examined the distribution of confidence ratings and agreement among reviewers regarding self-harm classification. Those analyses informed selection of a threshold of confidence ratings. Subsequent descriptive analyses examined the distribution of rater’s self-harm classifications limited to events with adequate ratings of confidence. Responsible institutional review boards at each participating health system reviewed and approved all trial procedures, including the chart review work described here. ## RESULTS Among all injury and poisoning diagnoses recorded in the study sample during the follow-up period, records data included 50 ICD-10 CM diagnosis codes for injuries or poisonings with undetermined intent, 94 codes for injuries or poisonings coded as accidental, and 3702 codes for injuries not accompanied by coding of intent. Among undetermined intent codes, the panel of investigators classified 43 as plausible mechanisms of self-harm (i.e. included in review) and classified 7 as unlikely mechanisms for self-harm (i.e. excluded from review). Among accidental injury/poisoning codes, the panel of investigators identified 26 as common mechanisms of self-harm (i.e. included in review). Among injury codes with no coding of intent, the panel of investigators identified 46 as common mechanisms of self-harm (i.e. included in review). The most frequent included and excluded codes in each group are shown in Table 1, and a complete list is available from the corresponding author on request. View this table: [Table 1.](http://medrxiv.org/content/early/2020/09/25/2020.09.24.20200998/T1) Table 1. Most common injury and poisoning diagnosis codes selected for inclusion and exclusion The selected list of undetermined intent codes identified 156 injury or poisoning events in this patient sample. Of those events, 70 had a definite self-harm code recorded on the same day, and 14 had a self-harm code recorded on a previous day, leaving 72 undetermined intent events to be reviewed. The selected list of accidental intent codes identified 308 injury or poisoning events in this patient sample. Of those events, 63 events had a definite self-harm code recorded on the same day, and 26 had a definite self-harm code recorded on a previous day, leaving 219 accidental events to be reviewed. The selected list of injury codes without coding of intent identified 499 injury events in this sample. Of those events, 121 had a definite self-harm code recorded on the same day, and 62 had a definite self-harm code recorded on a previous day, leaving 316 injury events without coding of intent to be reviewed. Review of clinical notes for all encounters within 14 days before or after each of those events found no relevant text (i.e. no encounters with any mention of injury or poisoning) in 12 (17%) of undetermined intent events, 46 (21%) of accidental intent events, and 28 (9%) of injury events without coding of intent. Exclusion of those events left 60 events receiving a code for undetermined intent, 173 events receiving a code for accidental intent, and 288 injury events with no coding of intent. Because some events received codes in more than one category, the total number of unique events was 508. Table 3 displays the distribution of confidence ratings and inter-rate agreement in each diagnosis group for events in with any relevant clinical text available for review. Raters expressed high confidence in ratings (summed confidence score of 6 or higher) for 77% of events coded as undetermined intent, 88% of events coded as accidents, and 95% of injuries without coding of intent. Agreement among reviewers was consistent with those confidence ratings; classification of self-harm was unanimous for the vast majority of events with high confidence ratings and a minority of events with low confidence ratings. View this table: [Table 2.](http://medrxiv.org/content/early/2020/09/25/2020.09.24.20200998/T2) Table 2. Examples of abstracted text and rater classifications View this table: [Table 3.](http://medrxiv.org/content/early/2020/09/25/2020.09.24.20200998/T3) Table 3. Distribution of rater confidence and inter-rater agreement. Confidence scores calculated as sum of three individual rater scores where 1=low confidence, 2=medium confidence, and 3=high confidence. Table 4 displays the distribution of self-harm ratings in each of the three diagnosis groups, limited to events with confidence scores of six or greater. In none of the groups, even with high confidence ratings, was the proportion of events with a majority of yes ratings higher than 19%. The total number of events with summed confidence score of six or greater and at least two reviewers classifying as self-harm was 58, including 9 events originally coded as undetermined intent, 8 originally coded as accidents, and 41 without coding of intent. View this table: [Table 4.](http://medrxiv.org/content/early/2020/09/25/2020.09.24.20200998/T4) Table 4. Distribution of self-harm ratings, limited to events with text available and summed confidence score of 6 or higher. ## DISCUSSION Findings in this sample are generally reassuring regarding false-negative errors in the use of ICD-10-CM encounter diagnoses to identify health care encounters for self-harm. In the three groups of diagnoses examined, between 5 and 15% of injury and poisoning encounters had clinical text documenting intent of self-harm. In this sample, inclusion of all additional events identified by chart review would increase the total number of self-harm events identified by approximately 10% (58 events added to approximately 550 events coded as definite self-harm). If the rates observed among encounters with adequate text to review were extrapolated to the remaining encounters without adequate text, that proportion of additional events would increase to only 12%. The low rates of self-harm we observed are notable given the selection of this sample. First, trial participants were selected for elevated risk of self-harm or suicidal behavior9. Our previous research suggests that outpatients reporting frequent thoughts of self-harm or suicide have a risk of subsequent self-harm more than ten times as high as the general population of health plan members11, 13. We would expect lower rates of self-harm in a population with a lower prior probability or expected prevalence14. Second, review was limited to diagnostic codes with higher likelihood of self-harm, eliminating injuries and poisonings unlikely to represent self-harm (among codes for undetermined intent) and selecting diagnostic codes more likely to represent self-harm (among codes for accidental intent and injuries with no coding of intent). We would expect even lower rates of self-harm events in these coding categories without this selection for codes more likely to represent self-harm. Findings in this sample regarding injury and poisoning diagnoses coded as having undetermined intent differed markedly from our previous findings regarding ICD-9-CM coding of undetermined intent7. If limited to people with recent mental health treatment, approximately 80% of injury and poisoning events with ICD-9-CM diagnoses of undetermined intent had clear documentation of self-harm. The discrepancy between these new results and our previous results regarding undetermined intent diagnoses may be explained by the marked decrease in undetermined intent diagnoses we observed with the transition from ICD-9-CM to ICD-10-CM8. If the transition to the more specific ICD-10-CM coding scheme led to increased coding of self-harm, then the events remaining in the undetermined intent category might be less likely to represent self-harm or suicide attempt. Our previous findings and these findings indicate that monitoring or comparison of self-harm rates across the transition from ICD-9-CM to ICD-10-CM would be seriously flawed. We are not aware of other data regarding prevalence of self-harm among injuries and poisonings coded as accidental or among injuries without coding of intent. Among injuries and poisonings coded as accidental, we found that only 5% were re-classified as self-harm after chart abstraction and blinded classification. Consequently, this group contributed only a small number of additional events. Among injuries with no coding of intent, we found that almost 15% were re-classified as self-harm. Given the relatively large number of injuries without coding of intent, this group contributed most of the additional events identified by chart review. We should emphasize that this work does not consider many sources of false-negative error in use of encounter diagnoses to identify self-harm. First, people experiencing self-harm might not seek medical care, so no encounter would appear in health system records. Second, treating providers might not record an injury or poisoning diagnosis. Third, some self-harm events might receive non-specific diagnoses not included in our record review process. Because we aimed to identify the first self-harm event among participants in this trial, we did not review injuries and poisonings occurring after a clear diagnosis of self-harm. We might observe different rates of misclassified self-harm events if we considered all potential events for any individual patient. We should also caution that our findings may not generalize to other health systems. While the ICD-10-CM taxonomy now applies across the US, recording of encounter or billing diagnoses is certainly subject to local influences3. Diagnoses from outpatient and emergency department encounters, typically recorded by treating clinicians, are influenced by how possible codes are presented and prioritized within the facility or health system EHR environment. Diagnoses from inpatient encounters, sometimes recorded by treating clinicians and sometimes by designated coding consultants, are influenced by local policies, practices, and EHR environments. Allowing for uncertainty regarding generalizability of these findings, we suggest the following implications. Researchers aiming to use health system data for population-based research, such as development of suicide risk prediction models, should not generally include diagnoses of undetermined intent in definitions of self-harm. Health systems aiming to measure or monitor population-level rates of self-harm should recognize that ICD-10-CM diagnoses of self-harm will modestly under-estimate true incidence rates. View this table: [Supplementary Table A.](http://medrxiv.org/content/early/2020/09/25/2020.09.24.20200998/T5) Supplementary Table A. Number of times the same event was assigned diagnosis codes from multiple code categories on the same day. View this table: [Supplementary Table B.](http://medrxiv.org/content/early/2020/09/25/2020.09.24.20200998/T6) Supplementary Table B. Distribution of events receiving diagnosis codes in more than one review category, limited to events with relevant text available for review. ## Data Availability We are not able to share clinical text from patient records. ## Footnotes * Supported by National Institute of Mental Health Cooperative Agreement UH3 MH007755 * Received September 24, 2020. * Revision received September 24, 2020. * Accepted September 25, 2020. * © 2020, Posted by Cold Spring Harbor Laboratory The copyright holder for this pre-print is the author. All rights reserved. The material may not be redistributed, re-used or adapted without the author's permission. ## References 1. 1.Hogan MF, Grumet JG. Suicide Prevention: An Emerging Priority For Health Care. Health Aff (Millwood). 2016;35(6):1084–90. [Abstract/FREE Full Text](http://medrxiv.org/lookup/ijlink/YTozOntzOjQ6InBhdGgiO3M6MTQ6Ii9sb29rdXAvaWpsaW5rIjtzOjU6InF1ZXJ5IjthOjQ6e3M6ODoibGlua1R5cGUiO3M6NDoiQUJTVCI7czoxMToiam91cm5hbENvZGUiO3M6OToiaGVhbHRoYWZmIjtzOjU6InJlc2lkIjtzOjk6IjM1LzYvMTA4NCI7czo0OiJhdG9tIjtzOjUwOiIvbWVkcnhpdi9lYXJseS8yMDIwLzA5LzI1LzIwMjAuMDkuMjQuMjAyMDA5OTguYXRvbSI7fXM6ODoiZnJhZ21lbnQiO3M6MDoiIjt9) 2. 2.Walkup JT, Townsend L, Crystal S, Olfson M. A systematic review of validated methods for identifying suicide or suicidal ideation using administrative or claims data. Pharmacoepidemiol Drug Saf. 2012;21 Suppl 1:174–82. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/pds.2335&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=22262604&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) 3. 3.Lu CY, Stewart C, Ahmed AT, Ahmedani BK, Coleman K, Copeland LA, Hunkeler EM, Lakoma MD, Madden JM, Penfold RB, Rusinak D, Zhang F, Soumerai SB. How complete are E-codes in commercial plan claims databases? Pharmacoepidemiol Drug Saf. 2014;23(2):218-20. PMCID: pmid:PMC4100538. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1002/pds.3551&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24453020&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) 4. 4.Iribarren C, Sidney S, Jacobs DR Jr..,, Weisner C. Hospitalization for suicide attempt and completed suicide: epidemiological features in a managed care population. Soc Psychiatry Psychiatr Epidemiol. 2000;35(7):288–96. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1007/s001270050241&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=11016523&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000088936400002&link_type=ISI) 5. 5.Rhodes AE, Links PS, Streiner DL, Dawe I, Cass D, Janes S. Do hospital E-codes consistently capture suicidal behaviour? Chronic Dis Can. 2002;23(4):139–45. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=12517321&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) 6. 6.Simon GE, Savarino J. Suicide attempts among patients starting depression treatment with medications or psychotherapy. Am J Psychiatry. 2007;164(7):1029–34. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1176/appi.ajp.164.7.1029&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=17606654&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000247872700013&link_type=ISI) 7. 7.Simon GE, Johnson E, Lawrence JM, Rossom RC, Ahmedani B, Lynch FL, Beck A, Waitzfelder B, Ziebell R, Penfold RB, Shortreed SM. Predicting Suicide Attempts and Suicide Deaths Following Outpatient Visits Using Electronic Health Records. Am J Psychiatry. 2018:appiajp201817101167. PMCID: pmid:PMC6167136. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=PMC6167136&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) 8. 8.Stewart C, Crawford PM, Simon GE. Changes in Coding of Suicide Attempts or Self-Harm With Transition From ICD-9 to ICD-10. Psychiatr Serv. 2017;68(3):215. 9. 9.Simon GE, Beck A, Rossom R, Richards J, Kirlin B, King D, Shulman L, Ludman EJ, Penfold R, Shortreed SM, Whiteside U. Population-based outreach versus care as usual to prevent suicide attempt: study protocol for a randomized controlled trial. Trials. 2016;17(1):452. PMCID: pmid:PMC5025595. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=PMC5025595&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) 10. 10.Shortreed SM, Rutter CM, Cook AJ, Simon GE. Improving pragmatic clinical trial design using real-world data. Clin Trials. (in press). 11. 11.Simon GE, Coleman KJ, Rossom RC, Beck A, Oliver M, Johnson E, Whiteside U, Operskalski B, Penfold RB, Shortreed SM, Rutter C. Risk of suicide attempt and suicide death following completion of the Patient Health Questionnaire depression module in community practice. J Clin Psychiatry. 2016;77(2):221–7. PMCID: pmid:PMC4993156. [PubMed](http://medrxiv.org/lookup/external-ref?access_num=PMC4993156&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) 12. 12.Linehan MM, Comtois KA, Murray AM, Brown MZ, Gallop RJ, Heard HL, Korslund KE, Tutek DA, Reynolds SK, Lindenboim N. Two-year randomized controlled trial and follow-up of dialectical behavior therapy vs therapy by experts for suicidal behaviors and borderline personality disorder. Arch Gen Psychiatry. 2006;63(7):757–66. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1001/archpsyc.63.7.757&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=16818865&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000238749600008&link_type=ISI) 13. 13.Simon GE, Rutter CM, Peterson D, Oliver M, Whiteside U, Operskalski B, Ludman EJ. Does Response on the PHQ-9 Depression Questionnaire Predict Subsequent Suicide Attempt or Suicide Death? Psychiatr Serv. 2013;64(12):1195–202. PMCID: pmid:PMC4086215. [CrossRef](http://medrxiv.org/lookup/external-ref?access_num=10.1176/appi.ps.201200587&link_type=DOI) [PubMed](http://medrxiv.org/lookup/external-ref?access_num=24036589&link_type=MED&atom=%2Fmedrxiv%2Fearly%2F2020%2F09%2F25%2F2020.09.24.20200998.atom) [Web of Science](http://medrxiv.org/lookup/external-ref?access_num=000327808900004&link_type=ISI) 14. 14.Ludman EJ, Simon GE, Whiteside U, Richards JE, Pabiniak C. Reevaluating Sensitivity of Self-Reported Suicidal Ideation. J Clin Psychiatry. 2018;79(3).