The Evaluation Center@HSRI: logo I-bar Chartpeopleevaluation

Toolkits & Materials HSRI Home

Product Catalog


Outcomes Measurement

PN-52 
The MHSIP Quality Report Toolkit-5/2004
Author: Dow Wieman, PhD., H. Stephen Leff, Ph.D., Amy DiRamio, Sarah Witham, Eric Goplerud, Ph.D., and Elise Young, M.S.W.
The purpose of this Toolkit is to provide guidance for those who have chosen to use the Mental Health Statistics Improvement Program (MHSIP) Quality Report to assess and report on the quality of behavioral health service systems in which they have an interest. The Quality Report is a modular performance measurement system designed to assess and report on the quality and efficiency of mental health services. Along with a set of universal indicators applicable to all persons receiving mental health services in any treatment setting, the Report Card consists of modules with indicators for specific populations and treatment settings.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-51 
Toolkit for Conducting Case Mix Adjustment of Mental Health Performance Indicators-2004
Author: Michael Hendryx, PhD.
The purposes of the tool-kit are, first, to provide a description of the definition, rationale, limitations, required tasks, and analytic methods of mental health performance indicator risk adjustment; and second, to provide the reader with computer exercises using a hypothetical database to practice conducting a risk-adjustment using either SAS or SPSS. A training module appropriate for distance learning adaptability will be released to accompany the toolkit.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-43 
Can We Measure Recovery? A Compendium of Recovery and Recovery-Related Instruments-2000
Author: Ruth O. Ralph, Ph.D., Kathryn Kidder, M.A. & Dawna Phillips, M.P.H.
This compendium includes a discussion on the conceptualization of and measurement issues related to recovery. Following this general discussion, the compendium is divided into two main sections – Recovery Measures, which covers instruments that purport to measure some aspect of recovery and Recovery-Related Measures which provides information on instruments that measure content that may be related to recovery. The actual instruments and related readings are included in an appendix to the compendium.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-04A 
Measuring Psychosocial Rehabilitation Outcomes: Toolkit Pilot Study Final Report-6/1998
Author: IAPSRS Research Committee
This report summarizes the results of multisite field testing of the toolkit for measuring psychosocial rehabilitation outcomes, an instrument developed by the research committee of IAPSRS.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-28 
Implementing Outcomes Management Systems in Mental Health Settings-8/1997
Author: G. Richard Smith, Jr., M.D., Carol Nordquist, M.A., Ellen P. Fischer, Ph.D., Cindy Mosley, Nancy S. Ledbetter, M.B.A.
Outcomes management is a recently described process to improve the outcomes and quality of care. The goal of outcomes management is to increase the effectiveness of health care interventions received by clients using a systematic assessment of clinical processes and outcomes and the provision of this information to those involved in treatment. Considerable work has been conducted to develop and adapt this methodology for the mental health care setting; however, little information exists on the actual implementation of a successful outcomes management system (OMS). This paper describes the process whereby an organization can implement such a system. While straightforward, implementation of an OMS requires thought and effort to be successful.
Publications can now be found on HSRI.org

PN-25 
Use of BASIS – 32 for Outcome Assessment of Recipients of Outpatient Mental Health Services-3/1997
Author: Sue Eisen, Ph.D., Marsha Wilcox, Ed.D., Elizabeth Schaefer, M.Ed., Melissa Culhane, BA, and H. Stephen Leff, Ph.D.
This study was undertaken to test both the technical and practical aspects of usefulness of BASIS-32 for outcome assessment of recipients of mental health outpatient services. The technical aspects focused on the ability of BASIS-32 to accurately and reliably assess the client’s difficulty in each of the domains that comprise the BASIS-32 and its potential for documenting change. Practical aspects of usefulness focused on the processes and costs of outcome assessment, including data collection, management, analysis, and reporting of results, as well as usefulness of results to clinical providers, managers, payers, and policy makers. This study also asked if the BASIS-32 and SF-36 measure the same thing. If not, it asked what the areas of overlap are and the unique areas of each measure.
Publications can now be found on HSRI.org

PN-15 
Report on Massachusetts Department of Mental Health Service Recipient Mortality 1991-1993-1/1996
Author: The Critical Incident Reporting Task Force
This report describes the results of a task force that investigated deaths among persons served by the Massachusetts Department of Mental Health (DMH) in recent years. The report has sections addressing the trend in the mortality rate, recommendations for improvement in data systems, preventable or postponable deaths, and the investigation process. The task force specified 38 recommendations to the DMH for system improvement. The Evaluation Center@HSRI provided technical assistance to the task force and is now disseminating the report in hopes that it may serve as a model for similar analyses elsewhere.
Publications can now be found on HSRI.org

PN-03 
Toolkit on Evaluating Quality of Life for Persons with Severe Mental Illness-1995
Author: Anthony Lehman, MD, MSPH.
The Quality of Life (QOL) Toolkit provides complete instructions for assessing quality of life for persons with severe mental illness. It includes a discussion of the concept of quality of life, a review of instruments that have been used in QOL assessments, the full version of the Lehman QOL Interview (in hard copy and on disk) with training manual; the Brief Version of the QOLI (in hard copy and on disk) with training manual; information on data analysis including instructions for determining sample size, scale construction, scoring guide, code books, and data specification sheets; data programming instructions in SAS and SPSSX (system files on disk); and examples of table results. Also included in the Toolkit is a selection of reprints describing studies that have utilized the Lehman QOL Interview.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-04 
Toolkit for Measuring Psychosocial Rehabilitation Outcomes-1995
Author: Prepared by The Research Committee of the International Association of Psychosocial Rehabilitation Services (IAPSRS).
This Toolkit provides for the ongoing measurement of a comprehensive set of targeted outcomes for psychosocial rehabilitation (PSR) programs and/or services. The Toolkit consists of the PSR Outcomes instrument and instructional materials necessary to use and interpret the instrument. The Toolkit materials are provided in hard copy and on a computer disk with the instrument forms included to facilitate tailoring them to local needs. This publication is copyrighted.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-05 
Toolkit on Evaluating Family Experiences with Severe Mental Illness-1995
Author: Richard Tessler, Ph.D. & Gail Gamache, Ph.D.
This Toolkit includes a documented discussion of the concept of care-giving, the burdens and gratifications associated with caring for a family member with severe mental illness, as well as a summary of current research. Full instructions for fielding a study using the Family Experiences Interview Schedule (FEIS) are provided. These instructions include the history and characteristics of the interview schedule, modular descriptions and rationales for each module, an outline of measures, procedures for scale and index construction, a question by question training manual, and complete instructions for interviewing. The Toolkit now includes an addendum with “New Measures of Family Member Evaluation of Mental Health Professionals, Client Services and Systems.”
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-06 
Toolkit on Evaluating Substance Abuse in Persons with Severe Mental Illness-1995
Author: Kim T. Mueser, Robert C. Drake, Robin E. Clark, Gregory J. McHugo, Carolyn Mercer-McFadden, & Thelma H. Ackerson.
This Toolkit provides complete instructions for assessing the detection of substance abuse in persons with severe mental illness. It includes a review of the literature in this area as well as the following instruments: Clinician Alcohol Use Scale, Clinician Drug Use Scale, Substance Abuse Treatment Scale, Drug/Alcohol Follow-Back Calendar. The instruments are also available on a disk included with the Toolkit. Extensive sections on training, data analyses, and a full bibliography complete the Toolkit.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-09 
Cross-Disability Integrated Health Outcomes Survey (CDIHOS)-1995
Author: Prepared by the Evaluation Center@HSRI.
The Cross-Disability Integrated Health Outcomes Survey (CDIHOS) is a health care outcomes assessment instrument. Its items cover an extensive range of domains such as social demographics; continuity and availability of medical and psycho-social services received; symptomatology; mental and physical functional status; work, school, and family activities; legal problems; preventative information offered by providers; substance abuse; general service satisfaction; and quality of life. The CDIHOS is designed to be administered to diverse populations, including persons with mental disabilities. For this reason, it utilizes straight-forward, structured questions, each of which is paired with an illustration to facilitate responding.
Publications can now be found on HSRI.org


Evaluation Methodolgy & Statistics

PN-50 
Applying Evaluation Knowledge and Operations Research Methods to Planning Mental Health Services: The Case of Housing for Persons Discharged from Psychiatric Inpatient Units in New York City-9/2002
Author: H. Stephen Leff, Ph.D.
This working paper includes two operations research models for planning the housing component of a mental health system. The paper has three purposes: 1) to bring the models developed, and operations research to those engaged in planning and evaluating housing and other services for persons with mental illness; 2) to show how data from needs assessments, information systems, and outcome evaluations can be used to provide the estimates required by the models outlined in the paper and by models of this type; and 3) to provide insights into the problem of meeting the housing needs of mental ill persons in a metropolitan area. One of the central themes of the paper is that even with incomplete data, operations research methods can provide important policy and planning insights in a timely manner.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-37 
Estimating Per Unit Treatment Costs for Mental Health and Substance Abuse Programs-3/1999
Author: Barbara Dickey, Ph.D., Jennifer K. Beecham, Ph.D., Eric Latimer, Ph.D., & H. Stephen Leff, Ph.D.
This toolkit is designed for program evaluators and others who wish to understand how unit costs of specific mental health services are estimated, especially where data available to carry out estimations vary in both quantity and quality. Per unit costs are the key to determination of per person expenditures. This toolkit has been developed to offer some practical methods for arriving at unit cost estimates when it is not possible to gain access to the cost accounts needed to calculate precisely the per unit cost of providing a specific service from a specific site. The toolkit also helps the reader to gauge the precision of such estimates.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-33 
A Methodology for Probabilistically Estimating Caseload Size and Overlap-1/1999
Author: Steven M. Banks, Ph.D., John Pandiani, Ph.D.
This working paper presents three applications of Probabilistic Population Estimation (PPE), a new statistical procedure that provides information on the unduplicated number of people who are involved in more than one institution, program, group, or activity, either concurrently or in sequence. The first is an estimation of the unduplicated number of residents of an entire state who were hospitalized for psychiatric or substance abuse services during a five-year period. The second applies the PPE methodology to construction of a global measure of access in a managed mental health care system. The third uses PPE to evaluate substance abuse treatment programs using incarceration.
Publications can now be found on HSRI.org

PN-32 
Integrating Process and Outcome Evaluation-3/1998
Author: Margaret Blasinsky, Barbara Cohen, Howard Goldman, Kevin Hambrecht, Matthew Johnsen, Wendy Landow, Rob Orwin, and Joseph Sonnefeld.
This Toolkit, developed by a team at ROW Sciences, Inc., describes an approach to organizing multi-level or hierarchical data, such as those collected in a multi-site study, to facilitate analysis of data from multiple levels (e.g., characteristics of sites, characteristics of provider or managed care organizations, consumer characteristics, service utilization, and consumer outcomes). The process for constructing a project-level database involves assembling a team, collecting relevant program/site level information from a variety of sources (e.g., contracts, census data, IMHO data) and organizing the information into a database along with more traditional data such as consumer survey and service utilization data. The Toolkit includes an example of analysis using a project-level database developed for a multi-site homelessness service demonstration study funded by the National Institute on Alcohol Abuse and Alcoholism.
Publications can now be found on HSRI.org

PN-16 
Practical Meta-Analysis Toolkit-1996
Author: Mark W. Lipsey & David B. Wilson.
The purpose of this Toolkit is to provide a practical guide to conducting meta-analysis with emphasis on its application to applied research, especially mental health research. Meta-analysis applies only to empirical research studies. It cannot be used to summarize theoretical papers, conventional research reviews, policy proposals, and the like. It applies only to research studies that produce quantitative findings. It should be noted that meta-analysis appears under many labels in published literature, e.g., quantitative synthesis, research synthesis, research integration, and the like. The toolkit comes with a disk that contains the MS Excel Effect Size Program, SPSS Macros, and Example Data Files.
Publications can now be found on HSRI.org

PN-18 
MIXOR & MIXREG: A Windows Computer Program for Mixed-Effects Regression Analysis with Autocorrelated Errors and Mixed-Effects Ordinal Regression Analysis Toolkit-1996
Author: Donald Hedecker & Robert D. Gibbons.
This Toolkit includes Windows computer programs for a regression analysis of longitudinal and clustered data that are dichotomous, ordinal, or normally-distributed with autocorrelated errors. The programs provided employ a mixed-effects regression model, which assumes some regression parameters other than residuals to be random. Hence, the program adjusts for the dependence among observations present in longitudinal designs due to subjects being repeatedly assessed over time; and in clustered designs due to subjects being nested within a larger context, for example hospitals or firms. The mixed-effects regression model can be applied to the unbalanced data which, in longitudinal context, may mean unequal numbers of observations per subject. Or it can be applied to even observations at different points in time and in clustered context, unequal numbers of subjects per cluster. Both average outcomes for a population and individual outcomes for each subject can be estimated with the mixed-effects regression model. In addition to the computer programs, this Toolkit provides summaries on the computational methods used in the programs, the programs’ descriptions, examples illustrating the programs’ uses, and relevant journal articles.
Publications can now be found on HSRI.org

PN-20 
Toolkit for Effective and Persuasive Case Studies-1996
Author: Lee Sechrest, Ph.D., Michelle Stewart, Timothy R. Stickle, and Souraya Sidani.
The aim of this Toolkit is to provide a framework for case studies that will substantially enhance their effectiveness and make them appropriately persuasive to the audiences for which they are intended. This Toolkit is meant to be comprehensive, addressing the entire range of purposes and types of case studies. It develops a general research protocol, however, that addresses potential deficiencies in case studies in such a way that it can be adapted to the needs of individual investigators and projects. The intent of this effort is to enhance the usefulness of case studies and to make their production as straightforward as that of any other research product.
Publications can now be found on HSRI.org

PN-21 
Evaluating Peer Providers-1996
Author: H. Stephen Leff, Ph.D., Jean Campbell, Ph.D., Cheryl Gagne, and Lawrence Woocher.
This working paper discusses issues related to evaluation of mental health services provided by peers also referred to as consumers/survivors/ex-patients. The first section of the paper offers a desired research process, combining participatory approaches to evaluation with traditional evaluation methods. The second section presents a conceptual model of research content for guiding future research on the effectiveness of peer providers. In presenting this model, the authors also discuss the research to date on its variables to suggest useful starting points for original research.
Publications can now be found on HSRI.org

PN-12 
Cluster Analysis of Service Data for Managed Care and Mental Health Systems Evaluation-6/1995
Author: H. Stephen Leff, Ph.D., Marcus Leiberman, Ph.D., & Matthew M. Wise, MPH.
The purpose of this short working paper is to introduce cluster analysis and illustrate its application to analyzing mental health service utilization data in the context of managed care and systems evaluation. While this exposition may not be sufficient to understand all the complexities of the procedure, it will enable people with elementary statistic knowledge to use software packages intelligently. The paper provides references to several texts on cluster analysis for those interested in furthering their understanding of this statistical technique.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-12 
Meta-Analysis Guidelines-4/1989
This is an exercept of guidelines on conducting a meta-analysis. Topics covered include selecting and coding studies, correction of coefficients for artifacts, interpreting results and resolution of conflicting meta-analyses. The guidelines are summarized succinctly by the authors and is a useful compendium to the HSRI toolkit on Practical Meta-Analysis (PN-16)
Publications can now be found on HSRI.org


Managed Care

PN-40 
Mental Health, Medicaid and Managed Care: Building a Unified System in Massachusetts-11/1999
Author: Robert A. Dorwart, M.D., M.P.H.
This case study explores the issues that arise when two state agencies—the Medicaid agency and the state mental health authority—come together to develop a managed care program for persons with serious mental illness. Reported from the perspective of the director of the SMHA, the study demonstrates how effective managers can resolve problems when autonomous state agencies, with different cultures and missions, join to blend funding streams in a program for a vulnerable population. The report also illustrates a particular form of case study investigation. This study was prepared by researchers at the Kennedy School of Government under the direction of Robert Dorwart, M.D., M.P.H.
Publications can now be found on HSRI.org

PN-27 
A Conceptual Framework for Evaluating the Intersystem Impacts of Managed Behavioral Health Care: Report on a Roundtable Discussion-4/1997
Author: Howard H. Goldman, MD, Ph.D. & Joseph P. Morrissey.
This essay presents an “open systems” perspective on current alcohol, drug abuse and mental health (ADM) reforms as a conceptual framework for evaluating their intersystem impacts. The framework focuses both on reforms within the ADM system itself (managed behavioral health care, MBHC, in particular) and those coming from outside that system (such as welfare reform). It is based on a review of the literature and discussions among a group of consultants attending a meeting held in November 1995. The first part of the report describes recent trends in MBHC and reviews some historic precedents of cost shifting following major reforms in the financing of mental health services. The second part examines an open system framework for understanding the impact of these changes on the ADM service system. The third part reviews some of the specific ways in which cost shifting between sectors may occur under MBHC arrangements.
Publications can now be found on HSRI.org

PN-22 
Regulating Managed Mental Health Care: A Policy Analysis and Discussion of the Role of Evaluation-12/1996
Author: David Rochefort, Ph.D.
This report analyzes recent efforts to regulate managed care practices in the provision of mental health care and discusses the relationship of these efforts to program evaluation. Regulatory measures described address areas such as information disclosure, consumer protection, mandated benefits, quality monitoring, licensing, selective contracting, and data collection. The author presents a framework of analysis that considers the concepts of “policy instruments” and “quality assurance” as they relate to regulation. Specific managed care regulatory efforts in Vermont, Rhode Island, and Massachusetts are also described. Finally, the author examines the role of program evaluation as a crucial regulatory tool.
Publications can now be found on HSRI.org

PN-23 
An Overview of Evaluations of the Massachusetts Medicaid Managed Behavioral Health Care Program-12/1996
Author: H. Stephen Leff, Ph.D., Virginia Mulkern, Ph.D., & Lawrence S. Woocher (Eds.).
This multi-authored report provides an overview of the evaluation activities conducted of the Massachusetts Medicaid managed behavioral health care program, the first statewide waiver program to deliver managed mental health and substance abuse services to Medicaid enrollees. The report includes chapters on service utilization and cost, service access and quality, provider changes, network development and maintenance, enrollment, and intersystem effects. Each of these chapters summarizes what was learned in the evaluations of the Massachusetts program and provides recommendations for future managed care evaluations. The report also includes a chapter that digests recommendations for managed care evaluation voiced by experts at a project symposium and a chapter of summary analysis and recommendations by the editors.
Publications can now be found on HSRI.org

PN-01 
Evaluating Models of Medicaid Managed Mental Health: Program and Evaluation Materials from States-1996
Author: by the Evaluation Center@HSRI.
This resource book provides original materials from states about their managed mental health care programs for Medicaid recipients as well as materials from the evaluations of these programs. In addition, Evaluation Center staff have prepared a brief overview of the program and evaluations of each of the eleven states included. States were chosen for inclusion that had Medicaid managed mental health care programs, and had program evaluations planned, in process, or completed. Since this has been an ongoing project, the states included do not necessarily represent all states that meet these criteria currently, but they do provide a substantial sample of such programs as of January 1996. For each state, where applicable and available, we have included a State Overview (prepared by The Evaluation Center), a Program Summary, Evaluation Design/Plans, Instruments, and Reports. The book also will contain introductory materials including a national overview and summary results from the states studied. The included appendix contains material relevant to the first multi-state study funded by HCFA and SAMHSA.
Publications can now be found on HSRI.org

PN-02 
Evaluating Managed Mental Health Care: A Sourcebook-1995
Author: Prepared by the Evaluation Center@HSRI.
This volume contains selected readings assembled to assist in evaluations of managed mental health care generally and the mental health components of Medicaid managed care waivers in particular. Review articles, commentaries, and original research reports are included. The particular topics that the book covers are: History of Medicaid Waivers, Classifying Managed Care Initiations, History and Assessment of Evaluation Approaches, Implementation, Evaluation and Quality Assessment, Consumer Level Outcomes, Family Experiences, Managed Care Costs, and Intersystem Impacts.
Publications can now be found on HSRI.org


Performance Measurement

PN-55 
"Measuring the Promise: A Compendium of Recovery Measures, Volume II" -10/2005
Author: Theodora Campbell-Orde, M.P.A., Judi Chamberlin, Jenneth Carpenter, M.S.W., & H. Stephen Leff, Ph.D.
This compendium offers an update to PN-43, Can We Measure Recovery? A Compendium of Recovery and Recovery-Related Instruments (Ralph, Kidder, & Phillips, 2000). Following a discussion of the need for recovery measures, Volume II offers a synopsis of the November 2004 invitational conference Measuring the Promise: Assessing Recovery and Self-Determination Instruments for Evidence-Based Practices. Reviews of the instruments themselves follow this section. The instruments are divided into two categories: measures of individual recovery and measures of recovery-promoting environments. Volume II concludes with recommendations for further developments in the measurement of recovery.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-13 
Consumer Research Activities in the States-6/2005
Author: Dr. Paul Weaver, Kentucky Center for Mental Health Studies
This report provides an overview of consumer involvement in mental health research and evaluation, aiming to inform technical assistance centers, such as the Evaluation Center through its Consumer Evaluator Network, where to direct resources in regards to consumer research and evaluation activities. The report discusses 1) consumers’ involvement in evaluation and research, 2) the areas in which consumers’ desire technical assistance in these fields, and 3) consumers’ familiarity with a consumer grant program that funds consumer evaluation and research activities entitled The Consumer Evaluator Network. Information for this report was collected through organized discussions with individuals knowledgeable of consumer activities, such as the directors of the State Office of Consumer Affairs (OCA).
Publications can now be found on HSRI.org

PN-49 
Keys to Quality: Conducting a Performance Improvement Project for Behavioral Health in Managed Care Based on the Principals of QISMC-9/2002
Author: Dow Wieman, PhD., H. Stephen Leff, Ph.D., Amy DiRamio, Sarah Witham, Eric Goplerud, Ph.D., and Elise Young, M.S.W.
This toolkit is designed to provide technical assistance to persons conducting or reviewing behavioral health quality improvement projects in managed care. For this purpose the toolkit presents a set of criteria defining exemplary behavioral health QI practices and case illustrations of these criteria. These criteria represent QISMC standards with added recommendations based on standards of research methodology or derived from other sources such as the MHSIP Consumer-Oriented Report Card. The projects described in the toolkit are intended to illustrate the QISMC standards and do not necessarily fulfill current or future HCFA requirements, nor are they HCFA requirements.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-47 
Selecting Process Measures for Quality Improvement in Mental Healthcare-7/2002
Author: Richard C. Hermann, M.D., M.S., H. Stephen Leff, Ph.D., & Greta Lagodmos, B.A.
This toolkit is a resource for identifying and selecting process measures for use in quality assessment and improvement activities. The toolkit includes guidance for using the National Inventory of Mental Health Quality Measures an interactive database of single item process measures of quality. The database will be available at http://www.cq aimh.org/quality.html and through a link provided by the Evaluation Center@HSRI (www.tecathsri.org). Both the inventory and the toolkit focus on process measures, a type of quality measure that evaluates components of the interaction between the healthcare system and consumers of healthcare.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-10 
QA/QI Poster-4/2002
This poster is a collaborative effort between the Evaluation Center and Harvard Medical School. The poster presents the results of a study in measuring whether quality measures used in QA/QI in mental health address evidence-based practices.
Publications can now be found on HSRI.org

PN-46 
Measuring Conformance to Treatment Guidelines: The Example of the Schizophrenia PORT-8/2001
Author: The Schizophrenia Patient Outcomes Research Team, University of Maryland School of Medicine, and John Hopkins School of Hygiene and Public Health (Anthony Lehman, M.D., M.S.P.H,, Principal Investigator, and Donald Steinwachs, Ph.D., Co-Principal Investigator).
The use of adherence to guidelines to measure quality of care has emerged as a major tool in the quality management arsenal. This toolkit has been developed to illustrate how to design a measurement strategy around a set of guidelines as a means of assessing how well an organization is actually implementing those guidelines. This toolkit is not a guide to the treatment of schizophrenia. A major reason is the PORT guidelines, as described in this Toolkit, may have been subsequently updated since the toolkit was compiled. Nevertheless, the PORT process described serves to illustrate how one might begin to build a guideline-based quality assessment system. A secondary purpose of this toolkit is to illustrate how evaluation evidence can be used to develop guidelines for specific disorders. The PORT Tools for assessing care (the PORT Mental Health Survey, and the Schizophrenia PORT Inpatient and Outpatient Record Review forms) are included in the toolkit.
Publications can now be found on HSRI.org

PN-44 
Psychiatric Rehabilitation Fidelity Toolkit-11/2000
Author: Gary Bond, Jane Williams, Lisa Evans, Michelle Salyers, Hea-Won Kim, Heather Sharpe, and H. Stephen Leff.
This toolkit presents a working guide for the development of fidelity measures to be used in assessing the implementation of psychiatric rehabilitation program models. This toolkit describes the origins of fidelity measures and discusses their research and practical applications, reviews current models in psychiatric rehabilitation, and provides a detailed guide for developing fidelity measures. The Appendix gives examples of instruments currently in use for psychiatric rehabilitation.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-42 
Linking Mental Health Consumer Files with State Death Records: A Guide to Obtaining and Using Mortality Information to Evaluate the Needs and Outcomes of Consumer Populations-4/2000
Author: Bruce Dembling, Ph.D.
This Toolkit provides planners, statisticians, and researchers with the technical assistance necessary to use mortality data as a performance indicator by linking consumer files with state death records. Data from individual death certificates can help clinicians understand the risks faced by consumers, and evaluate the needs and outcomes of their specific populations over time. This Toolkit addresses the technical aspects of data acquisition, processing and analysis, and requires significant computing resources. It includes a CD ROM with electronic files used to build translation tables for cause of death codes, electronic links to state and federal agencies and resources, as well as a Bibliography of Related Readings.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-11 
Excerpt: A Model of Indicators & A Report Card for Assessment of Mental Health Plans' & Systems' Performance-3/1999
This is a brief excerpt taken from a Model of Indicators & a Report Card for an assessment of mental health plans and systems performance. It includes various means of presenting and interpreting performance assessment data and disseminating findings. Sample Report Card formats with graphic illustrations are also included.
Publications can now be found on HSRI.org

PN-35 
Performance Measurement Using the MHSIP Consumer Oriented Report Card-11/1998
Author: by the Evaluation Center@HSRI.
This Toolkit has been developed to support users of the MHSIP Consumer-Oriented Report Card, a set of performance indicators and measures developed by the Mental Health Statistics Improvement Program (MHSIP). The Toolkit provides guidance to persons implementing a performance measurement system on a host of practical and methodological topics. The Toolkit is organized into six chapters: Introduction, Planning and Organizing your Performance Measurement System, Design and Methods, Data Management and Analysis, Reporting Results, and Evaluating a MHSIP Report Card Project. Finally, the Toolkit includes several appendices taken from the experience of states including an example of a state mental health authority’s budget for a performance measurement project, sample translations of the Consumer Survey, and a literature review on the use of consumer interviewers.
Publications can now be found on HSRI.org

PN-24 
County Behavioral Health Performance Measures-1997
Author: National Association of County Behavioral Health Directors (NACBHD).
This report describes the draft set of performance indicators developed and selected by representatives of the National Association of County Behavioral Health Directors for use by county mental health/substance abuse authorities. Following an introduction and description of the process of indicator selection, the report presents the draft performance measures in five domains: access, satisfaction, consumer outcomes, intersystem outcomes, and utilization. Tables of candidate measures that were considered by the group and instruments proposed as measurement tools for the indicators selected are appended.
Publications can now be found on HSRI.org

PN-30 
Indiana Hoosier Assurance Plan Packet-1997
Author: Fredrick L. Newman, Richard Deliberty, Kay Hodges, John McGrew, & Manual J. Tejeda.
The Hoosier Assurance Plan is a legislative mandate to provide care for adults with serious mental illness, or a chronic addiction, or both; and, children and adolescents with a severe emotional disturbance. The key system features are managed competition among service providers, informed consumer choice supported by a Provider Profile Report Card, an advisory panel (with consumer involvement) to guide system development, and empirical evidence to support Level of Care Determination/Reimbursement. This packet contains several materials related to the Hoosier Assurance Plan: • Report on Research/Implementation Strategy and Assessment Instruments to Support Level of Care Determination • Potential Format of CAFAS and HAPI-Adult Outcome Data for the 2nd Edition of the Provider Profile Report Card • Hoosier Assurance Plan Instrument for Adults [HAPI-Adult]: A Public Domain Assessment Instrument • Training Packet for the HAPI-Adult • Provider Profile Report Card, 1st Edition [November, 1996]
Publications can now be found on HSRI.org

PN-19 
Toolkit: A Model of Indicators and A Report Card for Assessment of Mental Health Plans’ and Systems’ Performance-1996
Author: Edna Kamis-Gould, Ph.D., & Trevor R. Hadley, Ph.D.
This Toolkit takes a comprehensive look at performance assessment of mental health plans and systems. The major objectives of the toolkit are to: (1) provide background information on performance assessment, performance indicators and report cards in the context of a managed care environment; (2) to describe and summarize existing models; (3) to present a model tailored to the needs of public mental health authorities; and (4) to describe the process of developing performance indicators and discuss such practical issues as cost, reliability, and validity.
Publications can now be found on HSRI.org

PN-11 
Stakeholder Perspectives on Mental Health Performance Indicators-1995
Author: Working papers prepared for the Mental Health Statistical Improvement Program (MHSIP) Phase II Task Force.
This material is a two-part document containing “A Compilation Of The Literature On What Consumers Want From Mental Health Services” and “Performance Indicators For A Consumer-Oriented Mental Health Report Card: Literature Review & Analysis.” Both are working papers prepared for the Mental Health Statistics Improvement Program (MHSIP) Task Force on the Development of a Mental Health Care Report Card.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org


Internet Evaluation Issues

PN-34 
A Tale of Two Networks: Possibilities and Perils for Mental Health Internet Discussion Groups-9/1998
Author: Prepared by the Evaluation Center@HSRI.
This paper explores the dynamics of two electronic discussion lists: one established to promote the discussion of mental health outcomes research and evaluation and the other established to discuss the application of evaluation to issues related to mental health and the law. After a year and a half, participants in the former continued to discuss issues according to their original goal. The latter discussion list, on the other hand, had changed from being the more active of the lists to being one marked by conflict, withdrawal of participants and hardly any activity. This paper examines how usage of the two lists evolved differently because of their content and membership. And it suggests, for mental health policy makers, consumers, planners, and evaluators, how, through the use of moderators who keep discussions respectful and on topic electronic discussion groups can be harnessed to bring together diverse stakeholders in to a learning environment.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-14 
Evaluating Internet Mental Health Applications: A Roundtable Report-1996
Author: Prepared by the Evaluation Center@HSRI.
This report describes the Roundtable discussions on evaluating Internet applications available to mental health system stakeholders. The objectives of the Roundtable discussion included: 1) sharing information about what mental health resources there are on the Internet; 2) learning what types of evaluation research have been conducted in this area; 3) considering the possible methods for evaluating on-line resources; and 4) identifying people who might be interested in becoming partners in this area of research. Participants included researchers, recipients of mental health services, and providers of Internet resources.
Publications can now be found on HSRI.org


Multicultural Issues in Evaluation

PN-45 
Science of Refugee Mental Health: New Concepts and Methods-2100
Author: Edited by: WooTaek Jeon, M.D., Ph.D., Masaya Yoshioka, M.D., and Richard F. Mollica, M.D., M.A.R.
This monograph presents scientific overviews of the major methods in the field of refugee mental health. The monograph also presents summaries of presentations from a 1992 conference sponsored by the Refugee Mental Health Program at the National Institute of Mental Health, the Harvard Program in Refugee Trauma at the Harvard School of Public Health, and the Indochinese Psychiatric Clinic at St. Elizabeth’s Hospital of Boston. This material will be helpful to mental health researchers and providers throughout the world working with survivors of mass violence and trauma.
Publications can now be found on HSRI.org

PN-54 
Toolkit on Tranlating and Adapting Instruments-4/2005
Author: Ligia M. Chávez, Ph.D., Glorisa Canino, Ph.D.
The purpose of this toolkit is to provide a step by step guide on the translation and adaptation of an instrument, using the latest standards for methodological approaches in cross-cultural research, in order to achieve cultural equivalency. Researchers and evaluators will be made aware of the conceptual and methodological challenges involved in translating and adapting instruments, and will be provided with guidelines and suggestions throughout the process.
Publications can now be found on HSRI.org

OT-14 
Annotated Bibliography on Cultural Psychiatry and Related Topics-2/2005
Author: Francis G. Lu, M.D.
The annotated bibliography contains brief guides to journals, videotapes and relevant websites concerning topics in multicultural mental health issues. Areas such as cultural competency assessment and promotion, as well as training materials for culturally and linguistically appropriate services are also included.
Publications can now be found on HSRI.org

FF-10 
Training in Cultural Competence: The Need for a Mental Health Cultural Assimilator-4/2003 Volume: 2 Issue: 3
Author: Sung-man Shin, M.S.
The increasing numbers and percentages of minority populations in the United States, along with the accumulating evidence of service disparities in access, continuity, and quality for people of minority cultures (Brach, & Fraser, 2002; Snowden, 2003; Sue, Fujino, Hu, Takeuchi, & Zane, 1991), create a profound sense of urgency among the public health service providers to resolve the treatment discrepancies between different cultural groups.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-41 
Ecocultural Research: A Mental Health Researcher’s Guide to the Study of Race, Ethnicity and Culture-3/2000
Author: Michelle Walsh, Rachel Smith, Alicia Morales, and Lee Sechrest.
This guide presents an ecocultural approach to cultural issues, taking into account the broad array of ecological and cultural variables that are likely needed to provide a causal explanation for difference in the provision of behavioral health services. It promotes a means of conceptualizing, measuring, and interpreting ecocultural variables in a way that will advance the provision of more effective behavioral health services, and fosters the growth of research skills that can be used to further pursue an understanding of culturally relevant variables in applied mental health service settings.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-07 
Providing & Evaluating Culturally Competent Care is a Growing Concern-2/2000 Volume: 1 Issue: 7
Author: The Evaluation Center @ HSRI
This issue discusses the conceptualization and delivery of culturally competent mental health services. It highlights two aspects of culturally competent care, measuring consumer identity and using culturally appropriate diagnosis, assessment and treatment.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-39 
Notes from a Roundtable on Conceptualizing and Measuring Cultural Competence-12/1999
Author: Western Interstate Commission on Higher Education (WICHE) Mental Health Program.
This report includes presentations and a summary of discussion from a two-day roundtable held in Denver, Colorado, on December 16-17, 1998. The event was sponsored by the Evaluation Center @HSRI (TEC), the Multicultural Mental Health Research Center (MMHRC), and the Western Interstate Commission on Higher Education (WICHE). The participants discussed and made recommendations for practical approaches to conceptualizing and measuring ethnocultural identity and cultural competence in the delivery of mental health services. Individual papers addressing measures of ethnocultural identity and cultural competence for use in evaluation of mental health systems presented are included as appendices.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-38 
Culture, Race and Ethnicity in Performance Measurement: A Compendium of Resources, Version 1-9/1999
Author: Dawna Phillips, M.P.H., H. Stephen Leff, Ph.D., Eva Kaniasty, Michael Carter, Marcel Paret, Theresa Conley and Manisha Sharma.
This compendium is a compilation of resources and readings for those interested in the area of providing or evaluating culturally competent mental health care. It is designed to be a dynamic resource. This compendium provides resources for several elements of a culturally competent mental health system as the Center has conceptualized it. Following are the topical areas covered: cultural competence standards; community needs assessment; measures of identity; culturally competent diagnosis, assessment and outcomes measures; and measures of adherence to cultural competence standards.
Publications can now be found on HSRI.org

FF-03 
Research Shows Ethnicity A Factor in Medication Response-10/1998 Volume: 1 Issue: 3
Author: Susan Milstrey Wells
This issue discusses the growing body of research that has begun to demonstrate how racial and ethnic groups respond differently to psychiatric medication.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-31 
Report of the Roundtable on Multicultural Issues in Mental Health Services Evaluation-3/1998
Author: Prepared by Evaluation Group for the Analysis of Data (EGAD) with collaboration from the Evaluation Center@HSRI.
A Roundtable discussion (working conference) of multicultural issues in Mental Health services evaluation was held in Tucson, AZ, on November 1-2, 1996. The goals of the Roundtable were to begin discussion among persons with multicultural expertise regarding the content, methodology, and design of mental health evaluations that would be sensitive to multicultural issues and to identify technical assistance options that could be implemented using the available resources of the Evaluation Center@HSRI. This report summarizes the proceedings of the presentations and discussions. Subsequent sections outline the central themes that emerged during the Roundtable. The report concludes with the consensus recommendations of participants.
Publications can now be found on HSRI.org


Evidence-Based Practices

FF-16 
Medicaid Performance Improvement Projects: A Means of System Transformation?-5/2005
Author: The Evaluation Center @ HSRI
The Centers for Medicare and Medicaid (CMS) have recently undertaken an initiative to promote systemwide performance improvement in Medicaid Managed Care Organizations (MCOs) by means of mandated Performance Improvement Projects (PIPs). The PIP initiative follows the general approach of conventional quality improvement methods, and draws extensively on earlier CMS initiatives as well as work done by others in the field, notably the National Committee for Quality Assurance. The protocol for conducting a PIP developed by CMS lays out a general framework for these activities and specific requirements to which they must conform. This FastFacts describes the ways in which the PIP protocol follows the general approach of conventional quality improvement methods.
Publications can now be found on HSRI.org

OT-15 
Making the Grade A Review and Comparison of Selected Evidence Grading Systems-0/2005
Author: 2005 H. Stephen Leff, Ph.D., Jeremy Conley, B.A., Shekihan Elmore, B.A.
Assessing the evidence produced by evaluations of interventions is at the heart of moving towards evidence-based practice. In collaboration with the National Academies of Science, the Evaluation Center@HSRI has written Making the Grade: A Review and Comparison of Selected Evidence-Grading Systems, a resource for anyone interested in the evidence grading process. This material describes and compares key features and mechanisms of influential evidence-grading systems in behavioral health fields, and provides an a introduction to the needs for evidence grading and the principles behind the process.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-08 
Getting to Systems That Promote Self-Determination Through Research and Evaluation-10/2003
Author: H. Stephen Leff, Ph.D., Jeremy Conley, Bphil., Theodora Campbell-Orde, MPA, Valerie Bradley, MA.
This is a working paper that was prepared for The National Self-Determination and Psychiatric Disability Invitational Conference: We Make the Road by Traveling on it, in Chicago-O’Hare, Illinois. The paper discusses self-determination in the field of mental health, exploring the construct of self-determination and the use of research and evaluation to develop systems which promote self-determination. The author presents challenges for quantitative research and evaluation posed by self-determination for persons with mental disorders: operationally defining and measuring self-determination, identifying services and practices that are effective in bringing about self-determination, and monitoring self-determination in report card oriented measures of quality assurance and consumer satisfaction.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-06 
Finding Evidence-based Practices: A Compilation of On-Line, Searchable Databases and Descriptions of How to Use Reviews-6/2003
Author: The Evaluation Center @ HSRI
This is a reference guide developed by the Evaluation Center on a list of on-line resources pertaining to evidence-based practices. Electronic resources include sites that focus on evidence-based mental health interventions, those that contain reviews of interventions for mental health and other medical disorders, sites that focus on understanding and using evidence-based medicine that are not specific to mental health, and printed references to evidence-based mental health practices.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-07 
A Brief History of Evidence-Based Practice and a Vision for the Future-10/2002
Author: H. Stephen Leff, Ph.D.
This is a DRAFT paper which will appear in Mental Health, United States, 2002, CMHS, SAMHSA, DHHS. The paper briefly reviews the history of evidence-based practice. Then describes the types of individuals and organizations currently focusing on evidence-based practices in mental health and nature of the information they provide. Following this, the paper considers the concerns raised about evidence-based practices in mental health. The paper then concludes with a vision of how evidence-based practices should be pursued in the future, taking into account the concerns that have been raised
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-05 
Implementation of Evidence-Based Practices: Bridging Science and Service-9/2002
Author: The Evaluation Center@HSRI
Resource materials for implementing Evidence-based Practices.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-48 
Toolkit on Manuals and Workbooks for Psychosocial Interventions-9/2002
Author: Teresita Camacho-Gonsalves, PhD., H. Stephen Leff, Ph.D., Toolkit Team, and William C. Torrey, M.D.
This toolkit is designed to provide a step-by-step guide for preparing a manual or workbook for a practice. The manual or workbook may be for a practice that is already evidence-based or for one that is about to be tested. The toolkit identifies essential elements of manuals and workbooks, describes these elements, and provides exemplary illustrations of the elements. Some of the elements described in the toolkit were suggested by research about and evaluation of the dissemination of new technologies. Other elements were suggested by theories about and observations of technological change. This is a big PDF (Adobe File) due to the images. Please understand that if you have a slow Internet connection this product could take about 5-7 minutes to save
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-04 
Knowledge Assessment: A Missing Link Between Knowledge Development and Application-7/2002
Author: H. Stephen Leff, Ph.D., Virginia Mulkern, Ph.D., Robert E. Drake, M.D., Ph.D., I. Elaine Allen, Ph.D., Clifton M. Chow, Ed.M.
Health care systems throughout the U.S. are adopting evidence-based practices. This paper addresses issues involved in evaluating evidence-based practices for psychosocial ehabilitation (PSR) interventions.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-09 
Quick Reference Guide: Finding Evidence-Based Practices-2/2002
This is an abridged version of our popular reference guide of resources on evidence-based practices. It includes resources on the internet as well as print.
Publications can now be found on HSRI.org


Other

PN-56 
Evidence-based Workforce Development Strategies for Evidence-based Practices in Mental Health-5/2007
Author: H. Stephen Leff, Judith A. Leff, Clifton Chow, Ben Cichocki, Dawna Phillips, Tamara Joseph
Implementation of evidence-based practices is one fundamental and essential requirement for transforming the nation's mental health services system as envisioned by the New Freedom Commission on Mental Health (2003) and related policy documents. It is also a key ingredient of system improvement in other countries as well. One of the factors influencing the implementation of evidence-based practices is an educated and skilled mental health workforce who can deliver and support the delivery of these practices. Developing such a workforce requires employing workforce development strategies for preparing large numbers of staff to proficient competency levels in desired skills. As much as possible, these strategies, like clinical treatments, should be evidence-based, i.e., shown by evaluation studies to be more effective than relevant alternatives.

In this paper, we provide a brief summary of some of the leading theories about learning and offer a theoretically derived set of 14 workforce development strategies for evidence-based mental health services. These strategies focus on both direct training as well as systems-level workforce development. The evidence for these strategies is assessed through a review of the relevant healthcare literature, and suggestions for improving the evidence base for workforce development are offered. In addition to offering a set of workforce development strategies, the appendices also include select instruments for measuring different levels of workforce development outcomes. The authors invite suggestions, comments.
Publications can now be found on HSRI.org

PN-36 
Report of the Roundtable on Mental Health Services Evaluation in Frontier States-1/1999
Author: Western Interstate Commission on Higher Education (WICHE) Mental Health Program.
This report summarizes presentations and discussion from the Roundtable on Mental Health Services Evaluation in Frontier States held in April 1998. the Evaluation Center@HSRI (TEC) and the WICHE Mental Health Program sponsored the roundtable. The purpose of the roundtable was to identify evaluation technical assistance needs of state mental health programs in selected frontier states. This report concludes with recommendations for individual and multi-state technical assistance, and a description of collaborative efforts between TEC and WICHE to provide technical assistance to frontier states.
Publications can now be found on HSRI.org

PN-29 
Seventh Annual National Association of State Mental Health Program Directors (NASMHPD) Research Institute Proceedings on Mental Health Research and Evaluation: Its Influence on Policy & Decision Making in an Era of Managed Care.-2/1997
Author: Presenters' materials compiled by the Evaluation Center@HSRI.
The proceedings contain papers and presentation materials from the Seventh Annual National Association of State Mental Health Program Directors Conference on State Mental Health Agency Services Research and Program Evaluation.
Publications can now be found on HSRI.org

PN-26 
What Policy Makers Want: A Guide for Evaluators-1997
Author: Leslie J. Scallet, JD.
Politics or self-interest may outweigh research findings and dictate results. But solid, well supported answers to policy questions can powerfully influence decisions. They can deter outcomes that contradict known fact or experience. They can suggest solutions to policy problems. And they can shape the underlying base of knowledge and understanding about an issue. Many sources inform the policy process and the policymaker. The purpose of this Guide is to help evaluators become more effective informers.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

PN-17 
Sixth Annual National Association of State Mental Health Program Directors (NASMHPD) Research Institute Proceedings on Mental Health Research and Evaluation: Creative Ideas for Austere Times-2/1996
Author: Presenters' materials compiled by the Evaluation Center@HSRI.
The proceedings contain papers and presentation materials from the Sixth Annual National Association of State Mental Health Program Directors Conference on State Mental Health Agency Services Research and Program Evaluation.
Publications can now be found on HSRI.org

PN-13 
Fiscal Year 1994 Directory of State Mental Health Agency Research & Evaluation Projects-1996
Author: The National Association of State Mental Health Program Directors (NASMHPD) Research Institute.
This directory contains information on over 800 projects conducted by, or in conjunction with, thirty-seven (37) state/territory mental health agencies. The information was compiled from a survey of all fifty-five (55) state and territorial mental health agencies. This book represents the most current version of a Directory of State mental health research and evaluation projects. Included with this book is a computer diskette containing the contents of the Directory in WordPerfect 5.1 format for keyword or phrase searches.
Publications can now be found on HSRI.org


TECScripts

The Topical Evaluation Network Program makes use of electronic mailing lists to allow subscribers who have access to the internet to participate in ongoing discourse about specific topics. Transcripts of on-line discussions as well as printed copies of archived documents are made available in TECScripts by mail to interested stakeholders, especially those who do not have internet access.

TS-10 
Who is Paying for Outcomes Management Programs?-6/2003
This is a compilation of unedited messages from the OUTCMTEN list that were received between June 13, 2003 and August 4, 2003. This discussion focused on methods of paying for service provider outcomes management programs.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-09 
Outcomes and Return on Investment (ROI)-5/2003
This is a compilation of unedited messages from the OUTCMTEN list that were received between March 25, 2003 and April 11, 2003. This discussion focused on incorporating outcome measurement activities into program evaluation, as well as ways to show organizations and funding sources that such activities show a return on investment.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-08 
Representative Sample for Outcomes-5/2003
This is a compilation of unedited messages from the OUTCMTEN list that were received between March 20, 2003 and March 25, 2003. This discussion focused on the practice of using outcome measures on a sample of program participants versus on the entire population of program participants.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-07 
Relationship Between Quality Improvement Activities and Relationship Between Quality Improvement Activities and Outcomes Outcomes Management-11/2002
This is a compilation of messages from the OUTCMTEN list that were received between November 19, 2002 and December 20, 2002. This discussion focused on how various quality improvement activities are defined and how they relate to outcomes management.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-05A 
Evidence-Based Treatments (EBTs)-1/2002
This is a compilation of unedited messages from the OUTCMTEN list that were received between January 12, 2002 and January 22, 2002. This discussion focused on the use of EBTs in clinical practice.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-05B 
Evidence-Based Treatments (EBTs) and Measuring Fidelity to EBTs-1/2002
This is a compilation of unedited messages from the OUTCMTEN list that were received between January 12, 2002 and February 3, 2002. This discussion focused on the use of EBTs in clinical practice and how to measure fidelity to EBTs.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-04 
Outcome Measures for Children and Adolescents-11/2001
This is a compilation of unedited messages from the OUTCMTEN list that were received between October 23, 2001 and November 13, 2001. This discussion focused on outcome measures for children and adolescents in the child welfare system. Some of the measures discussed are the Child Behavior Checklist (CBCL), Child and Adolescent Functional Assessment Scale (CAFAS) and the Ohio Youth Scales.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-06 
Objectives Based Treatment Plans-11/2001
An Unedited Compilation of Email Messages from the Outcomes Evaluation Topical Evaluation Network. Messages from 9 November 2001 – 19 December 2001
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-03 
The Need for Fundamental Change-4/2001
This is a compilation of unedited messages from the OUTCMTEN list that were received between April 24, 2001 and April 26, 2001. This discussion focused on suggestions made in the Institute of Medicine’s “Crossing the Quality Chasm: A New Health System for the 21st Century.”
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-02 
Written Treatment Plans and Mental Health Outcomes-3/2000
This is a compilation of unedited messages from the OUTCMTEN list that were received between March 21, 2000 and March 31, 2000. This discussion focused on “studies which examine the effects of a written treatment plan on outcome in mental health clinics.”
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

TS-01 
Dealing with Therapist Resistance to Outcomes-9/1999
This is a compilation of unedited messages from the OUTCMTEN list that were received between September 2, 1999 and September 16, 1999. This discussion focused on how to overcome therapist resistance to outcome measurement. The original author asked if therapists might be less resistant if outcome measures were a) truly useful to therapists in treating individual patients and b) actually helped improve their outcomes.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org


Evaluation FastFacts

Evaluation FastFacts is a 2-3 page briefing on new and current mental health services evaluations, resources, and/or methods. The function of this briefing is to be a quick and easy way for constituents to learn important and timely information in the field of evaluation. The following issues of Evaluation FastFacts are available by fax, mail ($1.00) or at our website. If you would like to subscribe to future issues, contact Atiya Dubose at (617) 876-0426 ext. 2501 or adubose@hsri.org.

FF-17 
Schizophrenia and suicide: a public health problem-10/2005 Volume: 4 Issue: 1
Author: Dow A. Wieman, Ph.D.
Recognizing that suicide is a multidimensional problem, SAMSHA, and other groups have funded interventions tailored to different populations. This article looks at the percentages of suicides that can be attributed to serious mental illness, using schizophrenia as an example. It also details a method that can be used to correlate state and mental health agency data, attributed to the Technical Evaluation Center, HSRI.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-15 
Case Mix Adjustment in Behavioral Health Care-11/2004
Author: Dow Wieman, Ph.D., H. Stephen Leff, Ph.D., Amy DiRamio, Sarah Witham, Eric Goplerud, Ph.D., and Elise Young, M.S.W.
The Evaluation Center@HSRI has just released Fast Facts issue #4 entitled "The most recent Evaluation Fast Fax discusses the use of case mix adjustment in behavioral health care. Also known as risk adjustment, case mix adjustment refers to statistical methods to controlling for group characteristics that that influence some outcome of interest. The FastFacts discusses uses in actuarial analyses such as capitation rate setting, services research (for post-hoc comparison of outcomes for non-equivalent groups), and in comparing groups for purposes of performance measurement. The FastFacts describes a toolkit produced by the Evaluation Center at HSRI, entitled Case Mix Adjustment in Behavioral Health, by Michael Hendryx Ph.D.
Publications can now be found on HSRI.org

PN-53 
The Performance Improvement Project: A Technical Assistance Manual-10/2004
Author: Dow A. Wieman, Ph.D., Robert Egnew, M.S.W., M.P.H., Ed Diksa, ScD
In 2003, the Centers for Medicare and Medicaid Services (CMS) promulgated new regulations requiring Medicaid managed behavioral health organizations to conduct annual Performance Improvement Projects (PIPs). This manual, developed in association with the National Association of County Behavorial Health Directors and the California Institute for Mental Health, provides a step by step approach to all phases of developing, implementing, and evaluating Performance Improvement Projects, with detailed information and resources presented in format and language designed to be readily utilized by local county-sponsored behavioral health authorities.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-14 
The National Registry of Effective Programs and Practice (NREPP): Recent Developments-7/2004
Author: The Evaluation Center@HSRI
The Evaluation Center@HSRI has just released Fast Facts issue #3 entitled "The National Registry of Effective Programs and Practices." This issue concerns recent developments and plans for the future of NREPP, a database of evidence-based practices for substance abuse and mental health treatment supported by the Substance Abuse and Mental Health
Publications can now be found on HSRI.org

FF-12 
Clinical Evidence Concise: A Model for the Dissemination of Evidence-based Treatment in Behavioral Health Care-2/2004
Author: The Evaluation Center@HSRI
The Evaluation Center@HSRI has just released a new issue of FastFacts entitled “Clinical Evidence Concise: A model for Dissemination of Evidence-Based Treatment in Behavioral Health Care”, (vol 3, issue 1). This briefing describes an initiative in general medicine named
Publications can now be found on HSRI.org

FF-11 
Not All Statistically Significant Differences Are Alike: Equivalence Analysis in Mental Health Evaluation-6/2003 Volume: 2 Issue: 4
Author: The Evaluation Center @ HSRI
Equivalence analysis is a new statistical tool for human services research. Stemming from pharmaceutical bioequivalence analysis, equivalence analysis offers a different perspective then traditional null hypothesis testing. More detailed and technical information about equivalence testing in psychosocial and services research may be found in the following references. The reader is also referred to the Evaluation Center@HSRI website, http://www.tecathsri.org. Coming soon: SPSS code for conducting equivalence analysis
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-09 
Measuring System Impacts on Mental Health Recovery:-2/2003 Volume: 2 Issue: 2
Author: The Evaluation Center @ HSRI
What is mental health recovery? What helps the individual in the process of recovery? What hinders? How do mental health systems and staff help, and how do they hinder? How can these effects be measured? A recent report entitled “Mental Health Recovery: What Helps and What Hinders?” represents the first step of a national project designed to address these questions.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-13 
The NTAC Training Curriculum for the Reduction of Seclusion and Restraint-9/2002
Author: The Evaluation Center@HSRI
THE NATIONAL ASSOCIATION OF STATE MENTAL HEALTH PROGRAM DIRECTORS (NASMHPD) and its National Technical Assistance Center for State Mental Health Planning (NTAC) have identified the creation of a violence- and coercion-free mental health treatment environment as a priority. Reducing and eventually eliminating seclusion and restraint is a
Publications can now be found on HSRI.org

FF-08 
HIPAA Transactions And Medical Codes: Implications For Behavioral Health Services Research And Evaluation-8/2002 Volume: 2 Issue: 1
Author: The Evaluation Center @ HSRI
This issue of Evaluation FastFacts addresses the HIPAA Transactions and Medical Codes and discusses their implications for behavioral health services research and evaluation.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-06 
CMHS Program Turns Knowledge Into Action-7/1999 Volume: 1 Issue: 6
Author: Susan Milstrey Wells
This issue summarizes an interview with Mike English, J.D., Director of the CMHS Division of Knowledge Development and Systems Change. Mike explains the Knowledge Development and Application (KDA) program.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-05 
Mortality Can Be A Powerful Performance Indicator-5/1999 Volume: 1 Issue: 5
Author: The Evaluation Center @ HSRI
The issue discusses the growing evidence that measures of mortality in a mental health service recipient population can be powerful indicators of system performance. Massachusetts’ Department of Mental Health is used as a case study.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-04 
Evidence Supports Medical Cost Offset-3/1999 Volume: 1 Issue: 4
Author: The Evaluation Center @ HSRI
This issue reviews the evidence for the potential of mental health services to reduce subsequent utilization and costs of other public services. The topic of this issue seems particularly timely considering the appearance of several articles on cost offset in March/April 1999 issue of Health Affairs.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-02 
Weaver Founds New Consumer Research Center-7/1998 Volume: 1 Issue: 2
Author: The Evaluation Center @ HSRI
This issue presents excerpts from an interview with Paul Weaver, Ph.D., Director of the newly formed Kentucky Center for Mental Health Studies, Inc. (KCMHS). KCMHS is a consumer-governed research institute designed to support mental health service consumers to conduct and collaborate in conducting mental health clinical and services research.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

FF-01 
Cochrane Library is Resource for Service Effectiveness-4/1998 Volume: 1 Issue: 1
Author: The Evaluation Center @ HSRI
This issue reviews the Cochrane Collaboration and Library, a resource for systematic reviews of literature on the effectiveness of mental health services.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org


Outlook

OT-03 
Outlook 3-4/2002
Author: A joint publication of the Center and NASMHPD Research Institute (NRI)
The issue highlights a few presentations and/or papers from the 2001 NRI conference. The focus of the conference was on “How Do We Stop Having the Same Conversations?” The presentations and/or papers point to different but related missing links in developing, testing, and implementing evidence-based practices.
Publications can now be found on HSRI.org
Publications can now be found on HSRI.org

OT-02 
Outlook 2-6/2000
Author: A joint publication of the Center and NASMHPD Research Institute (NRI)
This issue highlights a few presentations and/or papers from the 2000 NRI conference. The focus of the conference was on "Turning What We Know Today Into Better Research and Services For Tomorrow." The presentations and/or papers include the intersection between Addiction, Mental Health and Trauma, Mixed Methods in Mental Health Services Research, and On the Care and Feeding of Qualitative Researchers.
Publications can now be found on HSRI.org

OT-01 
Outlook 1-6/1998
Author: A joint publication of the Center and NASMHPD Research Institute (NRI)
This inaugural issue highlights articles with a focus on Performance Measurement. The articles discuss the development of performance measurement in mental health systems and address possibilities for a synthesis. Additional publications in the portfolio include updates on the NRI and NASMHPD Managed Care Performance Indicator Project
Publications can now be found on HSRI.org


Presentations

PS-04 
Should An FDA-like Process Be Adopted for Getting To Evidence-based Practices in Public Mental Health?-1/2100
Author: Steve Leff, Ph.D.
Health care systems throughout the U.S. are adopting evidence-based practices. This includes public mental health systems that serve primarily persons with severe and persistent mental illness. This think tank was designed to address whether an FDA-like process involving an ordered set of evaluations that make explicit criteria for determining an intervention’s safety and effectiveness should be established for public agencies to use in identifying evidence-based practices. This process would be crucial in deciding and in justifying which practices are encouraged and funded. An explicit process might also establish a fair and accessible pathway for clinicians, advocates, policymakers and evaluators to follow in demonstrating that interventions are safe and effective. The think tank addressed reasons why this might not be a good or practical idea. Participants were also be asked to envision such a process and what organization(s) might be responsible for the process (e.g., a federal agency, research collaboratives, or professional organizations, etc.)
Publications can now be found on HSRI.org

PS-07 
Identifying And Developing Evidence-Based Practices-9/2005
Author: Steve Leff, Ph.D.
Identifying And Developing Evidence-Based Practices
Publications can now be found on HSRI.org

PS-06 
The Second-Generation MHSIP Report Card-5/2003
Author: Dow Wieman, Ph.D.
This presentation was given at the National Conference on Mental Health Statistics in May, 2003. It focuses on the distinction between MHSIP Handbook 1 and the current version. Included are discussions about the new knowledge incorporated into MHSIP 2 such as survey methodology, use of consumers for data collection, measuring support for recovery, measuring cultural competence and other common measures.
Publications can now be found on HSRI.org

PS-05 
A Brief History of Evidence-based Practice and a Vision for the Future-2/2003
Author: H. Stephen Leff, Ph.D., Kevin Hennessy, Ph.D.
This presentation was given at the NRI (NASMHPD Research Institute)'s Annual Conference in February, 2003. It outlines the history and evolution of Evidence-based practices in mental health and addresses some concerns posed by Consumers, Providers, scientists/evaluators, multicultural groups and organizations.
Publications can now be found on HSRI.org

PS-03 
A Brief History of Evidence-based Practice and a Vision for the Future-2/2003
Author: Steve Leff, Ph.D., Kevin Hennessy, Ph.D.
National Research Institute Annual Conference 
February 11, 2003. Washington D.C.
Publications can now be found on HSRI.org

PS-02 
Getting to Evidence-Based Practices-6/2002
Author: Clifton M. Chow
International Association of Psychosocial Rehabilitation Services (IAPSRS)  27th Annual Conference 
Toronto, Ontario Canada  June 14, 2002
Publications can now be found on HSRI.org

PS-01 
Toolkit on Manuals and Workbooks for Psychosocial Interventions-1/2002
Author: Terry Camacho-Gonsalves, Ph.D.
PRISMe Meeting, January 24, 2002
Publications can now be found on HSRI.org