The Center for Nursing Research & Innovation  

Collaborative Alliance for Nursing Outcomes

CALNOC Ops Team, left to right:
Patricia McFarland, MSN, RN (Assoc. of Calif. Nurse Leaders),
Nancy Donaldson, RN, DNSc, FAAN (UCSF School of Nursing),
Diane Storer Brown PhD, RN, CPHQ, FNAHQ, FAAN (Kaiser Permanente No. Calif. Region),
Carolyn E. Aydin, PhD, and
Linda Burnes Bolton, Dr.P.H., RN, FAAN (Cedars Sinai Medical Center)

CALNOC Website (www.calnoc.org)

Announcement

Nursing Data on Patient Safety in California Will Expand to Include Two Pacific Northwest States
CALNOC Press Release 12/10/08 (.pdf)

Introduction

The Collaborative Alliance for Nursing Outcomes (CALNOC; formerly known as the California Nursing Outcomes Coalition) was launched in 1996 as the nation's first ongoing nursing quality database. CALNOC has a current membership of 225 hospitals from California, Washington, Oregon, Arizona, Nevada, and Hawaii. Self-funded by participant annual data management fees, contracts and grants, CALNOC is a joint venture between the Association of California Nurse Leaders and the California state affiliate of the American Nurses Association, ANA\California. CALNOC was established as one of six initial American Nurses Association Nursing Quality Indicator Report Card research and development pilot projects. Once enrolled, hospital participant attrition has been low, with less than 5% of member hospitals withdrawing since 1998. The demographic characteristics of CALNOC participating hospitals demonstrate the diversity of its member institutions.

Evolution

During its first decade, CALNOC surmounted inherent threats to the integrity of multi-site quality report card efforts identified by Jennings et al (Jennings, Loan et al. 2001), including indicator measurement standardization, feasible data retrieval, strategic relevance, and linking the work to the contemporary demands of clinical nursing. CALNOC’s vision is to advance patient care excellence. Its mission is to add value to participating hospitals and the healthcare system by:

• Collaborating with participating hospitals to support regional and national patient care safety, outcomes and performance measurement efforts. 
• Advancing research and evidence-based practice to optimize patient care excellence.
• Providing participating hospitals with actionable data to guide decision making and performance improvement.
• Building and sustaining a state of the art nursing outcomes database.
• Synthesizing & disseminating data to inform public policy, practice & education.

CALNOC Today

CALNOC has become a dynamic, professional research and development collaboratory with a 13-year history of robust nursing-sensitive quality measurement, benchmarking, web-based reporting innovation and research. CALNOC's measures, processes and findings contributed to the body of evidence considered by the NQF in its quest  to establish the nation’s first voluntary consensus standards for nursing sensitive care and the resulting initial set of 15 measures for hospital quality performance measurement included a measure (restraint use prevalence) attributed to CALNOC (National Quality Forum 2004). CALNOC has contributed intellectual property and methodological expertise to the establishment of the ANA's National Database for Nursing Quality Indicators (NDNQI), and provided methodological consultation to the Military Nursing Outcomes Database and the Veterans Administration Nursing Outcomes Database.

CALNOC has grown concurrently with exponential demand for data from hospitals. Since its inception, CALNOC has aggregated over 10 years (42 quarters) of data, representing nurse staffing data for >1,300 patient units and almost 46 million patient days. Falls data includes >130,000 patient falls, while over 315,000 patients have been evaluated for pressure ulcers and restraint use. The CALNOC Team maintains ongoing contacts with each participating hospital's Chief Nursing Officer, as well as their designated site coordinators. One example of CALNOC's strong operational support from member hospitals has been their support of emerging new initiatives and studies. For example, from 2001-03, 25 CALNOC hospitals volunteered to be study sites for CALNOC’s first AHRQ R01 research study, Unit Level Nurse Workload Impacts on Patient Safety Study.  From 2002-2004, 33 hospitals participated in the a second AHRQ funded study, CALNOC Partners to Reduce Patient Falls, contributing to understanding of how challenging it is to achieve changes in aggregated patient outcomes (Donaldson, Rutledge et al. 2008). CALNOC is currently recruiting over 100 medical-surgical nursing units from member hospitals to participate in a Robert Wood Johnson funded Interdisciplinary Nursing Quality Research Initiative (INQRI) study entitled "Measuring the Impact of Medical Surgical Acute Care Micro System Nurse Characteristics and Practices on Patient Outcomes."

Current CALNOC nursing-sensitive measures include quality indicators originally based on the ANA Acute Care Indicators advanced for testing in 1995 (American Nurses Association 1995); (American Nurses Association 1996a); (American Nurses Association 1996b). The proposed study builds on CALNOC data collection, reporting and educational methods that have been fully detailed elsewhere (Aydin, Burnes Bolton et al. 2004), (Donaldson, Brown et al. 2005), (Aydin, Burnes Bolton et al. 2008). CALNOC’s current measurement capacity includes the following indicators:

• Nurse staffing--direct care hours, skill mix, patient days, nurse/patient ratios, and contracted staffing utilization, workload intensity (admissions, discharges, transfers), staff voluntary turnover, and use of sitters
• RN education level, certification, and years of experience
• Patient falls--risk, incidence (rate per 1000 pt. days) and consequences
• Pressure ulcers--risk, prevalence, stage, and hospital acquired
• Pressure ulcers--incidence of reportable hospital acquired pressure ulcers stage 3 and above (new indicator implemented in 2009 as part of CALNOC's Robert Wood Johnson Foundation-funded INQRI [Interdisciplinary Nursing Quality Research Initiative] project)
• Restraint prevalence--type and clinical justification
• PICC-CABSI--prevalence on medical-surgical units
• Medication Administration Accuracy--observed prevalence of key safe practices and errors.

Design and Methods

CALNOC data are considered sufficiently robust to have been selected as a data source by the California Hospital Assessment and Reporting Taskforce (CHART) stakeholders, including consumer groups, employers, health plans, and hospital leaders, for the California 2006 hospital report card of carefully vetted clinical and patient measures. Affirmation of the validity of CALNOC methods and measures may be the research and measurement integrity of the other measure sponsors in CHART, including: The Joint Commission (TJC), Agency for Healthcare Research & Quality (AHRQ), Centers for Disease Control (CDC), and the California Office of Statewide Health Planning and Development. Procedures for all CALNOC data sharing are codified by well developed HIPAA compliant legal data-use agreements signed by CALNOC's member hospitals.
CALNOC has set the industry standard with its robust web-based reporting platform of nursing quality core measures, unique in its concurrent measurement of structures, processes and outcomes.

Findings

CALNOC has completed two intramural studies and a AHRQ funded study that are relevant to the proposed research effort. The first study builds on a preliminary report that examined the impact of mandated nursing ratios in California on key measures of nursing quality among adults in acute care hospitals. This self-funded study explored nurse staffing and nursing-sensitive outcomes comparing 2002 pre-ratios regulation data to 2004 post-ratios regulation data (Donaldson, Burnes Bolton et al. 2005); (Bolton, Aydin et al. 2007). For the 2007 follow-up and extension study, post-regulation ratios data from 2004 and 2006 were used to assess trends in staffing and outcomes. Findings for nurse staffing affirmed trends noted in 2005 and indicated that changes in nurse staffing were consistent with expected increases in the proportion of licensed staff per patient, however examination of the relationship between staffing and nursing-sensitive patient outcomes did not find anticipated improvements in patient outcomes.

The “CALNOC Partners for Quality to Reduce Patient Falls Project” (AHRQ PFQ Grant# 1U18HS1370401) was a four-year quality improvement demonstration project that used a novel telephone-based coaching intervention to support hospitals in systematic efforts to reduce the incidence of patient falls and severity of fall-related injury in California Hospitals. This demonstration project expanded and advanced CALNOC's efforts to use its quality benchmarking infrastructure as a vibrant network to expedite the transfer of evidence-based knowledge into practice as the basis for improving patient care quality and safety.

In conducting the CALNOC PFQ project, 92 medical surgical units drawn from 33 hospitals participated in a preliminary test of a coaching intervention to expedite falls improvement projects. Results from mixed regression models were used to examine trends in falls. Findings did not reveal hoped for significant differences in falls or fall injury rates. Changes in falls and injury rates were small and actual power to detect changes was low (8%). Baseline unit fall level was the best predictor of fall rate outcomes. Impact of adoption of evidence-based falls risk assessment tools reveals significantly fewer accidental falls. Analysis of the pre-post organizational level self assessment data reveals significant differences in a wide range of falls-related policies, protocols and clinician practices that would be expected to ultimately improve the rate of falls and fall related injuries.

Concurrently, in an effort to identify and investigate hospitals that appear to be "best practices" CALNOC has conducted a study of consistently best and worst performers for key outcomes measured in its dataset. Hospital case study approaches were used to better understand practices that determined sustained best quartile performance. The structure of care was explored related to nurse staffing skill mix, hours of care, and use of contracted staff. The process of care was explored related to risk assessment processes, risk identification, and implementation of preventive nursing interventions. Preliminary findings from our examination of best and worst performers for hospital acquired pressure ulcers on medical-surgical units (all units combined) showed that, as predicted, the best performers had a higher percent of care delivered by RNs and fewer patients per licensed staff. At the facility level, however, these findings were no longer significant. Finally, our facility-level examination of those hospitals with the lowest and highest use of patient restraints indicated that those with the lowest restraint use had more total hours of care and fewer patients per licensed staff than those with the highest restraint use. 

Research Team

The CALNOC Research Team is based in the University of California School of Nursing (UCSF) Center for Nursing Research and Innovation. UCSF School of Nursing was founded in 1939 as the first autonomous School of Nursing in any state University. The UCSF School of Nursing has 5 of 7 specialties ranked among the top 2 by the US News and World Report, and ranks first in NIH funding nationwide. UCSF is designated as a World Health Organization Collaborating Center in Nursing and is one of five John Hartford Centers for Geriatric Nursing Excellence.

The ongoing CALNOC project is conducted by the CALNOC Operations Team consisting of staff from the Center for Nursing Research and Innovation at University of California, San Francisco, The Association of California Nurse Leaders, Cedars-Sinai Research Institute, and representatives of the CALNOC User Members. Key CALNOC personnel including Dr. Nancy Donaldson (UCSF), Dr. Carolyn Aydin (Cedars- Sinai Research Institute) and Ms. Patricia McFarland (ACNL) coordinate and manage the work of CALNOC. The CALNOC Research Team, under the leadership of Co-Principal Investigators Drs. Nancy Donaldson and Diane Brown, is operationally coordinated by Dr. Donaldson, and is accountable for the integrity of CALNOC methods, studies and published reports. The Center works closely with the CALNOC Governance and Advisory Council that includes national experts, operational stakeholders, and public policy representatives and payers.  The proposed project will benefit from the interdisciplinary and seasoned expertise of the CALNOC team. It is noteworthy that the CALNOC Team has been recently selected by the American Academy of Nursing as the first team to be awarded Edge Runner recognition.

CALNOC Sites

CALNOC's partnership with its sites is codified by a fully executed Confidentiality and HIPAA Business Associate Agreement, a tie that binds the collaborative and establishes mutual understandings and accountabilities between partners and is signed by the Chief Nursing Officer and the Chief Executive Officer for each participating hospital.

Data Infrastructure

The CALNOC data infrastructure, including the database, the analytical software, and the website itself, is hosted at a centralized secure web-hosting facility on a high-speed dedicated server with adequate processing power and storage capacity for both the database and the website. The CALNOC System and its underlying data and program code are accessible to CALNOC researchers and the system development group (computer programmers and systems analysts) via a secure Virtual Private Network (VPN) link. CALNOC member hospitals access the reports authored by the programming team using Cognos, by specifying the desired data elements, time periods, facility, units, etc. 

Automated Excel data submission spreadsheets are pre-programmed with macros that check data at the hospital before the spreadsheet can be saved and submitted. The use of Excel is user-friendly for hospital site coordinators with varying skill levels and encourages the participation of small hospitals or hospitals with scarce resources that may not have extensive data processing capacity. Data from hospital automated systems can also be easily downloaded or pasted into the Excel files for submission. The spreadsheets are then emailed to the data entry mailbox on the CALNOC server and uploaded to the database using proprietary automated upload software.

Scannable data collection forms are used by hospitals to conduct pressure ulcer/restraint prevalence studies and are also an option for RN Education surveys. Hospitals have the option of mailing the forms to CALNOC for scanning and submission to the database if they do not have resources for data entry. Automated Excel data submission files are also available for hospital data entry. Scannable forms encourage participation of hospitals without extensive data processing resources.

Existing CALNOC hospital education programs, documentation and web-based tutorials support sites in existing indicator data capture, strengthening the reliability of measures in this study. In addition, CALNOC data reliability and validity are ensured in 4 steps:

• Hospital data entry--CALNOC uses the two data submission methods described above: CALNOC’s automated Excel data submission files and customized scannable forms. 
• Submission to CALNOC: Excel data submission files are emailed to the CALNOC data in-box, or Scannable forms are mailed to CALNOC and scanned into the database using CALNOC customized data checking.
• Error checks for all data uploaded to the CALNOC database include: correct unit identifiers, variable outliers, and validations specific to each indicator. Examples include, but are not limited to:
• Nursing hours per patient day validated against expected ranges by unit type and past performance for the same unit,
• Fall rates per 1000 patient days examined for outliers (high or low),
• Pressure ulcer prevalence coding checked for high/low outliers and for completeness.
• Data Management Validation--Following upload and data processing, additional data checks for outliers and invalid data are conducted by CALNOC Data Management staff and hospital representatives are contacted to resolve questions and make corrections.

Following final data processing, hospitals log on to the CALNOC website for customized reports and inherently examine accuracy of their own data and contact the CALNOC Data Management Team with any corrections. The CALNOC database continuously accepts data corrections, thus perpetually strengthening reliability and validity of the dataset.

References

Grobe SJ, Becker H, Calvin A, Biering P, Jordan C, Tabone S. Clinical data for use in assessing quality: lessons learned from the Texas Nurses' Association Report Card Project. Semin Nurse Manag. 1998 Sep;6(3):126-38.

Redmond G, Riggleman J, Sorrell JM, Zerull L. Creative winds of change: nurses collaborating for quality outcomes. Nurs Adm Q. 1999 Winter;23(2):55-64.

American Nurses Association (1995). Nursing care report card for acute care. Washington D.C., American Nurses Publishing.

American Nurses Association (1996a). Nursing quality indicators: Definitions and implications. Washington D.C., American Nurses Publishing.

American Nurses Association (1996b). Nursing quality indicators: Guide for implementation. Washington D.C., American Nurses Publishing.

Aydin, C., L. Burnes Bolton, et al. (2004). "Creating and analyzing a statewide nursing quality measurement database." Journal of Nursing Scholarship.

Aydin, C., L. Burnes Bolton, et al. (2008). Beyond Nursing Quality Measurement: The Nation's First Regional Nursing Virtual Dashboard. Advances in Patient Safety: New Directions and Alternative Approaches. K. Henricksen, J. Battles, M. A. Keyes and M. L. Grady. Rockville, MD, Agency for Healthcare Research and Quality. 1: 217-234.

Bolton, L. B., C. Aydin, et al. (2007). "Mandated Nurse Staffing Ratios in California: A Comparison of Staffing and Nursing-Sensitive Outcomes Pre- and Postregulation,  ." Policy, Politics and Nursing Practice 8(4): 238-250.

Donaldson, N., L. Burnes Bolton, et al. (2005). "Impact of California/s licensed nurse-patient ratios on unit level nurse staffing and patient outcomes." Policy, Politics & Nursing Practice 6(3): 198-210.

Donaldson, N. E., D. S. Brown, et al. (2005). Final Report: Impact of Unit Level Nurse Workload on Patient Safety, AHRQ: 1-20.

Donaldson, N. E., D. N. Rutledge, et al. (2008). The Role of the Coach in Advancing Research Translation. Advances in Patient Safety: New Directions and Alternative Approaches. K. Henricksen, J. B. Battles, M. A. Keyes and M. L. Grady. Washington, D.C., AHRQ. 1: 285-302.

Jennings, B. M., L. A. Loan, et al. (2001). "Lessons learned while collecting ANA indicator data." J Nurs Adm 31(3): 121-9.

National Quality Forum (2009) "National Voluntary Consensus Standards for Nursing-Sensitive Care Performance Measure Set Maintenance."  Volume,  DOI:

Back to top- - -