Chapter 36 OUTCOME EVALUATION
INTRODUCTION
The demonstration of quality of care requires evaluation from a number of aspects. Outcome measurement can occur at a variety of levels (government, service and individual), and can involve participation of the older person with a mental health problem, the carer and the mental health worker. This chapter briefly examines what type of outcome evaluation data are being collected at a national level, and then goes on to the other areas of symptom reduction, quality of life, and consumer and carer satisfaction. Constantly evaluating health services and implementing the recommendations from the evaluations will ensure that high-quality and responsive services are delivered.
LARGE-SCALE OUTCOME MEASUREMENT
People with mental health problems find benefit in routine outcome measurement and believe it leads to improvement in care (Guthrie et al 2008).
Health of the Nation Outcome Scales (HoNOS 65+)
The Health of the Nation Outcome Scales (HoNOS) (Wing et al 1996) are a British development designed to gather information from key areas of mental health and social function for the purpose of comparing mental health services and monitoring changes both within individuals and within services. Although the HoNOS were developed for the full age range, by 1999, arguments had been put forward that the scales needed to be modified to be more suitable for the assessment of older people with mental health problems. Modifications and revisions were made to capture greater rating accuracy for depression and dementia, while the essential structure of the HoNOS was maintained. This work produced the HoNOS 65+ (Burns et al 2004, HoNOS 65+ Implementation Group 2002).
HoNOS and various derivatives have been in routine use in many Australian mental health services since 2000. It appears that, in this country at least, the data are being collected, but they are only utilised in very general terms for service evaluation, rather than clinical management or service development (Andrews & Page 2005). This may be due to a number of problems with data collection, including:
Overall, as an outcome measure for inpatient and community services, it has been established that the HoNOS 65+ is acceptable to mental health workers, has satisfactory reliability, is sensitive to change, and has good concurrent validity and interrater reliability. However, it lacks content validity, construct validity, predictive validity and test–retest reliability (Cheung & Strachan 2007, Spear et al 2002, Turner 2004). The HoNOS 65+ is a global assessment tool; therefore, if some areas of assessment require other information, a more specific screening and assessment tool should be used.
The items are as follows (HoNOS 65+ Implementation Group 2004):
There are a few inpatient/community studies that have reported on the use of the HoNOS 65+. McKay and McDonald (2008) described the experience of using the HoNOS 65+ in a small integrated OPMHS in Sydney, Australia. The HoNOS 65+ data had been collected for many years, but they were not being utilised at a local level. A service review recommended integrating outcome measures into routine clinical practice to enhance individual care, team and service operations. At the individual care level, a ‘key item’ score is derived by clinical team consensus regarding the older person’s overall functioning and the main clinical issue that involves the OPMHS. The ‘key item’ is the most prominent HoNOS 65+ item (e.g. if the admission is for aggressive behaviour it will be item 1).
In another Australian study, Spear et al (2002) found the HoNOS 65+ is sensitive to change and that the community-based OPMHS was having a positive impact on treatment for mental health problems. A study conducted in the United Kingdom (MacDonald 2002) established that the HoNOS 65+ can determine:
Life Skills Profile 16 (LSP–16)
The Life Skills Profile 16 (LSP–16) is a shortened version of the LSP–39 (Rosen et al 1989). It measures the level of disability experienced by people with a mental health problem, particularly those living in the community. It is a 16-item scale administered by a mental health worker. The LSP–16 has four subscales, examining the domains of withdrawal, self-care, compliance and antisocial behaviour. The reference period is the previous 3 months. Information is obtained by direct observation or from other people who are in contact with the older person. It can be completed in 5 minutes. The scale has been rated well for clinical utility and feasibility (Pirkis et al 2005). Although recommended for use with older people in the NOCC (Pirkis et al 2005), it does not appear to have been tested specifically with OPMHS users.
Resource Utilisation Groups–Activities of Daily Living (RUG–ADL)
The Resource Utilisation Groups–Activities of Daily Living (RUG–ADL) is a component of the casemix tool RUG–III developed for use in North American RACFs (Fries et al 1994). It was conceptualised from the point of view that diagnosis is a poor predictor of resource use. Factors like objectives of care, functional deficits, rehabilitation problems and behavioural problems are better predictors. The RUG–ADL focuses on basic activities of daily living, such as toileting, bathing, eating and mobility. These four items are rated by a mental health worker and the reference period is ‘current status’. Psychometric testing has centred around the RUG–III as a casemix tool, rather than the RUG–ADL as an outcome measure. With that in mind, its content validity has been criticised for not being adequate for mental health problems such as dementia. It has good interrater reliability, but there has been limited or no other standard psychometric testing (Pirkis et al 2005).