CISCA Validity Reliability and Performance Analysis | May 2020

Introduction

Following the creation of CISCA, a series of analyses were run to assess the reliability and validity of the tool to ensure it accurately and empirically measures care capability.

Method

CISCA was administered to a sample of 339 staff and 354 consumers and family members across 14 teams of retirement living and aged care providers in Western Australia. Following data cleansing and initial testing an initial exploratory factor analysis was conducted on the data using a maximum likelihood extraction with direct oblimin rotation. Further regression analyses were conducted to further assess the tool’s validity.

Results

The Exploratory Factor Analysis revealed the CISCA survey accurately and reliably measures 5 care capability areas that also model the same 5 leadership capability areas from CILCA as hypothesized.

Caring with purpose had the lowest reliability and while it’s still acceptable (above 0.7) we have written three new questions to be included to improve the quality of this factor. We will conduct further reliability assessments on future samples to ensure these three questions achieve the goal of improving the overall reliability of Caring with Purpose.

The Regression Analysis found significant causational relationships between CISCA performance and organizational data as follows:

Team performance on CISCA can explain employee workplace culture scores by 27%

• As scores on ‘Caring for Team’ increase, employees’ perceptions of workplace culture increase by 25%.
• As scores on CISCA domain ‘Caring with integrity’ increase, employee’s perceptions of workplace culture increase by 17%

Team performance on CISCA can explain employee job satisfaction by 23%

• When team cohesion is high, job satisfaction increases by 22%
• When teams Care with Integrity, job satisfaction increases by 11%
• As teams Care with Purpose, employee’s job satisfaction improves by 14%
• When teams Care Safely, job satisfaction increases by 9%
• Team performance impacts personal / sick leave
• As teams Care less with Integrity, absenteeism and sick leave increases by 1.4%

Comparing site comparison data, we found Engagement correlated with the consumer experience, measured by Net Promoter Score and Customer Satisfaction. Note: this result is only based on a sample of 5 sites and caution must be applied when making conclusions about these relationships until we have a larger sample.

Discussion

The results indicate that the comprehensive CISCA tool accurately measures some key factors contributing to overall performance including engagement, culture and front line care capability.

It also provides evidence that investing in improving Care Capability is likely to improve staff experience which is also likely to improve the consumer experience and overall care outcomes.

Further research into this will be conducted with a bigger data set in the future.

Learn more about our CISCA Assessment here