Can the Federal Baldrige Survey Measure Workforce Well-being in an Academic Health Center?

Introduction
Experts suggest health care institutions switch focus from measuring burnout to measuring positive organizational psychology. Concerns include burnout being a late sign of organizational decline. The Baldrige survey is promoted by the U.S. Department of Commerce to measure positive worksite conditions (e.g., workforce wellbeing of industries, including health care and education). For years, the survey has been completed by managers within organizations, but now the same survey is promoted for completion by an organization's workforce. We tested the structure of the Baldrige survey when completed by an academic health care workforce. In addition, we tested whether the results in an academic worksite correlate with an example metric of an organizational mission.


Methods
In 2015, our academic health center surveyed faculty and staff with the Baldrige survey. The validity of the Baldrige was tested with confirmatory factor analyses. Within the School of Medicine, responses for the Baldrige's concepts were correlated against a measure of organizational outcome: graduates' assessments of Departmental educational quality.


Results
The structure of the Baldrige survey did not validate when assessed by a workforce (RMSEA = 0.086; CFI = 0.829; TLI = 0.815). None of its concepts correlated with learner reported educational quality.


Conclusions
The Baldrige survey, when administered to a workforce rather than managers, did not appear to measure workforce well-being within an academic health care center. We discourage use of the current survey for this purpose.


INTRODUCTION
The well-being of physicians in the United States, as measured by rates of burnout, is declining. 1 Accordingly, the well-being of health care personnel has been proposed as a fourth aim of health care in addition to the health of a population, the patient experience of care, and the cost of care. 2,3 The health care system's focus on burnout, rather than positive organizational psychology, has been questioned. 4,5 Burnout is likely an end state in organizational decline, so focusing on burnout may delay detection of organizational dysfunction.
The Baldrige survey is an excellent candidate for measuring positive organizational health and has been used by industries, including healthcare and education. 6 The Baldrige framework and criteria for performance excellence was created by the Malcolm Baldrige National Quality Improvement Act of 1987 and is managed and its use encouraged by the National Institute of Standards and Technology (NIST) within the U.S. Department of Commerce. The survey and national benchmarks are freely available. The Baldrige institutional assessment includes a questionnaire that measures seven concepts of organizational tactics. While the survey was originally intended to be completed by managers within an organization applying for the Malcolm Baldrige National Quality Award, administration of the survey to front-line personnel is now encouraged by the NIST. 7 Although the survey is encouraged for front-line personnel by the NIST, the survey's structure has not been validated for this purpose. The objective of this study was to assess the structural validity of the Baldrige questionnaire when front-line personnel of an organization are queried. In addition, we correlated the concepts within the Baldrige with a measurement of an institutional goal.

METHODS
Study design and setting. An analysis of existing, anonymous data that had been collected for operational purposes by the Organizational Improvement Office (OIO) at the University of Kansas Medical Center was performed.
Participants. Invitations to the online, anonymous survey were emailed by OIO to all faculty and staff in 2015. One reminder email was sent.
Measurements. We used the Baldrige "Are We Making Progress" 2011 questionnaire (Table 1). The questionnaire contains 40 Likert questions that query the presence of four to nine positive attributes per seven concepts (leadership, strategy, customer focus, workforce focus, information management, operations, and results).
The organizational outcome of interest was educational quality which was measured using the ratings of clinical clerkships by recent graduates of the school. The ratings were gathered from the 2014 Association of American Medical Colleges graduation questionnaire (AAMC-GQ). 8 The AAMC-GQ provides separate ratings for clerkships in internal medicine, family medicine, surgery, pediatrics, psychiatry, and obstetrics and gynecology. The ratings were normalized by using the national percentile ratings for each clerkship rather than use the raw ratings by our graduates.
Statistical methods. A confirmatory factor analysis (CFA) validated the structure of the Baldrige questionnaire. Statistical fit of the model was evaluated with the root mean square error of approximation (RMSEA), comparative fit index (CFI), and the Tucker-Lewis index (TLI). Acceptable fit is indicated by RMSEA less than 0.06, CFI above 0.90, and TLI above 0.90. 9 Analyses were done with the Lavaan Package for R Programming Language. 10 BALDRIGE SURVEY continued. For departments or divisions that sponsor a clinical clerkship within the School of Medicine, mean values of responses were calculated by personnel within the work unit to each item within the seven Baldrige concepts. Then, a mean for each Baldrige concept was calculated. The means of the responses were correlated for each concept to graduates' assessments of departmental educational quality. Calculations were done with the R Programming Language. 11

RESULTS
Responses were received from 877 faculty and staff for a response rate of 21%. The Baldrige did not validate by confirmatory factor analysis with all measures of fit not meeting thresholds for validity (RMSEA = 0.086; CFI = 0.829; TLI = 0.815).
None of the seven concepts of the Baldrige as assessed by departmental personnel significantly correlated with educational quality as assessed by recent graduates ( Table 2). The range of correlation coefficients ranged from -0.01 for the concept of "Results" to 0.58 for the concept of "Customer Focus".

DISCUSSION
Our academic health center, which like all academic health centers combines both delivery of health care and provision of higher education, did not validate the structure of the Baldrige for measuring its seven concepts. In addition, the Baldrige concepts did not correlate with our measure of an organizational goal, the AAMC-GQ. The negative results may reflect that most prior studies of the Baldrige queried managers and external assessors of organizations rather than front-line personnel. [12][13][14][15][16][17][18][19][20][21][22] In addition, most of these studies created custom surveys based on Baldrige concepts. 13,14,16,20,22 Four previous studies attempted validation of the original Baldridge questions. 12,15,21 Two studies surveyed leaders or managers in diverse industries and were able to validate the survey after modifying the structure. 12,15 In a third study, Jayamaha et al. 21 surveyed Baldrige personnel who formally assessed companies that applied for the Baldrige award. They found low discriminant validity suggesting questions belonged to multiple concepts. The only study that surveyed frontline personnel, like our study, was not able to validate the relationships between concepts of the Baldrige model. 23 Workforce conditions should move beyond current recommendations to measure burnout and instead measure positive organizational psychology. 4,5 Unfortunately, the Baldrige does not support this goal. Other studies support the concept of thriving (defined as a workforce that is both engaged and learning or improving) should be measured. 24,25 Thriving has been studied in industries other than health care and found to correlate with job performance of both individuals and groups. 24,25 On the other hand, burnout, while correlated with quality of care as perceived by physicians, 26 did not correlate with measured quality of care in the "Minimizing Error, Maximizing Outcome" study 27 or Healthy Work Place trial. 28 In addition, as previously noted, focusing on burnout may delay detection of organizational dysfunction. In addition to helping academic health centers meet organizational goals, successful measurement of workforce wellbeing may have larger impact by using controlled, public reporting to address physician burnout in clinical practice. 3 Our study is limited by a low response rate. However, this rate is typical of national studies of burnout. 1 With a larger study size, our correlation of 'customer focus' of departments with recent graduates' satisfaction with departmental clerkships might become statistically significant. However, even if this correlation is significant, the structure of the Baldrige does not validate and better surveys should be sought. In 2015, the Baldrige survey was revised; however, only a single question was reworded.
Our negative finding regarding the Baldrige's inability to measure the perspective of front-line personnel should not be generalized to other roles of the Baldrige framework and awards. For example, other studies using external examiners show receipt of the Baldrige award in health care correlates with organizational financial performance 29 and positive experiences by patients. 30 However, as noted previously, the only prior study that attempted validation of the structure of the Baldrige survey to measure the perspective of front-line personnel using the original questions in the survey was also negative.

CONCLUSIONS
We discourage use of the Baldrige survey to measure employee perceptions of well-being in academic health centers. The lack of validation studies of front-line personnel in any industry questions the use of the Baldridge by frontline personnel in any setting. Our negative findings are important as workforce well-being is an emerging issue and the NIST is promoting the Baldrige "Are we making progress" survey for measuring the front-line perspective.