A multidimensional approach to the structure of consumer satisfaction with vocational rehabilitation services.
by John F. Kosciulek
Rehabilitation Counseling Bulletin,
Wntr 2003 v46 i2 p92(6)
Full Text: COPYRIGHT 2003 Pro-Ed
The purpose of this investigation was to examine the dimensions underlying consumer satisfaction with vocational rehabilitation services. Participants were 223 consumers who received services from a state rehabilitation agency in a Midwestern state. Data were gathered via mail survey through participant completion of a self-report questionnaire. A two-dimensional configuration of 14 consumer satisfaction stimuli yielded the following dimensions: (a) satisfaction related to case management versus employment and (b) satisfaction related to consumer choice versus customer service. Findings suggest that consumers view satisfaction as a multidimensional construct. Implications for the measurement of consumer satisfaction and use of consumer satisfaction data in vocational rehabilitation program evaluations are discussed.
Due to the significant lack of progress in promoting the employment of persons with disabilities during a period of unprecedented economic growth in the 1990s, current public disability policy and accreditation mandates have greatly increased the requirements of vocational rehabilitation (VR) service providers and funders to demonstrate positive outcomes (O'Day, 1999; Seelman, 2000). For instance, the Rehabilitation Act amendments of 1992 and 1998 increased program evaluation documentation requirements for state rehabilitation agencies (Council of State Administrators of Vocational Rehabilitation [CSAVR], 1998). Similarly, recent standards established by the Commission on the Accreditation of Rehabilitation Facilities (CARF, 2000) now require community rehabilitation programs to document program effectiveness more thoroughly through ongoing outcome assessments. Furthermore, adopting a market-based approach to reimbursement for VR service provision, the Ticket to Work and Work Incentives Improvement Act of 1999 (TWWIIA) connects payment for services to quality employment outcomes for service recipients (Hoff, Vamey, Marrone, Butterworth, & Silverstein, 2001; Kosciulek, 2000).
One area of outcome evaluation that has received significant attention in terms of VR program effectiveness is consumer satisfaction. Koch and Merz (1995) aptly described how the importance of measuring consumer satisfaction has grown during the past decade as a result of federal legislation and the emergence of consumer choice approaches in vocational rehabilitation. Richard (2000) reported that effective consumer satisfaction assessment practices were necessary to promote consumer empowerment philosophies. Similarly, Kosciulek, Prozonic, and Bell (1995) emphasized that consumer involvement and the increased demand for accountability has resulted, in a need for VR programs to demonstrate high levels of consumer satisfaction with services. Further, Schwab and Fenoglio (1992) stated that a high level of consumer satisfaction was one of the key features of a quality rehabilitation system. Finally, a major conclusion of the CSAVR (1998) Committee on Client Services Consumer Satisfaction Report was that consumer satisfaction is one of the primary variables for evaluating the effectiveness of services and documenting positive rehabilitation outcomes (CSAVR, 1998). In this report, the CSAVR described the critical roles consumer satisfaction data play in state VR agency program implementation. The functions of consumer satisfaction data identified in this report included the following: (a) service improvement, (b) program planning and revision, (c) in-service training, (d) counselor performance appraisal, and (e) agency performance appraisal in relation to state and federal regulations. Thus, the literature has indicated that consumer satisfaction is an essential component to consider when assessing the efficacy of VR services.
Given its significance to the assessment of VR programs, accurate and comprehensive evaluation of consumer satisfaction is imperative. However, one of the criticisms of consumer satisfaction program evaluation research has been that there is a lack of sophisticated and holistic measurement of the construct. Numerous researchers have reported a dearth of empirical literature that captures the potential complexity of the construct (e.g., Janikowski; Bordieri, & Musgrave, 1991; Koch & Merz, 1995; Kosciulek, Vessell, Rosenthal, Accardo, & Merz, 1997; Richard, 2000). Janikowski et al. (1991) succinctly concluded, "Very little research has been conducted examining the varying dimensions of client satisfaction with rehabilitation programming" (p. 43). Similarly, Richard (2000) stated that VR researchers and program evaluation specialists too often have restricted themselves to using single global measures of consumer satisfaction. Thus, the need exists for reliable and valid data on the multiple cognitive constructs that VR service recipients use to arrive at their opinions of satisfaction or dissatisfaction.
Identifying and measuring multiple dimensions of consumer satisfaction would provide VR personnel with a comprehensive and accurate view of consumer perceptions of the services they receive (Richard, 2000). For example, from a pragmatic perspective in response to regulatory mandates, the CSAVR (1998) reported that consumer satisfaction surveys must address all the major aspects of the VR service delivery process, including interface with the VR system, the specific services received, service providers, consumer choice, and rehabilitation outcomes. More precise consumer feedback regarding satisfaction with the counselor's role as a case manager, informed choice opportunities, and the quality of employment outcomes would yield specific data useful for improving VR services.
The purpose of this investigation therefore was to respond to the need for additional information on the multiple dimensions that encompass consumer satisfaction. The specific intent of the study was to identify the dimensions that underlie consumer satisfaction with VR services. It was planned a priori for this study to be an initial step in identifying consumers' perceptions of satisfaction and to encourage researchers and program evaluation specialists to move toward the development of multi-dimensional measures of consumer satisfaction with VR services. It is anticipated that as specific dimensions that make up the construct of consumer satisfaction are identified, defined, and reliably and validly measured, useful information will be obtained to guide service provision modification that would both increase satisfaction levels and enhance employment outcomes for consumers with disabilities.
The purpose of descriptive research is to make assertions about particular aspects of populations or samples (Cozby, 1993). As an inquiry in the area of consumer satisfaction with VR services, the current study attempted to elicit salient features of consumer satisfaction. A descriptive research design was necessary to depict dimensions of satisfaction among a sample of consumers who had received VR services.
Participants were recruited from a state rehabilitation agency in a midwestern state that serves individuals with visual impairments. All consumers (N = 812) whose cases were closed by the agency during Fiscal Year (FY) 1999 were mailed a Demographic Information Sheet (DIS), the 14-item Consumer Satisfaction Survey (CSS), and a cover letter describing the study and ensuring confidentiality and anonymity. To maximize the overall response rate, a second set of study materials was mailed to individuals who had not returned the original set of materials within 2 weeks of the first mailing.
Because the selected state agency served individuals with visual impairments, a significant effort was made to make study materials and processes accessible to all consumers. First, as recommended by the agency's Rehabilitation Advisory Council, study materials were mailed in large-print format. Second, consumers were given the option of completing the survey via Braille format or through a telephone interview. Of the 812 study packets originally mailed, 82 were returned as not deliverable. Thus, 730 packets were delivered to the most current consumer addresses available.
Three consumers requested, were provided, completed, and returned study materials in Braille format. Five consumers requested and completed the 14-item CSS and the DIS via telephone interview. An additional 261 study packets were returned to the investigator. Of these packets, 26 CSSs were returned with no data and 20 were returned with missing data (i.e., not all 14 items completed). These 46 survey sets were not included in the final data analysis. The 20 CSSs with missing data were not included due to the full data matrix requirement of the statistical technique used in this study. As a result, full data sets from a total of 223 consumers, or 30.5% of the 730 possible participants, were used to identify the dimensions that underlie consumer satisfaction with VR services.
The population of interest in this study was consumers with disabilities who received VR services. The sample was drawn from consumers whose cases were closed by a state rehabilitation agency in a midwestern state during FY 1999. The participants were 223 consumers who completed and returned a survey of their level of satisfaction with the VR services provided by the state rehabilitation agency. Of the participants, 127 (57%) were women and 96 (43%) were men. In terms of race, 167 (75%) participants were ,White, 49 (22%) were African American, and 7 (3%) were Asian American. One hundred thirteen (51%) participants had never been married, 74 (33%) were married, 32 (14%) were divorced, and 4 (2%) were widowed. The participants' ages ranged from 18 to 56 years, with a mean of 29.25 years and a standard deviation of 4.31 years. Participants averaged 12.14 years of education completed, with a standard deviation of 2.79 years.
The primary data source in this investigation was the 14-item CSS used by a state rehabilitation agency to evaluate the level of consumer satisfaction with VR services. (The 14 survey items are included in Figure 1.) In addition, the DIS was developed by the investigator to collect demographic information (e.g., age, marital status) from participants. Data from the DIS were used to describe participant characteristics.
Participants were asked to rate each of the 14 satisfaction items on a 4-point Likert-type scale (1 = strongly agree, 4 = strongly disagree), indicating their level of satisfaction with each service delivery aspect. It is important to note that the Rehabilitation Advisory Council of the state rehabilitation agency developed the 14 items used as the stimuli for data analysis. Given the input of consumers and direct service providers (e.g., rehabilitation counselor) into their development, the satisfaction items related directly to the context of the VR services provided by the state agency. Such an item development process is consistent with the recommendations of Kosciulek (1999) and Zimmerman (1995), who both emphasized that open-ended constructs such as consumer satisfaction must be measured in context in order yield valid and useful results. The internal consistency reliability (Cronbach's alpha) of the CSS for the current study was .94. According to Power (2000) and Thorndike (2001), this finding indicates that the CSS had a high level of reliability in its application with the sample in the present study.
In accordance with the purpose of the investigation and the nature of the descriptive research design, multidimensional scaling (MDS) was used to evaluate the structure of consumer satisfaction with VR services. MDS is useful for obtaining information about the structure of a data set (Davison, 1992; Forgas, 1979). Given that the research literature emphasizes the complex nature of consumer satisfaction (CSAVR, 1998; Koch & Merz, 1995; Richard, 2000), MDS was identified as the most appropriate procedure to examine the data obtained in this study. The use of MDS enabled the identification of underlying dimensions that might not be apparent in the data with more conventional methodologies.
MDS is a mathematical procedure that systematizes data by representing the similarities of objects on a spatial map, or configuration, in which stimuli judged to be similar are points close to each other on the map and stimuli judged dissimilar are points distant from one another (Schiffman, Reynolds, & Young, 1981). MDS spatial representations differ from one-dimensional scales in that the former are able to portray the complexity (i.e., the underlying structure) of the data. Specifically, MDS uses iterative numerical methods to estimate the spatial coordinates for each of the stimuli in space. The resulting interpoint distances will match as closely as possible to the participants' response data and approximate a "good fit" to the model (Davison, 1992).
The coordinate axes that emerge represent perceptual dimensions that can be identified and defined by inspecting the properties of the stimuli positioned at the extremes of the axes. Theoretically, the positions the stimuli hold in the overall spatial configuration correspond to the salient attributes on which they are being perceived and can be conceived of as representing the respondents' "perceptual space" or "cognitive map." The relationships among the points on the map reveal how the respondents perceive the stimuli. In other words, the map or configuration uncovers the "hidden structure" in the data, thereby making it easier to comprehend (Kruskal & Wish, 1978). In relation to the current study, if distinct and interpretable satisfaction patterns can be identified from the consumer's perspective, more relevant and useful VR consumer satisfaction program evaluation processes can be developed.
The SPSS 10 for Windows (SPSS, 1999) nonmetric MDS ALSCAL procedure, using the symmetric, matrix conditional, and ordinal options, was applied to the ratings of the 14 consumer satisfaction items completed by the 223 participants. The Euclidean distance scaling model was used to transform the rating data into the dissimilarity matrix necessary for MDS. Solutions were specified in two through four dimensions. This number of potential dimensions is within the bounds of MDS stability considerations for 14 stimuli (Kruskal & Wish, 1978; Schiffman et al., 1981). As recommended by Young and Hamer (1987), measures of stress, fit, and interpretability were used to determine the number of dimensions that best represented the structure of the consumer satisfaction stimuli.
As illustrated in Figure 1, the results of the ALSCAL analysis revealed the location of each of the 14 consumer satisfaction stimuli in relation to each other within a two-dimensional space. This two-dimensional MDS solution was obtained with a Kruskal's stress value of .091 and an [R.sup.2] of .93. Stress is a lack-of-fit index; values of the stress index higher than .10 may indicate a poor solution and a lack of fit of the MDS model to the data (Kruskal & Wish, 1978). [R.sup.2] is an index of fit that indicates how well the MDS model corresponds to the input data (Hair, Anderson, & Tatham, 1987). The Kruskal's stress value and [R.sup.2] index indicate an adequate fit and high level of correspondence of the MDS model to the data. A three-dimensional solution, even though it had slightly lower stress and higher fit, was rejected as being not readily interpretable. The two-dimensional solution thus presents an accurate and reliable visual depiction of the data obtained from consumers regarding their satisfaction with VR services.
As recommended by Davison (1992), interpretation of the underlying consumer satisfaction patterns was conducted by examining the configurations and recalling from the existing consumer satisfaction literature what is known about the stimuli positioned at extremes of each of the coordinate axes. As shown in Figure 1, the two-dimensional MDS configuration indicated a case management (left) versus employment (right) satisfaction pattern (Dimension 1--horizontal). Results also indicated a consumer satisfaction pattern along Dimension 2 (vertical) of consumer choice (top) versus customer service (bottom).
The purpose of this investigation was to examine the dimensions underlying consumer satisfaction with VR services. Prior to discussing results, caveats regarding methodological limitations must be noted. The first is that the validity of the findings in this study is threatened by the 14 consumer satisfaction stimuli used in this investigation (as opposed to another group of items), the use of level of agreement (e.g., strongly agree) as the measure of consumer satisfaction, and a quantitative rather than qualitative approach to assessing consumer satisfaction. Numerous researchers (e.g., Janikowski et al., 1991; Koch & Merz, 1995; Kosciulek et al., 1997) have affirmed that consumer satisfaction with VR services involves many facets that can be assessed through various methods.
The second caveat relates to the degree to which the findings of this investigation can be generalized across persons, times, and settings (Parker, 1993). As per guidelines presented by Wilkinson (1999) and the Task Force on Statistical Inference of the American Psychological Association, because the sampling procedure was non-random, study results cannot be generalized to all consumers who receive services from state VR agencies. Similarly, the external validity of results in this study is threatened by the 30% response rate that limits generalization to the overall sampling frame and population of consumers with disabilities who receive VR services. Thus, the cross-sectional nature of the data collection procedure allows only for interpretation of results concerning the sample used at the time of the study (Heppner, Kivlighan, & Wampold, 1999).
The results of the MDS of the 14 consumer satisfaction stimuli suggest two dimensions along which consumers perceive satisfaction with VR services. The first dimension identified in this study represented satisfaction related to case management versus employment. As illustrated in Figure 1, satisfaction stimuli at the left end of this dimension (e.g., "I received services from the agency without excessive waiting") entailed the processes of coordinating, the VR program. At the right end of Dimension 1 were stimuli related to consumer perception of satisfaction with employment outcomes resulting from VR services (e.g., "As a result of the services provided by the agency, my present work situation is better than it was before I began the program"). Thus, consistent with previous quantitative reports (CSAVR, 1998; Janikowski et al., 1991)and conceptual papers (Koch & Merz, 1995; Richard, 2000) on consumer satisfaction with VR services, a major dimension underlying satisfaction as perceived by participants in this study focused on case management and employment aspects of the VR process.
The second dimension of the MDS solution was interpreted as satisfaction related to consumer choice versus customer service. Satisfaction stimuli at the top end of this dimension centered on consumers' perceived satisfaction with the availability of choice in the VR process (e.g., "I had the final say in the selection of a vocational goal"), whereas behaviors at the bottom end of Dimension 2 related to customer service aspects of the VR process (e.g., "The agency staff returned my phone calls promptly"). Consistent with the CSAVR (1998) Committee on Client' Services Consumer Satisfaction Report, this finding indicates that consumers perceive choice and customer service as distinct components of satisfaction to consider when assessing the efficacy of VR services.
The two dimensions underlying consumer satisfaction with VR services identified in this investigation suggest that consumers consider satisfaction a complex, multifaceted construct. The data gathered in this study indicate that consumers' cognitive map of satisfaction consists of multiple factors. An important practice implication of this finding is that state rehabilitation agencies may need to expand the brief survey format currently used in most states (CSAVR, 1998) in order to obtain comprehensive, reliable, and valid measures of consumer satisfaction. Various methods, including standardized quantitative instruments, qualitative focus groups, and individual interviews, are necessary to adequately capture the diverse aspects of satisfaction in program evaluation activities conducted by state VR agencies. In addition, no reports exist in the consumer satisfaction literature relative to the long-term assessment of consumer satisfaction in relation to employment outcomes following vocational rehabilitation. Studies assessing satisfaction levels during service provision, at case closure, following 90 days of successful employment, at 6 months post closure, and at 1 year post-closure would begin to shed light on the true efficacy of VR services in relation to consumer standard of living and quality of life.
The findings in this study also demonstrate that even with a relatively small number of stimuli, consumer satisfaction studies can yield distinct, interpretable, and stable dimensions. Results thus provide evidence of the complexity of consumer satisfaction and need for further study of the construct. Further elucidation of the construct of consumer satisfaction through similar studies is warranted. Additional research providing evidence that consumers view satisfaction as a multidimensional construct suggests the need for the development of a psychometrically sound, multifactorial instrument that assesses consumer satisfaction in the context of vocational rehabilitation. In this manner, if discrete satisfaction patterns can be identified from the consumer's perspective, more pertinent VR consumer satisfaction program evaluation processes can be developed.
Future research also should attempt to identify the patterns, dimensions, and themes of consumer satisfaction for consumer population subgroups (e.g., closure status, cost of services, disability type). In addition, it is necessary to conduct longitudinal studies that assess the relation of satisfaction to long-term employment outcomes (i.e., beyond 90 days of employment). A comparison of the satisfaction of consumers using agencies serving the visually impaired to that of consumers using general agencies would also be instructive. State VR agencies can use the dimension anchors of case management, employment, choice, and customer service to focus satisfaction survey development and processes, as well as to compare consumer population subgroups. Furthermore, given that response rates in most consumer satisfaction surveys conducted by state VR agencies are very low (CSAVR, 1998), valuable information would be gained by comparing demographic characteristics and case service processes and outcomes of those consumers who do and do not respond to satisfaction surveys. Finally, regarding measurement approaches, in order to yield data useful for program improvement, context-specific satisfaction instruments that address the VR service process as provided by state rehabilitation agencies and community rehabilitation programs are preferable to generic human service satisfaction assessment approaches, such as SERVQUAL (Zeithaml, Parasuraman, & Berry, 1990), which do not specifically address the VR process.
Commission on the Accreditation of Rehabilitation Facilities. (2000, December). Believing in consumers: CARF practices what it preaches. CARF Connection, 1-3.
Council of State Administrators of Vocational Rehabilitation [CSAVR]. (1998). CSAVR committee on client services consumer satisfaction report. Washington, DC: Author.
Cozby, P. C. (1993). Methods in behavioral research (5th ed.). Mountain View, CA: Mayfield.
Davison, M. L. (1992). Multidimensional scaling. New York: Wiley. Forgas, J. P. (1979). Social episodes: The study of interaction routines. New York: Academic Press.
Hair, J. E, Anderson, R. E., & Tatham, R. L. (1987). Multivariate data analysis with readings (3nd ed.). New York: Macmillan.
Heppner, E E, Kivlighan, D. M., & Wampold, B. E. (1999). Research design in counseling (2nd ed.). Belmont, CA: Wadsworth.
Hoff, D., Varney, E., Marrone, J., Butterworth, J., & Silverstein, R. (2001). The proposed Ticket to Work regulations. Tools for Inclusion: Family and Consumer Series, 9(1), 1-10.
Janikowski, T. P., Bordieri, J. E., & Musgrave, J. (1991, Summer). Dimensions of client satisfaction with vocational evaluation services.
Vocational Evaluation and Work Adjustment Bulletin, 24, 43-48.
Koch, L. C., & Merz, M. A. (1995). Assessing client satisfaction in vocational rehabilitation program evaluation: A review of instrumentation. Journal of Rehabilitation, 61 (4), 24-30.
Kosciulek. J. E (1999). Consumer direction in disability policy formulation and rehabilitation service delivery. Journal of Rehabilitation, 65(2), 4-9.
Kosciulek, J. E (2000, November). Implications of the Workforce Investment Act of 1998 and Ticket to Work and Work Incentives Improvement Act of 1999 for vocational rehabilitation. Paper presented at the Mid-South Professional Counselors and Case Managers Symposium, Memphis, TN.
Kosciulek, J. E, Prozonic, L., & Bell, D. (1995). On the congruence of evaluation, training, and placement. Journal of Rehabilitation, 61 (4), 20-23.
Kosciulek, J. E, Vessell, R., Rosenthal, D. A., Accardo, C., & Merz, M. A. (1997). Consumer satisfaction with vocational rehabilitation services. Journal of Rehabilitation, 63(2), 5-9.
Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Newbury Park, CA: Sage.
O'Day, B. (1999). Policy barriers for people who want to work. American Rehabilitation, 25(1), 8-15.
Parker, R. M. (1993). Threats to the validity of research. Rehabilitation Counseling Bulletin, 36, 130-138.
Power, E W. (2000). A guide to vocational assessment (3rd ed.). Austin, TX: PRO-ED.
Richard, M. A. (2000). A discrepancy model for measuring consumer satisfaction with rehabilitation services. Journal of Rehabilitation, 66(4), 37-43.
Schiffman, S. S., Reynolds, W. E., & Young, E W. (1981). Introduction to multidimensional scaling: Theory, methods, and applications. Orlando, FL: Academic Press.
Schwab, A. J., & Fenoglio, J. A. (1992). The Texas Rehabilitation Commission system for measuring quality rehabilitation. Journal of Applied Rehabilitation Counseling, 23(2), 21-25.
Seelman, K. D. (2000, July). Employment of individuals with disabilities: Opportunities and challenges. Paper presented at the Cornell University Employment and Disability Policy Summer Institute, Ithaca, NY.
SPSS. (1999). SPSS for Windows (Version 10.0) [Computer software]. Chicago: Author.
Thorndike, R. M. (2001). Reliability. In B. E Bolton (Ed.), Handbook of measurement and evaluation in rehabilitation (3rd ed., pp. 29-48). Gaithersburg, MD: Aspen.
Wilkinson, L. (1999). Statistical methods in psychology journals: Guidelines and Young, E W., & Hamer, R. M. (1987). Multidimensional scaling: History, theory, and application. Hillsdale, NJ: Erlbaum.
Zeithaml, V. A., Parasuraman, A., & Berry, L. L. (1990). Delivering quality service: Balancing customer perceptions and expectations. New York: The Free Press.
Zimmerman, M. A. (1995). Psychological empowerment: Issues and illustrations. American Journal of Community Psychology, 23, 581-599.
John F. Kosciulek, PhD, is an associate professor in the Office of Rehabilitation and Disability Studies at Michigan State University. His current research and teaching interests include research methodology, outcome assessment, Web-based learning, and family adaptation to brain injury. Address: John F. Kosciulek, 237 Erickson Hall, Michigan State University, East Lansing, MI 48824; e-mail: firstname.lastname@example.org