Validating the Adult Primary Care Assessment Tool

Agrowing body of literature at both individual and ecologic levels has demonstrated the association of primary care and health outcomes.^1-11 Franks and Fiscella,¹² using nationally representative survey data, showed that adult respondents who reported a primary care physician rather than a specialist as their regular source of care had lower subsequent mortality and lower annual health care costs after controlling for differences in demographic characteristics, health insurance status, health perceptions, reported diagnoses, and smoking status. Both Shi^4,6 and Farmer and collegues¹³ found better health outcomes in states with higher primary care physician-population ratios after controlling for sociodemographic measures (% elderly, % urban, % minority, education, income, unemployment, pollution) and lifestyle factors (seatbelt usage, obesity, and smoking). Recent studies further showed that primary care may mitigate the adverse effects of income inequality on health.^14-16 Taken individually, each of the main features of primary care (person-focused care over time, accessible care, comprehensive in the sense of meeting all common health needs, and coordination when people have to receive services elsewhere) are known to improve both the effectiveness as well as the efficiency of care.^1,7,17-24

The mounting evidence associating primary care with improved health outcome has led to a rapid increase in interest in assessing primary care achievement by consumers and patients.^{18,19,21,25-28} Despite its importance, there currently is no way to assess the extent to which people receive adequate primary care; receiving care from a physician or physician designated as a primary care physician is at best only a proxy for actual adequacy of provision of primary care services. As a result, there are efforts to develop instruments that directly assess the adequacy of primary care.^20,29,30

The Primary Care Assessment Tool (PCAT) instruments developed by The Johns Hopkins Primary Care Policy Center for Underserved Populations were designed to measure the extent and quality of primary care services at a provider setting designated by consumers as their main source of general care and consistent with a focus on attributes of primary care that have been demonstrated to produce better outcomes of care at lower costs.²² The PCATfamily of instruments includes the Child Consumer/Client Survey, the Adult Consumer/Client Survey, and the Facility/Provider Survey. All surveys are based on self-report by patients or providers. The Consumer/Client Survey (both adult and child editions) is designed to collect information from consumers or family caretakers regarding their experience using health care resources. It may be used to survey target populations as defined by geography (community surveys), health plans, sites of care, payment mechanisms, or specific health care needs. The survey, which takes approximately 40 minutes to complete, can be administered through either telephone or face-to-face interviews, or by mail. Ahigh school reading level is required to self-administer the questionnaire. The Facility/Provider Survey is designed to collect information about specific operational characteristics and practices related to providing primary care from the viewpoint of practitioners, clinics, group practices, and institutions. This survey can also be implemented either by mail or by face-to-face or telephone interviews. It is parallel in content to the consumer/client survey. All 3 instruments are available for general use on request.

We report on the validation of the Consumer/Client Primary Care Assessment Tool Adult Edition (PCAT-AE). Its companion instrument for children (PCAT-CE) was previously validated.³⁰ Specifically, we assessed the congruence between the theoretically derived measures and the empiric results in terms of the underlying structure of the principal primary care domains within a diverse sample of populations including health maintenance organization (HMO) members and community health center (CHC) users. The validation process also served to reduce the number of items needed to assess the adequacy of primary care.

Methods

Subjects

The study participants were members of 2 health plans in 2 counties of South Carolina. Both counties are part of Columbia, the states capital and third largest city. One of the health plans (referred to as HMO) is licensed as an independent practice association (IPA) HMO model, in which primary care physicians act as gatekeepers and health care managers. Referral to specialists must be made through primary care physicians, and specialists must be affiliated with the HMO. The primary market has been large group employers, including employees of the state agencies and national and regional companies. Members of this plan are primarily from middle-income households. The other health plan (referred to as CHC) is a coalition of 12 Columbia-based health and social services provider organizations, including the county hospital, health department, department of social services, community health centers, and other social service agencies that provide services to lower income persons, such as Medicaid recipients and low-income households. These 2 plans were selected because they represent typical South Carolina managed care organizations and health plans for low-income individuals, respectively. Samples drawn from these 2 plans allowed us to test the reliability of PCATwith a diverse sample of populations, including both middle-income and low-income individuals using regular physician offices and community health centers, respectively.

Estimation of the sample size for this study involved several steps. First, an estimate of the likely proportions or means and standard deviations for each primary care measure was derived from a previous study.²⁵ When data were not available, a conservative estimate (eg, a larger standard deviation or proportion closer to 50/50) was made. Second, the estimates of the proportions, means, and standard deviations for the dependent variables were entered into the standard sample size formula for a two-group, cross-sectional sample. Using a 95% confidence interval, the largest sample size required was 300 per group. The CHC group was oversampled because of additional planned within-group analyses (not the focus of this paper). Finally, the desired sample size was adjusted for anticipated survey nonresponse (anticipated to be higher for a mail survey than a face-to-face interview).

For the HMO group, a mail survey was used since it was deemed most efficient. In 2 previous longitudinal studies of the same HMO, we used mail survey and telephone interviews alternately with a cohort of HMO members and obtained comparable results.^31,32 For this study, we sent a letter with a PCAT-AE questionnaire to 1000 randomly selected adult members to invite them to participate in the project. Because of known frequent changes in addresses, we recruited the non-HMO plan individuals and conducted in-person interviews at all the community health center sites where members came to the clinics for non-urgent visits. Patients were systematically approached while waiting for their scheduled appointment (ie, every nth patient based on expected visits for a particular site) and recruited for the study during a period of 4 weeks for each site.

Measures

Identification of Primary Care Source.Three questions were developed to identify an individuals usual source of care and the strength of that affiliation: (1) Is there a doctor or place that you usually go if you are sick or need advice about your health? (usual source), (2) Is there a doctor or place that knows you best as a person? (knows best), and (3) Is there a doctor or place that is most responsible for your health care? (most responsible). Aperson was considered to have a usual source of care if he or she answered positively to any 1 of the 3 questions (95% for the HMO plan and 90% for the low-income plan). Anegative answer to all 3 questions rendered the individual as not having a usual source of care.

An algorithm based on response to these 3 questions identified the strength of affiliation with the primary care source. If all 3 physicians/places were the same, this was considered evidence of a strong affiliation. If the response to the usual source question was the same as for either of the other 2 questions then that site was used although the affiliation was considered less strong. If the response for a usual source question was different from the other 2 responses but the other 2 responses were the same, then the site where both were the same was used (weak affiliation). If all 3 responses were different (weakest affiliation), then the site identified for usual source was used. All subsequent questions asked about this specific person or place. For those with no identifiable source of primary care, subsequent questions were asked about the last place that was visited.

Domains of Primary Care.The PCAT-AE was modeled on the previously validated PCAT-CE and is consistent with the 1978 Institute of Medicine (IOM) definition of primary care as accessibility, comprehensiveness, coordination, continuity, and accountability³³ and with the 1996 IOM report definition of primary care as the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and the community.³⁴ When combined into scales, the PCATsurvey items dealing with primary care quality were designed to measure each of the core domains of primary care; that is, first contact, longitudinality, comprehensiveness, and coordination (definitions of the primary care domains are provided in the Appendix).

Nine experts were asked to rate the appropriateness and representativeness of the primary care domain items. These experts consisted of 3 policymakers in federal agencies, 2 directors of community pediatrics at major medical centers, a health research director at a major HMO, 2 family medicine professors, and a general internal medicine physician. Acard sorting technique was used to determine the degree of congruence between each item and the domain it was designed to measure. Each survey question with its response categories and descriptions of each of the primary care domains was printed on separate index cards and mailed to the experts who assigned each question to one of the defined domains and suggested revisions and/or addition of other items. The percent agreement among the experts was used to determine the degree of congruence on the placement of each item in a particular domain. In addition, students in a graduate course on primary care independently assigned each item to a domain as well as to its appropriate subdomain.

In addition to the 4 core primary care domains, 3 other related domains (family centeredness, community orientation, and cultural competence) were included; these domains were considered derivative in that their achievement would be related to the achievement of the major domains.¹ However, they were separately specified as ancillary domains because of widespread appreciation of their likely importance.

Thus, the PCAT-AE consists of 7 domains represented by 9 scales. Each of the 4 core domains of primary care is represented by 2 components, 1 representing a characteristic of the facility of providers service organization and 1 representing a behavior of the provider or consumer.¹ One of these 8 potential components (longitudinality strength of affiliation) is represented by an index rather than a scale and is scored from the responses to the 3 questions noted under the heading Identification of the Primary Care Source. One subdomain, the facility characteristics related to the achievement of coordination, is obtainable only from the facility or provider, since consumers would not be expected to know the nature of information systems that facilitate coordination of care. Thus, the PCATinstrument has 6 scales representing the 4 primary care domains: first contactaccessibility, first contactutilization (first contact domain), longitudinalityinterpersonal relationships or ongoing care (longitudinality domain), coordination of services (coordination domain), comprehensiveness services available, comprehensivenessservices received (comprehensiveness domain) and the 3 ancillary domains of family centeredness, community orientation, and cultural competence.

For first contactaccessibility 12 questions were developed to measure access to the source of care. For first contactutilization 3 questions addressed the extent to which the source of care is first used for various types of problems. Twenty questions addressed the nature and strength of the person-focused relationship with the source of care over time (longitudinality). Eight questions were used to address the coordination of services between a primary care provider and specialty care. The comprehensivenessservices available domain included 24 items of important primary care services. An additional 13 questions were used to measure comprehensivenessservices received. Two items were used to measure family-centeredness, 5 community orientation, and 3 cultural competence. Copies of both the original questionnaire and the revised condensed version are available on request.

For consistency in response and scoring, all items representing the primary care domains were represented by a 4-point Likert-type scale (1=definitely not; 2=probably not; 3=probably; and 4=definitely). The sum score for each domain was derived by adding (after reverse-coding where appropriate) the values for all the items under each domain. An additional Dont Know/Cannot Remember option was also provided for each item. At least 3 methods could be used to code this category. The missing value method treats this item as missing for those who answer Dont Know/Cant Remember. The median value method assigns a value of 2.5 for those who answer Dont Know/Cant Remember. The imputation method imputes the response based on the mean of the results from other items within the domain when at least 50% of the items have been answered. Since the internal consistency reliability (a) is the highest based on the imputation method, this method is adopted in coding the Dont Know/Cant Remember category. However, the other 2 methods also produced high internal consistency reliability (results available on request).

Analysis

The purpose of the validation was to assess the congruence between the theoretically derived measures and the empiric results in terms of the underlying structure of the principal primary care domains. Although conceptual framework was relied on in the construction of primary care measures, empiric validation was used to reduce the number of items so that the questionnaire became more concise.

The validation of PCAT-AE with the South Carolina sample involved several steps. First, principal component factor analysis was used to explore the structure of the PCAT-AE items and examine its construct validity by determining if the items fell into the hypothesized scales (factors; definitions of measurement-related concepts used in this paper can be found in the Appendix). Factor analysis was also used for item selection and placement into scales based on the pattern of the factor loadings.³⁵ Four criteria were used in deleting items and the determination of the final factors.^36-37 Afactor loading greater than 0.35 was considered meaningful and used as a criterion for retaining items. In addition, each retained factor should have at least 3 items with loadings greater than 0.35. All retained items should share the same conceptual meaning or construct. Also, all retained items should not have secondary loadings greater than 0.35.

Second, internal consistency reliability of the primary care scales was assessed by Cronbachs coefficient alpha (a)³⁸ and item-total correlation for items in each domain. Cronbachs coefficient alpha is based on the covariance among individual items in a scale and the number of items. It ranges from 0, indicating total lack of consistency, to 1, indicating complete internal consistency reliability. The item-total correlation is the correlation between an individual item and the sum of the remaining items that constitute the scale. If an item-total correlation is small, the item is not considered to be measuring the same construct that is measured by the other items in the scale. All the retained items exceeded the minimum acceptable item-total correlation of 0.30.³⁸

Third, the Likert scaling assumptions were tested for the final items related to the primary care scales. Likerts method of summated rating scales is based on the assumption that item responses in each scale can be summed without standardization or weighting.³⁹ The underlying assumptions that must be met include: (1) item-convergent validity (tested by item-scale correlations); (2) item-discriminant validity (tested using the scaling success rate, ie, correlation of each item with other items within the same scale is greater than with items from different scales); (3) equal item variance (tested by examining item means and standard deviations and the equivalence of the intraclass correlation and Scotts homogeneity ratio for each scale); (4) equal item-scale correlation (tested by examining the range of item-scale correlations); and (5) score reliability (tested by Cronbachs coefficient a.

Fourth, descriptive statistics were performed for the revised primary care scales, including mean, standard deviation, range, percentile, skewness, kurtosis, and interscale correlation. Since respondents who never saw a specialist did not answer the coordination questions, analyses were performed both with and without those questions, including the coordination domain.

Results

Subjects

For the HMO group, a total of 350 individuals responded after 3 mailings. Excluding the nonresponses due to wrong addresses and changed plans (n=340), the effective response rate was 53 percent (350/660). The respondents and nonrespondents were not significantly different in age, sex, race, and zip codes of mailing addresses. For the CHC group, a total of 1000 individuals were systematically selected and approached. Among them, 265 refused to be interviewed, 195 were not able to complete the interview prior to their appointment, and 540 completed the interview. Taking only refusal into account, the response rate was 67% (540/540+265). Men were more likely to refuse the interview than women. There were no significant differences in age and race between respondents and nonrespondents. All interviews were conducted by graduate public health students trained in interactive sessions and were completed in 1999.

The sample included 823 adults with an identified usual source of care. Among them, most (69% of HMO and 60% of CHC respondents) indicated a strong affiliation with their usual source of care (ie, all 3 doctors/places were the same). Very few (0.6% of HMO and 1.2% of CHC respondents) indicated the weakest affiliation with their usual source of care (ie, all 3 responses were different). Just over half of respondents (56%) were non-white (primarily black). Over half (55%) had an annual household income under $25,000. Most respondents (76%) had health insurance coverage all year and had been seeing their regular source of care for more than 1 year (82%). Sixty-three percent had seen their regular source of care for more than 2 years. The majority chose their own usual source of care (78%) and did not have trouble paying for their health care (74%). More than half of the respondents made at least 1 visit to a specialist (56%). This relatively high rate may be due to a somewhat elderly sample; more than 20% of the respondents were older than 65 years.

Table 1 compares the HMO sample with the CHC sample on sociodemographic and health care utilization measures. The HMO sample included predominantly white (81.6%) and higher income subjects (86.8% with annual household income of $25,000 or more). In contrast, the CHC sample included predominantly non-white (83.2%) and lower income subjects (85.9% with an annual household income less than $25,000). Compared with the CHC respondents, HMO subjects had been seeing their regular source of care for a longer time, were more likely to choose their own doctors and visit a specialist, and less likely to have trouble paying for their health care.

Factor Analysis and Construct Validity

In the initial exploratory factor analysis, all 92 applicable questionnaire items measuring the subdomains and domains of primary carefirst contact, longitudinality, comprehensiveness, coordination, family centeredness, community orientation, and cultural competencewere included. Based on the results of the initial factor analysis, 4 criteria were applied to reach the final solution (Table 2; initial factor analyses not shown but available upon request).

Seven common factors were extracted, corresponding to the hypothesized primary care scales: first contactaccessibility, first contactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, coordination, and community orientation (Table 2). Those extracted factors explained 88.1% of the common variance. Eigenvalues ranged from 16.17 to 1.16. All principal primary care domains were extracted as hypothesized. Only 1 of the 3 derivative features, community orientation, was separately identifiable.

Derivation and Reliability of the Primary Care Scales

Table 3 presents the results of the reliability analyses for both the original items and the final items (based on factor analysis). Item descriptive results (means and standard deviations) are also presented. Scale reliability measures include item-total correlation and alpha coefficient reliability. The distribution of the items varied significantly from a mean of 1.85 (ask about gun safety) to 3.73 (Provider answers questions in ways you understand) on the 4-point Likert-type scale. The distribution tends to skew toward more favorable answers (above 2.5). Apart from the gun safety item, only 2 items fell below a mean of 2 (1.94 for Provider knows neighborhood problems, 1.90 for Provider makes home visits). The first contactutilization and longitudiinalityinterpersonal relationships scales achieved the highest mean scores, whereas scales with lower means were community orientation, first contact-accessibility, and comprehensiveness-services received.

Eighteen of the 92 initial items were deleted on the basis of the criteria imposed for factor analyses. No items were deleted for first contact-utilization, coordination of services, comprehensiveness-services received, and community orientation scales. All items were deleted for family centeredness as were two thirds of the items for first contact-accessibility. Two items (out of 22) were deleted for longitudinality-interpersonal relationships and 3 (out of 24) for comprehensivenessservices available. Items from cultural competence were combined into first contact-accessibility. The revised scales demonstrate internal consistency reliability that was higher than or equal to the original scales, despite the reduction in number of items. Item-total correlations were also high and ranged from 0.34 (If sick, seen same day if office is open) to 0.91 (How to prevent hot water burns and How to prevent falls).

Testing the Likert Scaling Assumptions

Table 4 presents a summary of the results of the tests of Likert scaling assumptions using the revised items. All item-scale correlations well exceeded the accepted minimum (0.30) with the majority greater than 0.50 (Assumption 1). All 7 multi-item scales achieved 100% scaling success, indicating that all items in these scales correlated substantially higher with items in their hypothesized scale than with items in other scales (Assumption 2). Item means within each revised scale generally differed by less than six tenths of a point (except for first contact-accessibility) and item standard deviations within each scale by less than four tenths of a point (Assumption 3). Formal evidence of equal item variance was supported by the equivalence of the intraclass correlation and Scotts homogeneity ratio for each scale. Equal-item scale correlation (Assumption 4) was also observed through the range of item-scale correlations. As shown in column 1 (range of item-scale correlations), the range is relatively narrow (from .17 for coordination of services to .38 for comprehensiveness-services received). Finally, score reliability (Assumption 5) showed that except for first contact-utilization (only 3 items), all alpha levels exceeded .70 and were sufficiently high. Five of the 7 scales had alpha levels above .85.

Descriptive Feature of PCAT-AE

Table 5 displays estimates of central tendency and dispersion of scale score distributions for the 7 primary care scales in this South Carolina sample. Except for community orientation, all primary care scales were negatively skewed, indicating distributions with more positive ratings of primary care. The community orientation scale was positively skewed, indicating distributions with more negative ratings on the community orientation aspect of primary care. The full range of possible scores was observed for all scales except ongoing care.

The percentage of respondents scoring at the floor (the lowest score) or ceiling (the highest score) was acceptably low for all scales except first contactutilization, where 50% of the respondents scored the maximum score.

Table 6 compares the alpha coefficient and interfactor correlation for each primary care scale. The alpha coefficient of each scale substantially exceeded its correlation with all other primary care scales. None of the inter-factor correlations were excessively high, demonstrating that each primary care scale has significant unique contribution. All significant correlations were positive, indicating the complementary nature of primary care domains. Relatively high and positive interfactor correlations were observed between comprehensivenessservices received and comprehensiveness-services available (0.44), with the former and longitudinalityinterpersonal relationships (0.43), with the latter and coordination (0.38), and with comprehensivenessservices received and community orientation (0.37).

Discussion

Using patient-provided survey information collected within 2 health plans in South Carolina, we assessed the validity and reliability of the PCAT-AE. The results indicate that the hypothesized scales for primary care (first contactaccessibility, first con-tactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, and coordination) have substantial reliability and validity, consistent with the findings from the testing of the PCAT-CE.³⁰ The 2 versions of the instrument differ only in the comprehensiveness domains, as comprehensiveness implies that all common needs are met, and health needs in childhood are different from those in adults. In contrast, challenges to accessibility, to the nature of interpersonal relationships, and to coordination and community orientation are similar for both children and adults and thus can be assessed by the same items. Only 1 ancillary feature of primary care, community orientation, was retained as a separate dimension after factor analyses. The extracted factors explained 88.1 percent of the total variance in the item scores.

All of the 5 assumptions, including item-conver-gent validity, item-discriminant validity, equal item variance, equal item-scale correlation, and score reliability, were met. These results suggest that these items may be used to represent the primary care scales, and the scoring of these items may be summed without standardization or weighting, as with Likerts method of summated rating scales.³⁹

The resulting instrument has 74 items. Although the retained items adequately addressed first contactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, and coordination, and are consistent with the framework, those representing first contactaccessibility fell short. Only 4 of the 12 items measuring accessibility were retained. When more detail on accessibility is required, items that were deleted because they had lower item-total correlation may be added back in. Users should also review the comprehensiveness items to ascertain their relevance in the setting in which they are to be used. Items may be deleted if they are inappropriate in the context in which they are used; for example, in health systems that do not offer on-site testing for human immunodeficiency virus (HIV), because HIV is uncommon. Since continuity of care is an important component of primary care quality, a minimum number of visits or minimum duration with a regular source of care should be part of the assessment tool.

Separate factor analyses were performed with the 2 health plans. The results were largely comparable in terms of the factors that emerged as significant, indicating the generalizability of the tool to both vulnerable and middle-income populations. The only major differences are that the CHC subpopulation analysis yielded an additional significant factor, cultural competence, which the HMO subpopulation and the total population analyses failed to identify. In contrast, the HMO subpopulation analysis yielded an additional significant factor, family centeredness, which the CHC subpopulation and the total population analyses failed to identify. Thus, when using PCATon vulnerable populations (especially racial and ethnic minorities), questions measuring cultural competence might be retained. Family centeredness seemed to emerge as a distinct concept, primarily in the middle-income population.

There are a number of uses for a valid and reliable instrument such as the PCAT-AE. First, understanding primary care as a multidimensional concept is consistent with the IOMs conceptualization of primary care and more precisely captures the quality of primary care than unidimensional proxies, such as a clinicians medical specialty. With the 6 scales representing 4 core domains, the index representing strength of affiliation with a primary care provider, a scale for community orientation and the optional scales for family centeredness and cultural competence, all the important features of primary care are addressed. Second, PCAT-AE can be used as a quality measurement tool that assesses the adequacy of primary care experience rendered under different health care systems or settings, and for patients with different sociodemographic attributes. Third, PCAT-AE can also serve as a quality control tool that compares the quality of primary care given by providers of different types. The instrument can be used with other outcomes to assess the effect of policy interventions and systems changes on the delivery of critical aspects of primary care.

Limitations

Interpretation of our results should take into account some limitations. First, because our study was restricted to 1 locale, the generalizability of the PCAT-AE to other sites and states is not assured. Additional testing and validation is necessary to corroborate the current results. Second, the 74-item questionnaire remains lengthy and could have contributed to relatively high nonresponse and incompletion rates. Future validation work will concentrate on further reduction of the items to the very essential in order to reduce response burden. Regarding the ceiling effect of first contactutilization, future tests will be conducted in other settings with less of a managed care focus, as there well may be quite different distributions of responses in other settings. Third, outcomes of primary care are not the focus of the assessment tool. However, numerous studies have linked primary care to better health outcomes. Subsequent research may help explain which attributes are most conducive to better outcomes so that limited resources can be used to focus on them or a combination of them. Fourth, the measurement of primary care achievement is entirely based on respondents self-report. While self-report may be the best way to ascertain peoples experiences, it is subject to recall and response bias. Moreover, some aspects of technical quality cannot be assessed by patientsor consumers reports.

Despite these limitations, PCAT-AE is a valuable tool for capturing the principal domains of primary care. The next phase of our work seeks to assess the predictive validity of PCAT-AE, by examining the extent to which the principal attributes of primary care can be linked to the achievement of favorable health outcomes, their ability to manage their illnesses, and their satisfaction with the care received. Such work would advance our understanding of the relationship between how primary care is delivered and the health outcomes that result.

Related technical terms

Primary Care Attributes

First contactcare implies accessibility to and use of services for each new problem or new episode of a problem for which people seek health care.

Longitudinalitypresupposes the existence of a regular source of care and its use over time.

Comprehensivenessimplies that primary care facilities must be able to arrange for all types of health care services, including referrals to secondary services for consultation, tertiary services for specific conditions, and essential supporting services, such as home care and other community services.

Coordinationof care requires some form of continuity, either by practitioners, medical records, or both, as well as recognition of problems that are addressed elsewhere and the integration of their care into the total care of patients.

Family centerednessrefers to recognition of family factors related to the genesis and management of illness.

Community orientationrefers to the providers knowledge of community needs and involvement in the community.

Cultural competencerefers to the providers adaptation to facilitate relationships with populations having special cultural characteristics.

Measurement Concepts

Measurement validityrefers to the extent that important dimensions of a concept and their categories have been taken into account and appropriately operationalized.

Measurement reliabilityrefers to the extent that consistent results are obtained when a particular measure is applied to similar elements.

Construct validityis present when the measure captures the major dimensions of the concept under study.

Content validityrefers to the representativeness of the response categories used to represent each of the dimensions of a concept.

Concurrent validitymay be tested by comparing results of one measurement with those of a similar measurement administered to the same population and at approximately the same time. If both measurements yield similar results, then concurrent validity can be established.

Predictive validity exists when the results obtained from the measurement succeed in predicting the expected later-occurring event or circumstance.

Test-retest reliabilityinvolves administering the same measurement to the same individuals at 2 different times. If the correlation between the same measures is high, then the measurement is believed to be reliable.

Split-half reliabilityinvolves preparing 2 sets of measurement of the same concept, applying them to research subjects at one setting, and comparing the correlation between the 2 sets of measurement. To the extent the correlation is high, then the measurement is reliable.

Interrater reliabilityinvolves using different people to conduct the same procedure, whether it be interview, observation, coding, rating, and the like, and comparing the results of their work. To the extent that the results are highly similar, interrater reliability is established.

Item-convergent validityrefers to the substantial correlation between each item and its hypothesized scale.

Item-discriminant validityrefers to items within a scale that correlate more substantially with their hypothesized scale than with any other scale.

Equal item variancerefers to items within a scale that have approximately equal means and variances.

Equal item-scale correlationrefers to items in a scale that contribute approximately the same proportion of information about the underlying concept.

Score reliabilityrefers to scores of scales that are reproducible and reliable.

Skewnessrefers to distribution of observations that is not symmetric, ie, when more observations are found at one end of the distribution than the other.

Kurtosisrefers to the extent observations cluster around a central point more than in normal distribution.

Agrowing body of literature at both individual and ecologic levels has demonstrated the association of primary care and health outcomes.^1-11 Franks and Fiscella,¹² using nationally representative survey data, showed that adult respondents who reported a primary care physician rather than a specialist as their regular source of care had lower subsequent mortality and lower annual health care costs after controlling for differences in demographic characteristics, health insurance status, health perceptions, reported diagnoses, and smoking status. Both Shi^4,6 and Farmer and collegues¹³ found better health outcomes in states with higher primary care physician-population ratios after controlling for sociodemographic measures (% elderly, % urban, % minority, education, income, unemployment, pollution) and lifestyle factors (seatbelt usage, obesity, and smoking). Recent studies further showed that primary care may mitigate the adverse effects of income inequality on health.^14-16 Taken individually, each of the main features of primary care (person-focused care over time, accessible care, comprehensive in the sense of meeting all common health needs, and coordination when people have to receive services elsewhere) are known to improve both the effectiveness as well as the efficiency of care.^1,7,17-24

The mounting evidence associating primary care with improved health outcome has led to a rapid increase in interest in assessing primary care achievement by consumers and patients.^{18,19,21,25-28} Despite its importance, there currently is no way to assess the extent to which people receive adequate primary care; receiving care from a physician or physician designated as a primary care physician is at best only a proxy for actual adequacy of provision of primary care services. As a result, there are efforts to develop instruments that directly assess the adequacy of primary care.^20,29,30

The Primary Care Assessment Tool (PCAT) instruments developed by The Johns Hopkins Primary Care Policy Center for Underserved Populations were designed to measure the extent and quality of primary care services at a provider setting designated by consumers as their main source of general care and consistent with a focus on attributes of primary care that have been demonstrated to produce better outcomes of care at lower costs.²² The PCATfamily of instruments includes the Child Consumer/Client Survey, the Adult Consumer/Client Survey, and the Facility/Provider Survey. All surveys are based on self-report by patients or providers. The Consumer/Client Survey (both adult and child editions) is designed to collect information from consumers or family caretakers regarding their experience using health care resources. It may be used to survey target populations as defined by geography (community surveys), health plans, sites of care, payment mechanisms, or specific health care needs. The survey, which takes approximately 40 minutes to complete, can be administered through either telephone or face-to-face interviews, or by mail. Ahigh school reading level is required to self-administer the questionnaire. The Facility/Provider Survey is designed to collect information about specific operational characteristics and practices related to providing primary care from the viewpoint of practitioners, clinics, group practices, and institutions. This survey can also be implemented either by mail or by face-to-face or telephone interviews. It is parallel in content to the consumer/client survey. All 3 instruments are available for general use on request.

We report on the validation of the Consumer/Client Primary Care Assessment Tool Adult Edition (PCAT-AE). Its companion instrument for children (PCAT-CE) was previously validated.³⁰ Specifically, we assessed the congruence between the theoretically derived measures and the empiric results in terms of the underlying structure of the principal primary care domains within a diverse sample of populations including health maintenance organization (HMO) members and community health center (CHC) users. The validation process also served to reduce the number of items needed to assess the adequacy of primary care.

Methods

Subjects

The study participants were members of 2 health plans in 2 counties of South Carolina. Both counties are part of Columbia, the states capital and third largest city. One of the health plans (referred to as HMO) is licensed as an independent practice association (IPA) HMO model, in which primary care physicians act as gatekeepers and health care managers. Referral to specialists must be made through primary care physicians, and specialists must be affiliated with the HMO. The primary market has been large group employers, including employees of the state agencies and national and regional companies. Members of this plan are primarily from middle-income households. The other health plan (referred to as CHC) is a coalition of 12 Columbia-based health and social services provider organizations, including the county hospital, health department, department of social services, community health centers, and other social service agencies that provide services to lower income persons, such as Medicaid recipients and low-income households. These 2 plans were selected because they represent typical South Carolina managed care organizations and health plans for low-income individuals, respectively. Samples drawn from these 2 plans allowed us to test the reliability of PCATwith a diverse sample of populations, including both middle-income and low-income individuals using regular physician offices and community health centers, respectively.

Estimation of the sample size for this study involved several steps. First, an estimate of the likely proportions or means and standard deviations for each primary care measure was derived from a previous study.²⁵ When data were not available, a conservative estimate (eg, a larger standard deviation or proportion closer to 50/50) was made. Second, the estimates of the proportions, means, and standard deviations for the dependent variables were entered into the standard sample size formula for a two-group, cross-sectional sample. Using a 95% confidence interval, the largest sample size required was 300 per group. The CHC group was oversampled because of additional planned within-group analyses (not the focus of this paper). Finally, the desired sample size was adjusted for anticipated survey nonresponse (anticipated to be higher for a mail survey than a face-to-face interview).

For the HMO group, a mail survey was used since it was deemed most efficient. In 2 previous longitudinal studies of the same HMO, we used mail survey and telephone interviews alternately with a cohort of HMO members and obtained comparable results.^31,32 For this study, we sent a letter with a PCAT-AE questionnaire to 1000 randomly selected adult members to invite them to participate in the project. Because of known frequent changes in addresses, we recruited the non-HMO plan individuals and conducted in-person interviews at all the community health center sites where members came to the clinics for non-urgent visits. Patients were systematically approached while waiting for their scheduled appointment (ie, every nth patient based on expected visits for a particular site) and recruited for the study during a period of 4 weeks for each site.

Measures

Identification of Primary Care Source.Three questions were developed to identify an individuals usual source of care and the strength of that affiliation: (1) Is there a doctor or place that you usually go if you are sick or need advice about your health? (usual source), (2) Is there a doctor or place that knows you best as a person? (knows best), and (3) Is there a doctor or place that is most responsible for your health care? (most responsible). Aperson was considered to have a usual source of care if he or she answered positively to any 1 of the 3 questions (95% for the HMO plan and 90% for the low-income plan). Anegative answer to all 3 questions rendered the individual as not having a usual source of care.

An algorithm based on response to these 3 questions identified the strength of affiliation with the primary care source. If all 3 physicians/places were the same, this was considered evidence of a strong affiliation. If the response to the usual source question was the same as for either of the other 2 questions then that site was used although the affiliation was considered less strong. If the response for a usual source question was different from the other 2 responses but the other 2 responses were the same, then the site where both were the same was used (weak affiliation). If all 3 responses were different (weakest affiliation), then the site identified for usual source was used. All subsequent questions asked about this specific person or place. For those with no identifiable source of primary care, subsequent questions were asked about the last place that was visited.

Domains of Primary Care.The PCAT-AE was modeled on the previously validated PCAT-CE and is consistent with the 1978 Institute of Medicine (IOM) definition of primary care as accessibility, comprehensiveness, coordination, continuity, and accountability³³ and with the 1996 IOM report definition of primary care as the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and the community.³⁴ When combined into scales, the PCATsurvey items dealing with primary care quality were designed to measure each of the core domains of primary care; that is, first contact, longitudinality, comprehensiveness, and coordination (definitions of the primary care domains are provided in the Appendix).

Nine experts were asked to rate the appropriateness and representativeness of the primary care domain items. These experts consisted of 3 policymakers in federal agencies, 2 directors of community pediatrics at major medical centers, a health research director at a major HMO, 2 family medicine professors, and a general internal medicine physician. Acard sorting technique was used to determine the degree of congruence between each item and the domain it was designed to measure. Each survey question with its response categories and descriptions of each of the primary care domains was printed on separate index cards and mailed to the experts who assigned each question to one of the defined domains and suggested revisions and/or addition of other items. The percent agreement among the experts was used to determine the degree of congruence on the placement of each item in a particular domain. In addition, students in a graduate course on primary care independently assigned each item to a domain as well as to its appropriate subdomain.

In addition to the 4 core primary care domains, 3 other related domains (family centeredness, community orientation, and cultural competence) were included; these domains were considered derivative in that their achievement would be related to the achievement of the major domains.¹ However, they were separately specified as ancillary domains because of widespread appreciation of their likely importance.

Thus, the PCAT-AE consists of 7 domains represented by 9 scales. Each of the 4 core domains of primary care is represented by 2 components, 1 representing a characteristic of the facility of providers service organization and 1 representing a behavior of the provider or consumer.¹ One of these 8 potential components (longitudinality strength of affiliation) is represented by an index rather than a scale and is scored from the responses to the 3 questions noted under the heading Identification of the Primary Care Source. One subdomain, the facility characteristics related to the achievement of coordination, is obtainable only from the facility or provider, since consumers would not be expected to know the nature of information systems that facilitate coordination of care. Thus, the PCATinstrument has 6 scales representing the 4 primary care domains: first contactaccessibility, first contactutilization (first contact domain), longitudinalityinterpersonal relationships or ongoing care (longitudinality domain), coordination of services (coordination domain), comprehensiveness services available, comprehensivenessservices received (comprehensiveness domain) and the 3 ancillary domains of family centeredness, community orientation, and cultural competence.

For first contactaccessibility 12 questions were developed to measure access to the source of care. For first contactutilization 3 questions addressed the extent to which the source of care is first used for various types of problems. Twenty questions addressed the nature and strength of the person-focused relationship with the source of care over time (longitudinality). Eight questions were used to address the coordination of services between a primary care provider and specialty care. The comprehensivenessservices available domain included 24 items of important primary care services. An additional 13 questions were used to measure comprehensivenessservices received. Two items were used to measure family-centeredness, 5 community orientation, and 3 cultural competence. Copies of both the original questionnaire and the revised condensed version are available on request.

For consistency in response and scoring, all items representing the primary care domains were represented by a 4-point Likert-type scale (1=definitely not; 2=probably not; 3=probably; and 4=definitely). The sum score for each domain was derived by adding (after reverse-coding where appropriate) the values for all the items under each domain. An additional Dont Know/Cannot Remember option was also provided for each item. At least 3 methods could be used to code this category. The missing value method treats this item as missing for those who answer Dont Know/Cant Remember. The median value method assigns a value of 2.5 for those who answer Dont Know/Cant Remember. The imputation method imputes the response based on the mean of the results from other items within the domain when at least 50% of the items have been answered. Since the internal consistency reliability (a) is the highest based on the imputation method, this method is adopted in coding the Dont Know/Cant Remember category. However, the other 2 methods also produced high internal consistency reliability (results available on request).

Analysis

The purpose of the validation was to assess the congruence between the theoretically derived measures and the empiric results in terms of the underlying structure of the principal primary care domains. Although conceptual framework was relied on in the construction of primary care measures, empiric validation was used to reduce the number of items so that the questionnaire became more concise.

The validation of PCAT-AE with the South Carolina sample involved several steps. First, principal component factor analysis was used to explore the structure of the PCAT-AE items and examine its construct validity by determining if the items fell into the hypothesized scales (factors; definitions of measurement-related concepts used in this paper can be found in the Appendix). Factor analysis was also used for item selection and placement into scales based on the pattern of the factor loadings.³⁵ Four criteria were used in deleting items and the determination of the final factors.^36-37 Afactor loading greater than 0.35 was considered meaningful and used as a criterion for retaining items. In addition, each retained factor should have at least 3 items with loadings greater than 0.35. All retained items should share the same conceptual meaning or construct. Also, all retained items should not have secondary loadings greater than 0.35.

Second, internal consistency reliability of the primary care scales was assessed by Cronbachs coefficient alpha (a)³⁸ and item-total correlation for items in each domain. Cronbachs coefficient alpha is based on the covariance among individual items in a scale and the number of items. It ranges from 0, indicating total lack of consistency, to 1, indicating complete internal consistency reliability. The item-total correlation is the correlation between an individual item and the sum of the remaining items that constitute the scale. If an item-total correlation is small, the item is not considered to be measuring the same construct that is measured by the other items in the scale. All the retained items exceeded the minimum acceptable item-total correlation of 0.30.³⁸

Third, the Likert scaling assumptions were tested for the final items related to the primary care scales. Likerts method of summated rating scales is based on the assumption that item responses in each scale can be summed without standardization or weighting.³⁹ The underlying assumptions that must be met include: (1) item-convergent validity (tested by item-scale correlations); (2) item-discriminant validity (tested using the scaling success rate, ie, correlation of each item with other items within the same scale is greater than with items from different scales); (3) equal item variance (tested by examining item means and standard deviations and the equivalence of the intraclass correlation and Scotts homogeneity ratio for each scale); (4) equal item-scale correlation (tested by examining the range of item-scale correlations); and (5) score reliability (tested by Cronbachs coefficient a.

Fourth, descriptive statistics were performed for the revised primary care scales, including mean, standard deviation, range, percentile, skewness, kurtosis, and interscale correlation. Since respondents who never saw a specialist did not answer the coordination questions, analyses were performed both with and without those questions, including the coordination domain.

Results

Subjects

For the HMO group, a total of 350 individuals responded after 3 mailings. Excluding the nonresponses due to wrong addresses and changed plans (n=340), the effective response rate was 53 percent (350/660). The respondents and nonrespondents were not significantly different in age, sex, race, and zip codes of mailing addresses. For the CHC group, a total of 1000 individuals were systematically selected and approached. Among them, 265 refused to be interviewed, 195 were not able to complete the interview prior to their appointment, and 540 completed the interview. Taking only refusal into account, the response rate was 67% (540/540+265). Men were more likely to refuse the interview than women. There were no significant differences in age and race between respondents and nonrespondents. All interviews were conducted by graduate public health students trained in interactive sessions and were completed in 1999.

The sample included 823 adults with an identified usual source of care. Among them, most (69% of HMO and 60% of CHC respondents) indicated a strong affiliation with their usual source of care (ie, all 3 doctors/places were the same). Very few (0.6% of HMO and 1.2% of CHC respondents) indicated the weakest affiliation with their usual source of care (ie, all 3 responses were different). Just over half of respondents (56%) were non-white (primarily black). Over half (55%) had an annual household income under $25,000. Most respondents (76%) had health insurance coverage all year and had been seeing their regular source of care for more than 1 year (82%). Sixty-three percent had seen their regular source of care for more than 2 years. The majority chose their own usual source of care (78%) and did not have trouble paying for their health care (74%). More than half of the respondents made at least 1 visit to a specialist (56%). This relatively high rate may be due to a somewhat elderly sample; more than 20% of the respondents were older than 65 years.

Table 1 compares the HMO sample with the CHC sample on sociodemographic and health care utilization measures. The HMO sample included predominantly white (81.6%) and higher income subjects (86.8% with annual household income of $25,000 or more). In contrast, the CHC sample included predominantly non-white (83.2%) and lower income subjects (85.9% with an annual household income less than $25,000). Compared with the CHC respondents, HMO subjects had been seeing their regular source of care for a longer time, were more likely to choose their own doctors and visit a specialist, and less likely to have trouble paying for their health care.

Factor Analysis and Construct Validity

In the initial exploratory factor analysis, all 92 applicable questionnaire items measuring the subdomains and domains of primary carefirst contact, longitudinality, comprehensiveness, coordination, family centeredness, community orientation, and cultural competencewere included. Based on the results of the initial factor analysis, 4 criteria were applied to reach the final solution (Table 2; initial factor analyses not shown but available upon request).

Seven common factors were extracted, corresponding to the hypothesized primary care scales: first contactaccessibility, first contactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, coordination, and community orientation (Table 2). Those extracted factors explained 88.1% of the common variance. Eigenvalues ranged from 16.17 to 1.16. All principal primary care domains were extracted as hypothesized. Only 1 of the 3 derivative features, community orientation, was separately identifiable.

Derivation and Reliability of the Primary Care Scales

Table 3 presents the results of the reliability analyses for both the original items and the final items (based on factor analysis). Item descriptive results (means and standard deviations) are also presented. Scale reliability measures include item-total correlation and alpha coefficient reliability. The distribution of the items varied significantly from a mean of 1.85 (ask about gun safety) to 3.73 (Provider answers questions in ways you understand) on the 4-point Likert-type scale. The distribution tends to skew toward more favorable answers (above 2.5). Apart from the gun safety item, only 2 items fell below a mean of 2 (1.94 for Provider knows neighborhood problems, 1.90 for Provider makes home visits). The first contactutilization and longitudiinalityinterpersonal relationships scales achieved the highest mean scores, whereas scales with lower means were community orientation, first contact-accessibility, and comprehensiveness-services received.

Eighteen of the 92 initial items were deleted on the basis of the criteria imposed for factor analyses. No items were deleted for first contact-utilization, coordination of services, comprehensiveness-services received, and community orientation scales. All items were deleted for family centeredness as were two thirds of the items for first contact-accessibility. Two items (out of 22) were deleted for longitudinality-interpersonal relationships and 3 (out of 24) for comprehensivenessservices available. Items from cultural competence were combined into first contact-accessibility. The revised scales demonstrate internal consistency reliability that was higher than or equal to the original scales, despite the reduction in number of items. Item-total correlations were also high and ranged from 0.34 (If sick, seen same day if office is open) to 0.91 (How to prevent hot water burns and How to prevent falls).

Testing the Likert Scaling Assumptions

Table 4 presents a summary of the results of the tests of Likert scaling assumptions using the revised items. All item-scale correlations well exceeded the accepted minimum (0.30) with the majority greater than 0.50 (Assumption 1). All 7 multi-item scales achieved 100% scaling success, indicating that all items in these scales correlated substantially higher with items in their hypothesized scale than with items in other scales (Assumption 2). Item means within each revised scale generally differed by less than six tenths of a point (except for first contact-accessibility) and item standard deviations within each scale by less than four tenths of a point (Assumption 3). Formal evidence of equal item variance was supported by the equivalence of the intraclass correlation and Scotts homogeneity ratio for each scale. Equal-item scale correlation (Assumption 4) was also observed through the range of item-scale correlations. As shown in column 1 (range of item-scale correlations), the range is relatively narrow (from .17 for coordination of services to .38 for comprehensiveness-services received). Finally, score reliability (Assumption 5) showed that except for first contact-utilization (only 3 items), all alpha levels exceeded .70 and were sufficiently high. Five of the 7 scales had alpha levels above .85.

Descriptive Feature of PCAT-AE

Table 5 displays estimates of central tendency and dispersion of scale score distributions for the 7 primary care scales in this South Carolina sample. Except for community orientation, all primary care scales were negatively skewed, indicating distributions with more positive ratings of primary care. The community orientation scale was positively skewed, indicating distributions with more negative ratings on the community orientation aspect of primary care. The full range of possible scores was observed for all scales except ongoing care.

The percentage of respondents scoring at the floor (the lowest score) or ceiling (the highest score) was acceptably low for all scales except first contactutilization, where 50% of the respondents scored the maximum score.

Table 6 compares the alpha coefficient and interfactor correlation for each primary care scale. The alpha coefficient of each scale substantially exceeded its correlation with all other primary care scales. None of the inter-factor correlations were excessively high, demonstrating that each primary care scale has significant unique contribution. All significant correlations were positive, indicating the complementary nature of primary care domains. Relatively high and positive interfactor correlations were observed between comprehensivenessservices received and comprehensiveness-services available (0.44), with the former and longitudinalityinterpersonal relationships (0.43), with the latter and coordination (0.38), and with comprehensivenessservices received and community orientation (0.37).

Discussion

Using patient-provided survey information collected within 2 health plans in South Carolina, we assessed the validity and reliability of the PCAT-AE. The results indicate that the hypothesized scales for primary care (first contactaccessibility, first con-tactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, and coordination) have substantial reliability and validity, consistent with the findings from the testing of the PCAT-CE.³⁰ The 2 versions of the instrument differ only in the comprehensiveness domains, as comprehensiveness implies that all common needs are met, and health needs in childhood are different from those in adults. In contrast, challenges to accessibility, to the nature of interpersonal relationships, and to coordination and community orientation are similar for both children and adults and thus can be assessed by the same items. Only 1 ancillary feature of primary care, community orientation, was retained as a separate dimension after factor analyses. The extracted factors explained 88.1 percent of the total variance in the item scores.

All of the 5 assumptions, including item-conver-gent validity, item-discriminant validity, equal item variance, equal item-scale correlation, and score reliability, were met. These results suggest that these items may be used to represent the primary care scales, and the scoring of these items may be summed without standardization or weighting, as with Likerts method of summated rating scales.³⁹

The resulting instrument has 74 items. Although the retained items adequately addressed first contactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, and coordination, and are consistent with the framework, those representing first contactaccessibility fell short. Only 4 of the 12 items measuring accessibility were retained. When more detail on accessibility is required, items that were deleted because they had lower item-total correlation may be added back in. Users should also review the comprehensiveness items to ascertain their relevance in the setting in which they are to be used. Items may be deleted if they are inappropriate in the context in which they are used; for example, in health systems that do not offer on-site testing for human immunodeficiency virus (HIV), because HIV is uncommon. Since continuity of care is an important component of primary care quality, a minimum number of visits or minimum duration with a regular source of care should be part of the assessment tool.

Separate factor analyses were performed with the 2 health plans. The results were largely comparable in terms of the factors that emerged as significant, indicating the generalizability of the tool to both vulnerable and middle-income populations. The only major differences are that the CHC subpopulation analysis yielded an additional significant factor, cultural competence, which the HMO subpopulation and the total population analyses failed to identify. In contrast, the HMO subpopulation analysis yielded an additional significant factor, family centeredness, which the CHC subpopulation and the total population analyses failed to identify. Thus, when using PCATon vulnerable populations (especially racial and ethnic minorities), questions measuring cultural competence might be retained. Family centeredness seemed to emerge as a distinct concept, primarily in the middle-income population.

There are a number of uses for a valid and reliable instrument such as the PCAT-AE. First, understanding primary care as a multidimensional concept is consistent with the IOMs conceptualization of primary care and more precisely captures the quality of primary care than unidimensional proxies, such as a clinicians medical specialty. With the 6 scales representing 4 core domains, the index representing strength of affiliation with a primary care provider, a scale for community orientation and the optional scales for family centeredness and cultural competence, all the important features of primary care are addressed. Second, PCAT-AE can be used as a quality measurement tool that assesses the adequacy of primary care experience rendered under different health care systems or settings, and for patients with different sociodemographic attributes. Third, PCAT-AE can also serve as a quality control tool that compares the quality of primary care given by providers of different types. The instrument can be used with other outcomes to assess the effect of policy interventions and systems changes on the delivery of critical aspects of primary care.

Limitations

Interpretation of our results should take into account some limitations. First, because our study was restricted to 1 locale, the generalizability of the PCAT-AE to other sites and states is not assured. Additional testing and validation is necessary to corroborate the current results. Second, the 74-item questionnaire remains lengthy and could have contributed to relatively high nonresponse and incompletion rates. Future validation work will concentrate on further reduction of the items to the very essential in order to reduce response burden. Regarding the ceiling effect of first contactutilization, future tests will be conducted in other settings with less of a managed care focus, as there well may be quite different distributions of responses in other settings. Third, outcomes of primary care are not the focus of the assessment tool. However, numerous studies have linked primary care to better health outcomes. Subsequent research may help explain which attributes are most conducive to better outcomes so that limited resources can be used to focus on them or a combination of them. Fourth, the measurement of primary care achievement is entirely based on respondents self-report. While self-report may be the best way to ascertain peoples experiences, it is subject to recall and response bias. Moreover, some aspects of technical quality cannot be assessed by patientsor consumers reports.

Despite these limitations, PCAT-AE is a valuable tool for capturing the principal domains of primary care. The next phase of our work seeks to assess the predictive validity of PCAT-AE, by examining the extent to which the principal attributes of primary care can be linked to the achievement of favorable health outcomes, their ability to manage their illnesses, and their satisfaction with the care received. Such work would advance our understanding of the relationship between how primary care is delivered and the health outcomes that result.

Related technical terms

Primary Care Attributes

First contactcare implies accessibility to and use of services for each new problem or new episode of a problem for which people seek health care.

Longitudinalitypresupposes the existence of a regular source of care and its use over time.

Comprehensivenessimplies that primary care facilities must be able to arrange for all types of health care services, including referrals to secondary services for consultation, tertiary services for specific conditions, and essential supporting services, such as home care and other community services.

Coordinationof care requires some form of continuity, either by practitioners, medical records, or both, as well as recognition of problems that are addressed elsewhere and the integration of their care into the total care of patients.

Family centerednessrefers to recognition of family factors related to the genesis and management of illness.

Community orientationrefers to the providers knowledge of community needs and involvement in the community.

Cultural competencerefers to the providers adaptation to facilitate relationships with populations having special cultural characteristics.

Measurement Concepts

Measurement validityrefers to the extent that important dimensions of a concept and their categories have been taken into account and appropriately operationalized.

Measurement reliabilityrefers to the extent that consistent results are obtained when a particular measure is applied to similar elements.

Construct validityis present when the measure captures the major dimensions of the concept under study.

Content validityrefers to the representativeness of the response categories used to represent each of the dimensions of a concept.

Concurrent validitymay be tested by comparing results of one measurement with those of a similar measurement administered to the same population and at approximately the same time. If both measurements yield similar results, then concurrent validity can be established.

Predictive validity exists when the results obtained from the measurement succeed in predicting the expected later-occurring event or circumstance.

Test-retest reliabilityinvolves administering the same measurement to the same individuals at 2 different times. If the correlation between the same measures is high, then the measurement is believed to be reliable.

Split-half reliabilityinvolves preparing 2 sets of measurement of the same concept, applying them to research subjects at one setting, and comparing the correlation between the 2 sets of measurement. To the extent the correlation is high, then the measurement is reliable.

Interrater reliabilityinvolves using different people to conduct the same procedure, whether it be interview, observation, coding, rating, and the like, and comparing the results of their work. To the extent that the results are highly similar, interrater reliability is established.

Item-convergent validityrefers to the substantial correlation between each item and its hypothesized scale.

Item-discriminant validityrefers to items within a scale that correlate more substantially with their hypothesized scale than with any other scale.

Equal item variancerefers to items within a scale that have approximately equal means and variances.

Equal item-scale correlationrefers to items in a scale that contribute approximately the same proportion of information about the underlying concept.

Score reliabilityrefers to scores of scales that are reproducible and reliable.

Skewnessrefers to distribution of observations that is not symmetric, ie, when more observations are found at one end of the distribution than the other.

Kurtosisrefers to the extent observations cluster around a central point more than in normal distribution.

Agrowing body of literature at both individual and ecologic levels has demonstrated the association of primary care and health outcomes.^1-11 Franks and Fiscella,¹² using nationally representative survey data, showed that adult respondents who reported a primary care physician rather than a specialist as their regular source of care had lower subsequent mortality and lower annual health care costs after controlling for differences in demographic characteristics, health insurance status, health perceptions, reported diagnoses, and smoking status. Both Shi^4,6 and Farmer and collegues¹³ found better health outcomes in states with higher primary care physician-population ratios after controlling for sociodemographic measures (% elderly, % urban, % minority, education, income, unemployment, pollution) and lifestyle factors (seatbelt usage, obesity, and smoking). Recent studies further showed that primary care may mitigate the adverse effects of income inequality on health.^14-16 Taken individually, each of the main features of primary care (person-focused care over time, accessible care, comprehensive in the sense of meeting all common health needs, and coordination when people have to receive services elsewhere) are known to improve both the effectiveness as well as the efficiency of care.^1,7,17-24

The mounting evidence associating primary care with improved health outcome has led to a rapid increase in interest in assessing primary care achievement by consumers and patients.^{18,19,21,25-28} Despite its importance, there currently is no way to assess the extent to which people receive adequate primary care; receiving care from a physician or physician designated as a primary care physician is at best only a proxy for actual adequacy of provision of primary care services. As a result, there are efforts to develop instruments that directly assess the adequacy of primary care.^20,29,30

The Primary Care Assessment Tool (PCAT) instruments developed by The Johns Hopkins Primary Care Policy Center for Underserved Populations were designed to measure the extent and quality of primary care services at a provider setting designated by consumers as their main source of general care and consistent with a focus on attributes of primary care that have been demonstrated to produce better outcomes of care at lower costs.²² The PCATfamily of instruments includes the Child Consumer/Client Survey, the Adult Consumer/Client Survey, and the Facility/Provider Survey. All surveys are based on self-report by patients or providers. The Consumer/Client Survey (both adult and child editions) is designed to collect information from consumers or family caretakers regarding their experience using health care resources. It may be used to survey target populations as defined by geography (community surveys), health plans, sites of care, payment mechanisms, or specific health care needs. The survey, which takes approximately 40 minutes to complete, can be administered through either telephone or face-to-face interviews, or by mail. Ahigh school reading level is required to self-administer the questionnaire. The Facility/Provider Survey is designed to collect information about specific operational characteristics and practices related to providing primary care from the viewpoint of practitioners, clinics, group practices, and institutions. This survey can also be implemented either by mail or by face-to-face or telephone interviews. It is parallel in content to the consumer/client survey. All 3 instruments are available for general use on request.

We report on the validation of the Consumer/Client Primary Care Assessment Tool Adult Edition (PCAT-AE). Its companion instrument for children (PCAT-CE) was previously validated.³⁰ Specifically, we assessed the congruence between the theoretically derived measures and the empiric results in terms of the underlying structure of the principal primary care domains within a diverse sample of populations including health maintenance organization (HMO) members and community health center (CHC) users. The validation process also served to reduce the number of items needed to assess the adequacy of primary care.

Methods

Subjects

The study participants were members of 2 health plans in 2 counties of South Carolina. Both counties are part of Columbia, the states capital and third largest city. One of the health plans (referred to as HMO) is licensed as an independent practice association (IPA) HMO model, in which primary care physicians act as gatekeepers and health care managers. Referral to specialists must be made through primary care physicians, and specialists must be affiliated with the HMO. The primary market has been large group employers, including employees of the state agencies and national and regional companies. Members of this plan are primarily from middle-income households. The other health plan (referred to as CHC) is a coalition of 12 Columbia-based health and social services provider organizations, including the county hospital, health department, department of social services, community health centers, and other social service agencies that provide services to lower income persons, such as Medicaid recipients and low-income households. These 2 plans were selected because they represent typical South Carolina managed care organizations and health plans for low-income individuals, respectively. Samples drawn from these 2 plans allowed us to test the reliability of PCATwith a diverse sample of populations, including both middle-income and low-income individuals using regular physician offices and community health centers, respectively.

Estimation of the sample size for this study involved several steps. First, an estimate of the likely proportions or means and standard deviations for each primary care measure was derived from a previous study.²⁵ When data were not available, a conservative estimate (eg, a larger standard deviation or proportion closer to 50/50) was made. Second, the estimates of the proportions, means, and standard deviations for the dependent variables were entered into the standard sample size formula for a two-group, cross-sectional sample. Using a 95% confidence interval, the largest sample size required was 300 per group. The CHC group was oversampled because of additional planned within-group analyses (not the focus of this paper). Finally, the desired sample size was adjusted for anticipated survey nonresponse (anticipated to be higher for a mail survey than a face-to-face interview).

For the HMO group, a mail survey was used since it was deemed most efficient. In 2 previous longitudinal studies of the same HMO, we used mail survey and telephone interviews alternately with a cohort of HMO members and obtained comparable results.^31,32 For this study, we sent a letter with a PCAT-AE questionnaire to 1000 randomly selected adult members to invite them to participate in the project. Because of known frequent changes in addresses, we recruited the non-HMO plan individuals and conducted in-person interviews at all the community health center sites where members came to the clinics for non-urgent visits. Patients were systematically approached while waiting for their scheduled appointment (ie, every nth patient based on expected visits for a particular site) and recruited for the study during a period of 4 weeks for each site.

Measures

Identification of Primary Care Source.Three questions were developed to identify an individuals usual source of care and the strength of that affiliation: (1) Is there a doctor or place that you usually go if you are sick or need advice about your health? (usual source), (2) Is there a doctor or place that knows you best as a person? (knows best), and (3) Is there a doctor or place that is most responsible for your health care? (most responsible). Aperson was considered to have a usual source of care if he or she answered positively to any 1 of the 3 questions (95% for the HMO plan and 90% for the low-income plan). Anegative answer to all 3 questions rendered the individual as not having a usual source of care.

An algorithm based on response to these 3 questions identified the strength of affiliation with the primary care source. If all 3 physicians/places were the same, this was considered evidence of a strong affiliation. If the response to the usual source question was the same as for either of the other 2 questions then that site was used although the affiliation was considered less strong. If the response for a usual source question was different from the other 2 responses but the other 2 responses were the same, then the site where both were the same was used (weak affiliation). If all 3 responses were different (weakest affiliation), then the site identified for usual source was used. All subsequent questions asked about this specific person or place. For those with no identifiable source of primary care, subsequent questions were asked about the last place that was visited.

Domains of Primary Care.The PCAT-AE was modeled on the previously validated PCAT-CE and is consistent with the 1978 Institute of Medicine (IOM) definition of primary care as accessibility, comprehensiveness, coordination, continuity, and accountability³³ and with the 1996 IOM report definition of primary care as the provision of integrated, accessible health care services by clinicians who are accountable for addressing a large majority of personal health care needs, developing a sustained partnership with patients, and practicing in the context of family and the community.³⁴ When combined into scales, the PCATsurvey items dealing with primary care quality were designed to measure each of the core domains of primary care; that is, first contact, longitudinality, comprehensiveness, and coordination (definitions of the primary care domains are provided in the Appendix).

Nine experts were asked to rate the appropriateness and representativeness of the primary care domain items. These experts consisted of 3 policymakers in federal agencies, 2 directors of community pediatrics at major medical centers, a health research director at a major HMO, 2 family medicine professors, and a general internal medicine physician. Acard sorting technique was used to determine the degree of congruence between each item and the domain it was designed to measure. Each survey question with its response categories and descriptions of each of the primary care domains was printed on separate index cards and mailed to the experts who assigned each question to one of the defined domains and suggested revisions and/or addition of other items. The percent agreement among the experts was used to determine the degree of congruence on the placement of each item in a particular domain. In addition, students in a graduate course on primary care independently assigned each item to a domain as well as to its appropriate subdomain.

In addition to the 4 core primary care domains, 3 other related domains (family centeredness, community orientation, and cultural competence) were included; these domains were considered derivative in that their achievement would be related to the achievement of the major domains.¹ However, they were separately specified as ancillary domains because of widespread appreciation of their likely importance.

Thus, the PCAT-AE consists of 7 domains represented by 9 scales. Each of the 4 core domains of primary care is represented by 2 components, 1 representing a characteristic of the facility of providers service organization and 1 representing a behavior of the provider or consumer.¹ One of these 8 potential components (longitudinality strength of affiliation) is represented by an index rather than a scale and is scored from the responses to the 3 questions noted under the heading Identification of the Primary Care Source. One subdomain, the facility characteristics related to the achievement of coordination, is obtainable only from the facility or provider, since consumers would not be expected to know the nature of information systems that facilitate coordination of care. Thus, the PCATinstrument has 6 scales representing the 4 primary care domains: first contactaccessibility, first contactutilization (first contact domain), longitudinalityinterpersonal relationships or ongoing care (longitudinality domain), coordination of services (coordination domain), comprehensiveness services available, comprehensivenessservices received (comprehensiveness domain) and the 3 ancillary domains of family centeredness, community orientation, and cultural competence.

For first contactaccessibility 12 questions were developed to measure access to the source of care. For first contactutilization 3 questions addressed the extent to which the source of care is first used for various types of problems. Twenty questions addressed the nature and strength of the person-focused relationship with the source of care over time (longitudinality). Eight questions were used to address the coordination of services between a primary care provider and specialty care. The comprehensivenessservices available domain included 24 items of important primary care services. An additional 13 questions were used to measure comprehensivenessservices received. Two items were used to measure family-centeredness, 5 community orientation, and 3 cultural competence. Copies of both the original questionnaire and the revised condensed version are available on request.

For consistency in response and scoring, all items representing the primary care domains were represented by a 4-point Likert-type scale (1=definitely not; 2=probably not; 3=probably; and 4=definitely). The sum score for each domain was derived by adding (after reverse-coding where appropriate) the values for all the items under each domain. An additional Dont Know/Cannot Remember option was also provided for each item. At least 3 methods could be used to code this category. The missing value method treats this item as missing for those who answer Dont Know/Cant Remember. The median value method assigns a value of 2.5 for those who answer Dont Know/Cant Remember. The imputation method imputes the response based on the mean of the results from other items within the domain when at least 50% of the items have been answered. Since the internal consistency reliability (a) is the highest based on the imputation method, this method is adopted in coding the Dont Know/Cant Remember category. However, the other 2 methods also produced high internal consistency reliability (results available on request).

Analysis

The purpose of the validation was to assess the congruence between the theoretically derived measures and the empiric results in terms of the underlying structure of the principal primary care domains. Although conceptual framework was relied on in the construction of primary care measures, empiric validation was used to reduce the number of items so that the questionnaire became more concise.

The validation of PCAT-AE with the South Carolina sample involved several steps. First, principal component factor analysis was used to explore the structure of the PCAT-AE items and examine its construct validity by determining if the items fell into the hypothesized scales (factors; definitions of measurement-related concepts used in this paper can be found in the Appendix). Factor analysis was also used for item selection and placement into scales based on the pattern of the factor loadings.³⁵ Four criteria were used in deleting items and the determination of the final factors.^36-37 Afactor loading greater than 0.35 was considered meaningful and used as a criterion for retaining items. In addition, each retained factor should have at least 3 items with loadings greater than 0.35. All retained items should share the same conceptual meaning or construct. Also, all retained items should not have secondary loadings greater than 0.35.

Second, internal consistency reliability of the primary care scales was assessed by Cronbachs coefficient alpha (a)³⁸ and item-total correlation for items in each domain. Cronbachs coefficient alpha is based on the covariance among individual items in a scale and the number of items. It ranges from 0, indicating total lack of consistency, to 1, indicating complete internal consistency reliability. The item-total correlation is the correlation between an individual item and the sum of the remaining items that constitute the scale. If an item-total correlation is small, the item is not considered to be measuring the same construct that is measured by the other items in the scale. All the retained items exceeded the minimum acceptable item-total correlation of 0.30.³⁸

Third, the Likert scaling assumptions were tested for the final items related to the primary care scales. Likerts method of summated rating scales is based on the assumption that item responses in each scale can be summed without standardization or weighting.³⁹ The underlying assumptions that must be met include: (1) item-convergent validity (tested by item-scale correlations); (2) item-discriminant validity (tested using the scaling success rate, ie, correlation of each item with other items within the same scale is greater than with items from different scales); (3) equal item variance (tested by examining item means and standard deviations and the equivalence of the intraclass correlation and Scotts homogeneity ratio for each scale); (4) equal item-scale correlation (tested by examining the range of item-scale correlations); and (5) score reliability (tested by Cronbachs coefficient a.

Fourth, descriptive statistics were performed for the revised primary care scales, including mean, standard deviation, range, percentile, skewness, kurtosis, and interscale correlation. Since respondents who never saw a specialist did not answer the coordination questions, analyses were performed both with and without those questions, including the coordination domain.

Results

Subjects

For the HMO group, a total of 350 individuals responded after 3 mailings. Excluding the nonresponses due to wrong addresses and changed plans (n=340), the effective response rate was 53 percent (350/660). The respondents and nonrespondents were not significantly different in age, sex, race, and zip codes of mailing addresses. For the CHC group, a total of 1000 individuals were systematically selected and approached. Among them, 265 refused to be interviewed, 195 were not able to complete the interview prior to their appointment, and 540 completed the interview. Taking only refusal into account, the response rate was 67% (540/540+265). Men were more likely to refuse the interview than women. There were no significant differences in age and race between respondents and nonrespondents. All interviews were conducted by graduate public health students trained in interactive sessions and were completed in 1999.

The sample included 823 adults with an identified usual source of care. Among them, most (69% of HMO and 60% of CHC respondents) indicated a strong affiliation with their usual source of care (ie, all 3 doctors/places were the same). Very few (0.6% of HMO and 1.2% of CHC respondents) indicated the weakest affiliation with their usual source of care (ie, all 3 responses were different). Just over half of respondents (56%) were non-white (primarily black). Over half (55%) had an annual household income under $25,000. Most respondents (76%) had health insurance coverage all year and had been seeing their regular source of care for more than 1 year (82%). Sixty-three percent had seen their regular source of care for more than 2 years. The majority chose their own usual source of care (78%) and did not have trouble paying for their health care (74%). More than half of the respondents made at least 1 visit to a specialist (56%). This relatively high rate may be due to a somewhat elderly sample; more than 20% of the respondents were older than 65 years.

Table 1 compares the HMO sample with the CHC sample on sociodemographic and health care utilization measures. The HMO sample included predominantly white (81.6%) and higher income subjects (86.8% with annual household income of $25,000 or more). In contrast, the CHC sample included predominantly non-white (83.2%) and lower income subjects (85.9% with an annual household income less than $25,000). Compared with the CHC respondents, HMO subjects had been seeing their regular source of care for a longer time, were more likely to choose their own doctors and visit a specialist, and less likely to have trouble paying for their health care.

Factor Analysis and Construct Validity

In the initial exploratory factor analysis, all 92 applicable questionnaire items measuring the subdomains and domains of primary carefirst contact, longitudinality, comprehensiveness, coordination, family centeredness, community orientation, and cultural competencewere included. Based on the results of the initial factor analysis, 4 criteria were applied to reach the final solution (Table 2; initial factor analyses not shown but available upon request).

Seven common factors were extracted, corresponding to the hypothesized primary care scales: first contactaccessibility, first contactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, coordination, and community orientation (Table 2). Those extracted factors explained 88.1% of the common variance. Eigenvalues ranged from 16.17 to 1.16. All principal primary care domains were extracted as hypothesized. Only 1 of the 3 derivative features, community orientation, was separately identifiable.

Derivation and Reliability of the Primary Care Scales

Table 3 presents the results of the reliability analyses for both the original items and the final items (based on factor analysis). Item descriptive results (means and standard deviations) are also presented. Scale reliability measures include item-total correlation and alpha coefficient reliability. The distribution of the items varied significantly from a mean of 1.85 (ask about gun safety) to 3.73 (Provider answers questions in ways you understand) on the 4-point Likert-type scale. The distribution tends to skew toward more favorable answers (above 2.5). Apart from the gun safety item, only 2 items fell below a mean of 2 (1.94 for Provider knows neighborhood problems, 1.90 for Provider makes home visits). The first contactutilization and longitudiinalityinterpersonal relationships scales achieved the highest mean scores, whereas scales with lower means were community orientation, first contact-accessibility, and comprehensiveness-services received.

Eighteen of the 92 initial items were deleted on the basis of the criteria imposed for factor analyses. No items were deleted for first contact-utilization, coordination of services, comprehensiveness-services received, and community orientation scales. All items were deleted for family centeredness as were two thirds of the items for first contact-accessibility. Two items (out of 22) were deleted for longitudinality-interpersonal relationships and 3 (out of 24) for comprehensivenessservices available. Items from cultural competence were combined into first contact-accessibility. The revised scales demonstrate internal consistency reliability that was higher than or equal to the original scales, despite the reduction in number of items. Item-total correlations were also high and ranged from 0.34 (If sick, seen same day if office is open) to 0.91 (How to prevent hot water burns and How to prevent falls).

Testing the Likert Scaling Assumptions

Table 4 presents a summary of the results of the tests of Likert scaling assumptions using the revised items. All item-scale correlations well exceeded the accepted minimum (0.30) with the majority greater than 0.50 (Assumption 1). All 7 multi-item scales achieved 100% scaling success, indicating that all items in these scales correlated substantially higher with items in their hypothesized scale than with items in other scales (Assumption 2). Item means within each revised scale generally differed by less than six tenths of a point (except for first contact-accessibility) and item standard deviations within each scale by less than four tenths of a point (Assumption 3). Formal evidence of equal item variance was supported by the equivalence of the intraclass correlation and Scotts homogeneity ratio for each scale. Equal-item scale correlation (Assumption 4) was also observed through the range of item-scale correlations. As shown in column 1 (range of item-scale correlations), the range is relatively narrow (from .17 for coordination of services to .38 for comprehensiveness-services received). Finally, score reliability (Assumption 5) showed that except for first contact-utilization (only 3 items), all alpha levels exceeded .70 and were sufficiently high. Five of the 7 scales had alpha levels above .85.

Descriptive Feature of PCAT-AE

Table 5 displays estimates of central tendency and dispersion of scale score distributions for the 7 primary care scales in this South Carolina sample. Except for community orientation, all primary care scales were negatively skewed, indicating distributions with more positive ratings of primary care. The community orientation scale was positively skewed, indicating distributions with more negative ratings on the community orientation aspect of primary care. The full range of possible scores was observed for all scales except ongoing care.

The percentage of respondents scoring at the floor (the lowest score) or ceiling (the highest score) was acceptably low for all scales except first contactutilization, where 50% of the respondents scored the maximum score.

Table 6 compares the alpha coefficient and interfactor correlation for each primary care scale. The alpha coefficient of each scale substantially exceeded its correlation with all other primary care scales. None of the inter-factor correlations were excessively high, demonstrating that each primary care scale has significant unique contribution. All significant correlations were positive, indicating the complementary nature of primary care domains. Relatively high and positive interfactor correlations were observed between comprehensivenessservices received and comprehensiveness-services available (0.44), with the former and longitudinalityinterpersonal relationships (0.43), with the latter and coordination (0.38), and with comprehensivenessservices received and community orientation (0.37).

Discussion

Using patient-provided survey information collected within 2 health plans in South Carolina, we assessed the validity and reliability of the PCAT-AE. The results indicate that the hypothesized scales for primary care (first contactaccessibility, first con-tactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, and coordination) have substantial reliability and validity, consistent with the findings from the testing of the PCAT-CE.³⁰ The 2 versions of the instrument differ only in the comprehensiveness domains, as comprehensiveness implies that all common needs are met, and health needs in childhood are different from those in adults. In contrast, challenges to accessibility, to the nature of interpersonal relationships, and to coordination and community orientation are similar for both children and adults and thus can be assessed by the same items. Only 1 ancillary feature of primary care, community orientation, was retained as a separate dimension after factor analyses. The extracted factors explained 88.1 percent of the total variance in the item scores.

All of the 5 assumptions, including item-conver-gent validity, item-discriminant validity, equal item variance, equal item-scale correlation, and score reliability, were met. These results suggest that these items may be used to represent the primary care scales, and the scoring of these items may be summed without standardization or weighting, as with Likerts method of summated rating scales.³⁹

The resulting instrument has 74 items. Although the retained items adequately addressed first contactutilization, longitudinalityinterpersonal relationships, comprehensivenessservices available, comprehensivenessservices received, and coordination, and are consistent with the framework, those representing first contactaccessibility fell short. Only 4 of the 12 items measuring accessibility were retained. When more detail on accessibility is required, items that were deleted because they had lower item-total correlation may be added back in. Users should also review the comprehensiveness items to ascertain their relevance in the setting in which they are to be used. Items may be deleted if they are inappropriate in the context in which they are used; for example, in health systems that do not offer on-site testing for human immunodeficiency virus (HIV), because HIV is uncommon. Since continuity of care is an important component of primary care quality, a minimum number of visits or minimum duration with a regular source of care should be part of the assessment tool.

Separate factor analyses were performed with the 2 health plans. The results were largely comparable in terms of the factors that emerged as significant, indicating the generalizability of the tool to both vulnerable and middle-income populations. The only major differences are that the CHC subpopulation analysis yielded an additional significant factor, cultural competence, which the HMO subpopulation and the total population analyses failed to identify. In contrast, the HMO subpopulation analysis yielded an additional significant factor, family centeredness, which the CHC subpopulation and the total population analyses failed to identify. Thus, when using PCATon vulnerable populations (especially racial and ethnic minorities), questions measuring cultural competence might be retained. Family centeredness seemed to emerge as a distinct concept, primarily in the middle-income population.

There are a number of uses for a valid and reliable instrument such as the PCAT-AE. First, understanding primary care as a multidimensional concept is consistent with the IOMs conceptualization of primary care and more precisely captures the quality of primary care than unidimensional proxies, such as a clinicians medical specialty. With the 6 scales representing 4 core domains, the index representing strength of affiliation with a primary care provider, a scale for community orientation and the optional scales for family centeredness and cultural competence, all the important features of primary care are addressed. Second, PCAT-AE can be used as a quality measurement tool that assesses the adequacy of primary care experience rendered under different health care systems or settings, and for patients with different sociodemographic attributes. Third, PCAT-AE can also serve as a quality control tool that compares the quality of primary care given by providers of different types. The instrument can be used with other outcomes to assess the effect of policy interventions and systems changes on the delivery of critical aspects of primary care.

Limitations

Interpretation of our results should take into account some limitations. First, because our study was restricted to 1 locale, the generalizability of the PCAT-AE to other sites and states is not assured. Additional testing and validation is necessary to corroborate the current results. Second, the 74-item questionnaire remains lengthy and could have contributed to relatively high nonresponse and incompletion rates. Future validation work will concentrate on further reduction of the items to the very essential in order to reduce response burden. Regarding the ceiling effect of first contactutilization, future tests will be conducted in other settings with less of a managed care focus, as there well may be quite different distributions of responses in other settings. Third, outcomes of primary care are not the focus of the assessment tool. However, numerous studies have linked primary care to better health outcomes. Subsequent research may help explain which attributes are most conducive to better outcomes so that limited resources can be used to focus on them or a combination of them. Fourth, the measurement of primary care achievement is entirely based on respondents self-report. While self-report may be the best way to ascertain peoples experiences, it is subject to recall and response bias. Moreover, some aspects of technical quality cannot be assessed by patientsor consumers reports.

Despite these limitations, PCAT-AE is a valuable tool for capturing the principal domains of primary care. The next phase of our work seeks to assess the predictive validity of PCAT-AE, by examining the extent to which the principal attributes of primary care can be linked to the achievement of favorable health outcomes, their ability to manage their illnesses, and their satisfaction with the care received. Such work would advance our understanding of the relationship between how primary care is delivered and the health outcomes that result.

Related technical terms

Primary Care Attributes

First contactcare implies accessibility to and use of services for each new problem or new episode of a problem for which people seek health care.

Longitudinalitypresupposes the existence of a regular source of care and its use over time.

Comprehensivenessimplies that primary care facilities must be able to arrange for all types of health care services, including referrals to secondary services for consultation, tertiary services for specific conditions, and essential supporting services, such as home care and other community services.

Coordinationof care requires some form of continuity, either by practitioners, medical records, or both, as well as recognition of problems that are addressed elsewhere and the integration of their care into the total care of patients.

Family centerednessrefers to recognition of family factors related to the genesis and management of illness.

Community orientationrefers to the providers knowledge of community needs and involvement in the community.

Cultural competencerefers to the providers adaptation to facilitate relationships with populations having special cultural characteristics.

Measurement Concepts

Measurement validityrefers to the extent that important dimensions of a concept and their categories have been taken into account and appropriately operationalized.

Measurement reliabilityrefers to the extent that consistent results are obtained when a particular measure is applied to similar elements.

Construct validityis present when the measure captures the major dimensions of the concept under study.

Content validityrefers to the representativeness of the response categories used to represent each of the dimensions of a concept.

Concurrent validitymay be tested by comparing results of one measurement with those of a similar measurement administered to the same population and at approximately the same time. If both measurements yield similar results, then concurrent validity can be established.

Predictive validity exists when the results obtained from the measurement succeed in predicting the expected later-occurring event or circumstance.

Test-retest reliabilityinvolves administering the same measurement to the same individuals at 2 different times. If the correlation between the same measures is high, then the measurement is believed to be reliable.

Split-half reliabilityinvolves preparing 2 sets of measurement of the same concept, applying them to research subjects at one setting, and comparing the correlation between the 2 sets of measurement. To the extent the correlation is high, then the measurement is reliable.

Interrater reliabilityinvolves using different people to conduct the same procedure, whether it be interview, observation, coding, rating, and the like, and comparing the results of their work. To the extent that the results are highly similar, interrater reliability is established.

Item-convergent validityrefers to the substantial correlation between each item and its hypothesized scale.

Item-discriminant validityrefers to items within a scale that correlate more substantially with their hypothesized scale than with any other scale.

Equal item variancerefers to items within a scale that have approximately equal means and variances.

Equal item-scale correlationrefers to items in a scale that contribute approximately the same proportion of information about the underlying concept.

Score reliabilityrefers to scores of scales that are reproducible and reliable.

Skewnessrefers to distribution of observations that is not symmetric, ie, when more observations are found at one end of the distribution than the other.

Kurtosisrefers to the extent observations cluster around a central point more than in normal distribution.