Frequently Asked Questions

The goal of National Core Indicators is to understand the performance of state service delivery systems. To evaluate how these systems are working and where they need improvements, NCI maintains very broad inclusion criteria for each survey.

National Core Indicators – Intellectual and Developmental Disabilities (NCI-IDD)

·       In-Person Survey (IPS): Adults who receive at least one publicly funded service (not including case management) from the developmental disability services in their state are eligible to participate.

·       Family Guardian Survey (FGS): Families who have an adult family member (18 years of age or older) who does not live in the family home and receives one service (not including case management) from the DD services in their state are eligible to participate.

·       Adult Family Survey (AFS): Families with an adult family member (18 years of age or older) who lives in the family home AND receives one service (not including case management) from the DD services in their state are eligible to participate.

·       Child Family Survey (CFS): Families who have a child (less than 18years old) who lives in the family home and receives at least one service (not including case management) from DD services in their state are eligible to participate.

National Core Indicators – Aging and Disabilities (NCI-AD)

Adult Consumer Survey (ACS): People who receive long-term services and supports as defined as at least one “active service” (not including case management) at least twice a week. For instance, chore services, personal care attendant, day services, home delivered meals.

The NCI is a survey, not a census; participating states do not survey every person who receives services in their state. Instead, NCI uses rigorous sampling techniques to hear from a representative sample of people who use state services.

National Core Indicators – Intellectual and Developmental Disabilities (NCI-IDD)

IPS

·       States select samples based on which populations they want to examine. Some states include people in institutional and immediate care facility settings, while other states only survey people living in home and community-based settings.

·       Each state attempts to complete 400 or more surveys. Completing 400 surveys guarantees valid comparisons to be made across states with a 95% confidence interval and a margin of error of +/- 5%.  What this means is that a sample of 400 allows us to be confident that the sample we have drawn is representative of the majority of the population. Any additional participants would be expected to fall just 5% above or 5% below the average. This helps to make it possible to compare across states where service systems are sometimes quite different.

·       Participating states use simple random sampling, stratified sampling, or proportionate sampling techniques to select participants, or sometimes a combination of techniques. States that use a stratified or proportionate sampling technique typically do so based on types of services or funding source, race/ethnicity, or by region. After stratifying their sample, states uses random sampling to select participants.

Family surveys (AFS, FGS, CFS)

·       States are asked to administer the family surveys by selecting a random sample of at least 1,000 families. Assuming a 40% return rate, this would result in 400 responses. Some states send the survey to all eligible participants. We recommend that states mail paper surveys and include a URL link for families to complete surveys online, if desired.

‍National Core Indicators – Aging and Disabilities (NCI-AD)

·       States select their samples based on which populations they want to examine. States determine which long-term services and supports (LTSS) programs from which to sample, this may include: HCBS Medicaid Waivers, Older Americans Act (OAA) (cluster I services), Money Follows the Person (MFP), PACE, Nursing Facilities, etc.

·       Each state attempts to complete 400 or more surveys. Completing 400 surveys guarantees valid comparisons to be made across states with a 95% confidence interval and a margin of error of +/- 5%.  What this means is that a sample of 400 allows us to be confident that the sample we have drawn is representative of the majority of the population. Any additional participants would be expected to fall just 5% above or 5% below the average. Most states conduct more surveys in order to make comparisons across programs or other groups.

·       Participating states use simple random sampling, stratified sampling, or proportionate sampling techniques to select participants, or sometimes a combination of techniques. States that use a stratified or proportionate sampling technique typically do so based on types of services or funding source, race/ethnicity, or by region. After stratifying their sample, states use random sampling to select participants.

Background information includes details about a person’s characteristics and the services they receive from the state. For each of the tools, there are methods for obtaining background information.

National Core Indicators –Intellectual and Developmental Disabilities (NCI-IDD)

IPS

Sources of background information include:

·       Case management records

·       Provider records

·       Managed Care Entity records

·       State DD data systems

·       Medicaid billing records

·       Individual Service Plans

Background information from records includes:

·       Residential information, including whether people live in institutional settings or home and community-based settings

·       Demographic details such as age, sex, race and ethnicity, marital and parental status

·       Guardianship status

·       ID diagnosis and level of ID

·       Mental health conditions, disability type, chronic health conditions

·       Preferred language and means of communication

·       Mobility

·       Healthcare utilization, Height and weight, Health behaviors (e.g., use of tobacco products)

·       Medications for mental health and behavioral support needs

·       Amount of paid supports

·       Employment setting, hours, wages, and whether community employment goal is in service plan

·       Funding source and Medicare enrollment

·       Service plan goal re: relationships, re: participation in community, re: independence/functional skills

·       Use of self-directed supports option

Background information that can be self-reported:

·      Preferred communication modes and language

·      Mobility

·      Healthcare utilization

Family Surveys (AFS, FGS, CFS):

·       All background information is reported by the responding family member

National Core Indicators – Aging and Disabilities (NCI-AD)

ACS

Sources of background information include:

·       Case management records

·       Provider records

·       Managed Care Entity records

·       Medicaid billing records

·       Eligibility records

·       Individual Service Plans

·       State data systems

Background information from records includes:

·       Primary source of funding

·       LTSS services received

·       Length of receiving services

·       Participation in a self-directed supports option

·       Legal guardianship status

Additional information that often comes from background information but, if missing, can be asked at the end of the survey meeting includes:

·       Age

·       Gender

·       Race and ethnicity

·       Marital status

·       Preferred language and means of communication

·       Residential information

·       Disability (physical, TBI, or IDD)

·       Chronic health conditions

·       Mental health diagnosis

·       Mobility

·       History of falls

Self-reported background information that is always asked of the participant includes:

·       Self-identified race and ethnicity

NCI is committed to transparency and confidentiality. In support of this, we have a variety of procedures to protect privacy of participants as outlined below.

·      Data from standard NCI data collection are collected through HSRI’s propriety Online Data Entry System Application (ODESA). ODESA has several layers of security, including access control, authentication and authorization controls, encrypted data storage and transmittal, and physical security for data servers.

·      Data analysis files never include personal health information or personal identifiers.

·      When data is shared with collaborators, all data is deidentified by removing information about state of residence.

NCI wants to hear from everyone who uses services about their experiences. Sometimes, people who want to participate need additional supports to do so. For some, this may be a person who knows them well who can respond to questions on their behalf. If NCI did not include proxies, there could be a large percentage of individuals who would not be represented in NCI.

Proxy determination process

·       The process for figuring out if a proxy will be used is called the “proxy determination process.” This process involves asking participants sample questions. Based on the responses to the sample questions, the surveyor is trained to be able to decide whether a proxy is needed. If a participant cannot answer the questions validly, they ask proxy allowed questions only – the IPS and ACS have survey sections that consolidate proxy-only questions for ease of survey administration.

·       The use of proxy participants for the NCI tools is limited to questions that are observable and/or measurable and on which others may reliably report.

·       Of note, when administering the proxy-allowed questions, proxies can respond to the surveyor, but the surveyor confirms those responses with the person who uses services by saying things such as: “How does that sound to you?” or “Do you have more to add?”

·       To avoid conflicts of interest, service coordinators are not allowed to respond as proxies. Further, if both the participant and a proxy participant answer a question, the participant’s answer is recorded so long as his/her/their answer is deemed reliable by the surveyor. Surveyors also keep track of who responds to each question—the participant or the proxy—and the proxy’s relationship to the participant.

Participation from family/caregivers

·       In some instances, an NCI participant will request to have a family member or caregiver present during the interview. If the participant does not require a proxy, but the family member or caregiver is attempting to respond to the questions on behalf of the person, surveyors are trained to ensure the participant’s response is recorded and remind the family members/caregivers of the importance of hearing directly from the person who uses services.

Every state works with their own legal departments to determine the state-specific consent requirements. NCI shares recommended resources and guidance on consent, including using plain language. Some states also work with Institutional Review Boards (IRBs) to ensure that state-specific consent and survey administration protocols are in line with rules about human subjects research.

Some of the considerations for states as they think through consent processes include:

·      Legal guardians - If a potential participant has a legal guardian, most states require permission from the guardian before the can survey the participant. Usually, the participant is also asked for their assent to participate.

·      Documenting consent - Some states require a signature for written consent. Other states obtain verbal consent and document they have obtained consent.

‍

A key principle of NCI is to hear directly for people who use services. We use several methods to enable equitable participation in the surveys for all people with disabilities and older adults, such as:

Talking with the person

·      Questions are asked directly of people in a face-to-face method (either in-person or remotely by video conference or telephone). The interview is always conducted in place where the person feels comfortable.

·      Questions are asked in an open-ended fashion first, and surveyors are instructed to summarize what they hear with the appropriate answer. Surveyors then ask for confirmation, and in this way move people from their completely unique and authentic response to a more standardized response.

Survey tools

·      NCI-IDD and NCI-AD are routinely revised based on feedback from states, people with lived experience with disability, families, and surveyors about how to make questions accessible and easy to understand.

·      Cognitive testing of NCI explored allowing surveyors the option to "reword" questions to include familiar names or terms if a participant does not understand the question wording. The evidence of this testing demonstrated more reliable responses from the IDD population.

Training surveyors

·      NCI national staff facilitate a rigorous, standardized training required for NCI-IDD and NCI-AD surveyors each year. See below for more details.

·      Several states use peer-surveyors for the NCI-IDD IPS. Having peers with disabilities collect data from participants with disabilities can help people feel more comfortable responding to questions and help amplify the voices of people with disabilities.

Using interpreters

·      NCI-IDD surveys have been translated into Spanish and Chinese. The NCI-AD survey has been translated into Spanish. Some states translate the surveys into additional languages (e.g., Russian, Vietnamese, Farsi, or those of local populations). Trained surveyors who speak those languages can administer the translated tool. However, if there is not a trained surveyor who speaks that language, participants who do not speak English or prefer to take the survey in another language are offered an interpreter, who translates what the surveyor asks.

·      The surveyor is still the person responsible for recording the responses.

People with speech disabilities

·      People who cannot rely on speech alone to be heard and understood are offered accommodations to participate. For some people, this maybe a person who knows them and can interpret their communication, an ASL interpreter, the use of speech-generating devices and other forms of augmentative and alternative communication (AAC).

·      In the future, we hope to explore additional ways to support the full and equitable participation of people with speech disabilities.

Researchers both internal and external to NCI have examined the psychometric properties of the various surveys, including individual items and multi-item scales of NCI. In addition, ongoing monitoring of NCI tools allows us to respond quickly to any potential concerns about validity and reliability.

National Core Indicators – Intellectual and Developmental Disabilities (NCI-IDD)

Validity

·       Face and content validity - The IPS was designed with input from a Project Advisory Committee with extensive experience in developing survey tools and methods to measure service system outcomes. The tool is regularly evaluated with input from people who use services and other subject matter experts to maintain high levels of face validity and content validity.

·       Criterion validity - We examined the extent to which a state’s score on a measure corelates with another factor with which it would be expected to correlate. Across these analyses, NCI data elements correlate with each other as would be logically expected and align with other measures that assess similar outcomes.

Reliability

·       Inter-rater reliability - There have been numerous studies of inter-rater reliability and percent agreement of NCI IDD survey tools. Agreement is consistently higher than 90%, with Kappa scores ranging from 0.79-0.95. These are consistent with high levels of reliability.

·       Inter-unit reliability - Between-state variation is significantly larger than within-state variation. The IUR for 14 measures that were endorsed as quality measures ranged between 0.75-0.98.

·      Multi-item scales

       o   Internal consistency - depending on the scale ranged from 0.35-0.69

        o   Principal components analysis (PCA) - Exploratory factor analysis was used to select items for five multi-item scales. Confirmatory factor analysis was then used to test the factor structure of the five multi-item scales. Each multi-item scale constituted a single factor that explained between 44%-71% of the variance.

We are in the process of creating a manual of the psychometric properties of NCI, which will compile the results from the many individual studies of NCI psychometrics and our process for risk adjustment. Stay tuned for more!

National Core Indicators – Aging and Disabilities (NCI-AD)

Validity

·       Face and content validity - The ACS was designed with input from a steering committee of NASUAD (now ADvancing States) members from 19 state departments on aging and Medicaid services. These members provided input on the indicators, survey design, question word, and methods to compile performance data on an ongoing basis. Additionally, subject matter experts from 13 national associations and universities provided feedback on the initial survey tool. As with the NCI-IDD tools, the ACS is regularly evaluated by subject matter experts, including people with lived experience with disability, to maintain high levels of face and content validity.

·       Content validity - The ACS development process began through extensive review of existing validated tools for quality of life for the general population and people with physical disabilities, as well as tools to measure satisfactions with services. From these tools, 120 indicators from 17 domains were identified; a Delphi approach was used to narrow the list of indicators to 61. Subject matter experts provided feedback on iterative revisions to questions, and two focus groups with people who receive services helped inform subsequent revisions.

Reliability

·       Inter-rater reliability - A mini-pilot of inter-rater reliability demonstrated a high level of agreement (Cohen’s Kappa>0.80)

The NCI national team administers several trainings for surveyors throughout the year. The goal of trainings is to provide a standardized training to ensure uniform application of the survey. Some additional information about surveyors and training to note are:

·       States employ a variety of individuals to conduct the face-to-face meetings for the NCI-IDD and NCI-AD surveys, such as: state staff, private contractors, advocacy organizations, students, etc.

·       A participant’s service provider, relative, personal case manager, staff member, or other close contact is not allowed to administer the survey.

·       Training includes asynchronous and synchronous trainings with in-depth review of: the NCI-IDD and NCI-AD program, survey administration protocol and procedures, and the survey tool.

·       The time to complete the training is approximately 6 hours. Surveyors must pass a knowledge check during the online asynchronous training with a score of at least 80%.

·       Many states also require that new surveyors complete a practice survey and achieve acceptable inter-rater reliability scores, and new surveyors typically shadow and are shadowed by more experienced surveyors for their first several NCI surveys.

Our framework for individual quality of life is informed by the work of Dr. Robert Schalock and others around the domains of quality of life, including: Emotional well-being, interpersonal relations, material well-being, personal development, physical well-being, self-determination, social inclusion, and rights.

NCI Domains

NCI measures outcomes that can be categorized into three major areas: individual outcomes, family outcomes, system performance, and health, welfare, and rights. Specific subdomains can be found at the links below:

·     NCI-AD domains and subdomains

·     NCI-IDD domains and subdomains

Quality Measures

In addition to understanding individual level outcomes, NCI aims to examine system-level quality through annual measurement efforts. Recent efforts have focused on developing measures of quality of life that can be examined in aggregate (e.g., for people who using the LTSS service system) instead of at the individual level. These efforts include:

·       NQF Quality Measures: NCI successfully submitted 14 measures for endorsement by National Quality Forum (now the former consensus-based entity for CMS) as measures of HCBS quality in 2021. For more information about these measures, please click here.

·       HCBS Recommended Quality Measure set: As part of CMS’ ongoing efforts to promote quality measures for HCBS systems performance, in 2022 they released the first set of recommended quality measures. NCI-IDD and NCI-AD measures represented 34 of the recommended measures. More information about these measures can be found here.

States are encouraged to use NCI in conjunction with other state data sources to examine their system performance compared to national benchmarks. This can inform strategic planning, policy reports, and quality improvement initiatives. Many states also share the reports with Quality Councils, community partners, and service users to obtain feedback about priorities for services. Some examples of the way NCI data can be used are listed below.

Filter or stratify reporting and examine for inequities by population

·       As described in the Access Rule NPRM, CMS may ask that states look at indicators by various demographic groups. NCI data can be stratified by race, ethnicity, sex, age, rural/urban status, disability, language, and other factors.

·       Currently NCI-IDD reports display many outcomes stratified by type of residence, and NCI-AD reports display many outcomes by program type. These reports are intended to support exploration of how outcomes differ in association with where people live and which services they receive and showcase one example of how states can stratify their data.

Look at year over year comparisons

·       Based on changes to policies and programs, there is often interest in looking at trends in outcomes. For example, we have been asked before if employment outcomes are getting better as more states pass employment first legislation and phase out subminimum wage laws. Individuals who wish to look at trends over time are encouraged to look at different years of reports, and please don’t hesitate to reach out to us with any questions.

Examine alignment to and progress towards compliance with federal and state level policies and regulations

·       For example, the HCBS Final Settings Rule requires that people who receive services have access to the greater community, autonomy, rights, and that they drive the process of developing person-centered plans. There are a number of measures from NCI that can be used to explore state progress towards compliance. Some additional resources for further reading include:

-NCI-AD crosswalk

-NCI-IDD crosswalk

-Case study

There has been recent interest in value-based purchasing or value-based payment for Medicaid LTSS services. This means that providers would be rewarded if the people who they provide services to have better outcomes, as opposed to be paid based on the services they provide. In addition, some states have explored using participation in the State of the Workforce survey as a criterion for value-based reimbursement. Other states have added questions to the NCI surveys that are in line with work that is being done related to value-based payment.

NCI reports show the percent of people that report having a particular outcome, whether that is having a job, having friends, having access to transportation, or something else. Some of the responses to these types of questions including things like: “Yes,” “In-between”/”Maybe”/”Sometimes” or “No”. Sometimes, it makes sense to report just the percent that say “Yes,” and sometimes it makes sense to combine “Yes” and “Maybe” to see the bigger picture.

·       For example, when we ask people if they got to choose where they live, we often hear people talk about making choices by themselves, making choices with the input of their trusted friends and family, or not having input. If we report just the percent of people who chose, we would fail to understand people who may have a guardianship and cannot make the decision, but did provide input. We make decisions about how to combine responses in the way we present data in order to measure people who have the opportunity to participate in things like making decisions.

Participating states own their data, and de-identified NCI data is available for secondary analysis for a small fee. If you are interested in learning more about the data sharing process, please reach out to Dorothy Hiersteiner or Stephanie Giordano.

Selected articles from research partners featuring NCI data

·      Stancliffe, R.J., Tichá, R., Pettingell, S.L., Houseworth, J. & Bershadsky, J. (2023). Current services and outcomes of formerly institutionalised and never-institutionalised US adults with intellectual and developmental disabilities: A propensity score matching analysis  https://doi.org/10.1111/jar.13103  

·      Chong N, Akobirshoev I, Caldwell J, Kaye HS, Mitra M. (2022). The relationship between unmet need for home and community-based services and health and community living outcomes. Disabil Health J., 15(2):101222. doi:10.1016/j.dhjo.2021.101222.

·      Prohn, S.M., Dinora, P., Broda, M.D., Bogenschutz, M. & Lineberry, S. (2022). Measuring Four Personal Opportunities for Adults With Intellectual and Developmental Disabilities.  Inclusion, 10 (1):19–34. https://doi.org/10.1352/2326-6988-10.1.19  

·      Broda, M.D., Bogenschutz, M., Dinora, P., Prohn, S.M., Lineberry, S. & Ross, E. (2021). Using Machine Learning to Predict Patterns of Employment and Day Program Participation. Am J Intellect Dev Disabil, 126 (6): 477–491. https://doi.org/10.1352/1944-7558-126.6.477  

·      Houseworth, J., Stancliffe, R., & Tichá, R. (2019). Examining the National Core Indicators' Potential to Monitor Rights of People with Intellectual and Developmental Disabilities According to the CRPD. Journal of Policy and Practice in Intellectual Disabilities, 16(4), 342-351. 

·      Houseworth, J., Stancliffe, R.J., & Tichá, R. (2018). Association of state-level and individual-level factors with choice making of individuals with intellectual and developmental disabilities. Research in Developmental Disabilities, 83,77-90. 

·      Tichá, R., Qian, X., Stancliffe, R.J., Larson, S., & Bonardi, A. (2018). Alignment between the Convention on the Rights of Persons with Disabilities and the National Core Indicators. Journal of Policy and Practice in Intellectual Disabilities, 15(3),247-255.