Project Activities
The researchers will conduct a comprehensive needs assessment across adult education, workforce investment, and other adult programs to identify the most critical skills and competencies learners, adult educators, counselors, and employers need to support adult learners. They will leverage this information and existing assessment items and tasks to the ASAP assessment system that will be personalized to the learner, delivered digitally, and provide diagnostic and actionable information for adult educators, learners, and employers. They will ensure the validity and fairness of the assessments using psychometric procedures.
Structured Abstract
Setting
This research will take place primarily in Massachusetts and Nebraska but will also involve partners and adult education students across the U.S.
Sample
Approximately 12 experts will participate in the needs assessment. For the course of the development and validation work, approximately 20,000 adult learners from primarily adult education programs and also community college programs will participate. Approximately 100 educators and 60 other stakeholders will provide feedback on the ASAP system.
ASAP will be a digital repository of assessment modules that users can electronically assemble and configure to serve specific assessment needs in adult education and workforce development. The assessment "unit" in the ASAP system is a "task module" that represents a cluster of literacy, numeracy, or integrative skills of interest to learners, educators, or employers. ASAP will leverage technology to capture evidence of student learning, using a universal design for learning architecture and combining features to make ASAP a personalized assessment system. ASAP will also leverage content and standards from adult education and competencies from the world of the workplace by linking assessment task to job skills in the O*NET system (which are linked to specific jobs and occupational clusters). ASAP assessment modules will be organized by NRS educational functioning levels and PIAAC achievement levels. ASAP will be available on virtually any digital device in any setting and delivered using relatively small assessment modules that can be taken in isolation or systematically pieced together to form larger assessments tailored to specific assessment goals. The system design will support all adult learners by providing additional opportunities to learn and acquire the skills needed for gainful employment and upskilling.
Research design and methods
For the needs assessment, the researchers will conduct a literature review in adult education assessment, survey key stakeholders (adult learners, teachers, career counselors, employers), and conduct focus groups. For the ASAP system, they will use principled design frameworks to ensure the validity, fairness, and reliability of the ASAP and address potential sources of construct-irrelevant variance that may emerge when assessments are administered to diverse adult learners. They will use socio-cognitive extended evidence-centered design to help ensure alignment between the assessment argument, measured skills, tasks designed, scoring, and reporting of evidence collected. In particular, they will be attending to possible bias for subgroup variation such as variation based on to sex, age, race/ethnicity/cultural heritage, native language, educational history, and learning disabilities. Over the course of the project, they will build four modes of ASAP: a manual mode for learner to choose a particular skill to work on, a placement mode and a diagnostic mode for educators or counselors, and a workplace mode for employers and similar stakeholders. After developing the full ASAP, they will conduct studies of its implementation to ensure it functions as intended and studies explore how ASAP links to national and international assessments in adult education to compare ASAP performance relative to other validated assessments in adult education.
Key measures
Key measures include the Massachusetts Adult Proficiency Test for Mathematics and Numeracy, the Massachusetts Adult Proficiency Test for Reading, the Program for International Student Assessment, and other reading and numeracy adult education achievement test data (e.g., Accuplacer, WorkKeys).
Data analytic strategy
The researchers will use item response theory for test development; differential item functioning and other psychometric analyses to evaluate fairness, reliability, and validity; regression analyses to explore construct-relevant and irrelevant factors affecting learners' performance; exploratory analyses of log (response process) data to evaluate motivation and cognitive processes; equi-percentile linking and alignment studies to evaluate performance relative to PIAAC.
People and institutions involved
IES program contact(s)
Project contributors
Products and publications
Products: Products will include the online repository of assessment task modules; the supporting online testing system modularized for different assessment purposes that serve the learner, educator/counselor, or employer; and assessment resources such as online practice tests, test administration guides, and score report interpretation guides. The team also will disseminate research results through presentations at practitioner and research-oriented conferences and by publishing research findings in peer reviewed journals.
Additional project information
This project will serve as a network research team for the CREATE Adult Skills Network, which conducts research and dissemination to support the millions of U.S. adults who have low foundational skills.
Kick-off Webinar available for viewing here
Supplemental information
Co-Principal Investigators: Oliveri, Maria Elena; Sabatini, John; Zenisky, April L.
Questions about this project?
To answer additional questions about this project or provide feedback, please contact the program officer.
