Frequently Asked Questions About Licensing Exams
Job analysis and test blueprints/ factor analysis
CLEAR Exam Review
Norman R. Hertz
Question: We know that an occupational analysis should be performed to identify the subject matter that should be included in our examinations. The results of an occupational analysis often produce a multitude of statistics and data. We would like to maintain the same scientific rigor in constructing the examination plan as was used in the occupational analysis. What data should be used in constructing the examination plan?
Answer: In a sentence, it is best to consider all of the available data in constructing the examination plan. Once the examination plan is constructed, however, subject matter experts should review it to confirm the results from the data analysis and modify it as needed.
Occupational analysis usually takes one of two forms, and the form of the analysis influences how the data are analyzed and interpreted. One form consists only of data from judgments about the importance and/or frequency of performing tasks. The other form also collects data about tasks, but, in addition, collects data about the importance of knowledge, skills, and abilities (KSAs) and/or the difficulty of acquiring them.
When the occupational analysis is task based, the empirical data analysis is task based, the empirical data analysis provides evidence for the structure--subject matter areas--of the examination plan and the relative importance of each subject matter area. If more than one rating is obtained for each task, the mean ratings are usually multiplied to derive an overall critical value (Kane, Kingsbury, Colton, & Estes, 1989). Once the critical value has been computed, the subject matter areas can be established with a statistical procedure known as factor analysis. Factor analysis groups related tasks into subject matter areas, and the relative weights of the subject matter areas are calculated by computing a composite of the critical values for all the tasks in each subject matter area. Before this weighting, tasks with small critical values should be eliminated from the subject matter areas because examinations should be based only on tasks critical to effective performance.
When an occupational analysis includes both tasks and KSAs, the procedure for determining the subject matter areas and computing the relative weights is similar. Including KSAs requires an additional step, linking the KSAs to the tasks. The advantage of including the KSAs is that item writers will be able to develop test items directly from the KSAs instead of extrapolating them from tasks. However, subject matter experts should link the KSAs to the tasks before the test items are written. Factor analysis is useful for verifying the linkages of the KSAs with the tasks (Hughes & Prien, 1989).
The final results of an empirical approach are a structure (an organization) for the examination, area weights, and the linkage between tasks and KSAs; together these constitute an examination plan. The plan can then be independently confirmed by subject matter experts. By combining empirical and judgmental approaches, one can produce examination plans that use all the data available. A note of caution is in order; factor analysis requires a relatively large number of responses, which may not be available for some occupations.
© 2002 Council on Licensure, Enforcement and Regulation