Validation Shell Game
Validation is the most misunderstood part of assessment shopping. I have taken or reviewed hundreds of assessments. I have yet to find one that was not validated.
PROBLEM #1 – THE TERMINOLOGY
There is a misconception about the term validation. Some of the commonly understood definitions of validation are:
PROBLEM #2 – THROW THE BOOK AT THEM
Once upon a time, there were many seminars on selecting assessment tools. One of the key recommendations at these seminars was to ask for the technical manual or the validation manual. This is supposed to be a description of how the assessment instrument was developed. The manual normally explains the construct of the scales, the choice of item formats, the development of the items, the demographics of the sample population, and the statistical analyses of the data resulting from the study of that population.
In a novel, this is where the plot thickens. Few people can really understand the details in a typical technical manual. The psychometric terminology is foreign. The statistics are analyzed with specialized software that expresses the results in esoteric language with numerical data puzzling to a layperson. This works in favor of some of the more questionable assessment companies. They present “validation manuals” filled with statistics and bold statements about the instrument, knowing that it will be accepted as their bona fides by most buyers.
SOME STANDOUTS ARE:
PROBLEM #3 – VALID DOES NOT MEAN IT WILL WORK FOR YOU
The message of this site is for you to be a careful buyer. Just because an assessment is valid does not mean that it will do what you expect it to do or what the company promises it will do. “Valid” is not a Good Housekeeping Seal of Approval.
PROBLEM #1 – THE TERMINOLOGY
There is a misconception about the term validation. Some of the commonly understood definitions of validation are:
- determining whether a statement is true or false
- confirming that a product or service meets a set of specifications or quality standards
- giving legal force to something
- giving official sanction or approval
PROBLEM #2 – THROW THE BOOK AT THEM
Once upon a time, there were many seminars on selecting assessment tools. One of the key recommendations at these seminars was to ask for the technical manual or the validation manual. This is supposed to be a description of how the assessment instrument was developed. The manual normally explains the construct of the scales, the choice of item formats, the development of the items, the demographics of the sample population, and the statistical analyses of the data resulting from the study of that population.
In a novel, this is where the plot thickens. Few people can really understand the details in a typical technical manual. The psychometric terminology is foreign. The statistics are analyzed with specialized software that expresses the results in esoteric language with numerical data puzzling to a layperson. This works in favor of some of the more questionable assessment companies. They present “validation manuals” filled with statistics and bold statements about the instrument, knowing that it will be accepted as their bona fides by most buyers.
SOME STANDOUTS ARE:
- An actual hardback book filled with questionable statistics purporting to support the quality of an equally questionable instrument
- A manual in which the development of the instrument was attributed to “an employee of the Cincinnati Public School System”
- An integrity instrument validated using a prison population
PROBLEM #3 – VALID DOES NOT MEAN IT WILL WORK FOR YOU
The message of this site is for you to be a careful buyer. Just because an assessment is valid does not mean that it will do what you expect it to do or what the company promises it will do. “Valid” is not a Good Housekeeping Seal of Approval.
