A relatively easy way to judge the effectiveness of any assessment is to look at the item format of the questionnaire. Items are the questions that make up any assessment. Item construction is an interesting combination of art and science. Occasionally, someone questions the effectiveness of “self-reporting instruments” in general. It is certainly possible to have a bad experience with some assessment products. That is not because it is  “self-reporting.” I had a bad meal in a restaurant once upon a time, but that was not a reason to indict restaurants as a group. Similarly, the effectiveness of assessment instruments depend upon several specific factors. Among these are the constructs chosen, the presentation, the user experience, and primarily, the item construction. When proper attention is paid to these factors, self-reporting instruments can produce accurate data. Over the many years that psychometricians have worked to refine the science in the pursuit of ever more accurate methodologies, item construction has evolved dramatically. The following is a simple history of that evolution, pointing out some of the qualitative issues that marked some of the more common types.

ADJECTIVE LISTS

One of the earliest assessments offered a list of adjectives. The candidate was asked to select the adjectives that they felt “described them.” Some instruments added a duplicate list of adjectives and asked to select the ones that their friends would use to “describe them.” This was a simple way to engage a participant in a counseling session. Accuracy was not the central objective. It was a starting point for a discussion, “Tell me why you chose these descriptions of yourself.”

Issues with Adjective Checklists

• Candidates tend to select those adjectives that are most flattering or that seem most supportive of whatever job is being considered. Example: Sales candidates rarely choose timid or shy, choosing instead persuasive or direct.
• Adjective checklists do not actually measure traits of people. Instead, they use the person’s choices to sort them into a simple model of personality styles.

Product Examples :

• Predictive Index
• The Culture Index
• Omnia

FORCED CHOICE ITEMS

Recognizing the problems that were inherent in adjective checklists, test developers introduced forced choice items. The most common of these offer a choice of three or four words or phrases. The candidate then selects the one that MOST describes them and the one that LEAST describes them. This kind of item is found in DISC-type instruments. These word groups were first put together by William Marston in 1928. While the words or phrases have changed somewhat over the last ninety years, the fundamental forced choice format has not.

Issues with Forced Choice Items

• Forced choice items produce ipsative scores. Ipsa is from the Greek word for self. It means that forced choice items produce scores that are only meaningful for that one person. They cannot be compared to anyone else’s scores, even if they have taken the same assessment and even using the same model of results. The reason is that there is no way of knowing how big the gap is between the MOST and the LEAST choices. When two options are offered, again there is no way of knowing how the participant viewed those choices. Two candidates could make the same choice, but for one candidate, it was a difficult decision with both choices equally appealing. The other candidate answered easily, having no interest at all in the one not selected. It is relative to each individual’s particular situation and personality. One way of understanding this is to imagine asking several individuals if it takes a long time for them to go to where they work. The Wall Street banker who lives in Westchester replies, “No, I can be in Manhattan in an hour.” The executive who lives in the Atlanta suburbs replies, “No, I can be downtown on Peachtree Street in only thirty minutes.” The attorney who lives in Rome, Georgia, replies, “No, I can be in my office in ten or fifteen minutes.” The fourth person asked replies, “No, I just go downstairs to my office.” Each person answered the same “No” but the extent of the difference was dramatic. The same is true with forced choice assessments. You cannot compare two reports produced by forced choice items. You cannot meaningfully compare two individuals using forced choice items.
• Forced choice data cannot be used to create norms or profiles  for jobs. Since the scores are only meaningful for each individual who answered the questionnaire, there is no statistical metric common to other participants that can be used to establish a relationship.
• Many of the products using forced choice items rely on a phenomenon known as the P.T. Barnum Effect, named after that famous showman whose exaggerated claims fooled millions. To achieve that Effect, the report text is written in a generalized form that appears to be accurate for virtually anyone.

Examples of Products Using Forced Choice Items

• Caliper
• Performax
• DISC
• OPQ (ipsative version)
• Myers-Briggs (MBTI) uses a mix of item types

TRUE – FALSE ITEMS

True – False items present a statement describing a behavior or a belief  which the candidate is asked to confirm as being either True or False for them. True – False items are usually quick and easy to answer. Unfortunately they share the same issue that cripples forced choice items.

Issues with True – False Items

• It is impossible to know the extent of the gap between True and False. For example, the candidate responds “True” to the statement, “I follow the rules.” Do they follow all rules, some rules, one of the rules? The limited choice of answers blurs the actual meaning.

Product examples :

• Birkman
• Hogan Personality Inventory

LIKERT ITEMS

Items using Likert scales present a statement describing a behavior or a belief. The candidate is given a range of choices with which to rate the level of their agreement or disagreement with the statement. Example: Strongly Agree, Somewhat Agree, Neither Agree nor Disagree, Somewhat Disagree, or Strongly Disagree. Likert items commonly have 5 – 10 point ranges. The advantage of this format is that it is possible to assess the degree of agreement or disagreement because of the candidate’s increased choices. The more accurate instruments use this format, and so do almost all the serious business-oriented instruments developed in recent years.

Issues with Likert Scale Items

• The issues that occur with Likert scale items do not relate to the format as much as with the actual wording of the statements. Each statement must include only one behavior or belief. For example, this is an item from a product using Likert scale items but which contain errors in the construction of some of the items: “I like to work slowly and carefully.” One could work quickly but carefully. One could work slowly but carelessly. Each word has a slightly different connotation. Therefore, any response to the item is unclear as to which one or maybe both factors influenced the answer.
• The most common Likert scale items have a 5-point range for responses. Some have 6-point or 7-point ranges.  When the ranges are greater than 7, the data becomes less meaningful because the standard deviation becomes quite large.

Product examples :

• The Prevue
• BestWork
• Profile XT
• Try on Jobs
• Berke
• OPQ (Likert version)