SmartItem™
The Ultimate Guide
This is your definitive guide to SmartItem technology in 2024.
The SmartItem is a revolutionary concept in item and test design.
Without hyperbole, it can prevent all test theft and almost all forms of cheating.
In this complete guide, you’ll learn:
- What a SmartItem is
- The benefits of SmartItem technology
- How the SmartItem works
- The difference between AIG and SmartItem tech
- Case studies of programs already using SmartItems
- And lots more
So if you’re curious about all things (or any thing) SmartItem, you’ll love this guide.
Let’s get started!
Contents
What's a SmartItem?
In this chapter, you’ll get a handle on the fundamentals:
What a SmartItem is, why it’s important, and the industries and types of exams that can benefit from SmartItems.
Let’s learn!
What Is a SmartItem?
A SmartItem is a revolutionary concept in item and test design that uses special technology during exam development and delivery so that the item renders differently each time it is given or viewed on a test.
- Translation: no two examinees see the same thing during a test.
The SmartItem isn’t a new item type, it’s an item treatment—a way to enhance item formats and make them more efficient, have greater generalizability, and best of all, protect themselves from item piracy and cheating. The SmartItem is the future of exams. (More on that coming up, but in the meantime, here’s a helpful visual!)
Why Is SmartItem Technology The Future of Exams?
Good question!
Most importantly, the Smartitem can protect the security of your exams by preventing all test theft and almost all forms of cheating and fraud.
And when your exam is secure, you (and your stakeholders, your examinees, and all the people who rely on the results of your test to make important decisions) can be confident that the scores your exam produces are trustworthy and reliable.
And there are a ton of other important benefits of Smartitems. We’ll go over these in more detail in Chapter 2, but for now, here are a few things SmartItem technology is proven to provide:
In a post-SmartItem world, you’ll experience:
Increased:
Decreased:
Score validity
Accessibility
Convenience
Media & stakeholder respect
Prosperity for educators
Opportunities for honest, deserving people
Fraud, cheating, and deceit
Expenses
Overhead
Mistrust in assessments
Opportunities for dishonest, undeserving people
Who Can Use SmartItem Technology?
The SmartItem is helpful for all.
Here are just a few groups who can benefit from it, along with some example SmartItems within each industry:
- Certification and Licensure
- K-12 Testing
- International Organizations
- Higher Education Programs
1. Certification and Licensure
The reputation of your exams and the scores they produce matter for all exams, especially for certification and licensure programs.
SmartItems ensure more reliable program decision-making by making exams impervious to cheating and testwiseness.
And if the content of your current exams must be or is being constantly updated, SmartItem tech enables you to distill your content so that one item measures the entire breadth and depth of the knowledge or skill you are working to assess.
Then, with fewer items for you to build and manage, you provide your SMEs with an easier and shortened test development process and lower costs for you!
To see certification examples of SmartItems, please visit these links:
Example 1: Maintenance of Medical Certification
- Link: examples.caveon.com/medical
- Objective: Read and understand details of published scientific research.
Example 2: IT Certification: Docker
- Link: examples.caveon.com/docker
- Objective: Use the docker run command to start containers according to various requirements.
Example 3: IT Certification: Oracle
- Link: examples.caveon.com/oracle
- Objective: Define the scopes of variables.
2. K-12 Testing
Reduce item maintenance, replacement, and incident management costs (including investigation, litigation, and public relations costs) by immediately stopping cheating and testwiseness in their tracks.
In addition to stopping cheating and testwiseness, SmartItem tech also ensures that test preparation is an honest process—both for students and for teachers. No more teaching to the test!
Moreover, students won’t benefit from viewing the exam questions and answers ahead of time, because each exam is completely unique to each student, and still each student’s exam covers an entire skill!
To see K-12 examples of SmartItems, please visit these links:
1. Common Core ELA
- Link: examples.caveon.com/common-core-ela
- Objective: Demonstrate command of the conventions of standard English capitalization, punctuation, and spelling when writing.
2. Common Core Math
- Link: examples.caveon.com/common-core-math
- Objective: Determine the unknown whole number in a multiplication or division equation relating three whole numbers.
3. Question on the Periodic Table
- Link: examples.caveon.com/periodic-table
- Objective: Use the periodic table as a model to predict the relative properties of elements based on the patterns of electrons in the outermost energy level of atoms.
“The Smartitem can protect the security of your exams by preventing all test theft and almost all forms of cheating and fraud.”
3. International Organizations
Exams delivered internationally are more at risk for content theft than almost any other exam.
When international tests use SmartItems however, it doesn’t matter if somebody steals one of your items, or even your entire exam!
Every SmartItem is different and has the potential to render in millions of ways, so you can remove the time and cost of item redevelopment completely.
To see example SmartItems, please visit these links:
1. Weather Map Interpretation
- Link: examples.caveon.com/weather-map
- Objective: Correctly interpret information provided on a weather map. Bones Identification: examples.caveon.com/bones Objective: Identify bones of the body.
2. National Anthems
- Link: examples.caveon.com/national-anthems
- Objective: Listen and identify national anthems from countries around the world.
4. Higher Education Programs
SmartItem technology creates item pools that are secure enough to be confidently used in online learning environments, and expansive enough to be relied upon forever.
Save costs in the short- and long-run with immortal exams, where you need only update your exam content if your testing objective changes. Wouldn’t that be nice?
To see higher education examples of SmartItems, you can check out these links:
1. Research Methods
- Link: examples.caveon.com/research-methods
- Objective: The student will identify all main points of an introductory psychology chapter on Research Methods.
2. U.S. Constitution and Bill of Rights
- Link: examples.caveon.com/us-constitution
- Objective: Explain how civil liberties and civil rights are protected in the U.S. Constitution and Bill of Rights.
To conclude chapter 1, the SmartItem is an item treatment that uses special technology during exam development and delivery so that no two examinees see the same content during an exam.
It makes test questions more efficient, have greater generalizability, and last indefinitely (so long as the learning objective or skill doesn’t change). Best of all, SmartItems increase fairness and validity by protecting themselves from item piracy and cheating.
Any and every organization and industry can utilize SmartItem technology effectively. From K-12 programs to higher education programs, certification and licensure programs to international organizations, the SmartItem is for you!
Up next, we’ll learn more about the benefits of utilizing SmartItem technology.
Chapter 2 will go into depth about the 21st-century testing challenges that are solved by SmartItems and the many other benefits that follow.
Here we go!
The Benefits of the SmartItem (And the Problems It Solves)
In the first chapter, we went over what a SmartItem is, why it’s important, and the different industries and types of assessments SmartItems can serve.
In this chapter, we’ll explore the overall benefits of the SmartItem and the many problems it solves.
(Then, in Chapters 3-7, we’ll discuss how it works, the difference between SmartItems / traditional items / AIG, the evidence and case studies that support the use of SmartItems, and how you can get started using SmartItem technology.)
Onward!
The Short- and Long-term Benefits of SmartItems
How SmartItems Benefit YOUR Program
SHORT-TERM
LONG-TERM
- Fewer items to build and manage
- Easier, more interesting development process for SMEs
- Immediate reduction in risk from security threats
- Reduced security management costs
- Reduced incident management costs (investigation, litigation, public relations, etc.)
- Strengthened reputation for test security
- Reduced effect of testwiseness when DOMC™ is used
- Reduced item maintenance and replacement costs
- Improved test preparation process (test takers must gain knowledge and experience)
- More reliable program decision making (not affected by cheating and testwiseness)
- Tests that better fulfill your company’s mission
Testing Challenges Solved by SmartItems
To really get a handle on the benefits of the SmartItem, we must first know what exams today are up against.
The testing industry currently faces several serious challenges, and the SmartItem was created to decrease (and sometimes completely end) most of these problems:
Trends in Test Security Threats
THREATS MADE MORE DANGEROUS BY NEW TECHNOLOGY
On the rise...
Less to worry about today...
- Cheating and using pre-knowledge
- Theft of test content using hidden cameras
- Proxy test taking using false identification
- Proxy test taking using remote access
- Stealing test files whether stored or in transit
- Copying
- Using cheat sheets
- Memorizing test questions for brain dump sites
Here’s how the SmartItem addresses these challenges:
- No More Theft
- Significantly Less Cheating
- No More Testwiseness
- Increased Fairness
- Increased Cost Savings
- Deeper Learning and Preparation
- More Convenience
1. No More Theft
The SmartItem removes all test theft.
If your test has SmartItems, an examinee could steal and share your entire test, but it won’t matter. SmartItems make stealing test questions and sharing them with others a worthless exercise.
Here’s how: the SmartItem can render in millions of ways. With so many renderings, examinees can no longer cheat by sharing test content, by buying questions and answers, or by recording an entire test and sharing it with a friend. There is no guarantee that what they steal will be seen on future tests. So, why bother to record it and share/sell test questions online? It won’t do any good!
With theft made useless by SmartItems, cheating through pre-knowledge becomes obsolete as well. (More on that below.)
So if your exam development team is frustrated by working long hours to create high-quality items, only to find exam content online at a braindump a few weeks after launch (even though you’re already monitoring the web and sending DMCA takedown notices to the braindump sites), the SmartItem does the hard work by making content theft obsolete.
2. No More Cheating with Pre-Knowledge
Rather than just detecting cheating and theft after they occur, the SmartItem prevents cheating and theft in the first place—before any of it can happen!
So while one of the SmartItem renderings on your exam might be captured and shared online, it provides no useful information to other test takers, who will see different renderings of the same SmartItem. Pre-knowledge becomes pointless!
Most recently, when the major tech organization, SailPoint, converted their test questions to versions of the SmartItem, pass rates in India dropped from 88% to 8%.
This indicates that almost everyone in that region was using braindump material and cheating using pre-knowledge on the exam! When they began using SmartItems, it effectively stopped nefarious efforts dead in its tracks.
With that said, there are two types of cheating the SmartItem does not impact:
- Proxy test taking
- Having a third party look over your shoulder and divulge the answers.
Fortunately though, both of these methods of cheating can be caught by other methods or by systems that ensure a test taker’s identity.
After two case studies, three scientific experiments, and one data simulation, the evidence shows that SmartItems not only work, but they improve the validity, reliability, and fairness of exams more than traditional items.”
3. No More Testwiseness
Testwiseness is the ability some people have to use cues inherent in multiple-choice questions to “game the test” and get a higher score.
Because testwiseness affects test scores, it results in unfairness to individuals who don’t or can’t use those skills.
Essentially, SmartItems provide a fairer system for those who can’t access specialized training or who don’t want to cheat using stolen test content.
(To combat against testwiseness on an even deeper level, SmartItem technology can be applied to the Discrete Option Multiple Choice™ [DOMC] item format to reduce the effects of testwiseness on your scores by presenting items and their options in a binary way and preventing the use of extraneous skills to answer the question. Learn more in this section.)
4. Increased Fairness
SmartItems actually enhance the fairness of your exams by eliminating the two largest sources of unfairness: testwiseness and cheating.
- Testwiseness—the ability to strategically take tests. Success on an exam should depend on the depth of an examinee’s knowledge, not their ability to take exams.
- Cheating—With so many renderings, it’s not possible for examinees to cheat by sharing test content, buying questions and answers, or asking a friend to take the test and tell them what they saw. All test theft and almost all forms of cheating outlined in this white paper are completely resolved with SmartItem technology!
Moreover, SmartItems can be constructed to modify themselves depending on the characteristics of the test taker. (Cool, right?)
What that means is, they can provide a choice of language for non-English speakers, text and audio options for those who have impaired hearing, item-based timing to assist pacing, and more. On top of the fairness qualities this quality adds, these on-the-fly customizations can calibrate each assessment to examinee circumstances and get a truer measure of their abilities.
5. Increased Cost Savings
Tests today are expensive to create, maintain, and administer.
On top of that, any costs of redevelopment after a security breach or content theft can be financially draining for any testing program!
Utilizing SmartItem technology ensures a longer lifespan for your items and tests, eliminating continual authoring and associated expenses, as well as the need to spend money on a multitude of other detection tools.
Traditional item creation and re-creation can cost upwards of $2,500 for large testing programs to re-create a single item.
But unlike traditional items, a SmartItem typically only has to be created once, and then it becomes virtually immortal.
It doesn’t ever need to be revised or maintained throughout its lifespan, unless your core objective changes for the exam.
All of this means your testing program saves substantial costs from start to finish.
6. Deeper Learning and Preparation
The only way an examinee can prepare for an exam composed of SmartItems is to become well versed and competent across the entire set of skills the test measures. No more “teaching to the test.”
While this is a different way than we’re used to learning and preparing to take tests, the result is deeper and better teaching, training, and learning.
7. More Convenience
Because many of the security threats are neutralized, tests containing SmartItems can be given in circumstances that would previously have been considered “risky.”
For example, almost the entire reason for going to a testing center, or to a gymnasium on a Saturday morning, is for security reasons.
However, with SmartItem technology, none of that is required, because a SmartItem-based test can be securely given in a home and monitored by online proctors, who act more as administrators than a security measure.
As long as the proctors (online or on-site) can authenticate the test taker and prevent coaching and proxy testing, the degree of risk associated with other threats is greatly curtailed, and it is quite possible that the test will have equal or better security than in its in-person traditional item counterpart.
As you can see, the SmartItem has truly transformed the item creation and replacement process. The security of exams is protected while simultaneously limiting the time and cost associated with exam development and redevelopment.
“SmartItems are invulnerable to theft and cheating, relieving SailPoint of the same security concerns that have plagued every other IT certification program for the past three decades.”
Arthur Altman
Program Manager, Global Education Services
SailPoint Technologies Holdings
To end Chapter 2, the SmartItem is a simple but significant innovation that yields un-stealable tests—tests that last virtually forever, meaning you don’t have to update exam content unless your core objectives change.
The SmartItem has seven primary benefits. They are:
- No More Theft
- Significantly Less Cheating
- No More Testwiseness
- Increased Fairness
- Increased Cost Savings
- Deeper Learning and Preparation
- More Convenience
Now that you know the security benefits of SmartItem Technology, you might be thinking:
“That’s nice and all, but how is this possible?”
Well, that’s what our next chapter is all about!
Up next in Chapter 3, we’ll discuss exactly how SmartItem technology works.
Then, in Chapters 4-5, we’ll discuss the difference between SmartItems vs. traditional items and SmartItems vs. AIG. Then, Chapters 6-7 will provide evidence that supports the use of SmartItems, case studies for programs already using SmartItems, and how to get started implementing them for your test program.
Let’s go!
How It Works
In the last chapter, we went over the problems SmartItem technology solves, along with its overall benefits.
In this chapter, we’re going to show you the “behind the scenes” of the SmartItem.
How do they work??
After, in Chapter 4, we’ll check out the difference between AIG and SmartItem technology.
Here is a step-by-step guide for what it looks like to build a SmartItem:
First, Determine the Skill
Since the SmartItem covers the entire breadth and depth of a skill, creating a SmartItem starts with just that: the skill.
(Important to note: a SmartItem focuses on the skill more than a traditional item.)
The image below shows a development screen for a selected-response item for the Common Core State Standard CCSS.MATH.CONTENT.3.OA.A.4. This standard reads:
Determine the unknown whole number in a multiplication or division equation relating three whole numbers. For example, determine the unknown number that makes the equation true in each of the equations 8 × ? = 48, 5 = _ ÷ 3, 6 × 6 = ?
Notice how:
- The same standard covers both multiplication and division.
- The unknown number could be in different places in the equation.
- With that in mind, remember that with the unknown number being presented in different areas of this question, the SmartItem should likely present versions of the item where the unknown number changes the position before and after the equal sign.
Each of these points should be considered when creating the SmartItem for this standard.
After you truly understand the skill being tested, you can design a question that can cover it entirely.
Second, Create Item Variations
Now that you have a feeling for the skill description behind this item, let’s look behind the scenes of this particular SmartItem.
You can create a SmartItem using:
- A graphical user interface (GUI)
- Code, or
- A plethora of response options.
Referring to the Common Core Standard above, let’s beef up our item with various content to cover the entire domain:
1. Create the Item
As discussed above, there are a few different ways to create a SmartItem—you can use code, a Graphical User Interface (GUI), or you can write a plethora of items. (We’ll learn more about this in Chapter 7.)
The below graphic displays the item authoring tool, Scorpion, which supports creating SmartItems.
While coding necessitates a special skillset, using a GUI and/or extensive options to create a SmartItem can be done by any item writer.
Returning to our third-grade common core question, this SmartItem was created using 17 lines of code (seen in the image below), but it could have also been created using Scorpion’s GUI interface.
2. Use Response Options
This same third-grade item can then be enhanced by using a different response format, in this case the Discrete Option Multiple Choice™ (DOMC). The image below shows a sample DOMC item with a single option showing:
Using DOMC improves this item by enhancing security and improving fairness through decreasing the impact of test taking skills.
In addition, using DOMC increases the total potential pool of questions rendered by this SmartItem to an astonishing 1,317,226 item variations!
(If you want to run this item a few times to see how variations are produced, click here—you’ll see one SmartItem repeated five times. Be sure to pay attention to the changing numbers, position of the equal sign, and location of the unknown number.)
3. Input Content
Remember, in order to cover an entire objective, a SmartItem will need to include the appropriate amount of content.
For example, if the objective requires a student to differentiate between mammals and non-mammals, the SmartItem would incorporate the names and characteristics (live birth, hair, etc.) of hundreds of common mammals and perhaps even a greater number of obvious non-mammals to adequately cover that particular content domain.
As another example, if the objective requires the student to identify the amendments to the U. S. Constitution that protected civil liberties and civil rights, then the SmartItem would include all the relevant amendments.
Regardless of whether a large or a small amount of content is assumed in the objective, it is important that the SmartItem covers all of it as part of the item design.
“[With] Smartitems, an examinee could steal and share your entire test, but it won’t matter. SmartItems make stealing test questions and sharing them with others a worthless exercise.”
In summary, a SmartItem begins with creating a skill around the objective.
Next, a SmartItem creates item variations using either a Graphical user interface, code, or many response options.
Finally, the SmartItem is filled with content to cover the entire domain.
In the end, you have a SmartItem that covers an entire objective and can render in millions of ways!
Now that you know how SmartItem technology works behind the scenes, let’s zoom in a bit more to understand the difference between SmartItems and traditional items.
Then, in Chapter 5, we’ll see what the difference is between AIG and SmartItem tech.
Chapter 6 will then look into the evidence that supports the use of SmartItems on exams, and Chapter 7 will teach us how to get started using SmartItem technology on exams.
Buckle up!
The Difference Between Traditional Items and SmartItems
In many ways, SmartItem development is very similar to traditional item development. The items and exams look the same when presented to examinees.
But there are some key differences that set them apart.
Let’s discuss what they are…
We’ll start with a step-by-step guide for what it looks like to build a SmartItem…
Although they may look the same as traditional items when presented to examinees, there are seven main differences that set a SmartItem apart.
1. A SmartItem can cover the entire skill as described in a competency statement, learning or assessment objective, or educational standard.
Take this objective as an example:
The student can add or subtract 2-digit numbers.
The SmartItem for this objective will be built to use all 2-digit numbers from 10 to 99, and both operations of addition and subtraction. One item covers the entire skill set.
2. The SmartItem will present a different version of the item each time it is given to a test taker.
By definition, each item version is congruent with the objective standard or competency. You can learn more about this in our last chapter.
3. A SmartItem renders on-the-fly as part of the item presentation to the test taker.
Just prior to the test taker seeing the item, a version of the item is rendered. These were not created in advance and reviewed; they were rendered on-the-fly, and they would be one of hundreds, perhaps thousands, (or even millions!) of versions that would come from one single SmartItem.
4. The SmartItem cannot be, and does not need to be field tested prior to its use.
It just isn’t necessary. It is also technically impossible.
5. The SmartItem leans more heavily on design than on writing.
In traditional exam development, objectives are created and are weighted to determine the number of items needed.
But unlike traditional items, the SmartItem leans more heavily on design than on writing, because with the right design, each SmartItem can allow just a single item to cover the depth and breadth of the whole skill.
6. One SmartItem is created per each objective.
Item writers produce one SmartItem per each objective, and they ensure that the question covers the objective completely.
Then, they enter the content into the computer system by creating stems and options with variables.
These stems, options, and variables are then reviewed, and a quality assurance step is added to ensure the SmartItem is functioning properly once in its final state.
7. The SmartItem is immune to many threats that make exam content vulnerable to theft and cheating.
Studies in some testing programs have shown that entire exams are often leaked to braindump sites as quickly as three weeks after publication!
The SmartItem, by design, is immune to many threats that make content vulnerable to theft and cheating. Therefore, by using SmartItems, exam programs will not need to replace exams as often, if ever.
An Example
The true difference between traditional items and SmartItems is best laid out with an example.
Let’s start with how a traditional item is made:
Traditional Items
In traditional exam development, the writer creates a select number of items per the learning objective as stipulated by the blueprint.
The writer chooses which pieces of the skill to measure—in this instance, which planets to test on. The final product might look something like these two items:
SmartItems
In contrast, a SmartItem is created within a computerized tool to encompass the entire objective, not just a piece of it. Here’s what the scaffolding of the SmartItem for this objective could look like:
The computer will select from this list, at random, upon delivery.
One candidate may see “Is this the first planet from the sun?” and another may see “Is this the third planet from the sun?” and so on.
The correct option is dependent upon which list item is presented to the candidate. If “third” appears in the stem, “Earth” is the correct option.
For example, the incorrect options will be any planet in the solar system that is not the correct option and are also presented at random. The examinee will see:
And there you have it! Piece of cake!
To conclude this chapter, there are seven main differences that set SmartItems apart from traditional items:
- A SmartItem can cover an entire skill
- The SmartItem will present a different version of the item each time it is given to a test taker
- A SmartItem renders on-the-fly
- The SmartItem cannot be, and does not need to be, filed tested prior to its use
- The SmartItem leans more heavily on design than on writing
- One SmartItem is created per objective
- The SmartItem is immune to many threats that make exam content vulnerable to theft and cheating
Up next, let’s zoom in further to find out what the difference is between AIG and SmartItem tech.
Then in Chapters 6-7, we’ll look into the evidence and case studies that support the use of SmartItems and how to get started using this exciting technology.
Here we go!
The Difference Between SmartItem Tech and AIG
In the last chapter, we went over the seven main differences between SmartItems and traditional items.
But another question that we frequently get asked is “How is SmartItem tech different from AIG?”
They both improve test security, look the same to test takers, and are relatively easy to create and implement. So what sets them apart?
That’s what this chapter is all about!
What Is the Difference Between AIG and SmartItem Tech?
Automated Item Generation (AIG) creates item versions with the purpose of expanding an item bank with static items that will then be reviewed and perhaps field tested.
How AIG Is Different From SmartItem Tech
- AIG generates static items, SmartItem tech renders dynamic items.
- AIG is a process, the SmartItem is a product.
- AIG is many items, a SmartItem is one item.
- AIG needs every item field tested and reviewed, the SmartItem does not.
- AIG items need to be replaced when exposed, a SmartItem does not.
- AIG requires test forms, SmartItem tech does not.
- AIG supports current testing methods, SmartItem tech looks toward the future.
Check out this table to learn about some other frequently asked questions:
The Differences Between SmartItem and AIG
SmartItem
AIG
Number of Items
SmartItem technology reduces the size of an item bank to, ideally, one item per objective.
AIG expands an item bank by creating hundreds of different items (or more) automatically.
Goals
A SmartItem prioritizes security, fairness, cost savings, and convenience.
AIG expands item capacity, primarily because of security reasons.
Transitory vs. Statics
Transitory item versions are unlikely to be seen again. Test-taker data is captured.
Item versions created by AIG are static, saved, and formalized items.
Field Testing & Review
Item versions do not need field testing or review beyond each quality-assurance process.
Item versions are usually reviewed and/or field tested during the development process.
Unused Items
Item versions are created by SmartItems on an as-needed basis. There is no waste.
Many item versions created by AIG may not ever be used because there isn’t a need.
Security
With numerous item versions, a SmartItem enhances security by making theft irrelevant and most forms of cheating impossible.
AIG enhances security with a large number of replacement items to use after original items are compromised.
Forms
SmartItem technology eliminates the need for forms and many other features of traditional test design.
AIG creates items that fit within existing test designs, such as the use of equivalent or equated forms.
Random Error Variance
Item variations are not assumed to have equivalent statistical properties, increasing the random error variance to remove much larger systemic error variance due to cheating.
AIG attempts to reduce error variance by relying on item models that are pre-calibrated to produce item variations where the calibrations can apply.
With the above differences in mind, there are a few key characteristics to remember about AIG:
Key Characteristics of AIG
- AIG is a test development process, not an item treatment.
- The goal of AIG is to produce hundreds or thousands of static, traditional items.
- AIG uses item models and templates during the test development process to generate static items. These models and templates are not part of the operational tests.
- AIG supports traditional methods of testing, usually placing static items on test forms.
- AIG can use SmartItem technology as a step in the process.
To conclude Chapter 5, AIG creates item versions but with the purpose of expanding an item bank with static items that will then be reviewed and field tested; the SmartItem is an item treatment that renders dynamic items.
Up next, let’s go over the evidence that supports the use of SmartItem technology. Then, in Chapter 7, we’ll learn how to get started implementing SmartItems.
Evidence Supporting SmartItem Technology
By now you know that the SmartItem has a lot to offer…
It increases your item writers’ efficiency, it limits item exposure, it enhances the overall security of your exam, and more.
And while those benefits are quite a big deal, it leaves many wondering… “Does the SmartItem actually work?”
Well, research shows it does.
In this chapter, we’ll talk about the evidence supporting SmartItem technology.
Let’s jump in with some frequently asked questions…
1. Does the SmartItem Work?
After two case studies, three scientific experiments, and one data simulation, the evidence shows that SmartItem’s not only work, but they improve the validity, reliability, and fairness of exams more than traditional items.
Validity
Our research shows that the SmartItem supports and provides evidence for the valid use of test scores.
Reliability
Research indicates that tests comprised entirely of SmartItems have high reliability coefficients.
Fairness
Research shows that by making cheating more difficult and eliminating testwiseness, SmartItems contribute to the fairness of exams more than traditional items.
Now back to the original question on whether SmartItem technology works…
We’ve found it’s helpful to look at the question from another angle.
So here are some ways of re-stating our original question:
1. Do SmartItems contribute to reliable exams?
- Yes! SmartItems perform well psychometrically. Statistical analyses indicate that SmartItems perform in the range where psychometricians would support their use.
2. Do SmartItems contribute to making valid interpretations from exam scores?
- Yes! Tests made up of SmartItems demonstrate high levels of reliability and validity.
3. Do SmartItems actually prevent the theft of test content?
- Yes! All forms of test theft. See point #7 in Chapter 4!
4. Do SmartItems actually prevent cheating?
- Yes, almost all forms of cheating!
5. How easy are SmartItems to use?
- Very easy! Subject Matter Experts (SMEs) report being capable of (and even enjoying) the task of designing and building effective SmartItems.
6. Do you have any proof of these claims about SmartItems?
- Sure do! Read on…
We have designed, run, and collected data from research projects that “prove” SmartItem technology works as promised.
The results are very encouraging!
And while more research is always welcomed, there is more than enough empirical evidence today to encourage the serious use of SmartItem technology.
These results are surprising and almost unbelievable to most testing professionals. And we get it… It’s hard to believe that SmartItems, varying as they do, can be described using common item analysis statistics. And it is even harder to believe they can contribute to test reliability and validity. But they can, and they do.
(You can go to the below pages to read in-depth, or keep reading here for summaries and highlights.)
The multiple-choice item’s impact has been profound. But after 100+ years, it’s time to replace it for more secure items that meet the needs of our day. In this white paper, you’ll learn about the history, evolution, and future of the multiple-choice item.
In his new e-book, Dr. David Foster—founder and creator of SmartItem technology—discusses the “how” behind this incredible innovation and the science that makes it all possible.
2. Is the SmartItem Fair?
While it is true that the SmartItem is designed to vary between test takers, SmartItems are in fact fair for individuals whose test scores are being compared for two reasons: it relies on randomization and it eliminates systematic unfairness.
1. The SmartItem Relies on Randomization
The error introduced to a test by the randomization in SmartItems is random error. The principles of science and measurement tell us that random error balances out with more items. Therefore, test scores from sufficiently long tests are comparable.
Test theory also supports the use of SmartItems since they render randomly within the constraints of each skill being measured by the test. This is the single most important property of SmartItems, and it is supported by both classical and modern test theory. (Read more about Randomly Parallel Tests and Test Theory in our white paper.)
2. The SmartItem Eliminates Systematic Unfairness
SmartItems eliminate large sources of confounding or bias error from cheating and testwiseness, two of the largest sources of systematic unfairness in exams.
3. What About Item Variability, Difficulty, and Randomization?
High-stakes exam questions have always been static, fixed, and immutable, with exams that are always identical.
After a moment of contemplation, testing professionals usually wonder:
- Does the variability in SmartItem renderings have an unfair result?
- Is the effect of randomness tracked in such a way that final test scores can be statistically adjusted to be comparable?
- Could one test taker’s overall exam end up being more difficult than another’s?
These questions are valid.
The commitment to fairness rightfully belongs at the center of all measurement efforts in the field of testing and assessment. Let’s start out by zooming in on item variance.
Item Variance
Variance in test difficulty is not exclusive to the SmartItem, and is often a baked-in component of multiple-choice tests.
In fact, the variation of item difficulty from the same objective for different test takers is done routinely in most testing programs. Here are a few examples:
- Creating multiple equivalent forms
- Linear on-the-Fly Testing (LOFT)
- Computerized Adaptive Testing (CAT)
The most obvious example is creating multiple equivalent forms. These forms are equivalent overall in difficulty, making the tests fair, but at the level of individual skills or objectives, the items vary in difficulty.
For example, persons taking Form A and Form B, sitting next to each other on test day, see different items taken from the same objective that are not of the same difficulty.
So, logically, as far as variations in item difficulty for different test takers, we as an industry are already comfortable doing something similar, and have done so for as long as any of us remember.
Item Difficulty
The Standards for Psychological and Educational Testing does not require that items be equally difficult for all test takers.
Item difficulty variation is an inseparable part of every test taker’s experience—even when test takers view identical items on the same test form.
This is based on the concept of “perception” in psychology. An examinee’s perception of an item will be affected by many factors, including their reading ability, primary language, familiarity with the testing modality, and more. (More on this shortly!)
With that said, the difficulty of individual SmartItems may vary on a test. But it is the design of SmartItems—as well as the presence of a significant number of SmartItems on an exam—that makes them fair.
For example, I’ll use a single, fixed traditional item as the point of reference.
Imagine any single item on any test. When that item is given to two people of the same exact ability relative to the item content, its difficulty still changes. Why?
- The conditions under which it as presented are different, and it is those conditions that modify the difficulty of the item.
- There are differences in the testing location. Some testing rooms may be hot and humid, or cold, making it difficult to concentrate. Those rooms may have distractions on the
walls, or noises in the halls or outside of the building. - The test takers may differ in anxiety, reading skill, familiarity with test taking or the test format, motivation and interest, fatigue, illness, a known or unknown disability, language differences, and many others.
- External factors, such as the screen resolution or whether the item is shown on a computer or on paper.
- Lastly, scoring errors may occur for one person and not the other.
Research proves that all of these, and others, destroy the assumption that the item, presented under these different conditions, has a unitary difficulty level.
And psychometric theory has allowed the entire testing industry to move forward and consistently assess individuals despite these well-known differences in item difficulty—applying to each item on the test.
Randomization
The differences in difficulty of SmartItem variations are randomly determined, distributing difficulty over the test items.
What that means is a SmartItem produces differences in difficulty randomly. There is no bias.
For example, imagine we take the same test…
On the first question, you might get an easy variation of a SmartItem, and I might get a more difficult variation of the same SmartItem. The SmartItem is not giving me the more difficult variation because it doesn’t like me or thinks I’ve had too many easy ones in the past. The SmartItem is blind to individuals and circumstances, and therefore renders variations in an unbiased way. That means, on the second item, I might get an easier variation and you might get a more difficult one.
The same logic applies for all subsequent SmartItems. As more SmartItems are presented, both tests become more similar in overall difficulty.
So, at some point:
- Our tests are quite equivalent in difficulty, with only minor statistical variation.
- Our scores become comparable and useful
The point where enough SmartItem variations have been given for the tests to be considered equivalent for practical purposes is likely dependent on the domain of content for the test. For example, A math test might take fewer SmartItems than a history test.
Research will likely be needed to establish the optimal length of exams, but that kind of research is common today, even when the SmartItem is not used.
The Purpose of the SmartItem
What we absolutely cannot live with are the very large systematic effects of cheating and testwiseness on test scores. These effects, sources of systematic unfairness, are large and damaging compared to the random effects described above.
It is impossible to overstate the devastating effects of cheating and testwiseness on the valid interpretations of scores. Why we have accommodated those effects over the years is difficult to understand.
If we follow standards and best practices in assessments, we must do all we can to remove sources of unfairness and contributions to error in our test scores. The purpose of the SmartItem is to combat cheating and test fraud. And that it does—effectively.
Read more about the insidious effects of cheating on the fairness of exams, and the vital importance of addressing it here.
Case Studies
Whew, that’s a lot.
Thankfully, there are case studies that support these points.
One of the early adopters of Smartitem is SailPoint Inc.
After developing and testing Sailpoint’s SmartItems, the promised benefits of the SmartItem are confirmed, including stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation.
And since SailPoint’s two certification exams went operational, the SmartItem technology has performed well. Moreover, test-level reliability and validity evidence provided support for the use of SmartItems to produce test scores for high-stakes certification decisions.
If you’re interested in learning more, we invite you to read Caveon’s in-depth book, SmartItem™: Stop Test Fraud, Improve Fairness, and Upgrade the Way You Test–it discusses each of the concepts within this article in far greater depth.
Interested in helping facilitate further case studies? Reach out. We’ll help you get started.
How Can I Implement the SmartItem?
Getting started is a simple process. Whether you integrate with your current system or use Caveon Scorpion, you’ll need to train your team on how to design a great SmartItem.
This is not a prescribed process. In fact, we fully expect to learn a lot about the application of SmartItem technology from early adopters! Each content universe is unique, and the approach to building that SmartItem is like an art form.
With that in mind, there are three basic ways to get started experimenting with the SmartItem.
Ways To Get Started
1. Scorpion
- Scorpion, Caveon’s secure development-to-delivery testing platform, is the first to support SmartItem development.
2. Integrate
- There’s no need to switch test development tools to harness the benefits of SmartItem technology. The SmartItem API will integrate with your existing tools, bringing the power of the SmartItem to whatever system you’re already using!
- With the SmartItem API, coding experience isn’t required. The API is a GUI interface that makes it easy for a SME or item writer to produce code without actually writing the code. By designing the variables needed and creating lists of item components, an SME can combine it all into a SmartItem easily with the API.
- The SmartItem API enables your team to create SmartItems out of any of the item types currently supported by your systems. Custom item templates support any item type, in any format, with no interruptions to the test taker.
3. Innovate
- Interested in doing research using SmartItems? Although the technology that makes the SmartItem possible is patented, we’d love to work with anyone who is interested in trying the innovation on for size. Reach out. We’ll help you adopt with confidence!
Frequently Asked Questions
Here are some Frequently Asked Questions about implementing SmartItem technology:
What are the software requirements?
Using SmartItems is surprisingly easy if you are using Caveon’s item development and delivery tool (Scorpion). If you use other systems for item development and test administrations, we created an integration tool called the SmartItem API. This API quickly and easily integrates with your existing systems and brings the capability and benefits of the SmartItem to you without changing your testing infrastructure.
Do I need to know how to code in order to use SmartItems?
There are two general choices for item writers: design items that require code or design items that don’t. Item developers need to evaluate each skill or objective and determine whether code is needed. Some objectives are better measured by simply increasing the number of options rather than producing code.
Do Subject Matter Experts (SMEs) like SmartItems?
Creating a SmartItem is more of a design task than a writing task. SMEs read the skill description and then figure out how to design a single item to cover the entire skill. Amazingly, a SmartItem is not as difficult, nor as timeconsuming to create, as you might think. In fact, if a development project calls for more than one traditional item for a skill, it might actually be easier to create a SmartItem than traditional items.
How are cutscores set?
Not being able to review items in the “traditional” way poses a challenge to some standard setting procedures. For example, an Angoff requires that someone read the item and then judge its difficulty. Because SmartItems constantly render unique views of a SmartItem, the review procedure needs to be adapted. The Contrasting Groups methods as well as modified versions of the Angoff have been successful.
How to Communicate the Need for SmartItems to Your Team
This new innovation in testing technology may seem out of reach for many test programs, especially if resources are limited.
However, when approaching the conversation, it can help to go at it with helpful, team-oriented ideas. Try these conversation starters on for size when communicating the need for SmartItems to your team:
- “I believe we have a way to make our exams more secure!”
- “Let’s even the fairness playing field by creating items that are impervious to fraud, cheating, and testwiseness.”
- “I know how we can help save our subject matter experts time during the development process (and make it more fun!).”
- “I found a way we can reduce item maintenance and replacement costs!”
- “There’s a way we can eradicate the need to replace our items, while also significantly decreasing costs surrounding incident management!”
No matter how you start the conversation, it’s important we recognize the imperative nature of new, innovative processes for exams. So here’s one last conversation starter you can use to communicate the need for SmartItem technology to your team:
In the assessment industry, one of the most groundbreaking innovations was the multiple-choice item, which was born in 1914 out of a life-or-death need for standardized measurement.
However, over the past century, our testing and evaluation needs have changed. Issues such as security, unfairness, and ballooning costs now plague exam programs and undermine the validity of their test scores.
It’s imperative that the multiple-choice item evolve to meet these current needs. That 100+ year-old invention simply won’t do to protect any exams from the many challenges stacked against us.
And with the new and innovative ways examinees are coming up with to cheat and steal questions, shouldn’t our exams equally provide new and innovative ways to combat this unfairness?
It’s time for us to embrace innovation and technology and bring the multiple-choice question into the 21st century where it can evolve to address our current needs and proactively tackle the problems of the future.
Conclusion
We hope you enjoyed our Ultimate Guide to SmartItem Technology!
In summary, A SmartItem is a self-protecting item treatment that employs a proprietary technology to prevent all test theft and almost all forms of cheating.
SmartItem technology pinpoints and solves many of the modern challenges facing testing today, including test theft, test fraud, testwiseness, ballooning costs, and unfairness. As a result, the SmartItem increases the fairness of exams for all test takers (because one well-designed SmartItem can render in millions of ways, compute variable changes in real time, and present a different version of the item to each test taker.)
Lastly, and most importantly, SmartItem technology performs well psychometrically, and tests comprised of SmartItems demonstrate high levels of validity and reliability.
Now we’d like to hear from you:
What benefits of the SmartItem intrigue you the most?
Is it the time savings? A specific security benefit? The increased quality of your exam? Or maybe something else.
Either way, let us know right now!
More SmartItem Insights
Testing programs have already adopted and love them. They're fun to develop. They’re long-lasting and strong. They’re fair. More importantly, the scientific research results reflect all of that.
The SmartItem on SailPoint Certifications
Ever wonder if the SmartItem actually delivers on its promises? SailPoint testifies that it provides stellar protection, reduced costs, psychometric quality, and more appropriate candidate preparation.
The Future Of Multiple-Choice
Like so many inventions before it, it was created to solve an important societal need, and its impact has been profound. In this white paper, we discuss the history, evolution, and future of the multiple-choice item.
The SmartItem: Does it Work? Case Study Evidence
SmartItem technology solves several critical validity issues that plague high-stakes testing programs today, including testwiseness, cheating, and the theft of exam content. This report has been curated to further prove this claim with evidence in the form of case studies.
Experience the SmartItem in Action
We get it. SmartItems can be tricky to visualize. We find an example can be helpful. So, click the button below and keep reloading the example to see a SmartItem render in real time.
How the SmartItem Improves Tests
In the beginning, one of our biggest concerns about SmartItem technology was how it would contribute (or not) to the reliability of an exam… And we have the answers right here.
The SmartItem™ e-Book
Dr. David Foster discusses a revolutionary concept in item and test design: the SmartItem. Without hyperbole, the SmartItem can prevent all test theft, and almost all forms of cheating. Such a claim cannot be ignored.
The SmartItem: Does it Work? Data Simulation Evidence
To provide additional support and evidence for the use of the SmartItem, this report discusses two data simulations with the goal of demonstrating the equivalence of difficulty for tests with and without SmartItems and to show the effect of SmartItems on predicted cheating.
AIG vs. SmartItem
Okay, seriously—what’s the difference? Though they both achieve similar test security protection, this infographic explores eight ways that SmartItem™ technology differs from Automated Item Generation (AIG).
Curious If SmartItem Technology Is Right For Your Program?
Tell us a little about your organization’s needs and request your free demo today!
