Caveon’s “AI Item Writer” Wins IT Certification Council’s 2024 Innovation Award
Caveon received the IT Certification Council (ITCC) 2024 Innovation Award! This annual award recognizes organizations making transformational changes to an IT exam, a company, or the industry as a whole.
The Challenge:
In the Summer of 2023, Caveon was tasked by one of their clients in the tech space, a global leader in financial software, to generate a large number of high-quality items in a short timeframe. The goal was to develop 1,600 scenario-based items to be used to gauge the learning progress of the client’s trainees. The timeline was six weeks.
To accomplish this endeavor, Caveon built a powerful and innovative software tool called the AI Item Writer. This tool uses AI to not only generate new items, but to refurbish items, clone items, and create test blueprints. Using the AI Item Writer, Caveon was able to successfully develop 1,600 items at a 75% reduction in exam development costs to the client.
Recognizing the potential pitfalls of using AI for exam development, Caveon took a two-part approach.
First, to mitigate security risks without compromising efficiency in the item writing process, Caveon’s in-house team of software developers and data scientists built the AI functionality directly into Caveon’s own exam development platform, Caveon Scorpion™. The AI Item Writer app inside Scorpion has three main use cases:
- Item Writing: Generating items from things like PDFs, videos, etc.
- Item Refurbishment: Cloning existing exam items
- Blueprinting: Creating test blueprints
Developing the AI Item Writer served several purposes: it allowed test developers to ensure that AI-generated items were protected by security measures inherent in Scorpion; it created a secure enclave for the AI tool, preventing inadvertent sharing of sensitive content back into the generative AI model; it allowed generated items to be easily banked, reviewed, and approved; and it allowed for continual advancement of the app, as seen by the addition of the blueprinting functionality.
As the second part of the approach, Caveon’s exam development team collaborated to overhaul item writing processes to work in tandem with the new AI functionality. This necessitated changes in traditional methods for authoring, reviewing, and approving items.
Caveon’s item writers quickly learned how to use the new AI functionality, including how to manufacture prompts and push AI to create more robust items. They then developed a workflow that allowed for extensive reviews to be completed by both human SMEs and prompt specifications in the app to ensure the generated items were accurate and aligned.
In just six weeks, Caveon and the client had created a powerful AI tool, made their tests more secure, and improved their whole process for developing exams.
More than 4 of the 6 weeks of the project were spent developing the AI Item Writer and the new processes for developing items. 10 days were spent generating items. The results were astounding: more than 4,000 scenario-based items were generated and reviewed, resulting in 1,600 approved items delivered for the clients’ exams. Significantly, most of these were not rudimentary questions, but sophisticated, scenario-based items.
The AI Item Writer has continued to advance in the months following. New blueprinting functionality has proven to shorten the Job Task Analysis and blueprinting efforts. In a recent project (details withheld for confidentiality), the blueprint feature condensed a 2-day workshop into less than 2 hours when generated by the AI Item Writer and reviewed by SMEs.
Top Reasons to Use the AI Item Writer:
The AI Item Writer takes time-consuming and expensive traditional exam development processes and makes them agile and affordable, without sacrificing quality and while improving overall security.
Time Savings
The AI Item Writer brings unprecedented agility and speed to the exam development process.
In a conventional exam development timeline, it typically takes 3-4 months to develop 120 items, an average of 30-40 items/month.
In comparison, utilizing the AI Item Writer enabled Caveon’s client to create 1,600 approved items in a mere 10 days. This is an average of more than 3,000 items/month, an accelerated exam development process by nearly 100x.
Additionally, the blueprinting feature has proven to shorten a 2-day workshop into less than 2 hours, saving roughly 14 working hours for every individual in the workshop.
Cost Savings
Developing items using the AI Item Writer translates to a 75% cost savings/item.
Cost savings can additionally be extrapolated to other tasks like blueprinting, where substantial resources are saved by decreasing the time necessary to accomplish the task.
Preservation of Quality
These time and cost savings are achieved without sacrificing an item’s quality, validity, or reliability. In the client’s case, the final 1,600 AI-generated items were mostly complex, scenario-based items that underwent rigorous reviews to guarantee the high standards necessary for a valid high-stakes assessment.
Upholding Exam Security
By building the AI functionality into Scorpion, Caveon ensured that the client’s data remained secure and that content exposure risks were removed. Additionally, the app enables testing programs to refurbish exposed items, clone items, and create large item banks that reduce the risks of cheating and theft, improving the overall security of exams.
Conclusion
Caveon is now able to combine their expertise in exam security with the power of AI to evolve how credentialing programs can build quality exams at scale. Caveon’s AI Item Writer leverages AI to kick-start the item generation process, enables subject matter experts to focus on their domain expertise, and reduces the overall time and cost of high-quality exam development.
AI is real, and it’s spectacular. Above all, the use of the AI Item Writer affirms that when done right, AI in innovative exam development isn’t just a future goal, but a resounding, proven success.
Curious If Scorpion Is The Right Platform For You?
Tell us a little about your organization’s needs and request your free demo today!
