Automatic Item Generation by Cees van der Vleuten
A few months ago, we had a successful meeting of the European Board of Medical Assessors in Braga in Portugal. I attended a workshop on Automatic Item Generation (AIG) given Dr. Josè Pego from Minho University in Portugal.
I am aware that our life will be significantly affected by artificial intelligence in the near future in many different areas. I have seen how the computer suggests diagnosis in radiology and pathology. The power of the computer blows you away. I had heard about AIG. It seemed very attractive. Writing good questions and having a bank of questions is one of the costliest elements in education. Being able to generate new items through the computer would be a great asset. I thought that would require complex algorithms that I would never understand and that AIG is only possible in highly specialized organizations.
The workshop was an eye-opener to me and demystified what it entails. The essence is rather simple. Given a certain clinical problem or presenting problem (e.g. abdominal pain), a group of authors develop what is called a cognitive map. A cognitive map decomposes the problem in elements (e.g. age, gender, context, vital signs, cause, diagnosis). Each of these elements may have a set of different values. Then one lets the computer generate a set of items that represent different combinations of the elements of the cognitive map. In the workshop this was done in a relatively simple Excel sheet. Naturally, dedicated software would make life much more easier. I am sure such packages will be developed and made available to the market. In a plenary at the same conference, Dr André de Champlain from the Medical Council of Canada said that experienced developers need a few hours to develop a cognitive map. They then generate items instantly, are able to check the quality of the items generated, which may lead to changes in the cognitive map. With a few real-time iterations hundreds of items can be generated from a single map. In a recent publication1, it was found in the context of mathematics in primary education that AIG was cost beneficial compared to traditional item writing when a set of 173-247 items can be generated from one cognitive model.
AIG has the future. I hope we will see feasible applications soon in daily educational practice.
1Kosh, A. E., Simpson, M. A., Bickel, L., Kellogg, M., & Sanford‐Moore, E. (2018) A Cost–Benefit Analysis of Automatic Item Generation. Educational Measurement: Issues and Practice, 0, 1-6.