Researchers Laure Wynants of Maastricht University and Maarten van Smeden of UMC Utrecht are concerned about the present situation. Together with a group of international researchers, they assessed all previously available prediction models for the early detection and progression of COVID-19. They conclude that while some models do contain important information for healthcare providers, most are currently based on too weak scientific evidence. Their research has been published in the leading medical journal The BMJ.*

Good prediction models

‘Good prediction models are urgently needed,’ says Van Smeden. ‘GPs and specialists in hospitals are now using a variety of prediction models under high pressure. Who runs a greater risk of getting COVID-19? Which patients with symptoms actually have a COVID-19 infection? And what is the expected disease progression in patients with COVID-19? Specifically, you can envisage a model that estimates the chance that a 35-year-old man suffering from shortness of breath actually has a COVID-19 infection. For doctors, it sometimes takes too long to get a test result and they want to have a definite answer more quickly. In a situation like this, you can turn to prediction models, but they aren’t yet sufficiently accurate to serve as the basis for a medical decision.’

National healthcare systems

As well as for the individual patient, these types of prediction models are also important for healthcare as a whole. ‘COVID-19 is an acute threat to global public health, with numbers of infections and deaths increasing daily,’ says Laure Wynants. ‘Since the outbreak at the end of last year, the pandemic has threatened almost all national health systems with overload.’ Early detection and predictions about the progression of the disease are therefore essential to apply the correct prevention measures, diagnostics and therapy in patients. The more targeted the efforts, the more efficient the healthcare will be as a whole. ‘You want to prevent patients from ending up in hospital unnecessarily, for example, but also prevent them from being sent home wrongly and having to be admitted later.’

Various shortcomings

The researchers looked at 31 prediction models produced by 27 international studies, most of them (25) from China. The data for the studies was collected between 8 December 2019 and 15 March 2020. Many have serious shortcomings. ‘Non-representative control patients, for example,’ says Van Smeden, ‘and data sets that are too small.’ The latter may be a result of the rush to come up with a prediction model. Given the emergency, this is understandable to an extent, he says, but it does not alter the fact that the result is scientifically irresponsible. ‘There was one study that looked at how long patients spent in hospital. When the study came to an end after a fortnight, the patients who were still in hospital were removed from the results.’ As a logical consequence, the model based on this study underestimates the time COVID-19 patients are likely to spend in hospital.

Appeal to developers

The research did reveal a number of predictive factors that are important for healthcare practice and policy. Wynants, who also works at KU Leuven, says: ‘Age is a factor, and also gender, for example, but also certain lab values such as C-reactive protein and lactate dehydrogenase.’ Nevertheless, the scientific evidence on which the prediction models are based is generally too weak. For a rapid improvement, the first thing needed is more data. The researchers are therefore calling on developers of prediction models to immediately share their data publicly: ‘Share COVID-19 patient data, because only then can we develop, test and apply reliable and widely deployable prediction models in daily practice.' The researchers believe that an online platform coordinated by an organisation such as the WHO is urgently needed.

Fortnightly update

The researchers are continuing the current study as a so-called living review, with a fortnightly update in The BMJ. The aim is to continuously provide healthcare providers and policymakers with up-to-date information on the quality of COVID-19 related prediction models. They will add new models to their study as they become available and evaluate them critically, examining not only the official publications of prediction models in scientific journals, but also studies that are only available online in so-called preprint archives. The models from these preprints are often already available for use by healthcare providers.

* The publication in The BMJ has been produced by a consortium of international researchers, led by Laure Wynants of the Care and Public Health Research Institute at Maastricht University and Maarten van Smeden of the Julius Center at UMC Utrecht.