The increased flexibility of modern techniques implies that larger sample sizes may be required for reliable estimation. Support vector machines (SVM), neural nets (NN) and random forest (RF) have received increasing attention in medical research, since these hold the promise of better capturing non-linearities and interactions in medical data.

Most prediction models are based on logistic regression analysis (LR), but other, more modern techniques, may also be used. The interest in the development, validation, and clinical application of clinical prediction models is increasing. Prediction of binary outcomes is important in medical research. This implies that such modern techniques should only be used in medical prediction problems if very large data sets are available. Modern modelling techniques such as SVM, NN and RF may need over 10 times as many events per variable to achieve a stable AUC and a small optimism than classical modelling techniques such as LR. Data hungriness was defined by plateauing of AUC and small optimism (difference between the mean apparent AUC and the mean validated AUC 200 events per variable. The area under the ROC-curve (AUC) indicated the performance of each model in the development part and in an independent validation part. We applied each modelling technique to increasingly larger development parts (100 repetitions). We created three large artificial databases with 20 fold, 10 fold and 6 fold replication of subjects, where we generated dichotomous outcomes according to different underlying models. We compared three relatively modern modelling techniques: support vector machines (SVM), neural nets (NN), and random forests (RF) and two classical techniques: logistic regression (LR) and classification and regression trees (CART).

We performed simulation studies based on three clinical cohorts: 1282 patients with head and neck cancer (with 46.9% 5 year survival), 1731 patients with traumatic brain injury (22.3% 6 month mortality) and 3181 patients with minor head injury (7.6% with CT scan abnormalities). We aimed to study the predictive performance of different modelling techniques in relation to the effective sample size (“data hungriness”). Modern modelling techniques may potentially provide more accurate predictions of binary outcomes than classical techniques.

100 top nn model's site's