Predictive Accuracy Index in evaluating the dataset shift (case study) - Becker, Jarosław; Becker, Aneta (Procedia computer science, 2023)
A dataset shift takes place in a situation, where the joint distribution of inputs and outputs differs between the stages of training, testing, and using predictive models. If the distribution of current data for the implemented forecasting model changed significantly compared to the distribution of data used to develop it, them it could lead to its incorrect operation. The aim of the study was to compare the properties of two indicators, the Univariate Predictive Accuracy Index (UPAI) and Multivariate Predictive Accuracy Index (MPAI). The research procedure was carried out in 2 scenarios. The first involved a comparison of UPAI and MPAI for the distributions of categorical variables, while the second for distributions of continuous variables. The obtained MPAI values for both scenarios were summarized and compared with the UPAI and PSI values calculated and published in our previous article [1]. The results of the experiment proved that basing the decision on the need to calibrate the model or build a completely new model on the basis of MPAI, a measure that takes into account the multivariate distribution of variables, is superior to one-dimensional measures.