Skip to main content

Table 2 Forecast quality assessment’ methods

From: Large-scale forecasting of information spreading

# What is evaluated Forecast parameter Method for assessme
1 IM response accuracy for the fixed virality value (without data assimilation) Number of responses Fix \(\Delta t\), \(n\). Calculate \(n\). values of the predicted parameter (predicted parameter should be measured and be saved). Calculate \(MAE\left( n \right)\), \(MAPE\left( n \right)\). Calculate mean value of the metrics of accuracy er \(p\) IMs.
2 IM response accuracy with data assimilation Number of responses Fix \(T, n\) (in Fig. 7\(n = 9\)). A series of forecasts for \(n\). iterations is carried out: a) at the time of IM generation; b) after each batch from the set \(b \in A\). batch of assimilation (with the adjustment of the model parameters from the obtained data). Figure 7 shows assimilation of three batches. Additionally, batches of evaluation \(b \in C\). must be collected to ensure verification predicted values for all \(\left| A \right| + 1\) forecasts. The averaged values of the forecast accuracy metrics for different assimilation batch are calculated.
3 Impact of the batches’ frequency on the forecast accuracy Number of responses Calculate a series of averaged values of forecast accuracy metrics using the method (2) with a fixed length of the forecast period \(T\), varying the length of the time interval \(l\) between batches
4 A number of batches required to achieve desired accuracy Number of batches Fix the desired accuracy \(\varepsilon\), \(p\), \(\Delta t\), \(l\), \(T\), the maximum number of batches of evaluation \(A_{ \hbox{max} }\). The assimilation of the batch is performed while the batch accuracy is less than \(\varepsilon\) or the maximum number of batches is exceeded \(A_{ \hbox{max} }\). Accuracy assessment is made by the method (3)
5 Accuracy of the aggregated dynamics reproduction for the period Number of responses Fix, \(p\), \(l\), \(T\). The response is predicted for \(p\) IMs for the period \(T\). The numbers of responses in the model and the actual number of responses are measured. The value of \(\lambda\) is calculated by the Eq. (6). If\(\lambda < \lambda_{0.05}^{ '} = 1.36\), then the null hypothesis of sample homogeneity is accepted
6 Earliness Forecast time Fix \(\Delta t\), \(l, T\), the number batches of assimilation \(k\). For the moment of IMs generation (\(t = 0)\) and for each assimilation batch, the forecast is made for the period \(T - l \cdot b\). The time of the forecast calculation \(\tau_{0} , \ldots ,\tau_{k}\), is measured. The series of values for the forecast earliness \(z_{0} , \ldots ,z_{k}\) is calculated using Eqs. (7, 8)