A new method of large-scale short-term forecasting of agricultural commodity prices: illustrated by the case of agricultural markets in Beijing

Data preprocessing

Time series models are good at analyzing and forecasting long-term data, which has clear trend and regular fluctuation. Therefore, this paper uses weekly price in time series models, by calculating the average daily prices in 1 week [27]. The purpose is to raise forecasting accuracy, by avoiding the influence of fluctuation and abnormal amplitude.

ARIMA (p, d, q) model

This paper forecasts agricultural commodity weekly prices with ARIMA model as time factor forecasting model. ARIMA model is a classical and widely-used model. Parameters p, d, q respectively represents the order of auto-regression, the difference time of smoothing the time series, and the order of moving average.

θ _i ⁽¹⁾ (t) is a series of random variables, in this model is the weekly price change of time t. t represents for time. μ represents for mean value. B is backward shift operator, B(W(t)) = W(t − 1). ρ(B) is moving average operator, ρ(B) = 1 − ρ ₁(B) − …ρ _q (B). φ(B) is auto-regression operator, φ(B) = 1 − φ ₁(B) − … − φ _p (B). ɛ(t) is independent disturbance, or random error.

In this model, we first put the data set to ADF stationarity test (augmented DF stationarity test). If the data set fails the test, difference the data set until it can pass the test or abandon this group of data [28]. In fact, most agricultural commodity prices can pass ADF test within one order of difference, therefore we assign d = 1. The values of p and q are chosen by AIC (Akaike information criterion) test. Set the range of p and q within 1 to 10. Then put the training set to AIC test, and find out p and q of the least AIC value. It takes a long time to figure out p and q for each agricultural commodity and an alternative solution is to directly take p = 10, q = 8. The forecasting results are accurate.

Data preprocessing

Consumers will not react to price changes within the same day. Therefore there is a time lag in the influence of price changes in other markets. This paper takes weekly average value of price difference, in this way to retain the trend of price changes, and leave enough time for reaction time lag.

Besides the time lag, the relevance between different wholesale markets is another difficulty in model designing, as most methods in regression analysis require variables to be mutually independent. The purpose of the model designing in this part is to evaluate the influence intensity between agricultural markets. Therefore, this paper uses partial least squares (PLS) method to forecast prices based on the space factor [29].

PLS model

Partial least squares method includes one procedure which is similar to principal component analysis (PCA), therefore can be used on variables with multiple correlations. For an agricultural commodity in market i, we want to forecast weekly price change θ _i (t) at time t. Independent variables are the price changes of other markets (θ ₁(t − 1), …θ _i−1(t − 1), θ _i+1(t − 1), …, θ _n (t − 1)) at time t − 1. We preprocess the training set with procedures above and put it in PLS model, and obtain regression relations between the price changes of target market and the price changes of other markets at the last time point. Finally we get the forecasting value θ _i ⁽²⁾ (t) of space model by the regression relations.

Through PLS model, we can obtain regression coefficients between each pair of agricultural markets, which to some extent reflect influential relationship between agricultural markets.

Furthermore, here we use PLS instead of directly using multivariate ARIMA model because multivariate ARIMA model require variables to be co-integrated. However, the price changes of agricultural markets in China indicate that the price change series in different agricultural markets have different stationarity. Therefore, different markets, failing in co-integration test, are not co-integrated. So we consider about using PLS method, a more general method, which can deal with all types of multivariate series.

Hypothesis of price fluctuation

First, this model proposes a hypothesis that besides fluctuation around the mean value, all price changes are caused by the change of exogenous variables.

The agricultural commodity is a component of market economy. Its price is irreversibly influenced by other economic variables and exogenous variables including weather and price changes. This kind of change is definitely not a fluctuation around the mean value [31–33]. Therefore, it’s a reasonable hypothesis.

The influence brought by exogenous variables will accumulate as time goes by. Due to the uncertainty of the influence, analysis of the influence at a single moment has a huge error. Therefore the next section will propose several methods to deal with exogenous variables, in this way to synchronize the price changes with the accumulation of exogenous variables.

Definition of urgency and sample calculation

We set c no more than 7, thus we can divide one cluster to be the median value, three clusters higher and three clusters lower, to reflect the stability and huge fluctuation of the price, as showed in Fig. 3. There will be no more than 7 different values of agricultural commodity prices after clustering.

1. 3.

   Raising the dimension. To the price p _i (t) of an agricultural commodity in the market i at the moment t, set the nearest future price change to be δ _i (t), which happens N(t) days from now, therefore we can get a new daily data \((\delta_{i} \left( {\text{t}} \right), {\text{N}}_{i} ({\text{t}}))\).

    
2. 4.

   Obtaining new variables. After last three steps, we get \((\delta_{i} \left( {\text{t}} \right), {\text{N}}_{i} ({\text{t}}))\). Now we define some new variables. This paper expects to quantify the range of possible price changes from the values of δ _i (t) and N _i (t), therefore defines a variable of urgency U _i (t). Suppose that price θ _i (t) lasts for time T _i (t). Based on experimental effect and quantification purpose, we define the U _i (t) as:

    

$${\text{U}}_{i} (t) = \frac{{(T_{i} \left( t \right) - N_{i} (t))\cdot\delta_{i} \left( t \right)}}{{T_{i} \left( t \right)\cdot N_{i} (t)^{{\frac{1}{4}}} }}$$

If N _i (t) < 3, take N _i (t) = 3, in order to prevent the urgency from sudden change which makes training and forecasting difficult.

From the definition of U _i (t), we can see that the bigger the price rise is or the sooner the change happens, the stronger the urgency is. So U _i (t) can quantify the urgency degree of price changes and send warning messages. The urgency change of honeydew price in some market is shown as Fig. 4.

The transformation of exogenous variables and sample calculation

Some exogenous variables have their own change trends, therefore showing no conspicuous relationship with the urgency change trends of agriculture commodities. Meanwhile these variables are random. So it is inappropriate to directly use the daily data of these exogenous variables in fitting and forecasting. Because we cannot avoid the random volatility of exogenous variables and the influences caused by their own features.

This paper takes urgency as the independent variable, respectively takes the accumulating value of temperature change, whether snowy, foggy or stormy, takes the accumulated maximum values, average values and each day’s values of crude oil prices and the exchange rate and thus obtain 14 derivative exogenous variables.

Warning model based on neural networks

This paper builds a BP neural network model [35] to research on exogenous variables and urgency.

Set the number of hidden layers to be 1. We choose the node number of hidden layer by mean square error (MSE), and choose LM method as the training algorithm. After the parameters are determined, we can train the training set using neural networks [36, 37].

In fact, the purpose of urgency is to reflect the accumulated effect of exogenous variables. From the definitions of 14 derivative variables, we can see that some of them are monotone as time goes by, and some of them are accumulating.

Trained neural networks can adjust to the urgency every day. The definition of urgency indicates that urgency measures the trend of price changes. High urgency doesn’t indicate a certain price change. Instead, it indicates a wider range of price change (if the price really changes).

Here, \(med\left\{ {{\text{U}}_{i} \left( s \right), s = t - 6, \ldots t - 1,t} \right\}\) is the median of the urgency values from day t − 6 to day t.

Forecasting results of the same agricultural commodity in different markets

We forecast the prices of cowpea and watermelon in four markets in Beijing. From week 61 to 75, several price fluctuations of cowpea happened with long intervals and wide range. And price fluctuations of watermelon last for short time. Two agricultural commodities have quite different price change trends.

The forecasting results of cowpea prices in Xinfadi Agricultural Wholesale Market, Fengtai District, Beijing and Dayanglu Agricultural and Sideline Products Wholesale Market, Chaoyang District, Beijing are showed in Figs. 5, 6, 7 and 8.

We can see from the figures that the forecasting results of weekly prices trend are nearly consistent with the real data. The forecasting results of the warning values are quite satisfactory, too. We almost precisely forecasted the trend of the price change warning values and rising amplitude.

We can see that the error of the mixed model is relatively smaller, which means that we can decrease the errors by combining the time factor model and the space factor model. The error values are all within 0.4. Cowpea prices are relatively cheap and change in a wide range, so the forecasting effect of weekly prices are good, which we can also see from the figures.

Furthermore, the forecasting results of the weekly prices of cowpea are quite precise in the trend forecasting (whether the price goes up/down), but are less precise in the forecasting of a sudden rising or declining. Time series models have limitation in forecasting a sudden change.

The forecasting results of the watermelon prices in two markets in Beijing are shown in Figs. 9, 10, 11 and 12.

The prices of watermelon have sudden fluctuation in short time periods from week 61 to 75. From the forecasting results of weekly prices, the mixed model has better forecasting results of price changes. In the warning model, as exogenous variables change, the change trend of the forecasting warning values are nearly consistent with real situation.

We can see that the mixed model of this paper is superior to each sub-model from the error analysis. The mixed model decreased the maximum values of each single model. The forecasting results are more stable. So the mixed model is superior to each sub-model.

From the forecasting results we can see that the mixed model proposed in this paper is good in forecasting weekly price changes and whole trends. The mixed model cannot precisely predict a huge rising or declining. Daily warning values of urgency are a good supplementation to weekly prices forecasting. When the price stays constant for a rather long time, neural networks can precisely forecast the urgency. When prices have huge fluctuation in a short time, forecasting results will have bigger error as variables do not accumulate.

As for the forecasting of urgency, from a consumer’s perspective, the sooner the price changes, the more important the accuracy of the warning value is. Therefore some errors are allowed to the forecasting results of the warning values when the next price change is still far away.

The prices of the agricultural commodities chosen by this paper often stayed constant for a long while. Thus the forecasting results of weekly prices are slightly fluctuant around the zero value. Therefore some errors are allowed, and the error values are usually small.

Forecasting results of different agricultural commodities in the same market

The forecasting results are almost the same as the real situation. Time series models and regression analysis only deal with data set itself, and the warning model deals with exogenous variables. This indicates that agricultural commodity prices are influenced by seasonal factors, prices of all markets and economical variables.

Analysis of special cases

When the prices intensely fluctuate, both models have huge errors. The mixed model has huge errors because sudden changes of a single market has no direct relationship with price changes of other markets. The feature of ARIMA model makes forecasting of huge fluctuation difficult. Neural networks model learns the relationship between exogenous variables and price changes. The feature that huge fluctuations are different from historical data model makes forecasting difficult. Quick changes make exogenous variables hard to accumulate and obtain the best forecasting results.

In fact, the phenomenon mentioned above hardly happens. When price fluctuations are intense, the mixed model precisely predicts the trend. Taking blunt-snout bream in Baliqiao Agricultural Wholesale Market, Tongzhou District, Beijing as an example, its price declined over 10 yuan for once, thus the mixed model is unable to forecast such huge decline. MSE between forecasting values and real values is 7.62. Two errors of the urgent are more than 15. The forecasting of trend, however, is correct, as the model successfully forecasts the trend for four times out of total six times. Furthermore, these six changes can be regarded as three 2-week continuous changes. In this way, the mixed model forecasts correctly for all three times. The forecasting result is shown in Fig. 13.