Data analytics for crop management: a big data view

Abstract

Recent advances in Information and Communication Technologies have a significant impact on all sectors of the economy worldwide. Digital Agriculture appeared as a consequence of the democratisation of digital devices and advances in artificial intelligence and data science. Digital agriculture created new processes for making farming more productive and efficient while respecting the environment. Recent and sophisticated digital devices and data science allowed the collection and analysis of vast amounts of agricultural datasets to help farmers, agronomists, and professionals understand better farming tasks and make better decisions. In this paper, we present a systematic review of the application of data mining techniques to digital agriculture. We introduce the crop yield management process and its components while limiting this study to crop yield and monitoring. After identifying the main categories of data mining techniques for crop yield monitoring, we discuss a panoply of existing works on the use of data analytics. This is followed by a general analysis and discussion on the impact of big data on agriculture.

Introduction

DA, (also called digital farming or smart farming)¹ [78, 105, 130], is a modern approach that uses digital and smart devices [sensors, cameras, satellite, drones, the Global Positioning System (GPS)] in conjunction with Data Mining (or data analytics) to improve productivity and to optimise the use of resources. Digital Agriculture (DA) comes as a response to the increasing demand for improving productivity while reducing farming operational costs. Moreover, the improvement of productivity should not be done at any cost, e.g., overuse of natural resources and chemical products. DA can, for example, manage crop growth by finding appropriate fertilisation program for each farming field and can help farmers to reduce their operational costs and respect the environment by refining their farming operations based on the needs of each part of the farming field.

Since agriculture has a direct and significant impact on the population and therefore its economic environment, DA in its turn should be viewed as the next natural step to respond to the world population’s needs while protecting the environment, by taking advantage of the recent technological advances in digital devices, communications systems, and artificial intelligence. These allow us to construct multidimensional domains, where the farms and farmers are their central subjects. Figure 1 shows the agriculture ecosystem and its direct impact on other sectors of the economy.

Fig. 1

The interlocked sectors involved in DA

Besides, since DA involves the development, adoption and iteration with digital technologies [39], and Artificial Intelligence (data analytics, ...), these developments and interactions should be well-defined (laws, regulations and policies) to guarantee rights and benefits of all the involved actors (farmers, farm holders’, data owners’, developers and analysts, technology vendors’,...) [70, 77, 78, 92, 113, 146].

DA can be regarded as a data driven form of farming, in which decision-making processes are based on explicit information derived from data collected through various sources [148]. DA and Precision Agriculture (PA) seem to refer to the same thing, however, as stated in [148], DA involves the development and adoption of modern technologies in both collecting the data and its analysis in various farming contexts, while PA takes into account only the in-field variability [147]. DA aims to exploit advanced digital devices, ranging from a simple sensor to complex robots, to offer the required farmland treatment with high accuracy. DA can be applied in almost all agricultural fields. For instance, in crop production: DA allows accurate management of crops, which includes fields, wasteland, crop, pest, and irrigation management, soil classification, etc. In Animal production: DA allows monitoring the animal over its whole life cycle, its food quantity, health control and protection from diseases, and so on. Fishery, animal Husbandry, livestock and dairy farming are some examples [14]. In Forestry: We can efficiently manage forests by supporting the environmental and sustainable decision [36]. DA can help in detecting unhealthy trees, air pollution, discriminate different tree species, protect the wildlife, etc. From the economy point of view, the application of DA for forest management enhances the wood quality and its production, which can augment profits; reduce waste and maintain the environment [138].

Addressing DA from all the above mentioned views is a challenging task and cannot be achieved without the participation of specialists from all these sectors. In this study, we focus on the use of Big Data in crop management, it is, not only one of the pillars in agriculture but also it can profoundly affect biodiversity. Moreover, crop growth is a very complex process involving various endogenous and exogenous factors. Recent advances in digital technologies allow us to collect data about all these factors. DA has the ability to elucidate the correlations and interactions of these factor to help farmers and agronomists optimise the productivity while reducing the side effects on the environment. DA exhibits several benefits to agriculture as shown in Figure  2. These benefits were discussed in [10, 13, 70, 98, 104, 112, 113, 130, 135, 148] and summarised in the following:

Fig. 2

Role of DA in crop production process

• DA provides a farmer with useful information to support their decision-making processes, such as soil and weather monitoring and prediction, weed and pest monitoring, crop yield dynamic predictions, etc.

• DA can sustain the environment and improve the products’ quality, since it provides high quality information and measurements for optimal farming operations on each field.

• DA can provide farmers advanced management methods against climate change and other environmental challenges. The farmer can continuously monitor crop growth and protect them against diseases.

• DA offers valuable feedback to farmers and good assessment of risks, to minimise microbiological or disaster-related risks.

• DA can provide prediction and assistance to farmers against adverse weather incidence, disasters and market instability by assessing the loss at the farm level.

• Farmers/agronomists can benefit from advanced models to understand the market and forecast which products could be more profitable.

The contributions of this study are in the investigation of big data analytics applications to crop production. Crop farming is a complex task, and it depends on many factors that should be taken into account. To optimise the operational cost and reduce the impact on the environment, the big data analytics emerges as one of the most cost effective approaches nowadays. The contributions, therefore, include the following:

• A comprehensive overview of Digital Agriculture big-data with a presentation of the conceptual-layered framework to show the effectiveness of data analytics on Digital Agriculture, when some necessary steps have been implemented. For instance, large-scale data analytics can only be effective if the historical data is available, carefully collected, and it is of high quality.

• A highlight of the different types of data used in the existing studies, and a presentation of the classification of different techniques applied to crop yield monitoring and their effectiveness of the overall results.

• A review and analytical studies of the most widely used data mining techniques to crop farming, with a report of their advantages and shortcomings.

• A discussion on the advantages of big data in agriculture, and how this can be used efficiently for crop farming and its extension to the agricultural field in general.

• A discussion on Digital Agriculture applications for crop management in small and large scale holders.

• A discussion on Digital Agriculture challenges and potential paths for future research.

Methodology

To study the impact of data analytics and big data on DA based on previous works, we conducted a systematic review approach that consists of three steps: (1) collection of related work, (2) selection of relevant work, and (3) examination and analysis of the filtered related work.

In the first step, we performed keyword-based research and We gathered a large number of studies from well-known and popular online sources (Web of Science, Scopus, IEEEXplore, ACM, etc.). We used a combination of keywords from the two sets (Big data, data mining, data analytics, machine learning, Internet Of Things, sensors) and (Digital agriculture, smart farming, precision agriculture, agriculture, farming). We gathered more than 327 articles. In the next steps, We selected a small number of articles, which are considered relevant for further analysis, based on their ideas, methods, data types and sources, addressed problems, proposed solutions, tools used and quality of the results.

Through the literature analysis, the study aims to find responses to the following research questions and discuss findings in the following sections.

• What is the process of DA for crop management?

• What are the various data types generated by farms and used in DA applications for crop management?

• What role does big data analytics play in DA?

• How are big data analytics used for crop management?

• What are the influences of the farm’s scale on the application of DA?

• How big is the data used in the proposed DA solutions’?

• What are the challenges facing the DA?

Figure 3 summarises the overall approach, adopted from the PRISMA² flow diagram.

Fig. 3

The research methodology flowchart

Related work

Despite that DA and Big Data being relatively recent research fields, their scientific literature is rich and covers several concepts. As DA is at the cross boundaries between agriculture and ICT, three major dimensions have emerged as of a very high importance; technology, social economics and ethics, and decision-making based on Machine Learning. The first dimension focuses on the use of advanced technologies to improve practices and productivity [56, 124]. In Ref. [124], the authors studied the impact of sensor networks in agriculture, including remote sensing technologies, wireless devices, and other IoT devices. Ref. [56] reviewed some developments in remote sensing within Big Data processing and management in agriculture. The second dimension concerns legal, ethics, social and economic factors of DA, to provide insights into the impact of digitised information and its analysis on the farm management; farmer identity, skills, privacy, production, and value chains in food systems [39, 70, 77, 78, 92, 113, 146, 148]. The third dimension focuses on the application of big data analysis and machine learning (ML), to optimise and forecast the production and the use of resources. In this paper, we only consider this dimension.

Various studies have been conducted on the application of data analytics to crop yield management. For instance, [71] presented a systematic review on crop yield prediction using ML techniques, and extracted major ML algorithms, features and evaluation metrics used in those studies. Ref.[35] discussed the yield estimation by integrating agrarian factors in ML techniques. This allowed them to show a strong relationship between crop yield and climatic factors. Ref. [103] Provided a systematic review on the use of computer vision and AI to enhance the grain quality of five crops (maize, rice, wheat, soybean and barley), disease detection and phenotyping. Ref. [64] reviewed the application of big data analysis in some fields of agriculture. It highlighted solutions to some key well-known problems, used tools and algorithms, along with input datasets. The authors concluded that big data analytics in agriculture is still at its early stage, and many barriers need to be overcome, despite the availability of the data and tools to analyse it. To measure the level of usage of big data in DA, the authors defined big data metrics (low, medium, high) for each of its dimensions (volume, velocity, and variety). However, while it is a very simple model, it is not easy to specify thresholds, as some dimensions, such as volume and velocity depend on technological advances. Ref. [12] presented a review on the use of ML methods to detect biotic stress in crop protection. The authors analysed the potential of these techniques and their suitability to deal with crop protection from weeds, diseases and insects. In addition, they provided very good instructive examples from different fields of DA. An earlier similar study was presented in [89], where the authors studied four very popular learning approaches; Artificial Neural Network (ANN), Support Vector Machine (SVM), K-means, and K-Nearest Neighbour (KNN). Ref. [25] presented a survey on data mining clustering methods applied to food and agricultural domains. It first described major techniques of unsupervised classification, then it examined some existing techniques applied to agriculture products; like fruit classification, wine classification, analysis of remote sensing in forest images and machine vision.

This study is not just an update of previous surveys. The main objective is to examine the effectiveness of big data analytics in crop yield monitoring and discuss the challenges of such paradigm shift in the agriculture domain. Moreover, It is important to understand the sources of datasets, their types, and which ML techniques are more suitable to analyse them.

DA: it’s all about data

Digital Agriculture (DA) relies heavily on the data sources and techniques used to collect it. This data is then organised in agricultural data warehouses and analysed [93]. The results of this data analysis provide significant insights to farmers and agronomists about how to improve the production, minimise the farming operational costs, manage risks, and protect the environment. The process of deploying DA is derived from data science.

Digital agriculture process

Figure  4, adopted from the knowledge pyramid DIKW, shows a data-driven process, which is at the heart of DA. This usually shows how data from past experiences and models serve as input to techniques of mining and analysis to help in future decisions and acting accordingly. The newly collected data will be used to further refine the process and adapt it to an ever-evolving agricultural world.

This is a data-driven methodology derived from the overall knowledge discovery process. The first phase, data collection, is crucial to the validity of the whole analysis. One needs to carefully identify the type of data that should be collected and the approach of gathering it and maintain it through its whole life cycle. This is even more complex in DA, as the data is issued from various and heterogeneous sources, and contains a number of factors of uncertainties. The second phase, data representation and analysis, is very sophisticated, as there is no common standards in the way the data should be integrated, consolidated, to derive a unified representation that is suitable for its analysis, and in the choice of the analysis techniques. Finally, the decision-making is a laborious task, where the extracted knowledge will be associated to the expertise of farmers and agronomists, farming constraints and regulations to derive new management processes with the view to improve productivity and quality of products, reduce and their impact on the environment. Figure  5 depicts a diagram presenting the DA process for crop yield monitoring, as explained below.

Fig. 4

DA– a data-driven process

• Data collection and preparation It is important to identify the data types and attributes based on the problem at hand (e.g., crop management), and the level of granularity of the data. The required data sources should also be identified and assessed for their data quality. As mentioned above, the data is then prepared for analysis. This includes data integration, representation, selection, transformation, etc.

• Data analysis the complex nature of the agricultural data requires an elaborative analysis approach, ranging from methods of feature selection or extraction to various learning algorithms to discover models, patterns (or knowledge in general term) for data analysis. These will be evaluated against the expected quality of results and their suitability to a decision-making process.

• Decision-making The main goal of the DA process is the decision-making. Any decision should follow the state-of-the-art practice, be justifiable and scientifically sound.

Fig. 5

Big Data Analytics system architecture for crop yield monitoring

Digital agriculture data

In agriculture, Very large amounts of data can be collected from various sources. These include sensors, weather stations, satellite imagery, drone imagery, and many other instruments. The datasets include weather data, farm records, environmental conditions, soil parameters (nutrients, texture, moisture, and so on. The data is usually rich, large, very complex, and heterogeneous. Therefore, its analysis is not straightforward.

The heterogeneity is not only expressed by the data types and formats, but it can be collected using different equipment of different quality. In addition, historical data may be described with different sets of attributes compared to very recent data. This can present inconsistencies in naming conventions and measures when the data is collected from different locations and times. Moreover, the data can be static and historical, which is considered as offline data, and can be online weather data collected at regular intervals (streams of data values), such as weather data (e.g., every 15 minutes), satellite imagery, which is characterised of being spatio-temporal, such as Geo-spatial data, Moderate-Resolution Imaging Spectroradiometer (MODIS) images, etc.

As mentioned earlier, the data collection is not well tackled in the literature. Most of the studies assume that the data is known already, and the experimental setup was already in place. Therefore, more effort is allocated to the data analysis and interpretation rather than on the complete environmental parameters and conditions. In the following sections, we discuss the data analysis process. This discussion is structured based on the main categories of the data analysis; classification, and clustering [24]. Note that, for high quality results, the data needs to be pre-processed, as discussed in the previous section. The pre-processing includes cleaning (dealing with missing values, redundant data, noise and outliers), data transformation, dimensionality or data reduction, and so on.

Classification for crop monitoring

Big Data analytics system architecture is depicted in Fig.  5. While this system is targeted specifically to crop yield management, it can be adapted to any data-driven application. This architecture implements faithfully what we have highlighted in the previous sections. In this section, we will focus on the data analysis layer of the architecture, moreover, we will pay attention to the data types and their sources, techniques of data acquisition, the learning algorithms. The main objective of the crop management data analysis is to get some insights about the crop monitoring problems and show the potential of DA through big data analytics, also called data mining. Data mining and its techniques are involved in several roles in crop production. Farmers may want to know the future yield of their crop, specific areas of their farms suffer from the spread of weeds or under-nutrition. Researchers can look for information such as plant growth patterns, optimum growing conditions, best pest and disease control environment and so on. Data mining offers panoply of sophisticated techniques required to meet all of these needs.

There are two major categories of data analysis: Classification and Clustering. In the work of [24], authors studied applications of data mining techniques in crop management and proposed a classification of these applications. They found that the classification and clustering are the main used categories, where the classification includes prediction, detection, protection, and categorisation). The choice between classification or clustering analysis is very simple. If the models or classes we are looking for were known in advance and we have an annotated data to support the training of the learning algorithms, then classification is the right choice. However, the annotated data is not always available and easy to generate, and in many cases we do not know even which models or patterns we are looking for. In these situations, clustering analysis is the right alternative.

In this section, we focus on the studies that use classification methods for their data analysis. Clustering analysis will be covered in the next section. We structure these classification studies based on the application objectives or targets which arecategorisation, prediction, detection, and protection.

Categorisation

While the classification main objective is to assign a given object into one of the predetermined classes, in the agricultural world, the use of classification process may vary depending on the stakeholders interests. In this study, we report four different applications (or targets) which are widely used in agriculture categorisation, prediction, detection, and protection.

Categorisation aims at defining the classes (or class labels) based on the simple recognition of similarities that exist across a set of entities. For example, categorisation can be used to classify small fruit from fruit with normal to big size, to make an estimation of yields; which may have an economic impact if the farmer wants to make different packages or prices for each type of fruit separately. It can also be used to classify damaged crops from good ones in order to estimate losses, or to prepare for the harvest and marketing. Categorisation can also be applied for crop mapping (e.g., poor, average, high yield), which aims to provide information on farmed fields given a specific type of crops, or to identify a type of crops that are more suitable for a particular field. Based on the input data, categorisation can help improve the farming operations based on the meaningful categories (classes) predefined in advance.

Producing accurate crop maps is essential for effective agricultural monitoring [131]. Categorisation approaches can be applied to study regional crop distribution within or post growing season. For this purpose, it can offer:

• A good understanding of how crops are distributed at early stage of their development; allowing for an opportune decision making and management, as well as adjusting crop planting structure, is crucial. Besides, the timely available of (spatial or maps) distribution of crop types is required for statistical and economic purposes [131].

• The availability of crops maps is critical for the diverse agricultural monitoring activities, such as crop acreage estimation, yield modelling, harvest operation schedules [131, 144], etc.

Moreover, categorisation has been applied for agricultural field mapping [31], to quantify the cropping intensity for small-scale farms [58], to identify and map crops and to retrieve the area of major cultivation [100] and to classify land-cover and crop [76]. Table  1 highlights the major fields, ideas and tools used for crops categorisation. We can see that data issued from satellites and remote sensing, and the features with vegetation indices especially NDVI and EVI, the RGB colours, are the most used.

Table 1 An analytical study on examples of crop categorisation approaches; demonstrates: the type of categorisation application, the used learning algorithm, the data type, data pre-processing and selected features for each algorithm

Crop yield prediction

The estimation of crop yield is crucial in DA, as it enables efficient planning of resources. Economically, an early and accurate prediction of yields can help decision-makers to react to the crops market. Moreover, crop yield prediction permits the study of factors that influence and affect the production, such as climate and weather, natural soil fertility and its physical structure and topography, crop stress, the incidence of pests and diseases, etc.

The prediction of crop yields has been the subject of many studies. Ref. [71] presented a literature review on crop forecasting, where the authors highlighted the most used machine learning algorithms along with the applied metrics and measures. In this section, we examine the learning algorithms that have been used in crop yield prediction from different views: data types, the pre-processing methods, and features or the predictor variables used in each study. Tables  2 and  3 summarise some relevant studies.

The crop yield forecasting approaches follow two major types of sources of data. The first type is related to the sources that have direct impact on the crops. These sources are soil data, weather data, environmental parameter data. These are usually used to predict crop yield [27, 34, 42, 46, 51, 73]. The second type of sources are the use of advanced technologies and tools like satellite multi/hyper spectral images, remote sensing and sensors to collect the data [62, 83, 102, 114, 152]. Some advanced studies use both types of data sources [1, 40, 54, 59, 65, 67, 68, 97, 120, 121].

The forecasting models based on the first type of data sources provides a pre-season estimation of the yield, even before the beginning of the crop season. This allows farmers to decide which strategy to both optimise the farming operations and crop production. These decisions include choosing seeds and crop type, type of fertiliser and its applications. Moreover, This data can also be used for some crop monitoring during the growing season.

The monitoring systems based on the second type of data analysis - data imagery obtained from satellite, cameras, scanner, sensors - allow for on-season estimation (emergence, detect stress conditions of crop, harvest dates, ...). These models are complex since they have to analyse the data that consists of both spatio-temporal and non-spacial. While the spatial data is of high resolution, some images can be of very poor quality, (e.g., images with lot of clouds). Features or predictor variables used in this kind of applications depend on the type of data sources, NDVI and EVI are the most used vegetation indices for satellite and remote/approximate sensors’ data source, min/max temperature and precipitation for weather data source, soil moisture and nitrogen fertiliser for soil based data source.

Table 2 Part 1: an analytical study on examples of crop prediction methods; highlights: the applied learning algorithms, the crop type, data type and pre-processing, the other studied and considered parameters in each proposed approach and the predictor variables for each used algorithm

Table 3 Part 2: an analytical study on examples of crop prediction methods; highlights: the applied learning algorithms, the crop type, data type and pre-processing, the other studied and considered parameters in each proposed approach and the predictor variables for each used algorithm

Crop protection

Crop disease is considered as a major menace for food security in many regions of the world since it causes serious crops losses. While the detection of crop diseases correctly and timely when they first appear is crucial in crop monitoring, this remains a difficult task. One of the solutions to deal with this issue is to use data analytics approach. This will reduce yield losses and prevent farmers to take effective reactive actions. Forewarning can be seen as the outputs of data mining process. Usually, this consists of examining the features of a newly presented case and assigning it to a predefined class.

Several interesting efforts have been developed to prevent crops losses due to diseases, Tables  4 and 5 summarise some major studies. Ref. [7] presented an overview of ML techniques for crop disease classification. In addition, it presented to a case study where a deep learning algorithm was successfully used. Ref. [45] provided a review on advanced ANN techniques to process hyper-spectral data for plant disease detection. Recently, deep learning approaches have been emerged and widely used for plant disease detection and classification, with a variety of network architectures (CNN, AlexNet, googLeNet, CaffeNet, DenseNet, Inception, LeNet, VGGNet,...) and training methods (shallow, deep, from scratch) [9, 16, 21, 28, 38, 63, 79, 82, 125, 139, 143, 150, 155]. Moreover, [127] presented an interesting study on the potential of the use of deep learning for plant stress phenotyping.

Crop protection, that consists of disease, stress, and weed detection, aims to offer tools that detect plants disease caused by various biotic (pathogen, insect, pest, and weed) or abiotic (temperature stress, nutrient deficiency, toxicity, herbicide) variables [126]. The earlier the stress, disease or their symptoms are detected, the greater the chance of reducing the disease spread within a field. This has gained significant advantage from the advances in image collection and processing and their analysis using ML algorithm. The state-of-the-art is very rich. The large majority of studies carried out so far were using image processing, consequently image-based data and classification techniques. These are capable of detecting disease at the scale of leaf, canopy or field [126].

Disease detection at a leaf level uses images collected using digital cameras, which are stored in data warehouses. For instance, PlantVillage database [6, 9, 21, 28, 63, 79, 88, 106, 125, 129, 150] is created for this purpose. The objective of this repository is to build classifiers with high accuracy. The basic classifiers can simply assign to an unseen image a label healthy or infected, while more elaborated classifiers can identify the disease - in other words, classify unseen images to disease classes. However, this approach has some limitations. First, it depends on the quality of the images, as when taken in natural environment, these images are subject to different degrees of light, shadow, dust and leaves overlapping and requires sophisticated image processing, which is not an easy task. Second, usually the datasets sizes are small, which affect the learning phase of the classifiers and more importantly the potential of some advanced learning algorithms such as deep learning. Data augmentation (rotation, light shade’s variation, colour inversion, translation and changes in intensity and so on) is one of the methods used to overcome this problem to artificially increase the number of images [6, 9, 21, 63, 79, 129, 150], but it does not always work. Transfer learning is another solution to scarce/small data-set, where the knowledge obtained from solving a task in a given domain is transferred to the target domain in which the dataset is small [6, 11, 28]. The transfer learning can only be efficient if the source and target domains share some similarities in terms of diseases and their symptoms, for example. Moreover, it is very challenging to transfer knowledge from representations learned using RGB images to a target task using multi-spectral images from UAV or satellite [126].

Third, this approach cannot detect more than a single disease at a time, and the detection of diseases if the symptoms are manifested in another area than leaves. Plant canopy based-image was proposed as a solution to this problem. The idea is to collect data relative to disease in situations where single-leaf phenotypes alone would not provide sufficient information. Such features include the size, the height, the structure, and branching of canopy [126]. The canopy-based detection uses UAV equipped with (multi/ hyper) spectral cameras and sensors to collect the data [32, 49, 80, 82, 136, 143, 153, 155]. Then data needs to be processed to extract features which are usually related to vegetation indices like NDVI and EVI or colours like RGB and NIR. The benefit from UAV images comes with cost on complexity of analysis since images taken by UAV are susceptible to occlusion, overlapping, and atmospheric effects. Also, UAV is not able to fly at higher altitudes, which decreases the quality of the collected images. To cover larger zones and fields, satellite-based remote sensing and images has been proposed as a very good alternative [15, 81, 109, 156]. However, the problem with satellite remote sensing is the revisit time, which is 16 days on average, which makes protection applications difficult, and some diseases can spread rapidly in fields before they are detected. Moreover, passive sensors cannot penetrate clouds [149]. The integration of these data with additional data sources like field surveys, contextual information of field and crop rotation can improve the accuracy [15, 81, 109].

Detecting diseases only from one data source based on digital images or sensor data is not sufficient. Besides, variations in symptoms may lead to false positives due to dynamic nature of plant changes [126]. Consequently, the appearance-based identification of diseases is not reliable enough to accurately detect unhealthy plants, especially in the early growth stages. The use of multi-data sources can improve the accuracy of the detection. For instance, the use of physiological features and morphological characteristics (growth attributes, yield-related features, soil) [66], or the employment of satellite-based images and canopy-based images [156], where the disease can be identified at the plant canopy level and at the field level.

Table 4 Part 1: an analytical study on examples of crop diseases protection and weeds detection approaches; highlights the applied algorithm, plant and data type, data pre-processing and the extracted features

Table 5 Part 2: an analytical study on examples of crop diseases protection and weeds detection approaches; highlights the applied algorithm, plant and data type, data pre-processing and the extracted features

Crop maturity monitoring

Crop maturity is a kind of crop yield prediction, but it is based on image data. This technique has been used in fruit detection, like apples, tomatoes, oranges, etc, and provides an early estimation of yield. It is also used for crop monitoring to provide information to farmers with the view to plan their farming operations, adjust management practices before harvesting, etc. Such intelligent systems for monitoring crop implement the data mining process incorporating machine vision and image processing methods among with advanced learning algorithms, such as CNN, SVM and ANN. Unlike crop yield prediction process described above, this process is based on a single-data source; digital images [5, 23, 52, 75, 108, 122] or sensor based-images [117, 123, 153]. Table  6 summarises such techniques. The challenges of these systems are more or less the same as those of systems for crop disease detection and protection. For instance, images with different illumination and lighting angles, complex surroundings and backgrounds, noise, the presence of clouds, etc.

Table 6 An analytical study on examples of crop maturity monitoring (fruits detection and counting) approaches; highlights the applied algorithm, plant and data type, data pre-processing and the extracted features

Clustering for crop monitoring

Clustering techniques are not widely employed in DA, few efforts have been deployed to investigate the potential of these techniques for zones’ delineation within a field. There are several reasons for splitting an agricultural field into zones. Some traditional reasons include crop diversification within a field, crop-rotation, facilitating the management tasks, and more recently we defined the zones based on yield maps. This usually helps to improve the overall crop yield of the field, by managing the zones more effectively. Therefore, delineation of Management zones (DMZ) is a very important task for farming operations since determining zones of low-or-high yields, and understanding the reasons behind low yields, can help come up with specific solution for each zone with the view to increase the yields. In addition, it has other economic benefits, because we can target each zone with the right amount of fertilisers, water, and other nutrients.

According to [69], delineation of management zones is an effective way to manage the variability of soil within a field, such that each zone will receive specific management. In [145], a management zone is defined as a subregion of a field that has a relatively homogeneous combination of yield-limiting factors, for which a single rate of a specific crop input is appropriate to reach maximum efficiency of farm inputs. In [53], it is defined as a subregion of a field that is relatively homogeneous with regard to soil attributes.

DMZ is a complex spatial problem, which is addressed in the literature from several perspectives. This has attracted interest from many researchers [61, 85, 87, 110, 140]. A literature review has been presented in [90], where the authors discussed the delineation of soil management zones from the variable-rate fertilisation point of view. many other studies presented the delineation based on various criteria. Some techniques that have been used include topographic maps, direct soil sampling, non-invasive soil sampling by electrical conductivity equipment, soil organic matter or organic estimated by remote sensing, and yield maps built using data collected over several seasons/years [99].

Figure 6 depicts the general process of delineation of management zones designed according to methodologies followed by the majority of the literature.

Fig. 6

The delineation management zones process

The majority of problems that are related to crop management imply the management of fields and zones. Therefore, the collected data is usually characterised by geographic coordinates and time associated with each sample, which leads to the use of data mining techniques that are more suitable for spatial and temporal datasets. It is well recognised that agricultural datasets are typically spatio-temporal, as the data is always associated with location and time. However, these datasets contain a significant amount of noise, outliers, and even missing values. For instance, GPS capture devices introduce some noise, imprecisions, and even outliers in the data. Satellite imagery also faces huge imprecision and noise (such as clouds, ...).

Because of the type of the datasets, which is spatio-temporal, it is not surprising to notice that the majority of the clustering algorithms used are of type partitional. K-means and Fuzzy C-Mean (FCM) are considered among the most popular clustering techniques and heavily used to cluster agricultural data [17, 18, 84, 134, 137, 142, 151, 154]. The FCM approach has an advantage over K-means, as it deals better with imprecision and noisy data. Moreover, other types of clustering algorithms have also been proven to be efficient in DA, such as density-based and hierarchical-based clustering techniques applied to DMZ [48, 116].

As mentioned above, besides its huge importance in crop management, delineation of management zone (DMZ) has received much attention, as the data is now available not only from traditional sources but also from refined sources, including advanced data pre-processing techniques. In addition, the recently collected data integrates knowledge of experts and farmers experiences on their fields, which improves significantly the quality of the data [84, 141]. Advanced imaging enhancement techniques improve further the data quality, and they offer the ability to track the development of crops and provide a Geo-referenced data that can describe the spatial and the temporal variability of soil and crops variables at high resolution, covering large areas [17, 84, 101, 132, 133, 141, 151].

Systematic analysis

In the following we will explore the application of data analytics in DA and its extension to big data, and illustrate the practical challenges that hinder the full adoption of DA by farmers.

DA in (small /large) scale farming

Farming can be carried out on a small or large-scale fields depending on several factors like land size, capital, farmer skills, level of use of machinery and technology, etc. According to FAO³ and Grain⁴, over 90% of all farms worldwide are of small-scale holding on average 2.2 hectares (from 0.6 to 10 hectares), except for Northern America where small farms have an average size of 67.7 hectares⁵. Small-scale farms represent 25% of the world’s farmland today, where 73.12% are located in developing countries.

In [10] the authors described three categories of smart farming technology, which are complementary:

• Data acquisition technologies: they are used to acquire the data that is related to the farm. These include remote sensing, weather data, etc.;

• Data analysis and evaluation technologies: these technologies usually take as input the data that has been collected so far and deliver insight to the farmer. These include computer-based visualisation and decision models, farm management and information systems;

• Precision application technologies: these are focusing on variable-rate application and guidance technologies.

The application of smart technologies and data analytics for crop management are not restricted to one kind of farm. Nowadays, every farm should adopt smart technologies, as they are needed for variable rates applications (irrigation, pesticides, fertilisers) [72, 102, 154] while protecting the environment.

The size of the farm determines how these technologies will be used. Large farms tend to develop their smart technology to monitor their farming land, or to afford some of the existing sophisticated systems like CropX as they hold the scale and margins. While small farms tend to rent sophisticated machinery and smart applications on demand, especially with the proliferation of cloud technologies that makes these smart applications reasonable, the work of [30] is an example among others, of a smart irrigation system designed for smallholders. Besides, some technologies are more suitable for large-scale farms like drones and aerial vehicles used to monitor crops which are not as profitable or efficient for small scales because they have less difficulty visualising their crops. On the other side, large-scale farms are responsible for 70% of current deforestation⁶, the largest share of agriculture-related greenhouse-gazes emissions, agricultural water use and habitat disruption resulting in biodiversity loss. Generally, small-scale farms require considerably fewer external inputs and cause minor damage to the environment.

Table  7 summarises the main differences between small and large-scale farming from several perspectives. However, DA can be applied to any kind of farm without restriction. Yet, we have found that the number of papers that addressed large-scale farms is almost the same as works on large-scale farms.

Table 7 Comparison between small-scale and large-scale farming

Technologies for data acquisition Table  7 can be used to all types of farms, such as remote sensing, imagery data systems, and so on. The acquired data, over the years, can lead to the phenomenon of Big Data. If pre-processed and stored properly, this will give a significant competitive advantage to farms that collected them, whether they are small or large. Some of the applications and data analyses that can be performed of the collected are summarised in the Tables  1,  2,  3, 4 and  5,  6.

DA and big data

Big data is not just characterised by the volume, but also by velocity, variety, and others [86]. These are enough to challenge the existing data mining techniques, as trying to develop techniques to deal with large volumes of data (volume), various types of data attributes (variety or heterogeneity), and be able to analyse the new data as soon as it is collected (velocity) are extremely challenging tasks. Moreover, many other characteristics can be found in some big data-driven applications, these include veracity, value, viscosity, veracity, visualisation, etc. In this study, we added veracity, as the data, collected by various instruments and sensors, is of different quality, which creates a huge challenge to the data pre-processing task, and therefore its analysis. In the following, we discuss the impact of Big Data challenges on DA.

• Velocity: many studies that have been examined do not consider the data velocity during their data collection. In DA, the frequency of collecting data depends on its source and the problem for which the data was collected. Some applications need real-time data and others do not. For instance, crop yield prediction does not need real-time data or data streams. It is performed at ad-hoc, while crop protection and disease detection require high quality sensors and imagery data connected to efficient methods of data analysis, which need continuous control.

• Variety: this is very common in agricultural datasets, as multiple sources were used to collect all the necessary information about the farm and farming operations. The data values can be a simple number such as temperatures to more elaborated such as imagery data, NDVI, soil texture, etc. This makes the definition of distance measures and other parameters of the learning algorithms very difficult.

• Veracity: Agricultural data contains many missing values and collected from various sources of varying quality. The data is very noisy, and more importantly it contains many missing values. Therefore, it is very challenging to clean and prepare it for the analysis. This was the case in the work conducted by [37], and also in [93,94,95,96, 107] where data was collected from very large farming areas.

Table  8, summarises a set of representative papers reported in the paper according to their usage of big data. For each paper, we identify the type, the size, the heterogeneity of data used, and the frequency of its collection. Also, we consider the number and type of ML algorithms used, the complexity of the proposed analysis algorithms and devices used to collect data, data analysis applied to a given crop and problem to solve. One can notice while the data analysis algorithms and techniques were heavily used and varied, the rigorous process of knowledge discovery was not followed, usually the data is relatively small either in size (small observations) or the data has few dimensions (for instance, considering only weather data, or fertiliser, without taking into account other factors).

From Table  8, we can extract three classes of applications according to their usage of big data: Full usage (the data contains all the characteristics of big data), light usage (the data contains some characteristics), non-usage (the data does not contain any characteristic of big data).

Table 8 DA applications and their usage of big data concepts

To examine the degree of use of the big data concept and to figure out which of its dimension is more present, we conducted a statistical study where we classify works according to their employment of the 4Vs of big data.

Fig. 7

Distribution of works according to the used Big Data dimensions’

Figures  7 and  8 show that no work has a full employment of big data (4Vs). One can notice that the agricultural data is multidimensional and heterogeneous (variety). Moreover, we have found that the prediction applications display more use of big data, there exist studies that have used three dimensions such as DMZ applications. It is worth noting that these applications, either prediction or delineation of zones, have the potential to use big data to provide stable and accurate results.

Fig. 8

Percent of employment of big data dimensions

If we put aside the volume dimension (V1) (see Figure  7, only 7% of the reviewed studies used (V2, V3 and V4), and 32% of studies just employed data mining techniques for agriculture problems. The most employed data mining techniques are for prediction, including yield prediction, forecasting, prediction of fertiliser applications, etc.

DA practical challenges

There exist a number of challenges and obstacles impeding the potential benefit of DA. In [104], the authors studied the barriers that prevent the adoption of smart farming in their country, Brazil. Some of these barriers include lack of integration and compatibility between different agriculture systems, lack of advanced data manipulation of data obtained from different equipment, poor telecommunications infrastructure on rural areas, and finally, the lack of training in deploying and using new technologies. These barriers are common to the majority of countries in the world.

From the Table  7, we can see that over 73% of crop farms are located in developing countries. So that, the investment in high and sophisticated DA technologies is not there. Most of the main technologies used in DA systems (GPS, UAV, auto-steering and variable rate technology) are designed for relatively large-scale farms located in developed countries [10] or designed by developed countries. Some of these technologies are becoming available recently. For instance, since 2018 African scientists can have access to free and open-source satellite data as a result of a deal signed by the African Union with the European Commission’s Copernicus programme.

As DA is relatively new technology, there is a lack of standards and common solutions for data collection, preparation and storage. In addition, there is a lack of data for many reasons, farmers did not record their data and it takes time to build significant historical datasets [20, 39, 77, 78, 92, 146]. Another major barrier is that many farmers are relying more on their expertise and refusing to adopt these new and complex technologies [10]. Moreover, the transition from their traditional practices and farming habits to these technologies comes with a cost and energy (training and learning new skills).

[20] States that the legal and regulatory frameworks around the collection, sharing and use of agricultural data contributes to a range of challenges. Many laws potentially influence the ownership, control of and data access. Ref. [74] presented a set of socio-ethical imperatives associated with the use of data in agriculture, including dependency risks, data concentration, potential lock-in effects, and the peril of transformation of farmers into information tools, in addition to the sustainability challenges.

Finally, according to [47], the real economic value of the use of big data in farming is still unknown, especially for small-scale farming. Consequently, it will be hard to convince them to switch from process-driven towards data and machine learning driven. This is reaffirmed in [20], where the authors stated that on one side, farmers are enticed with promises of increased profits and farming efficiency, on the other hand the proofs are not there yet.

Conclusion

Digital agriculture (DA) is a data-driven approach that exploits the hidden information within the collected data to gain new insights; transforming the farming practices from intuitive-based decision-making to informed-based decision-making. DA relies on efficient data collection practices, efficient data preparation and storage techniques, efficient data analytics, and efficient deployment and exploitation of the gained insights to make optimal farming decisions.

In this study, we presented a systematic review of the potential use of the data mining process in crop production and management and highlighted serious gaps which can be considered in future studies. The majority of the current practices were dominated by statistical analyses and small machine learning systems. However, these can only give some ideas within a very limited view of the overall system. Agricultural data-driven applications collect a significant amount of data from various sources. This constitutes an excellent opportunity to the field to answer numerous research and practical questions that were not possible before. Nevertheless, despite all the advantages that can be gained from DA, there are several other challenges and obstacles that need to be addressed, among them lack of data, lack of skills, and lack of maturity and standards so that it can be adopted and deployed quickly and easily.

In this study, we cover approaches that deal the entire process of data mining; from data collection to knowledge deployment. We cover this process from big data view, with more focus on crop monitoring and management in an attempt to understand the challenges that DA is currently facing. We defined the research questions addressed by the study and provided a classification of data mining techniques used in the field. For each class, a set of representative existing works have been reviewed, and an analytical study has been provided to highlight the category of machine learning method applied and for which purpose. We discussed the big data concepts and its current impact on DA, and showed that from the data analyst’s view, the transition towards DA is ready to embrace big data analytics concepts. This provides new opportunities of investment into these challenges and allows for a efficient ways of managing crops. Besides, it will provide farmers with new insights into how they can grow crops more efficiently, while minimising the impact on the environment. It also promises new levels of scientific discovery and innovative solutions to more complex problems.