Enhancing oil palm segmentation model with GAN-based augmentation

Abstract

In digital agriculture, accurate crop detection is fundamental to developing automated systems for efficient plantation management. For oil palm, the main challenge lies in developing robust models that perform well in different environmental conditions. This study addresses the feasibility of using GAN augmentation methods to improve palm detection models. For this purpose, drone images of young palms (< 5 year-old) from eight different estates were collected, annotated, and used to build a baseline detection model based on DETR. StyleGAN2 was trained on the extracted palms and then used to generate a series of synthetic palms, which were then inserted into tiles representing different environments. CycleGAN networks were trained for bidirectional translation between synthetic and real tiles, subsequently utilized to augment the authenticity of synthetic tiles. Both synthetic and real tiles were used to train the GAN-based detection model. The baseline model achieved precision and recall values of 95.8% and 97.2%. The GAN-based model achieved comparable result, with precision and recall values of 98.5% and 98.6%. In the challenge dataset 1 consisting older palms (> 5 year-old), both models also achieved similar accuracies, with baseline model achieving precision and recall of 93.1% and 99.4%, and GAN-based model achieving 95.7% and 99.4%. As for the challenge dataset 2 consisting of storm affected palms, the baseline model achieved precision of 100% but recall was only 13%. The GAN-based model achieved a significantly better result, with a precision and recall values of 98.7% and 95.3%. This result demonstrates that images generated by GANs have the potential to enhance the accuracies of palm detection models.

Introduction

As the world’s population is poised to reach unprecedented levels in the coming decades [1], ensuring food security for this rapidly growing populace becomes a global imperative. Addressing this challenge necessitates a holistic and sustainable approach to agriculture. In recent years, digital agriculture has emerged as a transformative force, reshaping our conventional methods. This surge in automation, driven by innovations such as computer vision, the Internet of Things, and robotics, is causing a paradigm shift in the agriculture sector. Simultaneously, these technologies have also enabled automated phenotyping, which accurately assesses plant traits. This plays a key role in improving breeding selection and advancing precision breeding. These automated practices hold the potential to substantially boost agricultural yield per unit area.

Despite the potential of oil palm to yield up to 10 tonnes oil per hectare per year (t/ha/yr), the global average productivity has plateaued around 3 t/ha/yr. Unfortunately, progress in closing this gap has been sluggish for many years. Genomic selection initiatives have shown great promise in addressing this issue. This is particularly true for yield component traits with high genetic heritability such as shell or fruit mesocarp thickness, as various researchers have shown [2, 3]. However, when it comes to complex traits like total oil yield [4] and height [5], environmental factors account for a significant 60% or more of the variation, adding complexity to the improvement efforts. Furthermore, phenotyping for complex traits tends to be slow and labour-intensive, a significant challenge given labour shortages [6]. In response to these challenges, the integration of digital agriculture and automated phenotyping has emerged as a pivotal solution. Among the most cost-effective tools in this transformative journey are the usage of drones.

The utilization of drone technology in agriculture is an emerging and continuously evolving practice. Its applications covers various aspects, including crop classification, pest and disease detection [7,8,9] and phenotyping tasks such as height measurements [10]. While recent advancements in computer vision, especially convolutional neural networks (CNNs), have facilitated the development of highly accurate and automated agricultural object detection models [10,11,12], the persistent challenge lies in constructing models that can robustly generalize across diverse scenarios. This challenge is amplified by the vast diversity of environmental conditions encountered in real-world settings. Within the oil palm industry, the accurate detection and segmentation of palms across diverse age groups, sizes, and environmental conditions remains a significant challenge, impeding the widespread implementation of automated monitoring and management systems. Moreover, the manual annotation of extensive image datasets demands substantial resources in terms of time and labour.

The introduction of generative adversarial network (GAN) [13] offers an intriguing prospect for addressing these challenges. GAN provides a pathway to augment existing image datasets and generate new data [14, 15]. In agriculture, researchers have proposed GANs as a solution to mitigate overfitting and improve CNN classification networks, with successful applications in identifying diseases in crops such as tomatoes and grapes [16,17,18]. Given the relative novelty of this approach in agricultural context, the purpose of this study is to assess the influence of GAN on both detection and segmentation accuracies, with a particular emphasis on oil palm.

Methodology

Data collection and processing

To ensure a wide representation of diverse terrains, topographies, and other environmental factors in oil palm plantations, eight estates owned by SD Guthrie across Malaysia were selected using a stratified random sampling method. These estates contained immature or young palms (< 5 years old, before canopy overlap) with a planting density of approximately 180 palms per hectare. For this study, the DJI Mavic 2 Pro drone, equipped with a Hasselblad camera with an F2.8 EQV 28 mm lens, was employed for mapping purposes. The flight altitude was set at 80 m to capture detailed imagery. Flights were scheduled between 8 AM and 11 AM, and 2 PM and 5 PM on clear days with low wind conditions to capture varied lighting conditions while minimizing shadow effects. As for the drone settings, image overlap was set at 80%, sidelap at 60%, and a flight speed of 5 m/s was maintained to balance efficiency with image quality. Additional drone settings, including optimal camera parameters such as exposure, sharpness, and precise GPS accuracy, were kept at default.

The collected images were uploaded to our customized WebODM [19] server. After image processing, the stitched orthophotos were separated into individual tiles of 640 × 640 using gdal_retile.py script from GDAL library [20]. From this step, 7755 generated tiles were selected and annotated using LabelMe [21]. After manual inspection and quality checking, 6499 high-quality tiles were selected. Of these tiles, 5168 tiles were randomly assigned to the training set, while the remaining 1331 tiles were used as the validation set.

Two additional independent estates were chosen, and the images acquired followed the same tiling and processing procedures as previously described. From these estates, a total of 100 tiles were selected and designated as test/challenge set 1. This dataset consists primarily of palms older than 5 years. In addition, a separate challenge set 2 was assembled from an estate impacted by a destructive storm, resulting in the generation of 100 tiles for evaluation.

Detection and segmentation model

The palm detection and segmentation models were built with Detection Transformer (DETR) [22] on top of the Detectron 2 framework [23]. The model backbone architecture used was the “ResNeXt50_32 × 4D” [24], an extension of the ResNet architecture [25], featuring 50 network layers, 32 cardinality levels and width of 4 (Fig. 1).

Fig. 1

Representative backbone architecture for ResNeXt50_32 × 4D. The convolutional layers were labelled as kernel size, convolutional layer name, in channel, out channel, cardinality level. The dotted curve arrows represent skip connection with dimension correction (convolutional residual block), whereas the solid curved arrows represent skip connection without dimension correction (identity residual block)

For the transformer-based object detection training, the initial learning rate was set at 1e-4, batch size 16, weight decay at 1e-4 and learning rate drop at 50. Most of these parameters were determined through trial and error using grid search, while the batch size was optimized based on the available computing memory. The encoding and decoding layers were both kept at 6, embedding size at 256, dropout at 0.1, number of attention heads at 8 and number of query slots at 100 [22]. Training was stopped when both the train and validation loss converged. The segmentation head of the network was trained separately using the frozen weight from the previous training. The same parameters were used except for the learning rate drop, set at 20 and batch size at 4. The model trained was labelled as “baseline palm model”. The training was carried out on a Google Cloud Platform Virtual Machine with a single NVIDIA Tesla A100 GPU, 85GB RAM and 12 CPU.

During each validation step, Common Objects in Context (COCO) [26] evaluator function was used to assess the model quality/accuracy. The COCO evaluation metrics used in this study were mean Average Precision (mAP) (at Intersection over Union (IoU) of 0.50:0.95) [26], Average Precision (AP) (at IoU of 0.50) [27] and mean Average Recall (mAR) (at IoU 0.50:0.95), both for maximum detections of 100 and for all areas. Besides COCO metrics, a simpler and more practical metric, known as palm count precision/recall, which was based on precision and recall at a detection score of 0.9, was also calculated manually.

GAN-based augmentation

From the training dataset, individual palms were segmented out from the tiles and placed into the center of 256 × 256 pixel sized images with black backgrounds using a Python script. The images were manually reviewed, and those depicting complete and clear palm features were selected. 1,444 images were selected from this step. These images served as the dataset for training the Generative Adversarial Network (GAN) generator and discriminator from scratch. The GAN architecture used in this step was Style-based Generative Adversarial Network 2 (StyleGAN2) [28] with adaptive discriminator augmentation [29], implemented in Pytorch [30, 31]. The “kimg” parameter was set at 25,000 [29], learning rate at 0.0025 while the batch size was set at 64 with a single GPU. The other parameters were kept at default. The training process was stopped after the FID (Fréchet inception distance) score plateaued and no longer showed improvement on TensorBoard [32]. Using the resulting model, approximately 200,000 synthetic palm images were generated. Accompanying each of these synthetic palms, automated palm segmentations were generated in JSON [33] format using a customized Python script.

37 random drone orthophotos from diverse global locations were retrieved from OpenAerialMap [34] and subsequently partitioned into individual tiles. From this pool, a total of 20,225 tiles were selected, alongside an additional 29,775 background control tiles generated from vacant field images. This combined dataset of 50,000 tiles served as the background dataset for the subsequent phase of the study. Employing a custom Python script, four synthetic palm images were randomly inserted into each background tile, ensuring no overlap, and simultaneously generating the corresponding segmentation JSON file.

Cycle-Consistent Generative Adversarial Network (CycleGAN) [35] was utilized to enhance the realism of synthetic tiles. The dataset of this step comprised of 50,000 synthetic tiles and 13,422 real (unannotated drone-captured) tiles. To facilitate model training and evaluation, both the synthetic and real tiles were divided into training and validation sets. In this process, 90% of the tiles from each category were designated for the training set, while the remaining 10% were set aside for validation. In the context of the A-to-B direction, the synthetic tiles were utilized as training dataset A, and the real tiles were employed as validation dataset B. Conversely, for the B-to-A direction, the real tiles constituted training dataset B, and the synthetic tiles served as validation dataset A. The selected mode for the GAN was “lsgan” with the discriminator network (net_D) kept as “basic”. Conversely, the generator network (net_G) implemented was “unet_128”. The learning rate was set at 0.0002, batch size at 20, decay epoch at 10 and loading size at 640. These parameters were determined through iterative trial and error experiments, ultimately selecting the configuration that produced the best balance of image quality, training stability, and computational efficiency for the augmentation task. Training was stopped when the loss values for generator losses and the discriminator losses all stabilized. Attainment of acceptable level of image quality was another condition. Utilizing the final generator model, all the synthetic tiles were transformed to closely resemble the real drone tiles.

33,746 good quality transformed tiles were combined with the original 5,168 tiles and used as the new training set to build the new palm detection and segmentation models using the same network architecture and method as before (Supplementary Fig. 1). The resulting model, known as the “GAN Palm Detector” was also evaluated on the final test/challenge datasets.

The performance difference between the GAN palm model and the baseline model was quantified. Using the “numpy” and “scipy” Python library, 95% confidence intervals for palm count precision, recall and F1 score metrics were calculated across all challenge datasets. To quantify the uncertainty in our performance metrics, we implemented a bootstrap resampling procedure. 1000 bootstrap samples were generated, each comprising 30 randomly selected individuals from the original dataset, using sampling with replacement. The resulting confidence intervals were then analysed to determine the statistical significance of the observed performance improvements.

Software used

All steps in the methodology were carried out using Python 3.10.13.

Result

StyleGAN2-ADA network training was completed after 1500 epochs, and the final plateaued FID score was 16.82. Sample synthetic palm images can be found in Fig. 2.

Fig. 2

Sample GAN-generated palm images

As for CycleGAN, training was stopped after 50 epochs. The generator A had a loss of 0.166, and discriminator A’s loss was 0.266, while for generator B and discriminator B, the losses were 0.196 each. An example of synthetic tile before and after CycleGAN transformation can be found in Fig. 3.

Fig. 3

A) Synthetic palm tile before CycleGAN transformation. B) Synthetic palm tile after CycleGAN transformation

Tables 1 and 2 present the COCO-evaluated model performance for both the baseline and GAN-based models, focusing on detection and segmentation respectively. Additionally, Fig. 4 illustrates the CIs calculated for palm count precision/recall metrics. The result shows that the performance of both GAN palm model with the baseline palm model were comparable for both the validation set and the challenge set 1. The GAN palm model outperformed the baseline palm model for challenge dataset 2. As illustrated in Fig. 4, the baseline model achieved 100.0% precision but only 13.0% recall (F1 score 0.23), indicating that while it rarely misidentified palms, it failed to detect most of them. In contrast, the GAN-based model achieved both high precision (98.7%) and high recall (95.3%), resulting in a significantly higher F1 score of 0.97. Representative examples of the baseline and GAN-based model’s performance in detecting and segmenting palms were shown in Figs. 5, 6 and 7.

Fig. 4

Precision, Recall, and F1 Scores: Baseline vs. GAN palm models across validation and challenge datasets. Asterisks (*) denote statistically significant differences between models

Fig. 5

Comparison of A) raw tile B) baseline palm model and C) GAN palm model for validation dataset. Both models performed almost equally in detecting palms

Fig. 6

Comparison of A) raw tile B) baseline palm model and C) GAN palm model for challenge dataset 1. The baseline model mistakenly detected some of the low-resolution shrubs as palms

Fig. 7

Comparison of A) raw tile B) baseline palm model and C) GAN palm model for challenge dataset 2. The baseline model was not able to detect storm-affected palms. Comparatively, the GAN palm model was even able to detect the fallen palm in the middle

Table 1 COCO evaluation table for palm detection models

Table 2 COCO evaluation table for palm segmentation models

Discussion

The introduction of CNN-based object detection models has spurred advancements in agricultural automation. Notably, these models have found application in tasks such as automated weed identification in crops [36] and the detection of plant diseases [37]. In the context of oil palm industry, CNN-based models are being explored for automated phenotyping and drone or satellite-based palm detection and counting [38,39,40,41]. This current study builds upon the application of CNNs in palm detection and extends the concept into palm segmentation. Here, segmentation is specifically referred to as instance segmentation, focusing on pixels representing individual palms, instead of panoptic and semantic segmentation [42]. However, a significant challenge lies in the generalizability of these palm CNN models across diverse environments. Unlike crops grown in controlled environments like greenhouses, oil palms are cultivated in open fields, exposed to numerous unpredictable factors. Environmental elements such as weather conditions (rain, wind, and fog) and varying lighting conditions influenced by sunlight and time of day are known to impact drone image quality [43]. Additionally, factors such as drone camera type and flight altitudes have been identified as contributors to variations in image quality [44]. The intricacies of image processing and stitching further complicate this issue [45, 46]. Hence, developing a generalizable model necessitates a diverse representation of palm images. Rather than manually addressing every conceivable scenario, GAN [13] offer a potential solution to mitigate these issues.

GAN based background switch has been proposed as an augmentation method for object detection [47]. This study shares a similar augmentation principle as the referenced publication, with the object of interest being inserted into a new background image. However, one major difference is that the palms being used were generated via StyleGAN2 [28, 29]. StyleGAN2 comprises a generator and a discriminator; the generator produces synthetic images, while the discriminator evaluates and distinguishes them from real images. An essential feature of StyleGAN2 is its ability to independently manipulate high and fine-level details in images, known as style-mixing. It also introduces stochastic variation, adding randomness for greater diversity in synthetic image generation (Supplementary Fig. 2). Leveraging these capabilities, StyleGAN2, along with its predecessor StyleGAN, had been used in generating highly realistic human faces [48], aerial imageries [49], medical images [50], and microstructural images of alloys [51]. Given StyleGAN2’s exceptional ability in generating high-quality, diverse images with fine-grained control over style attributes, it was selected in this study to generate realistic synthetic palms. These synthetic palms expanded the dataset and introduced a wide range of variations in appearance, thereby potentially improving the robustness and generalization capability of the palm detection model. In this study, the synthetic palms were generated onto an empty background image, and the annotation masks - essential for pixel-wise class classification, were automatically derived from object boundaries.

While the synthetic palms introduced variability in individual palm characteristics during model development, accurately representing the full spectrum of background and environmental conditions across all possible plantation scenarios remained a challenge. To enhance the models’ robustness in diverse settings, we incorporated additional environmental variations using a random selection of drone images from various global flight missions, sourced from OpenAerialMap [34]. This approach aimed to expose the models to a wider range of real-world contexts. However, imprinting individual palms onto these background tiles presented challenges including color inconsistency, resulting in an artificial appearance. To address this, CycleGAN [35] was employed. Known for its image-to-image translation capability, CycleGAN maps images from one domain to another while preserving content. Beyond artistic style transfer, CycleGAN has found applications in X-ray image augmentation [52] and laser–visible image translation [53]. In our case, CycleGAN was used mainly to enhance the realism of synthetic tiles by harmonizing color and lighting conditions, effectively bridging the gap between synthetic and real tiles. While it’s acknowledged that a combination of manual mixing techniques can potentially substitute GAN for this purpose [54], this avenue was not explored here.

Data augmentation has been proven valuable in image classification [55] and its extension into object detection has been explored, albeit with a slightly lower impact on accuracy [56]. In addition to conventional augmentations such as random flipping which were done for both datasets in this study, the synthetic tiles generated through GAN-based augmentation were used together with the real tiles to develop the GAN-based palm detector. The DETR framework [22] implemented on top of the Detectron2 object detection framework [23] was used to build the palm detectors used in this study. DETR integrates a transformer encoder-decoder with a CNN backbone. The CNN backbone used for feature extraction was a variation of the ResNeXt architecture [24], which was built upon ResNet [25] with the introduction of the “cardinality” concept. Nevertheless, both these architectures shared similarities in that they were both based on residual learning, which involves the use of bottleneck blocks and skip connections. The cardinality feature of the ResNeXt architecture, which divides the input channels into multiple groups and perform separate convolutions for each group, helps the model capture diverse features and learn different aspects of the training images in parallel.

The transformer following the CNN backbone, consists of an encoder and decoder [22], is used for global context reasoning. The DETR architecture incorporates a set-based global loss with bipartite matching, enabling pairwise and parallel decoding of object embeddings and simultaneous prediction of object coordinates and class labels. DETR’s versatility has been demonstrated across various applications, including medical object and drone-based insulator defect detection [57, 58]. Given the typically structured and dense arrangement of palms in plantations, and that replanting is usually conducted on entire fields, DETR’s ability to capture global context and relationships between objects positions it as a particularly fitting choice for our application.

In their respective validation datasets, both the baseline and the GAN-based palm detectors demonstrated strong performance across all detection, segmentation and counting tasks. The GAN-based palm detector achieved impressive precision and recall values, reaching up to 98.5% and 98.6% respectively. These results were comparable, and in some cases, slightly superior to reported values for similar tasks involving various agricultural crops or plants [59, 60]. The mAP and AP values also stood on par with the findings of other relevant research works [61,62,63,64]. It is noteworthy, however, that the single-class focus of this study—oil palm—likely contributed to these high-performance metrics.

Upon applying the models to challenge dataset 1, in general a slight decrease in palm detection accuracies was observed. This can be attributed to the dataset’s characteristics, which include older palms and overlapping canopies, posing increased challenges for detection. Compared with detection, the segmentation accuracies declined slightly more. This decline can be attributed to the presence of dense canopies, which cast shadows and obstruct the visibility of individual palm canopies in the surrounding area. This limitation hinders effective observation and delineation of the palm canopies during the segmentation process. Comparing palm count accuracies between the baseline and GAN models revealed that the GAN-based model demonstrates a lower susceptibility to false positives. Though not reflected in the mAP and AP metrics, the baseline model displayed a slight inclination to mistakenly identify indistinct shrubs, which loosely resemble palm seedlings from top view, as palms (as illustrated in Fig. 6). It is important to note that the validation and challenge set tiles used in this study were predominantly from our plantations, with most tiles exclusively featuring palms. Instances where tiles contained both shrubs and palms were rare, and tiles exclusively featuring shrubs lacked annotations and resulted in their exclusion from COCO evaluations. Consequently, many of the falsely detected palms on these tiles could only be accounted for in palm count metric. While the precision improvement may not appear statistically significant within the confines of our current dataset, it’s important to consider the potential impact in more diverse plantation settings. The GAN-based model’s enhanced ability to differentiate palms from other vegetation types suggests improved generalizability, which could prove particularly valuable when applied to plantations with more varied flora. This generalizability can be attributed to the diverse background objects present in the drone background tiles sourced and processed from OpenAerialMap [34].

The challenge dataset 2, collected from a plantation affected by a destructive storm, represented an extreme scenario that served as a test for unforeseen and severe plantation circumstances. The palms found here were also > 5-year-old, similar to challenge dataset 1. The mAP, AP and mAR of the baseline palm model were all lower than 0.15. Though achieving count precision of 100%, the recall was only 13%, indicating that all detected palms were correct; however, the model missed a substantial number of actual palms. As a direct comparison, the GAN-based palm detector was capable of all detecting, segmenting, and counting the palms, achieving comparable accuracies with challenge dataset 1 and its validation set. In many cases (as shown in Fig. 7), the model was even able to detect fallen palms. From the top view, an individual palm canopy appears radial and almost symmetrical. The formation of the canopy is driven by the sequential growth and arrangement of fronds in a spiral pattern, known as phyllotaxis [65, 66]. These fronds are usually packed in an organized spiral, which contributes to the vertical and horizontal dimensions of the canopy. As for the individual fronds of an oil palm, they have a fan-like shape with a central axis and radiating leaflets. While some StyleGAN2-generated synthetic palms resembled the phenotypic outcome of these biological patterns, others did not and generated asymmetrical canopies with irregular fronds. In the challenge dataset 2, the storm has, to a certain extent, altered the canopy shapes and orientation of the fronds. It is likely that that the training dataset used to construct baseline model inadequately represented these structural changes. On the other hand, the GAN-based training dataset exhibited a broader range of “possible” palm canopy structures in real-life, thereby achieving high accuracies. The storm-affected palm dataset exemplified an extreme instance of palm canopy variations; typically, distortions to palm canopies in a plantation are usually not as severe. Nevertheless, the results demonstrate that a detection or segmentation model constructed using GAN-generated synthetic tiles in conjunction with raw tiles exhibits superior generalizability and robustness compared to a model relying solely on raw tiles.

While both the GAN-based and baseline models showcase effective performance across various age groups, with the GAN-based model demonstrating the ability to detect palms with canopy distortions, it’s important to acknowledge the limitations of our study. Our research focused on a specific set of conditions and did not explore several potentially influential factors. These include variations in drone flight altitudes, oil palm planting densities, drone camera models, and camera settings. Specifically, our study utilized a DJI Mavic 2 Pro, a consumer-grade drone. This choice was driven by its cost-effectiveness and widespread accessibility, aligning with the objectives of large-scale plantation digitalization. Flight altitude was optimized at 80 m above ground level to balance coverage area, battery efficiency, and image resolution. This configuration resulted in a ground sampling distance sufficient for palm detection but slightly limited the quality of the fine canopy details captured. Furthermore, this study primarily concentrated on the Elaeis guineensis species of oil palm. Although the developed methods may have some applicability to Elaeis oleifera, the extent of this cross-species effectiveness was not directly evaluated in this research.

Future research directions could include expanding the model’s capability to accurately detect palms across all age groups and more locations. In addition, the quality of the GAN generators could be significantly enhanced through the use of higher-resolution imagery. Future studies could employ professional-grade drones equipped with advanced sensors, operating at lower altitudes to capture finer ground sampling distances. These improvements would yield more detailed palm canopy feature representations, potentially improving the accuracy of the GAN generators and the subsequent detection and segmentation models. While this study focuses on oil palms, the challenge of CNN models not generalizing well in crop or weed detection has been highlighted by various researchers [67, 68]. One of the promising solution involves the use of a modified CNN architecture [69]. The GAN-based image augmentation approach presented in this study could be easily integrated as part of a comprehensive solution to improve model generalizability across diverse agricultural contexts.

The GAN-based models developed in this study have demonstrated robust performance in detecting and segmenting palms across a range of previously unseen scenarios, including diverse palm canopy variations and novel environmental contexts. This capability opens up several immediate practical applications in oil palm plantation management. They can be operationalized to automate counting of young palm (< 5-year-old) and to phenotype their canopy growth, enabling the identification of abnormal palms. Furthermore, when integrated with hyperspectral or multispectral imagery from drones or satellites, these models can facilitate accurate plant health assessment and early detection of diseased palms [70,71,72]. This automation forms the foundation of an integrated digital agriculture solution. When coupled with drone-based systems for precise fertilizer and insecticide application [73, 74], it creates a comprehensive approach to plantation management. The implementation of these technologies has the potential to advance the oil palm industry towards a new era of digital agriculture characterized by enhanced automation and precise phenotypic measurement.