Shallot Crop Harvest Time and Yield Prediction Using Machine Learning Based on Farmers’ Tacit Knowledge in Brebes Regency, Indonesia

Indonesia has a high demand for shallots, but seasonal production variations cause price fluctuations. The market is asymmetrical, as a significant increase in consumer-level demand directly impacts production centers. Conversely, supply shortages disproportionately impact farmers' prices, driven by high trade and transportation margins resulting from an extended supply chain [1]. Brebes Regency, in Central Java, is a shallot production center that supplies 60% of shallots to the Jakarta market. In Brebes, substantial changes in harvest area and production directly affect the sustainability of shallot farming. Market price information is easily accessible from Jakarta’s market.

Farmers often rely on the Ijon system, a pre-harvest selling arrangement that provides immediate financial relief but can cause them to lose control over their harvest and miss potential price increases. This issue highlights the need for precise yield forecasting to empower farmers in negotiating equitable repayment terms and achieving better financial outcomes. Data analysis and machine learning can help estimate crop yields accurately by integrating farmers’ experience with environmental and production variables [2].

Machine learning-based prediction models play a critical role in effective crop farming, assisting in decisions related to planting, irrigation, fertilization, harvesting, and trading [3]. Among the machine learning approaches, the MARS-ANN hybrid model demonstrates high prediction accuracy by combining ANN's predictive power with MARS's feature selection capabilities. For example, the model has been used effectively to predict wheat, rice, and maize yields based on meteorological and soil data [4]. Similarly, machine vision-based yield monitoring has been employed to create geotagged yield maps for shallot fields, achieving a 76% detection accuracy [5]. In Turkey, onion yield predictions utilized support vector regression and polynomial regression, while in Bangladesh, climatic data was combined with linear regression for shallot yield estimations [6, 7]. Furthermore, the SVM classifier has been used to assess shallot quality with 60% accuracy, highlighting its potential for integration into web or mobile tools [8].

The effectiveness of machine learning-based yield prediction can be evaluated across three dimensions: prediction horizon, scale, and crop type [9]. Predictions are feasible at all vegetation stages, but many studies focus on predictions right before harvest. The best grain forecasts for each model were made before harvest at the start, middle, and end of the growing season [10]. Scale is important because models at each scale serve different purposes, such as plant-scale models aiming to understand factors affecting crop growth, field-scale models assisting in crop management [11], and models at larger scales primarily informing policy-making in agriculture. Klompenburg's "Crop Prediction" model is a comprehensive framework that integrates 21 factors to enhance yield predictions on various scales [12].

Although machine learning has been extensively studied for crop yield prediction, its ability to integrate tacit knowledge, such as farmers’ intuition, experience, and observational skills, has not been thoroughly explored [13]. Tacit knowledge is qualitative and context-specific, often communicated through actions rather than words. Designing systematic yet simple questionnaires that elicit detailed observations from farmers can help encode this knowledge into machine learning processes, enhancing model accuracy and relevance.

This study directly addresses the challenges Indonesian shallot farmers face, such as market dynamics and the Ijon system, by leveraging machine learning for yield and harvest time predictions. Based on the discussion above, this paper aims to: 1) identify the most effective machine learning model for predicting shallot crop yield and harvest time, and 2) determine the critical variables, including tacit knowledge, that influence these predictions.

3.1 Experimental site

The study was conducted in Brebes, in the Central Java province of Indonesia (7°3′0″ S, 108°54′0″ E). This area has a tropical climate with temperatures ranging from 24℃ to 32℃ within the two months, with the lowest temperatures from June to August. The average annual rainfall is around 1200-2000 mm/year. In Brebes Regency, rain can occur in any month, although seasonal rainfall typically begins at the end of October and lasts until May. The probability of rain in a month is more than 80% in January and February.

3.2 Data collection

Data were collected via farmer interviews using questionnaires with the assistance of Brebes district extension personnel in nine sub-districts within the Brebes district, specifically Brebes, Wanasari, Ketanggungan, Larangan, Bulakamba, Kersana, Jatibarang, Bantarkawung, and Bantarharjo. These sub-districts were selected as the main shallot production areas. The diversity in agroclimatic conditions, soil properties, and farming practices across these areas ensures a comprehensive dataset for modeling shallot yield and harvest time.

The dataset gathered from Brebes' extension staff totaled 368, with the condition that the shallot plants were 45 days old when the samples were collected. This age was chosen as it is a critical growth phase where vegetative development and early bulb formation occur, allowing for accurate observation of key yield predictors.

Three samples were taken from each farmer's field to ensure representative and reliable data. This approach minimizes the impact of within-field variability caused by uneven soil properties, microclimatic differences, or irrigation patterns, thereby improving the robustness of the data. The sample size and distribution were designed to capture sufficient variability while maintaining feasibility for field data collection.

Tools such as rulers, scales, and vernier calipers are used to measure growth parameters. Information on the use of fertilizer in shallot plants was recorded. We also used smartphone weather applications to obtain climate data, including temperature, humidity, and precipitation. Agricultural extension conducts data gathering to minimize measurement mistakes. Farmers were then interviewed again after thirty days following the first interview to collect data on the harvest day and yields. All data was collected and stored in Microsoft Excel. They also take note of each field's soil type, texture, and pH. Farmers were also asked about how much water is consumed and how to water it. Several questions related to the condition of plant area, bed size, cultivation methods and farmers' tacit knowledge are detailed in the questionnaire in Table 1.

Table 1. The questions for farmer respondents

3.3 Dataset enhancement

The acquired data is then prepared and segregated into numerical and categorical data. The data underwent cleaning and pre-processing, including tasks such as encoding category data, normalizing numerical data, and scaling the data. Subsequently, both quantitative and qualitative data are merged. The dataset of 368 obtained when used directly in the machine learning process is still relatively small, so additional data is needed through data synthesis activities. The data synthesis process was conducted to replicate real-world conditions of shallot farming, ensuring the synthetic data retained the statistical and contextual integrity of the original dataset. Machine learning models require large volumes of data to be trained and validated. By using large data sizes, they can find meaningful patterns in real-life data [30]. It is known that most conventional machine learning methods produce good accuracy results when the dataset has high dimensionality [31].

The synthesis process involved the following steps:

1. Data preprocessing: The real data was first cleaned and analyzed for patterns, distributions, and correlations to understand the underlying statistical properties.

2. Synthetic data generation using SDV-CTGAN. The Synthetic Data Vault (SDV) library, in combination with the Conditional Tabular Generative Adversarial Network (CTGAN), was utilized. CTGAN is particularly effective in generating realistic tabular data with a small dataset, as it captures the distribution and dependencies between features. The model was trained iteratively, adjusting hyperparameters to achieve replication values above 75%.

3. Validation of synthetic data. The synthetic dataset was evaluated by comparing statistical properties such as mean, variance, and feature correlations with the original data. This step confirmed that the synthetic data retained the characteristics of shallot farming conditions.

4. Augmentation using Gretel AI. A cloud-based synthetic data platform was used to complement SDV-CTGAN. This platform employed advanced algorithms to generate additional synthetic data, offering a user-friendly interface and API for seamless integration. It also provided tools to compare synthetic data against real data for quality assurance.

5. Final Merging. The validated synthetic data was merged with the original dataset, creating a more comprehensive dataset for machine learning. This combined dataset was then split into training, validation, and test sets to ensure robust model evaluation.

3.4 Predictive modeling

A crucial step in machine learning is analyzing algorithm models using the train-test procedure. The machine learning process for estimating harvest time and yield of shallot crops consists of three phases: optimizing the hyperparameters of the base estimator model, optimizing the hyperparameters of the competitor model, and selecting the superior estimator between the base model and the competitor model. The training process uses synthetic data with 12 model algorithms, according to Figure 1.

The next step involves applying machine learning algorithms to the modeling process. We experimented with several methods to achieve improved performance. The algorithms used in this study included FNN, AdaBoost, XGB, SVR, Lasso, Ridge CV, decision tree, gradient boosting, ElasticNet, extra trees, and linear regression. Both hyperparameter tuning and optimum model selection utilize evaluation metrics given by the Sklearn library. Various metrics are utilized to assess the reliability of the optimal model across multiple tests. Based on 70% of the synthesis dataset, we trained the models to produce several alternative models, and the remaining 30% of the data was used in the validation phases. The models are compared using a mean square error (MSE) and a mean absolute error (MAE) to see which performs the best and has the lowest MAE and MSE values. Furthermore, the biggest R-squared value is also evaluated. All data sets without synthesis are reused in the testing and validation process.

1.png

Figure 1. Machine learning workflow was used in this study

3.5 Feature selection to coefficient model

The best prediction model can perform feature selection during the model fitting process. This means it can identify and potentially remove irrelevant or redundant features that might not contribute significantly to predicting yield or harvest time. The best prediction model shrinks coefficients of less important features and eliminates some, resulting in a more interpretable model. This reduces model complexity, enhances prediction accuracy, and ensures efficient models. This tool is used in machine learning models to identify important characteristics that determine shallot yield and harvest time, as well as key tacit knowledge variables.

3.6 Feature selection to coefficient model deployment machine learning model into Google spreadsheet

The model acquired from the training and testing analysis in the machine learning process in the form of linear programming is then entered into a Google spreadsheet with the code "= array formula (if (row (B:B) = 1)); formula calculation". After the farmers' data is input, all variables are normalized. This normalization method involves scaling the input variables to a specific range, often 0 to 1, which helps alleviate difficulties associated with varied input feature scales and improves the model's numerical stability. Each input's value is calculated using the linear programming equation, which includes categorical data. The model calculation results are transformed with the resulting data's normalized range value to provide the desired prediction results.

The model that was deployed in the Google spreadsheet was then evaluated on 15 shallot farmers who were interviewed, and samples of shallot plants were collected at the age of 45 days as in method 3.2 and then interviewed again following the first interview to discuss the harvest day and results. The data supplied into Google spreadsheet was then calculated and reported via an email sent by the system. The computation results must be compared to the data from farmer interviews to determine the difference in error.

This study utilized machine learning models, including classical linear regression, random forest, decision trees, and FNN, to estimate shallot harvest time and production yields. Among these models, the linear model demonstrated the highest R-squared value for both estimating harvest time (TTH) and yields. While RidgeCV outperformed the linear model in predicting TTH and yields by a small margin, the ElasticNet model was identified as the best prediction model through the best estimator application. These models effectively estimated the critical factors influencing shallot harvest time and yield. Harvest time is influenced by factors such as plant age and morning temperature, while yield predictions are determined by area, number of beds, bed width, temperature range, bulb weight, and phosphorus weight. Additionally, tacit knowledge such as farmers' calculations for harvest timing, day length, reasons for planting, and confidence in crop yields, was found to significantly impact the model's accuracy. Farmers can leverage this tacit knowledge, including historical data and environmental cues, to estimate the optimal harvest date and predict yield outcomes more accurately.

The study's findings have practical implications for farmers and agricultural planners. By applying these machine learning models and integrating tacit knowledge, farmers can make more informed decisions regarding planting and harvesting schedules, optimizing yield predictions. This could lead to better resource allocation and improved shallot production efficiency. The model performed well in Brebes Regency but may require further adjustments and validation for use in other regions or with different crops. The forecasted TTH deviation was 23.5%, with a yield inaccuracy of 13%. Comparing these results with actual farm data will be crucial for refining the prediction models and enhancing their applicability in broader agricultural contexts.