IntelliStream: A Machine Learning Framework Based on Regression for Improving Broker Performance and Throughput via Log Analysis

The need for real-time data streaming and processing has significantly increased in today's data-driven society. Platforms designed for these tasks must efficiently and reliably handle a vast amount of data. Apache Kafka, a widely used distributed event streaming platform, has emerged as a key technology for building real-time data pipelines and streaming applications [1]. It remains quite challenging to optimize Kafka's performance to meet the demands of a changing workload, despite its robust design.

Recent studies comparing manual, heuristic, and machine learning–based tuning methods in distributed systems are supported by several journal articles and surveys. Manual tuning of Kafka parameters generally shows only modest throughput improvements, whereas heuristic methods achieve stronger results, and machine learning approaches often provide the largest performance gains while also reducing latency variability [2-4]. Manual tuning in distributed systems like Apache Kafka generally results in limited performance improvements due to its static adjustment nature. The studies [3, 4] presented that manual tuning requires administrators to change distributions and configurations directly and typically only achieves around a 10% improvement in throughput, as changes are not adaptive to dynamic workload conditions. Heuristic-based methods, using rules or profiling to guide tuning, can deliver higher throughput gains (up to 25%) but still struggle to adapt under fluctuating workloads [4-6]. Whereas Machine learning techniques, such as reinforcement learning and regression-based optimization, routinely surpass manual and heuristic methods in both throughput and latency stability (with throughput improvements of 40–50%) [2, 6-8].

Performance tuning of Kafka involves adjusting numerous configuration parameters that influence throughput, latency, and stability. These parameters, such as heap size, buffer sizes, and concurrency settings, can interact in complex ways, making manual tuning a daunting and often inefficient task [9]. Moreover, the dynamic nature of workloads in production environments necessitates continuous and adaptive tuning to maintain optimal performance [10]. The problem has a promising answer in machine learning. Machine learning models can find patterns and correlations in operational logs, particularly garbage collection (GC) logs and broker logs, which are not immediately visible using conventional techniques. These insights can then be used to predict the effects of various configurations on performance metrics and to identify optimal settings [11].

The selection of GC logs and broker logs as primary indicators is motivated by their direct impact on system performance. GC logs capture information about memory allocation, collection frequency, and pause durations [12]. A high frequency of GC events or long GC pauses can significantly reduce throughput by interrupting message processing and increasing latency due to stalled threads. Similarly, broker logs provide operational metrics such as message throughput, request latency, error counts, and partition reassignments [10], all of which directly reflect system stability and responsiveness. For example, sustained error rates or rising latencies in broker logs indicate declining stability under load, while steady throughput values reflect efficient resource utilization. By analyzing both GC and broker logs together, the study establishes a holistic view: GC behavior explains memory-level bottlenecks, while broker metrics capture system-level performance outcomes. This dual-log perspective ensures that the machine learning framework links configuration parameters to the most critical performance dimensions.

The dependent variables that represent performance are explicitly defined as throughput (MB/sec), latency (milliseconds), and stability (measured through error rates and consistency under varying workloads). These metrics capture efficiency, responsiveness, and robustness in distributed streaming. The independent variables are the Kafka configuration parameters, such as heap size, buffer size, concurrency levels, and GC behavior that directly influence performance outcomes. To connect these inputs with measurable system behavior, the study leverages GC logs, which provide details on memory allocation, pause times, and heap utilization, as well as broker logs, which record throughput, request latency, and error statistics. This explicit mapping of independent and dependent variables strengthens the analytical foundation of the proposed machine learning framework, ensuring that the research targets the most relevant and impactful performance indicators.

We utilize the publicly available JVM Logs Dataset from Kaggle [13], which provides GC and Kafka broker logs under diverse workload conditions and serves as a reproducible benchmark for performance modeling studies. which comprises 471 files and 2,880 features derived from GC and Kafka broker logs. This rich dataset captures diverse workload conditions, providing a robust foundation for machine learning–based performance modeling and auto-tuning.

Through a series of experiments, the model shows the effectiveness of the strategy, exhibiting notable gains in performance stability and throughput. In addition to increasing system efficiency, the automated tuning process eliminates the need for significant manual intervention, which lowers complexity and operating costs. Given the complexity and non-linearity of distributed system behavior, relying on a single regression technique may not adequately capture the diverse relationships between configuration parameters and performance outcomes. For instance, while linear regression offers interpretability, it oversimplifies non-linear effects; conversely, tree-based models such as decision trees or gradient boosting can capture non-linear patterns but may risk overfitting [14]. To address these limitations, an ensemble of regression algorithms is often more effective, as it combines the strengths of multiple models to improve predictive accuracy and generalizability. Approaches such as Random Forest Regression and Gradient Boosting Regression have shown significant promise in prior system optimization studies [15], reducing variance and bias while delivering more reliable performance predictions. Incorporating ensemble regression methods into auto-tuning frameworks thus provides a fine-tuned and adaptive means of modeling system performance under dynamic workloads. The objective of this research is to develop a machine learning-based auto-tuning framework that improves Kafka's throughput and stability while reducing manual intervention.

The paper proposes an automated, machine learning-based approach to fine-tune real-time data streaming platforms for better throughput and overall performance. The method involves collecting and preprocessing log data, extracting relevant features, and training regression models to predict performance outcomes. The models are then used to adjust configuration parameters dynamically, ensuring the system operates at peak efficiency under varying conditions.

The remainder of the paper is structured as follows: Section 2 reviews related work in the field of performance tuning using machine learning. Section 3 describes the methodology, including data collection, feature engineering, and model training. Section 4 presents the experimental setup and results. Section 5 concludes the paper with a discussion of the findings, and finally, Section 6 presents potential future work in the area.

The section outlines the methodology for using machine learning techniques to auto-tune real-time data streaming platforms by analyzing GC and broker logs [1]. The approach involves several key steps: data collection and preprocessing, feature extraction, model training, and the application of regression models for dynamic configuration tuning [32].

Specifically, the study evaluates multiple regression algorithms, including Linear Regression, Ridge Regression, Lasso Regression, Support Vector Regression (SVR), Decision Tree Regression, Random Forest Regression, and Gradient Boosting Regression, to capture both linear and non-linear relationships in the data. The dataset, constructed from GC and broker logs collected from AWS EC2-based Kafka deployments, comprises several thousand log entries that encapsulate throughput, latency, memory usage, and garbage collection metrics. By benchmarking a diverse set of algorithms on this dataset, the methodology ensures not only robust model selection but also provides a plug-and-play framework where different regression techniques can be substituted depending on workload characteristics and prediction requirements.

A comprehensive collection of 471 files, including 226 raw log files, 244 structured CSV files, and 1 Python preprocessing script, comprised the publicly accessible JVM Logs Dataset on Kaggle [13], from which the experimental data were sourced. The dataset yields 2,880 columns of features after translation, comprising 1,204 decimal-valued metrics (such as latency, throughput, and memory utilisation) and 1,676 integer-valued parameters (such as counts and frequencies). By capturing crucial data from Kafka broker logs (throughput, latency, and error counts) and GC logs (heap consumption, halt periods, and collection frequency), these capabilities allow for the methodical modelling of configuration–performance interactions under various workloads.

3.1 Data collection and preprocessing

Log collection: The first step involves collecting logs from the data streaming platform [33]. Specifically, gather garbage collection (GC) logs and broker logs. GC logs provide insights into memory management and garbage collection events, while broker logs contain information about broker activities, including message throughput, latency, and error rates. Figure 1 shows the AWS EC2 [34] instances that were provisioned with the steps mentioned in Table 1.

Figure 1. AWS EC2 instances

Table 1. The installation steps on AWS EC2

Figure 2. Zookeeper and Kafka broker running on AWS EC2

Preprocessing: Raw logs are often noisy and unstructured. Preprocess these logs to extract relevant data points. After the EC2 instances are up, Figure 2 shows the zookeeper [21] and Kafka broker running, which is used to capture the logs for further processing.

Data integration: The preprocessed data from GC and broker logs are integrated into a unified dataset. The dataset captures the system’s state and performance metrics over time, providing a comprehensive view necessary for model training.

3.2 Feature extraction

Metric selection: As shown in Figure 3, identified key performance metrics that influence the throughput and stability of the system. For GC logs, important metrics include GC pause times, heap usage, and frequency of garbage collection events [35]. For broker logs, the primary focus is on metrics such as message throughput, request latency, and error rates [10].

Figure 3. Optimized feature selection

3.3 Model training

Regression models: Regression techniques are utilized to model the relationship between the extracted features and the performance metrics. Various regression models are considered, including linear regression, ridge regression, and more complex models like Random Forests and Gradient Boosting Machines [32]. For feature selection, we have adopted the OptiFeat approach, as detailed in prior work [36]. OptiFeat combines subject matter expertise with Recursive Feature Elimination (RFE) to ensure optimal feature selection, enhancing model interpretability and performance. The feature engineering process aligns with the methodology outlined in OptiFeat, providing only the most relevant features are retained for model training.

Training and validation: The dataset is split into training and validation sets (70:30). The training set is used to train the regression models. In contrast, the validation set is used to evaluate their performance [37]. Key steps include:

- Hyperparameter tuning using cross-validation to optimize model performance.

- Evaluating model accuracy using metrics such as R², MAE, and RMSE.

- Selecting the best-performing model based on validation results. The complete process is depicted in Figure 4.

Figure 4. Procedure chart

3.4 Dynamic configuration tuning

Predictive analysis: The trained regression models are used to predict the impact of different configuration settings on system performance. By simulating various configurations, we identified settings that maximize throughput and minimize latency [9].

Automated tuning: Developed an automated tuning mechanism that adjusts the system’s configuration parameters based on model predictions. The mechanism continuously monitors system performance and logs, dynamically updating configurations to maintain optimal performance. The steps involved are:

- Monitoring: Continuously collect and preprocess GC and broker logs.

- Prediction: Use regression models to predict performance under current settings.

- Adjustment: Automatically adjust configuration parameters based on predictions.

- Feedback Loop: Implement a feedback loop where the effects of configuration changes are monitored, and models are retrained periodically to adapt to evolving workloads.

Clear criteria, such as performance deterioration thresholds, prediction confidence ratings, and historical trend analysis, control the auto-tuning process. The model is guided by these measurements to determine the best time and way to make configuration changes, guaranteeing the system performance stays stable and adaptable to shifting workloads.

Implementation: The auto-tuning mechanism is implemented as a modular system that can be integrated with existing data streaming platforms. It includes components for log collection, feature extraction, model inference, and configuration management. The relatively poor performance of SVR and Decision Tree models can be attributed to both the characteristics of the dataset and model sensitivity. Since the underlying relationship between configuration parameters and throughput is largely linear, simpler regression models are better suited. In contrast, non-linear methods such as SVR and Decision Trees underperform without extensive hyperparameter tuning. This reinforces the importance of the proposed plug-and-play framework, which allows such models to be tested in baseline form while enabling future integration of optimized or alternative regression techniques.

Have chosen a variety of regression techniques to provide a comprehensive comparison of both linear and non-linear models, as shown in Table 2, ensuring all potential relationships in the data are considered. Linear Regression serves as a baseline, offering simplicity and interpretability. Ridge and Lasso Regression [39] introduce regularization to handle multicollinearity and feature selection, potentially improving model performance. Non-linear models like SVR [40], Decision Tree Regression, Random Forest Regression, and Gradient Boosting Regression [32] were included to capture complex and non-linear relationships. The diverse selection allows us to benchmark performance across different approaches, revealing that Linear and Ridge Regression perform exceptionally well, indicating a strong linear relationship in the dataset. Non-linear models like Random Forest also showed good performance, suggesting they capture some additional patterns. The comprehensive approach helps in selecting the best model based on empirical results, ensuring robust and accurate predictions.

To evaluate the accuracy and reliability of regression models for performance prediction, several statistical metrics are employed. The most widely used is the MSE, depicted in Eq. (3):

$M S E=\left(\frac{1}{n}\right) \sum_{\{i=1\}}^n\left(y_i-\hat{y}_i\right)^2$               (3)

As defined, it measures the average squared difference between the actual values $y_i$ and the predicted values $\hat{\mathrm{y}}_i$. Building on this, as shown in Eq. (4), the RMSE provides the error in the same units as the target variable by taking the square root of the MSE:

$R M S E=\sqrt{\left(\frac{1}{n}\right) \sum_{\{i=1\}}^n\left(y_i-\hat{\mathrm{y}}_i\right)^2}$          (4)

Another useful metric is the MAE, depicted in Eq. (5), which captures the average magnitude of prediction errors without squaring them, making it less sensitive to outliers:

$M A E=\left(\frac{1}{n}\right) \sum_{\{i=1\}}^n\left|y_i-\hat{y}_i\right|$               (5)

Finally, the Coefficient of Determination (R²) assesses how well the model explains the variance in the data and is expressed as in Eq. (6):

$R^2=1-\frac{\sum_{i=1}^n\left(y_i-\hat{y}_i\right)^2}{\sum_{i=1}^n\left(y_i-\bar{y}_i\right)^2}$           (6)

• $y_i=$ actual observed value for the $i^{\text {th }}$ data point $\hat{\mathrm{y}}_i=$ predicted value for the $i^{\text {th }}$ data point (from the regression model), $\bar{y}_i=$ mean of all observed values $y_i$.
• n = total number of observations.
• $\sum_{i=1}^n$ = Summation over all data points from i = 1 to n.

Among these, MSE and R² are particularly well-suited for system performance prediction tasks. MSE heavily penalizes large deviations, which is crucial for throughput predictions where significant errors can distort optimization decisions. R², on the other hand, provides an interpretable measure of the proportion of variance explained by the model, enabling straightforward comparison across different regression approaches. Together, they offer both absolute and relative perspectives on model performance, ensuring accurate and reliable assessment for auto-tuning distributed systems.

Table 2. Comparing different regression models

The performance of various models is compared in Table 2. MSE and R² were chosen for their balance of simplicity, interpretability, and comprehensive insights into model performance [23]. MAE is less sensitive to outliers and provides a simple average error. MSE helps penalize significant errors. RMSE offers a more interpretable error in the original unit of measurement, and R² shows how well the model explains variance in the data, with values nearer 1 denoting a better fit. They provide a clear and concise way to evaluate and compare the regression models, ensuring robust and accurate analysis. From the comparative results, it is evident that Linear and Ridge Regression outperform other techniques with near-perfect accuracy and stability, making them the natural choice for this study. Lasso Regression, while slightly less accurate, provides added value for feature selection, whereas non-linear models such as Random Forest and Gradient Boosting captured additional patterns but did not surpass the precision of the linear approaches. Support Vector Regression and Decision Trees, on the other hand, showed weaker generalization and higher error rates, rendering them unsuitable for this dataset. Although Linear and Ridge Regression were ultimately selected for their superior performance and interpretability, the framework is intentionally designed to be algorithm-agnostic, allowing future researchers to plug in alternative regression or advanced learning models that may yield better results under different data distributions or workload conditions. The results of this study provide a framework for automated configuration tuning that has been proven to improve the performance of data streaming systems while reducing the need for manual intervention.