Sentiment Analysis and Stock Data Prediction Using Financial News Headlines Approach

MKSVR, a system that quantitatively analyses news of financial market in intra-day and combines with stock tick price, was employed by Li et al. [1]. Over the course of a year, experiments were carried out utilizing market news and tick data from the Hong Kong stock exchange. The results show that the MKSVR can make greater use of hidden facts obtained from news articles. Information from historical stock prices are used than models which just make use of news artefact for stock price estimation. Authors also proves that the MKSVR method surmount models that employ just single source of information.

Li et al. [2] elaborated a developed system names eMAQT for profit analysis. This system supports an assumption mentioned previously about finance related to public. It clarifies that public information events are vulnerable to multiple explanations by investors. This outcome provides excellent trading possibilities for knowledgeable investors to benefit after the news release day. In other words, in the age of social media, stock markets are responsive to public information.

The following summarizes the contribution of the work undertaken by Nguyen et al. [3]. First, although earlier research analyzed general feelings in documents, this study elaborates a strategy for prediction of market data related to stock in terms of subject sentiment. Second, we presented two approaches for capturing topic sentiment connections. The first is technique which makes use of current topic model names as JST-based method, and the other is a method which uses defined way to recognize topics and feelings. This method is called as an Aspect-based sentiment method. Finally, this is the initial study to exhibit the usefulness of using sentiment analysis by investigating big scale test data. Although the average accuracy is only 54.41 percent, the suggested technique may foresee change in stock price with more than 60. Crone and Koeppel [4] showed that sentiment indicators may accurately predict market action. In terms of the non-linear descriptive model for continuous returns it has a validation set directional accuracy of 75.64% and a generalization set real-world rate of 60.26%. The observed findings proves that non-linear models performs way ahead than linear regressions and another benchmark models naive technique. The bivariate study also revealed that the relationship between the exchange rate and other sentiments strengthens after market moves. The investigation revealed the capability of employing sentiment indicators to define financial time series.

Nemes and Kiss [5] employed several sentiment analysis algorithms to emotionally inspect and categorize different news headlines from economics and assess their influence on distinct stock market value movements even when the context was missing. Emotions were divided into three categories: good, negative, and neutral. Text Blob and NLTK-VADER Lexicon tools have neutral categories, however Recurrent Neural Network (RNN) did not. The outcomes of the different sentiment studies were compared to the BERT result as a benchmark. Zuo and Kita [6] reported the P/E ratio prediction technique employing a Bayesian network. They digitized the P/E ratio data by using the uniform clustering or Ward technique to cluster the P/E ratio frequency distribution. The digitized P/E ratio data are used to generate a Bayesian network for the interdependence among prior P/E ratio distributions. Authors compared the correlation coefficient and accuracy of prediction of the actual stock price with algorithms of classic time-series forecast. These algorithms are ARMA, MA, ARCH and AR models.

Katayama and Tsuda [7] verified that the emotion determined by the dictionary of polarity had a stock market effect. Three hypotheses were developed and tested each one by authors. Hypothesis 1 states that if favorable news is released, increase in the stock price of company will be observed, and the outcome will be as predicted. Many investors feel that by reading the news and making investing decisions, they are assessing the substance. The second hypothesis states that hypothesis 1 effect will increases for front-page articles. The analysis based on regression confirmed that this hypothesis was valid. Sidogi et al. [8] used Long-Term Short-Term Memory (LSTM) networks to study the im- pact of financial news headline sentiment on the prediction of stock prices. The analysis is carried out on intra-day data with precise lag durations between headlines from the published article and realized stock values. They conducted a systematic comparison of the performance of LSTM models for stock market forecasting under identical settings, but with an objective evaluation of the relevance of including financial news emotions as model inputs. Sheta [9] developed fuzzy models for two nonlinear processes using the Takagi-Sugeno (TS) approach. They were the development effort estimate for a NASA software project and the stock market projection for the following week SP 500. The creation of the TS fuzzy model may be accomplished in two parts. 1) Using model input data, determine the membership functions in the rule antecedents. 2) estimate the consequence parameters. They estimated these parameters using least-squares estimation. The findings were encouraging.

Ho et al. [10] suggested an intraday financial trading system driven by a unique brain-inspired evolving Mamdani Takagi-Sugeno Neural-Fuzzy Inference System (eMTSFIS). When compared to current econometric and neural-fuzzy forecasting methodologies, the eMTSFIS predictive model contained synaptic mechanisms and in- formation processing capabilities of the human hippocampus, resulting in a more resilient and adaptable forecasting model. To create buy-sell trading signals, the suggested system’s trading approach was based on the moving-averages convergence/divergence (MACD) concept. The lagging aspect of the MACD trading rule may be addressed by including forecasting skills into the calculation of the MACD trend signals. The experimental findings based on the SP500 Index demonstrated that eMTSFIS could deliver highly accurate forecasts and that the resulting system could discover timely trading opportunities while minimizing wasteful trading trans- actions. These characteristics allowed the eMTSFIS-based trading system to provide investors with better multiplicative returns. Nagar and Hahsler [11] presented a unique approach to aggregate news from various sources using automated text mining approach to build a corpus for news. The corpus underwent filtration to isolate pertinent sentences, subjected to analysis through natural language processing (NLP) methods. A sentiment gauge, denoted as News Sentiment, was introduced, relying on the count of words displaying positive and negative polarities. Open-source tools were employed to construct tools for both news compilation and aggregation, along with a tool for sentiment assessment. The researchers observed a robust correlation between the temporal fluctuations of News Sentiment and the real-time shifts in stock prices. Su et al. [12] implemented a self-organized neuro-fuzzy model with five layers. This system uses technical indicators for modeling stock market dynamics. The model's predictive and forecasting capabilities were validated using a dataset comprising four indicators: Volume Adjusted Moving Average (VAMA), Stochastic Oscillator and Ease of Movement (EMV) which is basically sourced from TAIEX. To enhance stock price prediction, a proposed adaptation of the moving average method was introduced for generating the input set for the neuro-fuzzy model. Simulation outcomes demonstrated the effectiveness of the model in prediction and its accuracy in forecasting. The neuro-fuzzy model significantly mitigated input errors stemming from the modified moving average method, leading to improved prediction outcomes.

Zarandi et al. [13] implemented an expert system for stock price analysis based on type 2 fuzzy rules. A fuzzy logic system of interval type 2 was employed to account for uncertainties in modeling rules, with each feature membership value represented as an interval. The suggested type 2 fuzzy model incorporated technical and fundamental indices as input variables and underwent testing for predicting the stock price of an Asian automobile manufacturer. Rigorous experimental testing revealed the model's efficacy in accurately predicting price fluctuations across diverse sectors of stocks. The outcomes were highly promising and were integrated into a real-time trading system for the anticipation of stock prices during active trading periods.

Sim et al. [14] proposed a new method for generation of rules for selection of undervalued stock named as 3D subspace clustering. 3D subspace clustering proves to be a highly efficient method for handling high-dimensional financial data, demonstrating adaptability to new datasets. This approach remains impervious to human biases and emotions, providing easily interpretable results. Over an extensive 28-year experimentation period in the stock market (1980 to 2007), XII observed that applying rules derived from 3D subspace clustering algorithms, specifically CAT Seeker and MIC, yielded profits 60 percent higher than those achieved using Graham's rules in isolation.

Chowdhury et al. [15] elaborated a model based on prediction which happens after market close to analyze each investor or company sentiment. Over a 4-week sentiment tracking period, a notable association emerged between the initial stock price trend and the effective positive index curve produced by the predictive news mining model presented in this article. The investigation highlighted a robust correlation, approximately 67%, between news sentiment and the original stock price curve. This correlation strongly supports the efficient market hypothesis, indicating that sentiment is distinctly mirrored in the movements of stock prices. Sarma et al. [16] and Nalabala et al. [17] revealed the importance of social media aspects in consideration of sentiment analysis for stock prediction along with e-commerce field. Wahyuningsih et al. [18] provides the comparision of using various machine learning algorithms in stock prediction. An experiment is conducted by Tudor [19] on Romaninan stock market for understanding sentiment analysis importance. Lenten et al. [20] elaborates various parameters used for performance analysis in prediction. Venturini [21] focuses on various risk factors involved during prediction.  Importance of correlation in prediction is elaborated by Nalabala and Nirupamabhat [22]. Gupta et al. [23] explain the importance of sentiment analysis.  

From literature survey, it is observed that sentiment analysis for financial news can be used for stock prediction in real time environment. Most of the researchers used standard dataset which can be the limitation for many real time applications. Various techniques are used for sentiment analysis like fuzzy logic, clustering, classification and natural language processing. According to survey natural language processing is used wisely due to its efficient performance. AIFFN and TF-IDF methods are the most efficient and accurate methods which can be used in the stock prediction for financial news. These methods overcome limitations provided by data storage and vector-based methods used in prediction. It is can concluded that real time financial news fetch and prediction will provide better prediction and good sentiment analysis for stock data as compared to standard datasets and analysis used for stock predictions.

4.1 Feature selection techniques

In this system on Prediction of Sentiment Analysis and stock data using Financial News Headlines, the various feature selection techniques were employed to identify the most relevant and informative features for identified models. One of the techniques systems utilized was correlation-based feature selection. This method enabled us to analyze the correlation between each feature and the objective variable, enabling us to select features that had a strong impact on predicting stock performance and sentiment analysis. By identifying and selecting the most significant features, intends to increase the accuracy and efficacy of our prediction and analysis models.

4.2 Feature extraction methods

To extract meaningful and representative information from the financial news headlines, employed feature extraction methods. One of the primary methodologies used was TF-IDF (Term Frequency-Inverse Document Frequency) vectorization. TF-IDF is a widely used technique in natural language processing that allocates weight to words based on their frequency and prominence in a document corpus. By transforming the textual data into numerical feature vectors using TF-IDF, system will able to capture the relevance of each word in the headlines, thus facilitating more accurate prediction and sentiment analysis. Limitation of TF-IDF is it does not help in getting semantic meaning of words. However in sentiment analysis, semantic meaning is very important to generate positive or negative sentiments. On these sentiments user can take decision about its financial stock predictions.

4.3 Feature engineering approaches

In this system various feature engineering methods were used in addition to feature selection and extraction to improve the predictive capability of our models. Feature engineering is basically used to extract complex patterns and features from the data. This follows the process for creating new features or modifying existing feature. System use many feature engineering approaches specialized to financial news and market data in research. Sentiment word embedding was one of these technologies, which involves encoding sentiment-related information into numerical vectors to capture the sentiment polarity and intensity of news headlines. Due to this system hoped to increase the performance and resilience of stock prediction and sentiment analysis system by including such manufactured characteristics.

Overall, in this system various techniques were attempted to optimize the representation and relevance of features in Stock Data Prediction and Sentiment Analysis system by utilizing feature selection techniques, feature extraction methods such as TF-IDF vectorization, and feature engineering approaches such as sentiment word embedding. In word embedding polarity of sentiments is consider as positive or negative using vector based matrix. System was able to extract significant information from financial news headlines, increase the accuracy of system models, and get a better grasp of stock market dynamics and sentiment patterns using these approaches.

6.1 Prediction model

The system examined many models, including regression-based models, time-series analysis-based models, and machine learning-based models, and selected the best model for stock price prediction.

6.2 Regression based models

Linear Regression was one of the models we picked after researching numerous regression models for stock data prediction. Linear Regression was chosen because of its ease of use, readability, and ability to capture linear connections between variables. It is hoped to find underlying patterns and trends in the stock market by fitting a linear equation to the data. The R-squared statistic was used to assess the effectiveness of the Linear Regression model, which represents the percentage of the variation in stock price explained by the model. The Linear Regression model scored an impressive 96% accuracy, suggesting that it accurately captured the link between the given characteristics and the stock price. It is crucial to note, however, that the accuracy may vary depending on market circumstances and datasets, and more validation and testing are necessary to evaluate its effectiveness in various situations. Nonetheless, the findings underscore Linear Regression’s promise as a trustworthy technique for stock data prediction. Data captured online is nonlinear in manner so it need to be converted into the specific vector as mention in previous section for sentiment analysis. Also real time data capture is dynamic which also requires feature extraction for sentiment analysis. Though the data is dynamic and nonlinear the regression models provides more efficacy on stock data prediction. In real time data capture each feature is important for sentiment analysis whereas the features having more polarity towards positive sentiments are used to generate a linear based stock prediction model. 

6.3 Time series analysis based models

Further in this system another approach was investigated as time series analysis-based models for stock data prediction in system, and one of the models we developed was the Long Short-Term Memory (LSTM) recurrent neural network (RNN). LSTM was selected because of its ability to capture temporal relationships and successfully manage data sequences. We hoped to use LSTM to utilize past stock data and capture patterns and trends over time. The LSTM model’s correctness was assessed using relevant metrics such as Mean Squared Error (MSE) or Root Mean Squared Error (RMSE). The LSTM model has an accuracy of 71%, demonstrating that it can forecast stock values to some degree. It is vital to note that the accuracy will vary dependent on the dataset, hyperparameter tweaking, and LSTM model architecture. Further investigation and testing are necessary to optimize the performance and examine the appropriateness of the LSTM model for various stock market circumstances and datasets.

6.4 Evaluation metrics

To analyze the accuracy and performance of our prediction models, system employed assessment measures such as Mean Squared Error (MSE) and Root Mean Squared Error (RMSE). These metrics compute the average squared difference between anticipated and actual stock prices, giving a quantitative assessment of the model’s ability to forecast stock prices reliably. In addition, system looked at measures like Mean Absolute Error (MAE) and R-squared (R2) to assess the prediction models’ overall performance and capacity to incorporate volatility in stock price data. Sentiment analysis plays a vital role in stock data prediction when both the models are used in an ensemble approach. Both models help in compromising the limitations of each other to improve upon system performance. These assessment measures assisted us in assessing the efficacy and dependability of our prediction models and comparing their performance.

8.1 Data analysis and visualization

Sentiment Analysis is done as a part of data analysis where word embedding is possible with vectorization. Validation is done with the help of feature extraction for real time data. A rigorous data analysis approach was used in this study effort to get insights from stock data and financial news headlines. The stock data was preprocessed, which included managing missing numbers, outliers, and guaranteeing data quality. To investigate the data’s features, exploratory data analysis (EDA) approaches were used. To analyze the data’s central trends and dispersion, statistical metrics such as mean, median, and standard deviation were produced. To discover patterns and trends in stock price movements over time, time series analysis methods such as autocorrelation and trend analysis were used. In addition to data analysis, visualization was critical in successfully presenting the results. Various visualization tools and approaches were used to graphically display the data’s linkages and patterns. Line charts were used to display previous stock prices and track their patterns. The association between stock prices and sentiment analysis ratings generated from financial news headlines was investigated using scatter plots. Heatmaps were used to visualize sentiment distributions across various time periods or stock categories. These visualizations aided in comprehending the dynamics of stock market data as well as the effect of sentiment analysis on stock performance.

8.2 System performance evaluation

To test its efficiency, the system created for stock data prediction and sentiment analysis underwent a thorough performance review. The system’s prediction skills were evaluated using historical stock data and financial news headlines. The data was divided into training and testing sets throughout the assessment process, with the training set used to train the prediction model and the testing set used to evaluate its performance. To assess the accuracy and reliability of the system’s predictions, several assessment measures were used. To measure the difference between projected and actual stock prices, mean absolute error (MAE) and root mean square error (RMSE) were determined. These measures revealed the average size of the forecast mistakes. Furthermore, accuracy metrics were used to evaluate the system’s capacity to accurately categorize the emotion of financial news headlines. To assess the performance of sentiment classification, precision, recall, and F1- score were calculated.

8.3 Comparison with existing methods

The comparison of suggested approach to current techniques for predicting stock data and analyzing sentiment is done. To prove results a thorough literature analysis is done to discover previously published or implemented relevant research and methodologies. System concentrates on typical current approaches that used comparable data and addressed similar goals.

The performance of various existing approaches using the same test dataset and assessment criteria as our system was evaluated and tested. The evaluation of their accuracy in prediction, sentiment categorization, and overall system performance to our suggested approach is done. The system is illustrated for the efficacy and superiority of system in terms of predictive skills, sentiment analysis accuracy, and overall performance via this comparison.

8.4 Interpretation of results

System offered an in-depth interpretation and analysis of the data analysis, system performance assessment, and comparison with current methodologies outcomes. Study of all the patterns and insights gleaned through stock data analysis, such as finding repeating trends, seasonality, and association with market occurrences. System also explore about the results of the sentiment analysis, emphasizing any intriguing discoveries or correlations between sentiment and stock performance.

Furthermore, this paper critically assessed system’s shortcomings and prospective areas for development. Handling of some issues including data quality, model assumptions, and inherent uncertainties in stock market dynamics might have impacted the system’s performance is also considered. System also took into account any possible biases or limits in the sentiment analysis process, such as issues with sentiment categorization accuracy or the effect of news sources.

The goal of the findings interpretation was to offer a thorough knowledge of our system’s capabilities, limits, and implications for practical applications in stock prediction and sentiment analysis. The system can be enhanced for stock exchange where dynamic sentiment analysis can be done in real time environment. Collaboration of sentiment analysis in stock prediction in big extends or real time stock market can be considered as future enhancements.