Stock Price Prediction: Technical Analysis, Historical Data & Machine Learning Models

Stock Price Prediction involves analysing historical market information and other financial signals to understand patterns in stock movements. Stock prices can be influenced by factors such as previous prices, trading volume, market volatility, interest rates, exchange rates, commodities, and financial news.

For you as a student, this topic shows how data science, statistics, big data, and machine learning can be combined to analyse financial markets. Before a prediction model is applied, financial data needs to be collected, cleaned, and converted into a suitable format.

The study discussed here uses Hadoop MapReduce for stock volatility analysis and PySpark with machine learning models for stock prediction. It also uses an LSTM model to analyse financial news sentiment and study its relationship with short-term stock movements.

Role of Historical Data in Stock Price Prediction

Historical data provides the foundation for analysing stock movements and preparing machine learning models. The study uses daily stock variables such as Date, Open, High, Low, Close, Volume, and Adjusted Close.

For volatility analysis, the research uses 1000 stock datasets from the New York Stock Exchange (NYSE). Each dataset is available as a CSV file containing daily stock information.

The study calculates the monthly rate of return for individual stocks and uses these returns to measure volatility. Stocks can then be compared based on the variation in their returns.

The research also uses Apple stock data along with financial news. The Apple dataset covers January 1, 2013, to April 2, 2016 and includes Open, High, Low, Close, Adjusted Close, and Volume.

Combining historical market data with news information allows the study to examine stock movements from more than one source.

Stock Volatility Analysis Using Hadoop MapReduce

Stock volatility indicates how much stock prices or returns vary over a period. The study uses standard deviation or variance to measure volatility.

A higher standard deviation indicates greater variation in returns, while a lower standard deviation indicates relatively smaller variation.

The research uses a three-stage Hadoop MapReduce process for volatility analysis.

• First stage: The mapper reads CSV files and extracts the stock name, date, and adjusted closing value. The reducer groups the data by stock and calculates monthly returns.

• Second stage: The next mapper passes the results to another reducer, which calculates volatility using the monthly returns.

• Third stage: The mapper sorts stocks according to volatility, while the reducer identifies the 10 stocks with the lowest and 10 stocks with the highest volatility.

This process demonstrates how Hadoop MapReduce can distribute financial data processing across multiple stages.

Machine Learning Models for Stock Price Prediction

The study uses Apache Spark and PySpark to process financial data and apply machine learning models. The prediction dataset contains variables such as Date, Open, High, Low, Close, Volume, and a dependent variable called LABEL.

The study examines correlations among the available variables to select relevant features. These include:

• Open

• High

• Low

• Close

• Volume

• Interest Rate

• ExchangeRate

• VIX

• Gold

• Oil

• TEDSpread

Data Preprocessing

Before applying the models, the dataset is checked for missing values. Around 2% of the data is identified as missing and removed using the dropna() function.

Categorical variables are converted into numerical representations using techniques such as one-hot encoding and label encoding. The selected features are then combined into a feature vector using the VectorAssembler function.

The dataset is divided into training and testing sets in a 7:3 ratio. The training data is used to build the model, while the testing data is used to evaluate its performance.

The overall workflow is:

Data Collection → Preprocessing → Feature Selection → Training and Testing → Feature Transformation → Machine Learning → Evaluation

Decision Tree Model and Reported Results

The study uses a Decision Tree Regressor through Spark MLlib for stock prediction. The model is applied to the selected market variables to analyse whether the stock price will increase or decrease at the end of the day.

According to the study, the Decision Tree model performed better than the other models included in its comparison, including SVM and logistic regression.

The research reports:

• Accuracy: 90%

• Mean Absolute Error (MAE): 1.024

• R-square: 70%

MAE measures the average size of prediction errors, while R-square indicates how much variation in the dependent variable is explained by the model.

The study also discusses neural networks as a possible area for further improvement. Since these figures are specific to the dataset and methodology used in the research, they should not be treated as a general accuracy level for stock prediction.

News Sentiment and Stock Market Forecasting

Historical prices are not the only information considered in the study. Financial news is also analysed to understand whether news sentiment has a relationship with short-term stock movements.

The research collects financial news from sources such as The New York Times and news aggregators and combines it with historical stock information.

1. News Data Processing

Before sentiment analysis, the news text goes through several preprocessing steps, including:

• Tokenisation

• Removal of numbers and punctuation

• Removal of unnecessary words and spaces

• Removal of stop words

• Stemming

The researchers also create a finance-specific stop-word list. Words appearing in fewer than three documents are removed.

Stemming converts related words into a common root. For example, developed, development, and developing can be reduced to develop.

2. LSTM Sentiment Analysis

The study uses a Recurrent Neural Network with Long Short-Term Memory (LSTM) to classify financial news as positive or negative.

The reported model settings include a batch size of 24, 64 LSTM units, two output classes, a maximum sequence length of 550, and 400,000 training iterations. A threshold of 0.5 is used for sentiment classification.

The study reports more positive sentiment in the analysed AAPL news data.

However, the researchers also observe overfitting in the LSTM model. Training accuracy increases with additional epochs, while validation accuracy remains constant or validation loss increases. This shows why validation performance is important when evaluating a machine learning model.

Applications of Machine Learning in Stock Market Analysis

Machine learning and big data technologies can be applied to several areas of stock market analysis.

1. Volatility Analysis

Hadoop MapReduce can process large collections of stock datasets and calculate monthly returns and volatility. The study uses it to analyse 1000 NYSE stock datasets.

2. Stock Price Prediction

PySpark can process large financial datasets and support machine learning models. The study uses market and economic variables such as stock prices, interest rates, exchange rates, VIX, gold, oil, and TED Spread.

3. News Sentiment Analysis

LSTM can process financial news and classify sentiment as positive or negative. Combining sentiment with stock information can help analyse its relationship with short-term market movements.

4. Large-Scale Data Processing

Hadoop MapReduce supports distributed processing, while Spark provides distributed data processing and machine learning capabilities. Together, these technologies can handle different stages of financial data analysis.

Limitations and Future Scope of Stock Price Prediction

Stock Price Prediction remains challenging because financial markets are affected by changing conditions and several external factors.

Key Limitations

• Changing market conditions: Historical patterns may not account for every factor affecting future stock movements.

• Financial risk: Machine learning models can support analysis but cannot eliminate investment risk.

• Data quality: The quality, size, and time period of the dataset can affect model performance.

• Overfitting: The LSTM model shows that higher training accuracy does not always translate into better validation performance.

Future Scope

The study suggests using more refined datasets, different time periods, and advanced algorithms in future research. Neural networks can also be explored further.

Another suggested area is a real-time trading model using live streaming data. Such a system could process market information continuously and calculate returns or investments in real time.

The study also reports that a Feed-Forward Neural Network produced a more accurate prediction of the opening stock price, which can be explored in future research.

How PW Skills Can Help You Build Skills for Stock Price Prediction

Stock price prediction combines data analysis, Python, machine learning, and big data technologies. PW Skills can help you develop these skills through structured learning and practical applications.

1. Learn Python and Data Analysis

You can build a strong foundation in Python and learn how to work with datasets, analyse patterns, and perform data-driven calculations.

2. Develop Machine Learning Skills

You can learn how machine learning models work and understand concepts such as model training, feature selection, prediction, and evaluation.

3. Work With Real-World Data

Practical projects can help you apply your learning to datasets and understand how data preprocessing, missing values, and feature engineering affect model performance.

4. Build SQL and Database Skills

Learning SQL can help you collect, filter, and manage large datasets, which is useful when working with financial and market data.

5. Explore Data Science and Big Data

You can develop an understanding of data science workflows and technologies used to process and analyse large volumes of data.

6. Apply Your Skills Through Projects

Project-based learning can help you connect theoretical concepts with practical use cases such as market analysis, prediction models, and sentiment analysis.