YouTube Sentiment Analysis

A comprehensive sentiment analysis project that analyzes and visualizes sentiment patterns in social media comments using machine learning techniques.

📋 Project Overview

This project performs sentiment analysis on social media data (Reddit and Twitter comments) to understand the emotional tone and sentiment distribution across different categories. The analysis includes data preprocessing, exploratory data analysis (EDA), and visualization of sentiment patterns.

🚀 Features

• Data Preprocessing: Clean and prepare social media comments for analysis
• Exploratory Data Analysis: Comprehensive statistical analysis of the dataset
• Sentiment Distribution Analysis: Visualize sentiment categories across comments
• Character Frequency Analysis: Analyze character usage patterns in text data
• Word Frequency by Category: Examine most common words across different sentiment categories
• Interactive Visualizations: Generate insightful plots and charts

📁 Project Structure

Youtube_sentimental_analysis/
│
├── data/
│   ├── Reddit_Data.csv          # Reddit comments dataset
│   └── Twitter_Data.csv         # Twitter comments dataset
│
├── Preprocessing_and_eda.ipynb  # Main analysis notebook
└── README.md                    # Project documentation

🛠️ Technologies Used

• Python 3.x
• Pandas - Data manipulation and analysis
• NumPy - Numerical computing
• Matplotlib - Data visualization
• Seaborn - Statistical data visualization
• Collections - Counter for frequency analysis
• Jupyter Notebook - Interactive development environment

📊 Dataset

The project uses two main datasets:

1. Reddit_Data.csv: Contains Reddit comments with sentiment labels
2. Twitter_Data.csv: Contains Twitter comments with sentiment labels

Data Structure

• clean_comment: Preprocessed comment text
• category: Sentiment label (-1: Negative, 0: Neutral, 1: Positive)

🔧 Installation

1. Clone the repository:

   git clone <repository-url>
   cd Youtube_sentimental_analysis

2. Install required packages:

   pip install pandas numpy matplotlib seaborn jupyter

3. Launch Jupyter Notebook:

   jupyter notebook

4. Open the analysis notebook:

   • Navigate to Preprocessing_and_eda.ipynb

📈 Analysis Components

1. Data Exploration

• Dataset information and structure analysis
• Missing value detection and handling
• Duplicate data identification
• Statistical summary of the data

2. Sentiment Distribution

• Category distribution visualization using count plots
• Analysis of comment distribution across sentiment categories

3. Text Analysis

• Character frequency analysis across all comments
• Word frequency analysis by sentiment category
• Text length distribution analysis

4. Visualizations

• Count Plots: Show distribution of sentiment categories
• Bar Charts: Display word frequency by category
• Statistical Plots: Character and text length distributions

📋 Key Functions

plot_word_by_category(data, n=20, start=0)

Generates a stacked horizontal bar chart showing the most frequent words across different sentiment categories.

Parameters:

• data: DataFrame containing the comment data
• n: Number of top words to display (default: 20)
• start: Starting index for word selection (default: 0)

🔍 Usage Example

# Load the data
import pandas as pd
data = pd.read_csv("data/Reddit_Data.csv")

# Basic data exploration
print(data.info())
print(data['category'].value_counts())

# Visualize sentiment distribution
import matplotlib.pyplot as plt
import seaborn as sns

plt.figure(figsize=(10, 7))
sns.countplot(data=data, x='category')
plt.title("Comment Distribution by Category")
plt.show()

# Analyze word frequency by category
plot_word_by_category(data, n=20)

📊 Key Insights

The analysis provides insights into:

• Sentiment Balance: Distribution of positive, negative, and neutral comments
• Text Patterns: Common words and characters used in different sentiment categories
• Data Quality: Missing values, duplicates, and data consistency
• Language Usage: Frequency patterns of words across sentiment categories

🧠 Additional Components

Machine Learning Modeling and API

Model Training & Pipeline

• Model Selection & Training:
  Multiple machine learning algorithms were explored for sentiment classification, including:

  • Logistic Regression: A linear model suitable for multiclass classification.
  • Support Vector Machine (SVM): Effective for high-dimensional text data.
  • Random Forest: An ensemble method that improves robustness and accuracy.
  • Naive Bayes: A probabilistic model often used for text classification.

• Feature Engineering:
  The following features were engineered and used for model training:

  • clean_comment: The preprocessed text of each comment.
  • word_count: The number of words in each comment.
  • (Optionally) num_stop_words: The number of stopwords in each comment.

• Pipeline Construction:
  scikit-learn's Pipeline and ColumnTransformer were used to:

  • Apply text vectorization (TF-IDF) to the clean_comment column.
  • Scale numeric features (like word_count).
  • Chain preprocessing and model training into a single, reproducible workflow.

• Model Evaluation:
  Each model was evaluated using metrics such as accuracy and F1-score on a held-out test set. The best-performing pipeline was selected for deployment.

• Serialization:
  The final pipeline (including all preprocessing steps and the trained model) was saved as model/pipeline.pk1 using Python's pickle module. This allows for easy reuse and deployment without retraining.

FastAPI Model Serving

• API Deployment:
  The trained pipeline is deployed using a FastAPI application (app.py). This enables real-time sentiment prediction via a simple web API.

• Endpoints:

  • POST /predict:
    Accepts a JSON payload with a comment and returns the predicted sentiment.
    Example request:

    {
      "text": "I absolutely loved this video, it was very helpful and well explained!"
    }

    Example response:

    {
      "prediction": "Positive"
    }

  • GET /health:
    Returns a simple status message to verify that the API is running.

• Input Handling:
  The API automatically computes required features (such as word_count) from the input text, ensuring compatibility with the trained pipeline.

• Error Handling:
  The API is designed to handle invalid input gracefully and will return informative error messages if the input format is incorrect.

How to Run the API

1. Install FastAPI and Uvicorn (if not already installed):

   pip install fastapi uvicorn

2. Start the API server:

   uvicorn app:app --reload

3. Test the API:

   • Open your browser and go to http://localhost:8000/docs for an interactive Swagger UI.
   • Or use curl:

     curl -X POST "http://localhost:8000/predict" -H "Content-Type: application/json" -d "{\"text\": \"This is a great video!\"}"

4. Example Python Request:

   import requests
   response = requests.post(
       "http://localhost:8000/predict",
       json={"text": "This is a great video!"}
   )
   print(response.json())

Model Files

• All trained models and pipelines are stored in the model/ directory for reproducibility and deployment.
• The main file for inference is model/pipeline.pk1, which contains both preprocessing and the trained model.

---

Note:

• Ensure the pipeline file (model/pipeline.pk1) exists and matches the features expected by the code.

• The API will not work if the pipeline file is missing or if the input features do not match those used

• 🤝 Contributing

1. Fork the repository
2. Create a feature branch (git checkout -b feature/new-analysis)
3. Commit your changes (git commit -am 'Add new analysis feature')
4. Push to the branch (git push origin feature/new-analysis)
5. Create a Pull Request

📝 Future Enhancements

• Implement advanced NLP techniques (TF-IDF, Word2Vec)
• Create interactive dashboards

📄 License

This project is licensed under the MIT License - see the LICENSE file for details.

📧 Contact

For questions or suggestions, please open an issue in the repository.

---

Note: Make sure to have the required datasets in the data/ folder before running the analysis notebook.

---

---