Keyword Extractor

A step-by-step guide to automatically identifying the most important keywords from a document using TF-IDF.

Libraries Used

This pipeline uses a few key Python libraries. Here is a quick overview of their roles:

• sklearn: Provides `TfidfVectorizer` for keyword scoring.
• nltk: Used for text preprocessing like stop word removal.

1. Data Loading and Preprocessing

• What the code does: It takes a single document and preprocesses it by converting it to lowercase and removing punctuation.
• Behind the scenes: Similar to other NLP tasks, cleaning the text is essential. By converting all text to lowercase and removing irrelevant characters, we ensure that the model doesn't treat "Keyword" and "keyword" as two different words.
• Abstraction: We use a simple function to abstract the entire text cleaning process, making the code clean and reusable.

import nltk
from nltk.corpus import stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

# Download necessary NLTK data
try:
    nltk.data.find('corpora/stopwords')
except nltk.downloader.DownloadError:
    nltk.download('stopwords')

# Sample document for keyword extraction
document = """
The field of natural language processing (NLP) is a branch of artificial intelligence.
It focuses on the interaction between computers and human language, allowing machines
to understand, interpret, and generate human language. NLP is a fascinating area of study.
It has many applications, including sentiment analysis, machine translation, and chatbots.
"""

def preprocess_text(text):
    # Convert to lowercase and remove punctuation
    text = text.lower().replace('.', '').replace(',', '')
    return text

processed_document = preprocess_text(document)
print("Processed document:")
print(processed_document)

2. Calculating TF-IDF Scores

• What the code does: It uses `TfidfVectorizer` to calculate the TF-IDF score for each word in the document.
• Behind the scenes: TF-IDF scores a word based on its importance within a document relative to a corpus of documents. Words that are frequent in the document but rare in the corpus (e.g., 'NLP', 'language') get a high score. Words like 'the' and 'a' get a low score because they are common everywhere.
• Abstraction: `TfidfVectorizer` abstracts the entire TF-IDF calculation. It automatically handles tokenization, counting, and the mathematical formulas for both TF and IDF.

# Create a TF-IDF vectorizer
# Note: We need a small "corpus" for the IDF calculation.
# We'll just use the document itself as the corpus for this simple example.
tfidf_vectorizer = TfidfVectorizer(stop_words='english')
tfidf_matrix = tfidf_vectorizer.fit_transform([processed_document])

# Get the feature names (words) and their TF-IDF scores
feature_names = tfidf_vectorizer.get_feature_names_out()
tfidf_scores = tfidf_matrix.toarray()[0]

# Combine words and scores into a dictionary
word_scores = {word: score for word, score in zip(feature_names, tfidf_scores)}

print("\nTF-IDF scores for each word:")
for word, score in sorted(word_scores.items(), key=lambda item: item[1], reverse=True):
    print(f"{word}: {score:.4f}")

3. Extracting Top Keywords

• What the code does: It sorts the words by their TF-IDF scores in descending order and extracts the top 'N' keywords.
• Behind the scenes: The highest-scoring words are the most important words in the document, according to the TF-IDF algorithm. These words are the best candidates for representing the document's content.
• Abstraction: The `sorted` function with a `lambda` key provides a clean and elegant way to sort the dictionary of word scores and extract the top keywords without manual looping.

num_keywords = 5
# Sort the words by score and get the top N
top_keywords = sorted(word_scores.items(), key=lambda item: item[1], reverse=True)[:num_keywords]

print(f"\nTop {num_keywords} keywords:")
for keyword, score in top_keywords:
    print(f"- {keyword} (Score: {score:.4f})")