Text Summarization

A step-by-step guide to generating a concise summary of a text using an extractive method.

Libraries Used

This pipeline uses a few key Python libraries. Here is a quick overview of their roles:

• nltk: For sentence tokenization and stop word removal.
• sklearn: Provides `TfidfVectorizer` to calculate sentence importance.
• numpy: Used for numerical operations.

1. Data Loading and Preprocessing

• What the code does: It takes a block of text, tokenizes it into sentences, and preprocesses each sentence by removing stop words and punctuation.
• Behind the scenes: Extractive summarization works by identifying the most important sentences in a text and combining them to form a summary. The first step is to break the text down into its constituent sentences so that we can score each one.
• Abstraction: `nltk.sent_tokenize` and `nltk.word_tokenize` abstract the complex process of splitting a continuous text into meaningful sentences and words.

import nltk
from nltk.corpus import stopwords
from nltk.tokenize import sent_tokenize, word_tokenize
from sklearn.feature_extraction.text import TfidfVectorizer
import numpy as np

# Download necessary NLTK data
try:
    nltk.data.find('tokenizers/punkt')
except nltk.downloader.DownloadError:
    nltk.download('punkt')
try:
    nltk.data.find('corpora/stopwords')
except nltk.downloader.DownloadError:
    nltk.download('stopwords')

# Sample long text
text = """
The Amazon rainforest, the world's largest tropical rainforest, is home to an incredible
variety of plant and animal species. It is often referred to as the "Lungs of the Earth"
because it produces more than 20% of the world's oxygen. The rainforest spans across
several countries, with the majority located in Brazil. It is a critical part of the
global ecosystem, playing a vital role in regulating the climate and carbon cycle.
Deforestation, primarily for agriculture and logging, poses a significant threat to
the Amazon and its biodiversity. Efforts to protect the Amazon are crucial for the health
of the planet.
"""

# Tokenize the text into sentences
sentences = sent_tokenize(text)

# Preprocess sentences
stop_words = set(stopwords.words('english'))
def preprocess_sentence(sentence):
    words = word_tokenize(sentence.lower())
    words = [word for word in words if word.isalnum() and word not in stop_words]
    return ' '.join(words)

processed_sentences = [preprocess_sentence(s) for s in sentences]

2. Building the Summarization Model

• What the code does: It uses `TfidfVectorizer` to convert the preprocessed sentences into a numerical representation. It then calculates a score for each sentence based on the sum of its TF-IDF values.
• Behind the scenes: TF-IDF stands for Term Frequency-Inverse Document Frequency. Words that are both frequent in a sentence (high TF) and rare across the entire document (high IDF) are considered more important. By summing the TF-IDF scores for a sentence, we can get a measure of its importance.
• Abstraction: `TfidfVectorizer` abstracts the entire process of counting word frequencies, calculating inverse document frequencies, and converting the text into a numerical matrix.

# Create TF-IDF vectorizer
vectorizer = TfidfVectorizer()
tfidf_matrix = vectorizer.fit_transform(processed_sentences)

# Calculate a score for each sentence by summing its TF-IDF values
sentence_scores = np.array(tfidf_matrix.sum(axis=1)).ravel()

3. Generating the Summary

• What the code does: It ranks the sentences based on their calculated scores and selects the top 'N' sentences to form the summary. The original sentences are then joined together in their original order.
• Behind the scenes: This simple ranking method is the core of extractive summarization. We find the sentences that contain the most unique and relevant information and present them to the user.
• Abstraction: `np.argsort` provides a clean way to get the indices of the sentences in descending order of their scores. The list comprehension and `join` method then abstract the process of rebuilding the summary text.

num_summary_sentences = 2 # Number of sentences in the final summary

# Get the indices of the top-scoring sentences
top_sentence_indices = sentence_scores.argsort()[-num_summary_sentences:][::-1]

# Reconstruct the summary, keeping original sentence order
summary_sentences = sorted([sentences[i] for i in top_sentence_indices], key=lambda s: sentences.index(s))
summary = ' '.join(summary_sentences)

print("--- Original Text ---")
print(text)
print("\n--- Summary ---")
print(summary)