Language Translation

Language Translation

Language Translation

A step-by-step guide to building a simple sequence-to-sequence (seq2seq) model for language translation.

Libraries Used

This pipeline uses a few key Python libraries. Here is a quick overview of their roles:

  • tensorflow: The core deep learning framework for building the seq2seq model.
  • numpy: Used for numerical operations.

1. Data Loading and Preprocessing

  • What the code does: It uses a small, hand-crafted dataset of English-to-French phrases. It tokenizes the text, creates a vocabulary for both languages, and pads the sequences to a uniform length.
  • Behind the scenes: A seq2seq model requires the input (English) and output (French) sentences to be converted into numerical sequences. We also add `start` and `end` tokens to the target sequences to signal the beginning and end of a translation to the model.
  • Abstraction: We define a `Tokenizer` class to abstract the process of building the vocabulary, converting text to integers, and padding sequences. This makes the data preparation process reusable and clean. 
import numpy as np
import tensorflow as tf
from tensorflow.keras.layers import Embedding, GRU, Dense
from tensorflow.keras.models import Model
from tensorflow.keras.optimizers import Adam
import re

# Simple English-to-French parallel corpus
corpus = [
    ("hello world", "bonjour le monde"),
    ("how are you", "comment allez-vous"),
    ("i am fine", "je vais bien"),
    ("thank you", "merci"),
    ("good morning", "bonjour"),
]

# Preprocessing function
def preprocess_sentence(sentence):
    sentence = sentence.lower().strip()
    sentence = re.sub(r"([?.!,])", r" \1", sentence)
    sentence = re.sub(r'[" "]+', " ", sentence)
    sentence = re.sub(r"[^a-zA-Z?.!,']+", " ", sentence)
    sentence = sentence.strip()
    return sentence

# Tokenizer class
class Tokenizer:
    def __init__(self, texts):
        self.texts = texts
        self.word_to_idx = {}
        self.idx_to_word = {}
        self.vocab_size = 0
        self.max_len = 0
        self.build_vocab()

    def build_vocab(self):
        words = set()
        for text in self.texts:
            words.update(text.split())
        self.word_to_idx = {word: i+1 for i, word in enumerate(sorted(list(words)))}
        self.word_to_idx[''] = 0
        self.idx_to_word = {i: word for word, i in self.word_to_idx.items()}
        self.vocab_size = len(self.word_to_idx)
        self.max_len = max(len(text.split()) for text in self.texts)
    
    def texts_to_sequences(self, texts):
        sequences = [[self.word_to_idx[word] for word in text.split()] for text in texts]
        return sequences

# Prepare data
input_texts = [preprocess_sentence(pair[0]) for pair in corpus]
target_texts = [f" {preprocess_sentence(pair[1])} " for pair in corpus]

input_tokenizer = Tokenizer(input_texts)
target_tokenizer = Tokenizer(target_texts)

input_sequences = input_tokenizer.texts_to_sequences(input_texts)
target_sequences = target_tokenizer.texts_to_sequences(target_texts)

input_padded = tf.keras.preprocessing.sequence.pad_sequences(input_sequences, maxlen=input_tokenizer.max_len, padding='post')
target_padded = tf.keras.preprocessing.sequence.pad_sequences(target_sequences, maxlen=target_tokenizer.max_len, padding='post')

print(f"Shape of padded input: {input_padded.shape}")
print(f"Shape of padded target: {target_padded.shape}")

2. Building the Seq2Seq Model

  • What the code does: It defines an encoder-decoder model. The encoder processes the input sequence, and the decoder generates the output sequence one word at a time.
  • Behind the scenes: The encoder, a `GRU` layer, processes the entire input sequence and compresses its information into a single "context vector." The decoder then uses this context vector as its initial state to generate the output sequence.
  • Abstraction: The `Encoder` and `Decoder` models are built separately but work together to form the complete `seq2seq` model. This modular design abstracts the complex flow of information from the source to the target language. 
# Encoder model
encoder_inputs = tf.keras.layers.Input(shape=(input_tokenizer.max_len,))
encoder_embedding = Embedding(input_tokenizer.vocab_size, 256)(encoder_inputs)
encoder_gru, state_h = GRU(256, return_state=True)(encoder_embedding)
encoder_states = [state_h]

encoder_model = Model(encoder_inputs, encoder_states)
encoder_model.summary()

# Decoder model
decoder_inputs = tf.keras.layers.Input(shape=(None,))
decoder_embedding = Embedding(target_tokenizer.vocab_size, 256)(decoder_inputs)
decoder_gru = GRU(256, return_sequences=True, return_state=True)
decoder_outputs, _ = decoder_gru(decoder_embedding, initial_state=encoder_states)
decoder_dense = Dense(target_tokenizer.vocab_size, activation='softmax')
decoder_outputs = decoder_dense(decoder_outputs)

model = Model([encoder_inputs, decoder_inputs], decoder_outputs)
model.summary()

3. Training and Translation

  • What the code does: It compiles the model, trains it on the parallel corpus, and provides a function to make new translations by feeding the input phrase to the encoder and then iteratively predicting the next word with the decoder.
  • Behind the scenes: During training, the decoder is fed the correct target sequence one word at a time (teacher forcing). During inference, the decoder is fed its own previous output as the next input, creating an autoregressive loop.
  • Abstraction: The `model.compile` and `model.fit` methods handle the training. The custom `translate_sentence` function abstracts the complex inference loop, which involves a multi-step process of getting the encoder state, predicting a word, and feeding it back into the decoder. 
model.compile(optimizer='adam', loss='sparse_categorical_crossentropy')

# Prepare target data for training (shifted by one step)
target_padded_train = np.expand_dims(target_padded[:, 1:], axis=-1)

# Train the model
model.fit([input_padded, target_padded[:, :-1]], target_padded_train, epochs=20)

# Inference function
def translate_sentence(input_sentence):
    input_seq = input_tokenizer.texts_to_sequences([preprocess_sentence(input_sentence)])
    input_seq_padded = tf.keras.preprocessing.sequence.pad_sequences(input_seq, maxlen=input_tokenizer.max_len, padding='post')
    
    # Get encoder state
    states_value = encoder_model.predict(input_seq_padded)
    
    # Initialize decoder input with start token
    target_seq = np.zeros((1, 1))
    target_seq[0, 0] = target_tokenizer.word_to_idx['']
    
    translated_sentence = []
    
    # Inference loop
    for _ in range(target_tokenizer.max_len):
        decoder_outputs, h = decoder_gru(decoder_embedding(target_seq), initial_state=states_value)
        output_tokens = decoder_dense(decoder_outputs)
        predicted_token = np.argmax(output_tokens[0, -1, :])
        
        if target_tokenizer.idx_to_word[predicted_token] == '':
            break
        
        translated_sentence.append(target_tokenizer.idx_to_word[predicted_token])
        target_seq = np.zeros((1, 1))
        target_seq[0, 0] = predicted_token
        states_value = h
        
    return ' '.join(translated_sentence)

# Test the translation
print("\nTranslation of 'hello world':")
print(translate_sentence('hello world'))
print("\nTranslation of 'i am fine':")
print(translate_sentence('i am fine'))

Comments

Post a Comment

Popular posts from this blog

Named Entity Recognition (NER)

Named Entity Recognition (NER) Named Entity Recognition (NER) A step-by-step guide to developing a model that can identify and classify named entities in text. Libraries Used This pipeline uses a few key Python libraries. Here is a quick overview of their roles: nltk: The Natural Language Toolkit, used for tokenization and its built-in NER functionality. 1. Data Loading and Preprocessing What the code does: It takes a sample block of text and performs two crucial preprocessing steps: word tokenization (splitting the text into words) and part-of-speech (POS) tagging (labeling each word with its grammatical function). ...
Read more

Speech-to-Text with a Simple Model

Speech-to-Text with a Simple Model Speech-to-Text with a Simple Model A step-by-step guide to converting spoken words into text using the SpeechRecognition library. Libraries Used This pipeline uses a few key Python libraries. Here is a quick overview of their roles: SpeechRecognition: The core library for performing speech recognition. It acts as an interface for various speech recognition engines. pydub: Used for handling and playing audio files. os: For handling file system operations. 1. Audio File Setup What the code does: It creates a temp...
Read more

Keyword Extractor

Keyword Extractor Keyword Extractor A step-by-step guide to automatically identifying the most important keywords from a document using TF-IDF . Libraries Used This pipeline uses a few key Python libraries. Here is a quick overview of their roles: sklearn: Provides `TfidfVectorizer` for keyword scoring. nltk: Used for text preprocessing like stop word removal. 1. Data Loading and Preprocessing What the code does: It takes a single document and preprocesses it by converting it to lowercase and removing punctuation. Behind the scenes: S...
Read more