PROBLEM: "I am going home" → Translation

HOW RNN WORKED:
Step 1: Read "I" → Store in brain → Hidden State 1
Step 2: Read "am" + Remember previous → Hidden State 2  
Step 3: Read "going" + Remember previous → Hidden State 3
Step 4: Read "home" + Remember previous → Hidden State 4

FINALLY: Now generate output word by word

Imagine: You want to lift 100 books

RNN WAY:
Lift Book 1 → hold in hand
Lift Book 2 → hold in hand (feeling weight of Book 1)
Lift Book 3 → hold in hand (now weight of 1+2)
...
By Book 50 → Hand starts hurting
By Book 100 → Memory of Book 1 almost gone!

THIS IS THE "VANISHING GRADIENT" PROBLEM! 🎯

TRANSFORMER WAY:
Person 1: Books 1-10 (All together)
Person 2: Books 11-20 (All together)  
...
Person 10: Books 91-100 (All together)

ALL AT ONCE! FAST! 🚀

# Typical scenario in 2016:

Dataset: 10 million sentence pairs (English-Hindi)
Model: LSTM-based Seq2Seq  
Hardware: 8 NVIDIA GPUs (expensive!)
Training time: 2-3 WEEKS! 😱

# If hyperparameter was wrong?
# Wait another 2-3 weeks! 
# Total experiments: 50+
# Total time: 6 months+ just for one model!

SENTENCE: "The cat, which was sitting on the mat that my 
          grandmother bought from the market last week, 
          was sleeping peacefully."

QUESTION: What is "was sleeping" about?

RNN PROBLEM:
- 20 words between "cat" and "was sleeping"
- By the time we reach "was sleeping", memory of "cat" is weak
- Model confused: "market was sleeping"? "grandmother was sleeping"?

INFORMATION LEAKED IN BETWEEN! 💧

WHAT HAPPENED IN BACKPROPAGATION:

Forward Pass:
Word1 → Word2 → Word3 → ... → Word100 → OUTPUT

Backward Pass (Learning):
Word100 ← Word99 ← Word98 ← ... ← Word1

GRADIENT CALCULATION:
- Word100 gradient = 0.9 (strong)
- Word50 gradient = 0.9^50 = 0.005 (weak)
- Word1 gradient = 0.9^100 = 0.0000000003 (almost ZERO!)

RESULT: Early words don't learn! 😢

SENTENCE: "The animal didn't cross the street because it was too tired"

QUESTION: "it" = animal? or street?

HOW SELF-ATTENTION SOLVES IT:

Step 1: Create 3 vectors for each word:
        - Query (Q): "What am I looking for?"
        - Key (K): "What do I have?"
        - Value (V): "My actual information"

Step 2: Compare "it" Query with all Keys:

        "it" ↔ "The"      → Score: 0.05 (low)
        "it" ↔ "animal"   → Score: 0.80 (HIGH! ✓)
        "it" ↔ "didn't"   → Score: 0.02 (low)
        "it" ↔ "cross"    → Score: 0.03 (low)
        "it" ↔ "street"   → Score: 0.08 (low)
        "it" ↔ "tired"    → Score: 0.40 (medium)

Step 3: Create probabilities using Softmax
Step 4: Weighted sum of Values = Final meaning of "it"

RESULT: Model understood "it" = "animal" 🎉

Attention(Q, K, V) = softmax(QK^T / √d_k) × V

BREAKDOWN:
- Q × K^T = Similarity scores (who is like whom?)
- ÷ √d_k = Scale down (keep numbers stable)
- softmax = Convert to probabilities (sum = 1)
- × V = Weighted combination (important info gets more weight)

PROBLEM: 
  Transformer processes all words in parallel
  So "I home" and "home I" would look the same!

SOLUTION: Add position information!

FORMULA (Genius!):
  PE(pos, 2i)   = sin(pos / 10000^(2i/d_model))
  PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model))

WHY SINE/COSINE?
  - Unique "fingerprint" for each position
  - Also captures relative positions (pos 5 - pos 3 = 2)
  - Can extrapolate (works for 1000+ positions!)
  - NO LEARNABLE PARAMETERS! Free! 🎉

VISUAL:
Position 0: [0.0, 1.0, 0.0, 1.0, 0.0, 1.0, ...]
Position 1: [0.8, 0.6, 0.1, 0.9, 0.0, 1.0, ...]
Position 2: [0.9, -0.4, 0.2, 0.8, 0.0, 1.0, ...]
...

Each position has a unique pattern!

MHA:  Q1-K1-V1  Q2-K2-V2  Q3-K3-V3  Q4-K4-V4
GQA:  Q1─┬─K1─V1  Q3─┬─K2─V2
      Q2─┘         Q4─┘
MQA:  Q1─┬
      Q2─┼─K1─V1
      Q3─┼
      Q4─┘

Q: "What's 17 × 23?"

<thinking>
  17 × 23
  = 17 × 20 + 17 × 3
  = 340 + 51
  = 391
  Let me verify: 391 ÷ 17 = 23 ✓
</thinking>
Answer: 391

# ========================================
# OLD WAY: LSTM-based Seq2Seq (2016)
# ========================================
# Library: TensorFlow 1.x or Keras

import tensorflow as tf
from tensorflow.keras.layers import LSTM, Dense, Embedding
from tensorflow.keras.models import Model

class OldSchoolTranslator:
    """
    2016-style translation model
    Problems:
    - Sequential processing (SLOW!)
    - Vanishing gradients
    - Limited context
    - Hard to train
    """
    
    def __init__(self, vocab_size=10000, embedding_dim=256, hidden_dim=512):
        # Encoder: Reads input sequence one word at a time
        self.encoder_embedding = Embedding(vocab_size, embedding_dim)
        self.encoder_lstm = LSTM(
            hidden_dim, 
            return_state=True,  # Need final state for decoder
            return_sequences=True
        )
        
        # Decoder: Generates output one word at a time
        self.decoder_embedding = Embedding(vocab_size, embedding_dim)
        self.decoder_lstm = LSTM(hidden_dim, return_sequences=True)
        self.output_layer = Dense(vocab_size, activation='softmax')
    
    def encode(self, input_sequence):
        """
        Process input ONE WORD AT A TIME
        Word 1 → Word 2 → Word 3 → ... → Final State
        
        PROBLEM: By the time we reach word 100,
                 we've "forgotten" word 1!
        """
        embedded = self.encoder_embedding(input_sequence)
        
        # LSTM processes sequentially - NO PARALLELIZATION!
        # This is the BOTTLENECK
        outputs, state_h, state_c = self.encoder_lstm(embedded)
        
        return outputs, [state_h, state_c]
    
    def decode(self, target_sequence, encoder_states):
        """
        Generate output ONE WORD AT A TIME
        Can't predict word 5 until word 4 is generated!
        """
        embedded = self.decoder_embedding(target_sequence)
        outputs = self.decoder_lstm(embedded, initial_state=encoder_states)
        predictions = self.output_layer(outputs)
        return predictions
    
    def train_step(self, source, target):
        """
        Training was PAINFUL:
        - Gradient vanishing/exploding
        - Teacher forcing required
        - Weeks of training time
        """
        # Forward pass
        encoder_outputs, encoder_states = self.encode(source)
        predictions = self.decode(target, encoder_states)
        
        # Loss calculation
        loss = tf.keras.losses.sparse_categorical_crossentropy(
            target, predictions
        )
        
        return loss

# TRAINING TIME: 2-3 WEEKS on 8 GPUs! 😱
# BLEU SCORE: ~25 (decent but not great)
# CONTEXT: ~50-100 tokens before quality degrades

# ========================================
# NEW WAY: Modern Transformer (2024-2025)
# ========================================
# Libraries: PyTorch + HuggingFace Transformers

import torch
import torch.nn as nn
import torch.nn.functional as F
from transformers import AutoModelForCausalLM, AutoTokenizer

# ==== USING PRE-TRAINED MODEL (RECOMMENDED) ====
class ModernTranslator:
    """
    2024-style using pre-trained LLM
    Just few lines of code!
    """
    
    def __init__(self, model_name="google/gemma-2b"):
        self.tokenizer = AutoTokenizer.from_pretrained(model_name)
        self.model = AutoModelForCausalLM.from_pretrained(
            model_name,
            torch_dtype=torch.bfloat16,  # Memory efficient
            device_map="auto"  # Automatic GPU allocation
        )
    
    def translate(self, text, source_lang="Hindi", target_lang="English"):
        prompt = f"Translate from {source_lang} to {target_lang}: {text}\n\nTranslation:"
        
        inputs = self.tokenizer(prompt, return_tensors="pt").to("cuda")
        
        with torch.no_grad():
            outputs = self.model.generate(
                **inputs,
                max_new_tokens=100,
                temperature=0.7,
                do_sample=True
            )
        
        return self.tokenizer.decode(outputs[0], skip_special_tokens=True)

# THAT'S IT! Pre-trained model does the heavy lifting!


# ==== UNDERSTANDING THE INTERNALS ====
class SimpleTransformerBlock(nn.Module):
    """
    Simplified Transformer block to understand the architecture
    Real implementations are more complex but same principle!
    """
    
    def __init__(self, d_model=512, n_heads=8, d_ff=2048, dropout=0.1):
        super().__init__()
        self.d_model = d_model
        self.n_heads = n_heads
        self.d_k = d_model // n_heads
        
        # Multi-Head Attention components
        self.W_q = nn.Linear(d_model, d_model)  # Query projection
        self.W_k = nn.Linear(d_model, d_model)  # Key projection
        self.W_v = nn.Linear(d_model, d_model)  # Value projection
        self.W_o = nn.Linear(d_model, d_model)  # Output projection
        
        # Feed-Forward Network
        self.ffn = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),  # Modern activation (better than ReLU)
            nn.Linear(d_ff, d_model)
        )
        
        # Layer Normalization (Pre-LN is modern standard)
        self.norm1 = nn.LayerNorm(d_model)
        self.norm2 = nn.LayerNorm(d_model)
        
        self.dropout = nn.Dropout(dropout)
    
    def scaled_dot_product_attention(self, Q, K, V, mask=None):
        """
        THE CORE OF TRANSFORMER!
        
        Attention(Q, K, V) = softmax(QK^T / √d_k) × V
        
        - Q: What am I looking for? (Query)
        - K: What do I have? (Key)
        - V: What's my actual content? (Value)
        """
        # Step 1: Calculate attention scores
        # Q @ K^T gives similarity between each query and all keys
        scores = torch.matmul(Q, K.transpose(-2, -1))
        
        # Step 2: Scale by √d_k (prevents softmax saturation)
        scores = scores / (self.d_k ** 0.5)
        
        # Step 3: Apply mask if needed (for decoder)
        if mask is not None:
            scores = scores.masked_fill(mask == 0, float('-inf'))
        
        # Step 4: Softmax to get attention weights (sum to 1)
        attention_weights = F.softmax(scores, dim=-1)
        
        # Step 5: Weighted sum of values
        output = torch.matmul(attention_weights, V)
        
        return output, attention_weights
    
    def multi_head_attention(self, x, mask=None):
        """
        Multiple attention "heads" looking at different aspects
        
        Head 1: Grammar relationships
        Head 2: Semantic meaning
        Head 3: Entity tracking
        ... etc
        """
        batch_size, seq_len, _ = x.shape
        
        # Project to Q, K, V
        Q = self.W_q(x)  # [batch, seq, d_model]
        K = self.W_k(x)
        V = self.W_v(x)
        
        # Split into multiple heads
        # [batch, seq, d_model] → [batch, n_heads, seq, d_k]
        Q = Q.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        K = K.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        V = V.view(batch_size, seq_len, self.n_heads, self.d_k).transpose(1, 2)
        
        # Apply attention to each head IN PARALLEL!
        # This is why Transformer is fast - all heads computed together
        attn_output, _ = self.scaled_dot_product_attention(Q, K, V, mask)
        
        # Concatenate heads back together
        attn_output = attn_output.transpose(1, 2).contiguous()
        attn_output = attn_output.view(batch_size, seq_len, self.d_model)
        
        # Final linear projection
        output = self.W_o(attn_output)
        
        return output
    
    def forward(self, x, mask=None):
        """
        Full transformer block:
        1. Multi-Head Attention + Residual + LayerNorm
        2. Feed-Forward Network + Residual + LayerNorm
        """
        # Pre-LN Transformer (modern standard)
        # Attention block with residual connection
        normalized = self.norm1(x)
        attention_output = self.multi_head_attention(normalized, mask)
        x = x + self.dropout(attention_output)  # Residual connection
        
        # Feed-forward block with residual connection
        normalized = self.norm2(x)
        ffn_output = self.ffn(normalized)
        x = x + self.dropout(ffn_output)  # Residual connection
        
        return x


# ==== POSITIONAL ENCODING (RoPE - Modern Standard) ====
class RotaryPositionalEmbedding(nn.Module):
    """
    RoPE - Rotary Position Embedding
    Used by: LLaMA, Qwen, DeepSeek, Mistral
    
    Instead of adding position, ROTATE the embeddings!
    Position difference = Angle difference
    """
    
    def __init__(self, d_model, max_seq_len=8192, base=10000):
        super().__init__()
        
        # Precompute rotation frequencies
        inv_freq = 1.0 / (base ** (torch.arange(0, d_model, 2).float() / d_model))
        self.register_buffer('inv_freq', inv_freq)
        
        # Precompute sin/cos for all positions
        positions = torch.arange(max_seq_len).float()
        freqs = torch.einsum('i,j->ij', positions, inv_freq)
        
        # [max_seq_len, d_model/2]
        self.register_buffer('cos_cached', freqs.cos())
        self.register_buffer('sin_cached', freqs.sin())
    
    def forward(self, x, seq_len):
        """
        Apply rotation to embeddings based on position
        """
        cos = self.cos_cached[:seq_len]
        sin = self.sin_cached[:seq_len]
        
        # Split into pairs and rotate
        x1, x2 = x[..., ::2], x[..., 1::2]
        
        # Apply rotation
        rotated = torch.stack([
            x1 * cos - x2 * sin,
            x1 * sin + x2 * cos
        ], dim=-1).flatten(-2)
        
        return rotated


# ==== USING GOOGLE GEMINI API (Easiest!) ====
from google import genai
from google.genai import types

def translate_with_gemini(text: str) -> str:
    """
    Modern way: Just use API!
    No training, no infrastructure needed
    """
    client = genai.Client(api_key="YOUR_API_KEY")
    
    response = client.models.generate_content(
        model="gemini-3-flash-preview",
        contents=f"Translate to English: {text}",
        config=types.GenerateContentConfig(
            thinking_config=types.ThinkingConfig(thinking_level="minimal")
        )
    )
    
    return response.text

# COMPARISON TABLE:
# ┌────────────────┬────────────────┬────────────────┐
# │    Metric      │   Old (2016)   │   New (2024)   │
# ├────────────────┼────────────────┼────────────────┤
# │ Training Time  │   2-3 weeks    │  Already done! │
# │ Code Lines     │    500+        │     10-20      │
# │ Context Length │   50-100       │  1M+ tokens    │
# │ BLEU Score     │    ~25         │    45+         │
# │ GPU Required   │   8 GPUs       │ API call only  │
# └────────────────┴────────────────┴────────────────┘

# ========================================
# MODERN LIBRARIES FOR LLM DEVELOPMENT
# ========================================

# 1. HUGGING FACE TRANSFORMERS (Most Popular)
# pip install transformers accelerate
from transformers import pipeline

# One line to load and use!
translator = pipeline("translation_en_to_de", model="Helsinki-NLP/opus-mt-en-de")
result = translator("Hello, how are you?")

# 2. LANGCHAIN (For LLM Applications)
# pip install langchain langchain-openai
from langchain_openai import ChatOpenAI
from langchain.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4")
prompt = ChatPromptTemplate.from_messages([
    ("system", "You are a helpful translator."),
    ("user", "Translate to Hindi: {text}")
])
chain = prompt | llm
result = chain.invoke({"text": "Hello world"})

# 3. GOOGLE GENAI SDK (For Gemini)
# pip install google-genai
from google import genai
client = genai.Client(api_key="YOUR_KEY")
response = client.models.generate_content(
    model="gemini-3-flash-preview",
    contents="Explain quantum computing"
)

# 4. ANTHROPIC SDK (For Claude)
# pip install anthropic
import anthropic
client = anthropic.Anthropic(api_key="YOUR_KEY")
response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello!"}]
)

# 5. OLLAMA (For Local LLMs)
# Install ollama from ollama.ai
# ollama pull llama3.2
import ollama
response = ollama.chat(
    model='llama3.2',
    messages=[{'role': 'user', 'content': 'Why is the sky blue?'}]
)

# 6. VLLM (For Fast Inference)
# pip install vllm
from vllm import LLM, SamplingParams
llm = LLM(model="meta-llama/Llama-3.1-8B")
outputs = llm.generate(["Hello, my name is"], SamplingParams(temperature=0.8))