Devices That Keep Information from Disappearing: Residual Connections and Layer Normalization
Transformer''s Transformation: From BERT to ChatGPT
AI dealing with language must understand word order in sentences. Previously Recurrent Neural Networks (RNN) were used — processing words sequentially one by one. However, as sentences grew longer, earlier information faded and parallel processing was impossible. LSTM improved memory but structural limits remained. Transformers solved these problems fundamentally.
The Basic Principle of Attention: Selecting and Focusing on Needed Information
Transformers look at entire sentences at once and calculate relationships between words using attention — focusing on important words, like a student focusing on "exam," "chapter 10," and "chapter 11" when a teacher announces what topics will appear. Originally supplementary to RNN, researchers realized attention alone suffices, leading to the 2017 paper "Attention is All You Need" introducing the transformer structure.
Transformers process information much faster than RNN and maintain context accurately even in long sentences. GPT, BERT, and T5 are all based on transformers.
How Attention Selects Important Words: Similarity Calculation
Each word simultaneously has three roles: Query (what information am I looking for?), Key (what information do I have?), and Value (the actual information). Attention compares Query with all Keys using cosine similarity — measuring directional similarity between vectors. For "I drank coffee this morning," the word "drank" (Query) finds "coffee" (Key) as most similar, bringing its meaning (Value) to understand the sentence.
Multi-Head Attention: Reading Sentences from Diverse Perspectives
Understanding language requires multiple perspectives simultaneously — not just what was drunk but when, who, and how. Multi-Head Attention runs multiple attention calculations in parallel, each examining different aspects of word relationships. One head focuses on verb-object relationships, another on temporal expressions, another on subject identification. Together they provide a three-dimensional understanding of sentences. This enables accurate handling of long sentences, complex sentences, and ambiguous expressions.
Conveying Word Order: Position Encoding
Transformers process all words simultaneously, so they don''t inherently remember word order. Position Encoding adds position information using sine and cosine waves. These provide stable values always between -1 and 1, clearly distinguish positions, and have regular repeating patterns helping models learn positions. This enables transformers to consider both word meaning and precise position together.
Residual Connections and Layer Normalization
Stacking many layers risks important information fading or being transformed too much. Residual Connections (skip connections) add each layer''s input to its output — binding previous and new information together so models can learn which is important. Layer Normalization adjusts output values within appropriate ranges to prevent values from becoming too large (unstable) or too small (no learning). Together they ensure important information is stably maintained across deep networks.
Mixture of Experts: Improving Transformer Performance
As transformers grow larger, computation becomes expensive. Mixture of Experts (MoE) maintains multiple specialist sub-models and routes each input to only the most suitable specialist — like a hospital routing patients to appropriate specialists. This dramatically reduces computation while improving performance. Only a subset of experts (e.g., 2 of 64) are activated for each input, making large models computationally tractable.
Transformer''s Transformation: From BERT to ChatGPT
BERT focuses on deeply understanding sentences by simultaneously checking front and back context, excelling at question answering, subject finding, and named entity recognition. GPT focuses on naturally continuing sentences, reading left to right and predicting the next word — suited for creative writing and conversation.
GPT-1 (2018) established transformer-based language modeling. GPT-2 (10× larger) enabled more natural and consistent longer text. GPT-3 (175 billion parameters, 2020) demonstrated remarkable zero-shot capabilities across translation, summarization, question answering, and even math — without task-specific training.
InstructGPT solved GPT-3''s accuracy and safety issues by training on human preference comparisons. ChatGPT built on InstructGPT to enable natural multi-turn conversation — shifting from solo writer to conversational partner.
Multimodal AI: Understanding Both Speech and Pictures
Transformers expanded beyond language. DALL·E creates images from text descriptions by treating images as sequences of small pieces like words. L-Verse enables bidirectional translation between text and images — generating text from images or images from text, like an interpreter working in both directions. Flamingo can look at images and answer questions or connect different data types. These multimodal models are broadening AI capabilities across text, images, and video.
From Understanding to Expression
Transformers completely changed how AI views and understands the world — from following words sequentially to simultaneously looking at entire sentences and focusing on what is important. The core insight is simple: "let AI find what is important and focus." This one idea opened the way for AI to go from understanding language to creating pictures, deeply grasping human intent, and expressing it richly. Going forward transformers will continue expanding into even more fields beyond language and images.
