To talk about AI is to shoot at a target that appears to be not only moving but accelerating.
How does a Robot, a Droid, or an Autopilot car talk with us? What's the cutting edge technology help Artificial Intelligence Machines to speak, listen, write and read?
The Transformer is a neural network architecture based on attention mechanisms.
To understand its design concept, architecture, and working principles, we will examine two tasks: text translation and image captioning.
Task 1: Text Translation
English: Peach looks absolutely radiant.
Chinese: Peach 看起来光彩照人。
Task 2: Image Captioning
Peach is wearing a black face mask and dark blue overalls.
The Transformer was introduced in the 2017 paper Attention Is All You Need. Unlike traditional recurrent neural networks (RNNs), which process sequences sequentially, the Transformer uses attention mechanisms to establish direct connections between information at different positions.
In text translation, the model needs to understand not only the individual meanings of “Peach,” “looks,” “absolutely,” and “radiant,” but also how “absolutely” intensifies “radiant” and what state the sentence as a whole describes.
In image captioning, the model needs to observe Peach’s hat, glasses, black face mask, and dark blue overalls, and understand that these visual features belong to the same person, allowing it to generate an accurate description.
Although one task processes language and the other processes images, both require connections between different pieces of information.
The Transformer uses attention mechanisms to establish dependencies between information, enabling contextual understanding and sequence prediction.
The classic Transformer uses an encoder–decoder architecture.
The encoder interprets the input information, while the decoder uses the encoder’s output and previously generated tokens to predict new tokens step by step.
In text translation, the encoder receives “Peach looks absolutely radiant.” and produces contextual representations of the English sentence. The decoder then uses this information to generate the Chinese translation, “Peach 看起来光彩照人。”
The decoder receives its input from different sources during training and inference, but the same principle applies in both cases: when predicting the next token, it can use only preceding target tokens and cannot see the correct answer it is currently supposed to predict.
Consider the following example:
English: Peach looks absolutely radiant.
Correct translation: Peach 看起来光彩照人。
During training, the correct translation is already known. To teach the Transformer to translate, a method called teacher forcing is used: the correct target sequence is shifted one position to the right and supplied as the decoder’s input.
During inference, the decoder does not know the correct translation.
For example, after training is complete, a user enters an English sentence: “Peach looks absolutely radiant.” No Chinese translation is provided. The decoder must start with BOS and generate the translation step by step using the encoder’s output. Each predicted token is appended to the decoder’s input sequence before the next token is predicted. This continues until the decoder generates EOS.
The important distinction is that during training, the decoder uses preceding correct target tokens, whereas during inference, it uses tokens previously generated by the model itself.
In image captioning, a visual encoding module first extracts features from Peach’s image. The Transformer encoder then establishes relationships between visual regions. Using this visual context, the decoder generates “Peach is wearing a black face mask and dark blue overalls.”
A Transformer typically contains multiple encoder layers and multiple decoder layers. Each layer processes features through attention mechanisms, feed-forward neural networks, residual connections, and normalization.
The Transformer cannot directly process natural-language text or raw images. The input must first be converted into numerical representations.
In text translation, tokenization splits an English sentence into tokens, for example:
Peach looks absolutely radiant .
These tokens are converted into integer IDs and then mapped to fixed-dimensional vectors through an input embedding layer. In practice, models may use subword tokenization, so a word does not necessarily correspond to a single token.
In image captioning, visual encoding methods such as the Vision Transformer (ViT) can divide Peach’s image into image patches and convert them into visual tokens through linear projection. Alternatively, a visual encoder such as a CNN can extract image features, which are then mapped into inputs for the Transformer.
For example, some visual tokens may contain information about the face mask region, while others may contain features of the hat, hair, or overalls. Each token is a learned numerical representation; it does not directly store a text label such as “face mask.”
Text is converted into vectors through tokenization and embedding, while images are converted into visual features through visual encoding. Both ultimately become input representations that the Transformer can process.
Input embeddings provide content features, but self-attention does not inherently know the order or spatial positions of tokens.
In text translation, the word order in “Peach looks absolutely radiant.” affects both grammar and meaning. Positional encoding helps the model distinguish information such as “Peach” appearing at the beginning of the sentence and “absolutely” appearing before “radiant.”
Spatial positions are equally important in image captioning. The black face mask is located on Peach’s face, while the dark blue overalls are located on the body. Without spatial position information, the model would struggle to fully understand the arrangement of these visual features.
The classic Transformer uses sine and cosine functions to generate sequence positional encodings. Vision models may use two-dimensional positional encodings or other representations of spatial position.
Embeddings represent the content of information, while positional encodings represent where that information is located.
Self-attention is the core of the Transformer. It allows each token to obtain information from other tokens in the same sequence according to their relevance.
Each token undergoes three learnable linear transformations to produce:
In text translation, when the model processes “radiant,” its query can be matched against the keys of “Peach,” “looks,” and “absolutely.” The model calculates attention weights from the matching scores and then computes a weighted aggregation of the corresponding values. This allows the updated representation of “radiant” to incorporate relevant context.
In image captioning, suppose a visual token contains features of the black face mask. It can establish relationships with tokens representing the face, hair, and other image regions. Through this exchange of information, the model can learn the contextual relationship between the mask and the person’s face.
Multi-head attention extends this process by using multiple attention heads to learn relationships in parallel across different feature subspaces.
For example, different attention heads in text processing may show different sensitivities to modifier relationships and syntactic relationships. In image processing, different heads may attend to local appearance, spatial layout, or relationships across regions.
These relationships are learned automatically during training. Each attention head is not manually assigned a fixed function.
Self-attention allows tokens to exchange contextual information, while multi-head attention enables the model to learn relationships from different feature perspectives.
After computing attention, the Transformer processes the output through Add and Norm.
Add refers to a residual connection, which adds a sublayer’s input to its output. Norm refers to layer normalization, which normalizes a token’s features and applies learnable adjustments.
In text translation, residual connections help preserve the original word features while incorporating contextual information obtained through attention.
In image captioning, residual connections similarly help preserve the original features of visual tokens while combining them with new information obtained through interactions across regions.
Layer normalization helps maintain stable training in deep networks for both tasks.
The original Transformer uses a Post-LN structure, in which residual addition is performed before layer normalization.
Residual connections improve the flow of information and gradients, while layer normalization helps stabilize network training.
Self-attention handles information exchange between tokens, while the feed-forward network (FFN) applies further nonlinear transformations to each token’s features.
In text translation, after attention is computed, the representation of “radiant” already incorporates information related to other words. The FFN further transforms this feature representation.
In image captioning, a visual token containing information about the black face mask also passes through the FFN, allowing its features to contribute further to subsequent visual semantic modeling.
The FFN typically operates independently on each token’s feature dimensions. It does not directly exchange information between tokens.
Self-attention establishes relationships between information, while the feed-forward network performs nonlinear feature transformations.
A classic encoder layer consists of multi-head self-attention, a feed-forward network, and the corresponding Add and Norm operations.
The Transformer stacks multiple encoder layers so that each layer can further integrate contextual information based on the preceding layer’s output.
In text translation, the representations of “Peach,” “looks,” “absolutely,” and “radiant” pass through multiple layers, gradually becoming features that incorporate the context of the entire sentence.
In image captioning, features from visual regions such as the hat, black face mask, glasses, and dark blue overalls also interact through multiple attention layers, forming representations that incorporate the overall image context.
The encoder ultimately outputs a set of contextual features for the decoder to use.
The encoder ultimately outputs a set of contextual features for the decoder to use.
Through multiple layers of attention and feature transformation, the encoder gradually converts local token representations into representations containing contextual information.
After the encoder processes the input, the decoder generates the target sequence.
During training, the correct target sequence is shifted to the right and supplied to the decoder. A causal mask prevents the current position from accessing future target tokens.
In text translation, the target sentence is “Peach 看起来光彩照人。” When the model predicts “光彩照人,” it can use the preceding “Peach” and “看起来,” but it cannot read the correct answer it is currently supposed to predict.
In image captioning, the target sentence is “Peach is wearing a black face mask and dark blue overalls.” When the model is about to predict “mask,” it can use the preceding text, “Peach is wearing a black face,” but it cannot access target tokens such as “mask” or “overalls” in advance.
However, the decoder can still access the complete encoder output through cross-attention, including the encoded features of the entire English source sentence or the entire image.
Causal masking restricts access to future target tokens, allowing the decoder to learn to generate output step by step.
The decoder needs to refer not only to previously generated content but also to the input information provided by the encoder.
The classic Transformer therefore introduces cross-attention.
In cross-attention, the queries come from the decoder’s current hidden representations, while the keys and values come from the encoder’s final output.
In text translation, when the decoder is about to generate “光彩照人,” it can use its queries to aggregate information related to “radiant,” “absolutely,” and their context from the encoded English representations.
In image captioning, when the decoder is about to generate “black face mask,” it can use its queries to aggregate visual information related to the mask region and its context from the visual representations.
Path 1: The decoder’s initial input
This consists of preceding tokens in the target sequence.
• During training, the decoder uses the correct target sequence shifted to the right.
• During inference, it uses BOS and tokens previously generated by the model.
Path 2: The encoder’s output
The encoder processes the complete input sequence and produces contextual features. These features are supplied to the decoder through cross-attention; they do not directly replace the decoder’s initial input.
The decoder takes preceding target tokens as its initial input. It uses masked self-attention to understand the content generated so far, then introduces the encoder’s input context through cross-attention, and finally predicts the next token.
In other words, the encoder’s output does not first enter the decoder and then get integrated a second time. Cross-attention itself is the key channel through which the encoder’s output enters the decoder and participates in its computation.
Self-attention establishes relationships within the same sequence, while cross-attention establishes relationships between the output sequence and the input sequence.
After the decoder processes the information, the model obtains hidden features for the current target position.
A linear layer maps these features to scores, called logits, for candidate tokens in the target vocabulary. Softmax then converts these scores into a probability distribution.
In text translation, the model may predict “光彩照人” after “Peach 看起来.” Suppose the next-token probability distribution is:
In image captioning, the model may predict “mask” after “Peach is wearing a black face.” Suppose the next-token probability distribution is:
The model selects subsequent tokens according to the probability distribution and continues generating until it outputs an end-of-sequence token.
The linear layer computes candidate token scores, and Softmax converts them into prediction probabilities.
The Transformer’s embedding vectors, attention projections, and feed-forward network parameters must all be learned through training.
In text translation, the training data consists of English sentences paired with their Chinese translations. For example, the model learns to generate “Peach 看起来光彩照人。” from “Peach looks absolutely radiant.”
In image captioning, the training data consists of images paired with descriptive text. For example, the model learns to generate an English description from an image of Peach During training, the model predicts target tokens under the constraints of the causal mask. Cross-entropy loss measures the difference between the predicted probabilities and the correct answers.
Gradients are then calculated through backpropagation, and an optimizer updates the network parameters.
The training objectives of the two tasks are similar: both learn to predict the next target token given the input context and preceding target tokens.
By repeatedly learning from correct examples and prediction errors, the model optimizes its parameters and learns the mapping between inputs and target sequences.
After training, the Transformer can generate results for new inputs.
In text translation, the encoder first processes the complete English sentence. The decoder then starts with a beginning-of-sequence token and generates the Chinese translation step by step.
In image captioning, the visual encoding module first processes Peach’s image. The decoder then uses the visual context to generate an English description step by step.
During training, the model can use the correct target sequence shifted to the right to calculate prediction losses for multiple positions in parallel. During inference, it must predict tokens sequentially based on the tokens it has actually generated.
The core idea is that training teaches the model patterns for prediction, while inference uses those learned patterns to generate new sequences.
The Transformer’s main advantages are its ability to establish direct connections between information through attention mechanisms and its efficient use of parallel computation during training.
In text translation, these capabilities allow the model to learn dependencies between distant words and generate natural translations based on context.
In image captioning, attention interactions between visual tokens help the model relate different image regions. Cross-attention then makes this visual information available for text generation.
However, the computational cost and attention-matrix storage requirements of standard full attention generally grow quadratically with sequence length. For example, longer text sequences increase the cost of attention computation. High-resolution images divided into large numbers of visual tokens can also substantially increase the computational workload.
In addition, when either task uses autoregressive decoding, generation generally proceeds one token at a time. Output length therefore also affects inference latency.
The Transformer has strong contextual modeling capabilities, but it requires a balance between model size, token count, and computational efficiency.
Let us review the Transformer’s overall workflow.
For text translation, the model converts “Peach looks absolutely radiant.” into token vectors, uses the encoder to understand relationships between words, and then uses the decoder to generate “Peach 看起来光彩照人。”
For image captioning, the model converts a full-body image of Peach into visual tokens, uses the encoder to establish contextual relationships between regions such as the hat, face mask, glasses, and overalls, and then uses the decoder to generate “Peach is wearing a black face mask and dark blue overalls.”
Although the two tasks have different inputs, they follow a similar computational process:
Input encoding → Positional representation → Self-attention → Contextual features → Cross-attention → Autoregressive output
The core idea of the Transformer is to use attention mechanisms to establish dependencies between information, learn contextual representations through multiple neural network layers, and then use these representations to perform sequence prediction.
From text translation to image captioning, the Transformer demonstrates the ability to convert different types of information into feature representations and model them through a unified attention mechanism. This also lays the foundation for subsequent research into robotic learning models that combine language, vision, and action.