T5Gemma: A new collection of encoder-decoder Gemma models

T5Gemma is a new family of encoder-decoder LLMs developed by converting and adapting pretrained decoder-only models based on the Gemma 2 framework, offering superior performance and efficiency compared to its decoder-only counterparts, particularly for tasks requiring deep input understanding, like summarization and translation.

Google logo
Google
•
Biao Zhang, Paul Suganthan, Ben Hora
•5 min read•intermediate•
--
•View Original
Hugging FaceT5TransformerVertex AI

Overview

The article introduces T5Gemma, a new collection of encoder-decoder models derived from pretrained decoder-only models. It highlights the advantages of the encoder-decoder architecture in various applications and presents performance comparisons with existing models, showcasing significant improvements in quality and efficiency.

What You'll Learn

1

How to adapt pretrained decoder-only models into encoder-decoder architectures

2

Why encoder-decoder models can outperform decoder-only models in specific tasks

3

How to utilize T5Gemma checkpoints for research and development

Key Questions Answered

How does T5Gemma improve upon existing decoder-only models?
T5Gemma improves upon existing decoder-only models by utilizing a technique called model adaptation, which initializes encoder-decoder models with weights from pretrained decoder-only models. This allows for enhanced performance in tasks requiring complex reasoning, as demonstrated by significant score improvements on benchmarks like GSM8K and DROP.
What are the performance benefits of using T5Gemma models?
T5Gemma models achieve comparable or better performance than their decoder-only counterparts, leading the quality-inference efficiency frontier across benchmarks like SuperGLUE. For instance, T5Gemma 9B-9B shows higher accuracy than Gemma 2 9B with similar latency, demonstrating effective quality-efficiency trade-offs.
What types of models are included in the T5Gemma release?
The T5Gemma release includes multiple sizes of T5-sized models (Small, Base, Large, XL) and Gemma 2-based models (2B and 9B), along with pretrained and instruction-tuned variants. This variety allows users to explore different configurations and training objectives.

Key Statistics & Figures

GSM8K score improvement
9 points higher
T5Gemma 9B-9B outperforms the original Gemma 2 9B model on the GSM8K benchmark.
MMLU score increase
12 points
T5Gemma 2B-2B IT sees a significant jump in its MMLU score compared to Gemma 2 2B.
GSM8K score after instruction tuning
70.7%
The GSM8K score for T5Gemma 2B-2B IT increases from 58.0% to 70.7%.

Technologies & Tools

Model Architecture
T5gemma
Used for developing encoder-decoder models based on pretrained decoder-only architectures.
Model Architecture
Gemma 2
Serves as the base for adapting models into the T5Gemma framework.

Key Actionable Insights

1
Utilize the flexible model adaptation technique to create custom encoder-decoder configurations tailored to specific tasks.
By experimenting with different encoder and decoder sizes, such as pairing a large encoder with a smaller decoder, you can optimize models for tasks like summarization where understanding input is crucial.
2
Leverage the released T5Gemma checkpoints to accelerate your research and development efforts.
These checkpoints provide a valuable resource for exploring model architectures and performance, enabling you to build upon existing work and innovate in the field of large language models.

Common Pitfalls

1
Assuming that decoder-only models are always superior for all tasks.
This misconception can lead to missed opportunities for leveraging encoder-decoder architectures, which may provide better performance in tasks requiring deeper understanding and reasoning.

Related Concepts

Model Adaptation Techniques
Encoder-decoder Architecture Advantages
Performance Benchmarking In Llms