Tech Corner

Decoding Text-to-Text Transformers

Anwesh Roy
Jul 20
2-3 mins

Table of contents

Automate your business at $5/day with Engati


The progress in NLP took a great leap with the introduction of Transformer architecture.

Based on NLP's successful architecture, language models such as BERT have achieved state-of-the-art results in various NLP tasks. 

The big idea 

Here's the central idea behind these language models. It's to train a massive corpus in an unsupervised manner to learn language structure, grammar, and semantics.

These massive pre-trained models can then be used as an encoder to generate contextual and semantic representations of text. By using transfer learning, several downstream NLP tasks can be performed with ease. Tasks such as text classification, sentiment analysis, question answering, and summarizing.

Text-to-Text Transformers

Recently Google has made a significant advancement in this area by releasing a new model, Text-To-Text Transformer or T5. 

How text-to-text transformers work?

T5 reframes all NLP tasks into a unified text-to-text format where the input and output of the model is text. It takes text input from various NLP tasks and predicts text output for the respective task as shown below:

how text-to-text transformers work engati

Every task considered uses text as input to the model, which is trained to generate some target text. 

This allows the same model, loss function, and hyper-parameters across diverse sets of tasks, including translation (green), linguistic acceptability (red), sentence similarity (yellow), and document summarization (blue).

The model was trained on Colossal Clean Crawled Corpus (C4) dataset. Which is a cleaned version of Common Crawl and is two orders of magnitude larger than Wikipedia.

The largest model has 11 billion parameters and achieved state-of-the-art results on the GLUE, SuperGLUE, SQuAD, and CNN/Daily Mail benchmarks.

The pre-trained model can be used as is without any further fine tuning for NLP/NLU tasks such as sentiment analysis, NER, Retail POS, Question Answering, Translation, and Summarization.

Anwesh Roy

Anwesh is the Senior Vice President of Engati. Driven by a passion to deliver value through AI-driven solutions, Anwesh is on a mission to mainstream Natural Language Processing (NLP), Natural Language Understanding (NLU), Natural Language Generation (NLG) and Data Analytics applications.

Close Icon
Request a Demo!
Get started on Engati with the help of a personalised demo.
Thanks for the information.
We will be shortly getting in touch with you.
Oops! something went wrong!
For any query reach out to us on
Close Icon
Congratulations! Your demo is recorded.

Select an option on how Engati can help you.

I am looking for a conversational AI engagement solution for the web and other channels.

I would like for a conversational AI engagement solution for WhatsApp as the primary channel

I am an e-commerce store with Shopify. I am looking for a conversational AI engagement solution for my business

I am looking to partner with Engati to build conversational AI solutions for other businesses

Close Icon
You're a step away from building your Al chatbot

How many customers do you expect to engage in a month?

Less Than 2000


More than 5000

Close Icon
Thanks for the information.

We will be shortly getting in touch with you.

Close Icon

Contact Us

Please fill in your details and we will contact you shortly.

Thanks for the information.
We will be shortly getting in touch with you.
Oops! Looks like there is a problem.
Never mind, drop us a mail at