Megatron NLG, ERNIE, and BLOOM: A look at GPT-3-like Large Language Models

OpenAI’s AI chatbot ChatGPT is built on top of the GPT-3.5 language model (Express photo)

What are LLMs (Large Language Models)?

Large language models use deep learning techniques to process large amounts of text. They work by processing vast amounts of text, understanding the structure and meaning, and learning from it. LLMs are ‘trained’ to identify meanings and relationships between words. The greater the amount of training data a model is fed, the smarter it gets at understanding and producing text.

The training data is usually large datasets, such as Wikipedia, OpenWebText, and the Common Crawl Corpus. These contain large amounts of text data, which the models use to understand and generate natural language.

GPT-3

Generative Pre-trained Transformer 3 (GPT-3) is a language model that uses deep learning to generate human-like text. Introduced by Open AI earlier in May 2020 as a successor to GPT-2, the model can also generate code, stories, poems, and a lot more. The model gained widespread attention following the release of ChatGPT in November and it also forms the foundation for the image-generative model Dall-E. It boats 175 Billion trainable parameters.

ERNIE Titan LLM

Baidu, which made its name in search engines, has recently been stepping up its game in AI. The Chinese company has developed its own Large Language Model called ERNIE (Enhanced Language Representation through Knowledge Integration). “Titan” is an enhanced version of ERNIE, designed to improve natural language understanding and generation tasks. It is pre-trained on a massive corpus of text data and can be fine-tuned for specific NLP tasks.

Although models like GPT-3 are promising, it’s still difficult for users to control results and obtain factually-consistent output. ERNIE proposes to fix this shortcoming by using a special technique during the training phase where the model learns to tell the difference between real text and text generated by itself. This also allows the model to rank the credibility of the generated text, making it more reliable and trustworthy.

Yandex YaLM 100B

YaLM 100B, like its name suggests, leverages 100 billion parameters. Parameters are the values that are learned and adjusted during training to optimize the model’s performance on a specific task. They determine how powerful a model is. While the 100 billion figure is obviously smaller than GPT-3’s 175 billion parameters, YaLM stands out by being available for free. Training the model took 65 days, with 1.7 TB of online texts, books, and “countless other sources” being fed to a pool of 800 A100 graphics cards. Yandex claims that this LLM is “currently the world’s largest GPT-like neural network freely available for English.” The model has been published on GitHub under the Apache 2.0 license, permitting both research and commercial use.

BLOOM

BLOOM has been trained to continue text from a prompt on vast amounts of text data using industrial-scale computational resources, according to BigScience, the company behind it. BigScience is an open collaboration with hundreds of researchers and institutions around the world, hosted on Huggingface. It is able to output in not just one, but 46 different languages, and 13 programming languages, which the company claims is “hardly distinguishable from text written by humans.” BLOOM can perform tasks it hasn’t been specifically trained for by converting them into text generation tasks. Like GPT-3, BLOOM uses around 175 billion parameters. But it has one big difference – it is accessible to everyone. The model’s training lasted four months and started on March 11, 2022, utilising 384 graphic cards of 80 gigabytes each on the Jean Zay supercomputer in France.

Gopher

Gopher is an autoregressive transformer-based dense LLM. It employs a staggering 280 billion parameters, rivalled only by Nvidia’s MT-NLG (530 billion) in size. The model was trained on MassiveText, which is a dataset sized 10.5 terabytes containing sources like Wikipedia, GitHub, and Massive Web. The company the model, DeepMind, is a British AI subsidiary of Alphabet Inc. that was acquired by Google in 2014. Gopher reportedly beats models like GPT-3 in terms of performance across disciplines like math, reasoning, knowledge, science, reading comprehension, and ethics.

MT-NLG

Megatron-Turing Natural Language Generation is developed by NVIDIA in collaboration with Microsoft. It was first introduced in October 2021 as a successor to the Turing NLG 17B and Megatron-LM models. The Turing project was launched by Microsoft in 2019 with the goal of allowing AI-powered enterprise search. MT-NLG is the largest of its kind, with 530 billion parameters. It can perform a wide range of natural language tasks like completion prediction, reading comprehension, commonsense reasoning, natural language inferences, and word sense disambiguation. The model was trained on NVIDIA’s Selene machine learning supercomputer, which is the sixth fastest supercomputer in the world.

Related Stories

IndiGo employee alleges caste-based abuse by senior officials; airline ‘strongly refutes’ claims

Test Priyanshu Haldar

‘Attack on Constitution’: After BJP MPs’ comments on judiciary, Congress senses opportunity, plans to corner government

Why so many Indian students — especially from Kashmir — go to Iran for higher education