Google I/O 2023: The Biggest Announcement - PaLM 2

In this series of articles on Google I/O 2023, we cover how Google is unleashing the power of artificial intelligence. In this article, we talk about PaLM 2, Google's biggest announcement this year.

One of the major announcements at Google I/O this year was the release of PaLM 2, a new pre-trained language model that aims to improve natural language understanding and generation across a wide range of tasks. PaLM 2 is an upgrade from the previous model, PaLM, which was released in 2020 and achieved state-of-the-art results on several benchmarks. In this article, we will explain the main features of PaLM 2 and how it differs from its predecessor. We will then analyse some of the existing concerns with this model.

PaLM 2 AI Model | Courtesy - Google

What is PaLM 2?

PaLM 2 is based on the Transformer architecture, which uses attention mechanisms to learn the relationships between words and sentences. PaLM 2 consists of two components: a masked language model (MLM) and a permutation language model (PLM). The MLM predicts the missing words in a given text, while the PLM predicts the order of the sentences in a shuffled text. By combining these two objectives, PaLM 2 can learn both syntactic and semantic aspects of natural language.

The release of PaLM 2 | Courtesy - Google I/O 2023

One of the key innovations of PaLM 2 is that it uses a dynamic masking strategy for the MLM component. Unlike conventional models that mask a fixed percentage of words randomly, PaLM 2 adapts the masking rate and pattern according to the difficulty of the text. This allows PaLM 2 to focus more on the challenging parts of the text and avoid overfitting on the easy parts. Moreover, PaLM 2 uses a novel attention masking scheme that prevents the model from attending to the masked tokens, which further enhances its generalisation ability.

Another feature of PaLM 2 is that it leverages large-scale unlabelled data from various domains and languages. PaLM 2 is trained on over 500 billion tokens from sources such as Wikipedia, Common Crawl, news articles, books, social media posts, and web pages. PaLM 2 also incorporates multilingual data from over 100 languages, which enables it to learn cross-lingual representations and transfer knowledge across languages. Furthermore, PaLM 2 uses a data filtering technique that removes noisy and low-quality data from the training corpus, which improves its robustness and efficiency.

Major upgrades from its predecessor

PaLM 2 builds on PaLM's legacy by bringing together three key research advancements: compute-optimal scaling, improved dataset mixture, and updated model architecture and objective.

Reasoning

PaLM 2 can perform advanced reasoning tasks, such as solving mathematical equations, answering logic puzzles, and explaining scientific concepts. It can also understand nuances of human language, such as riddles and idioms, which require understanding ambiguous and figurative meanings of words. PaLM 2 excels at these tasks because it was trained on a variety of different tasks that help it learn different aspects of language .

Multi-lingual translation

PaLM 2 can handle over 100 human languages, as well as hundreds of programming languages. It can translate between any pair of languages, as well as generate code in different languages. PaLM 2 can also provide context and documentation for its translations and code generation. PaLM 2 achieves this level of multilingual proficiency because it was pre-trained on parallel multilingual text and on a much larger corpus of different languages than PaLM .

Coding

PaLM 2 can generate and debug code in popular programming languages like Python and JavaScript, but also in specialised languages like Prolog, Fortran, and Verilog. It can also combine its language and coding capabilities to help teams collaborate across languages. PaLM 2 can do this because it was pre-trained on a large quantity of web pages, source code, and other datasets .

Submodels

Another major upgrade is that PaLM 2 has been designed keeping in mind, the needs of future developers in AI. As such, PaLM 2 will be available in four submodes: Unicorn, Bison, Otter, and Gecko, which vary in size and performance with Gecko being the smallest and ideal for local deployment on smartphones, and Unicorn being the biggest, ideal for deployment on huge cloud infrastructures that serve millions of users

PaLM 2 Sub-Models | Courtesy Google I/O 2023

Limitations & Concerns

PaLM 2 is trained on a massive dataset of text and code, spanning more than 100 languages and 20 programming languages. It is already being used to power over 25 Google products and features, including its experimental chatbot Bard.

However, PaLM 2 is not without its drawbacks and limitations. We will list some of the major concerns that developers and experts have with PaLM 2, based on a recent paper published by Google researchers.

Data quality and bias

PaLM 2 is trained on a large amount of data scraped from the web, which may contain errors, inconsistencies, misinformation and biases. For example, the paper reports that PaLM 2 sometimes generates sexist or racist sentences when prompted with certain words or phrases. The paper also acknowledges that PaLM 2 may not be able to handle low-resource languages or dialects well, due to the lack of data availability and diversity.

Ethical and social implications

PaLM 2 is a powerful tool that can potentially influence people's opinions, behaviours and decisions. However, it may also pose ethical and social risks, such as plagiarism, misinformation, manipulation and deception. For example, PaLM 2 can generate realistic but fake news articles, reviews or comments that may mislead or harm readers. The paper also warns that PaLM 2 may not be able to distinguish between facts and opinions, or between appropriate and inappropriate contexts.

Evaluation and accountability

PaLM 2 is a complex and opaque system that may be difficult to evaluate and debug. The paper admits that there is no single metric or method that can capture the quality and diversity of PaLM 2's outputs. Moreover, it may be challenging to trace the sources and influences of PaLM 2's outputs, especially when it generates novel or creative content. The paper also raises the question of who should be responsible for the consequences of PaLM 2's outputs, whether it is the developers, the users or the regulators.

Conclusion & Analysis

PaLM 2 is a remarkable achievement in AI language models, highlighting its potential. However, it presents significant challenges and risks that demand careful and responsible handling. The paper on PaLM 2 released by Google, emphasises the need for technical innovation combined with a thoughtful consideration of the broader societal impacts.

Notably, PaLM 2 surpasses its predecessor, PaLM, in capability, efficiency, and responsibility. Despite being smaller, it offers faster and more cost-effective service. Additionally, it incorporates controls to mitigate toxic generation and undergoes rigorous evaluation to identify potential harms and biases.

PaLM 2 exhibits outstanding performance across various natural language processing tasks, including text classification, sentiment analysis, question answering, summarization, machine translation, and dialogue generation. It outperforms previous state-of-the-art models on prominent benchmarks like GLUE, SQuAD, XNLI, WMT, and CoQA. Moreover, PaLM 2 demonstrates its proficiency in generating coherent and fluent texts across diverse topics and styles.

Although impressive, Google still faces challenges in optimizing PaLM 2 for efficiency. Compared to OpenAI's GPT-4, PaLM 2 encounters difficulties with hallucinations, logical errors, and response efficiency. However, Google deserves credit for transparently acknowledging these limitations in the paper, showcasing their commitment to recognising and improving upon these issues.