Can LLMs Think About Thinking? Exploring Metacognition in AI

Deep Dive Series: Can LLMs Think About Thinking, and Can We Leverage Such Traits?

This article explores the fascinating intersection of Large Language Models (LLMs) and metacognition, drawing parallels with human cognitive development and examining how these advanced AI systems might possess and utilize self-awareness about their own knowledge and processes.

The Genesis of Metacognition

The concept of metacognition, defined as "knowledge and cognition about one's own cognitive phenomena," was first coined by American developmental psychologist John H. Flavell in 1979. His early research with children indicated that older children exhibited metacognitive abilities, such as understanding their learning processes, while younger children did not. Flavell presciently suggested that developing metacognitive skills could be instrumental in teaching individuals to make wiser decisions and learn more effectively.

Subsequent educational research has validated Flavell's insights, demonstrating that fostering metacognition enhances learning outcomes and resilience in students.

LLMs and the Question of Metacognition

Given the remarkable advancements in LLMs, capable of superhuman performance in many language-based tasks, researchers are investigating whether these models exhibit similar metacognitive traits. At Princeton Language and Intelligence (PLI), studies suggest that LLMs not only display aspects of metacognition but that these abilities can be actively leveraged to improve their performance.

Establishing Cognition in LLMs: The SKILL-MIX Framework

Before delving into metacognition, it was crucial to establish that LLMs possess genuine cognitive abilities, moving beyond the notion of them being mere "stochastic parrots"—systems that randomly recombine text from their training data.

Two key papers published in 2023 by researchers from PLI and Google DeepMind laid the groundwork:

"A Theory for Emergence of Complex Skills in Language Models" (Arora et al.): This work utilized random graph theory and neural scaling laws to link model size and training data to performance. It proposed that larger LLMs are inherently better at combining multiple skills required for complex language tasks. The theory posits that scaling up model parameters by an order of magnitude can double a model's competence in combining skills.
"Skill-Mix: A Flexible and Expandable Family of Evaluations for AI models" (Yu et al.): This paper introduced SKILL-MIX, an evaluation framework designed to measure an LLM's ability to combine skills. The evaluation involves prompting LLMs to generate text demonstrating a specific set of skills on a given topic, followed by automated and human assessment. SKILL-MIX provided empirical evidence that models like GPT-4 possess cognitive abilities, thus moving beyond the "stochastic parrot" description.

Leveraging Metacognition for Enhanced Performance

The research then explored how LLMs' metacognitive capabilities could be practically applied to improve their performance, particularly in challenging domains like mathematical problem-solving.

In a study by Didolkar et al. (involving PLI, Mila, University of Cambridge, and Google DeepMind), researchers investigated GPT-4's metacognitive knowledge concerning mathematical problem-solving:

Skill Identification: GPT-4 was prompted to identify the concepts (skills) necessary to solve problems from the GSM8K dataset (a collection of grade school math problems).
Skill Grouping: The model was then asked to group these fine-grained skills into broader, compound skills (e.g., "basic arithmetic operations").
Repository Creation: A repository of "skill exemplars" was created by associating each training question with its relevant compound skill and storing example question-answer pairs.
Performance Improvement: When a test question was presented to an LLM along with relevant skill exemplars (acting as in-context learning examples), the LLM's performance on the GSM8K dataset significantly improved. This demonstrated that the LLM could leverage its "knowledge about its knowledge" to enhance problem-solving.

INSTRUCT-SKILLMIX: Efficient Fine-Tuning with Metacognition

Building on these findings, the PLI team (Simon Park, Simran Kaur, and Anirudh Goyal) developed a method called INSTRUCT-SKILLMIX to efficiently fine-tune smaller LLMs using synthetic data generated from a larger LLM's metacognition.

Phase 1: Skill Extraction: A powerful "teacher" LLM (GPT-4-Turbo) was used to generate a diverse set of topics, the skills required for each topic, and associated tasks (e.g., "information seeking").
Phase 2: Data Generation: The teacher LLM created approximately 4,000 instruction-response pairs by combining random sets of skills and tasks. This synthetic dataset, INSTRUCT-SKILLMIX, was designed for supervised fine-tuning.

The results were striking: a LLaMA-3 8B base model fine-tuned on the INSTRUCT-SKILLMIX dataset achieved a 42.76% win rate on AlpacaEval2.0, outperforming larger models like Claude 3 Opus. This demonstrates that a small dataset, generated using metacognitive insights, can effectively endow smaller LLMs with advanced instruction-following capabilities.

Future Directions and Implications

The article concludes by highlighting the broader implications of viewing LLMs through the lens of metacognition:

Understanding Inner Workings: It provides a framework for better understanding the internal processes of these complex models.
Safety and Alignment: Metacognition can be used to steer LLMs towards behaviors that are safe and aligned with human values.
Emerging Research: Studies on LLM reasoning traces and self-consistency checks suggest that LLMs might be developing internal mechanisms akin to "ruminating" on their outputs.

The authors suggest that these explorations into LLM metacognition are just the beginning, promising further advancements in AI capabilities and alignment.

Image Credit: Princeton Laboratory for Artificial Intelligence Research