From a Theological Quarrel to Next-Token Prediction

From Markov chains and information theory to neural networks and transformers: the ideas that made modern language models possible.

[That’s not a typical blog post, I am testing this format and will appreciate your feedback.]

AI moves so quickly that even five years ago can feel like prehistory. New models, products and acronyms arrive every few months, and each wave makes the previous one look strangely primitive.

But when I started wondering where the idea behind next-token prediction actually came from, I found myself going back not five years, but more than a century—to a theological quarrel between two Russian mathematicians.

I wanted to understand how we got from counting vowels in a work of Pushkin to machines that can write, translate, generate code and hold a conversation. What I found was not a single moment of invention, or a straight road leading inevitably to ChatGPT. It was a series of ideas developed for very different reasons, which eventually came together.

This is my attempt to connect those dots—not through every technical breakthrough, but through the few big ideas that made modern language models possible.

A quarrel about free will

At the beginning of the twentieth century, the Russian mathematician Andrei Markov was engaged in a long-running dispute with Pavel Nekrasov, a fellow mathematician who had originally trained in theology.

Their disagreement was about mathematics, but it also reached into religion, politics and the question of free will. Nekrasov belonged to a group of Russian intellectuals who tried to use probability theory to support the idea that human choices were independent. Markov, an atheist and an outspoken critic of the church, considered this an abuse of mathematics.

The mathematical issue was whether reliable statistical laws required events to be independent. A familiar example of independence is a coin toss: the result of one fair toss does not determine the next. But much of the world does not behave like that. Tomorrow’s weather is related to today’s. The next move in a chess game depends on the position already on the board. And the next letter in a sentence is influenced by the letters that came before it.

Markov set out to show that dependent events could still produce patterns that mathematics could describe. An event could be shaped by what happened before it without the entire sequence becoming unknowable.

That idea would eventually become the basis of what we now call a Markov chain. But Markov was not trying to create a language model, let alone a chatbot. He was trying to win an argument about probability.

Counting vowels in Eugene Onegin

To demonstrate his idea, Markov chose an unusual dataset: Alexander Pushkin’s novel in verse, Eugene Onegin.

In 1913, he presented an analysis of 20,000 letters from the text. Working by hand, he classified the letters as vowels or consonants and counted how often one followed the other. He wanted to know whether the probability of the next letter changed depending on the letter before it.

It did.

A vowel was not equally likely to appear at every point. Its probability was different after another vowel than after a consonant. Written language, Markov demonstrated, contained measurable chains of dependence.

This was still a long way from text generation. Markov was not asking a machine to continue Pushkin’s sentences, and he was not predicting whole words. He was using literature as evidence for a mathematical argument.

Even so, his experiment introduced the question that runs through the history of language models:

How much can we learn about what comes next from what came before?

Shannon turns guessing into a science

Several decades later, that question reappeared in the work of Claude Shannon at Bell Labs.

Shannon was not initially trying to make computers speak or write. He was trying to understand communication. How much information can be transmitted through a channel? What happens when the signal is noisy? And how much of a message can be reconstructed from its surrounding context?

To answer those questions, he had to think about predictability.

In his foundational 1948 paper on information theory, Shannon produced several approximations of English. Randomly selected letters looked like noise. Letters chosen according to their frequency in English looked slightly more familiar. When the choice of each new symbol took previous symbols into account, fragments resembling English began to emerge.

The more context the process used, the more language-like the result became.

Shannon later conducted experiments in which people tried to guess the next character in a passage of English. Some characters were easy to predict; others carried more surprise. This helped him estimate the redundancy of written language—the extent to which a message remains understandable even when some of it is missing.

Markov had shown that symbols in a sequence could be dependent. Shannon showed that this dependence could be used to measure information. Predictability was no longer simply an observation about language. It had become part of a general theory of communication.

When machines learned language by counting

For much of the following half-century, statistical language technology was built around counting.

A system could look through a large collection of text and record which words tended to appear together. If “peanut butter and” was frequently followed by “jelly,” then “jelly” would receive a high probability as the next word. Models of this kind became known as n-grams: they made predictions using a short sequence of preceding words or symbols.

These models gradually found their way into everyday technologies:

• Speech recognition

• Spelling correction

• Early machine translation

• Predictive typing on mobile phones

• Familiar autocomplete suggestions in search boxes

Long before ChatGPT, millions of people were already interacting with systems that tried to predict what they were going to say or type next.

Counting worked remarkably well, but it had an obvious limitation. Language can produce an almost endless number of combinations. Many perfectly reasonable phrases may appear only once—or never appear at all—in the material used to build the model.

The models also struggled to recognize similarity. “Dog” and “puppy” are closely related to us, but in a table of word counts they are simply two different entries. Learning something about one did not automatically teach the system anything about the other.

The old systems could remember common sequences. The next generation would begin to learn what words had in common.

From counting words to representing them

This is where neural networks enter the story.

Instead of treating every word as an isolated symbol, neural language models learned numerical representations of words. In those representations, words used in similar ways could acquire similar characteristics. “Paris,” “London” and “Rome” did not need to be stored in exactly the same contexts for the model to notice that they played related roles.

In 2003, Yoshua Bengio and his colleagues published an influential paper describing a neural network that learned these word representations while learning to predict the next word.

The training process can be explained without going deeply into the mathematics. The model was shown part of a text and asked to guess what came next. It compared its prediction with the actual word, adjusted itself slightly and tried again. By repeating this process across a large collection of examples, it gradually became better at prediction.

The important shift was from storing phrases in enormous counting tables to learning reusable patterns. What the model learned in one sentence could help it make sense of another sentence it had never seen before.

Later neural networks became better at carrying information forward through a sequence. But they still had difficulty with long passages. Information from the beginning of a sentence could fade by the time the model reached the end. Processing text one step after another also made these systems slow to train at very large scale.

The next breakthrough began with a deceptively simple idea: perhaps the model should not try to remember everything equally.

Learning where to pay attention

Consider the sentence:

The animal didn’t cross the road because it was tired.

To interpret “it,” we naturally connect the word to “animal.” Other words in the sentence matter less for that particular relationship.

Attention gave neural networks a way to do something similar. When producing or interpreting a word, the model could examine different parts of the text and assign more importance to the parts that seemed relevant at that moment.

Attention first became prominent as a solution to a problem in machine translation. Earlier systems tried to squeeze the meaning of an entire sentence into a single internal representation before translating it. That worked for short sentences, but information was easily lost as sentences grew longer.

With attention, the system could look back at different parts of the original sentence while producing each part of the translation. Instead of relying entirely on a fading memory, it could decide where to focus.

This idea prepared the way for the transformer.

The transformer—and the unusually short road to market

In 2017, a team of Google researchers published a paper with the memorable title Attention Is All You Need. It introduced the transformer, an architecture built around attention rather than the older step-by-step recurrent approach.

The transformer could examine relationships between words while being trained much more efficiently on modern computing hardware. That practical advantage mattered enormously. Researchers could now train bigger models on far larger collections of text.

The transformer began as a machine-translation architecture, not as a plan for a universal chatbot. Yet the journey from research paper to widely used product was remarkably fast.

In 2018, GPT and BERT showed that transformers trained on large amounts of text could acquire abilities that transferred to many different language tasks. In 2019, GPT-2 demonstrated that a larger model trained on more data could produce surprisingly coherent passages. By 2020, GPT-3 could perform a range of tasks from instructions and examples included directly in a prompt, and these capabilities were becoming available to developers through an API.

Then, in late 2022, ChatGPT placed a conversational interface around this technology and made it accessible to almost anyone.

It took roughly three years for the transformer to travel from a research paper to a commercially usable platform, and about five years to become a mass-market consumer product. That pace helps explain why five years can feel like prehistory in AI.

But the transformer was only one ingredient. Modern language models became possible because several things arrived together: the architecture, enormous collections of digital text, powerful computing infrastructure, improved training techniques and a willingness to see how far performance could be pushed by scale.

The transformer did not invent language prediction. It made language prediction scalable.

When prediction started to look like something more

A GPT-style model is trained on a task that sounds almost disappointingly simple. It sees part of a text and tries to predict the next token. A token may be a whole word, part of a word, a number or a piece of punctuation.

Calling this “autocomplete” is not entirely wrong, but it can hide the difficulty of the task.

To continue a recipe, a model must learn patterns found in recipes. To continue a legal document, it must pick up something about legal language. To predict the next lines of a computer program, it must learn recurring structures in code. To continue a historical discussion, it must absorb associations between people, events, places and dates.

The model is directly trained to predict tokens. But becoming very good at prediction across many kinds of text encourages it to build internal representations of some of the structures behind that text: grammar, style, concepts, relationships and patterns of reasoning.

This does not settle the question of whether a language model “understands” in the human sense. Nor does it make its output automatically true. A plausible continuation can still be factually wrong. But it helps explain why “just predicting the next token” can produce abilities that appear much broader than prediction.

The same method also proved effective for software. Code, like prose, is a sequence of tokens—and real programs are highly repetitive. By training language models on public code alongside comments and documentation, researchers taught them to move from a description of a task to a likely implementation. In 2021, this idea reached developers through GitHub Copilot, powered by a production version of OpenAI’s Codex: next-token prediction had become a practical coding tool.

A language model is not yet an assistant

There is one more distinction that is easy to miss.

A model trained to predict text is fundamentally a continuation engine. If it has seen many questions followed by answers, it may learn to answer questions. But next-token training alone does not tell it that it should always be helpful, follow the user’s intention, acknowledge uncertainty or avoid harmful instructions.

Additional training is needed to turn a base language model into something closer to an assistant. People provide examples of desirable answers, compare different responses and help shape which kinds of behaviour the system should prefer.

That layer of training is a major part of what users experience as the personality and usefulness of a conversational AI product. The underlying model predicts a continuation; the assistant has also been trained to interpret that continuation as a response to a human request.

This final step also reminds us that ChatGPT was not simply the transformer placed inside a chat window. It was the product of architecture, scale, data, human feedback, interface design and engineering coming together.

Back to Markov

It would be tempting to end this story by drawing a straight line from Markov’s quarrel with Nekrasov to ChatGPT. But history is rarely that tidy.

Markov did not invent next-token prediction. Shannon was not building a chatbot. The researchers who developed attention were trying to improve translation, and the transformer itself was not originally presented as the foundation for a general-purpose assistant.

Each group was solving the problem in front of it. The connection becomes visible only in retrospect.

Markov supplied a mathematics of dependence. Shannon connected predictability with information. Statistical language models turned prediction into a practical engineering technique. Neural networks learned representations that counting alone could not capture. Attention gave models a better way to use context, and transformers made the process scalable. Vast datasets and computing power then pushed that machinery far beyond its original applications.

The story began with an argument about whether dependent events could still obey mathematical laws. More than a century later, the same broad insight sits beneath systems used around the world: what comes next is shaped by what came before, and those relationships can be learned.

Markov counted vowels and consonants to make a point about probability. He could hardly have imagined where that question would eventually lead.

---

Sources and further reading

• Gely P. Basharin, Amy N. Langville and Valeriy A. Naumov, “The Life and Work of A. A. Markov”, 2004.

• A. A. Markov, “An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains”, lecture delivered in 1913; English translation published in 2006.

• Claude E. Shannon, “A Mathematical Theory of Communication”, 1948.

• Claude E. Shannon, “Prediction and Entropy of Printed English”, 1951.

• Frederick Jelinek, “The Dawn of Statistical ASR and MT”, 2009.

• Yoshua Bengio et al., “A Neural Probabilistic Language Model”, 2003.

• Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate”, 2014/2015.

• Ashish Vaswani et al., “Attention Is All You Need”, 2017.

• Alec Radford et al., “Improving Language Understanding by Generative Pre-Training”, 2018.

• Alec Radford et al., “Language Models Are Unsupervised Multitask Learners”, 2019.

• Tom B. Brown et al., “Language Models Are Few-Shot Learners”, 2020.

• Mark Chen et al., “Evaluating Large Language Models Trained on Code”, 2021.

• Long Ouyang et al., “Training Language Models to Follow Instructions with Human Feedback”, 2022.