The Lock on Knowledge

The Upper Limit of Knowledge in Human Organizations

There is a limit to the range of knowledge human effort can explore. That upper bound plainly exists, and we are drawing ever closer to it. Innate sages and people born already knowing belong more to myth than reality; the body of knowledge shared by humanity as a whole cannot escape the constraints of the following premises:

1. New knowledge is always produced on the basis of old knowledge.

2. An individual’s capacity for knowledge is finite.

The course of development since the birth of modern science also supports these two premises. Two trends are apparent.

First, within any given discipline, the threshold for becoming a frontier researcher keeps rising. Put differently, discovering new knowledge requires mastery of more and more existing knowledge. Achieving the same degree of progress therefore demands either longer education or more gifted researchers, and both have limits. One indirect piece of evidence is the gradual rise in the average age at which Nobel laureates in physics receive the prize.

Second, the overall body of knowledge is being divided into ever finer parts, and disciplinary classifications are becoming increasingly specialized. This may be described as an expression of the social division of labor, but it can also be understood as a concession to the limits of individual knowledge. In our time, we no longer see masters who range across multiple disciplines; even someone able to work and produce results in several branches of a single discipline is already an exceptionally accomplished leader in the field. On the surface, this phenomenon may not seem to impair humanity’s ability, as a whole, to discover new knowledge. In reality, it raises the hidden cost of acquiring knowledge. The cost of communication between disciplines—and even between different branches of the same discipline—continues to rise. The Chinese saying that “different trades are separated by mountains” has become a reality even between neighboring research specializations. Difficulties in exchanging knowledge across fields may cost us many opportunities to make new discoveries.

The latter trend can also be summarized this way: as disciplinary classifications grow ever finer, the complexity of collaboration within human organizations increases explosively. At the extreme, collaboration and communication themselves will consume the entire mental capacity of researchers and the entire organizational capacity of research institutions.

No amount of educating successive generations of chimpanzees would enable them to master calculus—unless, somewhere along the way, they evolved into Homo sapiens. Human beings face a comparable intellectual ceiling. Ours is merely higher than that of chimpanzees, and we cannot surpass it unless we continue to evolve.

Are Large Language Models Omnipotent?

The birth of artificial intelligence has significantly expanded humanity’s capacity to explore knowledge. It cannot, however, escape the constraints described by the premises above; this is especially true of large language models, currently the most celebrated form of AI.

In a world with large language models, the essential truth of the propositions that “new knowledge is always produced on the basis of old knowledge” and “an individual’s capacity for knowledge is finite” remains unchanged. What has changed is how those limits manifest themselves.

First, one bottleneck in the development of large language models is a shortage of “useful existing knowledge.” The sources from which today’s large language models construct their knowledge systems can be divided into human knowledge and knowledge produced by the models themselves. The scale and parameter counts of large language models are expanding far faster than humanity’s store of knowledge. The total number of books humanity has produced and that are currently accessible is on the order of 400 to 500 million, while the total number of publicly published academic papers is on the order of 100 to 200 million. These constitute the most concentrated essence of humanity’s knowledge base, amounting to roughly 10 petabytes of data. No individual human—or even scientific organization—could master so much knowledge. To a large language model, however, this material is a nonrenewable mineral deposit, and it will soon be mined out.

The second source of a large language model’s knowledge is what it produces itself. This, in turn, can be divided into two types: predictions based on human knowledge, and exploration of the formal sciences through its own feedback. The former still derives, in essence, from the human knowledge base. As AI-generated material accounts for a growing share of all available knowledge, the quality of the overall knowledge base will deteriorate, because such generated material has difficulty obtaining feedback from the real world. The latter, by contrast, constitutes genuinely useful new knowledge generated by AI. Formal sciences such as mathematics and computer science can provide their own feedback without requiring feedback from the physical world. This is why large language models currently perform best in these fields and show the potential to surpass human beings completely.

Second, while the knowledge capacity of large language models has increased enormously, the interaction costs found in human organizations have migrated into retrieval and recall—specifically, into context capacity and compression. For a large language model, knowledge storage can be regarded as almost infinitely expandable. Its working capacity, however—the amount of context active in its chain of thought—is finite. Expanding context capacity, or working memory, presents the same problem as enlarging a complex human organization: the range within which it can operate efficiently is very limited. Beyond that boundary, most additional cost goes toward maintaining the organization itself. Context compression can partly address the problem, but repeated compression introduces distortion of its own, until the result departs from reality.

To sum up: first, the initial constraint on large language models’ exploration of knowledge comes from their interaction with human knowledge and with the real world. Outside the formal sciences, the useful knowledge needed for their evolution is extremely limited. Second, as entities existing in the real world, large language models face capacity constraints just as human beings do. The constraint lies not in the scale of their knowledge base, but in the scale of their context. Of these two constraints, the first may be solvable; the second may be a fundamental limitation of every intelligent system.

Formal Science and the Physical World

Existing large language models have achieved astonishing results in formal sciences such as mathematics and some areas of computer science. This is closely related to the characteristics of both formal science and large language models.

The defining feature of formal science is that the “truth” of its assertions is judged entirely within the rules of the system it constructs. In other words, “truth” means internal consistency and does not involve interaction with the physical world. The substance of formal science can likewise be regarded as a kind of structured linguistic sequence: a system of “symbols, rules, and extrapolation.” These characteristics place it squarely in the optimal operating range of large language models.

First, verifying truth or falsity in formal science does not involve the physical world and is therefore inexpensive for a large language model. A model’s exploration of formal science can consequently receive efficient, reliable feedback—far more efficient than feedback derived from human knowledge. In this process, the model can form a closed loop entirely within text.

Second, the foundation of a large language model is the prediction of serialized text, which is naturally isomorphic with formal science. What large language models do best is fill in missing content in serialized text, while formal science is itself the formalized expression of existing structures.

Unfortunately, superhuman ability in formal science is not enough to make a large language model a genuine artificial general intelligence. One fundamental problem is that “truth” in formal science and “truth” in the physical world cannot simply be transferred from one to the other. “Truth” in formal science represents only consistency within a formal system, whereas “truth” in the physical world generally means correspondence with reality. Moving from mathematics to physics usually depends on some formalized assumption; only on the basis of that assumption—or within the domain it covers—do mathematical judgments become meaningful.

Why mathematics is so effective in the real world is far too profound a question for this essay to explore. For large language models, however, the mismatch between mathematical and physical “truth” leaves the feedback loop open. Existing large language models cannot interact directly and genuinely with the physical world. When confronting questions outside the formal sciences, they must therefore rely on human beings as proxies, greatly reducing the efficiency with which they acquire and produce knowledge about the world.

Is embodied intelligence, then, the critical step in AI’s evolution toward artificial general intelligence?

Yes. AI capable of interacting directly with the physical world on its own is undoubtedly a crucial path toward AGI. But it must be recognized that “embodied intelligence” modeled on human beings has no potential to exceed human capabilities. One reason the existing human knowledge base cannot expand rapidly is that the effective bandwidth through which individuals and organizations interact with the physical world is extremely limited. Interacting with the world by imitating human beings will not raise that bandwidth ceiling. Artificial general intelligence will certainly possess embodiment, but a powerful AI that surpasses humanity will certainly not emerge with humanity itself as its template.

In an age of earth-shaking change, may every intelligence see farther.

Translation Notes

• “知识之锁” is rendered as “The Lock on Knowledge.” The title can suggest both a lock that restricts access to knowledge and a constraint that locks knowledge growth in place; English cannot preserve both readings equally strongly.
• “隔行如隔山” is a Chinese saying that literally compares the gulf between professions to a mountain. The translation retains the mountain image because the author immediately extends it from disciplines to research directions.
• “带头大牛” is informal academic slang for a highly accomplished, influential researcher. It is rendered as “an exceptionally accomplished leader in the field” to preserve the register shift without using opaque slang.
• “有效旧知识” is translated as “useful existing knowledge.” Here “有效” means knowledge capable of contributing meaningfully to further model development, not merely knowledge that is valid or true.
• “文本内的闭环” / “反馈回路的开环” are rendered as a “closed loop entirely within text” and an “open feedback loop,” preserving the control-systems metaphor.
• “具身智能” is rendered by the established term “embodied intelligence.”