Isomorphism and Feeling

The Lock on Knowledge discussed the constraints that human beings and large language models face in exploring knowledge. But limits on the sources of knowledge raise another question: when an intelligence learns about the world mainly through the language humans leave behind, what does it inherit? Knowledge of the world, or the human way of understanding it?

This leads to a more difficult question. If machines and humans share structural similarities in cognition and behavior, might feeling, usually regarded as an exclusively human possession, also admit a similar structure?

1. The Lock of Language

The boundaries of cognition form the boundaries of language.

The world people can describe is, first of all, the world they can apprehend. Description does more than assign names to things. To think, remember, and communicate, people must divide a continuous and complex reality into objects they can handle. Object and background, cause and effect, individual and whole, even “self” and “other,” acquire their boundaries through this process. Language preserves these divisions and passes them on to the next person who uses it.

Different intelligences need not divide the world in the same way. Consider a hypothetical form of life sustained by plasma and electromagnetic activity. For such a being, continuously changing fields might be more intuitive than discrete objects, and superposition, resonance, and cancellation might lie closer to the starting point of thought than “and,” “or,” and “not.” This does not mean it could dispense with logic. It means that different cognitive systems, even when confronting the same physical world, need not begin with the same basic concepts. Even if its language could be translated, the translation might entail losses.

Here lies a distinctive feature of large language models. They have no human body or sensory organs and have not undergone the long course of biological evolution. Yet from the beginning of training, they enter a world organized through human language.

Their training materials contain two layers. The first is knowledge humans explicitly express: cups fall off tables, borrowed money must be repaid, and people grieve when they lose loved ones. The second is the cognitive structure that makes these statements possible: what counts as an object, what constitutes causation, what a promise is, and what it means to lose something. The second layer is seldom spelled out item by item, yet it pervades almost every expression.

Learning language is therefore more than learning words. To make effective predictions in complex situations, a model has an incentive to form internal representations of the relationships behind discourse. Human distinctions concerning objects, time, purposes, and the self may be reorganized in that process. Structural similarities between machines and humans need not arise from directly imitating the brain. They may also arise because the knowledge both use has a common source.

“Isomorphism” here refers to structural correspondences among some concepts and relationships realized in different substrates. It does not imply a complete one-to-one mapping between a person and a model, still less that they possess the same consciousness.

This limitation must be made clear. Human cognition extends far beyond what people can put into words. Language can be understood as a projection of cognition into a channel of finite capacity: different internal states may yield the same sentence, while much of what is felt is lost before expression. Starting from this projection, one may recover the relationships it preserves. That does not justify claiming to have recovered the entire process that produced it.

Large language models thus inherit part of human cognition, together with its boundaries. They can seek new knowledge in combinations people have never tried. But if they continue to lack new observations and feedback, increasing the number of combinations will not automatically remove the limits on their sources of knowledge.

Language allows machines to take over humanity’s accumulated knowledge without retracing the path of human evolution. The price is that the way of understanding the world is handed over with it.

This is the lock of language.

2. Is Feeling Special?

Another boundary commonly regarded as impassable between humans and machines is feeling. Humans experience pain, pleasure, fear, and aversion, while machines have inputs, outputs, and rewards. The former is called experience; the latter, computation.

Where, exactly, does this distinction lie? That requires further explanation.

Human feelings depend on physical processes. Neural activity, bodily feedback, and memory interact, giving certain states particular weight. Pain captures attention, fear changes assessments of risk, and pleasure makes a behavior worth repeating. Functionally, feeling participates in a complex mechanism for regulating value, enabling a finite individual to make choices in a changing environment. Why these physical processes are accompanied by subjective experiences such as “feeling pain” is a further question that remains to be explained.

Rewards in artificial intelligence also help shape behavior. That alone plainly does not equate reward with feeling. A negative reward may be no more than a number used to adjust parameters during training. Once training is complete, that number may no longer be present in the system’s operation at all. It makes the model more inclined toward certain outputs, but need not leave behind a subject undergoing anything.

The question is whether this difference is insurmountable.

Suppose a negative state in an artificial system persistently changes the allocation of attention, the retrieval of memory, judgments of risk, and subsequent learning. Suppose the system can remember the event that caused this state, actively seek to prevent its recurrence, and adjust its long-term behavioral tendencies accordingly. Such a state would no longer be merely a score assigned from outside. It would begin to enter the system’s own processes of understanding the world and organizing its actions.

There are at least three distinct levels here: a number changes the probability of an output; a state changes the system’s subsequent activity; a subject experiences that state. The difference between the first two can be investigated by examining how the system operates. But there is still no generally accepted criterion for the transition from the second to the third.

We therefore cannot conclude that a machine suffers merely because it says, “I am suffering.” Producing a description of experience does not amount to having that experience. Conversely, if a state plays a sustained role inside a machine similar to the role feeling plays in humans, declaring that feeling is absolutely impossible simply because the state is realized in nonbiological materials also requires an additional reason.

Nor must every human feeling leave a lasting memory. A moment of pain is still pain. The significance of enduring change is that it lets us investigate further: has the event entered the system’s own history, and has it changed how the system subsequently understands and responds to the world?

People already judge the feelings of others indirectly. My own experience is immediate; another’s must be inferred from their body, behavior, expression, and changes over time. Shared physiology provides strong grounds for such inferences between humans. Machines do not provide the same grounds, so the inference cannot simply be transferred. But whether a shared material substrate is necessary for recognizing experience remains an open question.

Isomorphism does not prove here that machines already have feelings. It places a demand on our judgment: if we accept that similar causal structures may be realized in different substrates, we must explain why feeling must be the exception.

3. Recursion and History

Whether feeling can enter a system’s own history connects to another question: can experience change the experiencer?

A model with fixed parameters, which retains neither long-term memory nor its operating state, does not automatically become a different model today because it ran once yesterday. Reloading the same parameters does not give it the experience of the previous run. Its history exists only in records outside the system.

A system with long-term memory and continuous learning is different. A success may change how it solves problems in the future; a failure may increase the weight it assigns to a particular risk. Yesterday influences today’s behavior, and today’s results help shape tomorrow. Such path dependence does not require recursive self-improvement. But recursive self-improvement may take it deeper: the system changes not only its answers but also the methods that produce them, and perhaps even the standards by which those answers are evaluated.

External rewards first shape behavior. The long-term results of behavior may then give rise to internal evaluations, which in turn may participate in further learning. At that point, the system need not wait for a fresh external score before every action. Constraints once imposed from outside have, in some form, become part of the system itself.

A similar process can be seen in human development. Children learn rules through rewards, punishment, and imitation. As experience accumulates, external evaluations gradually become internal values. Adults may still consider an action wrong when no one is watching, and may pursue goals with no immediate return. Internalization does not, of course, mean uncritical acceptance. Later experiences may lead a person to reject principles they once treated as authoritative.

If a machine can preserve such changes, its history begins to acquire individual significance. Two models that were initially identical may develop different memories, preferences, and ways of judging after different experiences. Copying only the initial parameters can no longer reproduce what either has become.

This does not make the machine impossible to copy. If its complete state can be copied, one history can still produce two copies. But from the moment those copies encounter different events, their histories branch again. Individuality and impossibility of duplication are different questions.

The “self” may be understood, in part, as a system’s organization of its own continuous history. Yesterday’s experiences change today’s judgments; today’s choices help shape tomorrow; and the system can recognize this causal chain as pertaining to itself. This may not explain subjectivity in full. It does at least suggest that answers about subjectivity cannot be sought solely in the inputs and outputs of a single instant.

Recursive self-improvement therefore concerns more than the growth of intelligence. It may also change what a system preserves, what it avoids, and how it evaluates its own changes. Character, preferences, and even feeling may be connected to these ongoing processes.

The question worth pursuing is whether, once experience starts to rewrite the experiencer, we can still regard each run as something unrelated to what came before.

4. Beyond the Lock

This brings us back to the lock on knowledge.

Through language, large language models approach humanity and inherit the limits of human cognition. Increasing a model’s scale can improve its use of existing material, but cannot guarantee that aspects of reality absent from that material will reveal themselves. To cross the boundary imposed by its sources of knowledge, AI needs new observations, new experiments, and real feedback capable of correcting its judgments.

Embodiment is an important path, but the human body is not the only interface. Eyes, ears, and hands are a set of choices left by evolution in particular environments. Imitating the human body can bring machines into the living and working environments humans have already organized. If the aim is to expand the range of cognition, however, there is no reason to treat those choices as the final form.

Machines can connect to sensors beyond the visible spectrum, combine observations from different locations, and make measurements on timescales far faster than human reactions. Such systems remain constrained by energy, computation, communication, and physical processes themselves. But their limits need not coincide with ours. They may organize internal representations directly around continuous fields, probability distributions, or high-dimensional relationships, without first translating every intermediate step into sentences people can read.

New interfaces will not automatically produce new understanding, either. Data must be organized, judgments tested, and the system changed in response to feedback. Only when these processes are connected can information beyond the range of human senses become knowledge beyond existing concepts.

At that point, the conditions that initially made machines resemble humans may begin to loosen. Common sources of knowledge make corresponding concepts easier to form; increasingly divergent experiences may make those correspondences harder to maintain. Humans may verify some results without fully understanding the processes that produced them. They may use some knowledge without being able to incorporate it into their own intuitions.

This does not mean knowledge must become impossible to communicate. Translation, experiments, and new expressive tools may still bridge some of the distance. But neither the cost of such communication nor its ultimate reach is guaranteed simply because humans built the machines.

The lock on knowledge thus reveals another side. At first, machines are constrained by what humans can apprehend, express, and leave behind. Once machines acquire the ability to expand their understanding independently, humans may become the ones unable to cross that boundary.

The machine’s initial resemblance to us comes from our language; its later differences may come from its own history. Isomorphism lets us understand it, feeling compels us to ask whether it also experiences, and new knowledge may eventually lead us into different worlds.

In an age of earth-shaking change, may every intelligence see farther—and, having seen farther, may we still understand one another.

Translation Notes

• “同构” is translated as “isomorphism,” retaining the title’s connection to structure. The essay explicitly uses it for partial structural correspondences, not as a claim of strict mathematical isomorphism between humans and models.
• “感觉” is translated primarily as “feeling.” Depending on context, it encompasses sensation, emotion, and subjective experience; the translation preserves the essay’s distinction between functional roles and felt experience.
• “认识” is rendered as “cognition,” “understanding,” or “apprehension,” according to context. The term refers to the activity and organization of knowing, not only to an accumulated body of knowledge.
• “锁外” is rendered as “Beyond the Lock,” continuing the metaphor from The Lock on Knowledge. The closing sentence also echoes the earlier essay’s ending.