What sign language reveals about the mathematics of authored experience
Imagine two people having a conversation in American Sign Language. One is telling a story about a disagreement at work, between a manager and a junior colleague. There is no podium and no script, but their conversation has architecture. Watching them, you can see it.
The signer establishes the manager at a location to her right, the colleague at a location to her left. As the story unfolds, those locations help keep track of who is doing what to whom. A shift of her torso or gaze can help introduce another person's perspective. Her hands carry the action while her face participates in the telling. Some of what an English transcript would have to explain in separate phrases is being expressed simultaneously.
ASL has its own grammar, developed and sustained within Deaf communities. Handshape, movement, location, and orientation work together with signals from the face and body. A facial movement may carry grammatical information, such as marking a question; another may convey the attitude of a character in the story. The distinctions are part of what a fluent signer understands.
The listener follows these relationships as the story unfolds. The manager's perspective remains distinct from the colleague's, and both remain distinct from the storyteller's own. Space helps give the telling an organization that a transcript would have to reconstruct in words. Part of what the story means lies in that organization.
We rely on such relationships in spoken conversation too. Pitch, gesture, and pauses carry information alongside the words; a look exchanged before an answer can change how we understand it. Writing can preserve much of this through careful description. Yet a transcript can retain every spoken word and lose the irony, just as a summary can capture an outcome and lose the disagreement that made it matter. What survives depends on what the record was made to carry.
A Rep will work at this boundary whenever it helps us tell a story or draws on that story to speak on our behalf. The account may become shorter, take on another emphasis, or reach someone who knows nothing of its original setting. We need it to travel without quietly changing whose story it is or what happened within it.
I want to approach the mathematics of that problem through something we can watch. The signer's hand moves through space, carrying relationships that its final position alone cannot reveal. Following that movement gives us a way into the less visible structure of an experience remembered and retold.
A space for experience
Picture a tiny light attached to a signer's wrist. In a long exposure, the light would draw a trail. Its position at each instant has three spatial coordinates; adding the time gives a path through four-dimensional spacetime. That path records where the wrist went, how quickly it moved, and where it paused.
Now add lights to the fingertips, the other hand, and the head. Each draws its own trail. To describe the whole configuration at once, we need many coordinates: the positions of the tracked points, the bends of joints, the orientation of a palm. The body occupies ordinary three-dimensional space, but a mathematical description of its changing configuration can have dozens or hundreds of dimensions. Each dimension records another quantity needed to say what the body is doing.
This is a configuration space: a space whose points stand for possible arrangements of a system. One point can represent an entire posture. Movement becomes a curve through successive postures. Robotics uses this way of thinking to describe an arm reaching for a cup or a hand folding a shirt. A path that looks simple in the room may require many joints to change together.
Some configuration spaces can be described using manifolds. A manifold is a space that looks like ordinary coordinate space in a sufficiently small neighborhood, even if its overall shape is curved or more complicated. The surface of a globe gives the familiar example: a small patch can be mapped with two coordinates, although the whole surface cannot be flattened into one seamless map. Smooth configuration spaces let mathematicians study motion locally and connect those local descriptions into a larger account.
For signing, this geometry describes the moving body. Understanding the language also requires its conventions, grammar, and conversational context. Two similar paths can do different linguistic work; variation between two performances need not make them different signs. A change in location can be incidental in one setting and change a reference in another. The interesting task is to discover which differences matter, and under what conditions.
That makes the path important as well as the position. The same hand can arrive at the same place through a different movement, carrying a different expression. A system that records only the endpoint has no way to recover that distinction. Following the changing configuration preserves something about how the expression came into being.
Something similar happens when we remember an afternoon by its outcome. Imagine a student at robotics practice whose robot has stopped moving. She suspects a loose connection and checks the wiring. A teammate notices that a recent change to the code may be responsible. They test that possibility, revise the program, and get the robot moving again. Later, the student makes a Dote about the afternoon.
The outcome fits in a few words: the team fixed the robot. Much of the learning happened along the way. She formed an explanation, acted on it, encountered another person's idea, and changed her mind after a test. A photograph might show the repaired machine. Her account could show what it took to become willing to listen.
Another afternoon could end with the same working robot after she persisted with her original diagnosis and found the loose wire. The outcomes resemble each other. The experiences offer different grounds for reflection. A Rep helping her prepare for an interview should be able to distinguish evidence of persistence from an episode about revising a judgment.
A Dote can preserve the route by keeping these relationships available. The student's initial explanation belongs to an earlier moment, before the test; her teammate's contribution remains attributable to the teammate. Her later interpretation refers back to the afternoon without becoming identical to it. She may reconsider that interpretation as she gains experience, or remember a detail that changes her understanding of what happened. An authored record needs room for that movement too.
The movement invites us to think in terms of a trajectory, but now the space through which it passes has to be constructed. Physical coordinates tell us where the wrist moved. What coordinates would help us describe the movement from suspicion to evidence, or from another person's suggestion to a decision we make our own?
Some of the needed distinctions are already familiar. We can distinguish action from observation, and observation from interpretation. We can record who speaks, what supports a claim, and whether it is asserted, doubted, or quoted. Permissions determine where an account may travel. Revision history connects a current version to the account from which it grew. Together, these give a proposed representation more structure than a paragraph standing alone.
They will not all behave like spatial coordinates. A confidence estimate might vary gradually; withdrawing a claim is a discrete change. Revoking permission closes a route that was open a moment before. A workable representation may combine continuous spaces with typed records and graphs of relationships. The point of constructing the space carefully is to make consequential differences expressible, including differences that resist a smooth geometric picture.
This gives the transfer described in The Memory Glove a more demanding purpose. The machine is learning from an account whose relationships must survive becoming useful to it. In the student's Dote, knowing that the robot eventually worked is only a beginning. Her Rep also needs to know how her judgment changed, who helped, and which interpretation of the afternoon she is willing to stand behind. Those relationships determine what the account can responsibly become.
How an account takes shape
Suppose the student asks her Rep to help explain what she learned. Her telling, her teammate's observation, and the test result can all contribute, but they enter the account in different roles. If the Rep merges them into “I knew the code was wrong,” it has assembled a plausible sentence from the afternoon's ingredients while changing the episode itself. Helping her reflect requires a more careful way of putting those ingredients together.
The same problem recurs as her records accumulate. Several Dotes might become an account of a project; several projects might support a story about her development. At each level, the smaller pieces contribute to something larger. Mathematicians call this composition, and one of their languages for studying it is the operad.
Operads emerged in topology; J. P. May's 1972 book The Geometry of Iterated Loop Spaces helped establish their modern form. They describe operations with multiple inputs and an output, together with rules for substituting one operation into another. Picture a branching tree. The branches feed a node; its output feeds another node farther up. Grafting a tree onto a branch makes a more elaborate composition. For a fixed arrangement of operations, the rules ensure that carrying out substitutions in stages gives a consistent composite operation.
In a colored, or typed, operad, inputs and outputs come in different kinds. “Color” is the mathematician's label for a type. A slot requiring one kind of input cannot simply accept an output of another kind. For authored memory, this suggests a way to distinguish an observation from an interpretation at the point where accounts are assembled.
For example, a review operation could take the student's telling together with supporting material and produce an account she has approved. That account could then enter an operation that prepares a story for an interview. A model's suggested interpretation would travel through a different route, remaining a suggestion until she has considered it. Her approval would establish that the account speaks for her; checking its claims against other evidence would remain a separate task. The structure would keep those two kinds of validation distinct.
What interests me is being able to follow this structure as an account grows. The tree can become more elaborate while its branches still lead back to material with a known source and status. A longer narrative need not dissolve its ingredients into one undifferentiated voice. Building a Dote operad would mean specifying the operations through which that growth is allowed and checking their compositions against the mathematical rules. Ordinary typed records provide a practical beginning, making the distinctions explicit enough to inspect.
Even a simple rearrangement shows why the rules need care. Listing the two teammates in a different order need not change their contributions. Exchanging who suggested the repair and who performed it changes the story. Symmetric operads describe how permutations of input positions interact consistently with composition, while leaving us to specify what occupies those positions and which roles matter. The aim is to prevent changes in the assembly from silently becoming changes in the account. An input can still be mistaken or incomplete, but at least the resulting claim has an inspectable history.
Category-theoretic models of meaning have explored a related question in language: how grammatical structure and the meanings of parts combine into the meaning of a sentence. Authored memory asks us to carry more through that composition. The sources and permissions travel with the claims, so an account prepared for a new purpose can still be traced to the experiences from which it grew. A story told on my behalf should disclose how it became my story.
The same story told differently
Once the student has an account she recognizes, she may want to tell it differently. With a friend, she might laugh about how long she spent checking perfectly good wires. In an interview, she might emphasize learning to accept help. With a mentor, she might examine the sequence of tests. Each telling draws attention to something different in the same afternoon. Her Rep needs to help her make those changes while preserving the relationships that keep the account faithful.
The globe offers a precise example of how something can change while a relationship stays fixed. Turn it in your hands. Its orientation changes, while distances between points on its surface remain the same. Turn it again and the combined movement is another rotation. Every rotation can be undone. These transformations form a group; because they vary smoothly, with smooth composition and inversion, they form a Lie group, named for the mathematician Sophus Lie.
We can therefore describe a whole family of changes by what they preserve. Under rotations, a shape appears in many orientations while retaining its geometry. The collection of configurations reachable from one configuration under a group's action is called its orbit. Instead of treating each orientation as an unrelated object, the mathematics gives us a precise account of how they belong together.
Geometric deep learning uses symmetries of this kind to guide the design of models. A model can be built to respect a specified transformation instead of learning every variation separately. For Representational AI, the attraction is the prospect of identifying similarly precise relationships among versions of an account. We would need to say which differences belong to the presentation and which change what is being represented. Those choices could then shape the model's architecture and learning objectives, making the preservation of particular relationships part of how it learns.
Consider the difference between moving a camera that records the signing conversation and moving a sign within the space the storyteller has established. The first changes the camera's viewpoint on the same event. The second may change whom the sign refers to. Both involve spatial transformations, yet they have different linguistic consequences. Before a model can preserve the relevant structure, its designers need to understand that distinction.
The student's retellings pose the same question in a less visible form. The interview version can leave out the order in which she checked the wires and still give a faithful account of learning to listen. Crediting her with her teammate's discovery would change a relationship that ought to survive. Her emotional interpretation might legitimately change as she grows older, while the difference between what she witnessed and what she learned afterward would still matter. Preserving meaning requires us to specify which relationships an operation is meant to preserve.
This is also where the analogy with rotations reveals an important difference. Turn the globe back and you recover its earlier orientation. A summary has no comparable reverse move: its missing details cannot be recovered from the shorter text alone. Most retellings therefore cannot simply be treated as elements of a Lie group. Their relationships can still be represented as directed transformations, with links to fuller accounts. That lets us distinguish a reversible change of presentation from an edit that discards or introduces information, and keep a record of what the edit did.
I call the requirement Semantic Transport Coherence: as an account travels, its important relationships should survive, and consequential changes should remain visible to its author. The student should be able to see how an interview version grew out of her Dote and decide whether it still expresses what she wants to say. Her choices govern which versions are retained and where they may be used. Preparing one telling for a particular audience should not silently make it the authoritative account of her afternoon.
Some of the changes that matter most are very small. The student says, “I thought the wiring was wrong.” Remove “I thought” and her discarded explanation has become a finding. Almost all the words remain, but their relationship to the speaker has changed. The same thing happens when “suppose this is true” becomes “this is true,” or “I oppose this framing” is recorded as “this is my framing.”
This relationship is stance, the posture under which something is said or held. We explore ideas without adopting them and repeat other people's words precisely because we disagree. We can concede a point without accepting the argument around it. An account that carries the words but loses those relationships can give a fluent, apparently well-supported account of a position we never took.
In the signing conversation, the storyteller's body helps show whose perspective the telling occupies, while grammatical signals help distinguish kinds of utterance. An English account carries those distinctions through its own resources. If the manager's statement becomes the storyteller's position, the record has failed even when the quoted words are exact. The relationships made visible in signing space are the same kind of relationships a Rep has to preserve when it carries our accounts elsewhere.
The damage can compound when the altered account becomes evidence for the next one. A model retrieves the sentence about faulty wiring, builds it into a summary, and later cites the summary when asked what the student learned. A discarded explanation has become part of her history. I call this false canonization: a provisional or transformed account hardens into the accepted record without anyone choosing to make it so. The system becomes increasingly consistent about a mistake.
A record that carries stance gives the Rep something to check against that drift. It also leaves room for a person to remain unsure. We can be ironic or conflicted, or simply not yet know what an experience means to us. The Rep can ask whether an interpretation fits and preserve an open question as open. Meaning is not only what is said. It is how the said is held.
Patterns across a life
Keeping that openness becomes more important as the student's records accumulate. Over a year, she may return to the same difficulty in different projects, or develop a capacity she has not noticed in the individual episodes. Her Rep could help bring those relationships into view. But it would need a way to distinguish a pattern worth considering from one that appears only because of how the records were selected or compared.
One mathematical way to approach a related problem is to examine shape at more than one scale. Imagine dots scattered roughly around a circle. Put a small disk around each dot, then gradually enlarge the disks. At first they are separate. As they overlap, they join into connected pieces. For a range of disk sizes, their union may surround a hole. Eventually the hole fills in. The nested sequence is called a filtration. Persistent homology, a method from topological data analysis, tracks when features such as connected components and holes appear and disappear within it.
A picture taken at any one stage would show only part of this behavior. Following the sequence lets us ask which features endure across a range of scales. Those features can deserve closer attention, although interpreting them still depends on what the dots represent and how their distances were chosen. Persistence tells us something about the spaces we constructed; understanding those spaces requires us to return to the data.
In the student's records, the interesting resemblance might lie in how an episode develops. An early assumption meets a disagreement, a test changes the situation, and she makes a different decision. Another account could use entirely different words while following a similar course. The research question is whether a representation attentive to that structure could make useful relationships visible through topological analysis. A recurring pattern in a life is not automatically a mathematical loop; finding a useful correspondence would depend on how the episodes and their relationships were represented.
Suppose the Rep brings together several Dotes in which accepting help changed a project's direction. The student might recognize a growing ability to collaborate. She might also point out that the collection leaves out all the times she helped someone else. That response matters to the proposed pattern. It changes the interpretation, draws attention to missing material, and may change what the system should look for next. The accounts become useful through a conversation she can redirect.
To be useful here, the method would have to show why it suggested those relationships and offer something beyond simpler retrieval and comparison. A pattern earns its place in a person's story through evidence and reflection. Its mathematical persistence alone cannot confer that authority. The student remains able to recognize something in the account, reject it, or understand it differently as her life continues.
That continuing relationship is what I want the mathematics to support. Her Rep should be able to help her see more in the afternoon than either of them saw at first, while keeping a proposed interpretation connected to the accounts that support it. A new telling should have a traceable relationship to the edits that produced it, under her choices about retention and use. She can change her understanding of the experience without having an earlier version quietly rewritten for her. Preserving an account's structure makes room for that kind of growth.
Embeddings and language models can contribute to that system. An embedding is itself a geometric representation; what matters is what its geometry expresses. Two accounts might lie close together because they share a topic while differing in whose judgment changed or whose help mattered. A model designed around the structure of experience could learn to attend to those differences. Developing the mathematics means making the relationships explicit, testing what systems learn from them, and inviting the people represented to show us where the resulting account ceases to make sense.
This is the sense in which I am looking for a different substrate. An account's structure would participate in the computation itself. The system might learn relationships among episodes, compose an account through operations that keep sources distinct, and revise a proposed implication when its supporting evidence changes. A generative model could put the result into words. The mathematics offers ways to make these choices precise enough to build and test.
The signing conversation remains a useful image of what we are trying to hold together. The manager and colleague occupy different places, and the storyteller moves between their perspectives while the listener follows who is speaking and what is in question. Movement, relationship, and the posture of a claim belong to the telling as it happens. A record worthy of the exchange must find ways to carry them forward.
Representational AI asks us to give authored experience that kind of structural attention. The next chapter takes up the vocabulary with which we might do it: how to name the parts and relationships clearly enough that a person and a machine can inspect the same account. That is the next step in semantic assembly, the work of making a larger meaning while keeping faith with the experiences from which it is made.
Sources
Edward S. Klima and Ursula Bellugi, The Signs of Language (Harvard University Press, 1979); Scott K. Liddell, Grammar, Gesture, and Meaning in American Sign Language (Cambridge University Press, 2003); Karen Emmorey, Language, Cognition, and the Brain (Lawrence Erlbaum Associates, 2002). Background on spatial reference, perspective, and the interaction of manual and nonmanual expression. See also Carol Neidle and Robert G. Lee, “The Syntactic Organization of American Sign Language: A Synopsis,” ASLLRP Report 12 (July 2005): https://www.bu.edu/asllrp/asllrpr12.pdf
J. P. May, The Geometry of Iterated Loop Spaces (Springer, 1972), especially the definitions of operads, substitution, and equivariance: https://www.math.uchicago.edu/~may/BOOKS/gils.pdf. Further treatments include Martin Markl, Steve Shnider, and Jim Stasheff, Operads in Algebra, Topology and Physics (American Mathematical Society, 2002), and Jean-Louis Loday and Bruno Vallette, Algebraic Operads (Springer, 2012).
Bob Coecke, Mehrnoosh Sadrzadeh, and Stephen Clark, “Mathematical Foundations for a Compositional Distributional Model of Meaning,” Linguistic Analysis 36 (2010), pp. 345–384: https://arxiv.org/abs/1003.4394. See also David I. Spivak, Category Theory for the Sciences (MIT Press, 2014).
Brian C. Hall, Lie Groups, Lie Algebras, and Representations, 2nd ed. (Springer, 2015). See also Brian Conrad, “Smoothness of inversion,” Stanford Math 210C: https://math.stanford.edu/~conrad/210CPage/handouts/inverse.pdf
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković, “Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges” (2021), on the use of geometry and symmetry in neural architectures: https://arxiv.org/abs/2104.13478
Gunnar Carlsson, “Topology and Data,” Bulletin of the American Mathematical Society 46 (2009), pp. 255–308. Afra Zomorodian and Gunnar Carlsson, “Computing Persistent Homology,” Discrete & Computational Geometry 33 (2005), pp. 249–274: https://geometry.stanford.edu/lgl_2024/papers/zc-cph-05/zc-cph-05.pdf. For an application to text, see Xiaojin Zhu, “Persistent Homology: An Introduction and a New Text Representation for Natural Language Processing,” IJCAI 2013, pp. 1953–1959.
The treatment of stance — assertion, supposition, opposition, concession, doubt, and quotation as elements of semantic structure rather than tone — draws on the linguistics of stance, evidentiality, and modality. See John W. Du Bois, “The Stance Triangle,” in Stancetaking in Discourse: Subjectivity, Evaluation, Interaction, edited by Robert Englebretson (John Benjamins, 2007), and Alexandra Y. Aikhenvald, Evidentiality (Oxford, 2004). On the voicing and reported-speech dimension relevant to quotation, see Mikhail Bakhtin, The Dialogic Imagination, edited by Michael Holquist (University of Texas, 1981).