Ed.blog

This post is a birthday gift for Matt Mullenweg, who just turned 40, and modestly request blog posts as gifts. Matt cofounded WordPress, on which this blog is written; he’s been a wonderful conversational partner on topics of consciousness over the years; and he is also one of the chief sponsors and enthusiasts for Sonic Sphere, one of the topics of the below.

Happy Birthday, Matt, and may your second quartet of decades exceed the first in substance, style and serenity.

——–

A treasured early experience from my twenties was being a member of Kevin O’Regan‘s extraordinary laboratory of consciousness in Paris. I’ve ever since aspired to being in an atmosphere of such undiluted creative boldness: a place where passion, ideas, and agency thrive in unfettered flourishing.

It was formative intellectually and personally, and also for the inspirational example it provided for how a culture and way of working can animate and enhance the work being done when they are harmoniously attuned (in this case, by the central theme of agency). I’ve tried ever since to nurture teams to work with a comparable level of inventive joy, with elusive results.

I was mulling earlier today the internal connection between the highly agential culture of that lab and the chief topic we were investigating (the role of action in constituting consciousness). And I was struck by the thought that it surely wasn’t a coincidence that the message and the medium were in such harmonious alignment.

And that thought in turn has gotten me thinking about what we’re up to at Sonic Sphere, the culture of the team we’re growing, and the invention that we’re attempting to bring to the world: one that helps people experience richer horizons of conscious experience. Given the constitutive role of agency in consciousness, it even has me thinking that we need to give as much emphasis to interaction as to light and sound in our iteratively evolving invention- about which, more below.

So in the weave of these ideas, let me try to identify some patterns.

The O’Regan lab

For those who don’t know him, O’Regan is a physicist-turned-psychologist-turned-philosopher (gradually climbing the tree of intellectual heroism) who published in 2001 a theory of consciousness so good that when I first read it I became breathless with a kind of ecstatic intellectual eros.

The theory seeks to resolve what is often (pompously) declared to be the greatest unresolved mystery in human affairs, namely the ‘Hard Problem of Consciousness‘, i.e. why we have subjective experience.

The Hard Problem is always articulated with reference to the mysteriousness of secondary qualities (colours, smells etc), and it is in the explanation of these that the sensorimotor theory changes the game.

The thing to be explained is first of all the qualitative differences between modalities: the redness of red, the ineffable subjective feeling of the smell of a rose, why the texture of sand-paper is so different in quality from the taste of cheddar etc etc. Equally, differences the kinds of experiences these modalities individually produce (redness vs roundness in vision, for example) is the object of inquiry.

The sensorimotor theory explains these differences in startling simple terms, by proposing that can be read off from the different ‘sensorimotor contingencies’ that the embodied functionality of those modalities imply: the different patterns of movement they permit, and how that movement invokes further sensory stimulation, and how implicit knowledge of those dependencies between movement and the feedback from that movement sculpt shapes through time that can, perhaps, give us a fecund view of the qualities of our various modes of experience.

A simple example of sensorimotor contingencies (for context)

The simplest experimental example to get one’s head around the concept of a sensorimotor contingency can be found in experiments done by the great psychologist Paul Bach-Y-Rita, inventor of the world’s first sensory substitution device, called a Tactile-Visual-Substitution-System (or TVSS).

The purpose of the device in question was to translate visual information into tactile information. The (blind) subject would wear a pair of glasses with a camera attached. That camera would translate its visual images to a pad of 100 * 100 vibrating pins, which recreated in real time a tactile ‘picture’ in patterned vibration. A bright object in the middle of the visual field would, for example, produce a strong vibration in the middle of the pad. If the light were top left, that’s where on the tactile pad the vibration would be strongest too. Thus, the TVSS device would manifest roughly the same info that is normally found in vision onto the skin.

It turned out in the course of these experiments that subjects could gain a strikingly compelling sense of vision through touch.

The critical observation Bach-Y-Rita made, however, was that the tactile information alone was useless to participants if they didn’t control the camera. When someone else controlled the camera, no matter how much time was spent, it was impossible to gain any purchase on the meaning of the tactile input, which was experienced as a kind of random pattering.

But when subjects did control the camera (i.e. when it was mounted on their glasses and they could freely move their head and body) they were able to make sense of the stimulation. A bright object in the centre of the perceptual field would swing left on the tactile image as they moved their head to the right, and vice-versas. Subjects found that after a developmental process of thousands of movements, and the learning of the sensory consequences of those movements, that the tactile information began to gain meaning and form, and that a distinctively 3-D feeling for the world would emerge- within hours of first donning the device.

Soon subjects would forget that touch would from the pad was touch at all. Instead, the sense of a world of tables, walls, people, depth would emerge in their experience, their consciousness. A distinctively visual qualitative experience- but deriving through the body.

What accounts for the visual feel of the experience? Not visual the cells in the retina, nor visual cortical brain regions (neither of which is involved) and not of course the raw form of the input (which is tactile). Instead what is common between a distinctively visual experience of the world in the TVSS and the normal sighted experience, is instead the structure of the underling patterns of interaction, and the subject’s mastery/implicit knowledge thereof.

The lessons even from this one example go wide and deep, but the fundamental one is that input alone is not information by itself. The meaning of the input has to be constrained by its context in a sensorimotor nexus. Up, down, left, right, near, far, these are concepts that are only given reality in human perception when integrated into a sensorimotor nexus. We take them for granted when we think about vision, but we do so at our intellectual peril when we forget the developmental sensorimotor process that determines the code for such input.

The same conclusion can be reached also from first principles (for all information requires a code), and it is this: that the fundamental unit of perception isn’t sensory, it is sensorimotor: input is infinitely ambiguous until contextualised against the dependencies of action, and active interaction is necessary to decode energy and turn it into information.

It is difficult to overstate how profoundly this simple idea shapes one’s thinking on the nature of perception, memory, consciousness, and our human relation to our physical embodiment. It truly calls for a paradigm- shift, a new way of organising the world, a simple thought shift after which nothing looks the same.

And the fun thing about this insight (in 2005, and indeed today) is that it is simultaneously obviously correct and woefully underspecified. It opens doors to an incomparably richer intellectual engagement to the nature of consciousness than the then-and-now dominant dualistic information-processing metaphors, but at the same time a thousand questions remain(ed): what exactly are colour, space, smell, texture, taste, memory? How to formalise a philosophy and science of mind were the fundamental grounding was not an internal state (some neurons lighting up, a ‘representation’ of the world) but an attunement to or implicit knowledge of patterns of interaction?

And it was within these wide open fields of intellectual possibility that a ragtag crew of brilliant young scientists had come to roam. And they found an intellectual atmosphere of great freedom and agency.

O’Regan’s Laboratory of consciousness

Kevin O’Regan, running the lab, had many intellectual virtues, chief among which was to embody (as he does even now at the age of nearly 80) a child-like open-mindedness constrained by a purity of intuition for rich explanation.

I recall relating to him that I had been disappointed by Bergson’s Matter and Memory because Bergson seemed to think that memory existed outside of the head. Kevin reproached me for my closed-mindedness, ready as he always was to approach a radical idea with total receptivity.

Meanwhile, there were perhaps a dozen grad students wandering around the lab, each of whom was boldly exploring the consequences of the theory in some fun direction. Collectively, they were the actors in a kind of utopian

Some anecdotes to illustrate the vibe of the place.

Aline Bompas, pictured below, spent a year wearing specially created sunglasses that were yellow on one side and blue on the other. Her aim was to investigate if the adaptation that would inevitable take place (the eye learning in essence to discount the colour bias) was motor-dependent, i.e. when she made eye movements after finally removing the glasses, would the world turn blue when she looked right, and yellow when she looked left?

The there was David Philipona, an ingenious mathematician, playing with Sony’s AIBO robot and programming it to follow the path of maximum learning in the uncovering of sensorimotor contingencies. The little robot dog would from the outside look like it was expressing curiosity and boredom as it played with an object for a while before apparently losing interest and clumsily pootling off to the next object of examination. A startling evocation of animal-esque subjective experience in a simple robot (in 2005!).

Conversation would turn to how a neuron knows it is a sensory vs a motor neuron (it doesn’t, without sensiorimotor context) and the kinds of algorithm that the brain needs to ‘discover’ the dimensionality of the world, the difference between sensory and effector, the distinction between modalities. The answers are worth a read. (And it is interesting to note that in contemporary machine learning and robotics, companies like Tesla and Wayve have finally discovered that architectures that combine perception and action integrally are the path to successful real world agency).

My own research was on creating illusory pain and out of body experiences (using variants of the rubber hand illusion) and led to some memorable scenes, including successfully making a series of young students experience their bodies as being 1/10th their actual size, and the world correspondingly immense.

And meanwhile, there was a brilliant Canadian neuroscientist, Paul Reeve, as welcome as anyone else in the lab, who thought this was all nonsense and with whom we spent endless hours defending and honing the ideas and dreams of the theory.

The whole place had the feeling of a conclave of mad scientists, where no question was out of bounds.

If I try to summarise the conditions under which this explosion of creativity emerged, I might identify three factors:

1. A wide-open horizon of discovery

The first reason the creativity was so exotic was that a new theory had opened up a new way of thinking, and thus there was a wide field of discovery lade bare before us: lots to do.

Part of the reason that the vibe was so good was how obvious it was that we were on a path to truth, and how little this was recognised by the broader academic community (who were stuck at the time, and continue to be unaccountably muddied by, the endless seductions of dualism and the mereological falacy).

It’s fun being right when outsiders don’t get it: both because of the camaraderie, and because it feels more playful, collaborative and peaceful to be simply after the truth than competing for dominance (in business or

2. A diversity of personal and intellectual styles

The lab was extremely diverse, in terms of gender, race, and intellectual specificity. Neuroscientists, cognitive scientists, mathematicians, roboticians, physicists, philosophers all bustled and argued and invented.

Research output included mathematical proofs, cognitive and behavioural experiments, philosophical thought experiments.

It’s almost a cliché to praise diversity, for it is so obviously productive and just and in the service of genius; yet it remains startlingly rare to find it truly expressed in the service of effective worldly action.

3. A virtuous balance between centralised / decentralised

The lab was centralised (under Kevin) but decentralised in the sense that everyone was free to pursue their intuition, to get into the swing of their own agency, and to pursue whatever passionate idea attracted them in the context.

From this example alone, one might take the distilled ingredients of a context capable of maximising intellectual agency, as these three:

1. Rich possibility (animated by pursuit of a secret truth)
2. Diversity of participants
3. Balanced decentralisation

One can see that all three of these factors were in the service of enhaced agency: rich open fields of possibility animated motivated effort and playful exploration; a diversity of participants ensured constant exposure to new methods of though and action, new ideas and perspectives; and the autonomy of the participants in a balanced decentralised system meant that everyone could do their own thing, express their particular genius, while individual discoveries and ideas super-additively inspired and amplified the other pathways.

In this way, the lab was an ‘engine of consciousness’ devoted to the study of consciousness: a place where individuals could expand their minds in pursuit of truth, and perhaps this was somehow connected to the underlying thought-space, where we were all hyper-attuned to the neglected role of action in the constitution of consciousness.

Sonic Sphere

So right now I’m busy with a gloriously ragtag bag of inventors building out the Sonic Sphere concept, which aims to be an engine and indeed a laboratory of consciousness. I’m keen to apply my learnings from Kevin’s lab, and indeed the project is infused with a connected energy.

The basic idea of the sonic sphere is that it is a spherical concert hall / media plane, in which the audience lies surrounded on all sides by light and sound, which can be sculpted into wholly new architectures.

The original vision was by Stockhausen, from 1970, and we’ve been iteratively developing the concept for the last three years. In doing so, we’ve done our best to establish the conditions of maximal agency in the team.

Agency in the Team

Here a similar trio of agency-enhancing factors are in play, as were in O’Regan’s lab.

1. Rich possibility (animated by pursuit of a secret truth)

The goal is inherently very rich, since -with the possible exception of certain nightclubs- spaces of public gathering have fallen far behind the digital technologies which collectively shape our imaginations.

The invention of a digitally-native audiovisual concert hall in which the audience is truly immersed not just 360 (which is just a 2-D concept) but 360/360 opens up almost immediately a treasure-trove of interesting possibilities.

Now the complete spherical audiovisual media plane is already an extraordinarily high-impact stimulating device. And with the audience in an essentially directionless sphere of stimulation, in which every aspect of the audiovisual environment can be sculpted in the service of new experience, a whole new world of possibilities obtains.

For almost twenty years, people have been experimenting with spherical media formats, and of course Stockhausen’s original Kugelauditorium dates from 1970, so the ideas are by no means entirely new. Yet we’re still clearly incredibly early in unlocking the potentialities of such immersive media, and it’s not an exaggeration to say that almost everything remains to be discovered. We stand on the brink of a positive explosion of technique and knowledge in the service of creating new spaces of consciousness.

2. Diversity of participants

In building a machine that has architectural, structural, visual, auditory, software, experience-design, musical, sound-design, a team of dazzling diversity has -almost entirely on a volunteer basis- been attracted to the project.

To mention but a few of our team / collaborators:

• Jessica Lair, an American opera-singer turned software developer
• Nicholas Christie- a structural engineer whose past work includes designing the kinetic aspects of the world’s biggest Ferriss Wheel
• Merijn Royaards- a Dutch musician / sound artist / architect / philosopher whose PhD studied the connections between the Russian Avant Garde’s experiments in sound and light and those of the 1980s rave scene.
• Alex Bondarenko, Stephane Lee, Rob Rowland, three dazzlingly gifted structural / electronic / mechanical / software engineers who build autonomous electric flying cars in their day job
• Chris LeJeune, a pioneer of spherical video.
• Andrew Melchior, once CTO for Massive Attack
• Patrick Rowland, a 24 year old electrician from Toronto.
• Uniiqu3, the Jersey Club Queen who made epic 3D music in NYC
• Rachel Harris, who created an extraordinary set of light cloths for our NYC sphere
• Madame Gandhi, a pop star who builds her own underwater microphones and has created epic arctic soundscapes for the sphere.
• Fatemeh Miri, a software developer who led our entire lighting system design / build / configuration / programming in NYC
• Mark Slee (the author of an incredible LED-lighting rig)
• Jeremy Guillory, an autonomous vehicle inventor and entrepreneur.

And while the intellectual and practical and geographical diversity here is inspiring, it can be substantially enhanced in time!

3. Balanced decentralisation

Given the need for so much excellence, it is quite easy for many parallel furrows of genius to be ploughed all while the fact of sharing the same field to force collaboration and mutual challenge.

For the London sphere we’re plotting, we’re simultaneously innovating the architecture, structure, speaker arrangements, spherical screen, and content. Each is an independent path, and at the same time each must interact with and take inspiration from the others.

So the conditions are well set for a good balance between a singular centralised vision, and a lot of opportunity for autonomous invention.

Audience agency inside the sphere

Our aim is to pioneer a new space, a kind of ultimate ‘engine of consciousness’.

By dint of unerring commitment to a full spherical media plane, we have in some sense a mathematically unbeatable complete immersion in sound and light… for what is more immersive than a sphere? With skilled sound engineering, robust audiovisual correlations, ingenious art, this alone creates a platform through which artists and musicians can surely vehicle audiences to new spaces of consciousness.

But this much could just be passive, receptive. And we know that only by invoking the agency of an audience can the deepest results be achieved. And so the question of agency for the audience while having the experience of being in the sphere, naturally becomes a massive point of research for us.

For we know that the story of enlivened consciousness goes far deeper than the act of mere provocation by a stimulus, no matter how awesome and sophisticated it may be.

Indeed, according to the insights of sensorimotor theory, perceptual input is merely a prompt for the evocation of imagination / the activation of a rich set of expectations, the conjuring of a world of expectation. That is not to say the stimulus isn’t important, for the mere application of sound and light can produce exotic effects (cf the history of cinema), but by itself sound and light at greater levels of immersion are just more and better of the same, and inherently limited in their power.

This insight is the (correct, in my view) critique of related efforts to our Sonic Sphere, such as the MSG sphere in Vegas: that it just creates another, bigger, more dazzling, screen. That it therefore reproduces a distinctively masculine and dominant perspective on the world, imperial vs interactive, and that the docile audience with their phones out to record the experience are not as close to the richest pastures of their own experience as they could certainly be.

So if our intention is truly to propose a revolutionary engine of consciousness, there must be inherent space for agency in the experience of the art within. How will this manifest in our spheres?

The agency of audio

The first thing to say is that audition inherently leaves much more space for the imagination (a continuous, integral part of all perception, even in ants) to lean into than vision, due to its lower spatial precision, more qualitative relationship to reality, and the fact that sounds do not occlude each other, so sonic spaces are capable of being dramatically more layered and impressionistic than visual ones, which have the habit of reproducing the superficially cartesian space of environments.

In this way, leading with audio is already a choice for agency, since the roving imagination can project into and situate itself within already far more advanced spaces of meaning than it is easy to convey by visual means.

The direct perception of artistic agency

One mode of experience of agency is that borrowed by the audience from direct perception of a performer. When you’re watching a live performer (whether dancer, singer, DJ, pianist) you often directly experience the miracle of their inventive genius as if it were flowing free from your own soul. This is why cellists are so sexy. There is some quality of mirror-neuron-ish empathetic alignment with a live performer that, despite the fact the audience is just sitting in a chair, gives an experience of agency.

Internal architecture

Clambering across nets, and having a surface amenable to moving, dancing, interacting already transforms a space helpfully in the direction of supporting active inhabitation by its audiences.

Our direction of travel, then, is towards no fixed seating, complete internal reconfigurability and a floor that invites movement. This alongside a culture of participation, will doubtless convey us in interesting directions.

Audience Participation in the art itself

One of the finest and most interesting events at our experimental sessions at The Shed in NYC was an interactive choral experience led by the composer Rolfe Kent.

Rolfe led 200 people in a choral round, as they also moved through the space, and as their voices were projected back in three dimensions. It was an experience of startling depth.

The first example of participation when it comes to music is of course dancing, and that is one reason why nightclubs have for decades been the most interesting zones of experience and innovation in public space.

Audiovisual Interactivity

A very simple form of interactivity in Sonic Sphere is the application of a digital echo. If you’re in, say, a rendering of Chartres Cathedral and your shouts or footsteps echo back at you as if the sound has vehicled its way around the vast stone structure with 3second reverb, there is an unmissable amplification of reality: this is no longer a mere sensory stimulus, but a sensorimotor one.

But one can imagine the interactivity going much further. Thinking back to the experiments of the Russian avant-garde, take the Terpistone: an instrument that directly

Massively multiplayer interactivity

This is perhaps where things will really get going, when a multitude of audience members co-participate in the evocation of rich states of consciousness.

We already have a fantastic example of this in contemporary culture: the club dance floor. A dancefloor at a rave is an example of how light, sound, and movement combine to create exotic architectures of consciousness by means of collective action.

The synchronous movements, flashes, and beats create a participatively sustained experiential space of consciousness within which (sometimes absurdly peaceful) meanings can be collectively sculpted. Without the movement, the cogency and clarity of the space of shared perception would be impossible to attain, and out with that would go the meaning too.

So with this we already have a concrete example of how a massively multiplayer interactive audiovisual consciousness-engine works. A parallel example it might be worth keeping in mind is the experience of singing in a choir: where the action (singing- altering the tones emitted from the voice) is in a non-euclidean space of harmony defined by the other voices, and the experience of the beauty of the art is amplified internally by the act of participation compared to merely listening. And of course, the evidence goes to show that even when we listen, we sing implicitly.

It is clear that it should be possible to extend this principle in all sorts of fascinating and interesting ways.

a) Imagine if everybody’s heart-beats -visually perceived in infrared by omnidirectional sensors- directly affected the entouring spherical visuals.

b) Imagine if the sphere were a Terpistone played by hundreds simultaneously…

c) Imagine if we could entirely eliminate the DJ, and the audience could collectively dance the set into existence…

d) Imagine if communities could through new multiplayer visual interfaces conduct public debate in truly democratic, mutually comprehensible, real-time fashion?

From these examples, I think our tripartite rubric for when agency is fully supported also applies to Audience Experience…

1. Rich possibility (animated by pursuit of a secret truth)

We can see how the wide range of experiments and influences audiences can have on the, all animated by the secret truth: you are a part of the art.

2. Diversity of participants

We’d have to let in the weirdos, the bankers, the grannies, the hipsters, the burners, the bakers and the children.

3. Balanced decentralisation

Technology and norms for including the audience in the experience without obliterating the agency of the artist also.

Agency for all participants in the ecosystem

If we do with Sonic Sphere what we hope to do, which is to prove it out as a format and make it available as widely as possible across the towns and cities of this world, how will we attain reasonable agency for all participants?

How will the spheres be designed, built, operated, monetised? What is the accessibility and programmability of the platform we’ll be able to offer to creatives and communities and audiences and owners? And how differently might the conditions be expressed at the different sites?

If our aim is to amplify consciousness, it must be to produce reasonable and productive agency for all participants. (This will always, I think, be an Aristotelian virtue, in the sense of a balance between two extremes demarcating the desirable behaviour, in this case of total anarchy and total domination by HQ).

(Disclaimer: I’ve made the mistake before of trying to innovate both product and institution at the same time, and wound up in a less favourable place to if I’d just stuck to the basics of the product. Entrepreneurship is astonishingly difficult under any circumstances, and one always wants to incur no unnecessary complexity, as it can be mortal. So please take none of the following (or indeed preceding) explorations to be a commitment to how we’ll wind up operating this institution, but rather as some ideas we’d love to make real, if our competence and the exigencies of the world allow for it).

Artistic agency (diversity of participants)

A space of art dominated, as cinema is, by sequels and big-budget spectaculars is not truly vibrant, reduces competition and quality and desicates the fields of artistic opportunity available for practitioners. How I’d love to see indie films made locally, but the dynamics of cinema-economics seem not to permit this.

One ideally wants artists to be able to fully express themselves in as permissionless and accessible way as possible, and also to offer them a paying audience sufficient to reward them for their brilliance (a steady feature of reality over thousands of years seems to have been the outrageous economic oppression / undervaluing of artists).

The use of open-source technology, and well documented tools and learnings in a context of a friendly and open community of practice is going to be key to an agency-first approach: at the top of the funnel, there must be as many investors able to try new ideas as possible.

There must be an efficient 16-year-old-in-a-bedroom tool-set (probably involving VR) that would allow any kid to create dazzling shows, show them to others, and enter them onto a pathway to shows in real world spheres.

A secondary technological factor here will be highly integrated and potent generative-AI tooling. The work to make an amazing Sonic Sphere show with so many speakers to exploit, pixels to fill, interactions to define… this would if one were manually making each decision be the task of a thousand artisans for a year. Artists are then going to need to have hundreds of thousands of AI interns to flesh out and express their visions.

And of course, despite the economic temptation to play risk-free big-name nostalgia-powered shows that are guaranteed to sell out, a high proportion of programming timespace must be made available for experiment, risk and local participation.

Finally, the spheres themselves must offer as much artistic affordance as possible.

Agency for venues (diversity of venues)

If sonic sphere is to fulfil its potential as a new kind of cultural space, the freedom of operators of individual sonic spheres will then also be key. There is some balance of centralisation and decentralisation which can ensure consistent experience (so that one knows as a visitor that one will be experiencing something of predictable quality) without wiping every hint of individuality from each.

High levels of agency for the teams running spheres in programming, community building and developing new genres and formats for this new experience will again be in the service off the underlying aim of the project: to be an engine of consciousness, a flourishing-ground for new forms of ideas, community, discourse, experience.

From our London HQ it is surely impossible for us to imagine what the Sonic Sphere in Mexico City should be experimenting with. There will have to be constitutive local empowement to control what and how .

And of course ideally, that agency will extend to the economic. If the largest practical portion of $1000 of local revenue goes to the artists/operators/staff , all the better for consciousness.

Agency at all levels

Alrighty. so far we’ve toured our way through:

1. How agency is at the heart of consciousness at the lowest level, and up all levels flowing beyond.
2. How Kevin O’Regan’s consciousness laboratory reflected this insight in creating a space of pursuit-of-an-exciting truth, with a great diversity of participants each enjoying well supported autonomy.
3. How Sonic Sphere likewise aims to be an engine of consciousness, and must therefore maximise agency at all levels.
4. How our team structure has evolved to support team agency, and can continue to do so
5. How the sonic sphere space can encourage audience agency
6. How the agency of artists and the people who run the sites.

Sounds pretty cool in my view. Let’s see if we can make it happen.

Happy Birthday Matt, thanks for being such a wonderful supporter of this project, and wishing you many a nourishing journey into the hyperspace of your consciousness in the bosom of future sonic spheres.