Introduction
Extended Reality can be used in public spaces, it can be seamlessly integrated into our everyday life, and it has the potential to change entire societies.
Humans are social beings. As such, they desire relationships with other people and a sense of belonging to one or more groups. Being part of such groups is essential for their identity finding, self-esteem, and individual well-being. In this respect, it is no wonder that most people look for social contacts wherever they are―leaving aside one or the other escape from reality. From the very beginning of the Internet, its users have always looked for opportunities to socialize in virtual spheres, whether in VZones on the early Internet, in Massively Multiplayer Online Roleplaying Games (MMORPG’s), in chat rooms, or in precursors of what is now called a metaverse, e.g., in Lucasfilm’s Habitat or Linden Lab’s Second Life [1].
However, if we analyze the current direction of Virtual, Augmented, Mixed, and eXtended Reality in general, we notice an extremely one-sided orientation: Researchers and companies alike are striving for small or large virtual worlds, for tinyverses [2] and metaverses [3], which, however, are mostly virtual spaces. Usually, three basic assumptions are made:
It is always the user who enters such a virtual space instead of having it come to him or continuously surround him.
In order to participate, every user needs wearable technology such as head-mounted displays, controllers, and possibly trackers.
All these virtual spaces are limited―sometimes to several million people (massively multiuser), but never so large that the entire population of a big country or even the whole world could participate. Also, those people who cannot wear the technology they need on their bodies, such as some people with disabilities, young children, or people who cannot cope with the technology, such as the elderly, are always left out.
Approaches to overcome these three limitations are hardly explored. Social XR [4] allows several people or a group of people to experience social presence and co-presence by engaging in activities and real-time conversations with each other. In some cases, the participants wear head-mounted displays, are almost completely shielded from their real environment, and are embodied by avatars [5], accepting that the use of avatars can change the user’s own and other users’ behavior [6]; in other social spaces, the participants are recorded by the camera so that photos or live videos of them are integrated into virtual environments. However, these social spaces are always located in a virtual world. Spatial Augmented Reality; on the contrary, takes place in real environments [7]. It allows individual users and relatively small groups of participants, up to a few thousand people, to experience augmented reality in a limited but real space, usually in one common location. It does not require any participant to wear technology on their body, because the necessary equipment, at least powerful projectors and speakers, if necessary, are integrated into the environment instead. For this reason; however, it is not or only slightly interactive.
Figure 1 shows a classification of the Virtual, Augmented, and Mixed Reality, Social XR, and Spatial Augmented Reality paradigms on a three-dimensional diagram with the axes:
Technology mainly in the environment vs technology mainly worn on the body
Anchored in virtual space vs in real space
The number of potential users ranging from individual users to a small group, a massively multiuser experience, and availability for everybody.
Figure 1 makes it clear that although there are XR approaches for the presence of users in both virtual and real spaces, almost all of those approaches only allow users to expand reality who wear corresponding input and output technologies directly on their bodies. An exception here is Spatial Augmented Reality [7], which is mostly based on projections and projection mapping, requires a correspondingly large amount of preparation time, is spatially limited, and most often not or only slightly interactive.
This vision paper, therefore, proposes a new form of XR: Societal XR (highlighted in green in Figure 1). Its goal is to make XR interactively usable and experienceable for really everyone, by not having to immerse people in virtual worlds, but by bringing the virtual offers to them into the real world. This is made possible by cameras, tracking systems, spatial audio speakers, projectors, and displays of all kinds in the real environment. It also requires modern approaches to semantically modeling intelligent environments (such as [8]), interpreting user behavior, storing user profiles, generating user interfaces at run-time, automatically choosing, and applying XR patterns (see [9]), and many more.
Societal XR is intended as a public form of XR that can also be perceived by bystanders and passers-by and not only by its respective users. Yet it can serve individuals as well as public and societal benefits. For this purpose, it integrates aspects of Spatial Augmented Reality, Mixed Reality, Social XR, Ambient Intelligence, and Ubiquitous Computing [10], among others.
A Societal XR Story
It was a beautiful Saturday in May, with bright sunshine and pleasant temperatures, when Claire left her apartment, closed the front door behind her, and headed to the park to call her friend Emilie. The sidewalks and playgrounds on Claire’s street were busy; apparently, the whole city wanted to enjoy the wonderful weather. The shops were open and invited potential customers to buy with projection shop windows and virtual mirrors. Passers-by stood in front of almost every shop. Some got carried away by the personalized ads, pausing to look at promotional holographs chosen for them based on their customer profiles. Others stopped in front of clothing stores, pointed to items, and were then able to look and move in the virtual mirrors of the shop windows wearing exactly those items of clothing. The latest craze, however, was holographic fittings that allowed passers-by to point to one or more items of clothing, which were then projected onto their bodies using holographic projectors. So not only could the passers-by see themselves in these clothes, but also companions, relatives, and other people in the vicinity could. The passers-by could even walk a few meters in their holographic clothing until they left the range of the respective projector.
Claire was lucky to live on such a popular shopping district, which meant there were projectors installed all over the street. They were all connected to each other. While each store also had dedicated projectors so that only those projectors could project that store’s merchandise, advertising, and other content, the projectors commissioned and installed by the city administration were also networked, distributed across the city, and were therefore capable of holographic projections maintained almost seamlessly even over several kilometers. Many residents used this function as a matter of course. Just as Claire was about to start walking, two joggers ran past her together: one really present, the other projected. Of course, they both jogged, but not in the same place. The second jogger could be anywhere, on another continent or on the other side of the world. But they both jogged in straight lines, so one was projected next to the other. With the help of either headphones or spatial audio, they could hear each other. Via microphones or smartphones that they wore to record their words, they could talk as if they were really jogging side by side. Claire had tried it herself a few times, even though jogging was not one of her favorite pastimes. For the first half hour she had had trouble with traffic lights and obstacles, because whenever her running partner had to stop at traffic lights or obstacles that Claire could not see, she also had to stop, next to her running partner, in the middle of the sidewalk. But that was fine. She got used to it on their first run together.
After two minutes of leisurely strolling, Claire arrived at the travel agency where she had already booked her last summer vacation. In her early twenties, just a few years ago, Claire had never thought she would set foot in a business as old-fashioned as a travel agency; after all, she could have had her trips booked online. But since travel agencies had started offering virtual trips, virtual trip previews, and holographic travelogues, they had become more popular. Some travel agencies had even set up or rented adjoining rooms in order to be able to offer customers the most convincing, interactive projections. They did not yet agree on the names of these rooms: Some still used the old term CAVE, i.e., Cave Automatic Virtual Environment, while others preferred the term holo chamber. In any case, these rooms allowed their customers to be truly present in completely realistic-looking recordings or live transmissions from foreign locations.
Of course, the travel agency’s cameras had recognized and identified Claire a few meters earlier. Servers had then identified Claire as a recurring customer, compared her customer profile at the travel agency, her advertising profiles on social media and the current summer trends. Based on this, the algorithms had apparently decided on Fiji. And so, the beautiful beaches of the Coral Coast on Fiji’s main island Viti Levu were projected onto the virtual shop window, together with shots of the Kula Wild Adventure Park, the Savu Na Mate Laya waterfall and of course a fantastic hotel complex. Claire watched enthusiastically and wished she had booked directly, but first she wanted to speak to a friend who was in the Maldives at the time. Many travel agencies now also offered this service in their CAVE’s or holo chambers: One could simply go live to a place where another person was vacationing and where the respective hotel or travel agency had cameras, microphones, thermometers, and projectors installed specifically for virtual calls. And so, after entering the travel agency and having a short conversation with today’s service employee, Claire found herself back in the Maldives. She talked to her friend, felt the summer heat, which of course was simulated by the installed air conditioning, and enjoyed a few minutes of downtime in one of the most popular holiday paradises in the world. This service was not exactly cheap, but Claire was happy to pay a little more for a little break. It was definitely a good extra income for the travel agencies and hotels.
After almost twenty minutes, Claire said goodbye happily and contentedly and left the holo chamber. The service staff received her friendly. Because Claire still wanted to make some inquiries about her vacation idea, he led her to his desk in the service area and asked her about her preferred destination. Cairo and Giza, Claire replied. Both sat down. The employee typed briefly on his keyboard and after fractions of a second the floor of the travel agency turned into a desert. The walls showed a bright blue sky and the wall behind the employee, right in front of Claire’s eyes, showed the Pyramids of Giza. The air conditioner raised the temperature by a few degrees, creating a light, warm draft. The overhead lighting was intensified accordingly. Claire felt present right there, as if she really were in Giza. She wondered, once again, how much a display film with a transparent protective layer would cost for her own apartment. Only a few of her acquaintances had previously afforded this expensive luxury.
Apparently out of nowhere, a boy appeared next to the service employee. He was obviously a holographic projection. The employee excused himself with a smile and a gesture, turned to the boy, his son, who was on his lunch break at school, and spoke briefly with him about lunch and dinner and when to pick him up. Then they said goodbye with one of those virtual hugs that seemed surprisingly real, even though they could not actually touch each other physically. The employee then turned back to Claire, politely apologized, and spoke to her about the current Cairo and Giza travel deals.
With a concrete idea, but not yet having booked, Claire went back onto the street. Sometimes she wondered why she should travel at all when she could also experience vacation completely virtually, but in fact these were two completely different things.
It was actually a bit cooler back on the road, despite the sunny May day. Everywhere, virtually projected people walked next to real people. They spoke to each other, but even at close range, Claire could never hear their conversations because they spoke and heard each other through headphones, smartphones, or spatial audio. When a virtual person said goodbye, they simply disappeared. Some people even walked virtual dogs. In big cities like New York, Claire had heard, there would even be extravagant locals who would keep virtual crocodiles and lions as pets on a leash. Claire was glad that in her tranquil small town it was mostly just dogs.
Claire wanted to cross the street a few meters further, but just before she reached it, the pedestrian light switched to red. At the same time, virtual barriers rode up from the sidewalk, which in themselves were clear indications to pedestrians and cyclists―but as is so often the case, two pedestrians who were obviously in a particular hurry stepped through the barriers as if they were air, and quickly crossed the street. Claire preferred to give herself a little more rest. After less than a minute, the barriers went back into the ground and in the split second in which they completely disappeared into it, the pedestrian traffic light switched to green. At the same time, barriers had moved up the street and blocked the lanes, at least virtually. Now they formed an aisle, a closed path for pedestrians, in which they could supposedly safely cross the street.
Claire’s destination was the park just across the street. It was a small, idyllic park that families in particular used when the weather was good. Every June, July, and September, the city held virtual festivals there. While it used to be pure projection mapping in the past, with buildings and trees bathed in bright colors, for a few years now it had been interactive simulations with annually changing themes. Most of the time, however, the large lawns were available to visitors for picnics, Frisbee games, and other recreational activities. Stages were sometimes set up for plays and concerts, and many days street musicians performed live. This was also the case today: Already on the other side of the street, at the entrance to the park, a single violinist played covers of well-known rock and pop songs together with a virtual cellist and an equally virtual contrabass player. Claire stood for a while and listened to the unlikely trio. Four passers-by had transferred donations to the violinist with quick swiping gestures on their smartphones without even stopping. After three songs, Claire also picked up her smartphone, called up her mobile payment service and with a single wave of her right hand swiped two dollars towards the violinist, who smiled at her in thanks and continued playing undeterred. Since her smartphone had only identified one active payee in the vicinity, the assignment had occurred automatically.
When Claire entered the park, she already saw numerous children releasing virtual balloons. Within the field of view of the ubiquitous cameras, the children could display a selection of shapes and colors with a gesture, which was projected in front of them as a radial menu. The city had paid $350,000 to equip this park with a sufficient number of cameras, spatial audio speakers and projectors. The Skytracker projectors had been most expensive, and could be used to project virtual balloons, gliders, dragons, or occasionally alien spaceships over the park even in daylight. They could still be seen from a kilometer away, albeit faintly and blurry. And, of course, the projectors tricked. The small zeppelin, which appeared to be hovering several kilometers high and advertised a nearby amusement park, was of course much smaller and projected much closer.
A few steps further, Claire approached a fountain that was shut down for repairs. It was hard to see from a distance because the fountain continued to splash, at least virtually, while lettering rotated around it announcing its repair and reopening for next Friday. From time to time even fish jumped out of the water and plunged straight back in. As Claire approached the fountain to get a closer look at the fish, a virtual barrier rose out of the ground directly in front of her, blocking her path. It was only then that Claire noticed the holographic foil display that had been spanned around the construction site, completely blocking the view of the actual fountain. Everything Claire had seen was a deceptively real-looking simulation.
As Claire stood in front of the virtual barrier, she noticed a father and his five-year-old daughter. Unintentionally, Claire overheard her father calling her Dorothy as she begged, "May I? May I?". At first, Claire did not know what Dorothy meant, but when her father finally gave in, the girl opened a radial menu with a gesture and deliberately selected a specific option. The menu then disappeared and the paved sidewalk beneath her feet turned into a yellow brick walkway like the one that led to the Emerald City in The Wizard of Oz. The girl cheered happily, took her father by the hand, and pulled him along the path, further into the park. Claire looked after them with a smile and was surprised that she had never noticed that there was also a display foil on the asphalt and a transparent protective layer on top of it. Some things apparently only children knew!
Claire continued to stroll down the yellow brick path until she could no longer see Dorothy and her father. Instead, she discovered the sub-surface stage, which, exceptionally, was provided with real, physical barriers. Behind these was basically just a large, nearly cubic hole in the ground, but equipped with numerous projectors. As was usually the case, it was used to represent the enclosure of a zoo from some other city in the world. This week there was a time-delayed live broadcast from the Beijing Zoo’s panda enclosure, according to the displayed virtual signage. Due to the difference in time zones, Claire saw a recording that had actually been recorded at the same local time in Beijing, several hours earlier. Everything looked deceptively real. Families and couples stood behind the barrier and watched as three panda babies played with and besieged the zookeeper while he tried to clean up the enclosure. A real test of patience for him, but a huge spectacle for the children here in the park.
As she watched the pandas, Claire’s phone vibrated. Since she wanted to make the call undisturbed, without too many other people around her, Claire initially only accepted the audio call, but not the live projection call. Claire apologized to her friend Emilie because, while it was not considered rude these days, it was at least unusual to make a call without a live projection. But Emilie took it easy, and after a few meters, which she had walked quickly, Claire reached an unoccupied park bench. She sat down on one half of the bench and swiped the projection mode icon towards the other half. She knew that cameras were capturing her sitting on the bench and streaming it lifelike into Emilie’s living room. Emilie now also appeared lifelike next to her, sitting on the other half of the bench. She was probably sitting in a chair, an armchair or on a couch at home which made it easy for the algorithms programmed into the projection systems to position her so precisely on the bench that Emilie even seemed to be leaning against the backrest.
Claire and Emilie talked, laughed, and celebrated Emilie’s Ph.D. degree for over an hour. Claire was looking forward to hopefully visiting her in Paris next year. The Eiffel Tower was said to have some awesome societal XR specials in the program that Claire definitely wanted to see live.
Discussion
The vision presented herein will not be feasible in the 2020s; that is not its goal either. Although numerous presented technologies such as virtual mirrors [11] already exist, many of them are still too expensive, too vulnerable to weather and damage, too controversial in society, or not sufficiently regulated by law. For example, the vision in section 2 does not include any long-range audio recording, which is actually already commercially available via directional microphones such as shot-gun microphones and can be widely used. This possibility was intentionally left out because it seems unattractive for reasons of privacy and data protection, at least nowadays, even if this is only a weak justification, because phone calls or conversations can also be overheard, bugged, and recorded in plenty of other ways. Nevertheless, it is likely that without significant changes in the legislation of numerous countries and in public opinion, no acceptance could be achieved within a social discourse for plastering the public space with microphones―although analogously the public areas in many cities such as London, despite considerable concerns, rejection and resistance, have long been comprehensively monitored with cameras and have been recorded and made visible to everybody from anywhere in the world, by Google Street View and the like. In this vision paper, too, the limits of what is technically feasible and what is socially desirable must therefore be discussed.
For a vision of a socially acceptable, inclusive technology for the common good, it is not sufficient just to provide technologies, but legislative, legal, and societal preconditions must also be taken into account. For example, the vision outlined requires user profiles and the collection of user data on a scale that is already being pursued by social media and companies in the advertising industry, e.g., for personalized ads (for example, [12]), although many people refuse them for numerous reasons (cf. [13]). In order to fulfill the vision of a user-centered XR paradigm for the common good, which benefits the general public instead of bringing profits to the few, the path to Societal XR would therefore require a large number of regulations, from laws to voluntary commitments by the industries involved to user-friendly terms of service.
The path to Societal XR also requires rethinking in many other areas, particularly in the way user interfaces are developed. Nowadays, they are mostly developed at design time, long before the completion of their respective software and its use. User interfaces usually do not adapt at all or only slightly to the respective user and are usually limited to a single device, e.g., the smartphone or tablet PC currently in use. They are also designed to be used by only one user at a time who is usually identified via a login procedure. However, approaches such as Ubiquitous Computing, Ambient Intelligence and also Societal XR require shifting the generation of user interfaces from design time to run-time to allow for generating adaptive user interfaces on demand that reach across numerous devices, rooms or even environments. Actually, the whole environment may become the user interface. In the vision presented above, the user interface may even follow the user by jumping over from one device to the next one and may change significantly when another potential user arrives. The basics of run-time UI generation for intelligent environments have already been researched, e.g., in the context of the world’s first industry 4.0 factory [14]. It was found that they required extensive, semantically precise descriptions and models, e.g., of users and user tasks, of usage environments, and of automating UI design approaches such as the application of design patterns [15].
Many of the technologies mentioned above are already available today: virtual mirrors, personalized ads, radial menus, ambient lighting, spatial audio, etc. Others are currently in the process of gaining social acceptance, due, for example, to major virtual events such as the ABBA Voyage concert in London in 2022. And some are still futuristic, but, thanks to science fiction, nevertheless already known to a broad public, such as holo chambers, called holodecks or holosuites in Star Trek, among many others. A few movies such as Minority Report (Steven Spielberg, 2002) feature technologies that are close to the Societal XR vision presented above but focus on their potentially dystopian elements.
Yet, the basic idea behind this vision is not a technical, but a social one. Today’s Virtual, Augmented, and Mixed Reality paradigms, as well as XR in general, mostly focus on one or a few individuals who, either alone or in a small group, experience additional virtual content that everyone else around them does not perceive. This not only means immersion in a positive sense, for example immersion in a virtual world or a mixed reality, but also always means the exclusion of others, even nearby people. Immersing oneself in a virtual reality with all senses always means disengaging oneself from reality for the most part at the same time. Some studies such as Aardema et al., [16] suggest that the use of virtual reality can cause dissociative experiences such as depersonalization and derealization as well as a lessened sense of presence in the objective reality. With increasing user numbers, one also has to ask the question how a future society should function in which millions of people regularly disengage from reality in order to immerse themselves in virtual worlds, just as in Ready Player One [17]. But XR can also achieve completely opposite goals, such as mass events in which any number of people and, above all, all people can take part, regardless of age, technical skills, or disabilities. This is made possible, for example, by Spatial Augmented Reality or projection mapping (cf. [18]), not only on stages or in concert halls, but also outdoors, e.g., in a city center or park.
Approaches such as Spatial Augmented Reality are currently hardly researched. Technical development hardly takes place in those approaches, but focuses on head-mounted displays for VR, AR, and MR, and virtual worlds and spaces. It would be worthwhile, in addition to the development of virtual spaces that are decoupled from reality, to also research the supplementation of reality or the real environment with meaningful, generally accessible and socially relevant virtual content. The vision of Societal XR can provide a common concept for this.
Ultimately, however, Societal XR, like any technological vision, is limited and will ultimately be overtaken by the future—which is exactly how it has to be. This results in particular from the constantly falling costs for hardware and the democratization of technologies, as historically been the case with the PC and the tablet PC: Between 1975 and 1980 (the dates vary) Microsoft’s founders Paul Allen and Bill Gates envisioned "a (micro)computer in every home and on every desk", which has long since been overtaken by reality. In [10], as early as 1991, Mark Weiser wrote about three types of "ubiquitous computers ... in different sizes, each suited to a particular task": tabs, pads, and boards. All three device types are available by now, and "weave themselves into the fabric of everyday life" [10], but there are already significantly more device types such as wearable computers for many more use cases. Every technological vision will ultimately, in one way or another, be overtaken by reality, which is a positive sign because a vision must not be utopian and impossible to implement but should rather offer a guide for research and development in the coming years or decades. The vision presented herein could have easily turned out more utopian, featuring, for example, complete immersion through psychochemistry as in Stanislaw Lem’s The Futurological Congress (1971), totally natural haptic feedback, holodecks as in Star Trek, or ubiquitous speech recording and recognition devices that, as discussed above, could be installed everywhere in the environment instead of being worn on the body, just like today’s surveillance cameras. A vision of Societal XR which worked completely without any body-worn devices, would have been easy to formulate. However, in order to serve as a guide for XR research and development, a goal was needed that appears both socially desirable and technologically feasible within a medium-term period.
Because the Societal XR vision presented in this paper is limited, it cannot solve all current and future problems. For example, the digital divide that already exists today may be reduced if or when XR becomes freely available to everyone in the public space, so that individuals will not have to buy expensive devices such as mixed reality HMD’s that they may not be able to afford. The fact that reducing the digital divide is not equivalent to abolishing it is briefly addressed in Section 2: At one point the vision describes that Claire wonders how much a technology like that in the travel agency might cost, because only few of her acquaintances have installed something similar in their homes. This was not deepened in this discussion because the focus of the Societal XR vision is on public, freely accessible XR. However, a similar vision for individual needs, e.g., at home, would also be desirable.
Conclusion
Societal XR is a vision that challenges three core aspects of current XR research, development, and technologies: It presents a concept of a generally accessible, non-discriminatory, social, and public form of reality expansion that excludes nobody. In Societal XR, virtual elements become part of the perceived reality. They come to the users instead of the other way around. They do not require the wearing of smart glasses or head-mounted displays and do not create a limited virtual world or space for a limited number of participants.
Societal XR combines concepts such as Spatial Augmented Reality, Social XR, Mixed Reality, Ambient Intelligence, and Ubiquitous Computing to formulate a new vision of what can be achieved with XR when no longer each and every individual has to afford XR and its devices, but society as a whole decides to integrate virtuality much more into reality and everyday life than is usual today. It is intended as a utopian rather than a dystopian concept of an evolutionary further development of XR that breaks away from XR being used individually or in small groups, but instead making it publicly available to entire towns, societies and populations.
Although the Societal XR vision cannot be adopted instantaneously, but might become feasible only within a few decades, it is to be hoped that it will be adopted and thereby give XR research and development an additional new direction and goal.
Authors’ Information
Daniel Görlich studied informatics at Humbolt University of Berlin and holds a Ph.D. in Human-Machine interaction from the University of Kaiserslautern. Currently, he is a professor for virtual reality and game development at SRH University Heidelberg. His research focuses on modern and novel ways to design and develop virtual worlds.
Authors’ Contributions
Daniel Görlich is the sole author.
Competing Interests
The authors declare that they have no competing interests.
Funding
No funding was received for this project.