VR Games are Less Immersive than Pancake Games

When people first try VR, they often experience The Blu. It is a spectacular demonstration of the presence and immersion possible in VR. In the most popular sequence in the experience, you find yourself on the deck of a sunken ship. As you marvel at the beauty of this underwater environment you have been transported into, a gigantic whale comes into view, mere meters away from you. It pauses to eye you (curiously? balefully?), before swimming away and sending a rush of water past you.

Wow! The fidelity of the environment in its visuals and audio stun the senses at first. Then, you realize you can walk about this deck as if you were there, even crouching down to inspect objects and fish, or reaching out your hand to brush the fauna of this seascape. Now, you come face to face with a creature whose scale you have only been privy to before maybe once, in a natural history museum. But this whale is living; you can lock eyes with it, and the encounter is as ephemeral as a real encounter with a wild creature – before you know it, it is gone.

This is VR in its best form: you are truly transported to a realm that is better than real life. Even if you took the time to become certified for scuba diving and started exploring sea wrecks and swimming with sea life, it wouldn’t be the same. VR allows you to strip away the scuba mask and the hours of training, the cost of taking a boat out to a site, and the danger of entering an alien world. It condenses a transcendent experience into a package that is available to ANYONE, even the young and the handicapped. It is hyperreal: better than reality.

But most of what you can experience in VR does not match this level of hyperreality. It struggles to justify the friction of the medium – the setup time, the cost, the discomfort of strapping a device to your face. Even enthusiasts soon realize that the available virtual worlds of VR don’t offer a better experience, holistically, than sitting on the couch watching Netflix or playing a 3rd person action game on the computer. Why is this? There is a fundamental calculation being performed unconsciously:

Relative Value = Unique Benefits – Unique Downsides

VR has a lot of unique benefits as compared to traditional pancake gaming, as well as some unique downsides. One large benefit is the novelty factor. But this benefit decreases with exposure, and in the end this calculation of relative value results in a negative number for most people. The data backs this up. 40% of people only use their VR device once a week, 34% of people use it less frequently than that. [1] A lot of people mostly break their device out to show to someone else.

Some of the common unique benefits of VR:

  1. Presence – VR has a phenomenal ability to make your brain believe you are truly in a different space.
  2. Massive input space — 6 DOF VR’s motion controllers afford many degrees of freedom over traditional computer inputs. Not only are there the eponymous 6 degrees of freedom (position and rotation), but there is linear and angular velocity, two or more analog inputs (joystick, trigger, etc), and several digital inputs (face buttons).
  3. Physical Freedom – you can transcend the limitations of your mortal form, flying across the world like a superhero or diving deep into the depths of an abyss.
  4. No Consequences – this is a benefit shared, nominally, by traditional games. In Grand Theft Auto you can shoot, steal, and drive like a demon without consequences – you won’t feel bad for killing real humans, and you won’t go to jail for the rest of your life. In VR, you can explore the same kind of consequence-free space, but mapped much closer to reality. You aren’t pressing a button to smash open your opponent’s skull in the gladiatorial arena; you are actually doing it!

Some typical downsides of VR:

  1. Time, Space, and Monetary Cost – not only are the headsets and computers expensive, but you often need to dedicate space in your house to them, and spend time setting it up and maintaining it.
  2. Hassle – by this, I mean the stress imposed by the ensemble of equipment. In order to enjoy the benefits of VR, you must contend with adjusting the headset to fit your head, finding a proper IPD setting, adjusting the headphones or earbuds, picking up the controllers after putting on the headset, adjusting straps on the controllers, minding the headset cable during play, avoiding lens fog, etc. Then sometimes the computer will be acting up, requiring some troubleshooting and application restarting.
  3. Discomfort – even with a perfectly situated headset, it presses against your face and scalp and heats up. Most devices also introduce some ocular discomfort over time, whether due to pupil swim, IPD mismatch, or other subtle optical problems.
  4. The Experience Mapping Problem.

The Experience Mapping Problem

The human brain is fantastic at recognizing patterns and drawing connections. As young children, we gain an intuitive understanding of physics by observing how objects react to our inputs – we unconsciously construct an elaborate mental model of reality, which allows us to accurately predict the outcome of our actions. This is what lets you grab a mug and set it down elsewhere without spilling its contents, or push through a door and enter the room beyond. Sometimes, this model is incorrect: you misjudge the weight of the mug, or the door is locked. When this happens, your body often continues to execute a planned sequence, causing you to drop the mug or run into the door. Trained on hundreds of thousands of hours of experience, our mental model maps certain stimuli to certain responses – and it takes a lot to break that linkage and reform your model to account for a new reality:

VR, fundamentally, runs smack into this mapping problem. Traditional pancake games exist as a separate reality – one that exists on a screen, and which you interact with by pressing buttons. You construct a new mental model when playing pancake games, one that maps the stimuli on-screen to button-pressing responses. Virtual reality, on the other hand, intends to present a reality that mirrors real life. You see it and hear it the same way you see the real world, you can move around with your actual body and you can use your hands to interact with the world – just like real life!
So your brain, naturally, attempts to use the same mental model to react to stimuli in VR. But this rarely serves you well. Current digital worlds function with radically different rules. And this radical mismatch between your mental model’s expected outcome and the observed outcome breaks immersion.

This is the basis behind my claim that VR is actually less immersive than traditional interactive simulations. This runs contrary to what you would expect; what could be more immersive than actually being physically present in another world? But what is immersion? Immersion is the transportation of the spirit – when your subjective experience is completely subsumed by a piece of media. You can be sitting on your couch with a controller, but be completely immersed in the fantasy action game in front of you. Your entire consciousness is in the world described by the image on-screen. Yes, your bodily presence never leaves the couch, but your mind and soul are elsewhere. This can happen even with a movie or book. A book is not naturally immersive, but eons of narrative craft can be leveraged by a good author in order to transport you to another time and place.

The antithesis to immersion is any reminder of your bodily presence in the real world. If you must consciously acknowledge the duality of your existence (the body in one place, the mind in the other), your mind is drawn back to the mortal coil. A distraction causes you to look up from your book, someone walks in front of the TV, or a phone goes off in a theater.

On one hand, VR enables a powerful sort of pseudo-presence. Your eyes and ears are physically present in this other reality, as are your hands (sort of). On the other hand, this physical presence is hampered by invasions of external elements — the cables, the screen-door effect, limited field-of-view, inner-ear and proprioceptive discrepancies — which remind you that you are strapped into a headset.

In addition, many VR experiences are chock full of metaphorical phones-in-theaters. The unconscious attempt to apply your mental model of the real world to this virtual world results in constant discrepancies between expected and observed results, which must be resolved with conscious effort. This further drags your mind away from the constructed reality and back into real life.

What can we do? How do we create a virtual reality that delivers on the promise in The Blu?

How do we tip the calculation of relative value from red to black?

Simply posing the question suggests an answer: we leverage the unique benefits and minimize the unique downsides. Fortunately, lots of people are already working to this end. The monetary cost of VR is decreasing, along with the setup and hassle. Technological improvements in display technologies and audio simulations boost presence, and new apps give players new freedoms and consequence-free fantasy fulfillments.

People are working on these problems because they are obvious. But the Experience Mapping Problem is not obvious. Its results are obvious – a lack of immersion. But this is frequently diagnosed as a fidelity problem, to be solved by higher resolution displays, more realistic graphics, advanced physics simulations, and more “natural” controllers. Unfortunately, these things may actually worsen the Mapping Problem with current-gen VR.

[Asgard’s Wrath]

[Job Simulator]

Which is more immersive? Asgard’s Wrath is more realistic, but you are less likely to lose yourself in the simulation.

[Hand tracking]

[Oculus Touch]

Which is more immersive? Finger tracking is more realistic, but when you try to grab a virtual object, the outcome is VERY different from the expectation based on a lifetime of using your hands.

One solution is to create a set of stimuli that are so different from reality that the player does not make the mistake of assuming their IRL mental model will apply in this space. But this means we must abandon the benefits of VR related to player fantasy. Physical freedom and consequence-free spaces are less meaningful if they are completely unrelated to reality anyways. Nonetheless, some experiences like Tilt Brush and Oculus Medium leverage the input freedom and physical movement freedom to create engaging abstract experiences. But the promise of a hyperreal VR world that is “like real life but better” demands that we try harder to resolve the Mapping Problem.

There are two paths to resolution – the first (and more common approach) is to expedite the player’s development of a new mental model. The second path is to develop virtual realities that are designed to gracefully accept players’ actions and minimize mismatches between expected and observed outcome.

We need elements of both solutions in order to achieve VR’s potential — but modern games need more of the second path. Forcing the player to develop a complicated new mental model to interact with virtual reality means sacrificing a significant amount of VR’s hyperreality, and thereby decreasing its relative value. But we need to maximize the relative value of VR — it’s now or never. If VR doesn’t enter the mainstream now, it will likely never flourish.

The most popular apps and games can teach us how to minimize the mapping problem; games like SUPERHOT, Job Simulator, Robo Recall, and Thrill of the Fight.

Let’s make VR the immersive hyperreality we were promised by science fiction:

(video source)

[1] “77% of Virtual Reality Users Want More Social Engagement”

Why Scientific Philosophy Is Important

I recently talked to a person who was convinced that scientific theories, mathematical theories, mathematical theorems, knowledge, truth, and scientific laws were all basically synonymous. He said that physics could not exist without math, because math defined physics. He also was convinced that believing and agreeing were the same thing. I attempted to remedy these misconceptions using some basic arguments, but I was finally written off as “not understanding anything” and “unwilling to do the math”. When I asked him to define the word truth, he merely kept repeating, “I don’t know what you mean. Truth is just that which is.” When I attempted to explain that the word “truth” was a symbol referring to a concept, and that we couldn’t have a discussion if we were referring to different concepts with the same word, he said “you don’t need to define truth, it just is. It’s very simple.” He couldn’t understand why I kept “bringing up philosophy when we’re talking about simple truths here.”

Sigh. If I can’t break through that kind of rhetoric, I might as well just explain my thoughts here.

Why is it important to know about the philosophy behind knowledge, truth, and science when talking about it? Isn’t it possible to rely on a the natural human consensus of truth? Besides, while it is so hard to explain using language, people intuitively grasp the concept. Right?

Well, let’s give some examples. It’s true that if you drop an object, it falls, right? Well, yeah, that statement is true if you are on the surface of a planet, and not orbiting it. Or if you are underwater and you drop a buoyant object — it goes up! But wait, can you drop something underwater if it doesn’t go down? No, that wouldn’t be dropping it would just be… releasing? Hold on, when an object is in orbit, isn’t it actually just falling in a special way? It’s moving sideways fast enough that it misses the ground by the time it’s fallen far enough. But if an astronaut releases a wrench, and it float right in front of him, you wouldn’t call that “dropping”.

What we see is that the word “drop” has a definition, and we need to know what the definition of “drop” is before we can begin to assess the truth of the statement “if you drop an object, it falls”. As it turns out, “dropping” an object consists of releasing it such that it falls away from you. Uh oh. So yeah, “if you drop an object, it falls” is true, but it doesn’t actually convey any physical knowledge; it just defines a property of the word “drop” in terms of another word, “fall”.

So lets look at some more meaningful examples. Most people would say it’s true that planets orbit the sun in an elliptical manner. Except it isn’t true. It’s true that the movement of the planets can be approximated into ellipses, but in fact there are measurable deviations. “Okay, sure. The movement is actually described by Newton’s laws of motion, and the law of gravitation.” Okay, yes, an N-body approximation gets much, much closer to describing reality. In fact, it perfectly matched the observations Newton was working from. However, it’s still not true the Newton’s laws describe the motion of the planets.

We can look to general relativity to describe the motion of the planets even better. We have launched satellites to observe very minor fluctuations in the path of the Earth that would confirm the prediction made by general relativity. As it turns out, general relativity makes predictions that perfectly match our observations. Woof. Finally, we’ve found some truth. The path of the planets around the sun is described by general relativity.

But wait, can we say this in good conscience? No! Just like Newton, we’ve found a set of laws which create predictions that match our observations. But just like Newton, we cannot measure the motion perfectly. All we can say is that general relativity describes the motion of the planets as far as we can observe. We don’t know if there is some unknown mechanic that affects the motion of planets in a way we can’t measure right now. We can’t say that general relativity is “true”, we can only say that it is confirmed by all of our observations to date, much in the same way that Newton could not say that his laws of motion were true; they merely described the all physical data he was capable of obtaining.

This gets to the root of the problem. While mathematical notions can be “true” because they exist within an entirely constructed framework defined through logic, theories in science can never be “true”. The point of science is not to find things that are true, but to find the best explanation for why the world works the way it does. And just to get one thing clear, theories are explanation of “why”, and laws are explicit definitions of how physical quantities relate. So no, we don’t use “math to define physics”, physics uses math to explain the physical universe. But even without math, we can perform a sort of qualitative physics.

For instance, “things stay still until you push them, and things keep going straight unless you push them.” This phrasing of Newton’s first law of motion is simplistic and uses words like “thing” and “push” without really defining them, but it gets the point across. Similarly, “big things move less when you push them, and small things move more.” This is very simplistic, and doesn’t even mention the fact that acceleration changes linearly with force, but it communicates the basic idea of Newton’s second law of motion, without even getting into what “big”, “small”, and “move” really mean.

The point is that the traditional phrasing of Newton’s second law, F=ma (which, by the way, is more accurately ΣF = m * Σa), merely uses mathematical symbols rather than English symbols, which allows us to manipulate it using the rules of mathematics. But just because we are manipulating arbitrary quantities with math doesn’t mean anything physically. Just because I calculate that an object which masses 1 kg should accelerate at 1 m/s^2 when I apply 1 N of force doesn’t mean the thing is actually going to act that way if I perform the experiment. This is because “mass” is really a simplification of a whole range of things, as is “acceleration”. It doesn’t even account for internal forces, and only describes the movement of the center of mass.

Math may be true, but only within the realm of math. When we translate physical quantities into the mathematical universe, they lose they physical meaning. We may translate them back, but the results we get can only be an approximation, not a truth, not a reality. These approximations can be very useful, but we have to remember the limitations of our theories, and our instruments.

Separating Science and Religion

I read this article for school:
Lightman’s The Accidental Universe

When asked to write an essay about it, this is what came out. I don’t normally post essays like this, but I’ve been meaning to write a post much like this for a while anyways, so it’s convenient.

Lightman descends into the realm of religion, masking his language with a thin film of scientific consideration, but none of its hard, decisive, rational edge. Lightman never even touches the basic principles of science, but uses philosophical arguments to parade a seemingly-scientific theory around.

Falsifiability is a method for evaluating scientific theories popularized by Karl Popper. It contends that a theory cannot be proved by showing evidence in favor of it. A theory may be shown to be strong if it can make empirically confirmable and correct hypotheses, but a theory can never be proved – only disproved. So to be a scientifically valid, a theory must have a way to be disproven (thus by not being disproven, it continues as the dominant theory). This is one of the problems with the multiverse theory, the theory of intelligent design, and even string theory: it is most likely impossible to disprove them. If an intelligent creator revealed itself, such a turn of events would not inherently make the multiverse theory wrong, per se (a multiverse theory can coexist with intelligent design). It would only make it irrelevant. Of course, this reveals an even bigger fundamental problem with those theories: they don’t explain the mechanics behind physical phenomenon in the traditional sense. Instead, they provide a framework of thought into which actual scientific theories can be slotted. But the multiverse theory is only one framework among many, and there is no way to show that one framework is strictly better than another.

Is it not just as reasonable, just as falsifiable (or not, as the case may be), to conclude that the universe as we know it is the only one, albeit a very lucky one? One could posit that it is indeed accidental. How does this postulate contend with the others on the battlefield of scientific thought? In some regards it may triumph over its opponents, because it relies on any contrary observation to disprove it, while both the multiverse theory and intelligent design can be valid even in the face of one or the other being true. So really, the Random Chance theory is more falsifiable, and thus more scientific.

But of course the Random Chance theory is completely unpleasing to the philosophical human mind. A much more palatable theory is the multiverse theory, which, like a wolf in sheep’s clothing, slips in among the legitimate scientific advancements and completes a scientist’s world view satisfactorily. But is a scientist’s world view scientific? No. Science is a tool for developing a physically accurate view of the world, and we employ it because the human mind is not built to obey scientific rules. Our capacity for cognitive dissonance is astounding. Thus a scientist can in good conscience accept a non-scientific belief to assuage his existential conflicts by slathering the belief in the manner of other physics theories.

Another such unfalsifiable belief system is string theory. String theory is a self-consistent way of interpreting physical data using notions that fall out of mathematical equations but have no basis in experimental science. Indeed, string theory exists only as a way for some physicists and mathematicians to unify all of reality under some Platonic mode. But it is only that; a way to think about the universe, to help explain the Great Unexplainable, as Sax Russell calls it. String theory cannot produce hypotheses that can be tested to confirm the mode of thought. It can explain the observed, but only as well as previous existing theories. While it is nice that it can bring physical laws under a single wing, niceness is not a necessary quality of scientific theory. It is a subjective human measurement applied in the realm of philosophy.

Philosophy is not useless. It is a tool, like science, for examining the world. However, instead of measuring and describing physical phenomenon objectively, it takes human concepts or unimaginable realities (such as that beyond the realm of science) and compresses them down and creates a set of rules for the human mind to follow. It generates modes of thought that allow us to function and think about that which might otherwise turn us into quivering lumps of existential dread.

But assembling a philosophical system of thought only to pass it off as a product of science is dangerous. Besides preying on those incapable of evaluating the modes of thought on their own, it tricks the creator as well. Thus we can see the inevitable and unending conflict between the “rational” scientist and the “faithful” man of religion. Neither of them realizes that they are jousting with philosophical ideas, and as a result keeps hitting his opponent not at the weak spots, but at the bastions of his belief. The scientist calls his mode of thought “scientific truth” (a misleading term in and of itself), and the religious man calls his mode of thought “religion”.

Unfortunately for the world, nobody (certainly not the loud ones) seems to realize that science and religion are not diametrically opposed. Religion is not taken entirely on faith; while it does depend on some unfalsifiable core, it builds up a philosophical belief system around that which, beyond the basic axioms, is self-consistent and pretty damn useful. The scientist, used to tackling scientific theories, thinks that by attacking the core tenets of religion, he can bring down the entire system. But the core is unfalsifiable, so the methods of science are useless. Science and religion shouldn’t even overlap in their realms of explanation. In truth, they don’t. But unfalsifiable philosophy is given the title of science, and physical explanations are given the title of religion, so two incompatible systems are faced against each other. It would be better for everyone if both sides retreated to their realm of the human experience, but since they won’t, we get tripe like Lightman’s essay.