Augmented Reality and Neuroscience
With Augmented Reality (AR), we can use the space around us to visualize our imagination, organize our thoughts with our own hands and share it all with our peers, face-to-face. We call this the Spatial Interface, and it stands to replace the Windows Interface paradigm that has dominated computing since pioneers like Doug Engelbart and Xerox PARC began developing it almost a half century ago.
AR frees us from abstract 2D interfaces that must be taught to the brain, and enables natural 3D interfaces that the brain has evolved to understand. That’s why we’ve turned to neuroscience to learn what the brain “wants”, and base our designs on that insight. This document is the product of years of research by a diverse, ever-growing team: neuroscientists, led by Professor Stefano Baldassi (previously at Stanford), UX designers and developers, advisors from industry and academia, and yours truly. Working together, we’re all playing a role in turning the science of the brain into a well-tested and practical foundation for AR development.
With any new medium it’s tempting to rush, of course. It’s a lot easier to simply port a 2D interface from Windows or iOS into 3D space. But AR is a new form factor, and the old rules just don’t apply. That’s why it’s so important to embrace the challenge of rethinking flat designs from the ground up as truly spatial tools, tailor-made for the human brain. And since AR is a burgeoning medium, your work will likely be the first holographic application in its category, setting a standard for others to follow.AR is a new form factor, and the old rules just don’t apply.
Our ultimate vision is a zero-learning-curve machine, based so naturally on the architecture of the mind that it feels like an extension of it. But we won’t get there overnight. This document is the first word—not the last—in an ongoing conversation, meant to evolve alongside the science. Moreover, the hardware and algorithms are developing as well. As such, there will be times when these principles will have to be diluted to deal with technological limitations. We’ve done our best to identify such cases and provide strategies for achieving optimal results, and our philosophy for each is simple: compromise should be minimal and temporary.
The Windows Graphical User Interface defined the last 50 years of computing, from the Xerox Alto to the iPhone. The time has come to free ourselves from the constraints of flat screens and enter the Spatial Interface, where we might live for the next 50 years. Let’s make sure it’s a comfortable stay.
Spatial interface design
Who this document is for
We recommend all readers start with the next section, entitled The Neural Path of Least Resistance, as it summarizes the foundational idea that drives this entire project. After that, we present a series of self-contained principles that focus on a particular aspect of the interface design process. For first timers, it’s best to start at the beginning and read them in order, as one tends to build on the last:
Each principle is presented in a consistent format:
- For all readers, the summary is the simplest and most immediate glimpse of the idea behind each principle
- For UX designers, UI Design Suggestions explore the application of the principle to real-world design situations
- For science-minded readers looking to dig a little deeper, the Neuroscience and Further Study sections provide an outline of the literature that serves as the foundation for the principles.
- For developers facing the challenge of applying these principles in the real world, sections like ...Before you Object! attempt to anticipate and address likely follow-up questions and concerns
Finally, let’s not forget that AR is, for the time being, a transitional technology with some inevitable limitations. In such cases, sections marked Technical Consideration are meant to help developers apply our principle of minimal and temporary compromise, which helps balance today’s reality with tomorrow’s ideals.
Our ultimate commitment is to the science of the brain and its applications to Spatial Interfaces. To that end, we’ve reviewed hundreds of neuroscience papers for each principle listed in this document and done our best to identify the most relevant cross section for inclusion here. If you have questions about our methodology, disagree with our conclusions, or have research of your own to contribute, please contact us. These ideas are intended to evolve alongside a growing body of studies and debate.
Spatial interface design
A scientific framework for spatial computing: strive for zero learning curve
Ultimately, every principle in this document is founded on this goal, with its logical conclusion being the eventual arrival of a true, zero-learning curve experience. The gatekeeper here is the brain, of course. That’s why, in the world of spatial computing, the foundation of interface design is neuroscience.
The Bayesian Brain and the
Neural Path of Least Resistance
Our approach to the brain begins with a probabilistic model called Bayesian statistics, which provides a versatile way to predict a user’s response to a new interface. When the user needs to perform a task, their first instinct will be to look for an element that represents the tool associated with that task. The user will do so based on their mental model of the world, called the prior, and compare it to the new interface, called the input. By designing interfaces that match the expected priors of most users, the learning curve will be reduced and the user will experience faster, more accurate results. Therefore, the optimal interface is the one which leverages the user’s existing priors—their mental model of the world—as much as possible, thus reducing the time and effort to accomplish a task. In practice, the best priors tend to be those that mimic1, or at least closely allude to corresponding objects and tools in the real world.
This goal of minimizing effort, or cognitive burden, is what we call
The Neural Path of Least Resistance.
Different tools we use for our tasks have different fit to our priors, or mental models of what the tool for our task should be like.
If the design of the interface does not rely on priors from the real world, but is instead based on something arbitrary like abstract icons, memorized gestures or keyboard shortcuts, the user must undergo the mentally taxing process of redefining their mental model and memorizing new rules, which requires the help of the entire brain. This may be an appropriate trade-off in narrow use-cases for specific “power-user” domains, but as a general rule it violates the Neural Path of Least Resistance and is not recommended for general audience applications.
In short, reduce the learning curve by leveraging the user’s priors and experience in the physical world. The remaining principles in this document provide techniques for applying this concept to Spatial Interface design.
Griffiths, T. L., Kemp, C., and Tenenbaum, J. B. (2008). Bayesian models of cognition. In Ron Sun (ed.), Cambridge Handbook of Computational Cognitive Modeling. Cambridge University Press.
1 Does this mean Meta is officially embracing the controversial practice of skewmorphism? Not quite. In the past, skewmorphism has been justifiably criticized for its tendency to overwhelm the user with needless details like drop shadows, bevels and textures that serve no purpose other than aesthetics. In augmented reality, however, the application of lifelike details in an interface is the key to familiarity and, in turn, understanding. To us, skewmorphism is a means to an end, not a goal of its own.