Current in-flight experiences are two dimensional and provide little to no detail about the passing geography or flight updates. A view of landmarks below and geographical regions are not currently available from all seats on the plane. A passenger cannot see outside of the window easily when sitting in a seat that is located in the middle of the plane. Even when sitting by the window, the view is limited. The passenger has a passive experience from points A to B, reliant on the Captain's updates for temperature, visibility, and potential upcoming turbulence. Since this was a project for a hackathon that was sponsored by Microsoft and Boeing, the complete experience had to be created for the HoloLens device.
My role was to research, design, and test all UIs related to the project. I collaborated with a visual designer, product developers, and engineers to launch this project. How I began was that I researched and analyzed current problems with in-flight experiences for passengers as well as other types of vehicle-based tourism. The concept of a glass bottom boat (figure 1) and domed window train cars (figure 2) provided initial ideas that we molded into an aerial experience. I specifically drew upon an experience that I had on the Amtrak train called Empire Builder from Seattle to Chicago where during spring and summer, an onboard volunteer National Park Service ranger shares facts about the landscape. In contrast to Amtrak and Ripley's, the HoloFlight experience is meant to provide an augmented tour experience that is self-directed.
Figure 1. Photo Credit:
Figure 2. Photo Credit:
After envisioning what the experience would look like from the above photos, I created a storyboard (figure 3) of the potential experience to see how it would play out from start to finish in order to gain deeper insights. I also created a few sketches of how we could build out the technology with 3D models (figure 4) that allow for users to preview sights and book tours at their destination, and live updates of things like sporting events (figure 5).
In addition to user experience research and design, I designed the logo (figure 6) and the visual designer drew it in illustrator. It is the map icon with wings. I was inspired by the pins that captains hand out to first time flyers.
Putting it together
The rendering of the project in Unity using Mapbox took longer than expected and the resolution was not as high as we thought it would be, but we pressed on because we only had 32 hours to complete this project. Since this was a hackathon, the value driver for this project was to get a working concept into the hands of the judges, so we settled on the lower quality initially. We also made the decision to look for ways to optimize the performance during the rest of the build.
View of Mapbox through the HoloLens.
Adding Points Of Interest (POIs) into Unity.
We did not have access to external users, so we did the next best thing and tested out the experience on ourselves. We were able to do a simple proof of concept of a user seeing multiple points of interest as they are "flying" over Seattle and clicking to view the pop-up item info panel.
HoloFlight is a holographic application for HoloLens that provides a glass bottom boat experience for in-flight passengers. It removes the pain of sitting in an aisle seat by allowing all passengers access to an unobstructed view of the terrain below. HoloFlight displays a 360-degree view of the earth from any seat on the plane and utilizes a "glass bottom" concept to provide an aerial view of various points of interest. The experience is completely immersive from both a first person and third person point of view. HoloFlight engages the passenger in the flight experience, supported with visuals regarding geography, points of interest, future travel suggestions, flight updates, weather in arrival city, etc.
The menu screen (created by the visual designer in photoshop)
The Flight Tracker (created by the visual designer in photoshop)
The HoloFlight Experience when viewed in HoloLens.
How It Was Built
This experience was built for the Microsoft HoloLens using Unity assets and other free online resources. The developer used Mapbox for the aerial map rendering (https://www.mapbox.com). He hit a blocker trying to get the Mapbox to render at a high enough resolution when deploying to the HoloLens, but we were all very happy with the outcome. Built with Unity and C#. Try it out: GitHub Repo
What I learned
Overall, the experience was really great and the judges were pleased with our project. My favorite part of the project was really the brainstorming and answering open-ended questions like "what experiences are possible with this technology", "how can this technology be used to improve peoples livelihood while on a plane?", "what other use case scenarios might this technology be used for?", "what are the current limitations?". This concept has so much potential and user stories that have not even been explored. For example, the military could use it for surveying war zones, checking supplies below that are integrated with the IOT, getting a head count on how many soldiers are alive and what their stats are if they are wearing a Fitbit device, etc. Also, there could be an interface built into the POI's that allows the passengers to preview their destinations and make reservations in advance inside of the headset. For me, this project has stood the test of time since it's creation in 2017, and I still enjoy thinking about the possibilities.
Read more about the team at DEVPOST