Background
Large indoor spaces like shopping malls and department stores are crowded with many people as well as stores stretching one after another in a complex structure. It is easy to get lost even when you are looking for stores you have been to before. Hence, directions via augmented reality (AR) navigation can be most useful for such spaces.
To provide stable and reliable AR navigation in indoor spaces, it is first and foremost essential that users’ location information be accurately identified. A variety of unexpected variables in the actual environment that act as hurdles need to be overcome technologically. Many people crowded in front of a famous restaurant become noise to machines when determining the location, and lights in strong contrast and rapidly changing event structures also interfere with identifying the location due to their changing visual characteristics.
Our newly developed technology is showcased as an indoor AR navigation demo for the multi-floor shopping center of Hyundai Department Store–Pangyo. Demonstrating accurate and stable AR navigation in the real environment is the goal for which we challenged ourselves so that we can overcome technological limitations.
1. Integration of complex spatial data such as multiple floors and stairs
Visual localization (VL), which is used importantly in indoor AR navigation, begins with spatial information collected by Mapping Robot M1X. However, to provide seamless services for stairs, multiple floors and narrow hallways in the real environment, it is necessary to have spatial information that integrates complex spaces to which a mapping robot cannot travel. To do so, we applied COMET, an in-house developed backpack-type mapping equipment, along with M1X and successfully linked vertical data in outdoor and multi-floor spaces for Hyundai Department Store–Pangyo.
M1X/COMET - 3D Point Cloud (Hyundai Department Store, Pangyo)
Indoor and outdoor 3D point cloud around Hyundai Department Store–Pangyo created by M1X and COMET
2. Location correction in real time while the user is moving
The visual localization (VL) technology of NAVER LABS, which can accurately identify the current location even in the indoor space where GPS cannot work, has the highest level of precision. In situations where the user keeps moving, however, it is difficult to stably augment AR content with VL alone due to slight location deviations occurring in real time. As the first method to overcome this problem, we have applied visual-inertial odometry (VIO), which tracks the location in real time by analyzing sensors and image information, in addition to VL.
Nevertheless, the content often shakes and lags several meters compared to real time due to the camera’s rapid movements, network delays and temporary VL/VIO deviations. NAVER LABS has solved such problems with the in-house developed real-time camera pose tracking. It uses location correction algorithm for real-time precision positioning, which can augment and track content in a stable manner at an accurate location while the user is moving by combining various filtering logics, and a location prediction technique which analyzes previous traveling routes and calculates the current location.
Content unstable in VL/VIO (left) has become augmented in a stable manner with the location correction algorithm (right)
3. Ensuring that AR navigation moves seamlessly between floors
NAVER LABS has completed a multi-floor navigation scenario, where the user moves between various floors in an indoor space, for the first time with the demo at Hyundai Department Store–Pangyo. It was a challenging project that required us to identify not only spatial data for all floors, but also the precise location on the floor where the user is and link them with the scenario. This may be easy for humans, but not machines.
To connect directions seamlessly through multiple floors, which includes getting on and off the escalator, we needed various location-based technologies. Our AR navigation identifies the current location with VL, tracks the location in real time and determines the user’s surrounding environment with camera pose tracking and visual object tracking (VOT), and augments content at an accurate location.
Directions that naturally guide you through the escalator on 1F (left) to B1 (right)
4. Connecting AR content and real space by using 3D maps
AR can expand spaces into its interface, and lead to more diverse and useful services along with navigations. To make it easier to link with more diverse content, NAVER LABS has developed a location-based AR authoring system mediated by a 3D map made from a mapping robot. As the system has the same coordinates as real spaces, content appears at an accurate location if you set the content on the 3D map. Not only can it precisely correct location values, but specific content parameters and interactions can easily apply to real spaces.
AR content is augmented at the location of a real space corresponding to the 3D map.
Without going outside in person, you can easily browse the 3D map and augment at a location you want, which makes it possible to use a wide range of scenarios. The Hyundai Department Store–Pangyo demo also applies various possible scenarios such as interactively opening a store’s menu and promotional information.
Various scenarios applied to the demo - Examples of linking with store information, product coupons and promotional content
Next steps
Advanced technology by demonstrating in everyday life environments
Our technological know-how that we have acquired from demonstrations in real everyday spaces and projects that we have newly identified and learned from play a key role in rapidly advancing the technology of NAVER LABS. NAVER LABS has established a pipeline across AR technologies, which includes not only a variety of location-based technologies such as VL for mapping, camera pose tracking and VOT, but also our internal rendering and content production technologies. Furthermore, we are packaging and testing the technologies we have developed in real-life environments including Hyundai Department Store–Pangyo.
In the near future, new location-based services will be provided in various daily life spaces. Our goal is to prepare technologies so that more diverse services can be provided in actual spaces. We will connect indoor and outdoor spatial data seamlessly, and further expand our research and demonstration in more spaces relevant to daily life.