Because human eyes are approximately 6-7 cm apart, we view the world from slightly different angles. This is called “binocular parallax,” which we humans reconstruct in our brains to perceive the world in three dimensions. This principle is now being applied in robotics at NAVER LABS Europe.
CROCO: An AI model that shows the world to robots
Martin Humenberger, the director of NAVER LABS Europe, presented “AI for Robots” at the recent DAN23 conference, stating:
"For robots in uncontrolled environments, performing tasks autonomously is extremely challenging. This is because lack of control implies unpredictability. However, this challenge can be overcome with foundation models."
A “foundation model” is a model that identifies ad learns rules from vast amounts of data. Large language models (LLMs) such as NAVER's HyperCLOVA X and OpenAI's GPT-4 are also foundation models. NAVER LABS Europe has been developing a unique foundation model called "CROCO" since 2021. Interestingly, CROCO is not a language foundation model, but a vision foundation model designed for application in robots.
While LLMs learn from an immense number of sentences, CROCO learns from images of the same scene photographed at different angles, which enables it to understand the three-dimensional world. This learning method closely resembles how humans perceive the world in three dimensions, using their two eyes.
Applying CROCO to robots allows them to explore spaces like humans and quickly adapt to complex situations or environmental changes. The purpose of researching such robot AI is to enable robots to interact safely with humans in everyday spaces and to provide a greater variety of services.
DUSt3R: An AI tool that converts images into 3D
One of the reasons foundation models like CROCO are gaining attention is that they allow the creation of various AI tools from a single model. In other words, by fine-tuning a model for specific purposes, its versatility can be infinitely expanded. Researchers at NAVER LABS Europe are concurrently working on remarkable projects based on CROCO, one of which is “DUSt3R.”
DUSt3R is an AI tool that instantly converts 2D images to 3D with a single process. On top of this, its performance is outstanding. With a few images or even just one, DUSt3R can reconstruct a space in 3D in seconds and even extract geometric information. The ability of DUSt3R to simultaneously perform 3D reconstruction and mapping is an amazing feature not seen in any other tool. While traditional methodologies are complex and resource-intensive, DUSt3R only requires a few photos as input.
Because of this, DUSt3R has garnered significant attention since its release. After the acceptance of the paper on it at a global AI conference and the subsequent publicization of its code, DUSt3R has been praised as having “a unique methodology that sets it apart from traditional technologies."
DUSt3R is inspiring not only roboticists but also service developers, with its ability to easily extract 3D spatial information from photos. There are numerous applications for this, such as in real estate services that provide 3D information, immersive recreations of historical spaces from photographs, and space computing-based services that rapidly understand 3D spaces and perform various interactions, requiring only cameras instead of expensive sensors.
AI is bringing about countless innovations in the everyday physical world, with CROCO being the starting point and DUSt3R serving as a practical example. Even now, NAVER LABS Europe is developing more powerful models and upgraded AI tools. What will come next? We will continue to share exciting news.