The Reality-Virtuality Continuum
Although the ideas behind augmented reality have been around for some time, the technology is still very much in its infancy. For that reason, the terminology can get a bit confusing. For example, the title of this article implies that its focus is augmented reality, but technically it’s “mixed reality”, which is a broader category.
Both of these concepts can be understood in terms of a continuum, with real environments at one end and wholly virtual environments at the other. In between lies augmented reality, as well as augmented virtuality. This in-between space in its entirety constitutes mixed reality.
The notion of a reality-virtuality continuum was first introduced more than two decades ago by Paul Milgram, a professor of mechanical and industrial engineering at the University of Toronto.
To put the continuum in a manufacturing context, think of the far left (Real Environment) as the traditional factory floor: workers operating machines based on information displayed on dials, gauges or screens.
The far right (Virtual Environment) is exemplified by the use of virtual reality in factory floor planning.
It may be difficult to imagine examples of augmented virtuality in manufacturing, but one possible application could be using handheld peripherals to input commands, or testing ergonomics in VR factory-floor planning.
There are many examples of augmented reality in manufacturing, so let’s look at a few in more detail.
5 Applications for AR in Manufacturing
Many manufacturers have begun to explore the benefits augmented reality can offer in an industrial environment, a trend that has nowhere to go but up as the hardware and software for AR improves. But as with any relatively young technology, we’ve only begun to scratch the surface of what augmented reality can do for manufacturing.
Even so, the possibilities are already tantalizing.
1. Complex Assembly
Volvo has been experimenting with AR on its assembly lines using Microsoft's HoloLens. (Image courtesy of Microsoft.)
Modern manufacturing involves putting together hundreds or thousands of components in a precise sequence as quickly as possible. This is true whether you’re manufacturing smartphones or jet engines, and every new product requires a new set of assembly instructions.
“Your work instructions tend to be these PDFs that are hard to work through, plus they’re static documents, so they may be out of date,” commented Ash Eldritch, CEO and co-founder of Vital Enterprises, a developer of augmented reality software.
“We take those instructions and make them glanceable in your field of view at all times, hands-free and voice-controlled,” Eldritch continued. “So we break down the work instructions along with associated technical drawings and even video from the last person who did the procedure and put all that onto the [AR] glasses. That means you can keep your hands on your task and you don’t need to walk over to a work station to check something.”
Although the notion of projecting assembly information on a heads-up display may be fairly novel, assisting with complex assembly is already a tried-and-tested AR application.
Boeing's AR tablet tool for assembly lines. (Image courtesy of Boeing.)
“Technically, AR has been used for this for the last 20 years,” noted Milan Kocic, Hexagon Manufacturing Intelligence’s business development manager for product experience and innovation. Kocic has a very cool job concealed by a very long title, and it makes him uniquely qualified to weigh in on augmented reality in manufacturing:
“I’m lucky enough to get to go to places like CES and other forward-looking tradeshows which ordinarily have nothing to do with [Hexagon MI’s] business, but I go to look at where the trends are going. What are the big companies trying to do? How is the technology developing? I try to predict what the future might hold for us two, three, five or ten years from today.”
This has given Kocic a valuable perspective on up-and-coming technologies, as well as how far they’ve really come.
“In aerospace manufacturing,” he said, “you have assembly assistance, where there are projectors that show the next fastener placement and that sort of thing. What makes augmented reality different today is mostly size and capability. It’s like 3D printing: the printers were very slow and expensive 20 years ago. Now, they’re just cheaper and faster.”
2. Maintenance
(Image courtesy of Vital Enterprises.)
In addition to helping with the assembly of manufactured products, augmented reality can be used to assist in the maintenance of manufacturing equipment. Mitsubishi Electric has been developing maintenance-support technology using augmented reality based on a 3D model that enables users to confirm the order of inspection on an AR display and then enter inspection results with their voice.
“We think that the most useful application of AR is maintenance in a manufacturing environment,” said a representative from the Mitsubishi team developing this technology. “Currently, field workers confirm a maintenance target using a maintenance manual. The process is time consuming and tiring for the workers because the manual and the maintenance target should be confirmed together.”
(Left) A mechanic wearing a tracked head-worn display performs a maintenance task on a Rolls Royce DART 510 Engine. (Right) A view through the head-worn display depicts information using augmented reality to assist the mechanic. (Image courtesy of Columbia University.)
More specifically, Mitsubishi expects its AR system to be used for a variety of maintenance work, with some applications beyond manufacturing, including inspections of water-treatment plants and building electrical systems.
Being able to see a machine’s status simply by glancing at it through an augmented reality display could be incredibly powerful, but AR can be useful for more than just error detection. This video from the GRAphics and INtelligent Systems Group (GRAINS) at the Polytechnic University of Turin shows how augmented reality can be applied to maintenance procedures:
Last year, elevator manufacturer thyssenkrupp announced that its technicians would begin using Microsoft’s HoloLens technology as a tool in service operations. Using HoloLens, service technicians can visualize and identify problems with elevators ahead of a job, and have remote, hands-free access to technical and expert information when on site.
“As the first fully self-contained holographic computer running Windows 10, Microsoft HoloLens has been helping companies and industries innovate in entirely new ways,” said Scott Erickson, general manager for Microsoft HoloLens. “By utilizing the out-of-the-box Skype experience without any additional development required, thyssenkrupp’s 24,000 service engineers can now do their jobs safer and more efficiently. Triaging service requests ahead of the visit and getting hands-free remote holographic guidance when on site has reduced the average length of thyssenkrupp’s service calls by up to four times.”
With all these features, it doesn’t take much imagination to conceive of how a HoloLens or similar AR device could be put to similar use in an industrial setting.
3. Expert Support
The preceding videos hint at another fertile manufacturing application for augmented reality: expert support. This application also relates to maintenance, as Eldritch explained:
“We’re doing a lot of field service, where the manufacturer of a machine also needs to provide service for it, and that involves travelling to the site where the machine is. For very distributed manufacturing operations, you have a number of inspectors and a number of technicians, but not enough experts for when things, inevitably, don’t go according to plan.”
“In those instances,” Eldritch continued, “you normally need to have your experts travel to the worksite, but by providing a ‘see what I see’ kind of telepresence, the expert can look through the eyes of the technician who’s actually doing the maintenance. They can also annotate on the field of view of the technician, so they can point out particular features of interest in what the technician is seeing. That allows experts to offer support and perform inspections from anywhere.”
This suggests that the much-lamented skills gap in manufacturing could have a technological solution in the form of augmented reality. Rather than training every technician or machinist, companies could use AR technology to supplement their employees’ existing knowledge with engineering expertise delivered via telepresence.
This also opens up new possibilities for training.
As Kocic pointed out, “You can feed training information right on top of the actual parts and assemblies. So you can start mixing the execution with training, which ultimately makes the equipment easier to learn and use. That’s especially important given the general trend in manufacturing toward using less skilled labor than before. Before, metrology was done in air conditioned rooms, but that’s shifted to the factory floor, and to people who are not specifically metrologists but instead are usually manufacturing engineers or operators. So how can you serve that information in a better way? AR seems to fit in that environment.”
4. Quality Assurance
Metrology in general, and quality assurance (QA) more specifically, offers numerous potential applications for augmented reality. This video by CAQ AG Factory Systems, a developer of quality management software, offers a glimpse into how even manual inspections can be enhanced with AR.
Some manufacturers have already begun exploring the possibilities offered by using AR for QA.
Last year, SlashGear reported on a pilot program running at a Porsche assembly plant in Leipzig, Germany. A demonstration by the company showed Porsche technicians using augmented reality as a tool in the quality assurance process.
The basic idea is that quality professionals can take photos of parts or assemblies on vehicles under inspection, and then compare those images to ones provided by the company’s suppliers via an augmented reality overlay. Features that are out of specification can be highlighted by the overlay, enabling the Porsche technicians to identify the issue quickly and intuitively.
The eventual plan, as reported by SlashGear, is to link the cameras on the production floor directly to Porsche’s cloud-based parts database in order to enable real-time analysis of parts and assembled components. This could reduce inspection times significantly, though it would also seem to undermine the value of augmented reality in favor of automation.
Airbus has been using a “Mixed Reality Application,” or MiRA, to integrate digital mock-ups into production environments, giving assembly workers access to complete 3D models of the aircraft under production. According to Airbus, MiRA has been used on the A380 and A350 XWB production lines to check the integrity of secondary structural brackets, which hold hydraulics and other equipment in place.
“MiRA links a real object with its digital genome, transforming reality into an interactive world in which information about the object can be directly accessed,” said Nicolas Chevassus, head of industrial processes at Airbus Innovations. Airbus has reported that MiRA reduced the time required to inspect the 60,000 – 80,000 brackets in the A380 fuselage from 3 weeks to 3 days.
5. Automation
Given manufacturing’s seemingly inexorable trend toward increasing automation, one might naturally wonder whether augmented reality and automation are in competition with one another. Why worry about improving a worker’s access to information if that person will soon be replaced by a robot?
When asked whether AR is competing or cooperating with automation, a representative of DAQRI, the company which makes the DAQRI Smart Helmet, had this to say:
“[The] DAQRI Smart Helmet cooperates with automation because it uses common protocols and standards, it is designed to integrate to DCSs, PLCs, SCADAs, Histograms, CMMSs and ERPs, making information previously constrained to the control room available right on the helmet, wherever the worker goes.”
From the software side, Eldritch offered a somewhat different perspective:
“I think it is competing with automation, but in a good way. A lot of the companies we work with have unions concerned that by improving the efficiency of workers, we’ll be taking away jobs. But what augmented reality does is enable you to leverage context awareness via the Internet of Things to gather information about the environment and then feed that to the worker.”
“All these technologies that are coming together around artificial intelligence are going to augment the capabilities of the worker and that’s very powerful. I call it Augmented Intelligence. The idea is that you can take someone of a certain skill level and by augmenting them with artificial intelligence via augmented reality and the Internet of Things, you can elevate the skill level of that worker.”
“So, I think that as more jobs get automated we’re also going to be pushing more workers up the skill-chain, which will make them more competitive in the market. So augmented reality and automation may compete in the near-term, but I think there will eventually be cooperation between automated machines and augmented human workers.”
Kocic offered a similar view, with specific reference to the robotic cells already in factories today:
“If you look at any automation cell, there’s a big cage around it. There’s a bunch of robots moving around, loading stations, unloading stations, et cetera. How do you control that? There might be a dashboard or a small screen, but there’s very limited information pointing to where any problems might be.”
“I think AR is a much more collaborative tool, where you can walk up to a cell, ask some feedback questions and get responses through the AR that tell you where everything is. So, this robot might need service, or it will say ‘Look, you’ve got a problem on this load station,’ and then deliver some statistical data from measurements to try to point you in a better direction, affecting how the production runs.”
“So I always see AR in that way: allowing you to collaborate with the automation cells to address any issues that might come up,” Kocic added.
AR & VR in MFG
When it comes to manufacturing, AR is no mere marketing gimmick. The practical applications are there, and the potential benefits are enormous. There’s clearly a lot that augmented reality can do for manufacturing, but it may surprise you to learn that virtual reality can also be used to enhance manufacturing operations.
B-AIM