RemoteSpark: the future of mixed reality in mining

More than half of the country’s coal mines are managed by pro-Russian separatist militia.

Credit: DmyTo/Shutterstock.

As digital connectivity in mines is becoming essential, Kognitiv Spark chief technology officer and co-founder Ryan Groom thinks that the ability to use mixed reality underground is increasingly improving.

“Before, we used to go into the dark zone – your Wi-Fi didn't work, your cell phone didn't work, but now the mines are being wired for internet connectivity so [miners] can go to the deepest and darkest parts of a mine and still be able to get signal and get assistance,” he says.

RemoteSpark provides a video feed from HoloLens, which is a Microsoft-manufactured pair of mixed reality smart glasses. The HoloLens computer uses multiple sensors, advanced optics, and holographic processing to create holograms that can be used to display information, blend with the real world, or simulate a virtual one.

The feed is taken from the remote user to the expert, which ensures that the first person to observe a potential mining incident can do it hands free, which is quite critical. This initial step of the process is not mixed reality itself, it’s what comes next that is truly interesting.

Groom explains: “In addition, we have enough horsepower left to be able to do a true mixed reality call, which means we can create holograms while doing a call. Most products can do 2D, so you have video call and a PDF or a photo, while we can put 3D holograms right in front of you, whatever you're working on. These holograms can be annotated and animated.”

In cases where companies are working with complex machinery, if they can rely on a 3D model version instead of a 400-page manual, this would ensure a quicker and more efficient learning experience.

RemoteSpark is heavily reliant on data from mining sites in order to improve operational safety and focuses on things that cannot be easily seen with a naked eye, like gas leaks or radiation.

The system could also work with temperature, pressure, and pumps with high pressure, high heat liquid inside, which the HoloLens system recognises and creates holograms from.

“If you have a propane leak sensor that can detect a concentration of propane, you put a headset on and actually visualise that, or before you send a crew in to do a repair you can see what part the system has highlighted. It's early days for that, but through holograms we get the most natural way to consume data,” Groom says.

When it comes to testing of the system, Kognitiv Spark has worked with partners to test network bandwidth capability and figure out how low bandwidth interferes with the network. Testing has proven to be especially effective in rural Canada where there are plenty of spots with poor internet connectivity and it’s easy to detect how bad a network gets before the video signal breaks up.

Broom shares that, before Covid-19, much of the mixed reality testing was quite immobile as it took place in a boardroom or an office.

“But now people need it immediately, so the testing is actually having to deal with real situations. We've had people in South Dakota take it out to oil and gas sites where connectivity is very poor and they put on their headset and do real live testing. We can test it here in our own backyard, but the best test is when the customer takes it into harsh condition and it works. We find that’s a great test,” he says.

The creation of the technology has not gone without challenges and hardships, especially in the initial building stages of the system.

While a large percentage of people that build for HoloLens or mixed reality use a popular gaming engine application called Unity, the team found that it needed more horsepower for the application. They decided to build their own rendering, which proved to be no mean feat for a small team. They used existing materials, from which they rewrote about 50% to target what they needed to, building a video call application and encoding the video on a home unit.

“To be able to balance all those very heavy workloads and have that run on the HoloLens, which is just a mobile device, and then on an ARM processor…For that, we needed to be as efficient as possible,” Broom recalls.

Since its creation of the ‘RemoteSpark’, the mixed reality tool has been collaboratively used by truck sales company Mack Defense Canada and Canadian technology consulting group Valcom for remote work in the industrial sector.

Their joint efforts then led to the first remote support call initiated by a Valcom technician in early 2020. Since then, Mack Defense has reported positive results from trans-Atlantic mixed reality calls between field service representatives in Europe and experts in Canada.

These two companies in particular are helping Kognitiv Spark with trials by taking the HoloLens software to extreme locations all over Europe to support deployments and fix equipment.

According to Kognitiv Spark, the system is also being used by the Royal Canadian Navy, Royal Canadian Air Force, and the Canadian Army, as well as defence and industrial organisations.