RemoteSpark: the future of mixed reality in mining
Kognitiv Spark is one of the companies tapping into the ability to visualise data and respond to it in real time and in-situ, providing miners with a significant advantage. Yoana Cholteeva looks at the abilities of the company’s ‘RemoteSpark’ mixed reality tool and how it can benefit the mining industry.
M ixed reality could play a significant role in the future of mining, especially in the post-Covid-19 world where reduced travel and physical distancing are likely to continue for some time. Now more than ever, allowing for mining experts to be able to guide and support field service staff in remote areas or hazardous conditions is of vital importance.
One solution, the newly established RemoteSpark software platform, built by Canadian augmented and mixed reality support operator Kognitiv Spark, has been created to enable remote workers to establish a secure, low-bandwidth mixed reality connection between people anywhere in the world. It creates a shared audio and video connection while allowing experts to share and use images, 3D assets, and various documents, which appear as holograms in the users’ real-life environment.
More than half of the country’s coal mines are managed by pro-Russian separatist militia.Credit: DmyTo/Shutterstock.
More than half of the country’s coal mines are managed by pro-Russian separatist militia.
Capabilities of Kognitive Spark’s ‘RemoteSpark’ mixed reality tool
As digital connectivity in mines is becoming essential, Kognitiv Spark chief technology officer and co-founder Ryan Groom thinks that the ability to use mixed reality underground is increasingly improving.
“Before, we used to go into the dark zone – your Wi-Fi didn't work, your cell phone didn't work, but now the mines are being wired for internet connectivity so [miners] can go to the deepest and darkest parts of a mine and still be able to get signal and get assistance,” he says.
RemoteSpark provides a video feed from HoloLens, which is a Microsoft-manufactured pair of mixed reality smart glasses. The HoloLens computer uses multiple sensors, advanced optics, and holographic processing to create holograms that can be used to display information, blend with the real world, or simulate a virtual one.
“We can put 3D holograms right in front of you, whatever you're working on.”
The feed is taken from the remote user to the expert, which ensures that the first person to observe a potential mining incident can do it hands free, which is quite critical. This initial step of the process is not mixed reality itself, it’s what comes next that is truly interesting.
Groom explains: “In addition, we have enough horsepower left to be able to do a true mixed reality call, which means we can create holograms while doing a call. Most products can do 2D, so you have video call and a PDF or a photo, while we can put 3D holograms right in front of you, whatever you're working on. These holograms can be annotated and animated.”
In cases where companies are working with complex machinery, if they can rely on a 3D model version instead of a 400-page manual, this would ensure a quicker and more efficient learning experience.
AusProof is celebrating 25 years of business in Australia in 2019.
Increased safety and real-life testing of the technology
RemoteSpark is heavily reliant on data from mining sites in order to improve operational safety and focuses on things that cannot be easily seen with a naked eye, like gas leaks or radiation.
The system could also work with temperature, pressure, and pumps with high pressure, high heat liquid inside, which the HoloLens system recognises and creates holograms from.
“If you have a propane leak sensor that can detect a concentration of propane, you put a headset on and actually visualise that, or before you send a crew in to do a repair you can see what part the system has highlighted. It's early days for that, but through holograms we get the most natural way to consume data,” Groom says.
“The best test is when the customer takes it into harsh condition and it works.”
When it comes to testing of the system, Kognitiv Spark has worked with partners to test network bandwidth capability and figure out how low bandwidth interferes with the network. Testing has proven to be especially effective in rural Canada where there are plenty of spots with poor internet connectivity and it’s easy to detect how bad a network gets before the video signal breaks up.
Broom shares that, before Covid-19, much of the mixed reality testing was quite immobile as it took place in a boardroom or an office.
“But now people need it immediately, so the testing is actually having to deal with real situations. We've had people in South Dakota take it out to oil and gas sites where connectivity is very poor and they put on their headset and do real live testing. We can test it here in our own backyard, but the best test is when the customer takes it into harsh condition and it works. We find that’s a great test,” he says.
Technical challenges and partnerships
The creation of the technology has not gone without challenges and hardships, especially in the initial building stages of the system.
While a large percentage of people that build for HoloLens or mixed reality use a popular gaming engine application called Unity, the team found that it needed more horsepower for the application. They decided to build their own rendering, which proved to be no mean feat for a small team. They used existing materials, from which they rewrote about 50% to target what they needed to, building a video call application and encoding the video on a home unit.
“To be able to balance all those very heavy workloads and have that run on the HoloLens, which is just a mobile device, and then on an ARM processor…For that, we needed to be as efficient as possible,” Broom recalls.
Since its creation of the ‘RemoteSpark’, the mixed reality tool has been collaboratively used by truck sales company Mack Defense Canada and Canadian technology consulting group Valcom for remote work in the industrial sector.
“Mack Defense has reported positive results from trans-Atlantic mixed reality calls.”
Their joint efforts then led to the first remote support call initiated by a Valcom technician in early 2020. Since then, Mack Defense has reported positive results from trans-Atlantic mixed reality calls between field service representatives in Europe and experts in Canada.
These two companies in particular are helping Kognitiv Spark with trials by taking the HoloLens software to extreme locations all over Europe to support deployments and fix equipment.
According to Kognitiv Spark, the system is also being used by the Royal Canadian Navy, Royal Canadian Air Force, and the Canadian Army, as well as defence and industrial organisations.
AusProof is celebrating 25 years of business in Australia in 2019.