Remote access to operational industrial equipment is a thorny topic. General adoption is slow, but some end users—usually smaller, flexible, and less IT-heavy CPGs—are beginning to take advantage of what’s available with remote dashboards and uptime trackers. A few have gone a step beyond, allowing OEM partners past the firewall for one-time troubleshooting, or ongoing machine monitoring and data collection.
This recent traction is notable because, despite being available for some time, there hasn’t been much end-user pull. Safety was a huge concern. And early on, OEMs themselves looked at remote monitoring as a conflict, potentially reducing aftermarket revenues with real-time troubleshooting.
But recently, OEMs shifted the remote focus from active access to more benign monitoring from afar, a baby step forward on the path to adoption. Trade show attendees tend to consist of the functional people who look at the bottom line, not the maintenance and operations people on the machines. So remote troubleshooting became less of a push than remote monitoring for predictive maintenance and improved OEE.
The resulting output of dashboards and OEE monitors was a one-way, outbound street of data without much in the way of physical safety concerns for operators. Those end user engineers and operators tasked with increasing uptime, limiting unplanned downtime, and improving OEE immediately saw value, and continue to see value.
But with safety fears allayed, the focus swiftly changed to security—data pushed out of the facility’s four walls is less safe, and anything coming back in isn’t to be trusted at all. Of course, data could stay in-network, allowing IT folks at least some buy-in. But now that the remote monitoring toothpaste is out of the tube, and the benefits are apparent, opportunity-seeking OEMs are demonstrating their capability with a full complement of remote capabilities—ongoing remote monitoring, in-bound remote troubleshooting, even potential e-commerce and part ordering. It’s making end-user IT departments uncomfortable.
What’s not to love?
“It doesn’t take much effort to turn up plenty of stories about how improper use of network layering can cause catastrophic results,” says Rick Rice, application engineer, Crest Foods, Ashton, Ill. “While I had my own reservations about opening our actual packaging equipment to the risk of cyber intrusions, our IT department had major concerns. With any attempt to connect our assets to higher level communications, there is a fear of outside parties gaining access your critical operations.”
For all the potential upside that remote access or monitoring may hold for machine operators and their OEMs, it’s the opposite for the IT warriors charged with defending their company’s network turf. For them, there is no upside.
But that’s not to say that end users don’t see the potential. They understand the theoretical value of remote monitoring, diagnostics, access, maintenance, and improvements to facilitate uptime. The problem is that there are a lot of different ways to skin the remote access cat, and so many colloquial ways to describe the technology, that it becomes complex. Add to that the many different voices within the CPG community that are championing or objecting to each potential methodology.
It’s up to OEMs to clearly articulate, in a common language, the type of remote access that they propose, and how they plan to accomplish it.
Personas, transparency, and indemnity
OEMs that have successfully made their case recommend getting all stakeholders into the same room to lay bare the spectrum of potential positive to potential negative consequences. In this environment, an OEM should be able to demonstrate that it’s not a matter of equal but opposite forces. The good can, in fact, outweigh the bad.
According to Dr. Mohan Sawhny, the McCormick Foundation Chair of Technology at Northwestern University’s Kellogg School of Management, understanding the different personas involved is key to encouraging adoption of any technology, and remote access is no different. The globally recognized expert in business innovation and disruptive technology sketched a mental template of four stakeholders at the CPG/brand owner level whom remote might affect.
One might be a maintenance engineer responsible for mitigating downtime. Another might be a shift operator, responsible for actual production. A third would be an IT person, responsible for security and integration of the shop floor to the top floor. A fourth persona might be the sales person.
“You must understand what motivates each of these players, what they stand to gain, what their KPIs and biases are, and where they line up on a continuum of active support to active opposition to a new technology,” Sawhny says.
In this case, the ratio of support to opposition might be three to one. The IT persona, with his “do no harm” mentality, is not paid in the potential upside, but in protecting against possible downside.
“The second thing is that the actual risks need to be quantified and weighed against potential good, because the perception of risk often exceeds the actual risk,” he says. “And on the flip side, you should be able to quantify the benefits of reduced downtime and reduced maintenance, to make the economics more transparent. That can be very compelling.”
Another approach that Sawhny suggests is indemnification, a known dirty word among OEMs. But he points to the consumer world and the rise of Hyundai, with its 10-year, 100,000-mile warranty, as an example of it working. If OEM security protocols are up to the task, then some sort of risk-sharing program could go a long way, according to Sawhny.
Early adopters
Larger companies are known to be hesitant to be offloading their production and equipment data to outside servers and to open it up on the internet. In many cases, the dictate is that information isn’t leaving the four walls, and if it absolutely must leave, it will be on their network. Meanwhile, some smaller, more agile companies, often with smaller IT departments and less red tape to clear, have taken advantage of what’s available.
One such early adopter end user is Barrie House Coffee, Elmsford, N.Y. The full-line coffee and tea business worked with Israeli OEM Pack Line, Ltd. to develop a packing system around a patented new coffee capsule format. The line incorporates Canadian OEM Nuspark’s secondary and tertiary packaging systems and All-Fill Inc. auger fillers.
“We’ve given them 100 percent access to all the cameras in the production areas,” says Shay Zohar, Barrie House director of sales and marketing. “They have full access into software, they see our workflow, our work volume, our efficiency, our alarms. Everything we have is fed live to them. By our sharing information this way they can really understand what’s happening on their machines. Only by seeing it in real time can they have this kind of deep understanding. It’s an approach that allowed all of us to do amazing things.”
A big part of the real-time visibility he speaks of is delivered by an industrial M2M router and data gateway from eWON called Flexy. All connections run through industry standard VPN protocols to guarantee a safe and secure connection that prevents network intrusions. OPC UA technology offers interoperability between platforms from multiple vendors and enables new machines, as well as legacy PLCs for IoT integration.
“We share and collect all data in and out of the production line via the eWON hardware with custom software written by Pack Line’s chief engineer and software genius, Eyal Dafna,” Zohar says.
A West Coast CPG that didn’t wish to be named discussed their leap into remote access and monitoring by way of ARPAC’s BeConnected® system. The company aimed to replace a complete food packaging line from the filler onward, and enlisted ARPAC, a Duravant company, for the installation.
“We’ve put in several systems with this type of capability over the past five years,” the project manager says. “We realized that it becomes more efficient to be able to let qualified people troubleshoot. That could be plant personnel who don't happen to be onsite. So it could happen on a weekend, where maybe you don't have the management and maintenance support that you normally would during the week, so they could remotely access it. And, even more importantly, depending on how complicated the issue could be, the OEM, ARPAC here, would have the capability to go in and look at the program. We got our IT people onboard, let them in on what kind of technology it was, and what the benefits were so they could weigh out the risks versus benefits,” he says. “They felt comfortable with the level of security and gave us their blessing.”
Implementation consists of a single Ethernet connection supplied by a company’s IT department with standard security that allows an outbound connection to Microsoft’s Azure Cloud, plus security handled through methods developed by HMS Anybus, Rockwell Automation, and Microsoft. In addition to creating a conduit for data collection, this connection to the Cloud can be used by ARPAC’s engineers for remote access. BeConnected delivers real-time reports and data charts that are accessible from any web browser. Manufacturers can see graphical data representing how well their equipment is running and how much is being produced on an hourly, daily, weekly, or monthly basis. Reports are simple to read and easy to use, plus they are configurable to whatever time duration is desired, says Brian Ormanic, lead applications engineer, ARPAC.
"It's the tags and software defined by Rockwell Automation that enable the technology and make it easy to use. And then it's hardware that connects the machine, and communicates the information that's in the PLC. The data are placed into a format that can be sent to a server through the Internet. And it's just a web server, which really simplifies it. You don't need any special hardware or IT firewall access. You just need access to the web,” Ormanic says. “So, that does cut out a lot of security risk—you are confined to the data within that web page, which doesn’t even reside on the customer’s network, it is in the Microsoft Cloud Server. In a web browser, you punch in a web address, and that's really all you have access to. You can’t connect to a computer network, corporate server, or anything else. You can't see folders or files. It's just from the PLC to the Internet. Plus, network traffic is minimal because only small bits of data are used, and that just gets pushed up to the Internet.”
Acquiring performance monitoring capability
In mid-2016, Pro Mach acquired Canadian software company Zarpac. The company had been known for its ZPI Performance Monitoring Systems, building in-depth data collection systems into a simple interface to follow KPIs, unplanned downtime analysis, live data, and more. The system is built in such a way that may appeal to IT departments seeking to keep more control in-house.
“In our model, we take all the reporting, all the data collection, and all the drivers necessary and they reside within the four walls of a plant, giving that plant’s local IT departments a lot more control and accountability over the system,” says Chris Hough, general manager, ZPI. “With the server in the end user’s environment, we don’t have a proprietary solution for remoting in. It’s first up to the clients to see whether they want to make that data available, and then if they do, it’s up to the customer to decide which method we can use to remote in to troubleshoot.”
But since the acquisition, Zarpac has adjusted to support OEMs’ hardware- and equipment-selling. It now offers a remote service and remote monitoring option that allows access during any machine failures or support requests.
“Monitoring works very much the same way that we access the system here, but it would allow some of those OEMs to remote in, access the server and not only be able to troubleshoot that machine, but also view a historical record of what happened prior to any breakdown,” he says.
ZPI’s diagnostics require up-to-the-second information to paint an accurate picture of the production process. So, poor connectivity can raise data reliability questions. A lot of newer machinery is precision equipment with specific timing functions that need to be recorded properly with as few points of failure and latency between the action and the record as possible. Both data recording and retrieval can be slower when accessing external networks.
With ZPI, most active users of the system remain physically within the plant, so the system is and should be focused and optimized for those users. Maintaining the data collection in the four walls of a plant resolves connectivity and latency issues. While being tied directly to the line, the ZPI system is configured in a way that, should there be any issues with the system itself, production will not be affected.
“Data reliability is huge,” Hough says. “We have built in tools to help troubleshoot, resolve, and notify users of any potential irregularities in the data collection process, such as interruptions in collection, data backups, and restoration time. This automation is hardware-neutral because it needs to apply to as much hardware as possible. By tying these alerts into the maintenance and support software the system can be configured to alert both plant staff and vendors of any warning events signaling larger machine issues in the short term.”
ZPI supports the system in a way that doesn’t require OEMs to have dedicated, in-house resources. The remote service option negates as many of the security concerns as possible, while allowing the clients to maintain system and data ownership.
“Using the remote service option with a detailed diagnostic history helps speed up the initial investigation phase without the requirement of site time,” Hough says. “The ZPI System will still run without an active support contract, while external sources will likely stop with potential loss of all historical information. Cloud Services introduce more points of failure in the data collection process.”
End user retrofitting considerations
Even though end-user adoption isn’t ubiquitous, OEMs are increasingly baking remote access capabilities into their latest machine versions and offerings in anticipation of a tipping point. Dr. Sawhny thinks that’s a wise bet.
“It is absolutely an inevitability and what is happening today is that the capability has been hard-coded this into the OEM’s equipment. So, the GEs of the world, when they make wind turbines or when they make locomotives or when they make aircraft engines, all this complex machinery is coming with monitoring built in; with IIoT built in,” he says.
But if end users decide that they’ll just wait until their machines turn over to buy new ones with remote access capabilities, that will delay adoption.
“The challenge for the OEMs is to create a painless, seamless, and high-value retrofitting program to drive up adoption in the installed base,” Sawhny says.
An installed base is always going to be much larger than the number of machines that an OEM is going to sell in a particular year. So, getting those existing machines remote-ready will be the challenge for OEMs to vastly accelerate adoption. And OEMs who are able to do so may be able to help steer which directions the technology is going next, while strengthening ongoing ties between the OEM and its customers.
Crest Foods’ Rice shared an example of retrofitting in action via a commercially available platform called Redzone. The company’s vintage equipment led it to find ways to retrofit to capture OEE data.
“By vintage, we are talking about before the advent of the PLC,” he says. “This older equipment is based on relay logic and it was the challenge of getting production information out of these somewhat archaic machines that provided us with our answer. If the machine can't be connected to an Ethernet network, or any network for that matter, then there wasn't a way for an outside entity to drill down into the operation of the machine itself.”
So Rice and his team installed a mini-PLC as a data collection point on each line in a group of two lines. An input signal was brought back to the mini-PLC from each product producer, in this case a vertical or horizontal pouching machine. Another input signal was brought back from each finished product position. This might be the output from a case packer or sealer but could also be a photoeye detecting accepted pouches after a checkweigher or the accept signal from the checkweigher itself. Each of these cycle events triggers an individual counter in the mini-PLC.
“The OEE data collection system monitors the counter accumulator and uses that information to determine if the line is up or down and how much product was produced in order to make the end case or bulk container of products. Since the data collection point wasn't physically connected to the production machine then a gap (called an air gap) exists that provides the ultimate security from network intrusion. The individual machines have signals that are derived from the cycle of the machine and trigger a small mechanical relay to provide the trigger signal back to the data collection point. From this simple setup, we were able to get OEE data from devices that were made nearly 40 years ago,” Rice says.
The purpose of Redzone is OEE and quality monitoring but the added benefit is a means by which to remotely monitor the status of each production line. From this real-time status, the end-user could prompt further activity, still remotely, where an OEM would dial in to their particular machine or process to assist the on-site people in returning the line to peak operating condition. Redzone is capable of automatically sending out emails to prompt that further action.
In order to use that data, the company incorporated the Redzone platform, an Apple iOS based system with iPads used as operator interfaces. The interface uses the input and output counters to determine if the line is running. Based on that, problems are flagged on the iPad that prompt an operator to assign a cause. So, the machines are retrofitted, and primed for remote access. But at this time, there are no plans to remotely change machines’ states of functions–Redzone is for data collection only, and the ability to monitor and react in real time is the system’s benefit. Remote monitoring and troubleshooting hasn’t been brought up yet, but potential for their use is built into the system.
This represents a microcosm of where many end users are on the adoption spectrum, whether on newer or older equipment. With the benefits tantalizingly within reach, and the risks becoming better understood, it may be a question of time before a remote access acceptance tipping point is reached.
