MTConnect is a middleware that has an internal messaging protocol and standard data formats for general manufacturing systems. It consists of an adapter for each machine and an agent as a server. The adapters collect machine data such as position, feed rate, tool, workpiece, and sensors, which are frequently used in manufacturing. The adapters send the data with its time stamp to the agent using the TCP/IP protocol. The MTConnect agent works like a server that gathers the data, providing current data and its history as an XML document format after it receives the requests from user applications. The user application can be programmed with any computer language as it can send requests to an agent and parse an XML document; therefore, it can be a virtual module, such as a digital twin, a database repository, or an autonomous decision maker.

However, one of the drawbacks of MTConnect is that it has no standard communication rules to send the data or control commands back to machines. To add the function, an additional communication network node and an executive program to perform the request are required. Fig. 1 illustrates concepts of the bidirectional middleware with MTConnect, showing how the digital twin retrieves data and commands to machines. This allows an autonomous feedback control loop that integrates cyberspace with the physical world [16] so that the digital twin checks collision from received position data and commands machines to adjust current operation, such as changing robot paths. The following steps contain details of building the framework using bidirectional middleware.

1) Connect adapters to collect axis positions or joint angles from machine controllers. PCs for the adapter are connected to the controllers via RS232, Ethernet, and any other industrial protocols. The adapter program should access the controller, collect machine data, and send the data to the MTConnect agent.

2) The agent collects data from all the adapters. The PC for the agent is connected to multiple PCs for adapters via TCP/IP protocol.

3) A digital twin sends requests for machine data to the MTConnect agent, and the agent sends the requested XML document. Then, the digital twin constructs 3D geometries. In this paper, streams of previous positions are used to predict future trajectories and therefore the collision of 3D geometries in the digital twin.

4) After predicting a collision, the digital twin sends control commands to each executive program, and each machine takes actions to avoid collisions. The communication between the executives and the digital twin uses TCP/IP protocol, sending control commands and receiving replies to check the communication.

The digital twin consists of 3D geometries. The application program containing the digital twin sends queries to the MTConnect agent, and the agent replies with a list of position data. A 3D CAD model of each machine is separated by entities, and they are re-grouped according to their moving axis. Then, the grouped parts are allocated in a virtual 3D space according to positions or angles collected from MTConnect so that it mimics actual machines. Fig. 2 is a flowchart of the digital twin, which describes how to construct the geometries, perform collision prediction, and command machines to avoid the collision. The details of generating future trajectories and the collision checking algorithm are explained in the next sections.

Predicting a collision is based on generating future trajectories for each machine. The research assumes that the machines perform a job repeatedly, such as mass production in a factory. Future trajectory is calculated from two data sets: one complete cycle of trajectory X_t and history of the trajectory Y_t until when collision prediction is requested. X_t is defined already in the application and Y_t can be retrieved from either the MTConnect agent or continuously gathered and saved as a list of trajectories. Since the data gathered from the MTConnect adapters has slight variation of time interval, the time series X_t and Y_t are linearly interpolated for equal sampling time, 20 Hz. When the digital twin receives the order to predict collisions, the amount of remaining trajectory until the end of the cycle is calculated using cross-correlation as Eq. (1),

where μ and σ are the mean and the standard deviation, respectively. The time τ_max, the first maximum with a zero first derivative and negative second derivative of the cross-correlation ρ, represents the time at which X_t and Y_t coincide. After the remaining series of the one cycle is calculated, several cycles of X_t are concatenated for repetition of the cycle. The predicted trajectory Y_pred is calculated as Eqs. (2) and (3),

where n is an integer that satisfies 0 < L < T’. Fig. 3 illustrates the algorithm.

To detect a collision between two 3D geometries, the oriented bounding box (OBB) and the separating axis theorem (SAT) are used. OBB is a rectangular block with a minimal size that contains a 3D entity. After the OBBs are calculated, a pair of OBBs is tested for intersection using SAT. The theorem works by projecting the two OBBs onto a potential separating axis and checking if their projected intervals overlap. If a single axis can be found where the projections do not overlap, the two objects do not collide. In theory, all arbitrary axis of two free-form shapes should be checked to find the overlap. However, the minimum number of axis to be tested in OBB is 15: 3 face orientations of the box A (unit vectors A₁, A₂, A₃), another 3 face orientations of box B (unit vectors B₁, B₂, B₃), and 9 combinations of two orientations A × B [13]. The projected length l_A of the line from the center of the box A to the direction of the box B is described as Eq. (4),

where a_i, and b_i are the half length of the box’s edges, and L is a unit vector of the arbitrary testing axis. As in Eq. (5), the condition of no overlapping of boxes is that the sum of two projected lengths l_A and l_B does not exceed the projected line of distance T between two center points. Fig. 4 describes the concept of SAT.

In the digital twin, the collision between entities is checked using the principle. After translating or rotating 3D geometry entities from positions or angles that are sampled from the predicted trajectories, OBB with SAT are checked with all entity pairs. Collision time is predicted based on the time when two entities intersect. To reduce calculation time, one can define a list of blocks to check for collision or decrease the number of times to be checked.

One of the experimental setups was prepared in Yonsei University, Korea. The system was composed of two lab-developed PC-based CNC controlling a three-axes gantry type stage and a single rotary motor attached to a white plastic rod on its end, respectively. The CNC was equipped with an EtherCAT master stack to control its periphery devices over EtherCAT. The MTConnect adapter was developed to gather CNC data such as position, velocity, load, and G-code block numbers from each CNC using TCP/IP communication features provided by Boost.Asio library. The MTConnect Agent released by MTConnect Institute runs on the same PC with MTConnect adapters and collected data from adapters via localhost communication. MTConnect client, shown in Fig. 5(a) which works as a digital twin in this paper for visualizing and detecting collision was developed using a commercial 3D graphics library (Eyeshot, devDept). The Testbed #1 is shown in Fig. 5(b).

Another testbed is located at Purdue University in the United States. It has a 6-axis industrial robot and a 3-axis CNC router. Each machine has its adapter and executive inside the embedded computer (Raspberry Pi 3 B+, Raspberry Pi Foundation). JOpenShowVar [17] is used to connect between the robot and the MTConnect adapter. An MTConnect agent is installed on a PC and gathers data from the adapters. The digital twin, developed using the same library in Testbed #1, sends and receives data on another PC. Fig. 6 describes the experimental setup.

Testbed #1 shows a simple scenario of the material loading and unloading using a rotary motor and machining in 3-axis stage. The 3-axis stage drew a tilted diamond-shaped trajectory which involves its X, Y, and Z-axis, while the end of the plastic rod attached on the rotary motor followed a semicircle-shaped trajectory. The cycle-time was about 12 and 15 seconds for the rotary motor and the stage, respectively. The stage and rotary motor were controlled by each independent CNCs which do not communicate with each other. To check and predict collision, the digital twin application proposed in this work was utilized. The digital twin application gathered recent history of two motion systems from MTConnect agent in every 5 seconds, and the gathered history was resampled in time interval of 50 milliseconds, which is the time interval of the one-cycle trajectories of each system recorded beforehand. Cross-correlation was computed and the lag of the gathered trajectory with respect to the one-cycle trajectory was calculated. Using the lag, future trajectory that followed the collected trajectory was predicted. Figs. 7 and 8 show the trajectories which was gathered and predicted based on the proposed framework.

Next, the digital twin application simulated the predicted trajectories of each system and utilized OBB to check collision. Entities which were predicted to collide each other were highlighted in yellow, and the predicted time of the collision was shown in red text. As shown in Fig. 9(a), the digital twin application predicted that the collision between the z-axis part of the stage and the plastic rod on the rotary motor would occur at the time of 20:06:22. To verify the collision prediction feature, it was checked whether the stage and the rotary motor collided at the predicted moment of collision. Figs. 9(b)–9(d) show the images of the testbed before the moment of collision. It was observed that the 3-axis stage and the rotary motor actually collide at the predicted moment in a very similar appearance rendered by the digital twin application in Fig. 9(a).

In Testbed #2, shown in Fig. 10, an applicable scenario of real-time collision prediction and machine control is implemented using the framework.

Fig. 10 illustrates the scenario. At first, the robot stops, and the future trajectories of the router and the robot are predicted. Next, the digital twin checks if collision happens from several points of the predicted trajectories. If there is no predicted collision at the digital twin, the robot decides to go straight across the CNC router. Otherwise, the robot detours the CNC router to avoid the collision. One of the two decisions is sent to the executive program of the robot. The executive changes a mode variable of the robot controller, and the robot is programmed to change its path according to a mode variable. Therefore, the robot can change its path according to the command of the digital twin. While the robot path is controlled by the application, the CNC router runs a G-code program repeatedly without any disturbances.

Fig. 11 compares gathered trajectories from MTConnect with one cycle trajectory of the CNC router. The digital application gathered position data from the MTConnect agent, and performed linear interpolation with 50 milliseconds interval. The one cycle trajectories were captured from the gathered trajectory, and were saved as a file that the digital twin application can read anytime. Three X, Y, and Z axis show phase difference of 1.75 seconds (τ_max) which were the same as calculation using cross-correlation. Fig. 12 is a result of cross-correlation between three pair of trajectories. It shows that the first maximum cross-correlation happens at 1.75 second (τ_max) which is the phase difference in the Fig. 11. Therefore, as in Fig. 13, a part of X_t after 1.75 seconds with several cycles of X_t are concatenated at the end of gathered trajectory Y_t. In the testbed, the predicted robot trajectories are pre-loaded one-cycle trajectories with predefined 3 seconds delay from the start considering calculation time of collision at the digital twin. Fig. 14 illustrates the predicted trajectories of the robot with initial delays. One cycle trajectories of the robot are attached after 3 seconds. Therefore, trajectories of two machines are predicted to be used for collision prediction.

After predicted trajectories were acquired, the digital twin checked geometric collision at four designated points of the trajectories. Figs. 15 and 16 show the results of collision prediction and machine control. Each time frame has captured images of the digital twin and the web camera, respectively. In Fig. 15, no collision was predicted because the spindle unit, represented as a pink box, was retracted to the right side so that the robot could move above the router’s working area. However, Fig. 16 shows that the robot would collide with the movement because the spindle unit was supposed to be on the left side that the two would collide. The two yellow entities in the digital twin indicate collided blocks at the collision checking stage. As a result, the robot detoured the router to avoid collision. The result verifies that the digital twin combined with bidirectional middleware detects and predicts collision from acquired position values.