E-mail
Print
facebook
twitter
Linkedin
google+
1 Introduction
Directed Energy Deposition (DED) process is a type of additive manufacturing (AM) technology that utilizes a laser, electron beam, or plasma arc to melt and deposit metal powder or wire in a layer-by-layer. The DED process offers high design flexibility and material efficiency, making it advantageous for metal part repair and customized manufacturing compared to traditional subtractive machining. This technology is widely applied in the aerospace, defense, and automotive industries, proving particularly effective for restoring existing components or fabricating new structures [1].
However, as illustrated in Fig. 1, the DED process is highly sensitive to various process parameters, where a slight change in environment can lead to defects in the final product. Therefore, precise monitoring of the molten pool behavior during the process and the development of real-time quality assessment technologies are essential.
Monitoring technologies for the DED process have been evolving to address this necessity, focusing on various sensor data and AI-based quality diagnostics. Recent studies have adopted approaches that utilize multiple sensors (such as temperature, imaging, and acoustic) to collect process data and perform real-time fusion analysis.
In sensor data-based DED monitoring, various approaches have been explored, including infrared (IR) camera-based temperature monitoring [2], vision image analysis of molten pool [3], acoustic sensor-based evaluation of process stability [4,5], and air quality measurement during the process [6]. These technologies contribute to the early detection of various defects or hazards occurring during the manufacturing process.
Meanwhile, in AI-based quality diagnostics, data obtained from various sensors mentioned above are utilized for AI training to predict product quality. Particularly, studies have been conducted on detecting process anomalies through multi-sensor fusion [7–10]. Additionally, research has been carried out to predict the thermal and mechanical characteristics of the process using physics-based Finite Element Method (FEM) simulations [11].
While these studies contribute to enhancing process reliability, the integration of digital twin technology is essential for improving real-time analysis and user experience. A digital twin is a technology that simulates or visualizes physical data from the real world in a digital environment in real time. Its primary applications include remote monitoring and process control, as well as process visualization and optimization [12].
In the environment where the DED process is performed, the melting of metal powder can generate hazardous gases for humans, and the presence of fine powder particles can degrade air quality. By utilizing a digital twin, air quality can be monitored in real-time during the process, ensuring a safer working environment. Furthermore, integrating a digital twin synchronized with the actual process and Virtual Reality (VR) technology enables remote monitoring, allowing users to experience the manufacturing process in an immersive manner. This approach also facilitates a faster response to process anomalies, enhancing overall operational efficiency.
Therefore, this study aims to develop and implement a data transformation AI for predicting temperature data to enable immersive monitoring in a digital twin-based DED process. To achieve this, a loss function is applied to facilitate the training of a molten pool dataset using a 3D Convolutional Autoencoder (CAE) model, and its effectiveness is evaluated. Furthermore, the developed AI is integrated with the digital twin system to visualize temperature data in real time without the use of temperature sensors, thereby establishing an immersive monitoring environment.
2 Experimental Setup & Data Processing
2.1 Experimental Setup
In this study, first, a data collection testbed, as shown in Fig. 2, was constructed and utilized to acquire molten pool images and temperature data for DED process monitoring. A DED head capable of supplying powder, gas, and laser was installed on a CNC machine. A Charge-Coupled Device (CCD) sensor was mounted to capture images of the molten pool, while an Infrared (IR) camera was attached to collect a two-dimensional temperature map of the molten pool.
These sensors were mounted on a plate connected to the DED head, ensuring that they moved along with the molten pool as the system traversed each axis. This setup allowed for consistent positioning of the molten pool within each frame, facilitating systematic image data acquisition.
As mentioned above, the DED process involves a wide range of process parameters, making optimization particularly challenging. Among these, laser power and scanning speed serve as dominant process variables, as they have the most significant influence on the formation of the molten pool during the process.
Therefore, two conditions were considered in this study to distinguish the deposition energy density: (1) a state with a normal energy density and (2) a state where excessive energy is supplied to the molten pool due to high laser power and low scanning speed. Under these conditions, single-track and single-layer processes were conducted. Table 1 presents the fixed process parameters used during the experiments, while Table 2 details the laser power, scanning speed, and the number of experimental repetitions for each condition. The process parameter information for the normal energy density condition was referenced from a prior study conducted using the same equipment [13].
The qualitative assessment of the deposited tracks after the experiment is shown in Fig. 3. In the case of tracks deposited under normal energy density conditions, a uniform width and height were visually observed. In contrast, tracks deposited under excessive energy density exhibited significant variations in width and height, despite being processed under the same conditions. Surface defects, such as black scorched marks, were observed due to the rapid melting and solidification of the material powder. Additionally, the instability of the molten pool led to spatter formation, resulting in incompletely melted powder particles scattered around the track.
2.2 Data Processing and Analysis
Through the experiments, synchronized CCD and IR image data were acquired. The imaging system captured molten pool images at a frequency of 27 Hz. The image acquisition program was developed based on the Python Software Development Kit (SDK) provided by each sensor manufacturer.
A total of 5,854 CCD images and 2D temperature maps were collected during the experiment. Among these, 4,975 images were obtained from deposition time points where the maximum temperature in the temperature map exceeded 1400°C. Specifically, 3,357 images corresponded to the normal energy density condition, while 1,618 images were associated with the excessive energy density condition.
Subsequently, all CCD images and 2D temperature maps underwent preprocessing, as illustrated in Fig. 4, to prepare them for AI training. For CCD images, image thresholding was applied to extract the contours of the molten pool region. The contour coordinates were then centered within the image by cropping to a fixed size, followed by resizing to standard dimensions. For 2D temperature maps, the coordinates of the maximum recorded temperature were identified, and the images were cropped to ensure that this point was centrally positioned. The temperature values were then normalized to a 0–1 range, corresponding to the IR camera’s measurement range (900–2,450°C).
Through this preprocessing pipeline, a Vision-Temperature image dataset was constructed, enabling effective training of the AI model described in the following sections.
To verify whether the constructed Vision-Temperature image dataset shows distinguishable data distributions, a feature analysis was conducted on both vision and temperature images.
The process and results of this analysis are illustrated in Fig. 5. For vision images captured by the CCD, the area and aspect ratio of the molten pool were extracted to evaluate its stability. In the case of temperature images, the maximum temperature value was extracted. Subsequently, the probability density function (PDF) of the extracted features was visualized to examine the overall distribution characteristics of the dataset.
For the normal energy density condition, the molten pool’s area and aspect ratio tend to remain within a consistent range compared to the excessive energy density condition. Additionally, the maximum temperature is stably maintained above 2,200°C, indicating a controlled deposition process. In contrast, for the excessive energy density condition, the molten pool exhibits a larger area than in the normal condition. However, the maximum temperature predominantly falls around 1,600°C. This does not imply an actual reduction in molten pool temperature but is likely due to changes in surface emissivity caused by oxidation or carbide layer formation. Ultimately, the feature analysis confirms that the two distinct deposition conditions targeted in the experimental design were successfully achieved.
3 Data Transformation AI Model
3.1 Design of Model Architecture and Loss Function
A 3D CAE, compared to a 2D CAE, takes a three-dimensional sequence as input, incorporating the temporal axis in addition to spatial information. This allows the model to learn temporal continuity and motion patterns simultaneously. By effectively capturing inter-frame correlations, 3D CAEs enable more accurate reconstruction while also reducing noise, making them particularly useful in super-resolution applications for video and image sequences [14].
Since the objective is to transform different types of sensor data—specifically, generating temperature data from molten pool vision images—a 3D CAE was selected as the data transformation model due to its advantages described above. The model architecture was independently designed, as illustrated in Fig. 6. All input and output data were resized to a fixed width and height of 48 pixels, and the input video sequence length was set to 8 frames (0.3 seconds) to balance computational efficiency and temporal context. The upsampling and downsampling modules were designed with two stages, each consisting of a 3 × 3 convolution and batch normalization, repeated twice, followed by a Scaled Exponential Linear Unit (SeLU) activation function.
Based on this, the model was designed such that the encoder, responsible for downsampling the input video data, reduces the width, height, and depth (frames) by half at each stage, while doubling the number of channels. Conversely, the decoder, responsible for upsampling, doubles the width, height, and depth, while halving the number of channels at each stage. Additional details on the model training process are provided in Table 3.
Considering the importance of monitoring the molten pool center in the DED process and the dataset characteristics shown in Fig. 4, a 2D Gaussian Weighted Squared Error (GWSE) loss function was designed and applied. To evaluate the effectiveness of this loss function, two separate models, Model A and Model B, were trained under the same conditions (model architecture, dataset, and hyperparameters), except for the loss function.
Fig. 7 illustrates the difference between the conventional Squared Error (SE) loss and the proposed 2D GWSE loss. In Fig. 7(a), the difference between the scaled temperature map ground truth and the model prediction is shown. When applying conventional SE loss, the calculated error is represented in Fig. 7(c). However, by multiplying a scaled 2D Gaussian weight map, as shown in Fig. 7(b), at the pixel level, the resulting weighted error is shown in Fig. 7(d). This approach amplifies the error in the central region of the image, which corresponds to the molten pool.
When training Model B, the GWSE loss was applied to facilitate learning features in the image center (molten pool region), making feature extraction and transformation more effective in this critical area compared to the outer regions.
3.2 Training and Evaluation of the Model
The dataset for training the data transformation model was constructed as follows. To augment the data, all 5,854 vision and temperature images were repeated three times and concatenated sequentially, forming a continuous sequence of 17,562 images. These were then divided into 8-frame segments, resulting in a total of 2,195 vision and temperature video datasets. The dataset was then randomly split, as shown in Table 4, and the same dataset was used to train Model A and Model B under identical conditions.
During training, the Mean Squared Error (MSE) loss for both the training and validation sets was recorded at each epoch, and the MSE loss curves for Model A and Model B are displayed in Fig. 8. While the training MSE values showed little difference between the two models, the validation MSE values for Model B were slightly improved compared to Model A, indicating that the GWSE loss contributed to better generalization performance.
After training was completed for both models, the test dataset videos were input into the models to predict 2D temperature maps. A comparison of the maximum temperature in each frame was then conducted. As shown in Fig. 9, Model B, which was trained using the GWSE loss, demonstrated an average absolute error reduction of 37.6°C in predicting the maximum temperature of the molten pool compared to Model A. Notably, when energy density was excessive, resulting in abnormal molten pool formation and temperature values around 1,600°C, Model B exhibited significantly improved temperature prediction performance compared to Model A, effectively capturing the molten pool’s temperature variations more accurately.
Additionally, to verify whether the models successfully learned to extract appropriate features from the vision images based on changes in energy density, the latent space representations of Model A and Model B were analyzed. The normal and excessive energy density video data were input into each model’s latent space, and the extracted feature distributions were visualized using Principal Component Analysis (PCA), as shown in Fig. 10.
From the results, it was observed that Model B’s latent space exhibited qualitatively clearer separability compared to Model A’s latent space, indicating that GWSE loss contributed to a more distinct feature representation of the molten pool features.
Additionally, to quantitatively assess the statistical significance of the data distribution differences between normal and excessive energy density conditions, the Jensen-Shannon Divergence (JSD) was computed. JSD is a metric that measures the similarity between two probability distributions, where values closer to 0 indicate high similarity and values closer to 1 indicate greater divergence [15]. The results confirmed that Model B’s latent space exhibited a higher JSD value compared to Model A, demonstrating that Model B was quantitatively more effective in distinguishing between normal and excessive energy density conditions.
Additionally, to evaluate the feasibility of real-time transformation of molten pool images during the DED process, the inference time of the model was measured using the test dataset. The average inference time for a single 8-frame video sequence data was 15 milliseconds, using an NVIDIA GeForce RTX 4060 GPU. Given that acquiring one video sequence during the actual process takes 296 milliseconds (under 27 Hz of sampling rate), users of the framework integrated with data transformation AI can view the converted temperature map with a latency of less than 0.4 seconds relative to the real process. This result confirms the potential of the proposed model for real-time application in DED process monitoring.
3.3 Application of Data Transformation AI on Digital Twin
The developed data tr AI model was integrated into the DED process digital twin, which was implemented in a previous study [16]. The key functionalities of the previously developed DED digital twin include: Real-time remote monitoring and quality diagnostics of the DED process; Data collection during the manufacturing process; Virtual reconstruction and visualization of unit deposition bead based on process parameters.
Additionally, the AI model was integrated to enable molten pool monitoring using only vision camera data, eliminating the need for direct temperature measurements. As illustrated in Fig. 11, the system was designed to process real-time vision video (8-frame sequences) as input and generate corresponding 2D temperature maps as output. This functionality allows users to monitor temperature distribution in the molten pool without requiring dedicated temperature sensors. Furthermore, a user interface was developed to facilitate immersive process monitoring via VR devices, enabling users to observe and analyze the DED process in real-time within a virtual environment.
4 Conclusion
This study proposes a data transformation AI model that generates 2D temperature maps from molten pool vision images collected during the DED process. Particularly, a 3D CAE was trained with a 2D GWSE loss to enhance the learning of molten pool center features effectively. While conventional studies primarily relied on direct temperature measurement for DED process monitoring, this research introduces a novel approach that predicts temperature data using only vision images. Additionally, through an analysis of the model’s latent space, this study quantitatively validated, using JSD, that the proposed model effectively distinguishes different process conditions (normal and excessive energy density), demonstrating its capability to learn the differences of data according to process variations.
The proposed data transformation AI model was directly integrated into the DED process digital twin system. As a result, operators connected to the digital twin can monitor molten pool temperature without the need for dedicated temperature sensors, leading to several key advantages: Cost reduction in process monitoring by eliminating the need for temperature sensors; Simplification of equipment configuration, reducing hardware dependencies; Minimization of communication data load, enhancing system efficiency.
Furthermore, by enabling real-time temperature visualization in VR/AR environments, the system allows for more intuitive process monitoring and quality control. In the future, the proposed methodology can be extended to other metal additive manufacturing processes, broadening the application of sensor fusion and data transformation technologies in advanced manufacturing systems.
While the proposed data transformation AI demonstrated reliable performance under the specific experimental settings, its generalizability to a broader spectrum of DED process conditions remains a limitation. The proposed model was trained exclusively on data acquired using a single material combination (e.g., SAE 316L powder on AISI 1045 substrate), fixed gas types, and within a limited range of process parameters (laser powers and scanning speeds). Therefore, its performance in conditions involving different alloy types, such as titanium- or nickel-based alloys, may significantly vary due to differences in thermal properties.
Furthermore, the model has not yet been validated under various powder characteristics (e.g., particle size, flowability), or more complex multi-track and multi-layer depositions where thermal accumulation and particle interaction occur. These factors may influence the optical appearance of and temperature response of the molten pool, potentially degrading the model performance.
Therefore, to address the limitations above, future work should involve constructing the dataset with a wider range of material types, process parameters, and deposition strategies (e.g., thin wall). Additionally, domain adaptation techniques could be explored to enhance cross-condition adaptability in real-world applications.
PDF Links
PubReader
ePub Link
XML Download
Full text via DOI
Download Citation 
CrossRef TDM
