LogInSouth Korea is bringing home cultural assets stored in foreign museums with an innovative project to digitize them for use in real-time 3D environments with a pipeline built upon Unreal Engine and RealityCapture.
The three main periods when cultural heritage and relics were taken away are the Korean War, the end of the Joseon Dynasty, and the Japanese colonial era. In the early '90, the government began a campaign to retrieve cultural heritage, and according to the Technology Research Institute for Culture & Heritage, around 230,000 South Korean cultural assets were either legally or controversially exported.
Among the efforts to repatriate the lost cultural assets is the idea of digital sharing, a South Korean government-led initiative that leverages digital technology and equipment to replace the complicated and difficult physical retrieval process. With digital sharing, Korean artwork stored in foreign museums can be exhibited in its home country as a precise digital replica of the original artifact. Interactive 3D content is developed based on the digital replica, providing a realistic and engaging experience for viewers.
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
The first project under this initiative is the Digital Homecoming Project for the Korean Cultural Assets at the Cleveland Museum of Art. Kim Jikio, CEO of the Technology Research Institute for Culture & Heritage and the director of this project has explained that it has been in development for three years and will be exhibited simultaneously in Korea and the US from March:
"We're capturing digital data from 3D scans of the Korean cultural assets held by the Cleveland Museum of Art and producing realistic digital content based upon that data. The project has been ongoing for three years.
In the first year of the project back in 2022, we digitally archived 13 major Korean artifacts at the CMA. In 2023, we selected the 19th-century folding screen Seven Jeweled Mountain from those artifacts based on its suitability for large-scale exhibition display and began to produce immersive, realistic content based on the artwork. Now the project is in its third year, the results of our hard work will be shown in an exhibition that will take place in both Korea and the US."
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
The reason TRIC chose to employ Unreal Engine in the production pipeline for this project is because producing real-time content while still preserving the quality with minimal efforts was crucial for achieving the fidelity of the original artwork in the digital version.
As the mesh and texture get more precise, the data inevitably gets heavier, so the company put a lot of resources into optimizing the data since the real-time interactive content is primarily viewed in VR or AR, as well as in physical installations.
"Using Lumen and Nanite in Unreal Engine 5, the details of the digitized artifact can be depicted at a high visual quality. These tools also reduce the burden of the long optimization process that used to be necessary for real-time production, and enable a quicker delivery of the desired quality with a smaller team.
In addition, Unreal Engine has remarkably shortened the pipelines for all TRIC projects, including those for immersive multiplanar imaging that require complicated calculation and simulation from the base videos, anamorphic imaging, and content development for new output formats."
The project process was divided into two parts: scanning and digitizing the 13 artifacts and then producing the realistic Seven Jeweled Mountain content. For the scanning stage, TRIC mainly used RealityCapture and the Unreal Engine-based pipeline was built to create the realistic content.
"In the first year of the project, we captured the ultra-precise digital data for the 13 Korean artifacts kept in the CMA, including the Seven Jeweled Mountain. Precise 3D scanning ordinarily requires the use of professional equipment such as LiDAR or structured light scanners. However, the artifacts were overseas in this case, making it difficult to bring in the equipment or create an optimal scanning environment.
As an alternative, we leveraged a photogrammetry method to capture the data using RealityCapture, which we frequently rely on to produce 3D datasets for spaces and objects in the buildings housing the artworks. We took thousands of pictures for each artifact and created high-fidelity assets, both in mesh and texture, using RealityCapture. On a side note, RealityCapture was also crucial for completing the Charlottenburg Palace Ceramic Room project, despite the fact that the German border was closed due to COVID-19, in collaboration with an architectural photographer in Berlin."
The comparison of real assets to digital ones:
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
In the production of the realistic content, TRIC aimed to translate the original Seven Jeweled Mountain, which is painted on a wide folding screen, to three immersive digital screens while preserving the traditional art style and adding the sense of three-dimensional space:
"To achieve this, we first built a vast gigapixel data network connecting the thousands of images that made up each side of the folding screen using a camera rotator, then created separate layers for each object, color gamut, and level.
We used Unreal Engine to place all the elements in a three-dimensional space and filmed them to recreate the traditional 2D artwork of the Seven Jeweled Mountain as spatial, dynamic 3D content."
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
As explained by Kim Jikio, one of the most important parts of creating the three-dimensional effect is expressing the brushstroke of the character outline:
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
"We created a shader using Unreal Engine's Material Editor. The paper-like, high-quality texture was downloaded from Megascans, modified, and applied to the character's post-processing material.
The fog was added using Unreal Engine's Niagara Fluids plug-in to bring a dynamic feel to the artwork. Niagara Fluid enabled us to produce a simulation that reacts to other objects in real time or controls the desired timing via Sequencer.
Niagara Fluids is lighter than the previous method of using OpenVDB and any required edits could be made instantly. To display these well-made characters and effects on a triple-planar screen, the animation was effectively processed using a camera rig that binds three cameras into one. Each video was rendered simultaneously using Movie Render Queue which enables rendering from three cameras placed in Sequencer at the same time."
Image Credits: Digital Homecoming Project for the Korean Cultural Assets
Since content that uses traditional Korean art resources like the Seven Jeweled Mountain screen must be consistent in its aesthetics, a production tool that enables precise control in all stages is essential. Unreal Engine provided an environment where the developers could control the composition assets in detailed precision via Niagara, Sequencer, and other systems while preserving both visual quality and performance.
For TRIC, real-time rendering in Unreal Engine was the foundational technology of this project, but Unreal Engine Marketplace and Megascans also offered an array of resources from their libraries for various natural phenomena as well as helped to save time by providing the right materials, high-quality shaders, and textures for the traditional style.
Read the full interview and learn about the future goals of TRIC here. Also, join our 80 Level Talent platform and our Telegram channel and follow us on Instagram, Twitter, and LinkedIn, where we share breakdowns, the latest news, awesome artworks, and more.