Generating PBR Materials From Photogrammetry Scans Using ArtEngine

Introduction

My name is Federico Capriuoli. Based in Italy and a civil engineer by training, I am passionate about scanning. I work for Acas3D, an organization focused on surveying historical sites and structures in 3D using laser scanning, photogrammetry, and drones for the purpose of preservation. I have a particular focus on capturing underground sites, which can prove challenging, given space and lighting constraints in these conditions!

Though my work is a niche within architecture, I leverage very similar workflows to those used in gaming, and thus my toolset looks quite similar. Indeed, I rely heavily on tools such as Lightroom, Reality Capture, Meshlab, ArtEngine, and Marmoset Toolbag.

Recently, I began using ArtEngine, a material authoring tool that leverages an example-based workflow and AI to drastically speed up the material creation process. I’d previously used the software to clean up scans for past preservation efforts, but I thought with this project, it’d be interesting to see if I could extract high-quality PBR materials from some of my old photogrammetry scans using ArtEngine. 

For the subject matter, I chose a scan I did of a small medieval church in Pisa. The church, called the Cappella di Sant'Agata, was about to be restored, and so my task at the time was to create a high-resolution 3D replica to be used as a reference to create technical drawings.

Here were the end results of that scanning project:

My Scanning Workflow at a Glance

In terms of my workflow for this initiative, I leveraged laser scanning and terrestrial photogrammetry methods. I shot on a cloudy day to maximize diffuse light and minimize shadows and used a color checker to calibrate colors.

To generate the laser scans, I used a FARO® Focus Laser Scanner. The scans needed to be done at a very high resolution (3mm \ 10m). Each scan produced a point cloud of about 90 to 95 million points, which meant that the 19 scans totaled almost 2 billion points!

For the detailed photogrammetric survey, I used my Nikon D850 and 35mm and 50mm lenses. I shot in RAW, color corrected the photos in Lightroom, then brought them into Reality Capture for processing. I created a high poly model (145 million polygons) and five UV maps at 16K. I then extracted plans, sections, and orthophotos from the model to generate the technical drawings. Finally, to optimize for online viewing, I reduced the model to 100,000 polygons and five 4K textures.

Material Generation with ArtEngine

As mentioned, I recently started using ArtEngine to do surface scanning work. Indeed, another one of my projects involves mapping all the medieval walls of my city, Pisa, and creating a library of seamless, 8K textures—one material for each wall. 

For this project, I wanted to illustrate something of the opposite: Is it possible to extract several materials from a single scan? Spoiler alert: with ArtEngine, yes!

First, I generated several baking planes (Blender for baking plane creation, xNormal for texture reprojection) from areas of the scan from which I wanted PBR materials. The planes varied in dimensions.

For the floor materials, I imported only the normal and diffuse maps into ArtEngine.

Here is where the ArtEngine magic begins. For each material, I began by importing only the Diffuse and Normal Maps into ArtEngine. From the Normal Map, I generated the height and ambient occlusion (AO) maps. From the Height and Diffuse Maps, I created the Albedo Map. Between these steps, I employed a fair amount of adjustment nodes, such as Gradient Removal and Normal Intensity, to fine-tune the results.

After creating all the maps, I used ArtEngine’s Seam Removal node to remove seams. I then created a second version of this material using the Mutation and Mask Paint nodes to generate a larger material that tiled at scale without any visible artifacts.

Here are my end results for the brick and wooden door materials:

And there you have it: several full PBR materials from a single scan!

I hope you learned something new about scanning workflows and feel inspired to create and give ArtEngine a try. There’s a free 30-day trial as well as a GDC promotion to get the tool for $19/mo for the year (vs. the regular price of $95/mo) if you purchase before May 17.

If you’d like to learn more about my work, I invite you to visit my website or Sketchfab portfolio, connect with me on LinkedIn, or shoot me an email at f.capriuoli@gmail.com.

Federico Capriuoli, 3D Scanning Expert