ArtEngine: Setting Up Materials in Unity

Introduction

My name is Victor Kam, and I’m a Technical Artist primarily focused on modeling, sculpting, and texturing. Over the years I’ve worked at studios such as EA, United Front Games, and Blackbird Interactive. I would consider myself a generalist and have done everything from environments to vehicles, weapons, and characters. I also do a lot of scripting on the side and am always looking for the next innovative toolset.

I’ve recently joined Unity Technologies on the Artomatix team. My primary role is to help artists get familiar with the software, such as ArtEngine, and work with clients to get the most out of their material production and speed up creation processes.

An Intro to ArtEngine

ArtEngine is a node-based standalone app that works with any engine, focusing on the creation of materials. It utilizes something called the Example-based Workflow meaning the user inputs data and the software can manipulate it to fit their needs. This can be anything from single images, scan data, to purchased assets. This is the foundation of ArtEngine and requires a slight shift of mindset if you are used to building materials from scratch. We take this approach to allow material creation to be more accessible so that there's no need for you to have a ton of training or expertise to get started. The software also automates the time-consuming and repetitive tasks creators don’t want to do, helping speed up material creation tenfold and allowing creators to spend time on more value-add work.

How Did This Story Begin?

The Artomatix story, within Unity, began with Dr. Eric Risser who founded the company after finishing his Ph.D. at the intersection of AI and Computer Graphics. He saw a growing demand for content in the entertainment industry and set out to solve the problem using example-based synthesis algorithms at the cutting edge of research. Last year, Artomatix joined the Unity family to bring more artist-friendly workflows to the community. Today, Eric continues to lead research and product vision in the Artomatix team and is continuing to develop new algorithms, leading some new exciting Assisted Creation workflows at Unity.

What Can the Tool Do?

If you are familiar with other node-based image processing software, you can easily get into ArtEngine. Although ArtEngine uses an Example-based Workflow, you can still build material from scratch as there are several procedural features included. However, the real power comes in when you give the software more data to initially work with.  

There are the standard tools you would expect from any image manipulation software like Levels, Blend, Blur that take care of the common manual operations you would do.

What makes ArtEngine stand out is how the tool leverages AI to automate several processes, the most popular being Seam Removal, Mutation, JPEG Artifact Removal, and Up-Res.

What Are Its Features?

What I like most about ArtEngine is the ability to manipulate full material sets as a single node inside of the graph, as opposed to always working with individual channels. Although node graphs can get complex over time, the focus is to try and simplify the process and make material creation as streamlined as possible.

At any point, you can use the Compose Material node and start working as a full material set from then on.

Or you can build a Material Collection from your input data to keep the graph nice and compact.

The most impressive feature is its AI-based Seam Removal process. Literally with a click of a button you get seamless maps across all channels which alleviates the need to manually clone stamp-out seams.

Another fantastic feature that can give you lots of mileage out of existing materials is Mutation. This leverages AI and some input parameters to seamlessly generate variations.

Creating Materials

There are several ways to generate materials inside of ArtEngine. You can go as simple as using a single image to process a full material. This can be anything off your phone or from image searches online. This requires some manual tweaking to get the maps right but is easily the quickest way to get material finished.

The next process requires a bit of a rig setup, where we can capture a surface with varying lighting angles. This gets processed through our Multi Angle to Texture node to generate Albedo and Normal. What I like about this node is that it has a lot of extra features like Auto Sort and Radial Compensation.

For the most accurate material, you can do the typical 3D reconstruction and bake maps onto a plane. Inside of ArtEngine, there are Normal and AO generation nodes, so you are really only required to have the Color and Height information baked out. Sometimes in this case I would just do a canvas capture from ZBrush and push this right into ArtEngine to process the rest. This could be anything from removing seams, color adjustments, and creating other channels. This method requires the most work before getting into ArtEngine but will yield the highest quality result.

What Hardware Is Needed?

To run ArtEngine a Windows PC is required along with an NVIDIA card. The AI processing relies strictly on the GPU so the more VRAM the better.

An Approach to Lighting

When taking photos for material creation it’s always best to take images under an overcast sky without any direct shadows. With that said we do have a set of delighting tools such as Albedo Generation, Hard Shadow Removal, and Gradient Removal. These are all here to correct your images if they have undesired lighting captured.

Data Import

To get started with ArtEngine, it’s as simple as dragging and dropping your data set into the graph area. You can begin working right away without much setup time. In this example, I’m dragging and dropping my data set and immediately using the Compose Material node to start manipulating the full set as a single node.

The interesting thing about ArtEngine is the capability to work on multiple materials at once. So you don’t need a separate project per material. This example shows two materials and a decal sheet all in a single graph.

Removal of Unnecessary Elements

There are a couple of ways to remove unwanted elements. We have the classic Content Aware Fill which most users are familiar with. There is also the ability to generate new seeds if you are not happy with the first result.

You can also use something we call the Ignore Mask with the Seam Removal, which will essentially reassemble the seamless version but without the painted areas. This is great if your initial data was captured suboptimally, like missed projections. You can use this method to essentially fill in spots.

Using the inverse of this, you can get creative and start extracting parts into their own tileable versions. I like doing this and then re-blending them back via Height into another material.

Seams Removal

Instead of taking a procedural approach to seam removal, we will analyze the input and use AI to rebuild the seam area with sections that make sense. The user has control over the border of the seam, which indicates where you want the operation to occur. That’s all there is to it really, just execute and the process will run.

How Easy Is This Task with ArtEngine?

For organic or more stochastic materials it’s as simple as a click and your results are complete.

When it comes to more structured surfaces like brick and cobble, we need to help the AI along by generating a structure's guide, which is a black and white image to help the Seam Removal do its job while maintaining structure. You can paint this externally or inside of ArtEngine to guide the AI.

How Does the Tool Work?

The seam removal is a combination of ML and data science. In a nutshell, there is an analysis of the highlighted border area where the image will get split up into chunks. It then reassembles itself in a way where there are minimal differences, thus removing the seam.

Mutation

Mutation is very similar to splattering a material around to create something new based on tweakable parameters. Because this is AI-based the transitions are believable and seamless. Below is an example of mutation with different parameters on a debug texture for more visibility.

An Example of Mutation

Settings available for mutation are jitter amounts, random scale and rotations, seam border widths, and blemish removal. Utilizing all these parameters in conjunction with one another can give you infinite variation and when used along with a color match you can get even more variety.

Another great use of the Mutation mode is to upscale a material based on real-world dimensions. For example, you have a scan from an asset library that is 1m x 1m, we can mutate to a specified target dimension. What this will do is fill the space up to the desired resolution while respecting the input and output dimensions. This is great for achieving perfect texel density. Afterward, you can always use the Mutation Revision to spot areas that are too repetitive.

Tools for Colors

There are a few tools to manipulate colors inside of ArtEngine. First is the usual Color to Mask node, however, we automate the color picking process to generate a series of masks based on a threshold value.

The Color Transfer node is useful for essentially changing the colors of an existing image without the use of masks. You can even match the color of one image to the palette of another if you like.

Color Matching

Color Matching is very similar to what you would find in Photoshop. Except that, because it’s nodal-based, this is a great way to experiment with different looks right inside the graph. For example, if our material needs to fit within a certain biome, we can do this by trying out different samples to find a result that works for us.

Other Cool Features It Includes

ArtEngine offers a host of nodes dedicated to fabrics such as the Pattern Unwarp which will utilize AI to unwarp a patterned image.

Remastering old content is a common theme now that the new consoles have arrived.  Built into ArtEngine is AI-based uprezing for the remastering of old data.

Still in beta is Style Transfer, which allows users to combine images into something new. So if your project needs backdrop art or you’re just looking for something non-realistic, this is for you.

When working in full production we also understand that there may be bulk revisions of large amounts of data at some point. This is why we have Batch Processing where you’re able to feed it a folder of either individual maps, or full materials.

As an artist, I am super excited to see AI being implemented into the material creation process.  We see AI being used in everyday gadgets from phones to social media, it’s only a matter of time that AI is an essential part of content creation. Being able to automate these once mundane tasks frees up more time to be creative and paves the way for more creators in the world.

We are starting to see a growth in the use of content libraries, the only downside to using premade materials is the inability to manipulate them as is.  With the power of AI tools, we can now take existing libraries and adjust them to the needs of a given project, rather than build from scratch.

This lifts the burden on the once specialized domain of material creation and opens up many doors to allow individuals to simply create and build immersive worlds.

ArtEngine is only scratching the surface of what’s capable in terms of AI-assisted content creation. The nodes I’ve discussed are always being improved and newer iterations are developing as we speak.  I hope all of you are as excited as I am!

Victor Kam, Techincal Artist

Interview conducted by Arti Sergeev