Setting Up a Stylized Character Shader With Unreal Engine 5

Introduction

Hey there! I'm Michal Patton, and I'm excited to share my work with 80 Level! I always knew I wanted to work in Entertainment Arts. However, what really drew me to 3D art was seeing games like Okami and films like Into the Spiderverse, which blended everything great about 2D illustrations into 3D worlds.

My studies at the Think Tank Training Centre helped me bring my skills far beyond what I thought I could achieve. I loved every minute of my studies. As a 21-year-old, I've still got a lot I'm interested in. My Cadet project was a great opportunity to learn more about my inspirations.

To me, 2D will always have an advantage over 3D because of the sheer amount of artistic freedom. But with that gap gradually narrowing, I wanted to explore how to give a 3D character some of the beautiful qualities of an illustration.

The goal of my Cadet Shader was to make the lighting of a scene look painted on in real-time and bring out some hand-painted textures. This meant working with Unreal Engine's post-process filters, breaking down the scene's existing lighting setup, and recombining it. The character I made was based on a speedpaint by the incredible Adrian Bush, and it all wouldn't have been possible without the advice of my friends and mentors, and breakdowns from other incredible artists.

Here is the original painting by Adrian Bush:

Modeling

While the main focus was on the shader, I also did learn a lot about modeling here. Initially, I believed that creating a stylized character wouldn't involve using exact references. However, that wasn't the case because there were plenty of real-life objects or faces that looked bizarre enough to be "stylized". For example, the folds on a Swedish military motorcycle jacket are huge, and the overall design is fantastic because it matches the forms I was aiming at. Additionally, certain elements of modern Air Force helmets helped in achieving the desired aesthetic. Moreover, I could always stumble upon a picture of a person with unique features that fit my stylized reference. It's fascinating how everyone's appearance can be somewhat unusual, and that adds to the charm.

I used Ref Switcher in ZBrush to match my references, sculpted over Marvelous Designer simulations, and always tried to make this character anatomically correct. However, I think that the advantage of stylized works is getting to break a few rules. I tend to go over the top with my designs, allowing me to subsequently refine them to a "happy medium" using ZBrush's layers.

Texturing

To get the most "painterly" feel, I wanted to paint my base color like an artist. So I did a lot of hand painting, especially in the face texture. I painted in some light and shadow values to the base color and went crazy adding grudges to the character's clothes. Thiago Vanuchi taught me a valuable lesson with texturing, which is that sometimes hand painting or layers without fancy blend modes are much more direct and artistic than trying to make procedural anchor point set-ups. Don't get me wrong, I used both, but it felt nice to work a little more destructively here, like a real painter. Proceduralism would come later, in the shader.

While the bulk of my face texturing was a single paint layer, I added dozens of grunges to add color to the clothes and assets. I painted in lighting to help distinguish smaller details and object separations. But I also recalled Fred Arsenault teaching me to adjust overall gradients and colors to help accentuate the focal point which was the character's face.

To top it off, I used a lot of sharpening filters. They could pixelate and weirdly crunch colors if turned up high, but in my case, they were the perfect solution to bring out the color variations and make the brushstrokes stand out.

Here you can see the picture of the base color:

Planning the Shader 

There are a lot of ways of setting up a filter like this. However, I believe every method benefits from analyzing art styles. I spent some time organizing 2D art references to identify how an artist rendered in their style.

Adrian Bush utilizes a lot of textured brushes when painting transitions between light and shadow. He generally seems to focus on dark and mid-tones, rather than highlights. However, I think that where there are highlights, they are often very pronounced. It was little notes like this that helped me identify what I wanted the shader to do.

Most of the research I did on the technical side was on a paper written by Delphine Decuyper. Her work was my Bible throughout this project. Ben Cloward also is a master at breaking down complex shader topics, his videos made this project so much easier. 

The Shader Creation

Below are the pictures of my first and second versions of the shader. Everything is a lot more organized in my later material blend version, so have a look at it:

I started by deconstructing the scene's lighting into value masks. The Unreal Engine's scene texture node can only grab a few passes of the scene buffer information. I can extract the lighting alone by dividing the scene (post-process input 0) by the base color. The next trick is changing these values to mask out a specific area of the lighting. My mentor, Nicolas Nino, recommended taking screenshots and manipulating them in Substance 3D Designer. This was a great help to me because the software was a little more immediately visual, and the more familiar nodes made for a great testing ground.

My initial approach to manipulating the lighting involved using linear interpolation as a "levels" adjustment to remap the values. Although it required a considerable amount of trial and error to find the appropriate deals, it ultimately achieved the desired outcome effectively. Additionally, I created a material function to streamline an existing three-point level function. At that time, everything worked well.

However, I later discovered the elegance of Curve remapping nodes. In more recent iterations of this shader, I have incorporated these nodes to remap the lighting. This approach offered a more visual and intuitive way to work, giving me greater artistic control over creating sharp and soft shadow transitions.

By remapping the lighting, I could separate out areas where I wanted to designate highlights, midtones, and shadows. Having multiple lighting masks helped me create the distinct shadow shapes I wanted to recreate from Adrian Bush's work. 

Here is a tricky part which is figuring out how to add brushstroke textures to these lighting masks. I could multiply or add a tileable onto these masks, but it wouldn't adhere to the models like I wanted. 

Around this time, I ran into Billy Deloe at GDC who showed me how he was using the world position offset node in his work. This was huge to me because it meant I could apply these textures completely procedurally, and they'd look great no matter where the camera was! Using a similar setup, I projected a brushstroke tileable to accurately wrap around the assets.

Now that I had this projected tileable, I wanted to combine it with my lighting masks to make the lighting passes look painted on. Before adding it or subtracting it from my lighting pass masks, I manipulated the values with LERP nodes. I darkened, lightened, and increased the contrast on the tileable for each individual mask, so it would look unique and show off the texture the right amount for the given lighting pass.

It's worth noting that this technique only really works for objects that aren't moving. The projection is visible if the actual character moves. So to work around this issue, I found that projecting the tileable into the character material's specular provides the texture in a similar way. I used specular because it was easy to call upon with the scene texture input node, and in general, the specular makes very little difference when viewed through my filter.

Going back to comparing Unreal Engine's nodes to Substance 3D Designer's, I often use LERP or IF nodes to combine lighting passes similar to the Designer's blend node. With the textured lighting mask as the alpha, I can combine all my passes back together in a string of those "blend" nodes. I found that IF nodes could be very interesting to utilize since increasing the A=B tolerance allows room to plug in a third lighting pass. I didn't use these all the time because the IF node did sharpen the masks to binary levels where LERP blending allowed more room or gradients.

The masked lighting passes were originally the scene's diffuse color with some additional parameters to tint them to appear more lit or in shadow. Using the diffuse made it much easier to see the painted texture Maps, but couldn't capture things like multi-colored lighting or subsurface scattering.

The workaround I have recently developed involves using the entire scene instead of solely relying on the scene's diffuse color. This approach allows me to apply a consistent tint and resolves the issues that arose from using only the diffuse component.

Once everything was set up, I had a few iterations testing different tint colors, lighting levels, and tileable scaling.

A Note Of Reflections

Since this shader only manipulates base color, reflections can be hard to capture. If this were a painting, I could use a little brush to apply tiny dots in the highlights. Of course, real-time is a little trickier. I wanted some little reflections visible on the character's eyes and goggles that moved at different viewing angles. I found that it could be done using a Reflection Vector node.

By plugging in Reflection Vector to the UVs of a speckled or spotted tileable, I got what looked a bit like dots of highlight. The node wraps the texture around a surface, while also warping around based on the camera's position like a real reflection. How cool is that?

In the eye material, I simply added this on top of my existing base color. Meanwhile, I incorporated the Reflection Vector node tileable to both the base color and the Opacity Mask for the goggle glass material. Most of the glass was masked out, except for some dirt and scratches, and a thin fresnel to separate the edge of the glass from the rest of the model at glancing angles. This way, the geometry below would be clear and play along with the rest of the filter.

There are so many fun tricks available in Unreal Engine that allow for the manipulation of results. These tricks not only helped me get the reflections I wanted but also gave me some artistic control over them, which can sometimes be difficult to achieve with lighting alone!

Organizing the Material Layer

At first, one of the issues with my shader was that it was difficult to deconstruct the layers and manipulate individual lighting passes. I wanted to put my shader into a Material Layer blend system so that each lighting pass could be its own layer, and it would be so much easier to organize multiple passes and control them individually.

Thanks to some help from my friend Hanlouw Pretorius, I figured out how to set it all up. I copied the lighting mask nodes to Material Layer Blends and copied the nodes to adjust scene tint and values to the Material Layer. This setup was a lot simpler since the Material Layer Blend system took care of masking the lighting passes and stacking them on top of each other. 

Conclusion

This project was incredibly enjoyable to work on, and there are still numerous techniques I am eager to explore. My goal is to contribute to the development of workflows that enable artists to concentrate on their preferred aspects of the creative process while providing them with the freedom to push the boundaries of 2D/3D mediums.

However, what I'd like most is if this project could inspire other artists! I learned so much from this community's amazing work, after all. Thanks to 80 Level for giving me this opportunity to share my work here!

Michal Patton, 3D Character Artist