How to Sculpt & Render Desert-Themed Plants in UE with Lumen & Nanite

Introduction

Hello! My name is Aria Baskakova, and I'm a Berlin-based 3D Games Environment/Props/Materials Artist with a great love for vegetation. My passion for game art takes its roots in playing World of Warcraft: the magic of 3D representation of the whole imaginary world captured me all at once. I earned a Bachelor's degree in Graphic Design and have some painting experience, so I started my career as a 2D Artist for social games while taking any opportunity to learn 3D techniques.

My options for those years were only short-term courses and tutorials, nothing long-term and costly. I learned the basics of props creation in the Draft-Punk course by Igor Dyatlov, a founder of Retry School. A firm base for Substance 3D Painter and Substance 3D Designer was achieved in Anton Ageev's courses. Of course, Daniel Thiger and Javier Perez's tutorials were substantial for my learning curve.

That path and the skills acquired brought me to Dragonfly, an outsourcing studio focused on a stylized games approach. I was lucky to work as a 3D Texturing Artist there for two years for the Wayfinder by Airship Syndicate. Wayfinder's art just astonished me, especially the environment by Romain Durand. After some time, I realized that I would like to revive my other 3D skills and practice them holistically, from construction to engine implementation.

After this conclusion, I decided to take several courses in CGMA, the most valuable of which for me was Vegetation and Plants for Games with Jared Sobotta and Shading and Material Creation in Unreal Engine with Hugh Trombley/Chew. Jared Sobotta's course was a pure gem and I just felt like a druid with a new ability to create any plant in the world. And shaders in the engine of one's choice give an artist the final and often crucial key to the look development.

About the Euphrates Poplar and Tamarisk Project

The Euphrates Poplar and Tamarisk Project wasn't initially a foliage-only scene, nor was it created in Unreal 5.2 from the beginning. How did it come? Well, I decided to summarize the new things I've learned with Unreal Engine 4.27 real-time PBR game-optimized environment scene after an amazing concept. There weren't any live trees in the concept art, but I wanted to add them because I imagined some bright trees would compositionally complement my scene in real time. I needed the plant to be logically placeable in some kind of Asian desert according to the concept's style. The Euphrates Poplar, in its autumnal state and with its gnarly trunk, was ideal. However, I noticed some interesting red-toned bushes in the reference photos with the Euphrates Poplar trees, which turned out to be Tamarisk. With its feathers-like fluffy branches, Tamarisk provides a challenge in setting it up. Last but not least, the Cleistogenes was chosen as a grass plant suitable for deserted places.

After completing my scene in Unreal Engine 4.27, I decided to render the foliage separately in Unreal Engine 5.2, knowing that the difference in rendering would be significant. Therefore, I had to set the foliage up in two engine versions, without Lumen/Nanite and with them. Of course, things look much more oomph in Unreal Engine 5.2. Except for the final setup in the engine, the creation pipelines are similar, but they have a couple of details to consider.

Sculpting the Euphrates Poplar Leaves' Atlas in ZBrush

The cool fact about the Euphrates Poplar trees is that they have different shapes of leaves at once on one tree. I've never imagined that's even possible! My foliage creation pipeline usually starts from ZBrush for leaves atlas sculpting. And here goes the first thing to consider in terms of optimization: one can use the atlas as the high poly for the further backing of the plane and cutouts, or use it as high poly for backing on low poly meshes of the leaves to get the maximum from Nanite technology. I do it the first way because my computer setup is still not high-end.

I start the sculpting with the shape primitive that suits the future leaves shapes more: in this case, it's a low poly circle used in conjunction with Dynamic Subdiv in the Geometry section. Dynamic Subdiv allows using Dynamic Thickness. This technique helps me draft the main shape in error-free mode while having consistent virtual mesh thickness without overlapping. The actual mesh is still one-sided low poly, which means I can easily tweak the shape with the help of just the Move Brush. It's handy if you want to have the perfectly aligned front and back sides of the leaves. Dynamic Subdiv can be applied once you are happy with the results. This will turn your mesh into a regular mesh with lower and higher subdivisions. Once applied, the dynamic thickness can not be recalculated, so I always advise having the duplicates.

As a next step, I use ZSpheres to create the leaves' skeleton. There are some advantages to my taste in this technique. First, it's faster to make one basic ZSpheres skeleton and alternate it between different leaves. Second, I found it more convenient doing it with ZSpheres instead of using the usual Dam Standard (or equal) brush because ZSpheres allow me to have a consistent profile of the leaves' skeletons from two sides simultaneously and provide polygroups when converted to Adaptive Skin. The polygroups of the skeletons are noteworthy not only while sculpting in ZBrush and turning them into masks for better edge readability, but also for the further texturing process when the masks for color and subsurface scattering go to the stage.

When sculpting in ZBrush for further baking on a plane, it's important to check if all the final meshes in ZBrush are aligned to this plane frontally and in profile. When aligning frontally, one needs to think about texture padding, as well as minimizing the space left for opacity. For 2K texture, the padding is usually 8px to the sides and 16px between elements. It's needed to prevent the bleeding of the different texture elements into each other in the engine while viewing the final asset from various distances. The situation with the opacity is simple: the less opacity space you have in the engine, the better. Aligning the sculpts in profile allows us to achieve accordingly aligned normal depth during the following baking stage.

Baking & Texturing the Euphrates Poplar Leaves Atlas in Substance 3D Painter

At first, to prevent memory overdraw while baking, it's convenient to use decimated mesh from ZBrush.

The baking on the plane is perfectly achievable inside Substance 3D Designer. In the ID section of baking, I switched to mesh ID/Polygroup because it will give me the masks for each leaf and its skeleton. AO, Curvature, and Thickness are well baked when the secondary Rays parameter in their palettes is the highest number. It's important to remember to choose ACES tone mapping in the Display Settings; otherwise, it will be tremendously hard to understand how the color and Metallic/Roughness values will look in the engine. ACES tone mapping works closest to Unreal Engine rendering.

While texturing in Substance 3D Painter, I use many anchors. Once a mask is created, I can add an anchor to it in the mask's stack of layers and then re-use this anchor with any other masks. This saves time crucially.

There is a common rule in creating textures for games to have greyscale images packed when exporting. Generally, any greyscale image can be represented by the red, green, blue, or alpha channels inside one file. The final combination depends on the current pipeline: some teams use ORM (AO+Roughness+Metallic for red/green/blue channels accordingly), while others can use MRE (Metallic+Roughness+Emissive), et cetera. The common rule is to use a green channel for Roughness, as it's the channel with the highest detail representation. This technique allows for lowering memory usage when playing games. 

Low-poly Leaves in Maya from a Plane to a Mesh

After the texturing process for leaves is done, it's time to create the optimized meshes for them. I use Maya as a box modeling software, but it can be performed in any other suitable software. I start with a typical cut-from-a-plane process:

• Place a plane with material from our texturing phase (base color + normal map + opacity, no need for Roughness/Metallic here, just fully rough);
• Cut the main shapes of the leaves without thinking about topology at this stage;
• Form a logical but not too dense topology for each leaf separately, emphasizing their shape and aiming to leave as little opacity as possible;
• Center pivot points of each leaf at the start of the petiole and in the 0/0/0 of the world coordinates.

Tips and tricks: When operating move/rotate tools on vertexes, check the Preserve UVs box to maintain the texture borders and tweak the shape simultaneously. 

Tileable Bark Textures for Euphrates Poplar in Substance 3D Designer

Usually, trees have different bark on the trunk and the branches. Therefore, I created two tileable textures for my tree. As far as one is for branches and goes only into the corresponding atlas, the final quantity of the texture sets in the engine project will still be two: bark tileable and the branches atlas.

For the trunk bark, I used two Tile Generators, Anisotropic Noise and Triangle Grid Grayscale, as a base for the main pattern. I am a big fan of the Triangle Grid Grayscale node; it's tremendously useful as a driver for Slope Blurs when you want the forms to be more unpredictably angled. The process of creating this layered texture wasn't easy for me. I had a lot of iterations and back-and-forths before the desirable result was achieved. 

The branches bark nodes graph is far simpler: two Anisotropic Noises and one Crystal 2, many Slope Blurs, and BnW Spots 3, and voila, the base height is here. The rounded scars are made with randomized ringed bell shapes in Tile Generator, and then, the convexes at the bottom of these shapes were achieved with Non-Uniform Blur. I used Height Blend and Blend nodes to mix the basic Height and the round Scars: sometimes, the combination of both works better than one.

Tips and tricks: don't be afraid of using high numbers of normal intensity for your bark textures, but it's important to understand where the maximum depth goes in reality. The main surface of the small branches of my Poplar is quite still, while the rounded scars are deep and rough. In this case, I combine two normal maps, one with an intensity of 3 and the other with an intensity of 30, by the round scars mask with a Normal Blend node. 

Branches Atlas for Euphrates Poplar in SpeedTree

With all the tree textures and leaves low-polies ready, it's time to go to SpeedTree and create a branch atlas. I find SpeedTree to be almost the perfect software for vegetation creation because it provides a comprehensive procedural pipeline with the possibility of profound manual tweaking. In my opinion, the poetry of organic shapes is truly dependable on the artistic touch, so it's crucial for me to have the possibility to adjust things to my liking by hand, and SpeedTree, with its manual mode, gives me this freedom.  

The main rule for the canopy, either atlases or the final branch placement, is to have the "air" in between, being careful not to overpopulate with leaves. Things look good in the engine when there is a space for light and shadow in rendering. Too dense leaf/branch placement will lead to a heavy-looking dark canopy. 

If you use the branches at atlas cutouts, it's possible not to think about polycount optimization at this stage, because these meshes are only needed for the baking maps. But if you'd like to use the final branches as meshes, too, the Segments menu will give you many settings to keep the polycount sane (even with Nanite) while maintaining the desired shape. In my case, I use this atlas for further flat card cuts.

Quite a profound yet simple tutorial about branch atlas creation/textures exporting in SpeedTree can be found at Artstation in the tutorial by Decogon Studios and Cat Yang.

The only thing that doesn't work quite well for me in SpeedTree during the branches atlas creation is a final AO map, which lacks precision to my taste, so I usually re-bake it in Substance 3D Designer from a Height Map.

Tips and tricks: always use the manual mode as the last step of vegetation creation because it's destructive in terms of further node-tweaking. Implementing your artistic touch is recommended only when you're satisfied with what is achieved procedurally. Save copies of your file beforehand.

The Final Tree in SpeedTree

While SpeedTree is awesome, it's useful to set up a test scene in the engine of your choice as soon as possible, because things look different in SpeedTree and, for example, Unreal, due to the crucial difference in rendering. What looks airy enough is that SpeedTree can be easily too dense in Unreal. It's also true for the shape: checking your tree through the stages of its development in the engine from different angles will save you time.

My tree polycount is quite modest, though with Nanite technology in Unreal Engine, it's possible to go with a much more dense topology. The final choice depends on your computer setup and the project's needs. The small branches at the start of the trunk are the planes cut from the branches atlas inside the SpeedTree, while it's possible to place them as real meshes if your polycount budget is higher.

As for the interesting nodes, I used the Peeling Bark node with an opacity mask from Substance 3D Designer for additional chunks of the tree bark. Also, I implemented a technique of Spine-only fake branch placing: if you want your cut card to behave as a branch, you can create branches nodes, turn the Type in the Skin palette to Spine only, and then attach the Mesh node, in which your atlas cuts are assigned in Material palette. The Spine branches won't be visible this way and will work only as the drivers for operating the cut cards. 

One thing to note: I do not recommend using the materials' color variation setting for the final plant construction (but it is okay for the atlases). SpeedTree just duplicates the textures with color tints, in this case, which is not optimized for game engines. The optimal solution would be to achieve the color variation through the shader logic inside an engine.

Before exporting the final tree, it's possible to enable the Wind generator. The wind system in SpeedTree is versatile and has many settings inside the software and as a ready shader/material instance in the engine. In Generator -> Wind, check the Enabled box, and then go to the Wind Wizard. The last one is a short way to add some wind motion with ready-made presets.

Though SpeedTree creates an automatic Lightmap for our foliage, it can be better optimized. To check its state at the end of your modeling work, tap Ctrl+7 or go to View -> Render -> Lightmap. You'll see your vegetation model in greens and blues, with greens for the higher Lightmap density. Typically we want to distribute more Lightmap space for the elements closer to the player's eyes. Also, we want to get rid of too much free space in the top right corner of the calculated Lightmap. The Lightmap weight of each part of the vegetation can be adjusted in Lightmap properties. If you ever notice that some of your branches, for example, look too dark in the engine while the others seem to be ok, it's because of the Lightmap, and it can be corrected easily by adjusting its weights in SpeedTree. 

The final export from SpeedTree is pretty easy because this software has a ready-made exporter for Unity and Unreal Engine. You will get all your textures packed and linked in a special shader with instance material to tweak the parameters later in the engine.                        

Tamarisk Plant Approach

The process behind the Tamarisk branches is a bit different. The leaves of this plant are very thin feather-type, so there was no need for sculpting. I created this small atlas with three leaves in Substance 3D Designer with brand-new Spline nodes and one shape scattered along the final spline profile. 

The resulting mini-atlas were sent to SpeedTree, where I created the bigger atlas for the plain cards that I intended to use in a cross-plane technique. For this technique, one needs to rotate the plane along its center to create a mesh with volume, which uses only one atlas texture set. Therefore, the rotation axis of the object should be straight. I assemble the cross-planed meshes in Maya after cutting them out, like with the Poplar leaves atlas. As you can see, I used only two cross-planes for each branch, and they are very low-poly. After combining planes into one mesh, you can bend them with Lattice or Bend tools to get rid of unnatural straightness. 

One will probably need to correct the direction of the polygons' vertex normals because once we cut the planes out of the flat surface, it will behave as a flat surface in the engines if we don't fix it. First, we need to see the vertex normals. In Maya, it's Display -> Polygons -> Face Normals/Vertex Normals (with a mesh selected). You will see the arrows going from the vertexes of the polygons, and they are the Vertex Normals. They show the light mathematics in which direction the light vector goes. I needed the engine to calculate my branches as something roundish fluffy, so I created a simple sphere and transferred its vertex normals attributes to the branches meshes.

In the following screenshot, the yellow arrows are the vertex normals corrected with Transfer Attributes from the stretched sphere, and the green ones are not. While the result is not always visually obvious in Maya or SpeedTree, it will be important in the engine. This technique is often used with vegetation that should resemble something round, and, of course, you can create different shaped objects used for transferring the vertex normals.

I exported the final branch meshes into SpeedTree and constructed three bush types. The flat-cut cards are used here, along with cross-planed meshes, to keep the polycount adequate for my purposes. Of course, there is a lot of manual adjusting after the procedural construction of each plant: one can tweak each element in the scene in Node mode and Freehand mode, but a thing to remember is to do it only when you are happy with the procedural stage because it will be not that easy to replay things procedurally in some cases after manual editing.

Unreal Engine 5.2 Plants Rendering

I usually export the vegetation from SpeedTree to Unreal Engine in ST9 format. The wind system needs to be enabled before the exporting. Once a project for your plant scene in Unreal Engine is created, Lumen can be turned on. In my case with the Tamarisk, I needed to go to Edit -> Project Settings -> Engine -> Rendering -> Lumen and choose Detail Tracing. It will help in calculating light with all these intricate Tamarisk's fluffy branches.

In addition to Lumen, you will need a Direction Light and a Skylight. I made the Light Color of my Skylight light bluish, and its Intensity scale is 5. The Direction Light in my scene is 3.5 lux with tind driven by a Temperature of 6500. 

As I previously said, SpeedTree creates packed texture sets and ready Material Instances. There are two things to consider: the subsurface scattering profile sometimes needs to be adjusted dramatically, and it's convenient, non-destructive, and fast to do in the engine. For example, I adjusted the Brightness Curve of the Subsurface texture for Tamarisk branches to 0.5 from the initial 1. For the Poplar leaves atlas, the Subsurface texture correction of the Brightness Curve was even 0.2. 

The Nanite for each plant also needs to be manually enabled, and you will immediately notice that the precision of the plant rendering went far more neat with it.

In the folder of each plant, you will find a Material Instance with a whole bunch of settings, while Master Material can be found at the bottom of the settings in General -> Parent-SpeedTreeMaster. The good news is that we can rewrite the SpeedTree Master Material, adding some needed parameters or changing big chunks of its logic. For example, I added a top-down gradient and color variation for random instance logic to this shader, which helped me to emphasize Tamarisk bushes' colors.

While Euphrates Poplar and Tamarisk utilize the SpeedTree Shader with some adjustments, I created the Cleistogenes shader from the start. The wind system in this shader is simpler. As soon as I made the Cleistogenes meshes in Maya and didn't need to use SpeedTree (as it often takes place with grasses and flowers), I had to define the vertex colors for the wind there. It's usually two-toned vertex information used as a mask in the engine. In my case, I wanted to define the roots of the grass as not moving in the wind. It's possible to assign vertex colors in SpeedTree, too, by the way. After importing the mesh into the engine, you will need to choose the Vertex Color Import option as Replace, and what is important is that sometimes, after that, you have to reimport the mesh. 

And finally, the PostProcessVolume capacities can't be overlooked. It allows real-time color correction even in shadows/mid-tones/lights separately and the same by color channels. I adjusted the gamma of my scene to 1.05 with Shadows Max 0.9. The yellow channel saturation in the mid-tones and lights is also heightened to make Poplar's leaves stand out even more.   

In conclusion, I've learned a lot by experimenting with this project. The Tamarisk bushes were not trivial for me to set up, but the outcome was rewarding. Taking on new challenges sustains self-development, every iteration during attempting your project to look as you wish will lead to improvement. I'm always curious about future tasks and their challenges. 

I hope this article can be useful for fellow artists. Thank you for reading! 

Aria Baskakova, 3D Environment/Props/Materials Artist

Interview conducted by Amber Rutherford