Creating a Moody Treehouse Scene in ZBrush, Maya, Substance & UE4

Introduction

Since the last 80 Level article, I took a bit of a detour and decided to try making a character: a dwarf based on this cool concept by Joris Dewolf. It was something completely different from my usual day-to-day work and I took it as a personal challenge. It was a lot of fun, a nice project to work on during the first couple of lockdowns and I’m happy with the result. 😊 Once that was out of my system, I returned to environments. Before completely falling in love with Even Amundsen’s concept I was going back and forth between three projects. Two of them are based on artwork by some of my favorite concept artists, Andreas Rocha, and Hamish Frater. The third one is an original scene inspired by a recent-not-so-recent trip.
Fingers crossed they manage to see the light of day! 

Deciding on the Сoncept

I found Even Amundsen’s concept a few months ago while browsing the concept art channel on ArtStation. There are so many amazing pieces of art being posted daily that it doesn’t take long to run into something inspirational. I’ve followed him for a while on ArtStation and Instagram. The majority of his work is characters, but this was the first environment that I’d seen from him.

What I’ve always admired about his work is all the story and exposition that he puts into each piece, instantly giving them depth and a sense of place, something which really helped when building the scene.

Enrolling in a Mentorship

Around the same time that I was starting this project, I was feeling like my skills were plateauing a little bit and I decided to enroll in an Environment Art Mentorship with Kyle Bromley and the Mentorship Coalition. Some key areas that I wanted to focus on were creating tiling textures with ZBrush and Substance Designer. Kyle has a lot of experience with vegetation and organics, so we decided it would be great to take on this concept and make it the focus of the mentorship.

Gathering References

I started by gathering some photo references for rocks (particularly ones near rivers), moss, and old mossy trees just to get an idea of what needed to be made. I also had a look at some current-gen games, particularly the Demon’s Souls Remake (which I was trying to tear myself away from at the time!), and some of the trees in that game used similar techniques that would come to be used in this scene, such as a bake with a tiling tree bark, moss blends and foliage cards on top for extra support.

Creating the Asset List

I used Trello to create the asset list, jotting down everything that I would need for the scene, such as rocks, stone stairs, the large tree, grass/mud/moss tiles, water, etc. Each asset had a sub-list of how many shaders/textures would be needed and finally, how long they would take to make.

These were then arranged into three columns, “to start”, “for review” and “complete”, where I would move assets between when I had made progress on them. This might not be to everyone’s tastes, but it helped my productivity, forcing me to look at my own work more critically, and somehow calling things “half-done” meant there was always room for improvement, which helped me to push the overall quality at the end. The timings for each asset were not entirely accurate, but I found it useful just to have something to aim for.

Composition

Before starting with the blockout I wanted to first analyze and understand the composition of the artwork in order for me to translate it well into 3D.

There were several strong elements that I needed to preserve. The red door is the focal point of the concept. It’s a warm and saturated detail in high contrast with the cool and low hue tones of the environment. The composition and the surroundings are flowing towards the door. The tree, the branches, the river, and the stairs are all guiding the eyes to the focal point of the image.

Blocking Out

With those points in mind, I moved to the initial blockout pass in Maya. This consisted of spheres, cubes, and a very rough version of the tree (quickly made using ZSpheres in ZBrush). Basic shapes to sit in place of the main parts of the scene just to get the scale right.

Around this stage, I made a few sketches over the blockout in order to address things early and make sure I wasn’t veering too far away from the concept. For instance, the branches on the right-hand side of the frame needed to be pushed out further away from the house, and a small gap had to be visible between the monolith and the tree to make sure the silhouettes of the two assets didn’t get lost and confused. I also made a quick topology sketch over the tree to try and get an idea of how the model was going to flow.

Once the scale was looking OK, everything was brought over to ZBrush. I found it great to work in ZBrush at this early stage! It’s usually something that I use later in the pipeline, once all the assets are made, but after the mentorship, I really appreciate the benefits of using it early on, especially for organic set pieces. Having everything together at the beginning granted me so much more freedom and prevented me from becoming too bogged down organizing files and splitting things up. This approach helped to ground the scene, making everything sit together better, and giving an idea at the start of exactly what needed to be built. It is definitely a workflow that I will use more often in the future if the scene lends itself to it.

I spent a bit of time going back and forth between UE4 and ZBrush, making sure the composition was matching the concept. During this stage I started to put more work into the ground too, giving the impression that the house was positioned on a hill, with a path leading up and the river cascading downwards. 

Once the ground was shaping up, I swapped it for Unreal terrain to allow me to edit it on the fly. When the time came to place the ground assets this would eliminate the need to keep re-exporting the ground mesh every time I made a change.

Sculpting the Large Tree

Once I had the blockout at an advanced stage, the next thing was to focus more on individual objects and take them further. The first thing to build was the tree. Before the mentorship, the plan was to have a tiling tree bark with a decent amount of geometry that could allow me to create a detailed silhouette. After discussing it with Kyle, we decided it was best to sculpt, bake it and then add the tiling texture as a detail blend. The tree was the centerpiece of the scene and wasn’t going to be re-used elsewhere.

Using the ZSphere blockout mesh as a base, I started to work into the silhouette, using the Move brush to make it look more organic and gnarly. For sculpting, Clay Buildup, Dam Standard, and Standard were the main brushes used, adding broad strokes with the Clay Buildup brush first, and then creating more direction in the tree bark with Standard and Dam Standard.

This process was repeated a few times, working from big to small, until there was enough detail. 

In the past, I’d found that my sculpts didn’t always translate well from ZBrush to the final in-game asset. A lot of details were lost or were not pronounced enough in the sculpt. One of the ways around this was a brilliant method shown to me by Kyle.

I sculpted over the mesh using the Flake brush with a big brush size and a low intensity (around 6 or 7). This helped to give things a bit more of a “pop”, adding more depth to what would otherwise be small surface details that would probably get lost in the final bakes. A similar technique was used on the rocks and logs for the house later in the project. You can see in the gif below the effect it had on the sculpts. 

For the low-poly mesh, I used the Decimation Master for the first pass. It works very well for organic objects and does a great job of preserving the silhouette, although it can get a bit messy! So, after decimating the meshes to a reasonable amount they were brought over to Maya for a clean-up and UVing.

All the baking was done in Marmoset Toolbag. In the past, I had used it mainly for presenting, but Kyle introduced me to the baker, and it’s instantly become my favorite. It’s fast, very easy-to-use and the results are amazing. It’s well worth checking out for anyone who hasn’t had a chance to use it yet.

Here’s how the tree was looking in Maya before any foliage or vertex painting.

Now that the tree was baked and ready, it was time to leave it to one side and focus on the other objects.

Creating the Rocks

In the beginning, sculpting the rocks was tricky. The first few attempts were too angular and stylized from overuse of the Trim Dynamic and Hard Polish brushes. They looked too craggy and sharp, whereas in the ref they were smoother, eroded by their proximity to water. That was a look that needed to be preserved, so it took a lot of restraint to keep them simple, while also making them interesting enough to look at!

Rock alphas really helped here. I used the brilliant pack by Dannie Carlone to get some landmark cracks to start off, and then gradually layered up detail from there using a combination of the Standard and Clay Buildup brushes, followed by the same flake brush technique described above.

Following on from that it was just a case of using the Decimation Master for the lows, cleaning them up in Maya, UVing, and then taking them into Marmoset for baking. Substance Painter was used for the final texturing.

Here I started to put together the base color and roughness maps using the reference I gathered and some photos that I had taken from a recent trek. I blended them together using an amazing mask generator created by Josh Lynch.

This generator takes the normal map as an input, from which you can select various mask types and then tweak the ranges of them however you like. It is a very powerful tool that I would highly recommend.

Here’s another example of the generator being used in Designer when I was texturing the tree bark tile.

Now that I had several objects ready to go, I was eager to get started on the tiling textures for the blending!

Creating the Tiling Textures with NanoMesh

From the asset list, there were three ground tiling materials that were needed for the scene. Grass, moss, and the forest floor. For these, I would be using NanoMesh to create the high polys, bake them in Marmoset and then finally take them over to Substance Designer to finish them off.

Firstly, here’s the setup that was used in ZBrush.

The main sculpting plane is where the high geometry would be. The base tile plane underneath would be used to bake the details from the sculpting plane above (they sit closer together in ZBrush really. I moved them further apart just to take the picture). The two Unreal character meshes are to provide a reference for the texel density. Worth noting that the main sculpting plane is larger. This is to provide a run-off to ensure that, when it bakes to the base tile plane underneath, there are no seams visible in the bake.

I started by subdividing the main sculpting plane a few times (to around 1.2 million) and then grabbed a subtle displacement map from textures.com for some small base details. One of the hardest things I find about creating these materials is starting off, so to have a plane in there already, with some nice surface noise provides a nice canvas to work on.
Next, the Standard brush was used to sculpt in some undulation to the plane (ensuring that Wrap mode was set to 1 in the Brush > Curve settings). This is essential as it helps to avoid bakes from looking too flat, and one of the perks of the Standard brush is that it preserves details with each stroke, so there was no need to worry about losing any of the displacement map.

The next task was to create a small library of assets that included twigs, leaves, rocks, and moss clumps that would be used to fill up the tiles (these came with the added bonus of being used later for the alpha cards). I didn’t do anything too fancy here. Just more work with Clay Buildup, Dam Standard, and lots of DynaMesh!

Before using NanoMesh, it’s also worth decimating the objects, at least as much as possible without compromising their look, as the polycount can skyrocket very quickly using this method, which it did numerous times in my case, causing a lot of crashes. If your machine isn’t advancing in years like mine, then it shouldn’t be too much of an issue.

The next step was to duplicate the main sculpting plane and decimate this too, to around 2-3k polys. This is where I would be drawing the NanoMesh onto. The reason for this is because NanoMesh was going to create objects on every single face, and the sculpting plane was far too dense.

With the subtool selected (in this case one of the twigs), I went into the brush settings and hit “Create InsertMesh” and then “Create NanoMesh Brush”. Then using ZModeler I was able to click and drag onto the plane to add the twigs. At this stage, NanoMesh created the subtool on every single face, so the first thing to do was to go into the NanoMesh settings and play with the random distribution slider, which reduced the number of instances and scattered them around. Higher values add more instances and more density, but it was best to keep it fairly light in number as there were several more subtools that needed to be added. 

Once the first twigs were in, Freeze Placement was enabled in the NanoMesh settings and another twig subtool was added to the plane, using the same steps above. Enabling Freeze Placement ensured that the twigs that had previously been added wouldn’t be overwritten by the new ones.

I continued to add more subtools, playing with the random distribution and the rotation/length/width variation until there was a decent amount of randomness. It’s also possible here to cycle through the indices and tweak the settings of each group of instances. Disabling Show Placement is handy here too to hide the decimated plane.
Here’s how the NanoMesh looked once all the subtools were added. The turquoise boxes represent the boundaries of the tile that’s going to be baked. Worth noting that these also represent the corners of the base tile plane that’s sitting underneath.

Once all the subtools were in (was also worth saving a backup of the ZTool at this point!), the next step was to go into the geometry settings and click “Convert BPR to Geo”, converting the NanoMesh object into a normal subtool. Then I duplicated the tile 8 times, using Deformation > Offset to move each subtool around the main central one. This ensured that the NanoMesh was tiling across both axes. Once that was done, I merged the objects together, before masking off the bits around the sculpting plane. Finally, I hit delete hidden in the Geometry > Modify Topology settings to lower the polycount back down to sensible levels. 

As you can probably see here, I ran into an issue with this method at the beginning where the outer edges of the tile were far denser than the center. This is because the adjacent tiles are overlapping one another, adding more and more objects. The majority of these wouldn’t have been seen in the final bake as most of them are sitting outside the boundaries of the tile, but they were still noticeable.

To solve this, the number of objects around the edges was reduced so that when it came to duplicating, the other tiles would fill up the gaps. If it happened to still be too dense in some areas, I used the Standard brush on the plane to cover some of the twigs.

To create a base colour with a good amount of variation, a colour mask was needed to select different objects in Substance Designer. For this, I used polypaint in ZBrush.
The beauty of this workflow is that NanoMesh preserves the polygroups of each group of instances that you place, so automatically it’s possible to select each group with Ctrl+Shift and assign a different colour to it. This is slightly trickier with already-polypainted objects, such as the moss tile that was made later. In that case, I needed the polypaint for the base colour, so it was necessary to save two versions and export the polypaint for the base colour and the ID map separately. 

I selected each group of subtools and assigned a random colour using Colour > Fill Object. Then the NanoMesh geo and the high plane underneath were exported to Marmoset for baking. Here I saved out the Normal, AO, Curvature, Height, and vertex colour maps.

Once these were done, it was off to Substance Designer to create the final maps!

One of the things that I really enjoyed during this mentorship was the approach to texturing. Kyle taught me to approach Substance Designer as more of an addition to the texturing pipeline rather than a replacement, to be used in conjunction with ZBrush, and it’s a workflow that I’ve really come to appreciate since doing this project.

Here’s how the graph for the forest floor looked once it was finished.

Most of this graph consists of blend nodes that are used to combine different uniform colour nodes, with the vert colour map as a mask. For these, I used the “Colour to Mask” node which was perfect for colour picking each group of objects.

After looking at the ref though it felt like the textures weren’t as dense overall as they could’ve been. So, to save going back to the original NanoMesh, I used GrabDoc in ZBrush to quickly export the alphas from the twigs that I’d already sculpted. These were brought into Substance Designer as bitmaps like the other textures.

The Tile Sampler node was used to scatter these around for some smaller twigs, using the same vert color map as before to add them underneath the baked twigs.

And these were the final materials using the methods that I described above, rendered in Marmoset.

Creating the Grass

For the foliage texture sheet, I wanted to include several grass clumps, bits of moss to dress the assets with, and various plants/flowers for more variation.

For the grass, I needed about 3 clumps of varying sizes to get a decent amount of variety. For these I used FiberMesh. A quick plane was made to start with, which would represent the texture sheet. Then I added some smaller planes, sitting 90 degrees against the main plane, a bit like shelves on a wall.

Next, I used masking on these for the areas where I wanted the grass to appear. The preview of FiberMesh is camera-based, so the mesh needed to be front-facing to have the grass sprout up naturally.

The same method described above was used for the plants too. The leaves and flower heads were sculpted by hand, but the stems were generated with FiberMesh. You can see the full texture sheet before baking in the image below.

Creating the Moss Alphas

For the moss cards, the meshes that were used for the moss tiling texture were repurposed. They were already polypainted too so there was no need to worry about texturing them. This would ensure that they also sat together with the tiling texture in-game.

The other bits of foliage were made using the same FiberMesh method I used for the grass clumps. The various leaves and flower heads were sculpted using Clay Buildup and Dam Standard brushes with DynaMesh.

Creating Scattered Rocks and Leaves

It soon became apparent that some smaller rocks and pebbles would be needed to scatter around the scene, but I didn’t want to spend too long going back and baking them, especially if they were only going to be small.

Kyle introduced me to an amazing boolean script called DCBoolManager, which can be downloaded for free here.

Using this, it was possible to take the rocks that I had already made and create potentially dozens more from them (I decided to stop after about six). You can see a quick demo of it that I recorded below.

The beauty of this script is that, when using intersection to create new rocks shapes, it projects the UV mapping onto the intersected piece. This causes seams in the texture, but as the rocks were going to be small in the scene, there was no need to worry too much. 
This not only saved so much time and made it possible to create more variations of the rocks quickly, but it was also super cheap because everything was using the same textures!

Creating the Ground Twig and Leaf Stencils

One important piece of advice that Kyle said to me, which really stuck with me throughout the project, was to “not make anything flat”. Meaning, do not solely rely on just tiling textures to do the heavy lifting. Having this mindset really helped to push the scene further in terms of believability.

With that in mind, I decided to create some ground alpha objects to support the tiling materials. The twigs and leaves from the ground tile were already made, so it was just a case of creating the masks for the stencils.

Substance Designer was used to create these too. Using the same vertex color map that was used earlier, I added a Color to Mask node to mask only the bits that were needed (some of the twigs and leaves) and then combined it with a grunge map. The brush pattern in the grunge map settings was set to around 0.75 and a Levels node was used to make it solid white. Two variations were exported for these, one for the twigs and leaves and another with just leaves on their own.

Here are the ground alphas in Maya. I decided to add some of the branch alphas I made for the tree here too for some extra variation.

And here’s how they look on the ground in Unreal Engine.

Putting It All Together

Now that most of the materials and assets were close to a final stage, the next step was to start putting them together into prefabs. I looked at a lot of Megascans assets here for inspiration, specifically the ground pieces with rocks and grass, etc. 

Following advice from Kyle, I decided to use a similar method for my ground assets, treating each one as its own little set-dressed environment.

Most of this process was done in Maya. I made a rough shape for the ground to start with and then subdivided it for more density. Maya’s sculpting tools came in handy here too to create a bit of surface noise. I don’t often use them, but for things like this, it was much quicker than selecting verts.

Once the ground mesh was in, the rocks were added, followed by the vert paint for where the material blends were going to appear.

Finally, I added the foliage and small rocks.

This method was used for most of the assets in the scene, starting off with a ground piece and gradually adding more and more objects to it, layering it up, and using the vertex paint as a guide for set dressing the moss, grass, and rocks. I also used this same workflow for the large tree too, minus the ground.

The benefits of this workflow became quickly apparent when the assets started to be placed in Unreal Engine. There was a lot of preparation involved in setting them up, but once they were done, the scene came together very quickly. It was great to be able to populate the scene and already have everything set dressed. Another crucial advantage of this is being able to make changes to assets and then see the effects across the whole scene!

Creating the River with TharleVFX’s Shader

For the initial pass, I was using the Unreal water system, but I came across the amazing water shader by TharleVFX on the Unreal marketplace, which had everything that I needed, so I decided to go with that for the final presentation. It also came bundled with some particle effects too, which were perfect for around the rocks!

The first stage to create the river was exporting the terrain from Unreal Engine and bringing it into Maya. I used this as a guide to model around and create the shape of the river. I subdivided it a couple of times for better vert painting and to make sure the UVs weren’t distorting too much on the bends.

Here’s the result in Unreal Engine.

At this stage, I added an additional roughness blend on the rock material as well, for the ones that were sitting alongside the river.

For this, a simple scalar parameter was multiplied with the roughness map, which was then fed into a lerp using the green vertex channel as an alpha. There was also a constant3 added too to darken the base color slightly.

Lighting Setup

I decided it was best to stick with dynamic lighting for this scene, as I didn’t want to waste too much time with lightmap UVs and I was enjoying the freedom of being able to move assets around on the fly without having to relight. To start with, I added a Directional Light and a Skylight. These two did most of the heavy lifting in this scene. The lighting in the concept seemed quite cloudy and claustrophobic as if it’s nestled in the middle of a thick forest. To try and mimic this, the Directional needed to be quite subtle, with the Skylight being more dominant.

There was a lot of testing and trial and error around this stage, but eventually, I settled on the Directional intensity having a value of 2.0 as it seemed like a nice balance between picking out details on the assets but not being too overpowering.

I downloaded multiple HDR maps from HDRI Haven, which is a brilliant resource to have. In the end though after a lot of testing, the Skylight ended up using the captured scene itself rather than a custom cube map.

Here are the settings I had for the Skylight and the Height Fog.

The intensity of the Skylight was boosted up to 1.5 to compensate for the weaker Directional and I tinted the colour to a light blue to try and mimic the blue ambience of the concept.

Also, worth noting here is that the sky distance threshold was set to 1. This is an absolute must for capturing the scene if you aren’t going down the HDRI route because the default value doesn’t consider the immediate area surrounding the Skylight actor. William Faucher explains it very well in his video here. It’s well worth checking out the rest of his channel too, he has some amazing videos about lighting and cinematics in Unreal Engine.

I also disabled “Lower Hemisphere is Solid Colour” too. While it can give off a nice occlusion effect, it felt as though too much of the cube map was being lost here and it was important to preserve it, especially with the Directional being so subtle.

After a lot of tweaking, I dialled the Height Fog back a lot to make sure it didn’t overwhelm the scene too much, which is so easy to do! For the final renders, this was kept at 0.04 just to add some subtle depth to the scene. 

Aside from the Directional and the Skylight, there were several Spotlights positioned behind the tree. Unlike the main lights, these were really ramped up to give off the edge highlights, something which seemed important to emphasize with the subtle lighting.

There were several small point lights dotted around the scene too, with shadow casting disabled. These were used to give the ambience a little boost in places where things seemed to be getting a bit too dark.

Trying to figure out the lighting for this scene resulted in a couple of alternative lighting scenarios that ended up being included in the final project. The dusk version had an almost identical lighting setup to the main version. The only differences being the tint and intensity of the Skylight and the Directional.

The night-time setting, however, was a little bit different. For this I shifted the position of the Directional to behind the tree, to catch the edges of the foliage on the branches, and to provide extra contrast at the front for the orange light in the house. All the Point lights were turned off, however, the Spotlights to catch the edges underneath were still in there, but with a different tint. The Skylight was toned down to 0.6 and had a darker blue tint.

Here is where the Volumetric Fog was increased slightly too, to create the light rays coming from the window.

After taking another look at the concept, the lighting seemed quite desaturated, and I felt as though I was losing the focal point towards the end. To fix this, I decided to decrease the global saturation to 0.75 and boost the saturation of the door base color to emphasize it again.

Aside from that I boosted the global contrast to 1.5 and added a subtle chromatic aberration for the final shots. I also increased the toe value slightly to 0.6 in the final shots, for some added contrast.

Final Words

Overall, the project took about 3 months to finish. For sure the hardest parts of this were making a proper start on it and then knowing when to call it all finished. Knowing what assets needed to be made and how it all needed to be put together were problems that the mentorship really helped with. Having even just the most primitive assets at the start is a step in the right direction and after getting past that the project really gained momentum.

The next stumbling block showed itself at the tail-end of the project, particularly the final lighting. I spent several nights agonizing over the best lighting setup, making constant tiny tweaks, and then completely changing my mind about it the day after. It was many screenshots later before I finally settled on something that I liked!

I must say though, that it was an absolute joy to work on, and I owe that to the mentorship. Kyle really helped me to push myself further than I thought I could go. I learned so much over the course of one month. It was mind-blowing and quite overwhelming at times. I would urge anyone who was considering doing a mentorship to go for it. It’s well worth it and to have direct feedback from great artists such as Kyle is invaluable!

Thanks so much for reading! 

Michael James Cox, 3D Artist 

Interview conducted by Arti Sergeev