Using Photogrammetry and Unreal Engine to Create a Dark Medieval Scene

Introduction

80.lv: Please introduce yourself to those who still don’t know you. What have you been up to since the last 80 Level article?

Hello, we are Adam Jarosz and Michał Mierzejewski, the Founders of Scans Factory. As we mentioned in our previous article for 80 Level, our studio creates real-time scenes in cinematic quality. Packages are available on the Unreal Engine Marketplace, but this year we also started selling our scenes on the Unity Asset Store. We moved to the office with our work and expanded our small team with a colleague Łukasz Brożek. We have been cooperating with Łukasz for a long time and it is natural for our growing studio that people from an outsource team start to work full-time at the office. This significantly improves communication and allows us to run several projects at the same time, even in such a small team.

This year we have been overwhelmed by the number of external projects, but next year we want to focus on our scenes and we would love to do them in Unreal Engine 5. Expect some new announcements on our social media.

Today we would like to give you a little insight into how our newest scene was created with Unreal Engine 4.22 and Reality Capture: Medieval Fantasy Ruins – Dark Forest Environment.

The Project

80.lv: How do you get started with new asset packs? How do you choose the right direction? Why did you choose this Dark Souls-style fantasy vision this time?

Adam Jarosz: Our main idea has remained the same for years – we are still trying to prove that photogrammetry can be used in real-time. We love to call our standards game-ready cinematic quality. This time also in fantasy-style.

For me, the beginning was a bit less obvious because I just needed a break from the realistic style of lighting. Considering we had a nice crypt scan from last year, I thought it was time to finally use it for something interesting. After the initial research, I had an idea for a dark fantasy atmosphere in the swamps. A bit in Diablo-style. It also fits the company's plans to release something different than before. I was able to start working on the design.

I collected the first references and previews of the scans that best suited the idea to better illustrate the direction in which we will go.

After talking to the team, I took the day to prepare the content and see what would come of it. A large database of decimated scans is very useful to prototype the level. I try to design the levels as if I was actually creating a level for the game. Taking into account the structure and limitations of the game world. I take care of a few focus points.

Blockout should give us 2-3 main spots that look promising. We call it “Draft” and it usually takes 1-2 days. I leave the project to the team for a while so that they can work a bit on the most-needed content. Below you can see the comparison of this stage with the final stage.

From the very beginning, we were not sure if we would be able to create good vegetation. But as we delved into this process further and further, it turned out that we could achieve satisfactory quality. Then the idea came up of a more inaccessible place where the rays of light barely penetrate the layer of tree crowns. It changed our direction a bit. A darker and more atmospheric style of some Witcher or Dark Souls world. It was a very big and good change for our scene. Although we still believe that the original idea would be interesting as well.

This time I specified the world more precisely with references and further work on the level started. Here is a comparison of the changes that happened.

The level itself was being created for about 3-4 weeks. The full walkthrough, scratched lighting, and most of the meshes were already game-ready. At this stage, our outsourcing starts to work. I wanted to strengthen the dense mood of the world and asked our wonderful outsource for some effects. Running rats, birds circling over the ruins, moving mists, the sound of the wind or the raven. Nothing enhances the perception of the scene more than good sounds. Our clients do not see it at first glance but appreciate it when it comes to making the actual level.

Finally, I wanted to make some fire guides. It adds some warmth to this dark atmosphere and breaks dominant colors.

From that moment on, we tried not to make many changes. We gave ourselves time for polishing, lighting, and the last decimated scans proceeded into the final game-ready meshes. Time to fix collisions, lightmaps and adjust the scene to the marketplace guidelines. We optimize our levels to around 80-90 FPS full HD on GeForce GTX 1070. This process, along with marketing, took another 3 weeks.

Final results:

Photogrammetry

80.lv: Let’s talk about photogrammetry. Where did you take the data? What rigs did you use? How did you approach lighting? Could you provide an overview of the workflow for beginners?

Michał Mierzejewski: Due to the fact that we have a large base of our raw scans, the work could start quickly. Poland is full of wonderful objects such as old palaces, castles, factories. Especially those that interest us - abandoned. That's why we regularly expand our collection. The scanning process itself is also the shortest stage in creating the entire scene.

But “Medieval Fantasy Ruins – Dark Forest Environment” is not only ruins but also the forest mentioned in the title. We scanned many trunks, roots, sticks, grounds, etc. Not only in several outdoor scanning sessions, but also in the studio using a shadowless tent. I really like this stage of production when we turn off our computers and go beyond the four walls of the office, e.g. into the mountains.

Our scanning rig consists of a Nikon DSLR camera and a Quadralite Rx400 ring flash, which perfectly eliminates shadows. Additionally, this type of lamp as opposed to the classic flash can perform many shots without overheating. The only downside of such a set is probably only the weight, which starts to bother you after a few scans in a row.

We also very often scan without the use of additional lighting in optimal weather conditions, on a bright, overcast day without clear shadows. When the weather is bad for a scanning session, you can always choose the shaded part of an interesting object that you plan to scan.

When it comes to working in the studio, we have a large shadowless tent with an automatic turntable. It speeds up work and gives the ability to 360-scan objects. When we add to this the mask generation feature in Reality Capture it is a total game-changer.

Before taking pictures, it is also worth preparing the object and its surroundings. You don't always think about it, but it's helpful to get a few leaves of grass, plants and clean up before scanning. It is much easier to do this before scanning than to get rid of them at the stage of creating a game-ready model. That is why our standard equipment during outdoor sessions are work gloves, pruning shears, and a broom.

It is also worth mentioning that when deciding to use photogrammetry, sometimes the road to work can be difficult and bumpy.

Reality Capture Workflow

80.lv: How do you process the data with Reality Capture? How difficult is the whole process today? What are the steps? Perhaps you could break down one of the cases.

Michał Mierzejewski: Processing the data for us is extremely simple and fast. It is thanks to photogrammetry and the use of Reality Capture we can create so many assets in such a small team. Of course, the first step is to take pictures. We use DSLR cameras for this purpose, but it is not necessary. When I started my adventure with photogrammetry in 2014, I had an ordinary compact camera, and sometimes I did a scan with a smartphone.

But using DSLR cameras gives us the opportunity to take photos in RAW format, which then can be developed. Thanks to that we can get more information from the photo itself, partially get rid of some shadows, light burns, or create the correct color profile using ColorChecker.

Once the photos are developed, we can process them in Reality Capture. I like to do the process in stages, for full control of each of them. Usually, after the alignment, I slightly fix the orientation of the scan in relation to the grid. Then I edit the Reconstruction Region, narrowing it down to the area of interest to me. The next step is to build a mesh, and when I get the multi-million triangles model, I use the Simplify Tool twice. So we received 3 models: model high poly for bake textures, model high poly for retopology, and decimated version for level prototyping.

Next, we prepare the game-ready model. For smaller objects, such as roots or sticks we only decimate it and tweak gently, manually prepare UV, bake textures, and add collisions. In this way, only a few hours pass from the photo to the finished model in the Unreal Engine.

But in the case of buildings, we want them to be modular, which is why it is mainly a classic retopology. So, at the scanning stage, we have already thought about how we will split the model into pieces in the 3D program. After creating the final model, we also make variations based on it. It increases the usability of our content and saves a lot of texture memory space.

Of course, all the main parts of buildings are adapted to grid 50. As a result, this time the whole process is longer, but ultimately we gain because of its later usability.

Thanks to that, you do not have to limit yourself to the original shape of the scanned object, but you can let your imagination run wild.

This time we also went deeper into the foliage topic. We have scanned tree trunks, and the taller branches are based on seamless bark texture. Sounds like a piece of cake, but it took many, many iterations of each tree to find the right shape. The most difficult thing is to create something that we see every day, just like a tree. Because subconsciously we see that something isn't working. Also, it was necessary to strike a balance between the density of the branches and the amount of light we wanted to achieve on the level.

80.lv: What would you say are the main advantages of Capturing Reality as a photogrammetry toolkit? Have you tried using other solutions?

Michał Mierzejewski: As I have already mentioned, my adventure with photogrammetry started in 2014 and I had the opportunity to test most of the available software. As Lead 3D Artist and Lead Scanning Artist at The Farm 51 in 2015-2020, I also had the opportunity to do R&D and look for the best solutions for the needs of projects such as Chernobylite or Chernobyl VR Project. 

Based on these experiences I can say that Reality Capture is not only the fastest available software but also the most universal. It is simply possible to combine data from many different devices – the data from the drone, DSLR camera, and LIDAR scanner are connected without any problem.

Especially in Reality Capture, I liked the possibility of creating automatic masks, which I already mentioned when working with a shadowless tent. It speeds up work very much. Before that, such masks had to be made by hand. 

Other advantages include:

• The ability to import/export and Merge Components – so we can develop data on many computers to get the results faster.
• Pausing calculations process and creating so-called checkpoints – this is especially useful when you are working in such a small studio like us. And the same computer is used for daily work and 3D scans calculations.
• Autosave during calculation.
• Great working Clipping Box.
• Lossless mesh editing – component is duplicated with each change like for example simplification or smoothing.

At Scans Factory, we use the Enterprise version, but I recommend that everyone check out the free PPI version to make their own opinion.

The Lighting Setup

80.lv: The final presentation features several cool effects and great lighting that add to the whole atmosphere. Could you share some details on the setup here?

Adam Jarosz: I follow the principle that the idea is the most important. It is no different here. Once you get the idea that the light shining through the crown of trees will work, you just do it. The secondary is whether you do it because you've just been inspired by some film, vacation trip or you just know that Unreal Engine has the Volumetric Fog feature. It must enrich the scene in the chosen direction and that’s all.

Of course, experience in building a level design that will support it is another helpful thing to enhance this effect. The level on each side of the light will look different, and the challenge is to make the walkthrough look interesting. The world should be coherent and operate not only in every direction but also inside and outside of buildings. With that in mind, you should make each step from blockout to the final presentation.

Also, never be afraid to give yourself a chance for a complete change. Remove all lighting and set up again.

What is worth paying attention to is work on the style and consistency in details. VFX played a big role in this scene. But I think our friend and outsourcer Michał Gawron will talk about it better.

Michał Gawron: The most important aspect was to make the scene "alive" by adding dynamic elements. This has been achieved through several types of effects: particles, materials, and blueprints.

Ambient particle effects like fog, birds, and falling leaves add a subtle sense of movement and depth. On the other hand, the expressive color of a campfire or a candle attracts attention and is a reference point for the viewer.

The most dynamic effects use blueprints. One of them is a flying fire guide that shows the viewer around the scene. At the moment of approaching it, it begins to move along a spline path placed on the scene. On its way, it has several "breakpoints" where it waits for the viewer to continue the journey. The path and breakpoints are fully customizable from the editor.

Another effect is interactive foliage that reacts to the player. This applies not only to grass but also to larger bushes. By using the RenderTarget texture, foliage can react with any number of dynamic objects, not only with the player. For any foliage to start reacting with the player, all you need to do is add a special material function to its material and connect it to the World Position Offset input.

The last cool effect is running rats with procedural animation in the shader with control parameters. There are two versions of rats. The first moves the rats along a random spline path between two burrows. The second version has the option to align the rats to any ground geometry, without precisely set up the spline paths every time the level changes. The ground alignment is done in the shader based on the geometry depth map created from the orthogonal top view when the scene is loaded.

Conclusion

80.lv: How much time did it take to finish the whole project with over 400 assets? How did you optimize the process to save time? What are the tricks when working on such huge packs?

This is our third big pack for Unreal Engine Marketplace and our assets base has already over 1k + 2k in RAW scans. It may not be as large a library as the Quixel, but in such a small team we are more than happy with the results. Using decimated raw scans saved the most time, especially as they are great for prototyping levels with a final look. 

Another thing is that thanks to our previous packages, we don't need to create the same assets each time from scratch. We can reuse simple assets such as brick, grass, water, or base materials. 

It is important that we try to outline the shape of the level and style as soon as possible. This allows everyone to work much more accurately. Outsource also. Here is an example of new packages sketched on the content we already have with some raw scans decimated in Reality Capture:

But ultimately you need to know when to stop. Usually, at some point, we prepare a build for our friend for feedback. This time, that was also possible for our community, which often knows better what they need. This allows us to look at the project from a different point of view.

Working with photogrammetry is also very important here, it is thanks to scans that we can significantly optimize time. We don't have to recreate every detail and every layer of the dirty wall. Anyway, who is a better texture artist than time and nature itself?

We constantly strive to improve the quality of our scenes, we still see a lot of space for that. We look forward to what the new year will bring!

You can find more information about our projects on Facebook, ArtStation, Twitter, and Instagram.

Adam Jarosz and Michał Mierzejewski, the Founders of Scans Factory

Interview conducted by Arti Sergeev