17.5 Houdini Review: TOPs, Solid Bodies, Erosion

Olaf Finkbeiner thoroughly reviewed the new features and improvements brought with version 17.5 of Houdini.

Some time ago, Olaf Finkbeiner wrote an article for a German print magazine Digital Production in which he reviewed the new features and improvements brought with version 17.5 of Houdini. Today, we gladly present you its translation in English. If you prefer German you can download the article here.

This is the second Houdini review by Olaf on 80.lv. Read the first review of H17 here:

H17 Banshee Review from Olaf Finkbeiner

Olaf has also just released the first five parts of his new video tutorial on procedural organic modeling in Houdini. Make sure to check it out!

The viewport allows a real-time path-traced representation of volumes now. Both point and spotlights can be used.

* .hip or TOP?

Houdini 17.5 update provides not only minor improvements but also completely new functions and in this case, even a completely new and very innovative operator type called TOPs – Task Operators Networks. TOPs are certainly the highlight of this update. But let me start with a different topic first.

I’d like to begin with my own personal highlight: Houdini Nodes can have more than one output, while previously you had to connect a null node to the second or 3rd output if you wanted to display the output or if you wanted to look at the values in the geometry spreadsheet. Now you can switch the output used and the output displayed in the geo spreadsheet.

Unfortunately, this was not implemented very well. I would have wished for implementation right on the node and not in an old-fashioned drop-down menu or, in the case of the output, per Right-click sub-submenu. But maybe SideFX reserved this for H18, there is still hope. Nevertheless, it is very helpful. The node outputs can now be viewed in the geometry spreadsheet and this allows for much better handling of multi-output HDA workflows. Terrains and constraint setups are good examples for such multi-output workflows. My own tool currently under development, a plant generator (OGOP), is also going to benefit greatly from it since I am implementing a multi-lane/output workflow.

Multiple Output Nodes – Parameter Spreadsheet

Multiple Output Nodes – Show Output for Viewport

SOPs – Surface Operators

Since we’re already talking about SOPs, the place where geometry is created in Houdini, let’s continue with the innovations in this area.

Fuse, Measure, Blendshapes, Exploded View, the Scatter SOP, and Bound were improved.

Many more SOPs are now compilable. This is needed to build multithreaded node networks that will run on all CPU cores in parallel. Especially loops benefit greatly and execute much faster.

Some Interactive modeling tools have been added too: a polygon pattern selection, a squash and a stretch manipulator for non-uniform scaling. Construction Planes in the orthographic viewport have been improved.

Curvature visualization in the viewport with the measure SOP Node.

Tetrahedral Fiber Friction under Stress (Vellum)

The multi-softbody simulation tool Vellum that was just introduced in H17 got some enhancements. Friction attributes can now be set on per point basis. Tetrahedral soft bodies can now have fiber constraints and are additionally more harmonized.

The Visualization of the stress state is now also possible. These visualizations help a lot when setting parameter values. It saves you from having to simulate previews in order to identify problematic areas in your simulation.

Solid Bodies and Destruction

Updates in the area of destruction are, of course, mandatory, it is Houdini we are talking about after all. Again, RBDs have been improved and are faster now, especially if rigid bodies with instantiated geometry are simulated.

New tools have been implemented for the presentation and editing of constraint setups in the viewport. There is now a dedicated RDB constraint SOP. And the bullet solvers timeframe can now be manipulated.

Concrete now breaks with more detail on the edges (Material Fracture 2.0) and the boolean fracture SOP can now deal with open surface objects.

If you have a shadow, nobody can hurt you! The Viewport

The viewport was not very attractive when I started using Houdini: it was very technical and analytical, but not photorealistic in most cases.

The behavior of the Principled Shaders (Universal shader à la Disney) in the viewport is now comparable to Substance or Marmoset Toolbag. Now you can work with a PBR-workflow and get predictable results when rendering or displaying the textures in other software solutions.

If you want to create tools that require interactive manipulation in the viewport you will like the object level python support and the browser/inspector panel for python states. SideFX is obviously building the foundation for a more interactive Houdini. The functions I just mentioned are primarily made for the software developers and not for artists, as they are very advanced and require deep Python and Houdini knowledge. But this is typical Houdini and the tools are available for everybody, just ignore them if it is too much for you. SideFX used them to implement the new interactive constraint setup viewport manipulators and they are very cool and welcome if you need to set up an RDB destruction.

Terrain Erosion

There is not much to say here except that SideFX tackled exactly the pain points. Erosion requires 70% less disk space. I tested that, and yes, that’s the way it is.

However, I think World Machine and the new Gaea have even faster and/or better, more beautiful erosion solutions. But that does not matter. Why should I care? TOPs are here! Just read on and you will understand.

Game Dev Toolset

Alice Vision, an open-source photogrammetry tool, was connected to Houdini.
A dedicated hard surface shader will display game assets in Houdini’s Viewport close to their look in a game engine.

Other additions are:

blended linear and dual quaternion Skinning for FBX I / O
MikkT Support in PolyFrame SOP
a UV Attribute Transfer SOP
a Path Deformer
a building and spiral Generator

and many other improvements in existing tools.

Four Times the Speed of Light! (Mantra)

Unfortunately, not much happened here, we’ll probably have to wait for H18 for bigger advances in rendering. However, photon maps are now generated four times faster.

And Light Path Expressions can now save light without shadow. Really! Light without shadows! Wow!

TOP of the OPs

As I said, TOPs (short for Task Operator Network) are the highlight of this update. Procedural Dependency Graph technology (PDG) is the new technology behind TOPs. SideFX will also be offering Pilot PDG as stand-alone software, that is an indication of what we are dealing with here.

First of all, TOPs and PDG is about automating and scaling pipeline tasks.

It also allows to depict dependencies, analyze them and utilize existing computing power much more efficiently. At first, I thought: ok, nothing for me here, that’s something for big companies or at least not for freelancers. But that was far-fetched! I want to really stress this here so that you continue to read the article to the end. I will start, however, with the technical info aimed at larger studios and pipelines. TOPs have appeared at a perfect moment for me as my current project will benefit greatly from the new technology. I am sure you have noticed by now that I am over the moon.

Learning TOPs

The Houdini documents for TOPs are very detailed and mostly finished. Additionally, SideFX has also been producing tutorials just in time for the release of TOPs. Of course, all of the tutorials include the HIP scene files to download.

To start with, I recommend the text tutorial Create a task workflow to build cities, 20 pages by Robert Magee (SideFX), which describes the creation of a procedural city with TOPs.

Ari Danesh (SideFX) has also recorded three videos:

PDG – Conceptual Introduction explains the general concept of PDG/TOPs (20 minutes)
In PDG – Space Colonization he creates a simple PDG/TOPs network (30 minutes)
PDG – Particle Stream Wedging deals with so-called wedges using a particle example (over 75 minutes)

Finally, there’s a 5-part tutorial called PDG for design work (90 minutes) by Entagma from Munich, Germany. Moritz Schwind shows that TOPs can be also used by designers and freelancers and can certainly make their job easier. Plus, he shows the PDG on the render farm.

Unfortunately, the 17.5 help files for the engine plug-ins (Maya, 3ds Max, Unreal, Unity) are not yet accessible or ready. Tip: there is always some info in the Houdini changelog.

Kenny Lammer’s PDG for Indie Gamedev (or at indie-pixel.com) – features a 4-hour tutorial on procedural game level creation. PDG, TOPs, and the Houdini Engine for Unity are used. The Unity plug-in now has the PDG asset link. This makes it possible to integrate an entire environment created with PDG at once via Houdini Engine in Unity. The game environment designers can now work interactively in Unity and customize the procedural landscape.

Free Houdini learning edition.

Python

If the shipped TOPs function nodes are not enough, you can easily expand the functionality with custom Python scripts. VEX or Hscript don’t come much into play here. So for more complex TOPs task, you need to know Python well.

A very impressive example is the ML-Machine-Learning-Demo, check the launch event at minute 55.

The Python Library PyTorch was connected to TOPs to learn ultra fast terrain erosions. First, two image pairs are generated: a non-eroded terrain and an eroded terrain. Those images are afterward analyzed with ML. Once the AI has learned from the image pairs, it will be able to erode terrains in the same style but now ultra quickly with comparable results. This is obviously not that flexible, the AI would have to learn each type of erosion first.

The integration of the AI PyTorch Library is only an example of how TOPs can integrate Python Libraries into a workflow. The AI setup shown is also not part of the shipping Houdini, at least not for now.

TOPs – The Principle

At this point in this article, it will make sense to hold your breath for a moment as I’ll try to briefly introduce the TOPs principle.

TOPs are an integral part of Houdini, and HDAs (Houdini Digital Assets) can be seamlessly integrated. Python and Python libraries can be used intensively. But with TOPs, you can also achieve a hell of a lot, even without a line of code.

Houdini Digital Assets (HDAs) are processed in parallel in a PDG setup. This way you can utilize all the available compute power optimally, be it locally or on the farm.

TOPs enable the creation of large procedural environments (Landscapes) by connecting several HDAs (Houdini Digital Assets) into one dynamic and interactive system.

Adding TOPs to existing pipelines makes the most sense if Houdini is in use.

SideFX does not target very small studios or freelancers with Pilot PDG (Standalone TOPs), this is currently aimed at larger studios.

TOPs in Detail

TOPs work in conjunction with Tractor, Deadline or any other scheduler as it can be integrated via API. This should be possible with about 200 lines of Python code I was told. TOPs can also talk to asset management tools like Perforce. Messages can be senе to the гser via e-mail or even via Slack, but again, that would require a small Python script.

Image Magick is already integrated to perform typical image format and size changes automatically or to create so-called “contact sheets” – overviews of a series of images.

A command server will run as an independent process and connect to other applications via command line commands. Various nodes can now send commands to the server that will handle the communication with the connected software. This can be Houdini (in this case Python will be send) and also Maya. For Maya, the connection is prepared and shipped with TOPs. Any other software that permits that can be connected this way. The communication can be via XML-RPC or Raw-TCP commands. This way we should be able to connect to terrain generators like WM or Gaea, and that is why I don’t care too much about the still-sluggish erosion in Houdini. However, in the case of Gaea, we will have to be patient as the Gaea-API is not released yet.

1 of 2

There are also some shelf tools for TOPs that help with typical so-called wedging operations. Wedging is the creation of variations for testing settings of a simulation, LODs or variants of plants or terrains.

Of course, such a system has to read and write arbitrary data. CSV, JSON, XML, SQL nodes are provided. The generic generator creates work-items either exactly as many as being passed from above in the chain or a set amount. If that is not enough, you can use a Python-Processor or a custom node. “Environment edit” and “Filter by expression” – I leave it to your imagination.

TOPs also automate the loading and saving of geometry, textures, data, etc. TOPs can create folders or download data from an URL.

The HDA Processor, Invoke, and Geometry Import are the connections to Houdini SOPs, the geometry creation node in Houdini. This is what connects TOPs to the rest of Houdini.

Of course, even rendering can be automated with TOPs.

TOPs are a fully procedural network, like in the rest of Houdini. You can also create TOPs HDAs. TOPs can make use of loops, error handling is also possible.

And similarly to the geometry spreadsheet, there is a table view for TOPs. You can use this to analyze any stage within the setup.

Attributes

Anyone who knows Houdini knows that it is all about points and their attributes. I once made a Youtube tutorial just about that:

With TOPs, so-called “work items” are now processed similar to the points. The “work items” also have attributes, exactly like the points. And these “work items” can now be grouped just like the points and they are processed with a Houdini typical Node Network.

The “work items” attributes are now used to structure the following nodes, or they can be connected to SOPs to create different variants of an object or for other use cases.

Dependencies / Mapper / Partitioners

Creating, displaying and analyzing dependencies is one of the most important functions and one of the advantages of TOPs and PDG.

TOPs can control whether, for example, all tasks have to be finished before the next step can start, or if partial results are enough to commence working on the “work items”. The tasks can also be partitioned. So, for example, TOPs can start distributing trees if one of the generated terrain tiles is already finished simulating the erosion.

TOPs can also track partitioned changes and compute only necessary new calculations and not the complete terrain setup.

Conclusion

With TOPs, Houdini 17.5 includes a powerful extension to the scope of functions that both studios and freelancers can use to optimize processes.

The standalone application “Pilot PDG” is provided by SideFX only on request to studios – currently, a single user license from Pilot PDG cannot be purchased. I was trying hard to get an answer from SideFX as to the cost of the standalone Pilot PDG but had no luck. So currently people who want to work on their own with PDG will have to buy a Houdini license.

There are three commercial variants: Houdini-FX, Houdini-Core, and Houdini-Indie.

FX is the full version, Core cannot simulate. Indie is almost like FX, but can only be used by setups $100,000 in turnover per year and rendering is somewhat limited (4K). But for about 200 euros per year for rent, it is unbeatable. Check the full comparison here.

Olaf Finkbeiner, 3D Professional & Houdini Artist

Olaf Finkbeiner (born in 1968) works as a senior consultant in the automotive sector. He also has many years of experience in 3D visualization. After years of practice, mainly with 3ds Max and MentalRay, he found his new 3D software and renderer home in Houdini and Mantra.