Difference between revisions of "HoudiniTops"

From cgwiki
 
(165 intermediate revisions by the same user not shown)
Line 2: Line 2:
 
== Overview ==
 
== Overview ==
  
I thought Chops was my most hated part of Houdini. Then I tried using Tops. I've gone from confusion, to disdain, to fury, but after nearly a year of fighting it, I'm learning to, well, not ''love'' it, but can appreciate what it can do, and can make it do some useful things.
+
This page starts off very wordy, sorry about that. Feel free to skip down to the '''[[HoudiniTops#Examples|examples]]''' section if you just want hips vs my ramblings.
  
A huge problem with Tops is terminology, documentation, workflow. Lops was in a similar boat, but at least that had the excuse of deriving from USD. Tops should have made the bridge from existing Rops/Sops knowledge into Tops much easier.
+
A huge problem with Tops is terminology, documentation, workflow. Lops was in a similar boat, but at least that had the excuse of deriving from USD and having to stick to those concepts.
  
Anyway, I ''think'' I have a handle on it now, so here's an attempt to explain it in simple terms.
+
That said in December 2020 I had that 'click' moment. I've been able to do some complex work setups that would be difficult or impossible without Tops. It was a good time to update these notes, try and explain tops in terms Houdini artists can understand. If any of it isn't clear (or better, if you have fixes for my mistakes), let me know!
  
=== Tops vs Rops vs sops vs farms ===
 
Tops is a node graph, built in python, to do stuff. It's similar to Rops in that it lets you chain operations together and control execution order. Rops only lets you see what's going on at a high level, eg 'run mantra' or 'do a sim', while Tops is like going from /obj down into sops and playing with points; you can see the inner workings of the stuff you're running.
 
  
Because Tops has much more awareness of what each node is doing, it can do tricks that can be difficult with rops or standard renderfarm packages. Eg:
+
=== Tops vs Rops, obj vs sops, vs farms ===
 +
Tops is a node graph for tasks and processes. It's similar to Rops in that it lets you chain operations together and control execution order. A major difference between Rops and Tops is fine grain control. Layering analogies, Rops compared to Tops is like /obj compared to Sops.
 +
 
 +
* /obj is a container based view of things, you don't see what's going on inside the nodes, and it has a limited set of nodes
 +
* Rops is a container based view of rendering, you don't see whats going on inside the nodes, it has a limited set of nodes
 +
 
 +
vs
 +
 
 +
* Sops is an atomic view of things, you can get right down to the points, it's much more general in scope
 +
* Tops is an atomic view of tasks and processing, you can get right down to the frames/individual processes, it's much more general in scope
 +
 
 +
Because Tops has more awareness of what each node is doing, it can do things that would be difficult with rops or standard renderfarm software. Eg:
  
 
* ensure things are run in parallel as much as possible
 
* ensure things are run in parallel as much as possible
 
* never have tasks waiting unnecessarily
 
* never have tasks waiting unnecessarily
* use as many cpu cores as you have as much as possible
+
* use as many cpu cores as you have
 
* detect when tasks don't need to be re-run, skip over them
 
* detect when tasks don't need to be re-run, skip over them
* when connected to a farm, do complex control over many machines doing many tasks
+
* when connected to a farm, allow complex control over many machines doing many tasks
* put a simple gui on simple operations, a full python api for the complex things
+
* put a simple gui on simple operations
 +
* provide a full python api for complex operations
 
* has built in support to control other apps, eg maya, nuke, ffmpeg
 
* has built in support to control other apps, eg maya, nuke, ffmpeg
 
* has built in support to talk to asset management systems like shotgun
 
* has built in support to talk to asset management systems like shotgun
 
* has built in support to work with tractor, deadline, hqueue
 
* has built in support to work with tractor, deadline, hqueue
  
Some of those features work great, others feel a little early and rough, but the potential is there. It's all balanced by being a scary new Houdini context with very little crossover with existing workflows, but hopefully this page will demistify some stuff.
+
Some of those features work great, others feel a little early and rough, but the potential is there.
  
 
=== Workitems vs points ===
 
=== Workitems vs points ===
  
If going from /obj to sops lets you work with points, going from rops to tops lets you play with '''workitems''', the atomic unit of tops. At its simplest you can think of a workitem as a frame, so if you have a mantra top that renders frames 1 to 100, then when you run the graph you'll see 100 workitems. Tops visualises those as little dots on each top node.
+
If going from /obj to sops lets you play with points, going from rops to tops lets you play with '''workitems''', the atomic unit of tops. At its simplest you can think of a workitem as a frame. If a mantra top renders frames 1 to 100, you'll see 100 workitems in the GUI, represented as little dots.
  
 
[[File:tops_little_dots.png]]
 
[[File:tops_little_dots.png]]
Line 35: Line 45:
 
[[File:tops_workitem_types.gif]]
 
[[File:tops_workitem_types.gif]]
  
In a similar way that a point in sops can represent an actual point, or a packed primitive, or a RBD object, or whatever else, a workitem doesn't have to be just a rendered image. A workitem could be
+
Workitems are more than just frames of a render. In a similar way that a point in sops can represent a particle, a vertex of a shape, a packed primitive, an RBD object, a workitem could be:
  
 
* a frame of a render
 
* a frame of a render
 +
* an entire image sequence
 
* the location of a mp4 on disk
 
* the location of a mp4 on disk
* an image sequence
 
 
* an asset name
 
* an asset name
* etc
+
* whatever you want it to be
  
Workitems themselves can be collated together into groups called '''partitions''' should you wish to work on groups of things, and expanded out again later into individual workitems, similar to how you can unpack a packed prim back into points.
+
Workitems can be collapsed into '''partitions''' if you need that, and expanded out again later into individual workitems. This is similar to packing and unpacking geometry in sops.
  
 
=== No really, workitems can be anything ===
 
=== No really, workitems can be anything ===
Line 49: Line 59:
 
It's worth emphasising how workitems can be much more than frames.
 
It's worth emphasising how workitems can be much more than frames.
  
Everything in tops is a workitem, i.e. a unit of work, i.e. a process that is done.
+
Everything in tops is a workitem, i.e. a unit of work, a process that is done.
  
It's intentionally generic nature means it feels quite different to Rops, but means its capable of a lot more. A lot of pipeline tasks, even non vfx tasks can be handled in tops, because ultimately you're not controlling 3d or geometry, you're controlling processes and tasks.
+
It's intentionally generic nature means it can feel complex compared to Rops, but means it's capable of a lot more. A lot of pipeline tasks, even non vfx tasks can be handled in Tops, because ultimately you're not controlling 3d or geometry, you're controlling processes and tasks.
  
 
=== Generating vs cooking ===
 
=== Generating vs cooking ===
  
 
[[File:generate_vs_cook.gif]]
 
[[File:generate_vs_cook.gif]]
 +
<br>''Generate will make workitems, but do no work. Cooking will actually do the work.''
 +
 +
More sops analogies! Watch some of the masterclasses about sops, there's a LOT going on that users aren't aware of. Sidefx folk talk about dirtying sop nodes, traversing bottom up through the graph setting states to understand what nodes need to be cooked, then working top down to do the node cooking. In other words, it first ''generates'' a list of work to be done, then it goes and ''cooks'' the nodes.
  
More sops analogies! Watch some of the masterclasses about sops, there's a LOT going on that users aren't aware of. Sidefx folk talk about dirtying sop nodes, traversing bottom up through the graph setting states, then working top down cooking nodes and geometry as it goes. You could think of that first process of traversing bottom-up as generating a list of work to do, and then the working top-down as actually cooking that list of work.
+
While Sops hides all this from the user, Tops exposes those generate+cook steps to the user, and gives you explicit control to separate them if required.
  
While Sops hides that distinction from the user, Tops shows it all. For simple graphs you don't have to worry about it, tops will generate and cook in one hit for you, but it's good to know the difference when you get stuck on more complex graphs.
+
For simple graphs you don't have to worry about it, Tops will generate and cook automatically. For bigger graphs, you might have some nodes that could take hours to cook, maybe lock up all of a studios renderfarm. It would be hard to create or debug a Tops graph if you had to wait hours on every run!
  
The reasoning is you might have some nodes that take hours to calculate, but you don't always need to execute those nodes to know ahead of time how many frames (workitems) it could generate. In fact you might be able to do quite a lot of work designing your graph without ever having to execute any nodes, sort of like keeping Houdini in manual/nocook mode.
+
If you can work with the generate step seperate from the cook step, you might be able to do quite a lot of work designing your graph without ever having to execute any nodes, sort of like keeping Houdini in manual/nocook mode.  
  
You can right click on a node and choose 'generate'. If the node is smart enough, it will generate workitems for you, which you can then cook. Normally you just generate and cook at the same time, with the keyboard shortcut shift-v.
+
To use this you can right click on a node and choose 'generate'. If the node is smart enough, it will generate workitems for you, which you can choose to cook, or just continue to the next node in this high level 'design' mode. When you're ready to both generate and cook a node, you can do this from the r.click menu, or with the keyboard shortcut shift-v.
  
 
=== Inputs and outputs ===
 
=== Inputs and outputs ===
  
 
[[File:tops_ins_and_outs.gif]]
 
[[File:tops_ins_and_outs.gif]]
 +
<br>''The previous node output becomes this nodes input. This nodes output changes depending on what the node does!''
 +
 +
If tops nodes are to be chained together, they need to pass workitems betweeen them. Similar to Sops and point attributes, Tops workitems have '''workitem attributes'''. While Sop points have @ptnum and @P, workitems have an index, input and output defined. Broadly speaking, input is what the node will work on, and output is the finished result of that node.
 +
 +
In the gif above I run have a few nodes chained together. The inputs and outputs are as follows:
 +
 +
* Filepattern - makes workitems from a directory path, eg $HIP/myvideos/*.mp4
 +
** Input - nothing (it doesn't have an input from a previous node)
 +
** Output - a path to an mp4
 +
* Ffmpegextractimages - given a path to a video, create an image sequence.
 +
** Input - a path to an mp4
 +
** Output - an image sequence
 +
 +
You can see these by double clicking on a workitem, the input and output will be listed. Note that if you generate rather than cook, you'll only see inputs (but sometimes you'll see an 'expectedoutput' attribute if the node is clever enough to guess it).
 +
 +
=== Attributes ===
 +
 +
[[File:tops_cops_attrib.gif]]
 +
<br>''Workitem attributes can be directly called in the rest of houdini with the @ prefix.''
 +
 +
What if you need to get to those attributes in code? What if you want to define your own attributes? What if you want to use those attributes outside of tops?
  
If tops nodes are to be chained together, they need to somehow pass information betweeen them. Similar to sops and point attributes, tops uses workitem attributes. While points must have @ptnum and @P, workitems usually have an index, input and output defined. Broadly speaking, input is what the node will work on, and output is the finished result of that node.
+
The input and output are exposed as '''@pdg_input''' and '''@pdg_output''', as well as a few other built-ins. They're listed [https://www.sidefx.com/docs/houdini/tops/attributes.html#builtin here in the sidefx docs], but I've made a cheatsheet. Think of these like @ptnum, @P, @N, @v etc:
  
Eg, you have a filepattern top and a ffmpegextractimages top linked together. The filepattern top creates workitems from places on disk, so if you pointed it at $HIP/myvideos/*.mp4, when you cook that node and inspect a single workitem, there's no input attributes (it has no parent node giving it stuff), while output will be $HIP/myvideos/funnycat01.mp4.
+
* @pdg_input - the input to the workitem
 +
* @pdg_output - the cooked result of the workitem
 +
* @pdg_id - the unique id, usually appears like a many digit hash, eg 25325
 +
* @pdg_index - the 'public' index, closer to standard @ptnum 0, 1, 2 etc, but careful, its not guaranteed to be unique!
 +
* @pdg_name - combo of name of the node and the workitem id
 +
* @pdg_frame - the frame if you're doing frame based rendering, 0 if you're not
 +
* @pdg_inputsize - the size of the input (eg if the input is an image sequence, the length of the sequence)
 +
* @pdg_outputsize - the size of the output  
  
Moving to the ffmpeg node, if you r.click on it, choose 'generate', then inspect a workitem, you'll see that input is set to $HIP/myvideos/funnycat01.mp4. Ie, this is the input the node will use to do stuff.
+
These attributes are exposed to the rest of houdini, and are available in parameters wherever you'd use $HIP, $OS, $F, $T etc.  
  
There's no output yet, because the node hasn't cooked. Cook it, inspect the workitem again, there's now an output attribute, which contains an array of the image sequence generated by ffmpeg.  
+
In the gif I use a file pattern top to find *.jpg in a folder. I can then use `@pdg_output` in cops, and the file cop will load the image from the workitem.
  
Append another node, say a generic generator (kind of like a fancy null), and generate+cook it, select a workitem, now you can see that the input on this node is the same as the output from the previous node.  
+
Because these are most often used in parameters, the parameter/hscript rules apply; integers and floats usually work directly, strings will usually need to be wrapped in backticks.
  
Took me a while to get used to this, I couldn't follow the relationship between inputs and outputs. In hindsight it makes sense; outputs from the previous node are copied to inputs for the next node. What this node does to the outputs is up to the node! It might copy the input to the output untouched (say like this generator 'null', or a node that is just creating other attributes), or it could generate completely new outputs (say converting image sequences back into mp4's).
+
=== Tops vs standard workflows ===
  
=== Attributes ===
+
Tops can break regular Houdini workflows, keep this in mind if you're working with other artists who don't know tops.
  
What if you need to get to those attributes in code? What if you want to define your own attributes? What if you want to use those attributes outside of tops?
+
Imagine you hadn't read this page, didn't know Tops, and had been given a hip using the tricks so far. Would you be able to understand what's going on?
  
The input and output are exposed as '''@pdg_input''' and '''@pdg_output'''. A lot of work in tops is done using hscript expressions on parameters, so most of the time you have to escape them in backticks. There's several implicit pdg attributes like this.
+
On top of this, if you have a hip with a tops graph, when you load the hip the Tops graphs is uncooked. That means no workitems exist, meaning anything in the rest of Houdini using workitem attributes will be errored or broken.
  
Pdg attributes are available to the rest of houdini, and can be used where you'd do things like $HIP, $OS, $F, $T etc. When the tops graph is run, and that particular workitem within the tops node is being processed, those pdg attributes will be set and can be used. Eg, you use a Fetch top to run a rop somewhere else in houdini, say a cops graph. You could set the file cop at the top of your compositing network to use `@pdg_input` as the image source, and it will then be replaced with whatever image sequence you choose to find (or generate) in tops.
+
Tops is not forgiving to people who don't know Tops. I mean sure, neither is Chops, nor Lops, nor esoteric Dops stuff, but at least with most of those you can save the hip in a working state, and others can load it in that state.
  
=== Attributes and the rest of houdini ===
+
Another source of confusion is the highly parallel and distributed nature of Tops. This gets clearer over time, but again its another thing that can disturb workflow for existing Houdini users, who generaly fear change.
  
It's also worth noting that because these attributes are specific to the workitem, if you start using these in your houdini setups, they can be tricky to understand for other houdini users, especially those who aren't familiar with tops (which is most of the Houdini artist pool). Those @pdg attributes don't exist until the tops graph is cooked, and even when the graph is cooked, it won't do anything in the GUI until you click on a worktime dot, and even then, you probalby have to make sure you click the correct workitem dot on the correct tops node.
+
It's not just an instant drop in replacement for Rops, careful now.
  
=== Tops and the houdini gui ===
+
=== Generate mode, automatic vs other ===
  
Aka 'Why can't I see anything?'
+
Short version: I mentioned generate vs cook as two seperate steps in Tops, and Tops will usually automatically do the right thing. If you start getting weird results, change the automatic mode ( 'generate when' to 'each upstream item is cooked'), things should start behaving.
  
It's VERY confusing for new users, moreso because the state of the tops graph isn't remembered between saves. Ie you can cook the graph, save the hip, reload the hip, the tops graph looks uncooked again. Be careful.
+
Long version:
  
=== Generate mode ===
+
The ideal tops node should be able to generate workitems without cooking. For example a Mantra Top can look at the frame range parameters, and know you'll need 100 workitems without needing to render.
  
Short version: If things are acting weird, set the 'generate when' option to dynamic, you'll get a little purple icon to say it's now dynamic, stuff should work.
+
There's cases where that's not possible. Eg a ffmpeg extract images can't look at the parameters to know how many frames are in the video(s) it will process; it has to actually run ffmpeg, extract the frames, count the result.
  
Long version: Tops makes a distinction between 'generate' and 'cook' steps for nodes. Ideally you would know ahead of time what each node is going to produce, so you get a sense of how many images/sim caches/mp4's you're going to generate before the work is done. But sometimes you can't know that in advance, and you have to adjust workflow accordingly.
+
In these cases, the 'generate when' parameter tells tops not to try and predict the workitems. Even more accurately, it tells the node that it has to wait for the results of the previous nodes output before it tries to calculate its own workitems.
  
Eg, say you're using a Fetch Top to run a mantra render. You don't need to execute the render to find out how many frames you'll generate, its right there in the parameters on the mantra rop (or better yet on the fetch top itself). If you r.click the node and go 'generate', it'll populate with as many dots as you have frames in the render. Easy.
+
The chances of miscalculated workitems increases as Tops graphs get more complex. For example a graph might make temp folders, write results in those folders, count the number of files made in those temp folders, and do other things.
  
Now say you have a ffmpegextractimages top, and you want to use an attribute create top to set @framecount, the number of frames that were extracted. Tops cannot know this number until ffmpeg actually runs.
+
The 'count the number of files in the temp folder' node (a filepattern top) in its default state will happily look at a folder path, find the path doesn't exist yet, and generate no workitems. But then when you inspect the finished cooked graph, the folder exists (as now the upstream node to make the directory has cooked), the folder is full of files, and even if you recook the filepattern top now, you'll see it gets results and it all looks fine. Confusing!
  
The 'generate when' mode at the top of every node tells the node when to do its workitem calculation (the generate step). The default is 'automatic', tops will try and guess if nodes should wait for previous nodes to cook or not. But sometimes tops guesses wrong, and things will silently misbehave. In this case with the framecount setup, I had to set the mode to 'each upstream item is cooked'. So now when I'm processing many videos, only when each video has been extracted will the attribute create node start, and be able to create the right result.
+
Sidefx are trying to fix catch-22 situations like this, but its good to be aware that things like this can happen.
  
 
=== Caching ===
 
=== Caching ===
Line 115: Line 156:
 
Short version: If multiple runs of your tops graph look the same, delete results on disk, try again.
 
Short version: If multiple runs of your tops graph look the same, delete results on disk, try again.
  
Long version: Because Tops can get right down to that atomic workitem level, it can do some tricks that aren't possible in Rops or other systems. A big part of this is recognising when parts of the network have already been run, and don't need to be recooked.
+
Long version:
 +
 
 +
Because Tops can get right down to that atomic workitem level, it can do some tricks that aren't possible in Rops or other systems. A big part of this is recognising when parts of the network have already been run, and don't need to be recooked.
  
 
The example I keep coming back to here of processing a folder of mp4s. Say you had the following chain of tops nodes:
 
The example I keep coming back to here of processing a folder of mp4s. Say you had the following chain of tops nodes:
  
* extract images top converting mp4s to image sequences
+
* ffmpeg extract images to convert mp4s into image sequences
* fetch to to run a cops graph that processes those images to a temp location
+
* fetch to to run a cops graph that processes those images, renders to a temp location
 
* fetch to run a geometry rop that traces the images
 
* fetch to run a geometry rop that traces the images
 
* fetch to a opengl rop
 
* fetch to a opengl rop
 
* fetch to a VAT rop  
 
* fetch to a VAT rop  
  
etc. Obviously a lot of that only needs to run once if you're making minor changes here and there to the network, or adding a new video to a folder full of already processed videos.
+
Obviously a lot of that only needs to run once if you're making minor changes here and there to the network, or if you add a new video, it should only procees that video and skip the rest.
  
As part of the cook process, wherever possible top nodes will check if output exists where it expects to write files. If it does, it will mark that workitem as complete, saving a lot of processing time.
+
As part of the cook process, wherever possible top nodes will check if output exists where it expects to write files. If it does, it will mark that workitem as complete, skipping a lot of processing time.
  
 
Of course this is great when you expect it, infuriating when you don't.  
 
Of course this is great when you expect it, infuriating when you don't.  
Line 133: Line 176:
 
Most nodes have a r.click option after its been cooked, 'delete this nodes results from disk'. For the most part it does the right thing, and it will then of course force a recook of all the workitems on the next run.
 
Most nodes have a r.click option after its been cooked, 'delete this nodes results from disk'. For the most part it does the right thing, and it will then of course force a recook of all the workitems on the next run.
  
If you just want a single workitem recooked, you can go on disk and delete whatever cache that is. I've found there's a r.click menu per workitem dot too that also claims to do it, but then I also trusted that menu to show me the files linked to only the current workitem dot, instead it spawned 600 mplay windows, one for every frame of every workitem I had, and had to restart my computer. YMMV.
+
If you just want a single workitem recooked, you can go on disk and delete whatever cache that is. I've found there's a r.click menu per workitem dot to get that atomic with your deletes.
 +
 
 +
Note that sometimes tops gets confused and won't delete files, or will delete too much, best to keen an eye on your files while doing these operations until you get a feel for it.
  
Still, caching, be aware of it, make it work for you.
+
Caching, be aware of it, make it work for you.
  
== Tips ==
+
== Examples ==
=== Simple cache then render locally ===
+
=== Simple cache a sim then render locally ===
  
 
[[File:pdg_overview.jpg]]
 
[[File:pdg_overview.jpg]]
Line 146: Line 191:
 
Most existing Houdini users want the basics from PDG; cache a sim to disk, run a render. Maybe chaser mode as a bonus? FFmpeg the result into an mp4, why not eh, YOLO!
 
Most existing Houdini users want the basics from PDG; cache a sim to disk, run a render. Maybe chaser mode as a bonus? FFmpeg the result into an mp4, why not eh, YOLO!
  
Here's that setup. Click the triangle with the orange thingy on it to start.
+
Here's that setup. Click the triangle with the orange thingy on it to start. This was one of the first Tops setups I created, I've been told some things aren't ideal, like the map by index (apparently all mapping will gradually be phased out). One day I'll revisit this and clean up.
  
 
'''cache sim''' is a fetch top that points to a disk cache sop after a simulation. You DON'T want a sim cache running on multiple threads/machines, it should just be one job that runs sequentually. To do this enable 'all frames in one batch'.
 
'''cache sim''' is a fetch top that points to a disk cache sop after a simulation. You DON'T want a sim cache running on multiple threads/machines, it should just be one job that runs sequentually. To do this enable 'all frames in one batch'.
Line 164: Line 209:
 
Also note that the button with the orange thingy on it kicks off the output, looking for the matching node with the orange output flag. See in that screenshot how I've left it on the mantra node? That means it'll never run the ffmpeg task. I'm an idiot.
 
Also note that the button with the orange thingy on it kicks off the output, looking for the matching node with the orange output flag. See in that screenshot how I've left it on the mantra node? That means it'll never run the ffmpeg task. I'm an idiot.
  
=== Where's the tops task menu? ===
+
=== Wedging a rock generator ===
 +
 
 +
It was either this or a fence generator, lord knows we need more tutorials on both these important things.
 +
 
 +
[[File:tops_rockgen_pretops.jpg]]
 +
 
 +
Here's the hip before getting into tops if you want to follow along:
 +
 
 +
[[:File:tops_rockgen_start.hip]]
 +
 
 +
 
 +
In this hip is a straightforward rock generator. The sops flow is
 +
 
 +
* high res sphere
 +
* some scattered points with random scale and N+up
 +
* copy spheres to points
 +
* point vop to displace spheres with worley noise
 +
* attrib noise for more high frequency detail
 +
* attrib noise for colour
 +
* Cd converted to HSV and back again to roughly match the colour of the env background.
 +
 
 +
So with this all setup, we could randomise a bunch of stuff with tops.
 +
 
 +
The first thing we'll do is wedge the number of points in the scatter. We'll create a wedge top, which will make a pdg attribute we can reference on the scatter.
 +
 
 +
* Create a topnet
 +
* Create a wedge top
 +
* Set the wedge count to 5, so we get 5 variations
 +
* Add a new wedge attribute with the multilister
 +
* Attrib name '''scatternum'''
 +
* Attrib type '''Integer'''
 +
* Set start/end to '''2''' and '''6''', so we'll generate a minimum of 2 scatter points, a maximum of 6.
 +
* Shift-v on the node to cook it and see what we have so far.
 +
 
 +
[[File:tops_wedge_scatternum_range.gif]]
 +
 
 +
Middle clicking on each workitem, we can see that each workitem has a scatternum attribute, starting at 2 and ending at 6. That might be useful for other things, but here we don't want it to be gradually increasing, we want it to be a random integer between 2 and 6. Enable '''random samples''', cook, look again.
 +
 
 +
[[File:tops_wedge_scatternum_random.gif]]
 +
 
 +
That's better, random samples for each workitem.
 +
 
 +
To use this in the scatter sop, all we do is type '''@scatternum''' into the scatter force total count parameter, and bam, its connected.
 +
 
 +
[[File:tops_wedge_scatternum_connected.gif]]
 +
 
 +
Add more entries to the wedge multilister, fill in parameters over in sops, bosh, you have a wedge setup. Note that when you create vectors, you access the components with @attrib.0, @attrib.1, @attrib.2.
 +
 
 +
Eg here I create a noiseoffset wedge, and drive the point vop noise offset with it.
 +
 
 +
[[File:tops_wedge_noiseoffset.gif]]
 +
 
 +
So that's all wedging, how do we write this out? We could use a disk cache sop, and set the output name to use @pdg_index, which corresponds to the id of each workitem (ie, 0 to 4 in this case of 5 wedges). Or you could use a rop geometry output top which basically does the same thing.
 +
 
 +
*append a '''rop geometry output''' top to the wedge
 +
* set the sop path to the end of your sop chain, /obj/rocks/OUT_ROCK in my case
 +
* set the output file parameter to make a unique file per workitem, eg '''$HIP/geo/$HIPNAME.$OS.`@pdg_index`.bgeo.sc'''
 +
* Now if you click through the workitems in the previous node, you can see the file path change if you mmb on the label.
 +
* cook the node, and it'll be baked to disk.
 +
 
 +
[[File:tops_rocks_output_rop.gif]]
 +
 
 +
Here's the finished hip:
 +
 
 +
Download hip: [[:File:tops_rockgen_end.hip]]
 +
 
 +
=== Wedge a sim and generate a fancy QC mp4 ===
 +
 
 +
[[File:tops_wedge_50fps_sm.gif]]
 +
 
 +
Download hip: [[:File:top_wedge_sim.hip]]
 +
 
 +
Ooo, combinations of attributes, captions on the render, an mp4 to comfortably view on the couch, this the houdini dream! And not too hard to setup either.
 +
 
 +
While the previous example used a single wedge top to do all the randomising, in this case we'll use 2 wedge tops. The first will make 5 workitems, the second will make 5 workitems for each incoming workitem, so 25 in total.
 +
 
 +
Here's the popnet we'll be working with:
 +
 
 +
[[File:tops_popnet_pre_tops.png]]
 +
 
 +
I've found when setting up top wedging its handy to have the topnet visible in one panel, and the popnet (or whatever) visible in another panel. By using the P hotkey within each pane to toggle the parameters, and making sure each network is pinned, you can alter both networks pretty easily.
 +
 
 +
[[File:tops_split_views.gif]]
 +
 
 +
I want to wedge the popdrag air resistance, and the popwind noise amplitude. I make a wedge top, set wedge count to 5, create a tops_drag_air_resist attribute, and type @tops_drag_air_resist into the relevant field in the pop sim.
 +
 
 +
I connect another wedge top and repeat the process for tops_noise_amp. Cook the node, you'll see 25 workitems created, if you step through them you'll see each combo of the 2 attributes being set on the sim.
 +
 
 +
[[File:tops_dualwedge_setup.gif]]
 +
 
 +
Now to render this. Move over to rops, create an opengl rop, set the camera. The output files need to be unique per wedge combination, otherwise competing workitems will start overwriting results. A simple way to fix this is to make @pdg_index part of the output filename. Because this is a string parameter, it needs to be wrapped in backticks:
 +
 
 +
[[File:tops_opengl_rop_filename.gif]]
 +
 
 +
Don't bother setting the framerange, we'll do that from tops.
 +
 
 +
In tops append a fetch top, point it at the opengl rop you just made. Set Evaluate Using to 'Frame Range', set the frame range you want and enable 'Cook Frames as Single Work Item'. This means the workitems will end up being an array of the rendered images per wedge combination, rather than generating a workitem for each individual frame.
 +
 
 +
[[File:tops_sim_opengl_top.png]]
 +
 
 +
Cook now, you'll see renders start, and folders fill up with images. The last step is to make an mp4 out of these image sequences.
 +
 
 +
Append a partition by node top, cook it. You'll see all the workitem dots are now represented by a single rectangle. If you double click it and browse the output, you can see its an array of all the images from all the workitems:
 +
 
 +
[[File:partition_output.gif]]
 +
 
 +
That's the perfect array of images to feed to a ffmpegencodeimages top. Do that, cook, hey presto, a mp4 of all the output.
 +
 
 +
Buuut.... its just lots of particle variations, there's no way to tell what wedge values correspond to what render. We can fix this by using the comment feature on the opengl rop.
 +
 
 +
Go back to rops, bring up the parameters for the opengl rop, and in the scene tab go down to viewport comment. This can be used to create a text overlay on the render, so if we use tops attributes here, we can read that in the final renders. R.click on the parameter and choose 'Expression->Edit expression', and combine text and attributes to design a useful overlay:
 +
 
 +
noise amp: `@tops_noise_amp`
 +
air resist: `@tops_drag_air_resist`
 +
frame: $F
 +
 
 +
Except... it won't work. When you hit accept on the editor, it makes the parameter green, assuming its an expression, and the expression isn't valid hscript. Ugh. Luckily hscript is pretty forgiving, all you need to do is wrap the whole thing in double quotes:
 +
 
 +
"noise amp: `@tops_noise_amp`
 +
air resist: `@tops_drag_air_resist`
 +
frame: $F"
 +
 
 +
[[File:tops_opengl_comment_expression.png]]
 +
 
 +
Check it! Fancy wedge like the grownups do!
 +
 
 +
[[File:tops_opengl_comment_screengrab.png]]
 +
 
 +
=== Mosaic of all the things ===
 +
 
 +
[[File:tops_wedge_mosaic.gif]]
 +
 
 +
Download hip: [[:File:top_wedge_sim_mosaic.hip]]
 +
 
 +
Mosaic, or contact sheet, or montage as its strangely called within Imagemagick, the ultimate way for a time pressed fx lead to review their teams work and say 'I hate it all, start again'.
  
The help mentions some stuff in the task menu, I couldn't see it.
+
That I could do this without reading the docs surprised me, taking that as further proof that I'm starting to understand Tops.
  
Well, it's there. Hidden in plain sight.  
+
The '''Imagemagick''' node lets you do image processing. One of its tricks is a mosaic effect, which they call 'montage'. If you give it an image sequence of frames 1 to 10, you'll get those 10 frames laid out together in a grid.
 +
 
 +
That's almost what we need here, but rather than giving it frames 1 to 10 and lay them out in a grid, we'll grab all the first frames from the wedges, put those in a grid, save that image to disk.. Then all the second frames from all the wedges, lay those out in a grid, save that image to disk... one we have all the grids, make a mp4 out of those grids.
 +
 
 +
This is a roundabout solution, but basically its expand the wedges out to be individual frames, collect together the matching frames, montage them, ffmpeg them.
 +
 
 +
# copy @pdg_index to a new attribute, @wedgeindex
 +
# '''workitemexpand''' top so that we go from 25 items of 50 frames, to 1250 individual frames as workitems
 +
# the frame number isn't stored per workitem, so create a frame attribute by parsing the file path on each workitem
 +
# '''partitionbyattribute''' using @frame, and on distinct attribute values. This partitions all the frame 1 images together, all the frame 2 images etc
 +
# The results come back slightly out of order, but on the advanced tab, we can tell the partition top to sort by the @wedgeindex we created in step 1
 +
# '''Imagemagick''' top, montage mode, this makes a mosaic from each partition. Because we have all the frame 1's in a partition, they all get assembled into an image, as do frame 2, frame 3, frame 4...
 +
# '''waitforall''' (or partitionbynode), basically just get it to make a single array of all the images so that we can...
 +
# '''ffmpegencodevideo''' to make an mp4
 +
 
 +
=== Tops and sims further work ===
 +
 
 +
Looking at the result of the last 2 examples, there's a few things to note:
 +
 
 +
==== Nice names help discoverability ====
 +
 
 +
When you use tops attributes in parameters in the rest of Houdini, they end up looking just like regular attributes. I don't like this.
 +
 
 +
In my world, I'd want nodes to have a tops badge so I can easily see its being controlled by tops. The attributes themselves wouldn't re-use the @ prefix ( # maybe? Or ~, ie a tilde for tops? Or t@foo instead of @foo ?), and expressions would go a different colour when driven by tops attributes.
 +
 
 +
Until Sidefx see reason, you can help yourself and others by naming your attributes nicely. Don't use @noiseamp, use @tops_noiseamp. See? Now it's clear it's from tops!
 +
 
 +
==== Fetch top Frame by frame vs complete sequence ====
 +
 
 +
If you run a regular sim through a regular rop without caching the sim first, Houdini will sim each frame as required, then render. If you send that rop to a render farm, and the farm splits the job into one frame per render blade, then each blade will have to sim from frame 1 to the frame it needs to render. Not very efficient.
 +
 
 +
The better option would be to cache the sim first, then have each blade render the cache. Tops is no different. If this were a heavy sim, I would put a disk cache sop after the popnet, use a fetch top to execute it (making sure the cache had a wedge index so they don't overwrite each other!), and then I could run the opengl fetch top in whatever mode I want.
 +
 
 +
==== Workitems as frames vs workitems as image sequences, general efficiency ====
 +
 
 +
Related to the above, if I were rendering from a cache instead, then I could have the workitems render individual frames rather than the entire sequence. But think about this; each workitem is having to spin up its own instance of houdini, the opengl rop, render, then shut itself down again. For a simple render like this, it's probably doing more work just to startup and shutdown than to actually render. It's probably more efficient to startup, do the sequence, then shutdown. At a big scale this would also be impacted by licences; each tops graph in a render farm would consume a license per blade, you'd quickly eat up all your licences in a churn of blades starting and shutting.
 +
 
 +
==== Workitems as arrays elegance vs workitem busy work later on ====
 +
 
 +
This setup looks neat early on with with the fetch, partition, ffmpeg top to make a sequenece of renders, but gets ugly when making the mosaic as I have to do lots of work to make attribs, expand, partition, sort.
 +
 
 +
Alternatively I could have run the fetch to make a workitem for every frame (if I cached the sim first), which would mean I could directly partition the results by frame. Neater, but now potentially less efficient.
 +
 
 +
I don't know if one way is more elegant than the other (and I'm assuming there's probably a better way to make the partitions), but it's something to keep in mind.
 +
 
 +
==== Debug workflow with cached results, generate vs cook ====
 +
 
 +
I'm getting better at predicting when tops needs to do work, and when I can take shortcuts. I know that once the fetch top has run the opengl rop, any of the downstream nodes that are just manipulating workitems and attributes are basically instant. It's only when nodes have to do work (ie the ffmpeg and the imagemagick nodes) that I'd get a processing time hit.
 +
 
 +
Hence a lot of the debugging was appending nodes after the opengl fetch top, cook, inspect a result, frown, try another node, cook, inspect, grin, append a new node, cook that, tweak it, recook, inspect, etc. Tops is more efficient to work with for debugging workflows than I first gave it credit for.
 +
 
 +
Specifically for the debugging, the actual workflow is to say append a node, cook it, double click a workitem dot, look at the output. If its an array of frames, expand that to see if its doing what I expect. Eg for the mosaic workflow I could see it was collating all the frame 1s, frame 2s etc together in the partition sop, but the results weren't sorted:
 +
 
 +
[[File:tops_wedge_partitions_but_unsorted.png]]
 +
<br>''All the frame 1s together, but the order is random''
 +
 
 +
That's how I knew I had to store the workitem index so I could sort it properly later. It's not quite as fluid as using the geometry spreadsheet in sops, but it's not bad.
 +
 
 +
Even when hitting the heavy 'do work' nodes, tops will often aggressively lean on the cached results on disk, and rarely do work if it doesn't need to. I'm trusting it more and more to do the right thing, and generally know it'll only start doing heavy lifting if I right click on a node and select 'delete results on disk'.
 +
 
 +
==== Use the overlay text top ====
 +
 
 +
[[File:tops_overlay_text.gif]]
 +
<br>''Drop shadows? Font choice? Damnit.''
 +
 
 +
After working out that comment trick and feeling smug, I stumbled across another top, '''Overlay Text'''.
 +
 
 +
It's a HDA that gives you a nice big text editor in its parameter interface, and under the hood uses a cops graph to generate text and overlay it onto the incoming workitems (which would be images from a render). This means you can have much more control over size, layout, font etc than the opengl comment top, and it'll work over any render. Oh well. Still pleased with my trick.
 +
 
 +
=== Calling an external app (avconvert) with the generic generator ===
 +
 
 +
[[File:tops_generic_processor.png]]
 +
 
 +
Download hip: [[:File:tops_avconvert_generic.hip]]
 +
 
 +
FFmpeg is great, but there's some Apple specific stuff that it can't do. The always amazing [http://benjaminskinner.com.au/ Ben Skinner] pointed out that there's command line tools that ship with OSX to do Apple specific video stuff, was curious how to call them from Tops.
 +
 
 +
This page gave the clues, with a command line string we could port over to Tops:
 +
 
 +
https://kitcross.net/hevc-web-video-alpha-channel/
 +
 
 +
The '''generic generator''' is the top to use here. It provides a command line parameter, all you do is fill in the bits, and use the attribute and backtick stuff to construct the bits that will change per workitem.
 +
 
 +
So if the example command line string is
 +
 
 +
avconvert --preset PresetHEVC1920x1080WithAlpha --source source.mov --output output.m4v
 +
 
 +
The first the we need to do is swap source.mov for @pdg_input. Remember, this is a string parameter, so we need backticks to ensure it gets evaluated as an expression:
 +
 
 +
avconvert --preset PresetHEVC1920x1080WithAlpha --source "`@pdg_input`" --output output.m4v
 +
 
 +
Next is the output.m4v. We want to make sure each workitem writes to its own unique name. Cos I'm using a filepattern top right at the top of the flow, it generates @filename for me, so I can use that to create the name. Again, watch those backticks:
 +
 
 +
avconvert --preset PresetHEVC1920x1080WithAlpha --source "`@pdg_input`" --replace --output "$HIP/videos/`@filename`.m4v"
 +
 
 +
This will work, but for bonus points we can also fill out the 'expected outputs' tab. Set expected outputs to 'file list', and the output file parameter matches the end of the command string:
 +
 
 +
"$HIP/videos/`@filename`.m4v"
 +
 
 +
Why do this? Well, this means that if we just use generate instead of generate+cook, the workitems will have a @expectedoutput attribute, so we can build up the rest of the network, do testing, without needing to run this cook every time.
 +
 
 +
=== Partition files by suffix, convert between partitions and workitem arrays ===
 +
 
 +
Download hip: [[File:tops_partition_to_workitem_arrays.hip]]
 +
 
 +
I can use a filepattern top to search through $HFS/houdini/pic/* and generate 112 workitems, which are a mix of .pic, .png, .rat, other formats.
 +
 
 +
To group these by filetype, I can disable 'Extension in Filename Attribute' on that top, and append a 'Partition by attibute'. Set Partition By to 'Distinct Attribute Values', add an attribute to the multiparm, set name to ''extension''.
 +
 
 +
Cook, and you'll see there's partitions based on the extension.
 +
 
 +
[[File:partition_by_extension.png]]
 +
 
 +
Fine, but what if you need workitems arrays, like you get from the ffmpeg extract images top? For example I had a process where I started with those results, split some workitems off, did things, and wanted to merge with the original stream, but now I had a mix of partitions and workitems.
 +
 
 +
A handy trick is to use a generic generator to filter the results. Append one, enable 'copy inputs to outputs' and cook, now those partitions have been translated into workitems with arrays.
 +
 
 +
[[File:partition_by_extension_to_workitems.png]]
 +
 
 +
== Tips ==
 +
 
 +
=== Tops vs PDG? ===
 +
 
 +
They're essentially the same thing. PDG stands for Procedural Dependency Graph. You could get really pedantic and say that TOPs, or Task Operators, are the nodes within the Procedural Dependency graph.
 +
 
 +
You might also call Sops a PGG (Procedural Geometry Graph) and Cops a PCG (Procedural Compositing Graph).
 +
 
 +
BUT YOU DON'T. YOU CALL THEM SOPS AND COPS.
 +
 
 +
I think using PDG as a name is needlessly confusing (especially with the builtin @pdg_ prefix on attributes), so I'm trying to say Tops whenever I can.
 +
 
 +
=== Where's the Tasks menu? ===
 +
 
 +
The help mentions some stuff in the 'Tasks' menu, I couldn't see it.
 +
 
 +
Well, it's there. Hidden in plain sight. Lets play 'Which of these things is a menu, but doesn't look like a menu':
  
 
[[File:tops_menu.gif]]
 
[[File:tops_menu.gif]]
  
Hmmm.
+
=== What do the colours mean in the mmb info for workitems? ===
 +
 
 +
Colours represent the attribute types. Note that there's no distinction for arrays (see the float array in with the other floats)
 +
 
 +
* dark cyan - internal attributes
 +
* pink - string
 +
* green - integer
 +
* yellow - float
 +
 
 +
[[File:mmb_workitem_types.png]]
 +
 
 +
=== When pdg_input and pdg_output are blank ===
  
=== @pdg_input is blank ===
+
[[File:pdg_copy_in_to_out.gif]]
 +
<br>''A generic processor top wll not generate an output attribute unless you tell it to.''
  
 
In summary, check the nodes above the erroring one, especially if they have an unchecked '''copy inputs to outputs''' toggle.
 
In summary, check the nodes above the erroring one, especially if they have an unchecked '''copy inputs to outputs''' toggle.
  
[[File:tops_inputs_to_outputs.png]]
+
I've had a few occasions where I'll have a node error, go inspect a workitem and see that @pdg_input or @pdg_output is missing or empty. The culprit is always an upstream node. Related to what I mentioned before, nodes usually expect an input, and most of the times set an output.
 +
 
 +
This isn't mandatory though! If you insert nodes into an existing network, those node might not get the input nor set the output, always check their middle click info panel to see what going on.  
  
I've had a few occasions where I'll have a node error, go inspect a workitem and see that where I'm expecting to find a value for `@pdg_input` in a parameter, its actually an empty string.
+
For example:
  
The culprit is always a previous node. Related to what I mentioned before, nodes usually expect an input, and most of the times set an output. What can happen if you're not careful is some top nodes may not set an output, or don't do the implicit 'copy input to output' if they don't need to. The generic generator node, which I use occasionally as null to just have a place to see whats flowing through, is an example of this.
+
* Wedge tops if appened to a filepattern top will have inputs and outputs, but wedge nodes by themselves don't, even if chained together.
 +
* Generic processor tops won't generate output by default, unless you enable 'copy inputs to outputs'.
 +
 
 +
=== In bigger networks copy pdg_input to something else ===
 +
 
 +
You've seen by now that each node reads pdg_input from the previous node, and may or may not alter it to send to the next node. This is fine for smaller clean networks, but when you introduce branches, or insert lots of nodes in the middle of a network, nodes downstream might break because their expected pdg_input has changed.
 +
 
 +
It's a good habit to use an attribute create to name thing explicitly. Eg, you use a filepattern top to grab a bunch of alembic .abc files. Immediately put down a attribute create, name the attribute 'tops_abcpath', and the value will be `@pdg_input`. Any nodes from this point on can refer to @tops_abcpath, and you'll be confident its unlikely to change.  
 +
 
 +
The analogy here is using @Cd in sops to drive masks/geometry deletion/pops emission. Fine for small things, but as complexity grows, you're better off using explicitly named attributes to avoid confusion or unintended behaviour later.
  
 
=== Set a limit on the number of workitems ===
 
=== Set a limit on the number of workitems ===
Line 195: Line 532:
  
 
[[File:tops_only_first_n_items_per_partition.png]]
 
[[File:tops_only_first_n_items_per_partition.png]]
 +
<br>''Get the first 2 videos from each subfolder. Remember to turn on 'items in upstream partition'!''
  
 
I have a filepattern searching an inputs folder. That folder is full of subfolders, say animal names, and in each of those are mp4's I want to process. So the folder structure might look like this:
 
I have a filepattern searching an inputs folder. That folder is full of subfolders, say animal names, and in each of those are mp4's I want to process. So the folder structure might look like this:
 +
 +
[[File:tops_animal_paths2.png]]
 +
 +
Now say while testing I only want to grab 1 or 2 from each category?
 +
 +
A partition top lets you collate workitems, similar to packing geometry. You can partition in many ways, here I'd use a '''partition by attribute''' top. I'd use an '''attrib from string''' top to split off the animal type into its own attribute, @animal, then partition by 'animal'. Make sure 'partition attributes independently' is set.
 +
 +
Now we can unpack them again using a '''work item expand''', but handily this has several ways to do that unpack. In this case 'first N', sounds right, so we can expand only the first 2 from each animal.
 +
 +
On this node, make sure apply expansion to: is set to 'items in upstream partition', otherwise it can do odd things like duplicate the first item it finds per animal twice. Remember @pdg_index isn't guaranteed to be unique in the way @ptnum is, so when relying on tools to sort by index, it can easily grab stuff you don't expect.
 +
 +
Finally to make sure the output is sorted per animal, a '''sort''' top can be used, with the name parameter set to 'animal'.
 +
 +
=== Make workitems from a string array ===
 +
 +
[[File:tops_animal_paths_fake.png]]
 +
<br>''The alt title for this section is 'make a bunch of fake animal mp4s from some text cos I'm too lazy to make them manually'. ''
 +
 +
Download hip: [[:File:tops_animals.hip]]
 +
 +
Read this, but also read the next section for an even easier method...
 +
 +
The previous example used to have some quoted text where the screenshot of the files go, like this:
  
 
  /inputs/dog/poodle.mp4
 
  /inputs/dog/poodle.mp4
Line 210: Line 571:
 
  /inputs/bird/gull.mp4
 
  /inputs/bird/gull.mp4
  
etc.
+
I thought a screenshot of files+folders would look better, but how to make fake mp4s with all those locations quickly? That itch to [https://en.wikipedia.org/wiki/Eating_your_own_dog_food eat my own dogfood] was strong, so I figured there must be a tops way. Here's how:
  
Now say while testing I only want to grab 1 or 2 from each category?
+
# '''Attribute Create''', name 'test', paste in the text, which appears with newline symbols ( ¶ ) and spaces.
 +
# '''Attribute From String''' to split @test into an array using a space as the delimeter. This creates a new attribute @split, which is a string aray of the filepaths.
 +
# '''Work Item Expand''', expanding on 'upstream attribute', and the attribute is 'split'. Now I have 12 workitems, each has @expandvalue set, which is the name of the fake mp4.
 +
# '''Text Output''', set the path to '''$HIP/animals/`@expandvalue`''', cook it, hey presto, files on disk!
 +
 
 +
Note that the first workitem errors. The first split element is an empty string, so the Text Output node complains that it can't create that file. I could limit by range and remove the first workitem, or use a rule or something to remove it, but I have my fake mp4s now, no need. :)
 +
 
 +
==== Version 2 ====
 +
 
 +
An even easier method!
 +
 
 +
Download hip: [[:File:tops_animals_v02.hip]]
  
A partition top lets you collate workitems, sort of like packing geometry. You can partition in many ways here I'd use a '''partition by attribute''' top. I'd use an '''attrib from string''' top to split off the animal type into its own attribute, @animal, then partition by 'animal'. Make sure 'partition attribtes independently'.
+
I noticed that Attribute From String has another option for 'Store Result as', which is 'Seperate Work Items'. Perfect, can skip the Work Item expand, and just change the Text Output path to use @split rather than @expandvalue.
  
Now we can unpack them again using a '''work item expand''', but handily this has several ways to do that unpack. A handy one is 'first N', so here I can just say give me the first 2 from each animal, and I get just that.
+
=== workitem @pdg_index isn't unique like @ptnum ===
  
On this node, make sure apply expansion to: is set to 'items in upstream partition', otherwise it can do odd things like duplicate the first item it finds per animal twice.
+
For a while I was treating the workitem @pdg_index like @ptnum, in that I assumed it was always unique, would resort and reindex as you add and remove workitems.
  
'''MATT, YOU KEEP FORGETTING TO DO THIS, SET 'ITEMS IN UPSTREAM PARTITION'!!!!'''
+
I've since found that's not the case. If you do things like merge different streams, or do partitioning and unpacking, you could have 5 workitems that all have a @pdg_index of 0. That means if you do things like say limit by range to 0, thinking you'll just get the single zeroth entry, you'll get back 5.
  
Finally to make sure the output is sorted per animal, a '''sort''' top can be used, with the name parameter set to animal.
+
A bit confusing, you have to use things like sort tops and other nodes to force workitems to reindex, I'm still working out the best method for this.
  
 
=== Get framecount from ffmpeg extract images ===
 
=== Get framecount from ffmpeg extract images ===
Line 263: Line 635:
 
[[File:tops_attrib_split.png]]
 
[[File:tops_attrib_split.png]]
  
The tops method for getting array elements is @attribute.index, eg @split.0. Here I wanted the last element, but there's no python style -1 syntax, so instead I create the array, reverse it, read the 0th element. Specifically:
+
Tops array attributes can be accessed with @attrib.0, @attrib.1, @attrib.2 etc, but there's no python style @attrib.-1 to get elements from the end of the list. As a workaround we can simply reverse the array. So:
  
* attribute from string top, split by delimiter enabled, '/' as the delimeter
+
* attribute from string top, split by delimiter enabled, '/' as the delimeter. A @split attribute has been created, an array of the file path components.
 
* attribute array top, update existing 'split' attribute, reversed enabled
 
* attribute array top, update existing 'split' attribute, reversed enabled
 
* attribute create top, uses `@split.0`
 
* attribute create top, uses `@split.0`
 +
 +
=== Rename files ===
 +
 +
Be careful! The rename top works, but its very easy to destroy a bunch of files, delete them without a trace, so tread carefully.
 +
 +
I had a bunch of files with spaces in names, hyphens, brackets, all sorts of bad stuff. I wanted them to be cleaned up to only use letters, numbers, underscores.
 +
 +
A python script top can use regex to generate cleaned names:
 +
 +
<source lang="python">
 +
import re
 +
oldname = '`@filename`'
 +
newname = re.sub("[^0-9a-zA-Z]+", "_", oldname)
 +
newpath = '`@directory`'+'/'+newname+'`@extension`'
 +
work_item.setStringAttrib('newpath',newpath)
 +
</source>
 +
 +
You can then feed this to a rename top (make sure to enable 'copy inputs to outputs' on the python script top):
 +
 +
[[File:tops_rename.png]]
 +
 +
But like I said, be careful. It seems to run immediately, even in generate mode, so if you don't have your attributes right, it will happily rename everything to a null name, effectively deleting it.
 +
 +
=== Attribute from string expressions ===
 +
 +
==== Simple pattern ====
 +
This is a remarkably powerful node to let you split simple strings easily, and complex strings with powerful (but frustrating) regex hell.
 +
 +
Simple strings are nice and efficient. Say you have a bunch of files you pull in with a filepattern top that look like
 +
 +
beauty_indirect_v03.0035.exr
 +
 +
And you want to create nice attributes out of those various bits. Append a attribute from string, set the string to `@filename`. To do the pattern matching, it's easiest to explain it in 2 steps:
 +
 +
First, copy one of the filenames into the pattern parameter and wrap the bits of the file you want in curly brackets. Skip the bits you don't need, like the '_v' before the actual interesting version number:
 +
 +
{beauty}_{indirect}_v{03}.{0035}.{exr}
 +
 +
Now change the stuff in the curly braces to the attribute names you want:
 +
 +
{pass}_{aov}_v{version}.{frame}.{suffix}
 +
 +
Cook the top, hey presto, its tokenized the string and made attributes for you. It even change the string '0035' into a float attribute of 35, pretty clever. Easier to see in a gif probably:
 +
 +
[[File:tops_att_frm_string_simple.gif]]
 +
 +
==== Regular expression ====
 +
 +
Hoo boy. The help just gives a link to the official python regex docs, but thats it. Took a few goes to understand how to use it.
 +
 +
Make sure to bookmark this, keep it handy when trying to construct regex (thanks Jake Rice for the link!) : https://regexr.com/
 +
 +
In this case I'm using a filepattern top to grab butterfly* from $HFS/houdini/pic/butterfly*. The files look like this:
 +
 +
butterfly1.pic
 +
butterfly2.pic
 +
butterfly3.pic
 +
butterfly4.pic
 +
butterfly5.pic
 +
butterfly5Bump.pic
 +
butterfly6.pic
 +
butterfly7.pic
 +
 +
Notice that there's no separator between 'butterfly' and the number, and that one 'Bump' image in there too? This is more than the simple pattern mode can handle.
 +
 +
Enter regular expressions. Here's the pattern I used:
 +
 +
(\w+)(\d+)(\w*)
 +
 +
[[File:tops_regex.png]]
 +
 +
What does that mean? Similar to the simple pattern, regular brackets identify groups, they'll just be numbered rather than named. The actual tokens themselves:
 +
 +
* \w+ - find one or more letters
 +
* \d+ - fine one or more digits
 +
* \w* - find 0 or more letters
 +
 +
Simple right? No, not really, but powerful. Regular expressions can do a lot more, you can name your groups, do conditional logic, all kinds of text processing tricks. Just keep in mind the classic quote from Jamie Zawinski about regular expressions:
 +
 +
Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.
 +
 +
==== Regular expression and named outputs ====
 +
 +
Oh you don't like the index notation? You wanna go all the way and confuse your peers? Ok then.
 +
 +
Put ?P<groupname> at the start of the regex code. So to convert the previous example to create attributes called @prefix, @digits, @suffix:
 +
 +
(?P<prefix>\w+)(?P<digits>\d+)(?P<suffix>\w*)
 +
 +
Ugh.
 +
 +
[[File:regex_named.PNG]]
 +
 +
=== Update attributes based on value without code ===
 +
 +
[[File:tops_male_to_M.png]]
 +
 +
Download hip: [[:File:tops_attrib_male_female_to_M_F.hip]]
 +
 +
Requires sidefx labs to be installed (its now an option right on the installer, SUPER handy, just do it!). They're doing their best to reduce the need to know python to use Tops. Right now its almost unavoidable, but its a noble goal.
 +
 +
Use the labs 'filter by value' to split male from female, then attrib create to swap the value from 'male' to 'M' and 'female' to 'F', then merge.
  
 
=== Ffmpeg and non sidefx rops ===
 
=== Ffmpeg and non sidefx rops ===
Line 288: Line 762:
  
 
A last specific thing for our setup, our build of ffmpeg didn't understand the '-apply_trc' option, so I disabled it.
 
A last specific thing for our setup, our build of ffmpeg didn't understand the '-apply_trc' option, so I disabled it.
 +
 +
=== Python script and pdg_input ===
 +
 +
In a python script top, one of the examples in the drop-down shows you how to get index and name of the workitem:
 +
 +
work_item.index
 +
work_item.name
 +
 +
Great. So the one you'd most likely need, @pdg_input, that'd be obvious right? No.
 +
 +
work_item.inputResultData[0][0]
 +
 +
Wot? Or you can get ugly and mix your @'s with your python code, but you have to wrap it in backticks *and* in quotes. Ugh.
 +
 +
"`@pdg_input`"
 +
 +
=== Get modified time of workitem as attribute ===
 +
 +
[[File:modtime.png]]
 +
 +
Armed with the above stupid trick, we can do this:
 +
 +
<source lang="python">
 +
import os
 +
modtime = os.path.getmtime('`@pdg_input`')
 +
work_item.setFloatAttrib("modtime", modtime)
 +
</source>
 +
 +
Which is... what? Giant mystery number? It's the number of seconds since the epoch. What? Exactly. Different OS's define 'the start of all time' in different ways, Linux sets it as midnight, Jan 1, 1970. Humans want that in a more readable format, so you'll want to employ the python datetime module:
 +
 +
<source lang="python">
 +
import os
 +
from datetime import datetime
 +
modtime = os.path.getmtime('`@pdg_input`')
 +
work_item.setFloatAttrib("modtime", modtime)
 +
modtimestring = datetime.fromtimestamp(modtime).strftime('%Y-%m-%d %H:%M:%S')
 +
work_item.setStringAttrib("modtimestring", modtimestring)
 +
</source>
 +
 +
There, a nice giant number that's easy to compare for sorting files oldest/youngest, and a nice string attribute for humans to read.
 +
 +
=== Tops and imagemagick to make gifs via generic processor ===
 +
 +
[[File:tops_imagemagic_create_attribs.PNG]]
 +
 +
Download hip: [[:File:tops_imagmagick.hip]]
 +
 +
If you try the gif imagmagick tricks on the [[GeneralUtilties|General]] page with tops, you'll find nothing works. In the current build (18.5.633 in Oct 2021) you don't get enough control over the command string. It needs the options and the images in a specific order, tops won't let you.
 +
 +
Instead you can use a generic processor top to just call your completely custom command, which I used earlier to run the OSX native video conversion tool.
 +
 +
I tried dropping this command into the node, expecting the waitforall above it to send the correct images to imagemagick:
 +
 +
magick convert -loop 0 -delay 2  "`@pdg_input`" -layers OptimizeTransparency +map $HIP/out.gif
 +
 +
But it no worky. If you look at the log, you can see that @pdg_input just gets expanded to the first image, rather than the full sequence. Boo. JeffLMnt had the fix though; the '''pdginputvals''' command will expand that tops attribute into a full string, with each value separated by a space, and surrounded in double quotes. Handy! So this'll work:
 +
 +
magick convert -loop 0 -delay 2  `pdginputvals("",0)` -layers OptimizeTransparency +map $HIP/out.gif
 +
 +
But it's probably tidier to create attributes in a previous node, then have this command use cleaner attribs. Here's what my attribcreate looks like before the generic processor:
 +
 +
[[File:tops_imagemagic_create_attribs.PNG]]
 +
 +
And the line now becomes this:
 +
 +
magick convert -loop 0 -delay 2  `@images` -layers OptimizeTransparency +map `@output`
 +
 +
As a bonus, you can put @output into the second tab to define the expected outputs. This will mean you can r.click on the node and choose 'delete outputs from disk', and it'll do what you want. Neat.
  
 
== Farm ==
 
== Farm ==
Line 345: Line 887:
  
 
Moving the diary to [[TopsTractorDiary]].
 
Moving the diary to [[TopsTractorDiary]].
 +
 +
== Todo ==
 +
 +
Some neat ideas for examples/topics from Jeffy Mathew Philip:
 +
 +
# converting between one format to another(like ingesting a kitbash set and spitting out individual alembics, or fbxs or usds) - like building an asset zoo.
 +
# example running the hda processor for poly reduction or geo processing
 +
# using the invoke top for geo processing(can use the top geometry sop)
 +
# more uses of the partitioning tops, eg partitioning a wedged -> substepped simulations to give me the wedges, and then I combine at the end to use the partition by frame to run the correct set of frames

Latest revision as of 23:11, 13 October 2021

Overview

This page starts off very wordy, sorry about that. Feel free to skip down to the examples section if you just want hips vs my ramblings.

A huge problem with Tops is terminology, documentation, workflow. Lops was in a similar boat, but at least that had the excuse of deriving from USD and having to stick to those concepts.

That said in December 2020 I had that 'click' moment. I've been able to do some complex work setups that would be difficult or impossible without Tops. It was a good time to update these notes, try and explain tops in terms Houdini artists can understand. If any of it isn't clear (or better, if you have fixes for my mistakes), let me know!


Tops vs Rops, obj vs sops, vs farms

Tops is a node graph for tasks and processes. It's similar to Rops in that it lets you chain operations together and control execution order. A major difference between Rops and Tops is fine grain control. Layering analogies, Rops compared to Tops is like /obj compared to Sops.

  • /obj is a container based view of things, you don't see what's going on inside the nodes, and it has a limited set of nodes
  • Rops is a container based view of rendering, you don't see whats going on inside the nodes, it has a limited set of nodes

vs

  • Sops is an atomic view of things, you can get right down to the points, it's much more general in scope
  • Tops is an atomic view of tasks and processing, you can get right down to the frames/individual processes, it's much more general in scope

Because Tops has more awareness of what each node is doing, it can do things that would be difficult with rops or standard renderfarm software. Eg:

  • ensure things are run in parallel as much as possible
  • never have tasks waiting unnecessarily
  • use as many cpu cores as you have
  • detect when tasks don't need to be re-run, skip over them
  • when connected to a farm, allow complex control over many machines doing many tasks
  • put a simple gui on simple operations
  • provide a full python api for complex operations
  • has built in support to control other apps, eg maya, nuke, ffmpeg
  • has built in support to talk to asset management systems like shotgun
  • has built in support to work with tractor, deadline, hqueue

Some of those features work great, others feel a little early and rough, but the potential is there.

Workitems vs points

If going from /obj to sops lets you play with points, going from rops to tops lets you play with workitems, the atomic unit of tops. At its simplest you can think of a workitem as a frame. If a mantra top renders frames 1 to 100, you'll see 100 workitems in the GUI, represented as little dots.

Tops little dots.png

Workitems can be many things

Tops workitem types.gif

Workitems are more than just frames of a render. In a similar way that a point in sops can represent a particle, a vertex of a shape, a packed primitive, an RBD object, a workitem could be:

  • a frame of a render
  • an entire image sequence
  • the location of a mp4 on disk
  • an asset name
  • whatever you want it to be

Workitems can be collapsed into partitions if you need that, and expanded out again later into individual workitems. This is similar to packing and unpacking geometry in sops.

No really, workitems can be anything

It's worth emphasising how workitems can be much more than frames.

Everything in tops is a workitem, i.e. a unit of work, a process that is done.

It's intentionally generic nature means it can feel complex compared to Rops, but means it's capable of a lot more. A lot of pipeline tasks, even non vfx tasks can be handled in Tops, because ultimately you're not controlling 3d or geometry, you're controlling processes and tasks.

Generating vs cooking

Generate vs cook.gif
Generate will make workitems, but do no work. Cooking will actually do the work.

More sops analogies! Watch some of the masterclasses about sops, there's a LOT going on that users aren't aware of. Sidefx folk talk about dirtying sop nodes, traversing bottom up through the graph setting states to understand what nodes need to be cooked, then working top down to do the node cooking. In other words, it first generates a list of work to be done, then it goes and cooks the nodes.

While Sops hides all this from the user, Tops exposes those generate+cook steps to the user, and gives you explicit control to separate them if required.

For simple graphs you don't have to worry about it, Tops will generate and cook automatically. For bigger graphs, you might have some nodes that could take hours to cook, maybe lock up all of a studios renderfarm. It would be hard to create or debug a Tops graph if you had to wait hours on every run!

If you can work with the generate step seperate from the cook step, you might be able to do quite a lot of work designing your graph without ever having to execute any nodes, sort of like keeping Houdini in manual/nocook mode.

To use this you can right click on a node and choose 'generate'. If the node is smart enough, it will generate workitems for you, which you can choose to cook, or just continue to the next node in this high level 'design' mode. When you're ready to both generate and cook a node, you can do this from the r.click menu, or with the keyboard shortcut shift-v.

Inputs and outputs

Tops ins and outs.gif
The previous node output becomes this nodes input. This nodes output changes depending on what the node does!

If tops nodes are to be chained together, they need to pass workitems betweeen them. Similar to Sops and point attributes, Tops workitems have workitem attributes. While Sop points have @ptnum and @P, workitems have an index, input and output defined. Broadly speaking, input is what the node will work on, and output is the finished result of that node.

In the gif above I run have a few nodes chained together. The inputs and outputs are as follows:

  • Filepattern - makes workitems from a directory path, eg $HIP/myvideos/*.mp4
    • Input - nothing (it doesn't have an input from a previous node)
    • Output - a path to an mp4
  • Ffmpegextractimages - given a path to a video, create an image sequence.
    • Input - a path to an mp4
    • Output - an image sequence

You can see these by double clicking on a workitem, the input and output will be listed. Note that if you generate rather than cook, you'll only see inputs (but sometimes you'll see an 'expectedoutput' attribute if the node is clever enough to guess it).

Attributes

Tops cops attrib.gif
Workitem attributes can be directly called in the rest of houdini with the @ prefix.

What if you need to get to those attributes in code? What if you want to define your own attributes? What if you want to use those attributes outside of tops?

The input and output are exposed as @pdg_input and @pdg_output, as well as a few other built-ins. They're listed here in the sidefx docs, but I've made a cheatsheet. Think of these like @ptnum, @P, @N, @v etc:

  • @pdg_input - the input to the workitem
  • @pdg_output - the cooked result of the workitem
  • @pdg_id - the unique id, usually appears like a many digit hash, eg 25325
  • @pdg_index - the 'public' index, closer to standard @ptnum 0, 1, 2 etc, but careful, its not guaranteed to be unique!
  • @pdg_name - combo of name of the node and the workitem id
  • @pdg_frame - the frame if you're doing frame based rendering, 0 if you're not
  • @pdg_inputsize - the size of the input (eg if the input is an image sequence, the length of the sequence)
  • @pdg_outputsize - the size of the output

These attributes are exposed to the rest of houdini, and are available in parameters wherever you'd use $HIP, $OS, $F, $T etc.

In the gif I use a file pattern top to find *.jpg in a folder. I can then use `@pdg_output` in cops, and the file cop will load the image from the workitem.

Because these are most often used in parameters, the parameter/hscript rules apply; integers and floats usually work directly, strings will usually need to be wrapped in backticks.

Tops vs standard workflows

Tops can break regular Houdini workflows, keep this in mind if you're working with other artists who don't know tops.

Imagine you hadn't read this page, didn't know Tops, and had been given a hip using the tricks so far. Would you be able to understand what's going on?

On top of this, if you have a hip with a tops graph, when you load the hip the Tops graphs is uncooked. That means no workitems exist, meaning anything in the rest of Houdini using workitem attributes will be errored or broken.

Tops is not forgiving to people who don't know Tops. I mean sure, neither is Chops, nor Lops, nor esoteric Dops stuff, but at least with most of those you can save the hip in a working state, and others can load it in that state.

Another source of confusion is the highly parallel and distributed nature of Tops. This gets clearer over time, but again its another thing that can disturb workflow for existing Houdini users, who generaly fear change.

It's not just an instant drop in replacement for Rops, careful now.

Generate mode, automatic vs other

Short version: I mentioned generate vs cook as two seperate steps in Tops, and Tops will usually automatically do the right thing. If you start getting weird results, change the automatic mode ( 'generate when' to 'each upstream item is cooked'), things should start behaving.

Long version:

The ideal tops node should be able to generate workitems without cooking. For example a Mantra Top can look at the frame range parameters, and know you'll need 100 workitems without needing to render.

There's cases where that's not possible. Eg a ffmpeg extract images can't look at the parameters to know how many frames are in the video(s) it will process; it has to actually run ffmpeg, extract the frames, count the result.

In these cases, the 'generate when' parameter tells tops not to try and predict the workitems. Even more accurately, it tells the node that it has to wait for the results of the previous nodes output before it tries to calculate its own workitems.

The chances of miscalculated workitems increases as Tops graphs get more complex. For example a graph might make temp folders, write results in those folders, count the number of files made in those temp folders, and do other things.

The 'count the number of files in the temp folder' node (a filepattern top) in its default state will happily look at a folder path, find the path doesn't exist yet, and generate no workitems. But then when you inspect the finished cooked graph, the folder exists (as now the upstream node to make the directory has cooked), the folder is full of files, and even if you recook the filepattern top now, you'll see it gets results and it all looks fine. Confusing!

Sidefx are trying to fix catch-22 situations like this, but its good to be aware that things like this can happen.

Caching

Short version: If multiple runs of your tops graph look the same, delete results on disk, try again.

Long version:

Because Tops can get right down to that atomic workitem level, it can do some tricks that aren't possible in Rops or other systems. A big part of this is recognising when parts of the network have already been run, and don't need to be recooked.

The example I keep coming back to here of processing a folder of mp4s. Say you had the following chain of tops nodes:

  • ffmpeg extract images to convert mp4s into image sequences
  • fetch to to run a cops graph that processes those images, renders to a temp location
  • fetch to run a geometry rop that traces the images
  • fetch to a opengl rop
  • fetch to a VAT rop

Obviously a lot of that only needs to run once if you're making minor changes here and there to the network, or if you add a new video, it should only procees that video and skip the rest.

As part of the cook process, wherever possible top nodes will check if output exists where it expects to write files. If it does, it will mark that workitem as complete, skipping a lot of processing time.

Of course this is great when you expect it, infuriating when you don't.

Most nodes have a r.click option after its been cooked, 'delete this nodes results from disk'. For the most part it does the right thing, and it will then of course force a recook of all the workitems on the next run.

If you just want a single workitem recooked, you can go on disk and delete whatever cache that is. I've found there's a r.click menu per workitem dot to get that atomic with your deletes.

Note that sometimes tops gets confused and won't delete files, or will delete too much, best to keen an eye on your files while doing these operations until you get a feel for it.

Caching, be aware of it, make it work for you.

Examples

Simple cache a sim then render locally

Pdg overview.jpg

Download hip: File:pdg_basic_v01.hip

Most existing Houdini users want the basics from PDG; cache a sim to disk, run a render. Maybe chaser mode as a bonus? FFmpeg the result into an mp4, why not eh, YOLO!

Here's that setup. Click the triangle with the orange thingy on it to start. This was one of the first Tops setups I created, I've been told some things aren't ideal, like the map by index (apparently all mapping will gradually be phased out). One day I'll revisit this and clean up.

cache sim is a fetch top that points to a disk cache sop after a simulation. You DON'T want a sim cache running on multiple threads/machines, it should just be one job that runs sequentually. To do this enable 'all frames in one batch'.

map by index controls execution order and limits how jobs are created. If you have node A that generates 10 things, connected to node B that is also set to generate 10 things, PDG's default behavior is to generate 10 B things for each thing made by A. In other words, you'll get 10 x 10 = 100 total tasks. For situations like this, that's definitely not what you want.

The mapbyindex ensures tasks are linked together, so 1 frame of the cache is linked to 1 frame of the render. Further, it allows a 'chaser' mode, in that as soon as frame 1 of the sim cache is done, frame 1 of the mantra render can start, frame 2 of the sim cache is done, frame 2 of the mantra render can start etc.

mantra is another fetch top that points to a mantra rop.

waitforall does as implied, it won't let downstream nodes start until all the upstream nodes are completed. It also subtly adjust the flow of tasks; the previous nodes have 48 dots representing the individual frames, this node has a single rectangle, implying its now treating the frame sequence as a single unit.

ffmpeg top needs some explaining (and some adjustments to the fetch top that calls the mantra rop), which I explain below.

Note that the frameranges on the fetch tops override the ranges set on their target rops by default.

Also note that the button with the orange thingy on it kicks off the output, looking for the matching node with the orange output flag. See in that screenshot how I've left it on the mantra node? That means it'll never run the ffmpeg task. I'm an idiot.

Wedging a rock generator

It was either this or a fence generator, lord knows we need more tutorials on both these important things.

Tops rockgen pretops.jpg

Here's the hip before getting into tops if you want to follow along:

File:tops_rockgen_start.hip


In this hip is a straightforward rock generator. The sops flow is

  • high res sphere
  • some scattered points with random scale and N+up
  • copy spheres to points
  • point vop to displace spheres with worley noise
  • attrib noise for more high frequency detail
  • attrib noise for colour
  • Cd converted to HSV and back again to roughly match the colour of the env background.

So with this all setup, we could randomise a bunch of stuff with tops.

The first thing we'll do is wedge the number of points in the scatter. We'll create a wedge top, which will make a pdg attribute we can reference on the scatter.

  • Create a topnet
  • Create a wedge top
  • Set the wedge count to 5, so we get 5 variations
  • Add a new wedge attribute with the multilister
  • Attrib name scatternum
  • Attrib type Integer
  • Set start/end to 2 and 6, so we'll generate a minimum of 2 scatter points, a maximum of 6.
  • Shift-v on the node to cook it and see what we have so far.

Tops wedge scatternum range.gif

Middle clicking on each workitem, we can see that each workitem has a scatternum attribute, starting at 2 and ending at 6. That might be useful for other things, but here we don't want it to be gradually increasing, we want it to be a random integer between 2 and 6. Enable random samples, cook, look again.

Tops wedge scatternum random.gif

That's better, random samples for each workitem.

To use this in the scatter sop, all we do is type @scatternum into the scatter force total count parameter, and bam, its connected.

Tops wedge scatternum connected.gif

Add more entries to the wedge multilister, fill in parameters over in sops, bosh, you have a wedge setup. Note that when you create vectors, you access the components with @attrib.0, @attrib.1, @attrib.2.

Eg here I create a noiseoffset wedge, and drive the point vop noise offset with it.

Tops wedge noiseoffset.gif

So that's all wedging, how do we write this out? We could use a disk cache sop, and set the output name to use @pdg_index, which corresponds to the id of each workitem (ie, 0 to 4 in this case of 5 wedges). Or you could use a rop geometry output top which basically does the same thing.

  • append a rop geometry output top to the wedge
  • set the sop path to the end of your sop chain, /obj/rocks/OUT_ROCK in my case
  • set the output file parameter to make a unique file per workitem, eg $HIP/geo/$HIPNAME.$OS.`@pdg_index`.bgeo.sc
  • Now if you click through the workitems in the previous node, you can see the file path change if you mmb on the label.
  • cook the node, and it'll be baked to disk.

Tops rocks output rop.gif

Here's the finished hip:

Download hip: File:tops_rockgen_end.hip

Wedge a sim and generate a fancy QC mp4

Tops wedge 50fps sm.gif

Download hip: File:top_wedge_sim.hip

Ooo, combinations of attributes, captions on the render, an mp4 to comfortably view on the couch, this the houdini dream! And not too hard to setup either.

While the previous example used a single wedge top to do all the randomising, in this case we'll use 2 wedge tops. The first will make 5 workitems, the second will make 5 workitems for each incoming workitem, so 25 in total.

Here's the popnet we'll be working with:

Tops popnet pre tops.png

I've found when setting up top wedging its handy to have the topnet visible in one panel, and the popnet (or whatever) visible in another panel. By using the P hotkey within each pane to toggle the parameters, and making sure each network is pinned, you can alter both networks pretty easily.

Tops split views.gif

I want to wedge the popdrag air resistance, and the popwind noise amplitude. I make a wedge top, set wedge count to 5, create a tops_drag_air_resist attribute, and type @tops_drag_air_resist into the relevant field in the pop sim.

I connect another wedge top and repeat the process for tops_noise_amp. Cook the node, you'll see 25 workitems created, if you step through them you'll see each combo of the 2 attributes being set on the sim.

Tops dualwedge setup.gif

Now to render this. Move over to rops, create an opengl rop, set the camera. The output files need to be unique per wedge combination, otherwise competing workitems will start overwriting results. A simple way to fix this is to make @pdg_index part of the output filename. Because this is a string parameter, it needs to be wrapped in backticks:

Tops opengl rop filename.gif

Don't bother setting the framerange, we'll do that from tops.

In tops append a fetch top, point it at the opengl rop you just made. Set Evaluate Using to 'Frame Range', set the frame range you want and enable 'Cook Frames as Single Work Item'. This means the workitems will end up being an array of the rendered images per wedge combination, rather than generating a workitem for each individual frame.

Tops sim opengl top.png

Cook now, you'll see renders start, and folders fill up with images. The last step is to make an mp4 out of these image sequences.

Append a partition by node top, cook it. You'll see all the workitem dots are now represented by a single rectangle. If you double click it and browse the output, you can see its an array of all the images from all the workitems:

Partition output.gif

That's the perfect array of images to feed to a ffmpegencodeimages top. Do that, cook, hey presto, a mp4 of all the output.

Buuut.... its just lots of particle variations, there's no way to tell what wedge values correspond to what render. We can fix this by using the comment feature on the opengl rop.

Go back to rops, bring up the parameters for the opengl rop, and in the scene tab go down to viewport comment. This can be used to create a text overlay on the render, so if we use tops attributes here, we can read that in the final renders. R.click on the parameter and choose 'Expression->Edit expression', and combine text and attributes to design a useful overlay:

noise amp: `@tops_noise_amp` 
air resist: `@tops_drag_air_resist`
frame: $F

Except... it won't work. When you hit accept on the editor, it makes the parameter green, assuming its an expression, and the expression isn't valid hscript. Ugh. Luckily hscript is pretty forgiving, all you need to do is wrap the whole thing in double quotes:

"noise amp: `@tops_noise_amp` 
air resist: `@tops_drag_air_resist`
frame: $F"

Tops opengl comment expression.png

Check it! Fancy wedge like the grownups do!

Tops opengl comment screengrab.png

Mosaic of all the things

Tops wedge mosaic.gif

Download hip: File:top_wedge_sim_mosaic.hip

Mosaic, or contact sheet, or montage as its strangely called within Imagemagick, the ultimate way for a time pressed fx lead to review their teams work and say 'I hate it all, start again'.

That I could do this without reading the docs surprised me, taking that as further proof that I'm starting to understand Tops.

The Imagemagick node lets you do image processing. One of its tricks is a mosaic effect, which they call 'montage'. If you give it an image sequence of frames 1 to 10, you'll get those 10 frames laid out together in a grid.

That's almost what we need here, but rather than giving it frames 1 to 10 and lay them out in a grid, we'll grab all the first frames from the wedges, put those in a grid, save that image to disk.. Then all the second frames from all the wedges, lay those out in a grid, save that image to disk... one we have all the grids, make a mp4 out of those grids.

This is a roundabout solution, but basically its expand the wedges out to be individual frames, collect together the matching frames, montage them, ffmpeg them.

  1. copy @pdg_index to a new attribute, @wedgeindex
  2. workitemexpand top so that we go from 25 items of 50 frames, to 1250 individual frames as workitems
  3. the frame number isn't stored per workitem, so create a frame attribute by parsing the file path on each workitem
  4. partitionbyattribute using @frame, and on distinct attribute values. This partitions all the frame 1 images together, all the frame 2 images etc
  5. The results come back slightly out of order, but on the advanced tab, we can tell the partition top to sort by the @wedgeindex we created in step 1
  6. Imagemagick top, montage mode, this makes a mosaic from each partition. Because we have all the frame 1's in a partition, they all get assembled into an image, as do frame 2, frame 3, frame 4...
  7. waitforall (or partitionbynode), basically just get it to make a single array of all the images so that we can...
  8. ffmpegencodevideo to make an mp4

Tops and sims further work

Looking at the result of the last 2 examples, there's a few things to note:

Nice names help discoverability

When you use tops attributes in parameters in the rest of Houdini, they end up looking just like regular attributes. I don't like this.

In my world, I'd want nodes to have a tops badge so I can easily see its being controlled by tops. The attributes themselves wouldn't re-use the @ prefix ( # maybe? Or ~, ie a tilde for tops? Or t@foo instead of @foo ?), and expressions would go a different colour when driven by tops attributes.

Until Sidefx see reason, you can help yourself and others by naming your attributes nicely. Don't use @noiseamp, use @tops_noiseamp. See? Now it's clear it's from tops!

Fetch top Frame by frame vs complete sequence

If you run a regular sim through a regular rop without caching the sim first, Houdini will sim each frame as required, then render. If you send that rop to a render farm, and the farm splits the job into one frame per render blade, then each blade will have to sim from frame 1 to the frame it needs to render. Not very efficient.

The better option would be to cache the sim first, then have each blade render the cache. Tops is no different. If this were a heavy sim, I would put a disk cache sop after the popnet, use a fetch top to execute it (making sure the cache had a wedge index so they don't overwrite each other!), and then I could run the opengl fetch top in whatever mode I want.

Workitems as frames vs workitems as image sequences, general efficiency

Related to the above, if I were rendering from a cache instead, then I could have the workitems render individual frames rather than the entire sequence. But think about this; each workitem is having to spin up its own instance of houdini, the opengl rop, render, then shut itself down again. For a simple render like this, it's probably doing more work just to startup and shutdown than to actually render. It's probably more efficient to startup, do the sequence, then shutdown. At a big scale this would also be impacted by licences; each tops graph in a render farm would consume a license per blade, you'd quickly eat up all your licences in a churn of blades starting and shutting.

Workitems as arrays elegance vs workitem busy work later on

This setup looks neat early on with with the fetch, partition, ffmpeg top to make a sequenece of renders, but gets ugly when making the mosaic as I have to do lots of work to make attribs, expand, partition, sort.

Alternatively I could have run the fetch to make a workitem for every frame (if I cached the sim first), which would mean I could directly partition the results by frame. Neater, but now potentially less efficient.

I don't know if one way is more elegant than the other (and I'm assuming there's probably a better way to make the partitions), but it's something to keep in mind.

Debug workflow with cached results, generate vs cook

I'm getting better at predicting when tops needs to do work, and when I can take shortcuts. I know that once the fetch top has run the opengl rop, any of the downstream nodes that are just manipulating workitems and attributes are basically instant. It's only when nodes have to do work (ie the ffmpeg and the imagemagick nodes) that I'd get a processing time hit.

Hence a lot of the debugging was appending nodes after the opengl fetch top, cook, inspect a result, frown, try another node, cook, inspect, grin, append a new node, cook that, tweak it, recook, inspect, etc. Tops is more efficient to work with for debugging workflows than I first gave it credit for.

Specifically for the debugging, the actual workflow is to say append a node, cook it, double click a workitem dot, look at the output. If its an array of frames, expand that to see if its doing what I expect. Eg for the mosaic workflow I could see it was collating all the frame 1s, frame 2s etc together in the partition sop, but the results weren't sorted:

Tops wedge partitions but unsorted.png
All the frame 1s together, but the order is random

That's how I knew I had to store the workitem index so I could sort it properly later. It's not quite as fluid as using the geometry spreadsheet in sops, but it's not bad.

Even when hitting the heavy 'do work' nodes, tops will often aggressively lean on the cached results on disk, and rarely do work if it doesn't need to. I'm trusting it more and more to do the right thing, and generally know it'll only start doing heavy lifting if I right click on a node and select 'delete results on disk'.

Use the overlay text top

Tops overlay text.gif
Drop shadows? Font choice? Damnit.

After working out that comment trick and feeling smug, I stumbled across another top, Overlay Text.

It's a HDA that gives you a nice big text editor in its parameter interface, and under the hood uses a cops graph to generate text and overlay it onto the incoming workitems (which would be images from a render). This means you can have much more control over size, layout, font etc than the opengl comment top, and it'll work over any render. Oh well. Still pleased with my trick.

Calling an external app (avconvert) with the generic generator

Tops generic processor.png

Download hip: File:tops_avconvert_generic.hip

FFmpeg is great, but there's some Apple specific stuff that it can't do. The always amazing Ben Skinner pointed out that there's command line tools that ship with OSX to do Apple specific video stuff, was curious how to call them from Tops.

This page gave the clues, with a command line string we could port over to Tops:

https://kitcross.net/hevc-web-video-alpha-channel/

The generic generator is the top to use here. It provides a command line parameter, all you do is fill in the bits, and use the attribute and backtick stuff to construct the bits that will change per workitem.

So if the example command line string is

avconvert --preset PresetHEVC1920x1080WithAlpha --source source.mov --output output.m4v

The first the we need to do is swap source.mov for @pdg_input. Remember, this is a string parameter, so we need backticks to ensure it gets evaluated as an expression:

avconvert --preset PresetHEVC1920x1080WithAlpha --source "`@pdg_input`" --output output.m4v

Next is the output.m4v. We want to make sure each workitem writes to its own unique name. Cos I'm using a filepattern top right at the top of the flow, it generates @filename for me, so I can use that to create the name. Again, watch those backticks:

avconvert --preset PresetHEVC1920x1080WithAlpha --source "`@pdg_input`" --replace --output "$HIP/videos/`@filename`.m4v"

This will work, but for bonus points we can also fill out the 'expected outputs' tab. Set expected outputs to 'file list', and the output file parameter matches the end of the command string:

"$HIP/videos/`@filename`.m4v"

Why do this? Well, this means that if we just use generate instead of generate+cook, the workitems will have a @expectedoutput attribute, so we can build up the rest of the network, do testing, without needing to run this cook every time.

Partition files by suffix, convert between partitions and workitem arrays

Download hip: File:Tops partition to workitem arrays.hip

I can use a filepattern top to search through $HFS/houdini/pic/* and generate 112 workitems, which are a mix of .pic, .png, .rat, other formats.

To group these by filetype, I can disable 'Extension in Filename Attribute' on that top, and append a 'Partition by attibute'. Set Partition By to 'Distinct Attribute Values', add an attribute to the multiparm, set name to extension.

Cook, and you'll see there's partitions based on the extension.

Partition by extension.png

Fine, but what if you need workitems arrays, like you get from the ffmpeg extract images top? For example I had a process where I started with those results, split some workitems off, did things, and wanted to merge with the original stream, but now I had a mix of partitions and workitems.

A handy trick is to use a generic generator to filter the results. Append one, enable 'copy inputs to outputs' and cook, now those partitions have been translated into workitems with arrays.

Partition by extension to workitems.png

Tips

Tops vs PDG?

They're essentially the same thing. PDG stands for Procedural Dependency Graph. You could get really pedantic and say that TOPs, or Task Operators, are the nodes within the Procedural Dependency graph.

You might also call Sops a PGG (Procedural Geometry Graph) and Cops a PCG (Procedural Compositing Graph).

BUT YOU DON'T. YOU CALL THEM SOPS AND COPS.

I think using PDG as a name is needlessly confusing (especially with the builtin @pdg_ prefix on attributes), so I'm trying to say Tops whenever I can.

Where's the Tasks menu?

The help mentions some stuff in the 'Tasks' menu, I couldn't see it.

Well, it's there. Hidden in plain sight. Lets play 'Which of these things is a menu, but doesn't look like a menu':

Tops menu.gif

What do the colours mean in the mmb info for workitems?

Colours represent the attribute types. Note that there's no distinction for arrays (see the float array in with the other floats)

  • dark cyan - internal attributes
  • pink - string
  • green - integer
  • yellow - float

Mmb workitem types.png

When pdg_input and pdg_output are blank

Pdg copy in to out.gif
A generic processor top wll not generate an output attribute unless you tell it to.

In summary, check the nodes above the erroring one, especially if they have an unchecked copy inputs to outputs toggle.

I've had a few occasions where I'll have a node error, go inspect a workitem and see that @pdg_input or @pdg_output is missing or empty. The culprit is always an upstream node. Related to what I mentioned before, nodes usually expect an input, and most of the times set an output.

This isn't mandatory though! If you insert nodes into an existing network, those node might not get the input nor set the output, always check their middle click info panel to see what going on.

For example:

  • Wedge tops if appened to a filepattern top will have inputs and outputs, but wedge nodes by themselves don't, even if chained together.
  • Generic processor tops won't generate output by default, unless you enable 'copy inputs to outputs'.

In bigger networks copy pdg_input to something else

You've seen by now that each node reads pdg_input from the previous node, and may or may not alter it to send to the next node. This is fine for smaller clean networks, but when you introduce branches, or insert lots of nodes in the middle of a network, nodes downstream might break because their expected pdg_input has changed.

It's a good habit to use an attribute create to name thing explicitly. Eg, you use a filepattern top to grab a bunch of alembic .abc files. Immediately put down a attribute create, name the attribute 'tops_abcpath', and the value will be `@pdg_input`. Any nodes from this point on can refer to @tops_abcpath, and you'll be confident its unlikely to change.

The analogy here is using @Cd in sops to drive masks/geometry deletion/pops emission. Fine for small things, but as complexity grows, you're better off using explicitly named attributes to avoid confusion or unintended behaviour later.

Set a limit on the number of workitems

Tops filter by range.jpg

Say you have a folder full of images that you want to process, but for testing just want the first 5 images.

A filterbyrange top will let you do this.

Pick 2 items from each group of workitems

Tops only first n items per partition.png
Get the first 2 videos from each subfolder. Remember to turn on 'items in upstream partition'!

I have a filepattern searching an inputs folder. That folder is full of subfolders, say animal names, and in each of those are mp4's I want to process. So the folder structure might look like this:

Tops animal paths2.png

Now say while testing I only want to grab 1 or 2 from each category?

A partition top lets you collate workitems, similar to packing geometry. You can partition in many ways, here I'd use a partition by attribute top. I'd use an attrib from string top to split off the animal type into its own attribute, @animal, then partition by 'animal'. Make sure 'partition attributes independently' is set.

Now we can unpack them again using a work item expand, but handily this has several ways to do that unpack. In this case 'first N', sounds right, so we can expand only the first 2 from each animal.

On this node, make sure apply expansion to: is set to 'items in upstream partition', otherwise it can do odd things like duplicate the first item it finds per animal twice. Remember @pdg_index isn't guaranteed to be unique in the way @ptnum is, so when relying on tools to sort by index, it can easily grab stuff you don't expect.

Finally to make sure the output is sorted per animal, a sort top can be used, with the name parameter set to 'animal'.

Make workitems from a string array

Tops animal paths fake.png
The alt title for this section is 'make a bunch of fake animal mp4s from some text cos I'm too lazy to make them manually'.

Download hip: File:tops_animals.hip

Read this, but also read the next section for an even easier method...

The previous example used to have some quoted text where the screenshot of the files go, like this:

/inputs/dog/poodle.mp4
/inputs/dog/pug.mp4
/inputs/dog/terrier.mp4
/inputs/dog/labrador.mp4
/inputs/cat/black.mp4
/inputs/cat/striped.mp4
/inputs/cat/white.mp4
/inputs/cat/ginger.mp4
/inputs/cat/longhair.mp4
/inputs/bird/parrot.mp4
/inputs/bird/gull.mp4

I thought a screenshot of files+folders would look better, but how to make fake mp4s with all those locations quickly? That itch to eat my own dogfood was strong, so I figured there must be a tops way. Here's how:

  1. Attribute Create, name 'test', paste in the text, which appears with newline symbols ( ¶ ) and spaces.
  2. Attribute From String to split @test into an array using a space as the delimeter. This creates a new attribute @split, which is a string aray of the filepaths.
  3. Work Item Expand, expanding on 'upstream attribute', and the attribute is 'split'. Now I have 12 workitems, each has @expandvalue set, which is the name of the fake mp4.
  4. Text Output, set the path to $HIP/animals/`@expandvalue`, cook it, hey presto, files on disk!

Note that the first workitem errors. The first split element is an empty string, so the Text Output node complains that it can't create that file. I could limit by range and remove the first workitem, or use a rule or something to remove it, but I have my fake mp4s now, no need. :)

Version 2

An even easier method!

Download hip: File:tops_animals_v02.hip

I noticed that Attribute From String has another option for 'Store Result as', which is 'Seperate Work Items'. Perfect, can skip the Work Item expand, and just change the Text Output path to use @split rather than @expandvalue.

workitem @pdg_index isn't unique like @ptnum

For a while I was treating the workitem @pdg_index like @ptnum, in that I assumed it was always unique, would resort and reindex as you add and remove workitems.

I've since found that's not the case. If you do things like merge different streams, or do partitioning and unpacking, you could have 5 workitems that all have a @pdg_index of 0. That means if you do things like say limit by range to 0, thinking you'll just get the single zeroth entry, you'll get back 5.

A bit confusing, you have to use things like sort tops and other nodes to force workitems to reindex, I'm still working out the best method for this.

Get framecount from ffmpeg extract images

The ffmpeg extract images node usually generates a output attribute which is an array of the images its created. At some point this stopped working for me, so I had to find another way to count the images.

After talking to support it was due to putting quotes around the output parameter path. Doh. Still, leaving this here as it's still a good mini example of tops workflows.

So as I said, we could try and find the files in the ffmpeg output folder for the node. A filePattern top can do this. If the ffmpegextractimages node is using the default

$HIP/images/`@pdg_name`/frame%06d.jpg

The following filepattern node should point to the folder above it, ie

$HIP/images/`@pdg_name`/*

It also has 'split files into seperate items' turned OFF, so that way I just have a single workitem per image sequence.

If you try this now, it won't generate any workitems. The default generate mode will mean it tries to look in that folder straight away, finds no images, and as such returns no workitems. Change the generate mode to 'each upstream item is cooked', then it works as expected.

Ok great, but where's the actual number of frames? It's there, annoyingly hidden. There's some tops attributes that don't appear in the middle click info, one of those is @pdg_outputsize. In this case, unsurprisingly, it returns the amount of frames in the sequence. So with an attribute create node, you can create a new integer variable called framecount, and set an expression to use @pdg_outputsize.

Tops framecount.gif

Note that you don't need to change the generate mode on the attribute create. As soon as any upstream node is set to be dynamic (ie, it has to wait for previous items to cook), all subsequent nodes are also made dynamic.

Create attribute from string array element

Related to the previous example, I used an attribute from string node to split a directory path into components, and then wanted to create a new attribute based on the last part of that path. Annoyingly I couldn't work out how to get an array element from the standard attribute create node, so I gave up and used a python script node instead:

renderpass = work_item.attrib('split')[-1]
work_item.setStringAttrib("renderpass", renderpass)

TIME PASSES

Had another go, here's a purely node based approach:

Tops attrib split.png

Tops array attributes can be accessed with @attrib.0, @attrib.1, @attrib.2 etc, but there's no python style @attrib.-1 to get elements from the end of the list. As a workaround we can simply reverse the array. So:

  • attribute from string top, split by delimiter enabled, '/' as the delimeter. A @split attribute has been created, an array of the file path components.
  • attribute array top, update existing 'split' attribute, reversed enabled
  • attribute create top, uses `@split.0`

Rename files

Be careful! The rename top works, but its very easy to destroy a bunch of files, delete them without a trace, so tread carefully.

I had a bunch of files with spaces in names, hyphens, brackets, all sorts of bad stuff. I wanted them to be cleaned up to only use letters, numbers, underscores.

A python script top can use regex to generate cleaned names:

 import re
 oldname = '`@filename`'
 newname = re.sub("[^0-9a-zA-Z]+", "_", oldname)
 newpath = '`@directory`'+'/'+newname+'`@extension`'
 work_item.setStringAttrib('newpath',newpath)

You can then feed this to a rename top (make sure to enable 'copy inputs to outputs' on the python script top):

Tops rename.png

But like I said, be careful. It seems to run immediately, even in generate mode, so if you don't have your attributes right, it will happily rename everything to a null name, effectively deleting it.

Attribute from string expressions

Simple pattern

This is a remarkably powerful node to let you split simple strings easily, and complex strings with powerful (but frustrating) regex hell.

Simple strings are nice and efficient. Say you have a bunch of files you pull in with a filepattern top that look like

beauty_indirect_v03.0035.exr

And you want to create nice attributes out of those various bits. Append a attribute from string, set the string to `@filename`. To do the pattern matching, it's easiest to explain it in 2 steps:

First, copy one of the filenames into the pattern parameter and wrap the bits of the file you want in curly brackets. Skip the bits you don't need, like the '_v' before the actual interesting version number:

{beauty}_{indirect}_v{03}.{0035}.{exr}

Now change the stuff in the curly braces to the attribute names you want:

{pass}_{aov}_v{version}.{frame}.{suffix}

Cook the top, hey presto, its tokenized the string and made attributes for you. It even change the string '0035' into a float attribute of 35, pretty clever. Easier to see in a gif probably:

Tops att frm string simple.gif

Regular expression

Hoo boy. The help just gives a link to the official python regex docs, but thats it. Took a few goes to understand how to use it.

Make sure to bookmark this, keep it handy when trying to construct regex (thanks Jake Rice for the link!) : https://regexr.com/

In this case I'm using a filepattern top to grab butterfly* from $HFS/houdini/pic/butterfly*. The files look like this:

butterfly1.pic
butterfly2.pic
butterfly3.pic
butterfly4.pic
butterfly5.pic
butterfly5Bump.pic
butterfly6.pic
butterfly7.pic 

Notice that there's no separator between 'butterfly' and the number, and that one 'Bump' image in there too? This is more than the simple pattern mode can handle.

Enter regular expressions. Here's the pattern I used:

(\w+)(\d+)(\w*)

Tops regex.png

What does that mean? Similar to the simple pattern, regular brackets identify groups, they'll just be numbered rather than named. The actual tokens themselves:

  • \w+ - find one or more letters
  • \d+ - fine one or more digits
  • \w* - find 0 or more letters

Simple right? No, not really, but powerful. Regular expressions can do a lot more, you can name your groups, do conditional logic, all kinds of text processing tricks. Just keep in mind the classic quote from Jamie Zawinski about regular expressions:

Some people, when confronted with a problem, think "I know, I'll use regular expressions." Now they have two problems.

Regular expression and named outputs

Oh you don't like the index notation? You wanna go all the way and confuse your peers? Ok then.

Put ?P<groupname> at the start of the regex code. So to convert the previous example to create attributes called @prefix, @digits, @suffix:

(?P<prefix>\w+)(?P<digits>\d+)(?P<suffix>\w*)

Ugh.

Regex named.PNG

Update attributes based on value without code

Tops male to M.png

Download hip: File:tops_attrib_male_female_to_M_F.hip

Requires sidefx labs to be installed (its now an option right on the installer, SUPER handy, just do it!). They're doing their best to reduce the need to know python to use Tops. Right now its almost unavoidable, but its a noble goal.

Use the labs 'filter by value' to split male from female, then attrib create to swap the value from 'male' to 'M' and 'female' to 'F', then merge.

Ffmpeg and non sidefx rops

I had a few issues getting ffmpeg to make mp4's from a renderman rop. In the end the fixes were relatively straightfoward.

Pdg tags.JPG

The ffmpeg top needs to know a few things:

  1. what part of the upstream nodes are making images, set with output file tag
  2. what the location of those images are, set with output parm name
  3. that the images are bundled into a single unit of work, using a waitforall top.

Top nodes can tag their output(s), in this case the ffmpeg top expects the images to have a 'file/image' tag. On the fetch top for renderman rop enable 'output file tag' and use the dropdown to select 'file/image'

To know what file name to put in that tag, enable 'output parm name' and set it to 'ri_display_0'. This is the parameter on the ris rop where the image path is set.

To bundle all the frames into a single unit, use a waitforall top.

A last specific thing for our setup, our build of ffmpeg didn't understand the '-apply_trc' option, so I disabled it.

Python script and pdg_input

In a python script top, one of the examples in the drop-down shows you how to get index and name of the workitem:

work_item.index
work_item.name

Great. So the one you'd most likely need, @pdg_input, that'd be obvious right? No.

work_item.inputResultData[0][0]

Wot? Or you can get ugly and mix your @'s with your python code, but you have to wrap it in backticks *and* in quotes. Ugh.

"`@pdg_input`"

Get modified time of workitem as attribute

Modtime.png

Armed with the above stupid trick, we can do this:

 import os
 modtime = os.path.getmtime('`@pdg_input`') 
 work_item.setFloatAttrib("modtime", modtime)

Which is... what? Giant mystery number? It's the number of seconds since the epoch. What? Exactly. Different OS's define 'the start of all time' in different ways, Linux sets it as midnight, Jan 1, 1970. Humans want that in a more readable format, so you'll want to employ the python datetime module:

 import os
 from datetime import datetime
 modtime = os.path.getmtime('`@pdg_input`') 
 work_item.setFloatAttrib("modtime", modtime)
 modtimestring = datetime.fromtimestamp(modtime).strftime('%Y-%m-%d %H:%M:%S')
 work_item.setStringAttrib("modtimestring", modtimestring)

There, a nice giant number that's easy to compare for sorting files oldest/youngest, and a nice string attribute for humans to read.

Tops and imagemagick to make gifs via generic processor

Tops imagemagic create attribs.PNG

Download hip: File:tops_imagmagick.hip

If you try the gif imagmagick tricks on the General page with tops, you'll find nothing works. In the current build (18.5.633 in Oct 2021) you don't get enough control over the command string. It needs the options and the images in a specific order, tops won't let you.

Instead you can use a generic processor top to just call your completely custom command, which I used earlier to run the OSX native video conversion tool.

I tried dropping this command into the node, expecting the waitforall above it to send the correct images to imagemagick:

magick convert -loop 0 -delay 2  "`@pdg_input`" -layers OptimizeTransparency +map $HIP/out.gif

But it no worky. If you look at the log, you can see that @pdg_input just gets expanded to the first image, rather than the full sequence. Boo. JeffLMnt had the fix though; the pdginputvals command will expand that tops attribute into a full string, with each value separated by a space, and surrounded in double quotes. Handy! So this'll work:

magick convert -loop 0 -delay 2  `pdginputvals("",0)` -layers OptimizeTransparency +map $HIP/out.gif

But it's probably tidier to create attributes in a previous node, then have this command use cleaner attribs. Here's what my attribcreate looks like before the generic processor:

Tops imagemagic create attribs.PNG

And the line now becomes this:

magick convert -loop 0 -delay 2  `@images` -layers OptimizeTransparency +map `@output`

As a bonus, you can put @output into the second tab to define the expected outputs. This will mean you can r.click on the node and choose 'delete outputs from disk', and it'll do what you want. Neat.

Farm

Force python scripts to run on the farm

If you have a python script node, even if you have a tractor or deadline scheduler, it will run in your local houdini session by default.

To fix this, turn off 'Evaluate in process'.

Ensure a rop geometry top sim runs on a single blade

You don't want 240 machines all doing their own run up, that's silly. Go to the 'rop fetch' tab, enable 'all frames in one batch', that'll lock it to a single blade and run sequentially.

Selection 117.png

Tractor scheduler stuck

Happens too often less often after some fixes from sidefx. Tricks to unstick it in order of least to most annoying:

  • Make sure there's no active stuck jobs of yours on the farm, delete 'em all and try again
  • R.click on tractor scheduler, 'delete temp directory'
  • Select the tractor scheduler, ctrl-x to cut it, ctrl-v to paste it
  • Reload the hip
  • Restart houdini
  • Quit FX
  • Quit the industry

Rez and tops debugging

Running this in a python script top to see whats going on with rez and environment values:

print('debug info...')
key = 'REZ_RESOLVE'
print(key+'='+os.environ[key])
print ('')
import pdg
has_tractor = str(pdg.types.schedulers.has_tractor)
print('pdg.types.schedulers.has_tractor: ' + has_tractor)
print ('')


Tractor

Pdg tractor edit.gif

Mostly works, the long story can be found below, but here's the summary:

  • Your environment needs access to the python tractor api. If you use rez, make sure to bring in a package for tractor.
  • PDG assumes it'll find $PYTHON set correctly. We didn't have this, but even then I found I couldn't use the regular system python, but had to point it to hython ( $HFS/bin/hython )
  • If your farm is behind a firewall, make sure your IT department chooses 2 ports you can use, and enter those ports into the callback and relay port fields on the tractor scheduler
  • As of 18.0.502 retry support exists on the tractor scheduler, as well as options for better logging.
  • Cooking jobs by default expects to connect to your desktop machine to update information, give you blinky lights and dots. This means that if you close your houdini session, the job will stop working on the farm. Call me old fashioned, but that defeats most of the point of using a farm. If you don't want this, use the 'submit graph as job' option at the top of the tractor scheduler, and it will run independent of your GUI session. Getting these to work reliably was problematic for us, YMMV.

Tops and tractor diary

Moving the diary to TopsTractorDiary.

Todo

Some neat ideas for examples/topics from Jeffy Mathew Philip:

  1. converting between one format to another(like ingesting a kitbash set and spitting out individual alembics, or fbxs or usds) - like building an asset zoo.
  2. example running the hda processor for poly reduction or geo processing
  3. using the invoke top for geo processing(can use the top geometry sop)
  4. more uses of the partitioning tops, eg partitioning a wedged -> substepped simulations to give me the wedges, and then I combine at the end to use the partition by frame to run the correct set of frames