Adding MEL to render layers

The builtin render layers make Chapman's plugin redundant for the most part. Its builtin, its point-n-click, there's no MEL to write, great (hence Chapman has stopped developing his plugin). However there's always going to be tricky little 'gotchas' where point-n-click doesn't work, and currently the builtin system doesn't allow you to do MEL overrides, which sucks. An example of this is fur (grrr, maya fur, hate hate hate). You can't set layer overrides on any of the texture-mapped fur attributes. You can force it via setattrs in mel, but there's no method to bind this to render layers.

I faced this exact problem, and so decided to try and find a way around this limitation.

First thing was to have a container for the mel statements, similar to Chapman's editor. Scriptnodes are close enough, and are easily edited from the expression editor. Start the expression editor, choose 'select filter->by script node name', give the script a name (I name mine layerChangeScript), set the mode to 'before', enter some mel in the bottom window, click 'create'. Set the mode to 'on demand'. Mel-wise, I find the current layer name, then use a switch statement that does whatever it needs per case. Here's a quick example of setting 2 fur attributes and some material colours:

string $currentLayer = `editRenderLayerGlobals -query -currentRenderLayer`;
switch ($currentLayer) {
  case "whiteFur":
     setAttr lambert2.color 1 0 0;
     setAttr blinn1.color 0 0 1;
     setAttr -type "string" FurDescription1.BaseColorMap[0] "whiteFurMap.tga";
     setAttr -type "string" FurDescription1.TipColorMap[0] "whiteFurMap.tga";
     break;
  default:
     setAttr lambert2.color 1 1 1;
     setAttr blinn1.color 0.5 0.5 0.5;
     setAttr -type "string" FurDescription1.BaseColorMap[0] "BaseFurCol.tga";
     setAttr -type "string" FurDescription1.TipColorMap[0] "TipFurCol.tga";
     break;
}

Like Chapman's plugin, you'll have to state each change at least twice; once for the layer, and once for its default state.

Next was to get this to automatically run whenever a layer is changed. A scriptjob is designed for this, letting you bind a command to an event like scene open, time changed, or in this case, changing render layers. Run the following in your mel prompt:

scriptJob -killWithScene -e "renderLayerManagerChange" "eval(`getAttr layerChangeScript.before`)"

This returns a job id if you need to kill it later.

All good right? Nope. Scriptjobs only work for interactive sessions, and only for as long as maya is running. As soon as you exit maya or batch render, your scriptjob is lost. To fix the rendering side, add the eval command of the above line to your 'pre render layer MEL' in the render globals:

eval(`getAttr layerChangeScript.before`)

And to make sure the script is active when you restart maya and reload the scene, you create a second scriptjob that creates the first scriptjob on scene-load. I find this easiest to do from the scriptnode editor. Create a new scriptnode with the same scriptjob line from earlier, set its execute mode to 'open/close'. Save your scene, and you're good to go.

-- MattEstela - 01 Feb 2009