GeneralUtilties

From cgwiki

Apart from Houdini, what are the most important things to a 3d artist? Animated gifs and a working bash shell of course, regardless of platform.

Gifs

Creating animated gifs with licecap and gifcam

This is awesome and free for windows:

http://blog.bahraniapps.com/?page_id=21

For OSX/Windows/Linux (via wine), licecap is also pretty good.

http://www.cockos.com/licecap/

ffmpeg to generate gifs

Good guide to how to avoid dithering here, which involves running ffmpeg twice; once to generate a palette, the second to generate a gif using that palette.

http://blog.pkh.me/p/21-high-quality-gif-with-ffmpeg.html

Here's the important bits from that site in 2 lines:

ffmpeg  -i invideo.mp4 -vf "palettegen" -y out_palette.png
ffmpeg  -i invideo.mp4 -i out_palette.png -lavfi "paletteuse" -y out.gif


The following examples don't do the palette trick, but show off how baroque you can get with ffmpeg to do all sorts of resizing and manipulation. This one shows how to crop:

ffmpeg -i "pyro.%04d.jpg" -lavfi "fps=50,crop=in_w:in_h-85:0:0,scale=400:-1:flags=lanczos" -y out.gif
  • crop will use default width (in_h), and default height minus 85 pixels ( in_h-85), calculated from the top left corner (0,0).
  • the scale command here will scale horizontally to 400 pixels, and keep the correct aspect ratio in y (because the y-axis is told to scale to -1, which is interpreted as 'keep the aspect ratio correct)


This one drops every 2nd frame:

ffmpeg -i "pyro.%04d.jpg" -lavfi "select=mod(n\,2),setpts=N/(25*TB),crop=in_w:in_h-85:0:0,scale=300:-1:flags=lanczos" -y out.gif
  • select=mod(n\,2) -- every second frame, n is current frame, the \ is there to escape the comma
  • setpts=N/(25*TB) -- magic command I don't quite understand, but I know that it stops it holding frames. Ie, if the previous command was mod(n,10), it would get every 10 frames, but without the setpts call each of those would be held for 10 frames, rather than only playing for 1 frame.

ffmpeg to generate super small gifs

For maximum pidgin victory (where there's an 8k limit):

 ffmpeg -i input.gif -lavfi "fps=15,scale=20:-1:flags=lanczos" -y output.gif
  • fps is fps, obviously
  • scale=20:-1 the horizontal width is 20, the vertical will resize to suit.


Futzing with the fps and width can scrape most gifs under 8k.

Imagemagick/convert to make gifs

To my horror, I realised that my sweet 50fps mograph computationl art tumblr porn gifs were actually running at 24fps. Seems ffmpeg can't go any higher than that, despite specifying the framerate.

Had a go with convert (part of imagemagick) instead, and it lets you do 50fps in all its cpu melting glory. This command takes a sequence, resizes to 256x256 as its loading, then does a post-crop that lops off the lower 40 pixels. The delay of 2 means 2 miliseconds, ie 50fps.

convert -loop 0 -delay 2 flipbook/hexagon.*.jpg[256x256] -crop +0-40 hex2.gif

Super thorough notes on convert here:

http://www.imagemagick.org/script/command-line-processing.php#geometry

Here's another one with a few more tricks:

convert -loop 0 -delay 4 flipbook/cubesmarch.*.jpg[256x210+80+80] +repage -level 0,60% -layers OptimizeTransparency +map cubesmarch.gif
  • -delay 4 4ms per frame, so 25fps
  • [256x210+80+80] define a crop window of 256x210, starting 80 from the left and 80 from the top
  • +repage actually do a crop (without this, it just masks out the rest of the image, which isn't very helpful)
  • -level 0,60% a levels command, map the blackpoint to 0, and white point to 60%
  • -layers OptimizeTransparency use transparency to try and get some savings. This brought a 1.8mb gif that was mostly white wireframes on black down to 1.4mb.
  • +map generate a single optimised colour palette for the full gif, otherwise it tries to generate one per frame, which costs in file size

Video conversion

ffmpeg to generate mp4 from image sequence

The basics, old skool style.

 ffmpeg -i "shot150_v06.%04d.jpg" -qscale 2 -r 10 -b 10M test.mp4

Here's a better way. I understand newer builds use libx264 by default now for mp4's, but I explicitly call it out of habit anyway. Assumes for 4 digit pad jpegs, no prefix on name:

 ffmpeg -f image2 -r 24 -i folder/%04d.jpg -vcodec libx264 output.mp4

If the sequence doesn't start at frame 1, you need to specify that:

ffmpeg -f image2 -start_number 1000 -i myimages.%04d.jpg -vcodec libx264 out.mp4

ffmpeg batch convert many .mov to .mp4 with bash

probably more elegant ways, but this'll do in a pinch:

 for i in *.mov; do ffmpeg -i "$i" -vcodec libx264 "${i%.mov}.mp4"; done;

via http://stackoverflow.com/questions/5784661/how-do-you-convert-an-entire-directory-with-ffmpeg

Or if you have a specific list of files in various locations, can put the files in a text file with a 'file' word prefix:

# this is a comment
file '/path/to/file1'
file '/path/to/file2'
file '/path/to/file3'

And concat them all together:

ffmpeg -f concat -safe 0 -i mylist.txt -c copy output

mp4s have to have dimensions that are powers-of-twoish, this added filter rounds to the nearest power-of-two for otherwise tricky quicktimes:

for i in *.mov; do ffmpeg -i "$i" -vf "scale=trunc(iw/2)*2:trunc(ih/2)*2" -vcodec libx264 "${i%.avi}.mp4"; done;

and yet more options... disable audio export with -an, copy the video directly with -vcodec copy, and set a start and end with -ss HH:MM:SS -to HH:MM:SS.

ffmpeg -i very_long_movie.mp4 -ss 00:22:12 -to 00:23:50 -vcodec copy -an shortclip.mp4

and a handy trick when transferring big files and get a decent ETA on the command line:

rsync --progress /path/to/sourcefile /path/to/destination

h265 experiments and gearvr

Wip!

Man the gearvr is temperamental. After much fiddling and reading of The Internet, this works well:

ffmpeg -i in_4k_360.mp4 -vf "scale=3080:2160" -vcodec libx265 out_360_bt.mp4

The results are remarkable; as h264 a 4min 4k video is about 400mb, the h265 is 40mb, and looks way better. Now to play with the quality settings a little more. Its very slow to encode though, but the results feel worth it. Interestingly, I found occasionally that cross-encoding from one mp4 to a h265 mp4 can break it (feels like metadata from the original video leaks into the new one, and breaks stuff). In that case, encoding clean from a jpg sequence makes it happy again. Must look into an easier way to scrub any bad metadata from the video, see if that keeps the gearvr happy.

In a recent update, h264 videos won't play on the oculus 360 video app if they don't have an audio track (h265 is unaffected, strange). Here's how to add a null aac audio track onto an existing h264 video:

ffmpeg -f lavfi -i anullsrc=channel_layout=stereo:sample_rate=48000 -i invideo_tb.mp4 -shortest -c:v copy -c:a aac outvideo_tb.mp4

Note that top/bottom stereo requires the _tb suffx in the name, that's how oculus video knows to unpack it into stereo.

To do quick clips for testing specificy the start timecode with -ss, and the duration with -t. Ie, this starts 30 seconds in, and runs for 10 seconds:

ffmpeg -i input.mp4 -ss 00:00:30.0 -c copy -t 00:00:10.0 output.mp4

Or if lazy, can skip timecode and just use seconds. This starts at the start, runs for 5 seconds:

ffmpeg -i input.mp4 -ss 0 -c copy -t 5 output.mp4

ffmpeg top bottom stereo pano to mono

To cut a stereo top/bottom video back into a mono 720p clip for avid and such:

ffmpeg -i in_stereo_tb.mp4 -filter:v "crop=in_w:in_h/2:0:0,scale=1280x720" -vcodec libx264 out_mono_720p.mp4

that does odd things to the pixel aspect ratio in player that understand it , adding the 'aspect' flag forces the pixels back to square:

ffmpeg -i in_stereop_tb.mp4 -aspect 2048:1024 -filter:v "crop=in_w:in_h/2:0:0,scale=2048x1024" -vcodec libx264 out_mono_2k.mp4

make ffmpeg less messy

add

-hide_banner

to the arguments you give to ffmpeg, it stops all that super verbose output ffmpeg likes to vomit every time it runs.

Shells

Getting a shell under windows

Git bash is quick and simple, not as bloaty as cygwin, comes with all the major things you need (less vim cat curl git clone etc...)

http://msysgit.github.io/

Get a decent terminal by replacing cmd.exe with ConEmu

Bash is bash, and it runs within cmd.exe by default, the windows DOS shell. Use it for more than 30 seconds, you'll find cmd.exe is horrible. Can't resize it easily, copy paste is weird, ugh. ConEmu is a free cmd.exe replacement, much nicer. Tabs, resizable, better copy/paste etc.

https://conemu.github.io/

Instructions for getting a 'conemu git bash here' context menu on folders can be found here:

http://superuser.com/questions/454380/git-bash-here-in-conemu

My shell muscle memory always kicks in when I try to copy paste, on linux most shells allow you to double left-click to copy a word, triple left-click copies a line, middle click pastes. You can do this in conemu, in the prefs (win-alt-p), mark/copy, set the text selection mode to 'always'.

Python and git bash

Just install regular python, and append its path to your .bashrc:

PATH=$PATH:/c/python27
export PATH

The best bash prompt ever

Don't question this, it just is. current directory in green, newline, prompt. Stick this in your .bashrc:

export PS1="\[\e[32m\]\w\[\e[m\]\\n\$ "

Looks like this (I pixellated the project path, but shows that the path is nice and easy to see and copy)

Prompt.png

I never thought to look before, but now there's loads of interactive bash prompt builders online. Eg:

Make djv work with git bash

Again, just add the path, make an alias (I use rv cos old habits die hard)

PATH=$PATH:/c/python27:"/C/Program Files/djv-0.8.3-pre2_winxp-x64/bin"
export PATH
alias rv=djv_view.exe

There's still problems with calling it, you can't just do 'rv .' (it doesn't launch anything), and definitely not 'rv *' (it launches a separate djv for each image!). I suspect the * is interpreted by bash, so it just launches everything, while using hashes goes straight to djv. This works:

rv myimages.####.exr

Get a seqls/lseq/lss equivalent with pyseq

Pyseq is a python module that handles sequence listing, comes with a callable 'lss' command. Download it with git, install it, copy lss to your ~/bin folder

git clone git://github.com/rsgalloway/pyseq.git pyseq
cd pyseq/
ls
python setup.py install
mkdir ~/bin
cp lss ~/bin

You get this after all that hard work:

$ lss
  1 .DS_Store
  1 Thumbs.db
  1 _tmpuntitled.png
150 shot150_v03.%04d.jpg 1-150
  2 shot150_v%02d 6-7
  1 tmp

Other

Find a word in lots of files of specific type

grep --include=\*.{glsl,txt} -rnw '/path/to/somewhere/' -e "pattern"

Via http://stackoverflow.com/questions/16956810/how-to-find-all-files-containing-specific-text-on-linux

Rsync

rsync -havz source /path/to/destination
  • h human readable
  • a archive, recurse into folders
  • v verbose
  • z use compression
  • n dry run, handy to test what'll happen without actually doing the sync

Jquery

I have my reasons.

Bookmarklet to embed jquery on the current page, so you can call it from the chrome interactive console:

javascript:if(!window.jQuery||confirm('Overwrite\x20current\x20version?\x20v'+jQuery.fn.jquery))(function(d,s){s=d.createElement('script');s.src='https://ajax.googleapis.com/ajax/libs/jquery/1.8/jquery.js';(d.head||d.documentElement).appendChild(s)})(document);


Good quick intro: http://jqfundamentals.com/chapter/jquery-basics

$( '#header' ); // select the element with an ID of 'header'
$( 'li' );      // select all list items on the page
$( 'ul li' );   // select list items that are in unordered lists
$( '.person' );  // select all elements with a class of 'person'


Futzing with GWT tables in jQuery:

var thumbs = $('.tableThumbsPanel')
thumbs.children;

Hardware

MSI GS63VR 7RG

https://www.msi.com/Laptop/GS63VR-7RG-Stealth-Pro.html#hero-overview

Haven't found any useful reviews on this laptop, had it for about a week, figured I'd blat some thoughts down.

Requirements

I needed a laptop to run an Oculus Rift VR headset. When researching I could break the requirements into needs, wants, and unimportant:

Need to have:

  • GPU powerful enough to drive a Rift (meaning Nvidia 960 minimum, anything extra a bonus)
  • 3 regular USB ports (one for the rift, and 2 for the rift cameras)
  • Native HDMI port (displayport adapters have caused issues)


Nice to have:

  • 4 or more USB ports (add a mouse, game controller, 3rd camera for 360 tracking... native ports are better, usb hubs have caused issues)
  • Ethernet port (downloading vr apps, builds of unreal, houdini etc on a regular basis is a chore over wifi)
  • Quiet as possible (yes I know that's a crazy ask when talking about gaming laptops)
  • 15 inch (13 is too small, 17 is too big)
  • Not look like a stupid gaming laptop


And things that are NOT important:

  • 4k screen, touchscreen
  • Long battery life
  • Be super crazy bleeding cutting edge performance, as long as its within the current bound of 'vr capable' that's fine
  • Build quality, keyboard quality, trackpad quality etc (its mainly gonna be used for launching vr apps, so its all secondary, but nice if its at least 'ok')

Max-Q

I started looking in early June 2017, and had accepted that I wasn't going to be able to get all those things. Turns out my timing was pretty fortunate, as nvidia just announced a reference platform called 'Max-Q'. To my surprise it aligned almost perfectly with my shopping list; to make a thin, quiet, high performance laptop capable of gaming (and thus capable of VR). Asus, MSI, HP all announced they'd be shipping stuff 'soon', so I waited a little, and sure enough, out they came. The Asus was the first available, but pushed too far into 'stupid gaming laptop', so I was reluctant to get it. The HP Omen range was delayed (and also too gamey for my tastes), but the MSI looked ideal.

Design

The design is surprisingly understated (apart from their awful faux Ferrari dragon badge), and build quality pretty good. It sits between the 2011 macbook pro (before they went to the unibody design), the modern macbooks, and a lenovo thinkpad. My reference point for everything I didn't want is the Alienware 17 I had got about 6 months earlier; huge, cheap looking, heavy, loud. Even looks and noise aside, the main problem with the Alienware and the Rift was insufficient ports; even running a fairly up to date external usb hub, it would drop its connection randomly, incredibly frustrating.

The MSI is a perfectly reasonable size (15.6 inches), and about half the weight and thickness of the Alienware. I can carry it in a backpack without thinking about it (could never do that with the Alienware), and when not gaming or VR-ing is a sensible quiet machine.

VR

Regarding its primary purpose, 'can it do VR?', it does that flawlessly. The model I'm running has a Nvidia 1070 in the Max-Q config; that means its about 10% slower than a desktop model, but that substantially lowers the cooling and power requirements, hence it can squeezed into this form factor. Having handled a few high-end desktop Nvidia cards, its remarkable how small and thin this laptop is. Re performance, it's fine. Robo Recall, Lone Echo, Unreal in VR edit mode, all run smoothly and happily.

Noise

Noise wise, its pretty good. It's not as quiet as a macbook or a surface pro in idle; those 2 for all intents and purposes are silent. The MSI always has a fan running, but its very quiet in idle mode; quieter than most desktops. The airconditioning in most offices would be louder than the MSI when just typing and browsing. There's a certain 'growl' to the fan noise, not objectionable in any way, but again having used silent machines for so long, and knowing that other companies have tried to replace the 'whirrrrrrr' of old cooling fans with a less objectionable white noise 'shhhhhh', I notice it, but don't mind it.

When the GPU requires cooling during a VR or game, the fans kick up into 'noticable'. Think like your car air-con on 2, but less low frequency. Annoying to your partner if they were trying to watch tv, they'd have to turn the sound up to compensate, but they wouldn't demand you leave the room. If you're wearing headphones (or in the case of the Rift, you're using the built in headphones), they easily drown out the sound of the fan. Compare that to the Alienware, which sounded like a hair dryer, loud enough that I'd miss quiet audio cues with the Rift headphones, and took to building a little book fort around the laptop to try and muffle the sound. It got so tiring (combined with the USB hub dropouts) that I actively avoided using it, which isn't great. You might've guessed I'm quite prissy when it comes to system noise, so I really appreciate how pleasant the MSI is to use.

Nice things

  • You have the option to run a standard 1920x1080, non touchscreen display. Saves a few bucks if you don't need a 4k touchscreen (I don't)
  • Ethernet port, so I don't need to run a usb-ethernet dongle, or suffer through wifi when getting huge oculus downloads
  • Keyboard has a number pad and is nice n clicky
  • Touchpad is probably the best I've used on a PC. Macbooks have got that perfect, I always found its surprising that it took PC companies so long to match it. The surface pro 4 with its keyboard+trackpad cover came close, but was still a little buggy, the MSI has no issues. 2 finger scroll, light tap vs click to select, all that stuff finally works as it should
  • Very reliable with just closing and opening the lid to sleep/wake, again Macbooks mostly get this right, PC laptops mostly don't.
  • Display can fold flat to the desk, handy to get it out of the way of the rift cameras
  • Build quality for the most part feels great, nice brushed metal design, fairly minimal flex, and the underside is covered in a soft touch felt, mmm.

Issues

The issues I've had are fairly minor:

  • Trying to de-l33t the keyboard was an amusing ride. The default backlight mode is pure red, I wanted just white. Installing the keyboard drivers pushed it into full rainbow pride mode, the software interface was the typical gaming laptop brushed steel and non intuitive wierdness. After randomly stabbing I found how to create a new preset, pushed the colours to pure white, instead I get a sort of light purple, with occasional spots of green; its like a pastel take on the Joker. It also starts pure red when woken from sleep or a hard reboot, then pops to lilac once the drivers kick in. Ah gamer laptops.
  • Pc laptops can't get past their sticker fixation; a fun few hours removing intel stickers, nvidia stickers, l33t gamer tech stickers, then cleaning up all the icky residue.
  • When in battery or quiet mode, occasional system stutters; most apparent when playing video. It'll play fine for 20 seconds, then do a Max Headroom glitch for half a second. Similarly when typing at speed on battery power the keyboard will similarly get stuck and glitch out, when on full power the problems mostly go away. I'm sure a driver update will fix.
  • Just like the old macbooks, the metal around the ports is super thin, like the metal bit on a pen to hook it on a pocket that no-one uses. And just like those metal bits on pens, they've already warped a little after only a week. Utterly cosmetic and trivial, but each time I see it I think 'tsk'.
  • The shiny 'msi' logo on the bezel beneath the display is super shiny. So shiny that it casts a specular highlight onto the base of the laptop. When typing at speed, I see a distracting warpy reflection of my fingers dancing away. Tempting to cover it with some black tape.
  • The power button is on the side, right near the front right corner. Right where I keep holding the machine steady to unplug/replug cables, don't realise I'm pressing it, and power down the machine. I'm sure with time I'll learn to stop pushing that button by accident, but grrrr.
  • That red badge. Jeez.

Summary

All in all, very happy. I spoke to a few people before getting this setup, most warned against the idea of 'a laptop for VR development'. All laptops that claimed to be VR ready before the Max-Q design arrived tended to be filled with compromise or issues. The G63VR does everything I need it to do, in a design that doesn't make me cringe, or require me to wear earplugs. I'd recommend it to anyone in a similar situation to me.

Oh, and yes, Houdini runs great on it.  :)