MySQL Movie Magic

Weta Digital, the company on Peter Jackson films
Heavenly Creatures (1994) nominated for
The Frighteners (1996)
in 1997, Peter Jackston started working on King Kong, but Universal canned it because there were a lot of monster and disaster movies.
Contact (1997) with Robert Zemeckis — the zero-gravity ride was done by Weta Digital
Then they did the Lord of the Rings trilogy (2001-3)
Van Helsing (2004) (just a few bits and pieces)
I, Robot (2004) nominated for best digital effects — technology for armies in Lord of the Rings was used here.
King Kong (2005) and trying to get what Peter Jackson wanted.

(make a chart!)
Size of a movie is based on a shot (camera does not cut away). Visual effects movie typically had 500-1000 shots.

Year Movie Shots Processors
1994 Heavenly Creatures 30 1
1996 The Frighteners 450 2?
1997 Contact 48 60
2001 Fellowship of the Rings 450 384 more processors needed for the massive armies in the prelude, and all the CG creatures
2002 The Two Towers
760 1400 More armies, the Ents, more fantastic creatures including Gollem
2003 Return of the King 1400 3200 1/2 the movie, 90 minutes, was effects!

Kong 2005, made skull island, 1930’s New York was digitized, Peter Jackson gets seasick so all the water was digitally added.

300MB per second of film! how is it archived, after scanning the film & digitizing it? 1 petabyte of data for LOTR and King Kong — 5-6,000 tapes.

Artists get to work on the online copy. 120 Terabytes of storage isn’t enough to store all the data, so it’s copied from the archive to a server. 100G ethernet even to desktops. 700 linux workstations, win and mac boxes around too. 10G ethernet to connect rooms together [sic]. High-density blade servers. After visual effects are rendered, they have to be put back to film, using red, green and blue lasers and a spinning mirror to burn film.

Wow! Pretty neat.

He showed us the shots of the way they built New York, including the almost (or just, I forget) finished Empire State Building. Also, he had many shots of the studio and outdoors with green screens, and it was just fabulous. And how they animated Kong and how it was different from Gollem.

1999 Weta used MySQL 3.22, to be the backend an online recruiting system. Migrated from 3.23 to 4.0 in 2005, and to 5.0 in 2006. 5 production machines
10 replicas (replication)
100 dbs
thousands of tables
millions of rows

MySQL helps with
production management (who’s doing what)
HR System
User database and access control for all the different systems
System monitoring (nagios, internal tools, using MySQL as its backend)
Theater, conference rooms and event booking systems
Polls for employees
Online stores (for employees to buy movie swag)
Internal auction site

Why do they use it?
Simple
Reliable (hardware crashes, but db didn’t. No lost data to date)
Scalable

MySQL at Weta
The Cluster — persistent db connections from webservers are 2/3 of db connections into cluster. 50-100 cxns per second peak, up to 50MB/sec coming out. But they’re not running on high-end hardware, just using the old rendering hardware.
The Monster — one monster db. 40 cols, sparsely used, ENUMs that need to be updated all the time, 20 useless indexes. 750,000 rows, 2/3 are meaningless. No normalization.
The Work Horse — One db with dedicated hardware — the disk monitoring system. 30 file servers, every few hours they need to know updated disk space stats (because so much disk can be used by folks). Computing that stuff takes a lot of CPU. up to 3,000 queries/sec as it compares new data with stored data and updating if necessary.

ShotInfo
Tracks thousands of shots over multiple projects
Tracks all cuts and edit changes
Tracks all the plates and film rolls (so you can find a bit of film you want to recreate/duplicate)
Tracks assignments
Data originates from FileMaker, so normalization isn’t great, field names aren’t consistently named.
One way mirror

ShotSub
Key system
Shot review system
Tracks work in progress
Visual History
How they know where they are in a shot at a given time
35,000 submissions per month for King Kong!

Disk Space Management System
Load balances data
tracks data usage
looks like normal filesystsem — also must be cross-platform
Global Name Space Distributed File System
Transaction based (like a filesystem!)
Millions of allocations, thousands created per day.

Weta’s Future with MySQL:
refactor databases and code
More scalability, more reliability, and less simplicity.
Multi-Master clustering
Federated Database servers
64-bit platforms
Faster hardware

Comments are closed.