Movies into Data

A new pipeline

In previous entries, I discussed the idea of analyzing color data from movies in order to get some insight into how the content of movies is reflected in their color schemes.  I found that movies are relatively easy to distinguish by genre, and that if movies are any guide, the future will be electric blue.  It was all good fun.

But then the data ran out.  Spotmaps ceased to be made, and despite my best efforts to contact the party or parties responsible for their production in order to demand more, no more Spotmaps were forthcoming.  So I had to make my own data.

 

Building a better spotmap

I took the opportunity, in creating new Spotmaps, to attempt to improve the previous formula.  The original idea of a Spotmap was that each cell in a spotmap captured the mean color of an entire frame.  Frames constitute the whole image, and were captured about once per second, so there were on average about 6000 or more separate colors per movie.

One of the problems I noticed with this formulation, however, was that colors tended to be washed-out, rather pastel mixtures of all of the colors in a frame, rather than representing any single color within the frame.  Consider an example here:

A frame from a movie.  The upper left boxed color is the mean coloration of the whole frame.
A frame from a movie. The upper left boxed color is the mean coloration of the whole frame (except the black bars above and below, which are removed prior to any processing).

This is one frame of a movie, and in the upper left is the average color of said frame: but note that despite the tremendous variation of colors in the frame, the “mean color” isn’t actually found anywhere in the picture.  That’s because it represents a weighted combination of all of the colors, instead of any particular color within the frame.  Furthermore, coding the whole frame as a single color doesn’t allow me to explore how color is distributed throughout the screen, that is, the composition of a frame (I’ll come back to this momentarily).

So, when I made my spotmap capturing software, I programmed it to collect 144 different colors per frame, representing 144 regions (16 * 9) laid on the image in a grid.  I call each region a “pseudopixel” (but I’m open to better suggestions), because it is effectively a combination of all of the pixels in a region.  Here’s an image with some of the boxes drawn in (unmodified image is here):

Red lines represent the demarcations between pseudopixels.
Red lines represent the demarcations between pseudopixels.

Ideally, it would be nice to capture every single pixel and analyze all pixels simultaneously, but the math quickly becomes dizzying when you consider how much information that really is (a standard 1920 * 1080 HD movie frame contains a staggering 2 million pixels, each of which has three R, G, B color values).  When I actually collect the data, then, the pseudopixels looks like this:

A pseudo-pixel image.  Note that this is from a 30 pixel square capture, whereas the real algorithm captures rectangles 120*90.
A pseudopixel image. Note that this is from a 30 pixel square capture, whereas the real algorithm captures rectangles 120*90.

Recognize the image?  I’m cheating a little here, because the above image is actually capturing more pseudopixels per frame than I want to analyze, but I include this picture as a proof of principle to show that my scripts work.

 

Testing

I took my new spotmap creating pipeline for a spin on three different movies:  The Dark Knight; Samsara; and Ratatouille.  These were chosen precisely because they are quality representatives of three wholly distinct genres (and also, movies I watched recently): The Dark Knight, a dark and brooding action film; Samsara, a visually-astounding documentary; and Ratatouille, a Pixar-animated comedy.  I can’t claim that these three movies form any sort of robust sample, or that they cover the wide breadth of possible movies, but they are quite diverse nonetheless.  They’ll have to suffice for a test run.

A little background on each movie: The Dark Knight is the best Batman movie ever made, and if you haven’t heard of it, you have probably been trapped underneath Antartic ice for the last 10 years or something.  Ratatouille is one of my favorite Pixar movies, and it’s about a talking rat named Remy who wants to transcend his humble, murine beginnings to become the greatest chef in France.  Finally, Samsara is this amazing documentary about life on Earth.  It has no words, just images, and it is incredibly beautiful.  It was made by the guy who did Koyanisqatsi and Baraka, so if you’re into that sort of thing, and you can stand a movie with no particular narrative, I would highly recommend it.

 

Recapitulating the old

Using the data collected in the algorithm, I can build a spotmap (in this case, of Ratatouille):

Each square represents the color of frame; time goes from left to right..
Each square represents the mean color of a single frame; time goes from left to right..

Unfortunately, the original website didn’t collect data on these particular movies, so I have no point of comparison to prove that my algorithm works as well as theirs.  I’ll pick some movies with original spotmaps to compare to in the future, however.

I can also look at the three core characteristics I examined in my original post: color lightness (formerly intensity), spread, and speed.

Lightness

  • Dark Knight: 0.1159368
  • Samsara: .259
  • Ratatouille: .203

Spread

  • Dark Knight: .28617
  • Samsara: 0.4436807
  • Ratatouille: .3743

Speed

  • Dark Knight: 26.5
  • Samsara : 19.7
  • Ratatouille: 38.9

These scores mostly confirm my intuition.  Summarizing: the Dark Knight is, well, a rather dark and gritty movie, so low color lightness.  On the other hand, it’s an action movie with high color speed, including the single quickest frame transition I’ve ever observed (spoiler alert: it’s an explosion.  It’s about as extreme as any frame transition can be.).  Ratatouille has the highest average color speed, and is relatively bright, as befits a kid’s movie.  Samsara is the slowest of the trio, but shows the greatest color spread (unsurprisingly).  Thus, the new pipeline largely recapitulates the results of the old analysis so far, but there’s more that can be done by sampling multiple colors per frame.

 

New modes of analysis

The simplest new analysis to be done is to compute the color speed on a per-pseudopixel basis, so that’s what I did.  Results are displayed as heatmaps.

 

Heatmap of Samsara's color speed.  Each cell represents a pseudopixel, wherein the color scales with the mean color speed in that pseudopixel.  Hotter colors means that colors change faster in that region of the frame.
Heatmap of Samsara’s color speed. Each cell represents a pseudopixel, wherein the color scales with the mean color speed in that pseudopixel. Hotter colors means that colors change faster in that region of the frame.
heatmap_colordist_Dark_Knight
Same as above, but for the Dark Knight.
Same as above, but for Ratatouille.
Same as above, but for Ratatouille.

For color speed, there’s three distinct patterns for the three movies: Samsara spreads the color speed around in a sort of u-shape at the bottom of the frame; the Dark Knight clusters the color changes at the top of the frame; and Ratatouille keeps it towards the bottom, but with a seemingly more even distribution.  Notably, all of the movies seem to have more of the action (color speed) happening in the center of the frame, as opposed to the corners.  That makes sense.

 

Composition Effect

One last thing I tried examining was whether there’s any evidence for composition, in the visual artistic sense.  Consider this beautiful frame from Samsara:

Still frame from Samsara.  Temples in Bagan, Burma.
Still frame from Samsara. Temples in Bagan, Burma.

Notice how the darkness of the forest gives way to jutting crimson and gold peaks of the temples, which in turn give way to grey-blue and white of the mountains and then the sky.  I would argue that the image is beautiful at least in part because of how the brightness of the image varies with the height, so that one’s eye is drawn from the forest upwards through the temples and to the sky.  To get all philosophical on this pretty picture, I think there’s a deep metaphor here relating to how mankind seeks divinity out of nature, but let’s set that aside for a moment.  This image was obviously composed, that is, set up so that the brightness increased with the y-axis.  Do cinematographers prefer this sort of composition?  Using my pseudopixels, I asked whether color lightness of each pseudopixel tended to correlate with the height along the y-axis for 500 random frames of Ratatouille.  The same idea was also applied to the x-axis, that is, do colors increase in brightness moving left-to-right across an image?  I compared that data to an empirical null, namely a set of completely random pseudopixels drawn from the movie.

Histogram of composition indices.  Red is y-axis composition, green is x-axis composition, and blue is an empirical null distribution.
Histogram of composition indices. Red is y-axis composition, green is x-axis composition, and blue is an empirical null distribution.

We see that there is strong evidence that the cinematographers (or, actually, since it’s Ratatouille, animators) chose to structure their frames so that color brightness either increased or decreased with the y- and/or x-axis.  There are many more extreme values of correlation, both positive and negative, in the real data than in the permuted null.  There’s much more analysis here to be done, but it’s good to note that there is evidence of composition in these movies.

 

Conclusions

Can we draw any grand conclusions about genre and color and motion and composition?  Unfortunately, not yet, but rest assured that as I digitize more movies, such questions will become answerable.  Building a better pipeline for movie digitization ought to pay dividends in the long run, as I will soon cease to be data-limited.  Combined with the many inquiries relating to frame color composition which I can now investigate, I have a solid foundation to launch a more certain quantitative analysis of movie colors.

 

 

______________________________________________________________________________

Have any suggestions for further analyses?  Want to take a look at my code?  Contact me below.

Post a comment

You may use the following HTML:
<a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>