Sunday Afternoon Coding

Hi, Kev here. Last week I described how to load the gridded global temperature data into memory; this week I'll describe how to render it on a map.

First, let's get some structural bits done. I'm going to render one month per frame, which simply means I'll need a variable to hold the month number and another for the year. I'll set them both to the start date. As the contents of these variables will need to be retained between frames, the variables will need to be defined outside the draw() function like so:

int month = 1;
int year = 1880;
void draw()
{}

Inside the draw() function, I increment the month number every frame, and when it gets to greater than 12, I then increment the year and set the month back to 1.

void draw()
{
  println(month + " " + year);
  month++;
  if(month > 12)
  {
    month = 1;
    year++;
    if(year > 2017)
      exit();
  }
}

This should output:

...
7 1880
8 1880
9 1880
10 1880
11 1880
12 1880
1 1881
2 1881
3 1881
...

...in the console. These are our keys for the HashMap.

I then set the size to HD (1920 by 1080) like so:

void setup()
{
  loadData();
  size(1920,1080);
}

So now we are good to go.

I'm now simply going to plot the data in a square grid. First thing's first: I need to get the temperature data given the month and year. So, just before I increment the month, I add this code:

String dataKey = month + " " + year;
int[][] temperatureGrid = data.get(dataKey);

temperatureGrid is a 37 by 73 array, which means we can loop through it like so:

  for(int x = 0; x < 37; x++)
  {
    for(int y = 0; y < 73; y++)
    {
      print( temperatureGrid[x][y] + " ");
    }
    println();
  }

This prints out all the data to the console

DataPrint.gif

Now I'll draw dots for each temperature on the screen in a grid. For the colours I use a function called lerpColor(). "lerp" is a contraction of "linear interpolation". It merely means, given two numbers, find me a number in between. So, to find a number three quarters of the way between 10 and 20, you'd give a lerp function: 10, 20, 0.75 and it'd spit out 17.5. For lerpColor() you give it two colours and a fraction, and it'll spit out a colour in between those colours.

Colours are represented as three numbers between 0 and 255. Each of those numbers are the level of red, green, and blue respectively. So color(255,0,0) is red, color(0,0,255) is blue, and color(255,255,255) is white.

I'm colouring the above normal temperatures with red and the below normal temperatures with blue. Temperatures near normal are white.

I replaced print( temperatureGrid[x][y] + " "); above with the following:

float temperature = temperatureGrid[x][y]/100.0;
color tempColour;
if(temperature >= 0)
  tempColour = lerpColor(color(255,255,255), color(255,0,0), temperature);
else
  tempColour = lerpColor(color(255,255,255), color(0,0,255), -temperature);

stroke(tempColour);
point(x,y);

This creates the following:

temp.gif

Making pretty good progress with only a few lines of code! Evidently, I have got my x and y coordinates around the wrong way. Let's fix that and make them pixels larger:

for(int x = 0; x < 73; x++)
{
  for(int y = 0; y < 37; y++)
  {
    float temperature = temperatureGrid[y][x]/100.0;
    color tempColour;
    if(temperature >= 0)
      tempColour = lerpColor(color(255,255,255), color(255,0,0), temperature);
    else
      tempColour = lerpColor(color(255,255,255), color(0,0,255), -temperature);
    
    noStroke();
    fill(tempColour);
    int x10 = x * 10;
    int y10 = y * 10;
    rect(x10, y10, x10 + 10, y10 + 10); 
  }
}

Giving us:

temp.gif

Lovely. If you squint, you can make out the continents! Hang about---why are they flat blue? Ah! missing values are marked as -9999, so therefore, they are blue as that's very much below normal. I'll mark those as grey color(150,150,150). As the colour code is getting cumbersome, I'll extract it out as a function like so:

color getColour(float temperature)
{
  if (temperature < -90)
    return color(150,150,150);
  color tempColour;
  if(temperature >= 0)
    tempColour = lerpColor(color(255,255,255), color(255,0,0), temperature);
  else
    tempColour = lerpColor(color(255,255,255), color(0,0,255), -temperature);
  return tempColour;
}

 

temp.gif

Ok, that looks pretty good. But squares greatly distort the area of each segment of the earth's surface. What we need is a proper sphere projecting function of which there is no end of choice.

My favourite is the fabulously named Kavrayskiy VII projection:

Kavraiskiy_VII_projection_SW.jpg

The projection is defined as:

projection.PNG

DON'T PANIC

This is really very simple. It just looks complicated because of all the Greek symbols and what-not. λ (lambda) is the longitude, and φ (phi) is the latitude in radians. I'll stick with radians, as computers by default use radians, so no need to convert.

Right, from the get-go I can see that the y coordinate is merely the latitude. Happy days. The x coordinate is calculated as follows in code:

float x = (3.0 * lng / 2.0) * sqrt( (1.0 / 3.0) - sq(lat / PI));

The sqrt() and sq() functions are square root and square respectively.

Processing has the type PVector, which allows us to handle an x,y coordinate pair, which means I can create a function like so:

PVector getKavrayskiyVII(float lat, float lng)
{
  float x = (3.0 * lng / 2.0) * sqrt( (1.0 / 3.0) - sq(lat / PI));
  return new PVector(x, lat);
}

Which returns the x and y coordinates of a given longitude and latitude.

The problem I have now is that I'm looping through the x and y coordinates when I now need to loop through the longitude and latitude angles. The data is for 5° by 5° segments of the surface or π/36 radians. So my loops now need to look like:

float step = PI/36.0;
float halfPi = PI/2.0;
int latCount = 0;
int lngCount = 0;

for(float lng = -PI; lng < PI; lng+=step)
{
  for(float lat = halfPi; lat > -halfPi; lat-=step)
  {
    float temperature = temperatureGrid[latCount][lngCount]/100.0;
    latCount++;
  }
  latCount = 0;
  lngCount++;
}

As I am working out four points surrounding a point on a sphere, I get the top-left, top-right, bottom-left, and bottom-right points like so:

float halfStep = step/2.0;
PVector tl = getKavrayskiyVII(lat + halfStep, lng - halfStep);
PVector tr = getKavrayskiyVII(lat + halfStep, lng + halfStep);
PVector bl = getKavrayskiyVII(lat - halfStep, lng - halfStep);
PVector br = getKavrayskiyVII(lat - halfStep, lng + halfStep);

To draw a four sided irregular polygon I'm using the quad() function where s scales it to a nice size:

float s = 170;
quad
(
  tl.x * s, -tl.y * s, 
  tr.x * s, -tr.y * s, 
  br.x * s, -br.y * s, 
  bl.x * s, -bl.y * s
);

Which generates:

 

temp.gif

How about that, eh?

The complete code can be found on github. Fork it and have fun!

Next week, I'll overlay it on the map of the earth from above.

Monday Night Paint-Splashing

Ice Shelf, watercolor&nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; &nbsp; Copyright © Marlo Garnsworthy

Ice Shelf, watercolor                                                                      Copyright © Marlo Garnsworthy

I love Monday nights.

Monday night is creation night. It’s hastily-shoveled-in-leftovers-in-the-studio night or earbuds-in-laptop-on-local-pub night. This week, it’s also the first night of Polar Week! 

Kev and I have been furiously preparing to launch Pixel Movers & Makers and working on our first animation about Antarctica's Pine Island Glacier. 

But for the last couple of months, I’ve also been working feverishly on something I’ve called the Sea Ice Sketch Project. I’ve been painting watercolor (or watercolour, depending on where you are) sketches of sea ice… and ice shelves and polar ice in general. (Some of the pieces have become banners for our website.)

My passion for polar ice began, it’s true, with sea ice. But it’s spread quicker than katabatic winds can steal your gloves to all polar ice. And so now, I’ve renamed it the Polar Ice Sketch Project. (Let it be duly noted that one of my brilliant friends suggested I call it the Icy Poles project, but I wasn’t sure anyone but fellow Aussies would get the reference.)

I’ve been trying to film my painting process. My first effort was going well until disaster struck…

Not enough painter's tape on my desk lamp. But I'm working it out.

Here, I put the finishing touches on the ice shelf in the painting at the top of this post. 

I'll be using similar techinues — and some others — as we continue our first animation project. I look forward to sharing!

I always paint to music. Here's what was on my Spotify during the above video (and which YouTube made me record over, alas): Youme & Meyou by Einstürzende Neubauten.  

Sunday afternoon coding

This weekend, I made a quick sketch of an animation using global temperature anomaly data. My goal is to graph the proportion of areas of the earth that were cooler and warmer than average over time.

After a quick search, I found the data I was looking for on NOAA's website. It is basically a large text file with the average monthly temperature for every 5° by 5° section of the globe.

It was the most beautiful text file of data I have ever seen:

ncdc-trimmed_10fps_480x256.gif

Each line contains 72 temperature values for each of the 5° sections; there are 32 rows, one for each 5°s of latitude. Between each section is the month and year. -9999 is used to signal no data available. The temperatures are in degrees Celsius converted to integers (the counting numbers: 1, 2, 3, etc.) by multiplying by 100 — presumably to avoid those pesky decimal points. 

So, what now?

Well, for quick sketch animations, I use processing.org, which is a java-based animation framework, which means I get to play with all the java data structures! So, I gotta read all that lovely data into some of them. But what and how?

Ok, I figure I'll be rendering the temperatures onto a map of somesort, which means I'll need to read the data so I can get all the months' temperatures all at once. Basically, given a year and month, I need the temperatures.

Uh oh! I'm going to start to get a tad techie below, if that floats your boat then please continue, if you're a keen learner then I'll assume you've had a go at processing's excellent tutorials otherwise you can skip to the end to see the pretty teaser animation!

Well, the temperatures are in a two-dimensional grid. So, that pretty much calls for a two-dimensional array, which is a way of storing stuff in memory, just like a grid.

int[][] temperature;

The square brackets there indicate, in a cack-handed kind of way, that temperature has two dimensions. So far so good.

But I'm going to need 1,656 of them! One for each month of each year between 1880 and 2017. So, why not add a third dimension? Indeed, I could and just number the months 0 - 1,655, and it'd work just fine. But that's not very interesting. I need a challenge — I mean, it's Sunday afternoon coding for goodness sake!

What I chose to use was a HashMap for no other reason than I get to use generics! A hash map is a data structure that behaves kinda like an array, but the index can be anything you like and it doesn't need to be sequential. (In this case, I can actually just use integers, and they'd be sequential — I'm a sucker for punishment.)

I defined the HashMap like so:

HashMap<String, int[][]> data;

Much more interesting than a dull old multidimensional array! So what's going on here? What's with the angle brackets?

This type of data structure is known as a generic. Which means it can use any type of data; strings (basically, sequences of characters, AKA words and sentences), integers, objects (computery type objects, not those things on your table over there), etc. In this case I'm using a String and a two-dimensional array of integers, which I've listed inside the angle brackets. The first item is what is known as the "key" and the second is the "value."

I figured I could make the key something like "{month} {year}" eg: "7 1956" for July 1956, which would return the temperature grid for that month and year like so:

int[][] temperatureGrid = data.get("7 1956");

So, how do you get the data in there?

Well, first I can't really use the pretty data file, as it is much easier to have a single character to separate the data values, such as a tab. As you can see here in this close up there are multiple space characters between the values (the little grey dots).

A close up of the temperature data showing the multiple space characters delimiting the values.

A close up of the temperature data showing the multiple space characters delimiting the values.

So, open up your favourite text editor *cough* Visual Studio Code *cough* and replace all the multiple spaces with a single tab. I did this by using what are called regular expressions, basically wildcards on steroids. Good text editors will have this as an option in find dialogs.

Visual Studio Code's find dialog showing the regular expression option highlighted in blue.

Visual Studio Code's find dialog showing the regular expression option highlighted in blue.

In the above image I set the pattern to find as \s+. The \s means match any space character and the + means find at least one of the previous pattern. So this will match one, two, or twenty space characters. (You can literally type the space character followed by the + but this looks dumb in a blog post as it looks like I'm talking about nothing).

In the replace box the \t represents the tab character. This will replace all the spaces matched by \s+ with a single tab.

I saved the data in a file called ncdc-merged-sfc-mntp-tab-delimited.dat to the same folder as my Processing sketch where I can easily load the file into one large string array called lines:

String[] lines = loadStrings("ncdc-merged-sfc-mntp-tab-delimited.dat");

The string array lines now contains all the data from the dat file. I just need to loop through each line and split it into useful stuff.

for (String line : lines) 
{    
  String[] values = split(line, '\t');
}

This is called a "for loop", it's read like so: for each of the string line in the string array lines execute the code inside the curly brackets.

The variable line ends up with a single line from the data file which we had previously modified to have single tab (\t) between each value. The command split(line, '\t') uses those tabs to create an array of strings which are the temperature values we want.

You can now see why I went to the trouble of removing the multiple spaces. If I hadn't and split on the spaces above, I'd get many empty values and that wouldn't be nice. But now I just split on a single tab and get the right amount of values.

So, the idea is that I loop through the file, scoop up the temperature data, and anytime I come across a line with the month and year in it, store the temperature data away in the HashMap keyed on the date. Simple.

What follows is the complete code to load the data; this is called via

loadData();

in the setUp method in the processing sketch.

HashMap<String, int[][]> data = new HashMap<String, int[][]>();

void loadData()
{
  String[] lines = loadStrings("ncdc-merged-sfc-mntp-tab-delimited.dat");
  String theKey = "ignore";
  int latCount = 0;
  int[][] tempGrid = new int[37][73];
  for (String line : lines) 
  {    
    String[] values = split(line, '\t');
    if(values.length == 2)  // This detects the line with only the month and year
    {
      data.put(theKey, tempGrid);
      tempGrid = new int[37][73];
      theKey = values[0] + " " + values[1];
      latCount = 0;
      continue;
    }
    
    for(int lngCount = 1; lngCount <= 72; lngCount++)
    {
      tempGrid[latCount][lngCount - 1] = int(values[lngCount]);
    }
    latCount++;
  }
  
}

Be sure to come back next week, when I'll start actually using this data and do some pretty stuff like so:

out.gif

It also involves maths and Kavrayskiy VII projections, mmm good!

Pixel Movers & Makers Launch!

 

Hey, it’s Marlo and Kev here!

We’re quite excited to launch Pixel Movers & Makers in time for International Polar Week! We’ve been hard at work for a few months now, making pixels, moving pixels, and crafting this new grand adventure.

In February, Marlo attended the APECS workshop on Antarctic Hydrology and Ice Shelf Stability at Lamont-Doherty Earth Observatory, where we were thrilled to present a poster about effective polar science communication and our first sample animation — and equally pleased by the response our efforts received.

APECS POSTER sm.jpg

We look forward to sharing our process and journey to creating informational videos about our changing planet that entertain, inspire, and compel.

Follow us on Twitter:

Pixel Movers & Makers --- @PixelMnM

Kev, Pixel Mover --- @KevPluck

Marlo, Pixel Maker --- @MarloWordyBird

                                                                                                                ~ Marlo & Kev