Arrays, Numbers, and Colors

For the array img we created above, you can display it as a grayscale image using ImageView. But if you happen to be following along in Juno or IJulia, you might notice that img does not display as an image: instead, it prints as an array of numbers as shown above. Arrays of "plain numbers" are not displayed graphically, because they might represent something numerical (e.g., a matrix used for linear algebra) rather than an image. To indicate that this is worthy of graphical display, convert the element type to a color chosen from the Colors package:

Here we used Gray to indicate that this array should be interpreted as a grayscale image. (Note that the Images package re-exports Colors, so you can alternatively say using Images.)

Under the hood, what is Gray doing? It’s informative to see the "raw" object, displayed as text:

(Users of Juno or the Julia command-line REPL interface will see this representation immediately.)

You can see this is a 2×2 array of Gray{Float64} objects. You might be curious how these Gray objects are represented. In the command-line REPL, it looks like this (the same command works with IJulia):

dump shows the "internal" representation of an object. You can see that Gray is a type (technically, an immutable struct) with a single field val; for Gray{Float64}, val is a 64-bit floating point number. Using val directly is not recommended: you can extract the Float64 value with the accessor functions real or gray (the reason for the latter name will be clearer when we discuss RGB colors).

What kind of overhead do these objects incur?

The answer is "none": they don’t take up any memory of their own, nor do they typically require any additional processing time. The Gray "wrapper" is just an interpretation of the values, one that helps clarify that this should be displayed as a grayscale image. Indeed, img and imgg compare as equal:

There’s more to say on this topic, but we’ll wait until we discuss Conversions vs. views.

Gray is not the only color in the universe:

Let’s look at imgc as text (shown here in the REPL):

Here we see one of the primary differences between Julia’s approach to images and that of several other popular frameworks: imgc does not have a dimension of the array devoted to the "color channel." Instead, every element of the array corresponds to a complete pixel’s worth of information. Often this simplifies the logic of many algorithms, sometimes allowing a single implementation to work for both color and grayscale images.

You can extract the individual color channels using their field names (r, g, and b), but as you’ll see in a moment, a more universal approach is to use accessor functions:

Julia’s Colors package allows the same color to be represented in several different ways, and this can facilitate interaction with other tools. For example, certain C libraries permit or prefer the order of the color channels to be different:

or even to pack the red, green, and blue colors—​together with a dummy "alpha" (transparency) channel—​into a single 32-bit integer:

From first (the first two hex-digits after the "0x") to last (the final two hex-digits), the order of the channels here is alpha, red, green, blue:

These values are close to the channels of c, but have been rounded off—​each channel is encoded with only 8 bits, so some approximation of the exact floating-point value is unavoidable.

c24 does not have an r field, but we can still use red to extract the red channel:

This may look fairly strange at first, so let’s unpack this carefully. Notice first that the "floating-point" portion of this number matches (to within the precision of the rounding) the value of red(c). The N0f8 means "Normalized with 8 fractional bits, with 0 bits left for representing values higher than 1." This is a fixed-point number--rather like floating-point numbers, except that the decimal does not "float". Internally, these are represented in terms of the 8-bit unsigned integer UInt8

(Note that N0f8 is an abbreviation; the full typename is Normed{UInt8, 8}.) N0f8 interprets this 8-bit integer as a value lying between 0 and 1, with 0 corresponding to 0x00 and 1 corresponding to 0xff. This interpretation affects how the number is used for arithmetic and conversion to and from other values. Stated another way, r behaves as

for essentially all purposes (but see A note on arithmetic overflow).

This has a very important consequence: in many other image frameworks, the "meaning" of an image depends on how it is stored, but in Julia the meaning can be assigned independently of storage representation. For example, in a different language/framework, the following sequence:

might produce the following images:

The one on the right looks white because floating-point types are interpreted on a 0-to-1 colorscale (and all of the entries in img happen to be 1 or higher), whereas uint8 is interpreted on a 0-to-255 colorscale. Unfortunately, two arrays that are numerically identical have very different meanings as images.

Many frameworks offer convenience functions for converting images from one representation to another, but this can be a source of bugs if we go to compare images: in most number systems we would agree that 255 != 1.0, and this fact means that you sometimes need to be quite careful when converting from one representation to another. Conversely, using these Julia packages there is no discrepancy in "meaning" between the encoding of images represented as floating point or 8-bit (or 16-bit) fixed-point numbers: 0 always means "black" and 1 or something greater than 1 always means "white" or "saturated."

Now, this doesn’t prevent you from constructing pixels with values out of this range:

Notice that the first two yellows look identical, because both the red and green color channels are 1 or higher and consequently are saturated.

However, you should be aware that for integer inputs, the default is to use the N0f8 element type, and this type cannot represent values outside the range from 0 to 1:

The error message here reminds you how to resolve a common mistake, trying to construct red as RGB(255, 0, 0). In Julia, that should always be RGB(1, 0, 0).

16-bit images can be expressed in terms of the N0f16 type. Let’s compare the maximum values (typemax) and smallest-difference (eps) representable with N0f8 and N0f16:

You can see that this type also has a maximum value of 1, but is higher precision, with the gap between adjacent numbers being much smaller.

Many cameras (particularly, scientific cameras) now return 16-bit values. However, some cameras do not provide a full 16 bits worth of information; for example, the camera might be 12-bit and return values between 0x0000 and 0x0fff. As an N0f16, the latter displays as nearly black:

Since the camera is saturated, this is quite misleading—​it should instead display as white.

This again illustrates one of the fundamental problems about assuming that the representation (a 16-bit integer) also describes the meaning of the number. In Julia, we decouple these by providing many different fixed-point number types. In this case, the natural way to interpret these values is by using a fixed-point number with 12 fractional bits; this leaves 4 bits that we can use to represent values bigger than 1, so the number type is called N4f12:

You can see that the maximum value achievable by an N4f12 is approximately 16 = 2^4.

Using this N4f12 interpretation of the 16 bits, the color displays correctly as white:

and acts like 1 for all arithmetic purposes. Even though the raw representation as 0x0fff is the same, we can endow the number with appropriate meaning through its type.

Sometimes, being able to construct a color values outside 0 to 1 is useful. For example, if you want to compute the average color in an image, the natural approach is to first sum all the pixels and then divide by the total number of pixels. At an intermediate stage, the sum will typically result in a color that is well beyond saturation.

It’s important to note that arithmetic with N0f8 numbers, like arithmetic with UInt8, overflows:

Consequently, if you’re accumulating values, it’s advisable to accumulate them in an appropriate floating-point type, such as Float32, Gray{Float64}, or RGB{Float32}.