Using Masks and Channels For Object Extraction

Being a pretty diverse tool, Photoshop suggests many ways to accomplish each task. And each has its pros and cons. One of the more powerful tools in photoshop is masks. It is probably also one of the more complex tools. We are going to tackle making today, and hopefully making them a bit less complex.

What are masks and why do we need them?

Masking is the feature that enables selecting a certain object and cutting it, ally a color to it, apply different effects to different areas and apply color looks to selected areas and so on…

Strictly speaking, a mask is nothing more than a black and white (and gray) pixel map. The mask marks black areas as transparent; white areas retain the same levels they has without the mask; and every gray in between is semi-transparent. Let’s have another basic look. (don’t worry though, we are going to pick up the pace quite quickly)

Let’s start with a fresh layer and fill it with orange, then we add a second layer, which is filled with blue, and apply a mask on the top layer. By default, a white mask is applied and has no effect over this top layer. Everything is blue.

But if you create a gradient mask on the top layer (make sure to select the white mask icon first), the image will look like this:

What just happened?

The left side (where the mask is black) is revealing the underlying layer, while the right side, where the mask is white has no effect. (and of course, everything in between)

It is quite common to use masks in photoshop. It is also quite common to use different tool to create those masks. The general rule is to use what works best for you.

Here are some example for when you actually need masks

Masking for Dodging and burning: Here is a photo and its matching Dodge & Burn Mask (we dodge and burn using a curves layer and masks, but more about this in a separate article). You can see the areas that have been brightened and dimmed with the mask.

Masking for selective adjustments: Here is another example using the same photo, where I masked the photo to select only the yellow dress and adjust its color. Here is the mask for the dress:

Note: I know which areas are going to be edited. I create masks for each area and apply it to a folder. Now I can put any adjustment layers in this folder and it will only apply to the relevant locations. This way, when I refine the mask, it will be updated for all the adjustment layers inside the folder.

Masking for selecting an object: You can also use a mask to cut a person from a background, here is how it looks like:

Such a mask is the basis for a composite like this one:

Both the dancer and the backdrop were “borrowed” from our image manipulation store.

It does not really matter how you create the mask, as long as it looks good when you’ve made it. There are, however, several tools and methods that suit one job better than others.

For example, the path tool is perfect for masking out a cube, but try and mask hairs or a dress with it and it will only take a few seconds to get you 100% frustrated.

One exception to this ‘horses for courses’ rule is selection via channels. Channel selection and masking works well for almost any task and except from some exclusions (which I can’t think about right now) can probably be used as a sole tool for masking. This is why I am going to take some time to explain channel masking.

But first a bit channels 101:

If we move in RGB mode, we can see that the layers window now becomes the channel window and displays four channels:

Each of these three channels is actually a mask and represents the proportion of each color in the photo. If you apply pure red, green and blue masked with their respective channels, you will regain the complete image again

If you don’t trust me on this one, you can reverse the channel effect like so:

• create three layers, each with a primary color
• apply the relevant channel as a mask to this layer
• set the blending more for each layer as liner (add).
• add a black background layer.

TADA! here is your original photo again.

You can also edit any individual channel: copy the channel (using right-click -> channel duplicate) to create a copy of the channel. Now you cna edit this copy with any of of the tools you are used to like the brash toll and so on. The problem is that any edits you do to the original image would not be reflected in this duplicated channel. There is a trick though:

If you apply an adjustment layer to a photo, it will obviously have an impact on what “goes into” each channel. This is usually not a desired effect, unless, of course, you use it to hack your masks.

Let’s try this, I want to mask create a mask of all the skin tones in the photo so I can manipulate them with no effect on the rest of the photo. This is where adjustment layers and channels can work wonderfully together. I will add a few adjustment layers with the goal of making the skin tone (red) as bright as possible and the rest as dark as possible.

I used these three adjustment layers to obtain the wanted result. This, of course, is different from photo to photo.

After applying those adjustment layers, my channel output changed significantly:

If you look at the red channel, it is quite ripe for masking, the skin is in a really nice contrast to the background (and everything else).

Use this Chanel as  “base” for further work (right click -> duplicate channel). On this new channel, you will work with the brush tool. As for the temporary adjustment layers, feel free to delete those. They are no longer needed.

Tweaking Masks

As you recall, everything that is masked white, is being affected by changes to the layer. So the jewelry must be repainted with a black brush (we only wanted skin, remember?) . Don’t be afraid though, this work is non-destructive and can easily be brought back and reworked again.

Applying Masks

Once you have your channel “well selected” here is what you need to do:

• cmd+click (or ctrl+click) on the layer thumbnail (this creates a selection)
• go back to the levels view (out of channel view)
• Create a folder
• create a mask (it should already be filled properly because you have the selection from the channel)

In this folder, you can adjust the skin separate from everything else. You can, of course, use the very same mask only inverted if you want to manipulate everything but the skin.

There you have it. With this technique, you can create even complex masks quite rapidly. Remember that yellow dress above? It needed two separate masks: one for the dress and one for the skin. Now who wants to do that with the path tool? It is five or six days off right there.

Luminosity masking

The last channel, the RGB channel is the combination of all three channels. Therefore it represents the brightness in the image. By using this channel, you can refine your masks further, here is how it’s done:

For Shadows:

• cmd-click on the RGB channel (created a selection)
• cmd + shift-click on a layer (reduces the selection)
• cmd + shift-click on the layer (yes, twice!)
• Create new channel
• Fill Area with white
• Remove selection
• Invert the new channel

For Highlights:

• cmd-click on the RGB channel (created a selection)
• cmd + ALT + Shift-clicking on the icon in the new channel
• cmd + ALT + Shift-clicking on the icon (yes, again)
• Create new channel
• fill new channel with white (yes, white)

For Mid-tones:

Based on these two masks you can now create a mask only containing the mid-tones. The base is, again, the RGB channel, just without the shadows and without highlights.

• cmd-clicking the icon in front of the RGB channel
• Create new channel (“mid-tones”)
• Fill the channel with a white fill (first luminance)
• cmd-click on the shadow channel you created earlier
• fill in the selection on the “mid-tones” channel with black
• cmd-click on the highlight channel you created previously
• fill in the selection on the “mid-tones” channel with black

This is essentially subtracting both the highlights and the shadows from the mask, leaving only the mid-tones.

With these three new luminance channels you can now adjust tint in the shadows and the highlights. Natalia Taffarel showed this technique beautifully in her workshop – especially in connection with color adjustments and pallets. This opens up many opportunities for quick and efficient retouching.

There are no shortcuts, of course, for this to become fast, you have no choice but to practice and practice (and practice…)