A layer mask is an 8-bit grayscale channel that has white or black as its background color. White areas on a layer mask permit pixels to be seen, black areas on layer masks hide pixels and gray areas partially hide pixels with the drakest greys hiding most of the image and the lightest grays allowing most of the image to show through. The best part about layer masks is that you can access them from both the Layers and Channels palettes.
The easiest way to think of masks is to think of fading an image out. Lets say you have an image that is at 100% opacity but you want to start to fade the opacity out halfway through the picture to make it fade out. You can't do that with opacities but you can do it by using layer masks.
To create a layer mask Select your layer to which you want to apply the layer mask to, then click on the little icon that looks like a circle inside a square in the layers window (it's at the bottom of the layer's window) then your image will look like it has 2 thumbnails. The one on the left is your image and the one on the right is the layer mask. When using layer masks the only colors that you can use are grayscale colors. You can put in colors too, but they will just be converted to grayscale on the layer mask, so it's best to stick with just grayscale colors.
How to work with a layer mask Once you have created your layer mask you can use a variety of different tools and filters, but if you wish to fade the opacity out then you would want to use the gradient tool (G) (when using the gradient tool, you will want to only use it inside the image, using it outside will not produce much of an effect or no effect at all)
*NOTE* Make sure that you have selected the correct thumbnail on the layer mask before doing anything. An example is if you wanted to invert your picture you would have to click on the left hand side thumbnail to make the picture invert, if you were still on the right hand side thumbnail and you inverted then the layer mask would invert and whatever fading you had would reverse, because the layer mask would shift from black to white and from white to black, therefore the part with black that hid the layer (opacity fading) would become white and would show the layer (increase the opacity) and vice versa.
If you wish to temporarily hide the layer mask then hold shift and click on the right hand side thumbnail in the layers window. You will know it is hidden because you will see a RED "X" through the thumbnail and your image on your canvas will be completely solid. You can also right-click on the layer and select "Disable Layer Mask" and to show it again you would either SHIFT-click on the right hand side thumbnail or right-click on the layer and select "Enable Layer Mask".
Once you have finished modifying your mask and you want to make it go back to a normal layer with the newly added mask, click on the little trash can icon in the layers window, and then click "APPLY".
If you decide that you don't want a layer mask anymore and you are going to do something completely different, click on the little trash can icon in the layers window but click "DISCARD".
If you decided that your layer mask wasn't quite in the right spot then you can move the layer mask by itself by clicking on the little chain link in-between the 2 thumbnails on the layer and then select which ever thumbnail you wish to move and drag it. Then to lock the layer and the layer mask back together you would click in-between the 2 thumbnails on the layer to reveal the chain link.
Using quick masks and alpha channels Basically there is only 1 difference between these 2 techniques. Quick masks don't get saved automatically whereas alpha channels do. These 2 techniques are just a way of using and/or making abnormal shaped selection marquees.
Alpha Channels To make an Alpha Channel you would use: lasoo tool, marquee tools and even the magic wand tool to select certain areas of your image, then goto SELECT|SAVE SELECTIION, and then the selection will be saved in the channels window ready to be called for later use.
It will appear to be only black and white. The black part is the outside of the abnormal shaped selection and the white part is the part that would have a marquee around it. To recall the saved marquee (or selection) simply goto the layers window and goto SELECT|LOAD SELECTION then in the drop down menu choose which alpha channel you wish to load. Then hit ok and your abnormal shaped marquee will appear.
Remember that the only way to move the marquee around is to have the marquee tool selected and your pointer must be inside the marquee, otherwise if you are on the arrow tool and you try to move it, you will actually cut out part of the image and move it somewhere else. As a personal preference I like to move the marquee to the spot I want and then I copy and paste in to a new layer, for future movement.
Quick Masks To make a quick mask you would press the letter "Q" and notice how the selected layer's bar goes from being black to being grayish, that is how you can tell if you are in quick mask mode.
Once you are in quick mask mode you can only use grayscale colors again, with the same principles: "White areas on a layer mask permit pixels to be seen, black areas on layer masks hide pixels and gray areas partially hide pixels with the drakest greys hiding most of the image and the lightest grays allowing most of the image to show through."
The spots that you paint are the spots you don't want to have selected. Think of it as using "masking"-tape to stop paint from going over it. If you wanted to create an abnormal selection marquee around one part of the image, then press "Q" to go into quick mask mode then select BLACK as your foreground color and paint over the area you don't want to turn into a selection. It will look like you are painting a slightly translucent red over top of your image - this is normal. Whereever there is red - that is the part of the image that will not be part of the selection when you exit quick mask mode. If you accidentally put more "RED" where you didn't want it to go, then select white as your foreground color and paint over the spots that aren't supposed to be there. Painting with white is basically like using the eraser to erase the red parts, and painting with black makes the red spots. Once you have made your mask (the RED part) press "Q" again and you will exit quick mask mode, and any areas that weren't red will be selected.
Are you confused yet, I sure am!!! LOL
I will recap the quick mask mode and how to simplify things to take less time. Lets says you have a picture of an animal in a forest. You want to make an abnoraml selection marquee only around that animal.
You can do 2 different things to accomplish this. 1 - The long way: enter quick mask mode then paint all around the animal but not on the animal, then when you exit quick mask only the animal will be selected. 2 - The short way: enter quick mask mode then paint over just the animal, then press Ctrl+I and everywhere that wasn't red will become red and vice versa. Then exit quick mask mode and only animal is selected.
As you become more confident with quick mask and Photoshop, you will be able to learn how to harness the effects of inverting a selection save you time.
Alternatively there is one other way to do this: 3 - Enter quick mask mode, paint over just the animal, then exit quick mask mode, and all parts around the animal will be selected, now you want the animal to be selected so with the Marquee still showing goto: SELECT|INVERSE (Shift+Ctrl+I) and then just the animal will be selected.
If you want you can choose to goto SELECT|SAVE SELECTION and save the selection as an Alpha Channel to be called for later use.