Basically, it's just the editing selected layer(s), as opposed to editing the complete RGB. Say, you want to reduce the brightness of the sky in the photo of a sunrise (too much contrast in the sky, while the details residing below the horizon, are predominantly green&grey). You choose the red layer ("select"), click on RGB again, edit curves so that the contrast is reduced (only the red layer will be affected) and you save it. The "new" photo will retain the dark details in lower part (below the horizon) while the sky will look less bright (less "highlighted")...
If you would edit the RGB (not just a "masked" layer), the reduced brightness would have "eaten up" the details in the dark areas (below the horizon)...