I had heard of this, but I was fascinated by some pictures I found last night. It is possible to place a small device on an ATM or other machine that takes credit or debit cards and read the card number. This is often supplemented by a camera or pin pad overlay that records your PIN number too. The really sophisticated ones have a cell phone built in that transmit the data via SMS.
Fortunately, this is not rampant, but it is something to watch for. Security measures include looking closely at the ATM or card slot for odd things, the CVV code on the card that cashiers sometimes ask for, and the chips that are embedded in the cards. Something that is more frightening are reports of skimmers placed inside of gas pumps where they cannot be seen. I don't know how you get inside a pump without being noticed, but all gas pumps have tamper-evident stickers on them from your state's bureau of weights and measures that should help you tell whether someone has been inside the pump since it was last inspected.
More information can be found in the links at this Krebs on Security post.