Millions of Twitter users are actually fraudulent bots, sold to real Twitter users (including many celebrities and media personalities) to inflate their stats and make them look more influential. Last week the New York Times investigated one of the most influential bot sellers and called up their celebrity clients. In the fallout, the Chicago Sun-Times suspended film critic Richard Roeper for a couple of days.
Even if you didn't buy fake followers, some may have followed you. But unless you've been selling your tweets to advertisers or putting your follower count on your résumé, no one cares. While it's against Twitter's rules to buy followers, Twitter hasn't even punished most users who bought thousands of followers. (Why would they? Twitter is desperate to keep celebrities using the site.) They certainly won't punish you for getting a few accidental fake follows.
But if you really want to, you can identify and block all your fake followers at TwitterAudit. The site will give you a free estimate of how many bots are following you. Then for $US5-30 ($6-38, depending on your follower count) they will identify each account, and let you block some or all of them. Note that TwitterAudit might accidentally flag real people as bots. If you block them, they won't be able to see your tweets. (Blocking them doesn't report them to Twitter as bots or anything, so don't worry too much.)
Media-coverage site Poynter points to some more tools for cleaning up your follower list and auditing other users' followers. You narc.