Even if you plan your trip to avoid your destination's busy season, there are just some sites that are so popular they will always have a crowd. Take advantage of the natural thinning around lunchtime so at least you can see the sights with fewer people. Image from kevingessner.
Most tourists, and many tour groups, break for lunch between 12PM and 2PM, so if you visit popular sites during that time crowds are likely to be smaller. If the site you're visiting is too large to go through in that two hours, plan to see the most popular part during the lunch break. For example, the Vatican Museum can take more than two hours, so visiting one of the most popular parts, the Sistine Chapel, during lunchtime makes more sense than trying to cram the whole museum in.
You can apply this idea to dinner, too. Most museums will be closed by dinner, but you can visit outdoor sites or sites that are open in the evening (like the Eiffel Tower, Lincoln Memorial and Tokyo Tower) for thinner crowds when most tourists are busy eating dinner.
How to Get the Most Out of Visiting World-Famous Sites [The New York Times]