It's one thing to install WordPress on your server, it's another to get it running smoothly -- and responsively. There are some common, almost mandatory tweaks -- grabbing a caching plug-in, minifying scripts and creating sprite atlases -- but these supposedly fire-and-forget techniques can hurt performance, if not done properly.
One point I found interesting was the use of gzip to serve compressed content. With WordPress, you can install myriad plug-ins that will activate this functionality via PHP, but it's better to use your server's implementation, if it's available. Apache allows this via regular expressions in htconfig files, though many HTTP server packages provide ways to configure the module.
Pandjarov also looks at the fickle nature of CSS sprites and Apple Safari, as well as issues that can arise with reverse proxies.
WordPress Performance Improvements That Can Go Wrong [Smashing Magazine]