There's no piece of software, be it an application on your computer or a content management system on a server, that you'll be happy with out of the box. Rather than turn away from the configuration "phase" of post-installation, just find yourself a good guide and get tweaking. In the case of WordPess, Digital Inspiration has you covered with an expansive list of pointers.
Amit Agarwal has compiled 30 possible changes you can make to your fresh WordPress install to boost security, improve SEO and make administration easier.
While most of the tips are recommended, there are a few I disagree with. Disabling post revisions, for example, is going to cause you headaches in the future, especially if your blog has multiple authors.
Agarwal's other suggestion of a revision limit however is a good idea, though it's unlikely a few extra text entries in your WordPress database are going to cause speed and / or storage issues.
As for the security tips — you should do most, if not all of them. The important ones require a little bit of PHP, but won't take long to implement and are worth the peace of mind.
Things You Should Do After Installing WordPress [Digital Inspiration]