Windows 10 will support a much larger variety of video codecs and container formats, but until it arrives (and you make the decision to upgrade), you can install a shell extension in Windows 7 and 8 to get, at the very least, thumbnail previews in Explorer.
The extension is called Icaros, and it's available for download via MajorGeeks. It supports the most popular container formats, including MKV, MP4 and AVI, as well as M2TS, OGM and even RealMedia's RMVB.
Once installed, configuration is simple -- just select the formats you want to see thumbnails for and activate the extension.
While it won't be amazingly useful for those who use some sort of media centre software to browse their video library, it's handy to have when organising your collection via Windows, which can be easier in most cases.