Most often, washing your clothes is as simple looking at them (or smelling them), and making a judgement call. However, it's not just about that obvious stuff, Brit Lab takes a look at all the gross stuff your body produces that you can't just use the smell test for.
The obvious stuff here includes washing socks, underwear, and t-shirts. This is all the clothing that's closest to your body's primary sweat glands, shedding skin cells, and skin oils. These should be washed regularly, preferably after each use. For towels, which pick up tons of skin flakes and bacteria, its advised to wash them after three uses. Jeans? Unless they getting sweaty or dirty, you don't really need to wash them at all. The same goes for any other pieces of clothing that are far away from your sweat glands.
Of course, everyone's laundry rules are a little different, but if you're just worried about bacteria build-up, it's best to wash anything that sits close to your genital area, breasts, or armpits regularly.