It turns out that, despite the noise, it's perfectly fine to yank up on your handbrake without pushing the button. Here's why. At some point you've probably been one of these two people: the person that loudly yanks the handbrake up without pushing the button (to your passenger's horror), or the horrified passenger wondering why the driver hates their car so much. In this video from the Engineering Explained YouTube Channel, Jason Fenske explains how your handbrake works and why the button is really only for releasing the brake.
Basically, it's a simple ratchet system. As you pull up on the handbrake, a latch moves over several teeth (the noise you hear) until you stop and it gets caught and can't go down without the button being pressed. As Fenske explains, it's designed to work that way and it's the same way a socket wrench works -- just on a larger, much louder scale. So, if you don't mind the noise, pull the handbrake without pushing the button as much as you like. Just don't ever try to force the handbrake down without pushing the button.