Money is just a tool, so you shouldn't necessarily feel bad about splurging on a $4 latte. On the other hand, it's easy to use this as a justification to spend money, and that can undermine your goals.
If you want to build wealth, you have to keep more money than you spend. Saving money is one way to go about it, but earning more money will yield bigger results. At the same time, you don't want to use this advice as an excuse to spend.
I fell into this trap recently. My spending has been higher than usual, so I took a look at my budget and noticed a bunch of small purchases that added up. It was mostly crap I didn't need or even want that badly. With most of these purchases, I remembered thinking, "I'm tired of worrying about money. It's a tool and I'm going to spend it." So I did, and that's fine, but I could've used that money on stuff I care about a little more. In short, I have the freedom to spend, but that doesn't mean I should spend. I like the way Physician on FIRE puts it:
Freedom to spend doesn't mean you should or will spend all those additional dollars. It means you've got enough cash to do so without putting your financial future at risk. If you... strive to achieve a net worth that qualifies you not only as independent but also financially free, you must decide how you will use those additional dollars.
In so many words, they explain there's a balance between saving so much that you don't enjoy your financial freedom and spending so much that you lose your financial freedom.
In other words, money is meant to be used, but you also want to make sure you have enough of it to use. So it's important to revisit your habits every now and then, especially when your financial philosophies have changed, just to make sure your habits are still in alignment with your goals. For more on this topic, head to the full post at the link below.
Financial Independence versus Financial Freedom [Physician on FIRE]