Hrm. I started by only collecting one character. That allowed me to buy repaints and other madness and still stay pretty cheap.
Then I branched out to a few other characters. I look for the older ones on Ebay and have gotten some great deals, especially for unpopular lines like, oh Cybertron or Machine Wars. (The, uh, rightly unpopular Machine Wars, I might add).
At the moment, having like a full time job and stuff, I can afford more but I go to the store with these priorities:
1) do they have any of the main core characters I collect (the Seekers, Barricade, etc)? If yes, buy that. If no, move on to question 2.
2) do they have new Decepticon figures (yeah I collect cons--we're a little Autobot thin here at chez Hellkitty)?
3) Does the figure look good? (If you saw Universe Onslaught, you'd know why that was an immediate buy!) I'm actually really picky so if the figure has like a goofy face in bot mode, and it's not one of my Question 1 figures, I'll pass!
4) IS this a repaint? If so, unless it's one of my Question 1 characters, I skip it. (Sorry, 8000 repaints of Lockdown).
5) Is it a vehicle I like? I collect jets and helicopters more than any other kind. Yeah. Don't judge me. So I'm more likely to pass on a regular car, no matter how accurate (unless it's Barricade), and save my money.
6) Know your toy market!! Where I live, it takes FOREVER for toys to get out here: We JUST got TFA Arcee!! But I know that some stores a) charge way more than others (looking at YOU Kmart) and some other stores stuff just...sits on the shelves forever. Knowing that, I can pass and rethink and think about buying a figure for a bit and it won't disappear.
7) Set a price limit or toy limit before going in, UNLESS you know exactly what you want (in which case, that's your price limit). So I can say, I'll buy *one* toy this week. That means I am going to be REALLY picky about which one I get. I find that in most cases, if I buy one toy, I actually appreciate it MORE than if I buy a whole bunch at once--the get sort of lost in the shuffle. The new lines won't disappear overnight, and I can wait and give each new toy its due. Buying one toy a week (or if it's like...robot heroes or something two packages, right?) still gives you four new to your collection in a month! And if the new toy market is slow, and I'm scouring ebay, same rule: one toy per week. It makes Sunday afternoon combing through ebay listings really fun!

Buy brick and mortar when you can!! Shipping costs are NOT love! While my must have toys do get the BBTS markup and shipping, most others can wait for The Hunt through the toystore.
9) Garage sales/goodwill, etc. I am NOT a kitbasher by any stretch. But I've found quite a few, say, mostly intact toys for super cheap. In fact, I once put together an intact Thundercracker from two partials right there in the goodwill bin. BINGO. It takes patience and luck, but you can find amazing stuff!
HK, hahah it almost sounds like I know what I'm talking about!