This is one of those questions where you start out thinking Volkswagen and everybody tries to talk you into a Ferrari.
I have cut parts on a mill that was ready for retirement a generation ago. I have also used the latest state-of-the-art CNC. It's great to have a surface grinder, lathe, plasma cutter . . . But you learn work-arounds or take it outside. Every place I have worked I have had access to their machinery, but it's not as good as having a Bridgeport in the garage.
You really have to fix a budget number and work around it. a CNC only makes sense if you have money to throw away. The CNC is perfect for multiple parts, but in LSR everything is a one-off. It it's really that complicated of a part have someone else do it.
You can get good work done from a used pile of junk, it's just harder. Everything in your price range is going to have backlash, the amount you turn the wheel and nothing happens. You learn to compensate. You also learn the difference between climb cutting and conventional cutting.
I'll chime in with everybody else, tooling, tooling, tooling. You can't have enough collets, clamps, 1-2-3 blocks, you name it. At some point you have to hold the part to machine it, and a lot will fit in a decent vice. If it won't, you chew up a lot of time trying to figure out how you are going to hold the part.
There is no equal to a sharp tool. A sharp tool cuts with very little effort and not much sound. A dull tool isn't cutting, it's wearing through the part with great effort and racket. Coolant is the key to keeping it sharp by pulling the heat out of the part and the cutter.
What you cut is almost as important as what you have already cut. The chips have to leave the area. Sounds simple but you would be surprised how often you see the chips come back around and you end up cutting the part and the chips, causing double the load on the cutter.
The rigidity of the mill determines how much of a cut you can take. The table top mills won't take much of a cut. The more iron the better. The mill, tooling and cutter determines the speeds and feeds you can get away with.
You very rarely need to hog large amounts, and when you do, patience can be a substitute for iron. You also learn to cut the excess off with the chop saw or bandsaw before you start milling.