Decent entry level recorders include the Sony M10 and Tascam DR-2D these are in the 120-220 price range.
External mics, external mics, external mics. Higher quality and better ability to place them where the sound is good.
Other than the "I was there feeling", the chances of recordings with these handhelds having musical value is almost nil.
Start saving and when you have $300 or more, we'll point you in a direction that'll make you and the people you share with happy.