I built some speakers in college, out of necessity... I sold my speakers to buy my woman a ring, and had no tunes.
I bought some drivers from madisound, designed a crossover (not their LEAP design, I used the Dickason book), and built the cabinets out of MDF (unfinished). I tried a few different cabinets... One was pretty good, so I stuck with it.
They were very colored, and something wasn't very coherent about that crossover. My second try was a bit better... But in the end I just settled with those until I could afford a pair of manufacturer-built speakers.
It was a very instructive lesson, tho--I learned alot about how different parts of the speakers interacted to build a good-sounding speaker. And I learned how much effort really goes into a good design!
If you really want to DIY, I'd go with an accepted "good" design. The ProAc Response 2.5 was widely copied and a DIY favorite. I heard a pair of homebuilt 2.5 copies and they weren't the real deal, but close enough for some folks. The problem is once you have all your money (drivers, crossovers, cabinets, wood, hardware) and time factored in I think a pair of used ProAc's starts looking mighty attractive