I'm one of those "If it ain't broke, don't fix it" people. So long as you have "unused" RAM, you can increase the buffer size. You'll find that your system is more responsive when it's at its smallest, yet still working well. I think I on'y use a buffer of 1024 (not sure if that's samples or bytes or something else, didn't check) for most stuff.
-Matt