The latest issue of TapeOp (issue #51) has an interview with one of
the developers of the Apogee units in the "Behind the Gear" section
that addresses this question quite well. It's a very good read. To
summarize the main points are:
1) High sampling rates are basically a waste EXCEPT for the fact that
it improves the aliasing of filters (as Paul noted) and offers better
quality of DSP processing.
2) 24 bit offers excellent dynamic range, but anything beyond that is
silly since it would far exceed the dynamic range of the human ear.
3) an excellent clock (like Big Ben or M-Clock) will improve your
sound quality more than anything.
The section on the unique type of dithering he uses was interesting as well.