The Digest protocol specification dates from 1999, which was a time when SSL was still viewed as a tool too expensive to be employed generically. The RFC begins with this paragraph, which gives the context:
"HTTP/1.0", includes the specification for a Basic Access
Authentication scheme. This scheme is not considered to be a secure
method of user authentication (unless used in conjunction with some
external secure system such as SSL 5), as the user name and
password are passed over the network as cleartext.
This is the only reference to SSL in the whole document. This makes things clear: Digest was designed to try to cope with the glaring issue of sending passwords in the clear, as Basic does when not used with SSL. We know, however, that not using SSL is a big problem anyway, because attackers have advanced a bit since last century: they no longer content themselves with passive eavesdropping, they actually hijack connections and implement man-in-the-middle attacks. No amount of Digest will save you in that case.
Note that one possible advantage of Digest over Basic is that you do not reveal the password even to the server itself, in case that server is hostile -- by which I mean that you are talking to a fake server, controlled by the attacker. Digest will not protect against a MitM, so in that situation you are already doomed, but only locally doomed: the attacker can see your data, and alter your requests, but he does not learn your password so he will not be able to come back later by himself. Several points make this advantage very small:
Even though Digest with a fake server does not reveal the raw password to the attacker, it still give enough to run a dictionary attack, and a very efficient one since it is only a couple of MD5 hashes (nothing even remotely comparable to bcrypt).
The inherent MitM attack is serious enough to mandate proper data integrity checks, which means SSL (with full server certificate validation, please !). If MitM is defeated, then the advantage of Digest over Basic evaporates as dew in the morning sun.
On the other hand, as you have noted, using Digest implies that the server must store the passwords themselves (possibly encrypted, but in a reversible way), and that is a big drawback of Digest authentication.
For the best of all worlds, use TLS with SRP which is SSL/TLS with a kick-ass password-based key exchange, where:
- the server needs not store the password, only a verification token (a kind of hash);
- there is no certificate at all;
- the authentication is mutual and relative to the password;
- even an attacker impersonating the server, the client, or both, cannot learn enough data to run an offline dictionary attack.
The only, "minor" trouble is that TLS+SRP is not widely supported (yet). GnuTLS can do it. We can hope for more extensive support in the future. Meanwhile, use Basic authentication within SSL, do not forget to thoroughly validate the server's certificate, and use bcrypt to store a password hash server-side.