HTML 4 became the HTML standard in 1997. At the time it was believed that people writing HTML would write to the current standard and would understand that deprecated tags would be removed in the next version.
HTML 5 will become the standard in 2015 or 2016 (probably) and has reintroduced many long obsolete tags previously flagged for complete removal but which people insist on continuing to use instead of the correct replacement tags. It has also introduced a small number of additional tags that are useful for things few people wanted web pages to do 20 years ago when the HTML 4 standard was being developed.
About 95%+ of the web still uses mostly HTML 3.2 with an occasional HTML 4 or 5 tag thrown in. The HTML 4 transitional doctype indicates that the page is still using HTML 3.2 tags and has yet to finish transitioning to strict HTML 4. HTML 5 currently uses a short form of the doctype that is valid for any version of HTML from HTML 2 onward.
All versions of HTML standards are mainly for those creating browsers. The tags those writing web pages should be using has always been a subset of the tags the standards allow because the standard has to allow for pages written a long time ago. No browser has actually implemented the SGML standards for HTML 2+ and all interpret the tags independently of which standard you identify that your page is using via the doctype (which is why HTML 5 can use an HTML 2 doctype and still work).