There is much hand-wringing going on in Washington these days about the inability of Republicans and Democrats to compromise even on seemingly unimportant issues. I think it is the inevitable result of long-term trends 100 years in the making.
The movement started in 1913 when the 17th Amendment was added to the Constitution. This is the one that requires senators to be elected by popular vote. Previously, as established by the Founding Fathers, senators had been elected by state legislatures.
Before the 17th Amendment, senators represented states as states. This made the states much more significant players in national politics -- collectively coequal to the national government in our federal system. But once senators became popularly elected, the states lost any real influence in Washington. Senators stopped representing their states as states and simply became super-congressman.
The impact of this fundamental constitutional change was not apparent for a long time, and in many ways we are still seeing the consequences play out. One reason is that senators from the Deep South retained a pre-1913 attitude long afterward. For cultural and historical reasons, the term "states rights" had real meaning in the states of the Confederacy. Unfortunately, the term became widely viewed as a code word for racism and therefore discredited as a valid constitutional principle.
But these same cultural and historical factors also kept the South solidly in the Democratic Party's control, even though it was the national Democratic Party that was leading the charge on racial integration, eroding what was left of states rights, and other policies that were fundamentally at odds with the attitudes of white Southerners. They were an outpost of political conservatism in an increasingly liberal party.
Northern Democrats were embarrassed by their Southern contingent, and Southern Democrats seeking national office -- such as Lyndon Johnson, Jimmy Carter and Bill Clinton -- quickly adopted Northern attitudes to distance themselves from their roots. But national Democrats could not expel the Southerners from their ranks because they had a powerful base in Congress, especially the Senate, and because no Democrat could win the White House without Southern votes.
Because Southern Democrats tended to be re-elected more easily and more often made a lifetime career of serving in Congress, they rose in seniority and eventually controlled many of the most powerful committees. In this position, they were able to block federal intrusions into state affairs. At the same time, their de facto alliance with Republicans on many issues, such as national security, protected Southern Democrats from two-party competition.
All of this began to change in 1974, when Democrats won huge majorities in Congress in the wake of Watergate. Flush with power, insurgent liberals decided they didn't need the Southerners any more, and they began taking away their chairmanships and treating them with the same disrespect usually reserved for Republicans. This led many conservative Southern Democrats to seriously question their place in the party.
At the same time, the Republican Party began to pour resources into the South for the first time and recruit attractive candidates to run against Democrats who hadn't had opponents in decades. Eventually, Republicans were successful in replacing the conservative Democrats, and the South became the new base of the party.
This trend accelerated after Southerner Newt Gingrich, R-Ga., joined the Republican House leadership. He realized that as much as Republicans might be philosophical brothers with conservative Southern Democrats, they had to be defeated if the Republican Party were ever to achieve majority status in Congress. Newt increased the pressure on them in many ways, eventually forcing almost all either to retire or switch parties. It was one of the most brilliant strategic moves in American political history, which paid off in 1994 when Republicans took control of the House for the first time in 40 years.
The most important result of the extinction of the conservative Democrat has been that the two parties have become aligned along ideological lines in a way that wasn't the case for at least 100 years. As the conservative element of the Republican Party was strengthened by the addition of the Southerners, liberal Republicans found themselves increasingly pushed out, completing the ideological solidification of both parties.
As a consequence, ideology and partisanship have become merged together in the 21st century in a way that was not the case in the 20th. Liberals mostly were liberals first and Democrats second, and conservatives were conservatives first and Republicans second. Now, it is much harder to maintain those distinctions. There is tremendous pressure on ideologues of both parties to be partisans first and support the party, even if it means compromising their principles.
This, I believe, is at the root of the current impasse. Purely partisan fights have been suffused with ideological fervor, thus making deals impossible for now.