The UN's highest court has cleared Serbia of direct responsibility for genocide during the 1990s Bosnian war.
Bosnians sought a guilty verdict at the court in The Hague
But the International Court of Justice did rule that Belgrade had violated international law by failing to prevent the 1995 massacre at Srebrenica.
Bosnia brought the case and would have sought billions of dollars from Serbia in compensation if successful.
The case is the first of a state being charged with genocide. Individuals have been convicted of genocide in Bosnia.
The Bosnian Muslim leader expressed disappointment at the ruling, which was welcomed both in Serbia and the Bosnian Serb Republic.
At least 100,000 people died in the 1992-1995 war, triggered by the break-up of the former Yugoslavia. Bosnia's Muslims and Croats wanted to cut ties with Belgrade, a move opposed by Bosnian Serbs.
The case, Bosnia and Herzegovina versus Serbia and Montenegro, began a year ago and a panel of judges has been deliberating since hearings ended in May 2006.
UN's highest legal body, resolving disputes between states
Based in the Peace Palace in The Hague, began work in 1946
The court has 15 judges of different nationalities elected to nine-year terms of office
If one state fails to comply with a ruling, the other party can take the issue to the UN Security Council
Bosnia argued that Belgrade incited ethnic hatred, armed Bosnian Serbs and was an active participant in the killings.
Belgrade said the conflict was an internal war between Bosnia's ethnic groups and denied any state role in genocide.
In the ruling, the president of the court, Judge Rosalyn Higgins, said: "The court finds that the acts of genocide at Srebrenica cannot be attributed to the respondent's (Serbia) state organs."
However the court added that the leaders of Serbia failed to comply with its international obligation to prevent the killings and punish hose responsible.
The court also rejected Bosnia's claim for reparations.
"Financial compensation is not the appropriate form of reparation," the ruling said.
The war crimes tribunal in The Hague has already found individuals guilty of genocide in Bosnia and established the Srebrenica massacre as genocide.
Under a 1995 peace accord, Bosnia remained a single state, but power was devolved to a Muslim-Croat federation and a Bosnian Serb Republic.
The BBC's Nicholas Walton in Sarajevo says many Bosnian Muslims were hoping for a clear ruling that Serbia as a state was responsible for pursuing a genocide in Bosnia during the 1990s.
The Bosnian Muslim member of the country's tripartite presidency, Haris Siladzic, told the BBC there was "disappointment" at the outcome.
However he welcomed the fact that the court had "ruled that Serbia and Montenegro had violated the Convention on the Prevention and Punishment of the Crime of Genocide by not preventing or punishing the perpetrators of the genocide".
In the Serb Republic, Krstan Simic, a senior member of the governing ruling Union of Independent Social-Democrats, said he was pleased that the judges had taken "real facts " into account.
In Serbia itself, President Boris Tadic welcomed the judgement and urged parliament to pass a declaration "condemning the crime in Srebrenica without any doubt".
The German presidency of the European Union urged Serbia "to use today's judgment as a further opportunity to distance itself from the crimes committed by the Milosevic regime".
The ruling comes with Serbia still facing challenges linked to the break-up of the former Yugoslavia.
Admission talks with the EU have been stalled over Belgrade's failure to hand over war crimes suspects for trial.