Atoms formed precisely when the average energy density (i.e. temperature) dropped below the ionization energy of hydrogen at ~ 4000 K.* At that moment, everywhere in the universe, there was a mass condensation of hydrogen atoms. Before that everything was an ionized plasma (of disassociated electrons and protons), which is quite opaque, and light was trapped to a life of not getting very far (though it certainly existed, unlike what others here have incorrectly said). After all that plasma collapsed into hydrogen atoms things became much more transparent and everywhere in the universe light, with a specific spectrum associated with the black-body (i.e. thermal radiation) emission of a body at the temperature of hydrogen formation), was set free. This light still exists today, everywhere and in every direction. We call it the Cosmic Microwave Background Radiation or CMB. This event was called Recombination (or Photon Decoupling, and yes I know that's sliiigghhhttlllyy different) and happened about 400,000 years after the Big Bang.
Of course I'm assuming by 'matter' you mean atoms. Before atom formation their were hadrons (Hadron epoch) and before that quarks (Quark epoch).
*A realistic model is a little more complex than this, don't take this value super seriously.