How do you avoid having sex with your close relatives? Well, not you, specifically, but how is it done generally, or perhaps among mammals in particular?
One obvious way would be to use a part of the genome that seems to evolve rapidly, and that would be able to distinguish between even moderately closely related individuals. For instance, the Major Histocompatibility Complex (MHC) is big, diverse, and changes rapidly. If two potential mates could somehow compare their MHC’s, they could estimate their degree of relatedness, and thus avoid inbreeding.
An article just out in Current Biology examines this question in mice. Noting that the MHC is a commonly invoked likely candidate for kin recognition, Sherborne et al (2007) also note that very little direct evidence indicating exactly what mechanism is used exists. SHerborne et al also notes that there is another highly variable genetic system, the Major Urinary Protein Gene Cluster (MUP).
This is a pretty obscure gene cluster. You won’t even find it in Wikipedia. (Enter “MUP” and you get things like “Melbourne Universityi Press” and Ministarstvo Unutrašnjih Polsova — the Bosnian Ministry of the Interior.)
These researchers designed an experiment to test whether mice use the MHC or the MUP to avoid inbreeding. They used “wild” house mice (Mus musculus domesticus). (Which kind of brings up the degree to which biological scientists get the concept of “wild.” … house mice are never wild. They are commensal. But that is another story.) The point of this was to avoid genetically homogeneous lab mice all of whom have undergone severe bottlenecking in their histories. They independently measured actual genetic similarity as a proxy for relatedness (of course, in real life, relatedness is a proxy for actual genetic similarity … proving that sometimes tautologies are useful). They also let the mice run around in a “natural” enclosure … meaning very large … and do whatever they wanted. They ran this experiment for a few generations.
The result was a reduction in the frequency of successful matings betwen full sibs, but a statistically weak one. There was no evidence for avoidance on the basis of MHC similarity. But, the MUP genetic complex had a “strong and highly significant effect on the likelihood of successful mating … there was no [reduction] when only one MUP haplotype was shared, but there were many fewer matings between mice that shared both [of the] MUP haplotypes … than expected under random mating conditions.”
This experiment was carried out among four populations, and the same results found in all cases.
In the end:
The use of MUP alone was sufficient to explain inbreeding avoidance in this study. Although the deficit in mating between mice of the same MUP type was very strong, there was no evidence that animals used genetic markers other than MUP to improve their level of inbreeding avoidance.
This interesting study challenges the commonly held assumption that the mammals would use odor cues linked to the MHC to avoid inbreeding. This idea was very attractive because, while inbreeding has its special problems, avoiding inbreeding (i.e., out breeding) has the independent benefit of enhancing MHC variability, which confers enhanced immune system capacity on the offspring of such out-matings. However, MUP cues for out breeding result in the same sort of MHC diversification anyway, according to this study.
Furthermore, while the MHC is related to immune system function and also could be (but maybe is not) used for inter-individual recognition, MUP appears to do only one thing: Signal individuality via scent. Nobody knows of any other purpose of MUPs.
Sherborne, Amy L., Michael D. Thom, Steve Paterson, Francine Jury, William E.R. Ollier, Paula Stockley, Robert J. Beynon and Jane L. Hurst. (2007) The Genetic Basis of Inbreeding Avoidance in House Mice. Current Biology In Press. doi:10.1016/j.cub.2007.10.041