OUP user menu

Chemical Communication in Rodents: from Pheromones to Individual Recognition

Robert E. Johnston
DOI: http://dx.doi.org/10.1644/BLe-010 1141-1162 First published online: 21 November 2003


In many species of rodents, sense of smell is the most important source of information about the social and nonsocial world. I selectively review the literature on chemical communication in this group, primarily as studied in the laboratory. I start with a discussion of the term “pheromone,” the types of chemical signals, and the mechanisms underlying responses to chemical signals. I discuss the chemical complexity of odors in rodents and the significance of multiple sources of odors. Efforts to chemically identify signals are reviewed briefly; some progress, especially with domestic house mice (Mus musculus domesticus), has been made, but new collaborations between chemists and biologists are needed. Three areas of current research are reviewed in detail: the role of the major histocompatibility complex in the production of body odors and the role of these odors in mate choice, the use of odors in kin recognition, and the functions of scent overmarking.

Key words
  • chemical communication
  • hamster
  • house mouse
  • kin recognition
  • major histocompatibility complex
  • mate choice
  • MHC
  • pheromones
  • scent marking
  • vole

Currently, we are on the leading edge of a revolution in our understanding of the chemical senses. This revolution is coming about because of advances in molecular biology, including discovery of genes that code for receptor proteins and several different 2nd messenger systems that mediate transduction of chemical signals into neural responses within receptor cells. This knowledge, primarily from research on house mice, Mus domesticus, has led to the development of new methods for investigating the structure, organization, and function of olfactory and vomeronasal systems. These new approaches are beginning to provide new insights into how animals detect and analyze odors, and further advances should soon help us understand chemical communication.

I begin with a brief overview of terminology and conceptual issues relating to chemical signals because I think that ambiguous use of terms has often hindered both theory and specific hypotheses tested in experiments. Then I review some exciting areas of current research concerning chemical communication in rodents. I also refer to studies of chemical communication in other taxa because I think it is important to integrate ideas, terminology, and concepts across a wide range of species. As in any branch of biology, this kind of comparative approach will, one hopes, result in theories that have broad applicability.


Types of Chemical Signals

It is essential to define the terms used in this field. In particular, the term “pheromone” has been used many ways and has been the source of controversy (Beauchamp et al. 1976; Johnston 1998; Müller-Schwarze 1977; Preti and Wysocki 1999; Tinbergen 1951). Some scientists use this term to refer to a single chemical compound that elicits a specific response (behavioral, endocrine, etc.), a usage similar to the original definition of a pheromone, which was viewed as a prime example of the ethological concept of a releaser (Karlson 1960; Karlson and Butenandt 1959; Karlson and Lüscher 1959; Tinbergen 1951; Wilson and Bossert 1963). This usage generally implies that the underlying mechanisms for detection of and response to such compounds are predominantly controlled by genetic mechanisms. At the other extreme, some scientists use the term to refer to any body odor that influences interactions between individuals of a species, even if the effect is subtle, is unpredictable, occurs only in certain circumstances, or is based entirely on learning by individuals. This diversity of usage often results in muddled thinking and difficulties in communication, both among scientists and between scientists and the general reader. For example, if an odor that identifies an individual rodent or members of a colony of insects is referred to as a pheromone (e.g., Dani et al. 2001), some readers might assume that such odors elicit 1 particular response, that a single chemical compound is the signal, that the production of the signal is genetically determined, that 1 olfactory receptor cell and 1 receptor gene are responsible, or, in some vertebrates, that the responses are mediated by the vomeronasal organ, all of which are incorrect. One option would be to drop the term “pheromone” entirely, but it has gained such wide acceptance that this tactic would not be effective. At the very least, authors should be explicit about how they use this term. Preferably, some general agreement should be reached.

My definitions of the types of chemical signals focus on the chemical composition of the signal, specifically the number of chemical components in a signal, as the primary distinguishing feature (Johnston 2000). I use the term “pheromone” to refer to a single chemical compound that is released by 1 individual of a species and that has a behavioral, physiological, or combined effect on a receiver of the same species. Examples include substances that attract mates, cause alarm reactions, or induce hormonal changes (Beroza 1970; Cardé and Bell 1995; Cardé and Minks 1997; Novotny et al. 1999b; Vandenbergh 1983). A “pheromone blend” is a mixture of a few chemical compounds in relatively precise proportions that influences behavioral, physiological, or both responses; the effects depend on the proper proportions and do not occur, or occur at a much lower level, if these proportions are altered. That is, the signal is the “blend” (a mixture with precise proportions of component compounds). The majority of insect sex attractants are pheromone blends rather than single chemical compounds (Cardé and Baker 1984; Linn and Roelofs 1989). A “mosaic signal” contains a large number of compounds; the proportions might vary considerably across individuals or classes of individuals (Johnston 2000). Examples of such signals are the odors used for individual and kin recognition among mammals (Gorman 1976; Smith et al. 2001) and colony, hive, or kin recognition in insects (Dani et al. 2001; Gamboa et al. 1996; Howard 1993).

These definitions relate to just 1 important aspect: the chemical composition of the signal. In most other classification schemes, the definitions of the types of pheromones also have included aspects relating to genetic control of the response mechanisms, the efficacy of the signal in causing a response, and the types of responses influenced by the signal. Although it is useful to study and classify these other aspects, it can be confusing to use such effects as part of a definition. I briefly review some of these issues.

Controversial Issues Relating to Types of Chemical Signals and Their Effects

I discuss 3 issues that cause confusion when used as part of the definition of the types of chemical signals: the efficacy of a signal in causing a response with short latency and high probability; the degree to which the mechanisms underlying detection, perception, and response to chemical signals are “innate” (i.e., do not depend on specific types of experiences during development); and the division of responses into discrete classes on the basis of the organ system involved (i.e., behavioral or endocrine).

The causation of responses.—Pheromones were originally defined as substances that elicited or released a response in a conspecific; the 1st examples to be identified were sex attractants in insects in which the response was upwind flight or locomotion (Karlson 1960; Karlson and Butenandt 1959; Karlson and Lüscher 1959; Whitten 1966; Wilson and Bossert 1963). Releasing stimuli, as defined by ethologists, had 2 distinguishing features: that the signal elicited just 1 specific behavioral response and that this response occurred immediately and with high probability. These characteristics are not particularly useful as definitions of a general class of signals, however, because they vary across functions of signals, social contexts, and taxonomic groups. For example, signals with certain functions (e.g., alarm signals) are likely to be more effective in eliciting quick, highly probable responses than signals with other functions (e.g., sexual attractants, threats, or signals for recognition of sex or kin) because the costs of not responding are much greater in the case of alarm signals. Responses also can vary with social context (e.g., time of day, temperature, presence of a predator, or presence of conspecifics), and the importance of context will vary across different types of signal function and taxonomic groups. For example, responses of males to a sexual attractant from a female will probably be eliminated in virtually all species in the presence of a predator but will be eliminated in only some species in the presence of another, more dominant male. Species that have relatively simple mechanisms for control of behavior (e.g., invertebrates) are more likely to have quick, highly probable responses than species with more complex mechanisms (e.g., mammals). Yet it does not make sense to say that female insects have sex attractant pheromones but female mammals do not, simply because there are some circumstances in which male mammals do not show a response to such signals or the response is delayed until the situation changes (e.g., a dominant male leaves the area). Similarly, the specificity of the response to a chemical signal can vary across taxonomic groups. One specific response (e.g., move upwind or deposit sperm) is more likely in species with a relatively small repertoire of behaviors and control mechanisms than in species with a larger repertoire of behaviors and more mechanisms for evaluating the physical and social context (e.g., presence and identity of nearby individuals).

Thus, I suggest that the mechanisms related to the causation of behavior should not be a defining characteristic for types of signals. These mechanisms should be investigated and categorized in order to understand how communication occurs and social behavior is coordinated.

Development of responses to a signal.— Another feature that was part of the original concept of a pheromone was that the mechanisms underlying responses to such signals were innate (not dependent on experiences of the developing individual). Everyone now realizes that the dichotomy between “innate” and “learned” is not useful in understanding the development of the nervous system or behavior because development is a series of interactions between the environment and the organism and the activity of its genes. Nonetheless, it is clear that some mechanisms are relatively closed (not influenced much by individual experience), whereas other mechanisms are relatively open (strongly influenced by such experience). For example, it seems likely that the development of mechanisms underlying a response to most single-component signals (pheromones) or to pheromone blends will in fact be primarily controlled by genetic mechanisms (Rössler et al. 1999; Schneiderman et al. 1986). The degree to which experience influences responses to a signal, however, is an empirical question that should be investigated; it should not be part of the definition of a category of signals. One might even argue that research into the possible role of experience in development of responses to sex pheromones has been retarded by the assumption that there is no effect of experience.

I am not aware of any case in rodents or other mammals in which a single compound with an identified function has been investigated developmentally, but a variety of experiments with arbitrary odors or whole secretions is consistent with the hypothesis that learning can have a role. Among Norway rats (Rattus norvegicus), for example, exposure to an arbitrary odor during early development can result in this odor becoming attractive in a mate choice context in the adult (Fillion and Blass 1986). Mere exposure of house mice or humans to some odorants leads to increased sensitivity to them (Pierce et al. 1993; Yee and Wysocki 2001). In addition, pairing exposure to odors in infants with tactile stimuli that mimic licking by the mother causes long-term changes in the response to these odors in the olfactory bulb (Leon 1992; Leon et al. 1987). Similarly, exposure of male rats to an artificial odor during sexual activity can result in this odor releasing a hormonal response in these males (Graham and Desjardins 1980). The opposite type of effect has also been demonstrated. Although a male hamster's attraction to the vaginal secretion of females is largely independent of sexual experience, pairing of illness with exposure to vaginal secretions will result in an avoidance of this odor and a decrease in sexual behavior toward females (Johnston et al. 1978; Zahorik and Johnston 1976).

Categorization of signal types by level of effect: primers and releasers.—Primer pheromones were originally defined as substances that elicited physiological effects, such as changes in hormone levels, whereas releaser or signaling pheromones had effects on behavior (Bronson 1968; Whitten 1966; Wilson and Bossert 1963). There are several problems with the dichotomy between behavioral and physiological effects as a means of categorizing types of signals. First, chemical signals can influence both behavior and the endocrine system. This has been shown most thoroughly in goldfish, in which several steroids and prostaglandins that are released into the water by 1 fish cause behavioral and hormonal changes in other fish (Sorensen and Stacy 1999; Sorensen et al. 1989). Similarly, several individual chemical compounds cause both behavioral and endocrine responses in house mice (Novotny et al. 1999b). Many glandular secretions or other sources of body odors cause both kinds of responses in a variety of rodents and other mammals (Johnston 1990; Johnston et al. 1999; Meredith and Fernandez-Fewel 1994; Vandenbergh 1983). Rather than call a signal both a primer and a releaser, it seems less confusing to say that the signal has 2 different kinds of effects. Finally, the distinction between behavioral and physiological effects is to some extent arbitrary because all signals affecting behavior work by way of underlying physiological mechanisms.

Complexity of Odors and Signals in Mammals

Topography of information on the body surface.—In most mammalian species, individuals use several sources of odors for communication (Albone 1984; Brown and Macdonald 1985; Müller-Schwarze 1974). In golden hamsters (Mesocricetus auratus), for example, these sources include urine, feces, vaginal secretions, flank glands, ear glands, Harderian glands (located in the eye socket behind the eyeball), saliva, and glands on the feet (Gray et al. 1984; Johnston et al. 1993; Payne 1977, 1979). To what extent is each source specific for a particular kind of information or communicative function? To what extent do different sources contain redundant information? Is a particular type of information restricted to the area of the body where it is produced, or do chemical substances get spread around on the body surface such that the whole body is covered with information derived from different sources?

Several years ago when we began to study individual discrimination and recognition, we set out to answer these questions for individually distinctive information (Johnston et al. 1993). We hypothesized that such information would be obtained from all parts of the body. We tested 11 different sources of odors by a habituation–discrimination procedure and found that golden hamsters discriminated between odors of different individuals from just 5 specific sources (urine, feces, vaginal secretions, ear glands, and flank glands; Fig. 1). They did not discriminate between odors from 6 other locations (feet, saliva, fur just behind the ear, fur on the belly, fur on the shoulders, and fur from the flank region of flank-glandectomized males). We concluded that individually distinctive information exists at specific locations on the body and suggested that at least some of these sources had been selected for individual differences in the course of evolution (Johnston and Robinson 1993; Johnston et al. 1993). We found a similarly discrete distribution of individually distinctive information on the body of Campbell's hamster Phodopus campbelli (Lai and Johnston 1993).

Fig. 1

Schematic representation of 6 sources of odors on the body of a golden hamster and the distribution of 3 types of information in those sources (individual identity, sex, and reproductive state).

In other experiments, we investigated locations of information for sex and reproductive state in golden hamsters, Campbell's hamsters, and meadow voles Microtus pennsylvanicus (Ferkin and Johnston 1993, 1994; Ferkin et al. 1995; Lai et al. 1996). For example, when testing male hamsters for attraction to odors from females as compared to males, we found that all sources of individual odors also contained information about the sex of the donor (ear glands, flank glands, urine, feces, and vaginal secretions). Saliva and Harderian glands also contain information about sex (Fig. 1; Payne 1979; N. Vasilieva and R. E. Johnston, in litt.), but saliva does not contain individual information (Harderian secretions have not been tested). We also investigated which odors changed in their attractiveness to males as a female's reproductive state changed. For golden hamsters, we found that just 2 odors provided this information: vaginal secretions and flank glands (N. Vasilieva and R. E. Johnston, in litt.). This general pattern suggests that each scent source provides many different messages or types of information about the sender and that different scent sources have overlapping information but also can have some unique information. Our studies of meadow voles strongly support this hypothesis of multiple types of information in each scent source (Ferkin and Johnston 1993,1994, 1995; Ferkin et al. 1994, 1995).

Rodents and other animals can obtain even more subtle or precise information by comparing odors from different parts of the body, as suggested by a study with Campbell's hamster. We found that responses of males to 4 different odors from females changed with the female's reproductive state, but whereas the pattern of male attraction to 2 of these odors were the same across days, the changes in response to the other 2 odors were unique (Fig. 2; Lai et al. 1996). If males track these patterns of change over time, they should be able to determine a female's reproductive state with great precision. This information could decrease the costs involved in courtship and mate guarding because males of this species travel considerable distances to visit the burrows of several females (Wynne-Edwards et al. 1992).

Fig. 2

Relative attraction of male Campbell's hamsters (Phodopus campbelli) to 4 different odors from females across 3 reproductive states. Each of the 3 graphs shows a different pattern of changes in attraction across the 3 states, suggesting a rich array of information about changes in hormonal status of females (Lai et al. 1996).

It seems likely that most rodents and other mammals have discrete sources of chemical signals in specific locations on the body and that different sources contain signals or information that differs to some extent. This conclusion is suggested by the mere existence of multiple types of scent glands in many mammalian species and by behavioral observations that suggest that some glands are used or investigated by other individuals in some situations but not others (Adams 1980; Albone 1984; Brown and Macdonald 1985; Ewer 1968). In addition, different odor sources are under different types of neural and hormonal regulation (Albone 1984; Thody and Shuster 1989). At the same time, different sources of odors on the body are likely to have some overlapping kinds of information because there are only a few types of glands that produce chemical signals (e.g., sebaceous, apocrine, eccrine), and each type has a specific type of secretory process. Thus, the types of chemical compounds produced are similar across locations on the body (Albone 1984; Brown and Macdonald 1985; Thody and Shuster 1989).

Consequences for perception and communication.—Virtually all sources of odors in rodents and other mammals consist of a mixture of a large number of chemical compounds. How do animals analyze these mixtures to extract different kinds of information? Are there specific molecules or molecular blends that provide such information within the mixture? Alternatively, is information encoded in the overall odor quality of the mixture, which is in turn dependent on the identity and relative proportions of the many components of the mixture? Each of us knows from personal experience that humans perceive complex mixtures (e.g., scent of coffee, peanut butter, or orange peel) as a unitary odor quality rather than as a set of individual elements (Laing 1991). This means that, even if a glandular secretion does contain a pheromone and even if there is a specific receptor protein, specialized receptor cell, or both for this pheromone and there are dedicated neural pathways in the central nervous system for response to this chemical, the pheromone will still be perceived in the context of the odor quality of the mixture. Thus, for example, if a male rodent smells the urine from another member of its species, it might have hard-wired sensory mechanisms for 1 or more individual compounds, but in addition, it will also perceive information on the basis of the overall quality of the odors (e.g., the donor's species, individual identity, major histocompatibility type, or genetic relatedness [kin or nonkin]). These types of information might influence the male's behavior.

There are several important consequences of the association of different types of olfactory information in each odor source. First, in nature, mammalian pheromones might never be detected without associated contextual information. If so, responses to a pheromone might be modified depending on the attributes of the individual that produced the pheromone. It would still qualify as a pheromone if it caused similar responses in a wide variety of individuals in a variety of circumstances, but not necessarily in all circumstances. Thus, the communication process will be more complex than the original notion of communication by a pheromone, in which a single chemical would have a specific and relatively invariant effect on the receiver. Second, if it is true that pheromones are usually perceived in a rich context of other information from body odors, the standard types of bioassays for discovering pheromones or pheromone blends, in which 1 or a few specific compounds are presented in isolation to subjects in an experiment, might be an abnormal situation and would result in no response or an abnormal response. This type of complexity adds to the difficulty in identifying mammalian pheromones.

Current Topics in Chemical Communication in Rodents

In this section, I review some areas of research that I find particularly interesting. Similar work has been published on other species of mammals, but research with rodents generally leads the way because of the ease of studying many rodent species in captivity and because we already know so much about the anatomy, behavior, and physiology of a number of species of rodents.

Chemical Identification of the Signals

At present, relatively few active research programs are aimed at identifying mammalian pheromones; the time is ripe for new efforts to do so. Although molecular biologists have identified and characterized numerous genes for receptor proteins, little is known about the ligands for these receptors. Studying molecules with specific functions in social behavior might be the most productive approach to discovering what particular genes and receptor proteins do.

There are numerous examples from rodents in which odors from 1 individual have strong effects on the behavior, physiology, or both of conspecifics (Brown and Macdonald 1985; Johnston 1983; Novotny et al. 1990). An interesting, recently described example is an odor from female Norway rats (R. norvegicus) that causes erections in males (Sachs 1999). In this case, and in the vast majority of other studies in mammals, relatively little is known about the particular chemical compounds that are responsible for the effects observed (Albone 1984; Brown and Macdonald 1985; Johnston 2000).

Most progress has been made with the laboratory house mouse, M. m. domesticus. In this species, urine seems to be the most important source of chemical signals, and urine or urine marks have been shown to influence a wide range of endocrine and behavioral responses. Seven specific compounds have been isolated from mouse urine, and each of these has been shown to have effects on reproductive physiology or behavior (Novotny et al. 1999b). For example, 2,5-dimethyl pyrazine, an adrenal-dependent substance found in the urine of group-housed females, caused delay of puberty (as measured by uterine weight) in young female mice that were continuously exposed to this odor from the day of birth to sacrifice at 30 days of age (Jemiolo and Novotny 1994). Exposure to whole urine from group-housed females had a similar effect. Five unique, androgen-dependent compounds found in adult male urine each have the effect of accelerating female puberty (Jemiolo et al. 1989; Novotny et al. 1999a, 1999b). Two of these compounds, both farnesenes, also stimulate estrous cycles in females whose cycles have been suppressed due to group housing (Novotny et al. 1999b). Some of these compounds have effects on behavior as well; for example, 2-sec-butyl-dihidrothiazole and 3,4-dehydro-exo-brevicomin are attractive to adult females, as measured in a sniffing preference test, and increase the aggression of a resident male to a castrated intruder (Jemiolo et al. 1991; Novotny et al. 1985; for a more detailed review of this interesting program of research, see Novotny et al. 1999b). Given the uniqueness and importance of these findings, it is surprising that this research has not been replicated or more fully exploited by other laboratories. It would be valuable to know more about how all of these identified compounds work together to regulate behavior and endocrine processes.

One recent extension of these findings follows from previous work that indicated that the vomeronasal organ is primarily responsible for mediating hormonal responses to odors (Wysocki 1989; Wysocki and Meredith 1987). Six of the 7 substances that were identified in mouse urine by Novotny's group evoked physiological responses in receptor cells in the vomeronasal organ (Leinders-Zufall et al. 2000). Furthermore, the cells were sensitive to very low concentrations of these substances, and each cell responded to only 1 of the 7 compounds, suggesting that there might be receptor cells specialized for responses to these substances and for mediating hormonal responses (Leinders-Zufall et al. 2000). More recent results indicated that receptor cells in the main olfactory epithelium also are extremely sensitive to these substances, but each cell usually responded to more than 1 of the 7 chemical compounds (Ziesmann et al. 2002). Thus, receptor cells in both the main olfactory epithelium and vomeronasal organ might be involved in mediating responses to these compounds, but the mechanisms that process information about them could differ in the 2 sensory systems.

Another interesting topic is the role of proteins in communication, either as signals or as carriers of other compounds that constitute the signal. It was claimed, for example, that a protein in female hamster vaginal secretions stimulated male copulatory behavior and it was given the name “aphrodisin” (Singer et al. 1986, 1987, 1989). The gene for aphrodisin was cloned and inserted into bacteria in order to produce it easily, but aphrodisin produced by bacteria did not have the same effects as the same protein extracted from hamster vaginal secretions. This suggests either that sexual arousal in males is due to 1 or more ligands bound inside the 3-dimensional structure of the protein or that the protein produced by the bacteria differed in some crucial way from that produced by female hamsters (Jang et al. 2001; Singer and Macrides 1990).

Proteins also have been implicated in communication among laboratory house mice. One prominent feature of mouse urine is the presence of large amounts of proteins, called mouse urinary proteins (Pes et al. 1999). The exact roles of these proteins are not fully understood, but their 3-dimensional structure is like that of aphrodisin in having a pocket in which smaller molecules can be bound. Mouse urinary proteins do bind smaller molecules and promote the slow release of these substances, thus prolonging the effectiveness of urine marks (Beynon et al. 1999). They might also serve as carriers of such molecules, aiding in presentation of the molecule to the olfactory receptors by binding with the receptor or with adjacent molecules on the surface of the receptor cell membrane (Novotny et al. 1999b). Finally, there is some evidence that mouse urinary proteins might be signals (Brennan et al. 1999; Mucignat-Caretta and Carretta 1995), although this claim is controversial (Novotny et al. 1999b).

Difficulties in Bioassays of Mammalian Pheromones

Why have so few pheromones been chemically identified among mammals, while hundreds of pheromones have been identified among insects? One problem is that the secretions or excretions that contain the signals are much more complex in mammals; for example, glandular secretions in insects used as sexual attractants usually contain fewer than 10 chemical compounds, whereas those used in mammalian communication contain at least 40–50 and usually contain hundreds of compounds (Albone 1984). The sheer number of compounds makes it more difficult to find a few that have specific effects, especially because active compounds are often present in low concentrations (e.g., parts per million—Novotny et al. 1999b). A 2nd problem might be that most people who have tried to identify mammalian pheromones have used the classical pheromone model; that is, the signal is a single chemical compound. If one is looking for a single compound, a reasonable strategy is to use the 20-questions approach—divide the mixture into halves or 3rds and see which fraction the signal is in. If the signal is a pheromone blend or a complex mixture, however, this approach will quickly fail. A 3rd problem is that mammalian behavior is usually influenced by a number of social and environmental stimuli. If behavior is under the control of many different factors, as opposed to just a few factors, it is more difficult to discover the role of any 1 factor and it is thus more difficult to design effective assays to identify the essential components of a signal.

I will discuss 1 example that illustrates some of the difficulties in developing reliable behavioral assays—namely, the search for sexual attractant compounds in the vaginal secretions of female golden hamsters (M. auratus). Male hamsters are attracted to the vaginal secretion of females; they spend a large amount of time sniffing it and, if they have access to the secretion, licking it (for reviews, see Johnston 1985, 1990). This secretion also causes increased sexual arousal, an increase in circulating levels of luteinizing hormone and androgens, and a reduction in the aggressive tendencies of males (Johnston 1990; Macrides et al. 1974; Pfeiffer and Johnston 1992, 1994). An assay in which males were exposed to odors coming through the bedding on the floor of the subject's home cage showed that 1 component of the vaginal secretion, dimethyl disulfide (DMDS), was attractive to males but that a number of other components were not attractive, either by themselves or in a mixture (O'Connell et al. 1978, 1979; Singer et al. 1976). Thus, it was concluded that DMDS was an important component of a signal that was attractive.

Subsequent experiments suggested that the behavioral assay used was not optimal because male hamsters investigated a wide variety of odors in this situation (Johnston 1981), probably because the odors were a novel stimulus in an otherwise familiar and consistent environment. Using another type of assay, in which odors were presented across a screened door in a neutral arena, male hamsters were attracted to vaginal secretions but were not attracted to dimethyl disulfide (Petrulis and Johnston 1995). In this situation, both the test odor and the odors in the arena were novel. These results suggest that DMDS is not, by itself, an attractive compound. The important methodological point is that bioassays should be thoroughly evaluated by testing a wide variety of stimuli to determine whether the reaction to the biological fraction is unique. Ideally, at least 2 assays should be developed to show that the results from both tests agree. This is particularly important when the behavior measured is investigation time because animals investigate odors for many reasons. In addition, investigation is influenced by many situational and experiential variables.

Odors, the Major Histocompatibility Complex, and Mate Choice

In the process of studying the immune system and the creation of mouse strains that were genetically identical except at loci in the major histocompatibility complex (MHC), scientists at Sloan-Kettering discovered that mice could detect differences in the odors from urine of 2 such strains of mice (Yamaguchi et al. 1981; Yamazaki et al. 1980). Since then, these findings have been confirmed and elaborated, and similar effects have been shown in domestic Norway rats (Brown 1985; Brown et al. 1990; Yamazaki et al. 1990, 1999). This research suggests that the MHC, although not the only part of the genome that influences body odor, nonetheless contributes strongly to body odors in mice and rats. Indeed, it has been found that highly trained mice are capable of distinguishing the odors of 2 strains of mice that are genetically identical except for mutations in MHC genes that result in a difference in 3 amino acids in 1 protein (see Yamazaki et al. 1991 for review).

Why should the MHC have such a large effect on body odor? First, the MHC region is the most variable region of the genome, so the potential for individual differences from genetic differences are greater in this region. Why is there such variability in this region of the genome? This variability is related to the variability of the proteins produced by these genes, and 1 hypothesis for the value of such variability is that the greater the variety of proteins, the greater the range of responses to pathogens. Although there is little support for this hypothesis in experimental tests with single parasites (Penn and Potts 1999), direct evidence for an association between variability in the MHC and survival has been reported on an unmanaged population of sheep, Ovis aries (Paterson et al. 1998).

If heterozygosity in the MHC is advantageous, one would expect animals to select mates on the basis of their MHC type. Evidence consistent with this hypothesis comes from studies of house mice, in which it has been found that mice prefer mates that have an MHC type different from their own (Potts et al. 1991; Yamazaki et al. 1976, 1991). These preferences presumably are based on differences in the odors of the 2 strains, although other possible types of information have not been rigorously excluded. The advantage of such a mate choice strategy would be that the offspring of parents with different MHC types should have more variability in their MHC region and, therefore, a greater range of response to pathogens. Another possible advantage is that preferring a mate with a different MHC type is a mechanism for reducing inbreeding and its associated disadvantages (Penn and Potts 1999; Potts and Wakeland 1990).

If the MHC is important in mate choice because it influences the viability of offspring, one would expect to see the same effects in other species as well. Perhaps the most fascinating evidence among mammals has come from studies with humans (Homo sapiens). In several studies, human subjects judged the attractiveness of odors from donors of the opposite sex as less attractive if the MHC types of the donor and the subject were similar (Eggert et al. 1999; Wedekind and Füri 1997; Wedekind et al. 1995). These preferences also depended on the reproductive state of female subjects—women in the fertile phase of their menstrual cycle showed these preferences but women in the postovulatory phase of their cycle, and women on birth control pills did not. This pattern is consistent with the hypothesis that the preferences are related to preferences for sexual partners.

Perhaps most intriguing, 1 study suggests that actual choice of marriage partners might be influenced by the degree of similarity in the MHC (Ober 1997). This study was conducted on 31 colonies of Hutterites in South Dakota. The genetics of these colonies had been previously investigated because of the lack of marriages outside of these communities, the extensive genealogical records of families within the communities (dating back to the mid-1700s), and the lack of extramarital sexual relations (Ober 1995). Analyzing 411 married couples and knowing the population of possible marriage partners, it was found that there were fewer matches in MHC type among spouses than expected by chance (Ober 1997). There is no evidence for or against the hypothesis that such choices might have been based on odors, but this is 1 possible mechanism. Other mechanisms could be at work, however, because there was also a high frequency of multiple marriages between specific families. The MHC was also implicated in fecundity in this population. Married couples sharing MHC types at specific loci had longer intervals until pregnancy and increased fetal loss compared to couples not sharing types at these loci (Ober et al. 1992). Two reports have failed to demonstrate marriage preferences based on MHC type (Hedrick and Black 1997; Ihara et al. 2000), but in both cases much less was known about the populations involved (also see Penn and Potts 1999).

Kin Recognition

Differential response to close relatives compared to unrelated individuals is a common phenomenon among animal species. Particularly thorough work has been carried out among insects, amphibians, and mammals (Blaustein and Porter 1990; Fletcher and Michener 1987; Halpin 1991; Hepper 1991; Waldman 1991). By far the most research among mammals has been on rodents, including field studies on several species of ground squirrels (Sherman and Holmes 1985) and laboratory studies on Norway rats, several species of voles (MicrotusFerkin 1990), spiny mice (Acomys cahirinusPorter 1989), and Belding's ground squirrels (Spermophilis beldingiHolmes 1986, 1988; Mateo 2002, 2003).

Much of the laboratory work indicates that odors are widely used as a means of identifying kin. Little is known about the chemistry of the signals that allow such recognition in rodents or other mammals, but it has been studied among invertebrates. The general conclusion from this work is that such recognition is based on the relative proportions of chemical compounds in mixtures (mosaic signals—Bonavita-Cougourdan et al. 1987; Howard 1993; Howard et al. 1982; Morel et al. 1988). In the social paper wasp (Polistis fuscatus), for example, the ratios of cuticular hydrocarbons differ across nests of related wasps, and the quantitative patterns of these differences are sufficient to identify the colony from which a single wasp is taken (Gamboa et al. 1996). Several studies concerning individual recognition in mammals have similarly concluded that the relative amounts of chemical components in a complex mixture supply sufficient information for such recognition (Gorman 1976; Smith et al. 2001); these same cues are probably used to discriminate between kin and nonkin.

Despite the lack of information about the specific chemical substances involved in kin recognition among mammals, the general principle is clear. The proportions of each chemical compound in a mixture produced in a gland should be more similar for closely related individuals than for distantly related or unrelated individuals. In a parallel fashion, the perceived odor quality of these mixtures should be more similar for closely related individuals than for unrelated individuals. Recent experiments from my laboratory provide evidence for this latter hypothesis. In 1 set of experiments, we investigated kin recognition in golden hamsters by using 2 scent-marking behaviors as a behavioral assay (Heth et al. 1998). Female hamsters deposit vaginal secretions as a means of sexual advertisement (Johnston 1977, 1979); thus, it would be expected that they would vaginal mark more toward males or odors of males that were not related to themselves compared to cues from males that were closely related. Male and female hamsters flank mark in agonistic contexts to advertise their presence and to defend resources such as their burrow and food hoard (Johnston 1975a, 1975b, 1977, 1979); thus they should also flank mark more in response to stimuli from nonkin. The data support these predictions; for both flank and vaginal scent marking, hamsters marked more toward odors of nonkin than toward those of full siblings (Fig. 3; Heth et al. 1998). Furthermore, when we tested subjects with odors of half-siblings, frequency of marking fell between that for marking in response to the odors of full siblings versus unrelated individuals (Fig. 3; Heth et al. 1998). These results indicate that hamsters discriminate between siblings, half-siblings, and nonsiblings by odor cues and suggest that the odors of half-siblings were perceived as intermediate between those of full siblings and unrelated individuals. Evidence for the latter hypothesis also was obtained in laboratory rats, in which it was found that amount of sniffing investigation was inversely proportional to degree of relatedness (Hepper 1987).

Fig. 3

Number (mean ± SE) of vaginal marks (left graph) or male flank marks (right graph) by golden hamsters in response to the odors in the flank glands of male donors that were either familiar full siblings (Sib), unfamiliar half-siblings (Half-sib), or unfamiliar unrelated individuals (Non-sib). Data is reorganized and regraphed from Heth et al. (1998).

A 2nd set of experiments supports the idea that odor similarity is related directly to degree of relatedness. We used a habituation–discrimination technique in which a male subject was 1st exposed to flank gland secretions from 1 male donor during four 3-min trials (habituation trials) spaced 15 min apart and then tested in a 5th trial with the flank odor of a different male. We measured the time that the subject spent investigating these odors. Across habituation trials, males investigate new samples from the same individual's flank gland for shorter and shorter periods. On the test trial, investigation increases if a different individual's flank gland odor is presented (Johnston 1993; Johnston et al. 1993); this spontaneous increase in investigation indicates that the subject discriminated between the 2 flank gland odors. In our experiments with odors of kin, we varied both the relatedness of donors to subjects and the experience that a subject had with a donor; that is, subjects were either familiar with donors because they were reared together or unfamiliar because donors were reared in another litter.

Subjects readily discriminated between flank odors from 2 unrelated donors or from familiar donors that were siblings. They did not, however, show discrimination between the flank odors of 2 unfamiliar siblings from another family or between their own unfamiliar siblings (Todrank et al. 1998, 1999). Our interpretation of these results is that odors of full siblings are so similar that subjects did not notice the difference between them. Subjects did discriminate, however, between flank odors of full siblings (related or unrelated to the subjects) that they had been reared with and flank odors of full siblings (related or unrelated to the subjects) that they had not been reared with but that they had briefly interacted with (ten 1-min exposures) before the tests (Todrank et al. 1998, 1999). These results suggest that some actual interaction is necessary before male hamsters learn differences in odor between full siblings or before they notice that there are slight differences in odor that have significance. These data provide direct support for the hypothesis that close relatives have extremely similar odors (e.g., “family signatures”).

The marking experiments described above also provided evidence that the mechanism underlying kin recognition in golden hamsters is that of phenotype matching. That is, hamsters responded by scent marking differently to odors of kin and nonkin whether or not the donors of odors were familiar to the subjects (Heth et al. 1998). Thus, subjects must have had a mechanism, such as a memory of what kin smell like, on which to judge the relatedness of unknown individuals on the basis of their odor (i.e., phenotype matching—Holmes and Sherman 1982). How do hamsters obtain this information? The primary ways that have been suggested are through familiarity with close relatives (mother or siblings) during the nestling period or by comparison to their own cues (i.e., self-referent matching or the “armpit effect”—Dawkins 1982; Holmes and Sherman 1982). Although this latter mechanism has been proposed as a theoretical possibility for many years and some evidence is consistent with this view (Holmes 1986), all of this evidence could be interpreted in other ways as well. In the experiments discussed above, we could not discriminate between these 2 hypotheses because young were cross-fostered at 7 days of age into another litter along with 1–2 siblings; subjects could have developed a representation of what kin smelled like from either their siblings and mother before cross-fostering, from their siblings after cross-fostering, or from their own odors.

In another series of experiments, we investigated whether hamsters recognized kin on the basis of self-referent matching or familiarity with their mother and siblings (Mateo and Johnston 2000). In this experiment, we cross-fostered 1 female young per litter into another litter shortly after birth. Young were weaned at 30 days of age and caged individually. We tested these individuals when they were sexually mature (41–60 days of age) for their responses to odors of male and female flank glands from an individual that was familiar but unrelated, an unfamiliar sibling, and an unfamiliar and unrelated individual. Females were tested on the day before sexual receptivity when they advertise for mates (Johnston 1979; Lisk et al. 1983). Females investigated flank gland odors of unrelated, unfamiliar males more quickly and for a longer period of time than they investigated odors of their own, unfamiliar brothers, which would be expected if their investigation relates to preferences for mates (Fig. 4). Flank marking (an agonistic behavior) was performed less by pre-estrous females toward unrelated males than to related males (Mateo and Johnston 2000). Thus, these data show that cross-fostered females responded differently to odors of unfamiliar brothers compared with unfamiliar, unrelated males.

Fig. 4

Latency (mean ± SE) to sniff (left), duration of sniffing (center), and number of flank marks (right) produced by proestrous female hamsters in response to flank gland odors of males that were either nonsiblings reared together with the subject (NSRT), full siblings reared apart from the subject (SRA), or nonsiblings reared apart from the subject (NSRA); asterisk indicates P < 0.05. Data reorganized and regraphed from Mateo and Johnston (2000).

Self-referent matching is likely because subjects had no opportunity to learn the flank odors of their siblings since these glands do not develop until sexual maturity, subjects had little exposure to flank gland odors of their mother, and hampsters do not have a fully developed olfactory epithelium until 1 day after birth (Taniguchi et al. 1982). Projections to the entorhinal cortex, which might be necessary for discrimination of individual odors, is not complete until approximately day 13 (Leonard 1975; Petrulis et al. 2000). Subjects did have some exposure to their mother (up to 12 h), but due to the immaturity of the olfactory system, we think it is unlikely that they would be able to discriminate between odors of 2 individuals or remember these subtle differences until at least several days of age, and perhaps as much as 10–12 days of age. These experiments thus provide the best evidence to date for self-referent phenotype matching as a means of kin recognition among mammals.

Scent Overmarking

Virtually all terrestrial mammals have at least 1 type of scent-marking behavior by which individuals deposit scents from specialized glands, other secretions (e.g., saliva), or excretory products (e.g., urine or feces—Brown and Macdonald 1985; Ewer 1968; Johnson 1973). In many species, individuals also scent overmark; that is, they deposit scent so that it at least partially overlaps a scent mark previously deposited by another individual of the same species. Only recently, however, has it been suggested that overmarking might have unique functions and that there could be specialized mechanisms to promote overmarking and to analyze information in such marks (Ferkin et al. 2001; Hurst and Rich 1999; Johnston 1999; Johnston et al. 1994, 1995; Rich and Hurst 1999).

Terminology.—Overmarking refers to cases in which the scent mark of 1 individual partially overlaps or completely covers the mark of another individual. The term counter-marking has also been used in this context, but it seems to me that counter-marking is more appropriate as a generic term for both overmarking and adjacent-marking (i.e., cases in which marks are placed in close proximity to marks of others but do not overlap these earlier marks—Johnston et al. 1994). Although species that overmark also often deposit adjacent marks, there may be species that deposit adjacent marks but rarely overmark or that overmark in some situations but adjacent-mark in others. Thus, it seems reasonable to maintain this distinction.

Function.—Overmarking could be a form of competition between rivals in which individuals can target specific individuals by covering their scent marks. Such competition is likely to be energetically expensive because rivals are likely to respond in kind, calling for a counterresponse, and so on, and because each individual is likely to have several rivals. Although the number of scent marks deposited or time spent marking by animals in nature has been quantified in several cases (Gorman and Mills 1984; Peters and Mech 1975), I am not aware of any studies that have provided either measurements of frequency of, or a time-budget specifically for, overmarking. This is no doubt due, in part, to the difficulty in knowing every single location in which a scent mark has been deposited. Nonetheless, it seems likely that the extent to which an individual overmarks its rivals will be an honest signal of phenotypic quality in many species. This information could then be used by other individuals to evaluate competitors or the quality of potential mates.

Results consistent with this hypothesis have been reported for meadow voles and prairie voles (Microtus ochrogaster). Both male and female meadow voles, after being exposed to glass plates containing over-marks deposited by 2 individuals of the opposite sex (e.g., a female subject exposed to overmarks from 2 males) later preferred to approach and investigate the individual that provided the top scent or the whole-body odors of that individual (Ferkin 2001; Ferkin et al. 2001; Johnston et al. 1997a, 1997b). If the choice is between a novel individual and the individual with the top scent in the overmark, opposite-sexed meadow voles chose the individual providing the top scent. No preference was shown when the choice was the individual providing the bottom scent versus a novel individual. Results for prairie voles were mostly similar, although both male and female prairie voles actually preferred odors of a novel individual over those of the individual providing the bottom scent, suggesting that the latter individuals were devalued (Woodward et al. 1999). These results indicate that voles can tell which individual's scent is on top or most recently deposited and suggest that both meadow and prairie voles might use overmarks as a means of evaluating potential mates.

Mechanisms for evaluation of over-marks.—How can individuals determine which animal has overmarked another? A series of experiments with hamsters and meadow voles have at least partially characterized the mechanisms involved. The most obvious hypothesis is that animals might use relative freshness of 2 individuals' scents as a cue and later prefer or remember better the fresher scent. This does not, however, seem to be the case in hamsters or meadow voles. Freshness differed by only 15–60 s in a variety of experiments with both species, and it seems unlikely that such small differences in age of scents would be detected or, if they were, would result in differential evaluation of the markers (Cohen et al. 2001; Ferkin et al. 1999; Johnston and Bhorade 1998). In other experiments, this hypothesis was investigated directly. Scent marks that differed by 24 h in age were placed adjacent to each other. Subjects, after investigating these marks, did not prefer 1 donor (Johnston et al. 1997b) or show a preferential memory for 1 scent over the other (Ferkin et al. 1999; Johnston and Bhorade 1998). Thus, freshness of the scent, by itself, was not sufficient for differential responses to 2 individuals in either golden hamsters or meadow voles; for house mice, however, Rich and Hurst (1999) suggest that freshness is important in evaluation of potential mates.

A 2nd way that animals might evaluate overmarks is to determine which individual's scent covers the most area or which individual has the greatest number or density of scent marks. On a functional level, placing more marks in an area would presumably reflect vigor. Experiments with meadow voles and golden hamsters, however, do not support this hypothesis (Ferkin et al. 1999; Johnston and Bhorade 1998; Johnston et al. 1997a, 1997b; Wilgox and Johnston 1995). For example, female meadow voles were allowed to investigate the home cage of a male that had been briefly investigated and marked by a 2nd male. Females preferred whole-body odors of this intruder male over the home-cage male, despite the probability that the home-cage male had many more marks and covered a greater area with his marks (Johnston et al. 1997a). Again, however, house mice are different. Female mice preferred a male whose territory was exclusively marked by the owner compared to a male whose territory contained marks of another male as well as those of the owner (Rich and Hurst 1998).

A 3rd possibility is that animals can determine the position of the scents of 2 individuals (i.e., which is on top and which is on the bottom in an overmark). This hypothesis has been supported in a number of experiments (Cohen et al. 2001; Ferkin et al. 1999; Johnston and Bhorade 1998; Johnston et al. 1997a, 1997b; Wilcox and Johnston 1995). In the most dramatic case, we rubbed the anogenital scent of a male vole on a glass microscope slide so that it covered a large portion of the slide. We then placed a small spot of scent of a 2nd vole in the center of the slide, on top of the scent of the 1st vole (Fig. 5, left side). After females were exposed to this simulated overmark, they preferred the whole-body odors of the top-scent individual. If, however, we left a clean spot in the middle of the slide when we deposited the 1st vole's scent and then placed the 2nd male's scent on this clean spot, so that there was no scent overlap (Fig. 5, right side), before exposing females to the slide, they did not show a preference for either scent donor (Johnston et al. 1997b). This experiment shows that female voles can determine when there is an overmark, as opposed to 2 closely adjacent scents, and the relative position (top or bottom) of the 2 individuals' scents. Thus, voles are exquisitely sensitive to the overlap of 2 individuals' scent marks. The females' preference for the top-scent male suggests that this information might influence their choice of mates (Johnston et al. 1997a).

Fig. 5

On the left, scent of male #2 was deposited on top of that of male #1; on the right, scent of male #1 was deposited on a clean portion of the plate, surrounded by the scent of male #1. Above: pattern of anogenital scent on glass slide from male meadow vole #1 (vertical lines) and male #2 (black). Below, mean (+SE) time that females spent sniffing bedding odors from male #1 (vertical lines) and male #2 (black) in a Y-maze after exposure to the scent marks of these males in the pattern shown above the graph. Data is reorganized and regraphed from Johnston et al. (1997b).

Experiments in golden hamsters also demonstrate this ability to determine which scent is on top or bottom (Johnston 1999; Johnston and Bhorade 1998). In addition, hamsters might have yet another mechanism to determine relative position; they might be able to use the geometry of 2 linear scents in an overmark to determine which is on top. That is, by analogy with visual occlusion of 1 object by another, the top scent should be a continuous streak, whereas the bottom streak will be interrupted by the top scent. We tested this hypothesis by placing the scent mark of 1 individual in a continuous streak and the scent of another individual at a right angle to this such that it came up to the mark of the 1st individual but stopped short by 1–2 mm. After exposure to this apparent over-mark, hamsters preferentially remembered the continuous scent, as if it were on top (Cohen et al. 2001; Johnston and Bhorade 1998). Thus, hamsters can form a finegrained spatial representation of the location of individuals' marks to determine which is the top scent.

Experiments with meadow voles, golden hamsters, and house mice reveal previously unsuspected abilities of animals to determine the relationship between scents of individuals in an overmark. These experiments also raise numerous additional questions about the perception, production, and functions of scent overmarking. It is likely that functions of overmarking vary greatly across species and, thus, that mechanisms underlying the causation of marking and the perception of overmarks also might vary. For example, prairie voles are monogamous and live in family units, whereas meadow voles are polygynous, with females defending small territories against other females and males having large home ranges that overlap ranges of other males and territories of several females. There are indications that overmarking behavior and responses to overmarks differ between these 2 species (Ferkin 2001; Ferkin et al. 2001), but at present, these results are not easily interpreted. Studies with house mice suggest that they saturate their environment with urine marks (Hurst 1987). Although in some situations they do not display much overmarking (Hurst 1987, 1989), in other situations they do overmark, but the extent of overmarking depends not only on the sex of the individuals but also their familiarity (Hurst 1990a, 1990b, 1990c).

Studies of scent overmarking have revealed that a number of species of rodents analyze and understand complexities in the olfactory environment that were not even imagined by naturalists or scientists a few years ago. Further comparative investigations of the functions and mechanisms of scent overmarking are likely to be extremely rewarding, especially if laboratory and field studies can be combined.


Among mammals, studies of rodents have provided the majority of fundamental insights into chemical communication and the physiological mechanisms underlying such communication. With the recent publication of the complete mouse genome, research on the sense of smell is likely to be concentrated on it and closely related species of rodents in the near future. This provides an exciting opportunity to learn more about chemical, behavioral, endocrine, and ecological levels of analysis so that a fully integrated picture can be obtained.


I thank N. Solomon and E. Lacey for inviting me to participate in the symposium and for comments on the manuscript, and J. Johnston for help with the manuscript and figures. Preparation of this manuscript was supported by a grant from the National Institute of Mental Health, grant 5R01 MH58001-01A1.


  • Associate Editor was D. M. Leslie, Jr.

Literature Cited

View Abstract