Group selection is a proposed mechanism of evolution in which natural selection acts at the level of the group, instead of at the level of the individual or gene.

Early authors such as V. C. Wynne-Edwards and Konrad Lorenz argued that the behavior of animals could affect their survival and reproduction as groups, speaking for instance of actions for the good of the species. In the 1930s, R.A. Fisher and J.B.S. Haldane proposed the concept of kin selection, a form of altruism from the gene-centered view of evolution, arguing that animals should sacrifice for their relatives, and thereby implying that they should not sacrifice for non-relatives. From the mid-1960s, evolutionary biologists such as John Maynard Smith, W. D. Hamilton, George C. Williams, and Richard Dawkins argued that natural selection acted primarily at the level of the gene. They argued on the basis of mathematical models that individuals would not altruistically sacrifice fitness for the sake of a group unless it would ultimately increase the likelihood of an individual passing on their genes. A consensus emerged that group selection did not occur, including in special situations such as the haplodiploid social insects like honeybees (in the Hymenoptera), where kin selection explains the behaviour of non-reproductives equally well, since the only way for them to reproduce their genes is via kin.

In 1994 David Sloan Wilson and Elliott Sober argued for multi-level selection, including group selection, on the grounds that groups, like individuals, could compete. In 2010 three authors including E. O. Wilson, known for his work on social insects especially ants, again revisited the arguments for group selection. They argued that group selection can occur when competition between two or more groups, some containing altruistic individuals who act cooperatively together, is more important for survival than competition between individuals within each group, provoking a strong rebuttal from a large group of ethologists.

Charles Darwin developed the theory of evolution in his book, Origin of Species. Darwin also made the first suggestion of group selection in The Descent of Man that the evolution of groups could affect the survival of individuals. He wrote, "If one man in a tribe... invented a new snare or weapon, the tribe would increase in number, spread, and supplant other tribes. In a tribe thus rendered more numerous there would always be a rather better chance of the birth of other superior and inventive members."

Once Darwinism had been accepted in the modern synthesis of the mid-twentieth century, animal behavior was glibly explained with unsubstantiated hypotheses about survival value, which was largely taken for granted. The naturalist Konrad Lorenz had argued loosely in books like On Aggression (1966) that animal behavior patterns were "for the good of the species", without actually studying survival value in the field. Richard Dawkins noted that Lorenz was a "'good of the species' man" so accustomed to group selection thinking that he did not realize his views "contravened orthodox Darwinian theory". The ethologist Niko Tinbergen praised Lorenz for his interest in the survival value of behavior, and naturalists enjoyed Lorenz's writings for the same reason. In 1962, group selection was used as a popular explanation for adaptation by the zoologist V. C. Wynne-Edwards. In 1976, Richard Dawkins wrote a well-known book on the importance of evolution at the level of the gene or the individual, The Selfish Gene.

From the mid-1960s, evolutionary biologists argued that natural selection acted primarily at the level of the individual. In 1964, John Maynard Smith, C. M. Perrins (1964), and George C. Williams in his 1966 book Adaptation and Natural Selection cast serious doubt on group selection as a major mechanism of evolution; Williams's 1971 book Group Selection assembled writings from many authors on the same theme.

It was at that time generally agreed that this was the case even for eusocial insects such as honeybees, which encourages kin selection, since workers are closely related.

Experiments from the late 1970s suggested that selection involving groups was possible. Early group selection models assumed that genes acted independently, for example a gene that coded for cooperation or altruism. Genetically based reproduction of individuals implies that, in group formation, the altruistic genes would need a way to act for the benefit of members in the group to enhance the fitness of many individuals with the same gene. But it is expected from this model that individuals of the same species would compete against each other for the same resources. This would put cooperating individuals at a disadvantage, making genes for cooperation likely to be eliminated. Group selection on the level of the species is flawed because it is difficult to see how selective pressures would be applied to competing/non-cooperating individuals.

Kin selection between related individuals is accepted as an explanation of altruistic behavior. R.A. Fisher in 1930 and J.B.S. Haldane in 1932 set out the mathematics of kin selection, with Haldane famously joking that he would willingly die for two brothers or eight cousins. In this model, genetically related individuals cooperate because survival advantages to one individual also benefit kin who share some fraction of the same genes, giving a mechanism for favoring genetic selection.

Inclusive fitness theory, first proposed by W. D. Hamilton in the early 1960s, gives a selection criterion for evolution of social traits when social behavior is costly to an individual organism's survival and reproduction. The criterion is that the reproductive benefit to relatives who carry the social trait, multiplied by their relatedness (the probability that they share the altruistic trait) exceeds the cost to the individual. Inclusive fitness theory is a general treatment of the statistical probabilities of social traits accruing to any other organisms likely to propagate a copy of the same social trait. Kin selection theory treats the narrower but simpler case of the benefits to close genetic relatives (or what biologists call 'kin') who may also carry and propagate the trait. A significant group of biologists support inclusive fitness as the explanation for social behavior in a wide range of species, as supported by experimental data. An article was published in Nature with over a hundred coauthors.

One of the questions about kin selection is the requirement that individuals must know if other individuals are related to them, or kin recognition. Any altruistic act has to preserve similar genes. One argument given by Hamilton is that many individuals operate in "viscous" conditions, so that they live in physical proximity to relatives. Under these conditions, they can act altruistically to any other individual, and it is likely that the other individual will be related. This population structure builds a continuum between individual selection, kin selection, kin group selection and group selection without a clear boundary for each level. However, early theoretical models by D.S. Wilson et al. and Taylor showed that pure population viscosity cannot lead to cooperation and altruism. This is because any benefit generated by kin cooperation is exactly cancelled out by kin competition; additional offspring from cooperation are eliminated by local competition. Mitteldorf and D. S. Wilson later showed that if the population is allowed to fluctuate, then local populations can temporarily store the benefit of local cooperation and promote the evolution of cooperation and altruism. By assuming individual differences in adaptations, Yang further showed that the benefit of local altruism can be stored in the form of offspring quality and thus promote the evolution of altruism even if the population does not fluctuate. This is because local competition among more individuals resulting from local altruism increases the average local fitness of the individuals that survive.

Another explanation for the recognition of genes for altruism is that a single trait, group reciprocal kindness, is capable of explaining the vast majority of altruism that is generally accepted as "good" by modern societies. The phenotype of altruism relies on recognition of the altruistic behavior by itself. The trait of kindness will be recognized by sufficiently intelligent and undeceived organisms in other individuals with the same trait. Moreover, the existence of such a trait predicts a tendency for kindness to unrelated organisms that are apparently kind, even if the organisms are of another species. The gene need not be exactly the same, so long as the effect or phenotype is similar. Multiple versions of the gene—or even meme—would have virtually the same effect. This explanation was given by Richard Dawkins as an analogy of a man with a green beard. Green-bearded men are imagined as tending to cooperate with each other simply by seeing a green beard, where the green beard trait is incidentally linked to the reciprocal kindness trait.

Kin selection or inclusive fitness is accepted as an explanation for cooperative behavior in many species, but the scientist David Sloan Wilson argues that human behavior is difficult to explain with only this approach. In particular, he claims it does not seem to explain the rapid rise of human civilization. Wilson has argued that other factors must also be considered in evolution. Wilson and others have continued to develop group selection models.

Early group selection models were flawed because they assumed that genes acted independently; but genetically based interactions among individuals are ubiquitous in group formation because genes must cooperate for the benefit of association in groups to enhance the fitness of group members. Additionally, group selection on the level of the species is flawed because it is difficult to see how selective pressures would be applied; selection in social species of groups against other groups, rather than the species entire, seems to be the level at which selective pressures are plausible. On the other hand, kin selection is accepted as an explanation of altruistic behavior. Some biologists argue that kin selection and multilevel selection are both needed to "obtain a complete understanding of the evolution of a social behavior system".

In 1994 David Sloan Wilson and Elliott Sober argued that the case against group selection had been overstated. They considered whether groups can have functional organization in the same way as individuals, and consequently whether groups can be "vehicles" for selection. They do not posit evolution on the level of the species, but selective pressures that winnow out small groups within a species, e.g. groups of social insects or primates. Groups that cooperate better might survive and reproduce more than those that did not. Resurrected in this way, Wilson & Sober's new group selection is called multilevel selection theory.

In 2010, Martin Nowak, C. E. Tarnita and E. O. Wilson argued for multi-level selection, including group selection, to correct what they saw as deficits in the explanatory power of inclusive fitness. A response from 137 other evolutionary biologists argued "that their arguments are based upon a misunderstanding of evolutionary theory and a misrepresentation of the empirical literature".

Wilson compared the layers of competition and evolution to nested sets of Russian matryoshka dolls. The lowest level is the genes, next come the cells, then the organism level and finally the groups. The different levels function cohesively to maximize fitness, or reproductive success. The theory asserts that selection for the group level, involving competition between groups, must outweigh the individual level, involving individuals competing within a group, for a group-benefiting trait to spread.

Multilevel selection theory focuses on the phenotype because it looks at the levels that selection directly acts upon. For humans, social norms can be argued to reduce individual level variation and competition, thus shifting selection to the group level. The assumption is that variation between different groups is larger than variation within groups. Competition and selection can operate at all levels regardless of scale. Wilson wrote, "At all scales, there must be mechanisms that coordinate the right kinds of action and prevent disruptive forms of self-serving behavior at lower levels of social organization." E. O. Wilson summarized, "In a group, selfish individuals beat altruistic individuals. But, groups of altruistic individuals beat groups of selfish individuals."

Wilson ties the multilevel selection theory regarding humans to another theory, gene–culture coevolution, by acknowledging that culture seems to characterize a group-level mechanism for human groups to adapt to environmental changes.

MLS theory can be used to evaluate the balance between group selection and individual selection in specific cases. An experiment by William Muir compared egg productivity in hens, showing that a hyper-aggressive strain had been produced through individual selection, leading to many fatal attacks after only six generations; by implication, it could be argued that group selection must have been acting to prevent this in real life. Group selection has most often been postulated in humans and, notably, eusocial Hymenoptera that make cooperation a driving force of their adaptations over time and have a unique system of inheritance involving haplodiploidy that allows the colony to function as an individual while only the queen reproduces.

Wilson and Sober's work revived interest in multilevel selection. In a 2005 article, E. O. Wilson argued that kin selection could no longer be thought of as underlying the evolution of extreme sociality, for two reasons. First, he suggested, the argument that haplodiploid inheritance (as in the Hymenoptera) creates a strong selection pressure towards nonreproductive castes is mathematically flawed. Second, eusociality no longer seems to be confined to the hymenopterans; increasing numbers of highly social taxa have been found in the years since Wilson's foundational text Sociobiology: A New Synthesis was published in 1975. These including a variety of insect species, as well as two rodent species (the naked mole-rat and the Damaraland mole rat). Wilson suggests the equation for Hamilton's rule:

rb > c

(where b represents the benefit to the recipient of altruism, c the cost to the altruist, and r their degree of relatedness) should be replaced by the more general equation

rb_k + b_e > c

in which b_k is the benefit to kin (b in the original equation) and b_e is the benefit accruing to the group as a whole. He then argues that, in the present state of the evidence in relation to social insects, it appears that b_e>rb_k, so that altruism needs to be explained in terms of selection at the colony level rather than at the kin level. However, kin selection and group selection are not distinct processes, and the effects of multi-level selection are already accounted for in Hamilton's rule, rb>c, provided that an expanded definition of r, not requiring Hamilton's original assumption of direct genealogical relatedness, is used, as proposed by E. O. Wilson himself.

Spatial populations of predators and prey show restraint of reproduction at equilibrium, both individually and through social communication, as originally proposed by Wynne-Edwards. While these spatial populations do not have well-defined groups for group selection, the local spatial interactions of organisms in transient groups are sufficient to lead to a kind of multi-level selection. There is however as yet no evidence that these processes operate in the situations where Wynne-Edwards posited them.

Rauch et al.'s analysis of host-parasite evolution is broadly hostile to group selection. Specifically, the parasites do not individually moderate their transmission; rather, more transmissible variants – which have a short-term but unsustainable advantage – arise, increase, and go extinct.

The problem with group selection is that for a whole group to get a single trait, it must spread through the whole group first by regular evolution. But, as J. L. Mackie suggested, when there are many different groups, each with a different evolutionarily stable strategy, there is selection between the different strategies, since some are worse than others. For example, a group where altruism was universal would indeed outcompete a group where every creature acted in its own interest, so group selection might seem feasible; but a mixed group of altruists and non-altruists would be vulnerable to cheating by non-altruists within the group, so group selection would collapse.

Social behaviors such as altruism and group relationships can impact many aspects of population dynamics, such as intraspecific competition and interspecific interactions. In 1871, Darwin argued that group selection occurs when the benefits of cooperation or altruism between subpopulations are greater than the individual benefits of egotism within a subpopulation. This supports the idea of multilevel selection, but kinship also plays an integral role because many subpopulations are composed of closely related individuals. An example of this can be found in lions, which are simultaneously cooperative and territorial. Within a pride, males protect the pride from outside males, and females, who are commonly sisters, communally raise cubs and hunt. However, this cooperation seems to be density dependent. When resources are limited, group selection favors prides that work together to hunt. When prey is abundant, cooperation is no longer beneficial enough to outweigh the disadvantages of altruism, and hunting is no longer cooperative.

Interactions between different species can also be affected by multilevel selection. Predator-prey relationships can also be affected. Individuals of certain monkey species howl to warn the group of the approach of a predator. The evolution of this trait benefits the group by providing protection, but could be disadvantageous to the individual if the howling draws the predator's attention to them. By affecting these interspecific interactions, multilevel and kinship selection can change the population dynamics of an ecosystem.

Multilevel selection attempts to explain the evolution of altruistic behavior in terms of quantitative genetics. Increased frequency or fixation of altruistic alleles can be accomplished through kin selection, in which individuals engage in altruistic behavior to promote the fitness of genetically similar individuals such as siblings. However, this can lead to inbreeding depression, which typically lowers the overall fitness of a population. However, if altruism were to be selected for through an emphasis on benefit to the group as opposed to relatedness and benefit to kin, both the altruistic trait and genetic diversity could be preserved. However, relatedness should still remain a key consideration in studies of multilevel selection. Experimentally imposed multilevel selection on Japanese quail was more effective by an order of magnitude on closely related kin groups than on randomized groups of individuals.

Gene-culture coevolution (also called dual inheritance theory) is a modern hypothesis (applicable mostly to humans) that combines evolutionary biology and modern sociobiology to indicate group selection. It is believed that this approach of combining genetic influence with cultural influence over several generations is not present in the other hypotheses such as reciprocal altruism and kin selection, making gene-culture evolution one of the strongest realistic hypotheses for group selection. Fehr provides evidence of group selection taking place in humans presently with experimentation through logic games such as prisoner's dilemma, the type of thinking that humans have developed many generations ago.

Gene-culture coevolution allows humans to develop highly distinct adaptations to the local pressures and environments more quickly than with genetic evolution alone. Robert Boyd and Peter J. Richerson, two strong proponents of cultural evolution, postulate that the act of social learning, or learning in a group as done in group selection, allows human populations to accrue information over many generations. This leads to cultural evolution of behaviors and technology alongside genetic evolution. Boyd and Richerson believe that the ability to collaborate evolved during the Middle Pleistocene, a million years ago, in response to a rapidly changing climate.

In 2003, the behavioral scientist Herbert Gintis examined cultural evolution statistically, offering evidence that societies that promote pro-social norms have higher survival rates than societies that do not. Gintis wrote that genetic and cultural evolution can work together. Genes transfer information in DNA, and cultures transfer information encoded in brains, artifacts, or documents. Language, tools, lethal weapons, fire, cooking, etc., have a long-term effect on genetics. For example, cooking led to a reduction of size of the human gut, since less digestion is needed for cooked food. Language led to a change in the human larynx and an increase in brain size. Projectile weapons led to changes in human hands and shoulders, such that humans are much better at throwing objects than the closest human relative, the chimpanzee.

In 2015, William Yaworsky and colleagues surveyed the opinions of anthropologists on group selection, finding that these varied with the gender and politics of the social scientists concerned. In 2019, Howard Rachlin and colleagues proposed group selection of behavioural patterns, such as learned altruism, during ontogeny parallel to group selection during phylogeny.

This section needs expansion. You can help by adding to it. (September 2023)

The vast majority of behavioural biologists have not been convinced by renewed attempts to revisit group selection as a plausible mechanism of evolution.

The use of the Price equation to support group selection was challenged by van Veelen in 2012, arguing that it is based on invalid mathematical assumptions.

Advocates of the gene-centered view of evolution such as Dawkins and Daniel Dennett remain unconvinced about group selection. Dawkins suggests that group selection fails to make an appropriate distinction between replicators and vehicles. The evolutionary biologist Jerry Coyne summarizes the arguments in The New York Review of Books in non-technical terms as follows:

Group selection isn't widely accepted by evolutionists for several reasons. First, it's not an efficient way to select for traits, like altruistic behavior, that are supposed to be detrimental to the individual but good for the group. Groups divide to form other groups much less often than organisms reproduce to form other organisms, so group selection for altruism would be unlikely to override the tendency of each group to quickly lose its altruists through natural selection favoring cheaters. Further, little evidence exists that selection on groups has promoted the evolution of any trait. Finally, other, more plausible evolutionary forces, like direct selection on individuals for reciprocal support, could have made humans prosocial. These reasons explain why only a few biologists, like [David Sloan] Wilson and E. O. Wilson (no relation), advocate group selection as the evolutionary source of cooperation.

The psychologist Steven Pinker states that "group selection has no useful role to play in psychology or social science", since in these domains it "is not a precise implementation of the theory of natural selection, as it is, say, in genetic algorithms or artificial life simulations. Instead [in psychology] it is a loose metaphor, more like the struggle among kinds of tires or telephones."

• Bergstrom, T. C. (2002). "Evolution of Social Behavior: Individual and Group Selection" (PDF). Journal of Economic Perspectives. 16 (2): 67–88. CiteSeerX 10.1.1.377.5059. doi:10.1257/0895330027265. S2CID 125841.
• Bijma, P.; Muir, W. M.; Van Arendonk, J. A. M. (2007). "Multilevel Selection 1: Quantitative Genetics of Inheritance and Response to Selection". Genetics. 175 (1): 277–288. doi:10.1534/genetics.106.062711. PMC 1775021. PMID 17110494.
• Bijma, P.; Muir, W. M.; Ellen, E. D.; Wolf, Jason B.; Van Arendonk, J. A. M. (2007). "Multilevel Selection 2: Estimating the Genetic Parameters Determining Inheritance and Response to Selection". Genetics. 175 (1): 289–299. doi:10.1534/genetics.106.062729. PMC 1775010. PMID 17110493.
• Boyd, R.; Richerson, P. J. (2002). "Group Beneficial Norms Spread Rapidly in a Structured Population" (PDF). Journal of Theoretical Biology. 215 (3): 287–296. Bibcode:2002JThBi.215..287B. CiteSeerX 10.1.1.405.1548. doi:10.1006/jtbi.2001.2515. PMID 12054837. Archived from the original (PDF) on January 27, 2005.
• West, S. A.; Griffin, A. S.; Gardner, A. (2008). "Social semantics: how useful has group selection been?". Journal of Evolutionary Biology. 21: 374–385. doi:10.1111/j.1420-9101.2007.01458.x. S2CID 14122417.
• Sober, Elliott; Wilson, David Sloan (1998). Unto Others: The Evolution and Psychology of Unselfish Behavior. Harvard University Press. ISBN 978-0674930476.
• Soltis, J.; Boyd, R.; Richerson, P. J. (1995). "Can Group-functional Behaviors Evolve by Cultural Group Selection? An Empirical Test" (PDF). Current Anthropology. 63 (3): 473–494. doi:10.1086/204381. S2CID 43998139.
• Wilson, David Sloan (1987). "Altruism in Mendelian populations derived from sibling groups: The haystack model revisited". Evolution. 41 (5): 1059–1070. doi:10.2307/2409191. JSTOR 2409191. PMID 28563418.
• Wilson, David Sloan (2006). P. Carruthers; S. Laurence; S. Stich (eds.). Human groups as adaptive units: toward a permanent consensus (PDF). Oxford University Press. ISBN 978-0195310146. Archived from the original (PDF) on 26 February 2009. {{cite book}}: |work= ignored (help)

• "Altruism and Group Selection". Internet Encyclopedia of Philosophy.
• Lloyd, Elisabeth, "Units and Levels of Selection." The Stanford Encyclopedia of Philosophy, (Fall 2005 Edition), Edward N. Zalta (ed.)
• The Controversy of the Group Selection Theory – a review from the "Science Creative Quarterly" (a blog)
• Binghamton: D.S. Wilson