Long after Neanderthals disappeared, fragments of their DNA continued a quiet struggle inside modern human genomes. Instead of vanishing all at once, many inherited genetic variants appear to have been slowly stripped away over tens of thousands of years. The remarkable part of that story is not that natural selection acted against Neanderthal ancestry, but how weak that selection was at any single location—and how thousands of almost imperceptible effects may together explain one of the most enduring signatures of ancient interbreeding.
Modern people outside Africa inherited a small amount of Neanderthal DNA after humans and Neanderthals interbred roughly 47,000 to 65,000 years ago. Previous work had already shown that this inherited ancestry is not spread evenly across the genome. Regions packed with genes, or regions where genetic recombination occurs less frequently, generally contain less Neanderthal DNA. The X chromosome also carries substantially less Neanderthal ancestry than the autosomes, while Neanderthal ancestry is absent from both the human Y chromosome and mitochondrial DNA.
Those patterns have long pointed toward natural selection removing Neanderthal genetic material after interbreeding. The more difficult question has been why.
One possibility has been that humans and Neanderthals had already evolved widespread genetic incompatibilities before they met again, causing many Neanderthal variants to become harmful once introduced into modern humans. Another possibility is that differences in population history, rather than incompatibilities, produced much of the pattern.
The new study set out to measure the average strength of selection acting against Neanderthal DNA across the genome in order to distinguish between these possibilities.
Following the fading signal of ancient ancestry
Rather than attempting to identify every harmful Neanderthal mutation individually, the researchers developed a model that predicts how common Neanderthal-derived DNA should remain at locations throughout the modern human genome.
The approach combined several pieces of information. It considered how close a neutral stretch of DNA lies to protein-coding regions, how frequently recombination separates neighboring genetic variants, the amount of Neanderthal ancestry present immediately after interbreeding, the time elapsed since admixture, and the probability that protein-coding DNA contains a harmful Neanderthal allele.
Because the precise locations of all harmful variants are unknown, the model treated every exonic base as having a small probability of carrying one. It then calculated the expected frequency of Neanderthal ancestry at each location by averaging across those possibilities.
The predictions were compared with previously published estimates of Neanderthal allele frequencies throughout the genomes of present-day Europeans and East Asians. The researchers estimated their model’s parameters by minimizing the differences between predicted and observed Neanderthal ancestry and evaluated uncertainty through large-scale block bootstrap analyses across the genome.
Many tiny disadvantages instead of a few powerful ones
The resulting estimates painted a remarkably consistent picture.
Across the autosomes, the average selection coefficient against harmful Neanderthal alleles was approximately 4.1 × 10⁻⁴ in Europeans and 3.5 × 10⁻⁴ in East Asians. Although statistically distinguishable from zero, these values are extremely small.
The estimated probability that any individual exonic base carried such a deleterious Neanderthal allele was also very low—about 8.1 × 10⁻⁵ in Europeans and 6.9 × 10⁻⁵ in East Asians. In practical terms, this corresponds to fewer than one deleterious Neanderthal allele per 10,000 exonic base pairs on average.
Combining those values produced an average selective cost per exonic base pair on the order of 10⁻⁸.
Individually, these effects were weak. Collectively, however, they could shape the broad genomic landscape of Neanderthal ancestry.
The model also reproduced a prominent pattern seen in real genomes. Regions with higher densities of exons consistently retained lower average levels of Neanderthal ancestry, while regions with fewer exons preserved more. At a scale of one centimorgan, predicted and observed levels of Neanderthal ancestry showed Pearson correlations of 0.897 for Europeans and 0.710 for East Asians, indicating that the model closely matched the observed relationship between recombination, functional genomic regions, and surviving Neanderthal DNA.
Population size may explain much of the pattern
The researchers argue that the estimated strength of selection is so weak that it points toward a different explanation than widespread hybrid incompatibilities.
Neanderthals are thought to have maintained a much smaller long-term effective population size than anatomically modern humans. In small populations, weakly harmful mutations can behave almost as if they are neutral because random genetic drift overwhelms selection.
To test whether this demographic difference alone could generate the observed pattern, the researchers simulated populations after the split between Neanderthals and modern humans. They assigned mutations at exonic sites selection coefficients drawn from an empirically supported distribution of fitness effects and tracked how those mutations accumulated before the two populations interbred again.
The simulations consistently produced an excess of weakly deleterious alleles that became fixed within the smaller Neanderthal population while remaining absent or rare in humans. The range of selection coefficients inferred from the genomic analysis fell within precisely the region where these simulations predicted the greatest excess of Neanderthal-specific harmful variants.
The simulated fraction of exonic sites carrying these Neanderthal-specific weakly deleterious alleles was also on the order of 10⁻⁵, closely matching the study’s estimate for the density of selected sites.
Taken together, the genomic analyses and simulations support the idea that many of the alleles later removed from human populations had probably not been strongly harmful within Neanderthals themselves. Instead, they likely drifted to high frequency in the smaller Neanderthal population before becoming subject to purifying selection after entering the much larger human population.
The authors note that another recent simulation-based study independently reached a similar conclusion.
Why East Asians appear to have inherited slightly more Neanderthal ancestry
The analysis also revisited a longstanding difference between present-day populations.
After accounting for selection, the estimated initial proportion of Neanderthal ancestry remained somewhat higher in East Asians than in Europeans.
The best-fitting initial admixture proportions were approximately 3.60% for East Asians and 3.38% for Europeans.
However, the confidence intervals overlapped because estimates of initial admixture and the overall strength of selection are partially confounded. Similar present-day levels of Neanderthal ancestry can result from combinations of lower initial admixture with weaker selection or higher initial admixture with stronger selection.
The researchers found that if the average fitness cost per exonic base pair is assumed to be the same in both populations, the inferred initial admixture proportions become distinct, remaining consistent with earlier work indicating somewhat higher initial Neanderthal ancestry in East Asians.
The X chromosome raises additional questions
The X chromosome contains much less Neanderthal ancestry than the autosomes, a pattern that has often been interpreted as possible evidence for reproductive incompatibilities.
The researchers adapted their model specifically for the inheritance of the X chromosome, accounting for its different transmission pattern and the absence of recombination in males.
Because much less data are available for the X chromosome, estimates became substantially more uncertain.
For Europeans, the analysis still found evidence consistent with selection acting against Neanderthal introgression on the X chromosome, with confidence intervals excluding zero for the combined measure of selection density and strength.
For East Asians, however, the confidence intervals included zero, leaving weaker statistical support for selection on the X chromosome in that population.
The researchers emphasize that the uncertainty surrounding X chromosome estimates makes firm conclusions difficult.
They also explored whether the lower amount of Neanderthal ancestry on the X chromosome could partly reflect sex-biased mating during ancient interbreeding.
Based on their point estimates, if the entire difference between the X chromosome and autosomes resulted solely from mating patterns, pairings between Neanderthal males and human females would have been roughly three times as common as the reverse pairing.
The authors stress that this interpretation is highly provisional because of the large uncertainty in the X chromosome parameter estimates.
Thousands of weakly harmful alleles may have mattered together
Although each individual Neanderthal allele appears to have had only a very small effect, the researchers argue that the combined burden may have been substantial.
Their estimates imply that selection has reduced the average frequency of these deleterious Neanderthal alleles by only about 56% over tens of thousands of years. They estimate that roughly 7,000 loci, corresponding to the product of the estimated density of selected sites and approximately 82 million exonic sites, still segregate for weakly deleterious Neanderthal alleles introduced through ancient interbreeding.
Because Neanderthal ancestry began at a low frequency, they estimate that a typical present-day European or East Asian individual carries roughly 100 of these weak-effect alleles.
The earliest human-Neanderthal hybrids may have faced a much heavier burden.
Assuming the effects of thousands of weakly deleterious alleles combined multiplicatively, the authors estimate that an average first-generation hybrid could have experienced approximately a 94% reduction in fitness relative to modern humans lacking nearly all of those alleles.
The paper immediately cautions that this calculation depends strongly on assumptions about how fitness effects combine. Selection may have been soft rather than hard, interactions among mutations may not have been multiplicative, Neanderthals may have evolved compensatory adaptations, and they likely possessed their own ecological or cultural advantages. Because of these uncertainties, the authors regard estimates of Neanderthal genetic load and hybrid fitness as provisional.
A different explanation for selection against Neanderthal ancestry
The researchers conclude that the observed genomic patterns do not require widespread intrinsic genetic incompatibilities between humans and Neanderthals.
Their results instead support a simpler explanation in which differences in long-term effective population size allowed numerous weakly deleterious alleles to accumulate in Neanderthals. Once those alleles entered much larger human populations through interbreeding, natural selection slowly began removing them.
The authors do not argue that larger-effect alleles or genuine hybrid incompatibilities never existed. Such variants may well have occurred, particularly those affecting ecology or behavior, but they conclude that they are not necessary to explain the broad relationship between gene density and Neanderthal ancestry observed across the genome.
More broadly, they suggest that differences in effective population size and the nearly neutral accumulation of weakly deleterious mutations may influence patterns of genetic introgression whenever populations with different demographic histories come back into contact.
Publication details
Juric I, Aeschbacher S, Coop G (2016) The Strength of Selection against Neanderthal Introgression. PLoS Genet 12(11): e1006340. DOI: 10.1371/journal.pgen.1006340
