When small groups of modern humans interbred with Neanderthals tens of thousands of years ago, they did not simply inherit a few useful genes—they may also have acquired thousands of weakly harmful genetic variants that linger in human genomes today. By reconstructing that ancient genetic exchange with large-scale evolutionary simulations, the study argues that the long-lasting consequences of Neanderthal ancestry can largely be explained by the extinct relatives’ small population size and accumulated mutation load, without requiring widespread genetic incompatibilities between the two groups.
For years, genetic studies have painted a complicated picture of humanity’s ancient encounters with Neanderthals. People living outside Africa inherited roughly 2–4% of their DNA from Neanderthals after the two groups interbred, and some of those inherited genes appear to have helped modern humans adapt to new environments. Yet the same inherited DNA is strikingly scarce around many important genes, raising a major evolutionary question: why did so much Neanderthal DNA disappear from precisely the regions where genes matter most?
The study tackles that mystery from an unexpected angle. Instead of assuming that modern human and Neanderthal genes were broadly incompatible with one another, the authors explored whether Neanderthals simply carried a heavier burden of harmful mutations because they had lived in much smaller populations for hundreds of thousands of years. If so, natural selection acting against those mutations alone might explain many of the genetic patterns still visible in people today.
A long period of isolation may have changed the genetic balance
Previous analyses of ancient Neanderthal genomes had already shown that Neanderthals possessed unusually low genetic diversity compared with living humans. Earlier work also indicated that they experienced an extended population bottleneck lasting roughly ten times longer than the bottleneck associated with modern humans leaving Africa.
Small populations face a fundamental evolutionary problem. Natural selection becomes less efficient at removing mutations that slightly reduce survival or reproduction because random genetic drift plays a larger role. Over many thousands of generations, these weakly harmful mutations can accumulate.
To investigate the consequences, the authors constructed forward-time evolutionary simulations that followed the accumulation of protein-altering mutations across realistic human exomes. Their simulations used estimates of the distribution of fitness effects for nonsynonymous mutations derived from previous human studies, while assuming that humans and Neanderthals experienced the same types of new mutations. Any differences therefore emerged solely from their contrasting demographic histories rather than from different mutation processes.
The simulations began with a shared ancestral population of 10,000 individuals evolving for 44,000 generations to establish equilibrium. The population then split into modern humans, which remained at 10,000 individuals, and Neanderthals, whose simulated population size dropped to 1,000 individuals. The two groups evolved separately for another 16,000 generations, corresponding to an estimated divergence roughly 400,000 to 470,000 years ago, before simulated interbreeding occurred.
Smaller populations accumulated a much heavier mutation load
The results pointed to a dramatic genetic imbalance before modern humans and Neanderthals ever met.
When harmful mutations were assumed to have additive effects, the median simulated Neanderthal possessed only about 63% of the fitness of the median modern human. Under a model in which harmful mutations were completely recessive, the difference became even larger, with median Neanderthal fitness falling to about 39% of the human level.
The simulations indicated that this disadvantage was not caused by rare catastrophic mutations but largely by the accumulation of many weakly harmful variants whose selection coefficients fell within a narrow range. These mutations were weak enough to persist in the smaller Neanderthal population while being more efficiently removed in the larger human population.
The authors argue that this difference alone could have profoundly influenced what happened after the two groups interbred.
Harmful mutations could explain why Neanderthal DNA disappeared from important parts of the genome
One of the most widely discussed observations from previous genetic studies is that Neanderthal ancestry is unusually rare near conserved regions of the human genome, where mutations are more likely to be harmful.
Some researchers had interpreted this pattern as evidence that human and Neanderthal genes often interacted poorly, producing widespread reproductive incompatibilities.
The new simulations offer another explanation.
If Neanderthals entered modern human populations carrying many weakly harmful mutations, natural selection would gradually eliminate those mutations. Because neighboring stretches of DNA tend to be inherited together, nearby neutral Neanderthal DNA would also decline in frequency simply because it remained linked to harmful variants.
The authors developed mathematical models alongside their simulations to estimate this process. They found that selection acting on a single weakly harmful mutation could reduce Neanderthal ancestry across approximately one million base pairs surrounding that mutation.
In one example examined by the study, a harmful mutation with a selection coefficient of 5 × 10⁻⁴ initially entered the human population at a 2% frequency through Neanderthal ancestry. After 2,000 generations, the mutation’s average frequency fell to roughly 0.7%. Even DNA located about 60,000 base pairs away from the harmful mutation still experienced about a twofold reduction in Neanderthal ancestry because of genetic linkage.
The authors argue that if many such mutations were scattered throughout the Neanderthal genome, they could collectively produce widespread depletion of Neanderthal ancestry around functional regions without requiring large numbers of incompatibility genes.
The earliest human–Neanderthal hybrids may have faced strong natural selection
The study also explored what happened immediately after interbreeding.
Under the additive mutation model, selection against the inherited Neanderthal mutation load proved extremely rapid. Simulations suggested that to end up with the roughly 2.5% Neanderthal ancestry observed after 2,000 generations, the initial population would have needed about 10% Neanderthal ancestry immediately after admixture.
Most of the reduction occurred within only about 20 generations.
During these earliest generations, individuals differed substantially in how much Neanderthal ancestry they carried, allowing natural selection to remove those with larger inherited burdens. Later, once ancestry became more evenly distributed across the population, selection became much less efficient because harmful variants were spread across many different genomic regions.
This slowing occurred even though weakly harmful Neanderthal mutations continued to persist.
Recessive mutations produced a very different outcome
The simulations changed dramatically when harmful mutations were assumed to be recessive.
In this scenario, Neanderthal DNA initially entered modern humans at only about 1% frequency but gradually rose to around 3% over the next 2,000 generations.
The increase occurred because Neanderthal chromosomes often carried harmful mutations at different positions than human chromosomes. When paired together in hybrids, each chromosome could mask the other’s recessive harmful variants. This phenomenon, known as dominance heterosis, temporarily made mixed ancestry genetically advantageous despite Neanderthals carrying a greater overall mutation burden.
Most of this increase happened during roughly the first 500 generations after interbreeding.
The authors emphasize that this effect depends on assumptions about how harmful mutations behave genetically. When they simulated partially recessive mutations with a dominance coefficient of 0.1, the results resembled the additive model much more closely than the fully recessive one. Neanderthal ancestry declined from an initial 10% to approximately 5.5%, although it briefly increased before settling toward its long-term level.
Some harmful Neanderthal variants may still remain today
Although natural selection removed many harmful variants, the simulations indicate that it did not eliminate all of them.
To estimate what remained, the authors compared three simulated human populations: one with constant population size, one experiencing only the out-of-Africa bottleneck, and one experiencing both the bottleneck and Neanderthal admixture.
They divided harmful mutations into weakly deleterious variants, with selection coefficients below 0.0005, and more strongly deleterious variants above that threshold.
The simulations showed that neither the population bottleneck nor Neanderthal admixture substantially altered the burden of strongly harmful mutations. Instead, their effects were concentrated among the weaker mutations.
Both the out-of-Africa bottleneck and Neanderthal introgression independently reduced fitness by roughly 1% through these weakly harmful variants. The additional burden introduced through Neanderthal ancestry declined only slowly over the following 2,000 generations, indicating that many weakly deleterious variants could persist for long periods.
According to the authors, these surviving variants are unlikely to play a major role in severe single-gene disorders because they generally have small individual effects. However, they may contribute more substantially to complex traits influenced by many genes.
The findings challenge one explanation without ruling out another
The authors do not argue that genetic incompatibilities between humans and Neanderthals never existed.
Instead, they conclude that the widespread reduction of Neanderthal ancestry near genes can largely be explained by differences in mutation load alone.
They note that several observations remain consistent with the possibility of at least some reproductive incompatibilities. These include long stretches of the genome almost completely lacking Neanderthal ancestry, reduced Neanderthal ancestry near genes expressed in the testes, and unusually low Neanderthal ancestry on the X chromosome.
However, they suggest that only a relatively small number of such incompatibilities may be needed. In contrast, explaining the broad genome-wide depletion solely through incompatibility genes would require hundreds or even thousands of subtle interactions, which they argue seems less likely given the relatively recent divergence between humans and Neanderthals and the substantial evidence for successful interbreeding.
The implications extend beyond ancient human evolution
The authors argue that their findings may apply far beyond Neanderthals.
Denisovans, another extinct human relative, also appear to have lived in relatively small populations and contributed DNA to some present-day human groups. Because Denisovans experienced similarly low genetic diversity, the same evolutionary processes may have shaped the fate of Denisovan DNA after admixture.
More broadly, the simulations point to a principle that could affect conservation biology whenever populations with very different histories exchange genes.
When individuals from a more inbred population contribute DNA to a larger, more genetically diverse population, weakly harmful mutations may spread rapidly into the larger population but require many generations for natural selection to remove. The authors suggest this possibility should be considered during conservation programs involving genetic rescue, where gene flow between populations is intentionally encouraged.
Questions that remain
The authors emphasize that they have not precisely measured the amount of harmful Neanderthal genetic material that survives in living people. Doing so will require better estimates of how mutations affect fitness across different genes and a deeper understanding of recent human demographic history.
They also did not include selection for newly beneficial mutations in their simulations. Such beneficial variants could have influenced how much Neanderthal DNA survived, particularly during the earliest generations after interbreeding.
Finally, the authors suggest that additional ancient human genomes may eventually allow scientists to track how Neanderthal ancestry changed through time, potentially offering new insight into how different classes of harmful mutations behave and how ancient interbreeding shaped the genomes carried by people today.
Publication details
K. Harris et al, The Genetic Cost of Neanderthal Introgression, Genetics (2016). DOI: 10.1534/genetics.116.186890
