When pollution occurs in an environment, populations present suffer numerous negative and immediate effects on their life history traits. Their evolutionary potential to live in a highly stressful environment will depend on the selection pressure strengths and on the genetic structure, the trait heritability, and the genetic correlations between them. If expression of this structure changes in a stressful environment, it becomes necessary to quantify these changes to estimate the evolutionary potential of the population in this new environment. We studied the genetic structure for survival, fecundity, and early and late growth in isogenic lines of a Caenorhabditis elegans population subject to three different environments: a control environment, an environment polluted with uranium, and a high salt concentration environment. We found a heritability decrease in the polluted environments for fecundity and early growth, two traits that were the most heritable in the control environment. The genetic structure of the traits was particularly affected in the uranium polluted environment, probably due to generally low heritability in this environment. This could prevent selection from acting on traits despite the strong selection pressures exerted on them. Moreover, phenotypic traits were more strongly affected in the salt than in the uranium environment and the heritabilities were also lower in the latter environment. Consequently the decrease in heritability was not proportional to the population fitness reduction in the polluted environments. Our results suggest that pollution can alter the genetic structure of a C. elegans population, and thus modify its evolutionary potential.