Genetics do matter!
Genetics play an important role in both how we respond to training as well as how much muscle we can potentially build. For both men and women, completing the exact same training regime (12 weeks in this study) leads to a wide variety of results with some individuals gaining more strength and size compared to others.
Myostatin is an example of a protein which limits muscle growth, probably to stop too much muscle from affecting your metabolic and mechanical efficiency. Different individuals have different levels of myostatin and thus different maximal muscle building potentials.
That being said, results can be achieved with a solid training routine, diet and determination no matter what genetic cards you have been dealt (though your determination may also be genetically controlled!) and most people probably don’t reach their genetic maximal potential. It all comes down to how fast you progress and what type of exercise your body responds best to, though more research is needed on this front.
Sources
Hubal MJ et al. Variability in muscle size and strength gain after unilateral resistance training. Med Sci Sports Exerc. 2005 Jun;37(6):964-72.
Thomas M et al. Myostatin, a negative regulator of muscle growth, functions by inhibiting myoblast proliferation. J Biol Chem. 2000 Dec 22;275(51):40235-43.