Yes, it's possible. Varicocele is a varicose structure that increases the temperature of the testicle, often affecting the production of sperm. If the affectation is important, severe oligospermia may appear, which will be difficult to recover, even when the varicocele is surgically operated.
This is because if the germ line of the testicle (sperm producing cells) is damaged, this damage will be irreparable, which can occur in varicoceles over time.
Suggested for you: What Diseases Are Associated with Oligospermia?.