Yes, there exist cases of varicocele where the man has experienced a decrease of libido, probably due to abnormal testosterone levels. Nonetheless, although it is possible, it is uncommon.
Also, this diminished libido as a result of varicocele can lead to sexual impotence, but only in very severe or extreme cases.
Suggested for you: What is a Varicocele? – Causes, Symptoms & Treatment.