Why do people ociate "growing a pair" with the fear of asking a girl out?

...but then they go on and ociate "growing a pair" with the willingess to get in a boxing ring and beat the crap out of a guy. How do you put asking a girl out and clocking a guy until he is purple together? Isn't that a bit morbid? I think the phrase needs to change.