To answer your question, yes. The time you take in between sets does affect your muscle growth. If you take 20 minute breaks you are truly wasting your time. Your rests should be as little as 20 seconds(fat burning) or as much as 5 minutes but never over. (5 minute breaks means your lifting as heavy as possible) Only take a break as long as your body needs so when you feel the burn stop do another set. Otherwise, you're not accomplishing anything during all those hard workouts.