Without a suitably matched Amplifier, your speakers / headphones will sound dead.
Yes the headphones / speakers themselves will have the greatest impact, especially regarding the range of frequencies produced (as you can't produce 20hz from 2" speaker cones). However, the Amplification of the source is critical as better Amps will greatly increase the sound stage and provide rich detail throughout the spectrums.
Often when people upgrade their amplifiers, they say they hear new levels of sound. Richer bass and more tight mids. Highs often sound much more crisp and the clarity across the ranges greatly increases.
Sometimes people buy very expensive speakers / headphones and find that they sound dull / quiet or 'flabby'. Often this isn't the result of the speakers / headphones, but instead it's an issues with the headphones / speakers not being driven correctly. The correct amp should match the hardware accordingly!
Please contact us if you have any queries, we will be happy to help you find the right solution. Often amplifiers come with all kinds of interesting additional functionality allowing you to get more for less.