I believe Vergo though too much of himself with his fighting capabilities. When Vergo was fighting Sanji, he only fought with his legs to counter Sanji's legs. With Smoker, Vergo used his hardened bamboo stick to counter Smoker's sea prism stone weapon, and also planned to use his hardened bamboo stock against Law in order to counter Law's sword.
I believe Vergo was top cocky and thought himself so competent at fighting that he could copy anyone's fighting style and take them down.
He was wrong.