Well they have held grudge against Luffy ever since he defeated Crocodile and realise that Luffy is Joyboy and tried to eradicate him in Egghead