User blog comment:AYET/The Dressrosa arc traitor/@comment-6920514-20130912052510/@comment-5962458-20130914101935

Well, in chapter 700 Kin'emon said that after the shipwreck only 2 samurai and Momo made it the island of Dresrossa. So the other one either drowned or drifted on another island.