Look at it this way, 95% of Marines are good and 95% of pirates are evil, and the last 5% of pirates are not even good, they more 'neutral'. Garp even said that Roger went on tantrums that destroyed entire towns (when a crewmember was 'offended'), do you think people went: "yeah, the house I worked for my entire life is destroyed but at least Roger is happy!"