Would someone that has mastered Observation, like Luffy, Zoro & Sanji, forget someone affected by the Hobi hobi no mi, like Robin?
They weren't shown reacting to have been affected by it, but that could've just been then focusing on their fights.
If Armament lets you touch a logia, then shouldn't Observation let you resist mind altering fruit?