I said this before and I'll say it again, Observation Haki will only help to a certain extend against strong opponents, they are fast and accurate to the point where knowing their attacks won't mean a 100% chance dodging them.
Imagine Luffy fighting Kizaru or Akainu or Shanks or some powerful guy, even if Luffy was in his best conditions, he still won't be able to avoid hits completely.
Luffy was totally exhausted and even if he knew that attack was coming there is a good chance he won't be able to react. Not to mention the fact that he was surrounded by strong enemies from every direction, in other words he senses "Danger" from all directios making Observation Haki partially insufficient.
Another point is the fact that "Haki" in all it's form is nothing but the manifestation of one's very soul, and Luffy was in a situation where he was extremely tired both physically and mentally, so his Haki wasn't as powerful as usual.
Sanji was no different, his trauma was being revived and his soul is suffering and he was living in shock, so it wasn't weird that his observation haki wasn't at it's best, and again, Niji was super fast so lets not neglect this fact.
Finally, my theory about Observation Haki is that it works both ways, it may help you identify others presence but I'm assuming it could also hide your "own" presence, an example to that is Aokiji appearing right behind Doflamingo, a strong Haki user, and he had no clue whatsoever.
Ah, last but not the least, Why didn't Luffy use Conqueror's Haki? This reminds me of why didn't Whitebeard use Conqueror's Haki in the Summit War. To be honest with you, this one's been bothering me as well for a long time now, and sadly, I have no good guess at this one :(
That's all I got ! I hope this helped you be a bit more acceptant to what happened :)