Your instinct is correct and worth keeping. It is legal in a public place, which is not the same as it being all right. What works for me: shoot the food, the stall, the queue, the hands. Nobody minds. If you want a person's face, buy something from them first and then ask. Almost everyone says yes once you are a customer rather than a tourist with a lens.