Some people do that to me as well. I find that if I linger around the corner for a while and photograph other stuff, then my presence becomes less obtrusive. By being there for a while I become a regular fixture like anything else on the street. People ignore me. Then I do my thing. Blend in by belonging there.
Even when trying to walk slowly my legs walk quickly, always in a rush to take me somewhere where I don't them to. Force of habit of always walking fast to cover as much of the large area. I stare downwards at the screen, never bring the camera to my eye, so it should look like I'm just toggling settings.
If I sit down and people are walking past, they can see my taking photos, but my reckoning is; i was here first and you're walking in my shot. Yesterday i walked past a barbers, took a quick photo, stopped, took a deep breathe, walked back, aimed, a few seconds felt like minutes, and they all stared and I ran away...no they didn't, they didn't notice. It helps seeing other people with mammoth cameras who look like photographers, then I make sure to photo them and want them to notice.
I take hip shots with the compact camera by my side walking past people, a very observant homeless man (all people sat/stood watching things past by catch me) who was too threatening to take a photo of but i was too stupid to care called my back after the second time I walked past...but just because the camera is by my side sat vertically doesn't mean i take a photo...
That's the trouble with 'sneaky' things like hip shorts. I use them too at times, but I find just being direct about it is better. Then just smile and be friendly. Tell them you think they look really cool and if you feel they are safe then offer to show them the photo. If people get angry, apologize and walk away. I doubt most average people will pick a fight over photos.