I'm a recently retired photojournalist & stock photographer. Also a heterosexual white male. I've made thousands of candid photos of people in Los Angeles (my home town) & NYC, & sold many of them to agency.
Yet I believe you're [mostly] in the right here. What that vendor did was wrong on multiple levels. First, it might have been illegal, given that deliberate "upskirt style" photos are banned in many jurisdictions. Not sure about LA. There's also the possible issue of harassement. I do think if you gone to a police officer s/he might have at least cited the man.
Even if legal, however, it was gross & inconsiderate. I don't think there's anything wrong with taking candid photos of interesting people at a place like Farmers Market. One might say, "Well you should ask." Okay. But that produces an entirely different kind of picture. People naturally tend to pose. Of course, too, if you intend to use the photo commercially, then you need a signed model release.
But clearly, no one wants to be photographed bending over, reading product labels. That's simply beyond reason. He had to know he was doing wrong.
Finally, there's the issue of self-interest. Were I sitting at a table at the market, drinking my coffee, seeing you walk by in your shorts might have made my day just a little brighter. Why would I ever want to do anything that would make you uncomfortable, or self-conscious to dress that way?
While people do have the right to photograph one another in public places, like any right, if it's abused, it could be rescinded or seriously restricted. Hungary, I'm told, banned the taking of candid photos, even in public, some years ago. With no exception for journalists, either. Imagine how crippling that would be to the profession.
So I'm left angry at the man on your behalf, & I'm also angry at him on mine.