For some reason people have a problem with ma'am and sir. I don't get this. It's like in this world where disrespect is now the norm, any appropriate titles are considered rude. With men, if they say "I'm not my father, don't call me sir.", sometimes I'll ask if they like ma'am better, since sir was acknowledging that they were male. I know, rather rude of me, but still. Once I had an older woman almost yelling at me because prostitutes used to be called madams and ma'am is a shortened word for madam. Personally I thought it was better than "Sorry, old lady."
Ma'am is simply a term to use if you are married. Miss if you're not. Ms. in front of a name for divorcees. Sir is good at all times. So I just look when people call their four-year-old daughters ma'am. But it's not something to get upset about. I usually try to not call them that if a person asks me not to, but get over it. You could be called a lot worse.