At what age is it considered rude to call someone a "boy" instead of a "man"

It depends on the context and the person saying it. There are several ways to say it in a derogatory manner or a familial manner.

I don't really mind, you'll always be a boy to the elders that watched you grow. My grandma is kinda like that, even though I'm much taller than her now.

It's not that big of a deal, but I'd say calling a 18 yr old male a boy is a bit insulting. How about "young man"? Or does that make me sound really old.

I was a boy in my twenties. Thought I knew stuff, when obviously I still had much to learn about becoming a man.

My son's, will always be my boys and so will all of their friends. If they are "men", they have no problem with an older person calling them boys.

I don't think it matters much.

I don't know! As Will said sort of said, it really does depend on the circumstances in which one is calling a young "adult", OR an older person a boy or a girl, or referring to them that way. People use those references all the time without any offense intended or taken.

Age doesn't matter. Some "men" will always be boys...and I don't mean that in a good way.

Unfortunately it has become more of a political and racial question thanks to political correctness. The word 'boy' is not offensive otherwise.