Many men will inherently associate a trans man with some degree of vestigial femininity. As such, any association to being trans is an insult to their masculinity, and by extension, their perceived value as a person.