I know that the chromosome that determine gender comes from your father, but is it possible for me to share traits with the men on my mother's side of the family?
Children are a blend of the two parent's chromosomes. You get half from your mother and half from your father. Just one chromosome determines your gender and some characteristics that are only found on that chromosome. The rest control other features. Some are on/off type genes - either the gene from your mother or your father will dominate. Others are blends, where both genes combine to give a unique combination to you.
For any specific feature that your mother inherited, there is only a 25% chance that one of her siblings has the same feature. There is only a 50% chance that you inherited that feature from your mother. So while you might share something in common with an uncle, the odds aren't very high.