Hi, I'm Peter ~There's something that always get's to me and makes me fell less of a person...Can you help?

What does it mean when some people at work, or in cl get quiet around me, or refrain from talking to me?( keep in mind I'm not talking about everybody).I personally think that maybe it has to do with how I carry myself~I need more confidence, is something wrong with me? Then I think, "The people that are more mature minded don't do that to me". Also, I 'can' be a little inhibited....okay, a lot,but still most people just don't know me ~ Sometimes it's hard being me.