From professional huggers to rent-a-cuddler businesses, some individuals have a stronger need for physical affection than others. And science explains these individuals may reap several health benefits. Hugging and touch has been linked to everything from a stronger immune system to lower stress to decreased risk for depression.

But not everyone likes to be touched, and a simple, short hug can make them feel as frosty as that pint of ice cream in your freezer.

“Unfortunately it’s unknown why some people avoid hugs. It could be that those folks are hard to find and it doesn’t seem like a prevalent enough issue to warrant research monies,” explains Tiffany Field, PhD, director of the Touch Research Institute at the University of Miami School of Medicine.

What we do know: Hugging has been around for millennia, when apes and other animals began greeting each other in a similar manner, according to a 2012 study in the journal Comprehensive Psychology. In humans, some are first exposed to this feeling as infants when they are held by their guardian during feeding sessions.

Even though it’s been a part of their lives, some people feel uncomfortable with closeness. We spoke to Field and Aline Zoldbrod, PhD, a licensed psychologist and sex therapist in Boston and the author of SexSmart: How Your Childhood Shaped Your Sexual Life and What to Do About It to find out why, plus how to show you care in other ways. (Speaking of going hands-free, avoid touching these places in public restrooms.)

Lack of physical affection during childhood

Even though our endocrine systems (aka hormones) and emotions are set to respond to human-to-human contact, if you don’t learn early that touching is okay and safe, it doesn’t come naturally.

“In our culture, people learn or don’t learn to enjoy touch, and to link it to good things like love, safety, warmth from their experiences with their caregivers as children,” Zoldbrod says. “Simply put, people who are hugged in their families are more likely to enjoy being hugged.”

Those who grew up in broods who were kind and empathic, trustworthy, physically affectionate, and sexually appropriate likely view physical touch as an expression of:

• Safety

• Caring

• Warmth

• Soothing

• Love

• Pleasure

• Relaxation

That translates to an openness to physical closeness as an adult.

“Lots of people grew up in families where they were not this lucky,” Zoldbrod says.

Those who grew up in frightening, neglectful, or abusive homes—even if the abuse wasn’t physical—may have totally different associations to touch. They possibly associate it with:

• Fear

• Controlling

• Pain

• Awkwardness

• Numbness

• Guilt

• Danger

Negative experiences with touch in the past

Sad but true, according to Zoldbrod: “More than a third of the population has had experiences with physical sexual and emotional abuse.”

The National Domestic Violence Hotline reports that about 25 percent of adult women and 14 percent of adult men in America have experienced physical violence by a partner at least once in their lives (not to mention other family, friend, or workplace traumas—or witnessing someone else being abused).

“These experiences are all stored in the body, and they interfere with experiencing pleasure from touch. Negative associations [with] all kinds of negative experiences leave their mark on the body. When trauma is stored in implicit memory in the body, people don’t like to be hugged or touched. It makes them feel out of control and vulnerable,” Zolbrod says.

Zolbrod adds, that these individuals may not even understand why they feel this way, because “we often only define ‘trauma’ as overt sexual trauma like molestation rape or incest. That’s only a part of the kind of trauma that is stored in people’s bodies.”

Feelings and traumatic memories of many kinds are stored in the body as images and physical sensations, she continues, and they can interfere with experiencing pleasure from touch.

Anxiety or depression

During and after a hug or a cuddle session, the body pumps out oxytocin, the “love hormone.” According to University of California San Diego scientists, low levels of this brain chemical is a contributor to both social anxiety and depression. And if it’s already low—and doesn’t get a boost from touch—physical bonding doesn’t feel all that great, natural, or desired. (In case you missed it, here’s what happens inside the brain and the body as a whole when you’re hugged.)

Discomfort with physical appearance

Those junior high school bullies can have a long-lasting impact.

“A young girl who is objectified or teased about her breasts can wind up feeling uncomfortable and ashamed of her breasts for her entire lifetime, thereby not feeling good about being hugged by anyone. Even by someone who loves her,” Zoldbrod says.

This trauma need not be direct, either. “Microaggressions” from society as a whole that make you self-conscious about your body can make you hesitant to touch.

What to do if you don’t enjoy hugs, but would like to

Say you feel slightly uncomfortable, yet desire to be a bit more physical touch-friendly. Zoldbrod suggests that you get curious about your past and what may have lead you to steer clear of physical affection. You may also find body mapping helpful to determine if any zones, in particular, are particularly sensitive.

“Did anything happen in your family to make you not like touch? If so, consider exploring the topic with the help of a trauma-informed mental health professional. You can change your emotions related to touch, but it takes patience, compassion, and special training,” she says.

Regarding that special training, certain modes of psychotherapy focus heavily on how feelings are stored in the body. Experts in sensorimotor therapy and somatic experiencing can be particularly helpful in the treatment of related conditions.

How to show your affection in other ways

Regardless of if [the person is] pro- or anti-hug, it’s best to ask first, “Hey, can I give you a hug?” before diving in. Just like with more intimate acts, consent is key here until you are familiar with each other’s preferences.

“If someone doesn’t like being hugged you can just show you care by being verbally interactive,” Field says.

Or you can try these alternative greetings, which, according to a 2014 study in the American Journal of Infection Control, is less likely to spread germs, too:

• A friendly nod

• A wave

• A fist bump

Whatever option you decide on, “let the people who don’t like touch call the shots. Give them their space,” Zoldbrod says. “Touching them against their wishes will just make them even more aversive to touch.”