Michael Richards is getting candid about his life in his memoir Entrances and Exits, out June 4, and he’s not sugarcoating any of the trauma — including discovering that he was lied to about who his father was for his entire childhood.

The Seinfeld star’s mother had maintained his father had been killed in the war, but as he got older, he realized the math didn’t add up. She then told him he’d died in a car crash.

Eventually, after trying to track down a man he thought could be his dad, he pressured her for the truth, and she confessed that his conception had been a product of her sexual assault.

Richards’ mother wanted to get an abortion, but they were illegal, so she initially put him up for adoption. She later changed her mind, however, and raised him as a single mom. The last name Richards had been made up, and the actor, now 75, was devastated.

Richards says the truth had a profound impact on him, and he had a lifelong feeling of not being lovable. He also says that he’d been harboring anger for a long time.

“I had to come to terms with knowing I was unwanted or that my mother wanted to get rid of me,” he says.

“It is definitely something that I have had to look into over the years to discover how my anger arises out of a feeling of inferiority. I have a temper, and it stems from that unwantedness, not being acceptable, not being understood, not being good enough to be liked or even loved,” Richards continues.

The comedian says that insecurity led him to turn down opportunities. “I said no to the offer of a star on the Hollywood Walk of Fame. I didn’t feel deserving,” he says. “I said no to hosting Saturday Night Live twice because I didn’t feel good enough.”

Never miss a story — sign up for PEOPLE’s free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer​​, from celebrity news to compelling human interest stories.

That feeling of not being good enough carried over to when he was starring as Kramer on Seinfeld.

“I felt every week during Seinfeld I was never really satisfied with my performance and that I could always do better. And as the stakes got higher, the pressure to always be good was difficult. And to accept myself even when my audiences were loving me and the awards, accolades and all the offers were coming in — it felt almost impossible sometimes, way too overwhelming,” he says.

“I’d think, ‘I don’t like myself as much as they like me. They wouldn’t like me if they knew the real me, the person behind the character that they’re laughing at,’ ” Richards continues.

Entrances and Exits will hit shelves June 4, and is available now for preorder, wherever books are sold.

To read more about Michael Richards, pick up the latest copy of PEOPLE on newsstands now.

If you or someone you know has been sexually assaulted, please contact the National Sexual Assault Hotline at 1-800-656-HOPE (4673) or go to rainn.org.