Elsa/Getty; KENA BETANCUR/AFP/Getty
Serena Williams didn't always want her 5-year-old daughter in the stands during her tennis matches.
In a clip from Wednesday's episode of The Drew Barrymore Show, the newly retired tennis pro, 40, explains why she didn't let her daughter Olympia come to her tennis matches until recently.
"Is it true that you didn't want Olympia to go to matches because you were afraid you would lose killer instinct?" Barrymore asks Williams, who shares daughter Olympia with husband Alexis Ohanian.
"I just was worried I'd get distracted because I would be like, 'Wait is she drinking? Is she doing this?' " Williams shares. "She came to a match once, like super brief, and I was like, 'Oh my God, is she wearing sun cream?' It's nuts."
"I would get stressed out and be like 'Wait a minute, oh my goodness I've got to serve. This makes no sense,' " she adds with a laugh. "So yeah, I've never really had her at matches until this summer actually."
Never miss a story — sign up for PEOPLE's free daily newsletter to stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human interest stories.
Olympia showed her support for her mom during the athlete's final U.S. Open this year. Williams' big moment came after she announced her 27-year tennis career was coming to an end following the tournament.
Noting that Olympia came to the match that is "the most watched moment in tennis history," Barrymore asks Williams if having her daughter there did anything to "assuage [her] fears" or "calm something down."
"Absolutely," Williams says without hesitation. "Apparently she was saying like, 'Go Momma, I'm so proud of you,' and I was like, 'What?' "
"So then she comes back after I had lost in Cincinnati, she came back and she's like, 'It's okay, Momma, you just have to do what you feel,' and I literally wanted to cry. It was like, 'Do what you feel. Do what your heart tells you.' It was the sweetest thing ever," she adds.
Williams lost what was likely the final match of her legendary career at Arthur Ashe Stadium on Sept. 2. After battling it out over three intense sets, the 23-time Grand Slam champion lost to Australia's Ajla Tomljanović.
The Drew Barrymore Show airs weekdays on CBS. Check local listings.