During the interview, the Pearl Harbor actor also appeared to claim his marriage to Garner was the reason for his addiction to alcohol.
According to Page Six, one insider from the industry who is very familiar with the former couple called his statement "disgusting."
"Jen has been the only one picking him up time and time again when he's done horrible things - fallen off the wagon or worse," the source explained. "Jen stood by him because she loves him. This is the mother of his children."
"Anyone who knows addict behavior knows how untrue those comments could be for any addict, but particularly Ben," another source said. "It's one of the first tenets of AA not to place blame for your disease."
As Radar previously reported, the 49-year-old actor spoke candidly about his alcoholism addiction, revealing he'd "still be drinking" had he not separated from his ex-wife.
"It's part of why I started drinking...because I was trapped," The Last Duel actor confessed. "I was like, 'I can't leave' cause of my kids, but I'm not happy. What do I do?"
Affleck added, "What I did was drink a bottle of scotch and fall asleep on the couch, which turned out not to be the solution."
After his comments sparked outrage, the father of three tried to clean up his comments on The Jimmy Kimmel Live.
Affleck claimed the media had manipulated his words to make him look like the "worst, most insensitive, stupid, awful guy" after noting that the situation "hurts" him.
A source close to the actor defended him, saying, "I do think that the quotes have been taken out of context. If you actually listen to the interview, he says everything in a different context." Although, the insider admitted, "he didn't need to act like he was talking to his shrink" during the interview.
Affleck and Garner first met while working on the set of 2003's Daredevil before tying the knot in 2005. Their 10-year marriage ended in 2015. They share kids Violet, 16, Seraphina, 12, and Samuel, 9.
The actor recently reconciled with his ex-fiance, Jennifer Lopez, 17 years after calling off their engagement. In an interview with WSJ Magazine, he gushed that their reunion is "beautiful to me."
However, Radar reported that the singer is also "livid" after learning what her man said about Garner.
Sources tell us the pop star felt her partner had betrayed their agreement concerning sharing personal details with the public.
"They both agreed that they wouldn't talk about their private lives in any depth after what happened last time they were together," the insider dished. "At events, the press is told 'no questions about JLO and Ben, only questions about the film.' Now he shot off his big mouth and dragged her into it."