Now the actor, who recently began dating actress Minka Kelly, claims in court papers that Drake-Lee is "punishing" him for his "hard work" on the hit ABC drama, according to People.
As Radar previously reported, Drake-Lee headed to court this summer in hopes of gaining full custody of the couple's two children, Sadie, 3, and Maceo, 2, even after Williams requested joint legal and physical custody following their split.
Recent legal documents from the Superior Court of Los Angeles have indicated that Williams, 36, claims that "Aryn uses the fact that I work to support our family to marginalize me as a parent.
"I am in my 9th year on Grey's Anatomy. The structure and demands of my work schedule are the same as they were when we married and when we decided to have our wonderful children. Working full time did not keep me from being a doting parent then and it should not now. When Grey's is on hiatus (approximately April through late July), it is still my responsibility to earn an income, provide for our family and further my career," the documents said.
"Despite Aryn's refusal to recognize the time, effort and strategy required to be a self-employed professional in my line(s) of work, and her attempt to paint my work as purely recreational, all of the trips she listed in her Declaration generate the income used to support our family. I am always trying to work more efficiently so that I can free up as much time as possible to be an even more present parent to Sadie and Maceo," Williams' documents state.
According to the star, "My hard work has allowed Aryn the incredible privilege of being a stay-at-home mom, with a full-time nanny, able to be with our children when she wants. She is now punishing me for providing that privilege."
But Drake-Lee is reportedly objecting to Williams' posting photos of their kids on social media and in a statement to PEOPLE, her attorney said, "Aryn continues to work tirelessly to maintain Mr. Williams' bond with their children and it is important to remember that his allegations are one-sided. It is a shame that Mr. Williams has chosen to place the children and his family's transition in the public domain."
Drake-Lee has allegedly demanded sole custody of their two kids and also asked the court to restrict Williams from introducing new girlfriends to the kids before any future relationships hit the six-month mark.
Was that a slap at Williams for dating former Friday Night Lights star Kelly?
As Us Weekly reported, his romance with Kelly, with whom he shot a video game in Paris, was revealed when they went to the movies together on July 10.
It's not the first time a Grey's star has been embroiled in scandal, as Radar has reported.
Williams, who split from his wife after five years, has complained about the controversy, "All of a sudden mother***ers are writing think-pieces that I somehow threw a…relationship — like, the most painful experience I've had in my life with a person I've loved with all of my heart — that I threw a person and my family in the trash because a girl I work with is cute."
Williams has insisted in legal papers about his children, "I am equally capable of caring for Sadie and Maceo. When I was not working on Grey's Anatomy, and during the show's hiatus, I fed the children, changed them, bathed them, dressed them, designed activities, played with them, read to them and put them to bed. I have always played a constant and significant role in their lives.
"Aryn is unilaterally parenting our children without any input from me; her marginalizing of me as their father is deeply disconcerting," he alleges.
On Wednesday, Drake-Lee responded to Williams' previous "claim that he has a First Amendment right to publicly post images of the parties' children" on social media.
In court documents obtained by PEOPLE, Drake-Lee stated that she felt Williams' Instagram posts with photos of their kids threaten their safety and welfare.
Williams has reportedly already deleted photos of the kids from his social media account.
We pay for juicy info! Do you have a story for RadarOnline.com? Email us at email@example.com, or call us at (866) ON-RADAR (667-2327) any time, day or night.