Shields, 51, died on Friday in his Trump Tower apartment in New York after a suspected overdose. A close friend revealed to Radar that the family is heartbroken by his passing.
"They're all crushed," said Gregory Broccoli. "His children are really upset. There are counselors with them."
Shields shares four children with his estranged wife, Jill Schwartzberg.
The New York city businessman dated Frankel, 47, on and off for two years.
As Radar first reported, Shields even bought Frankel an engagement ring.
The Skinnygirl founder was photographed sobbing uncontrollably at his funeral on Monday.
The friend also shared how Shields' mother is reacting to the news.
"His mom is really upset as well. She keeps saying, 'That's my baby,'" said the friend.
Broccoli believes Shields' death was "100 percent" an accident, Radar reported.
Broccoli also revealed Shields took medications for his bad back, which he complained about for years.
Celeb addiction expert and author Kimberly Friedmutter told Radar exclusively that an overdose death can be the most crushing type of loss.
"Devastation can be more easily processed when one feels the tragedy was inevitable. In the case of losing a loved one to addiction/accidental overdose, it never really heals," the expert explained.
We pay for juicy info! Do you have a story for RadarOnline.com? Email us at email@example.com, or call us at (866) ON-RADAR (667-2327) any time, day or night.
For the biggest celebrity news stories of the day, controversies, crime, and other hot topics listen to our new podcast ‘All Rise’ below!