"They've sent message to his people via their people that he needs to keep his mouth shut about their mother," a source exclusively told Radar.
The trouble started earlier this year, when rumors popped up linking the Grammy winner to the late princess.
Diana's former butler, Paul Burrell, reportedly claimed he used to sneak the singer into Kensington Palace to see the late royal in the '90s.
After learning the surprising news, Diana's kids were quick to strike the rumors down.
"Bryan was shocked they took it so personally but is telling friends that perhaps if their father was nicer to their mother while they were married, Diana wouldn't have sought comfort from him in the first place," the source said.
As for Adams, his admiration for Lady Di seems rooted in truth. He even released the song "Diana," with lyrics boasting how the princess drove him "wild," how she is the "queen" of his dreams and questioning what she was doing with a guy like her husband.
"Bryan thinks it's incredibly rude and rich of William and Harry to react like this," said the source.
We pay for juicy info! Do you have a story for RadarOnline.com? Email us at firstname.lastname@example.org, or call us at (866) ON-RADAR (667-2327) any time, day or night.