Armstrong stepped away from Livestrong in May but that hasn't dissuaded former supporters like Michael Birdsong from going ahead with the legal action.
"It's like when you find out there is no Santa Claus, along those lines, you know?" Birdsong told CBS News.
Birdsong estimates he personally donated $50,000 to the charity and as a volunteer "mentor," the avid amateur cyclist pulled in another $65,000 from family and friends.
For one thing, Birdsong said that he believed -- and that most other donors did too -- that Livestrong gives the millions it raises to clinical research projects. Not so.
"What we began investing in, instead of clinical research, was an ongoing dialogue with patients, with survivors, with their families, to gauge the long-term effects that cancer takes on a family and on one's livelihood and on one's psyche, on one's body," Livestrong communication's chief Katherine McLane told Sharyl Attkisson for her CBS Sunday Morning cover story.
That's not good enough for Birdsong who claimed he saw many "questionable expenses" during the years he was involved.
"They sent, like, five staff members for 21-plus days to France to just follow him Armstrong around -- and tweet about it," he said. "And they had this really big party on the Champs-Elysees the final day of the tour. How is this effective?"
Livestrong, like all non-profits, has a .org website name, but it brought in $2.5 million by selling its name to a for-profit website, livestrong.com, that markets health and fitness products.
"It's unheard of -- I've never heard of anything like that," Mark Zimbelman, an accounting professor at Brigham Young University who's looked into Livestrong's practices," told CBS's Attkisson. "It's just questionable, to sell their name, and allow somebody to take a website and make money doing other things."
Although Livestrong reports that 82 cents of every dollar raised goes to programs, services and grants, that number actually includes staff salaries, lobbying costs and legal fees.
Livestrong's 2011 IRS tax return show it put $13 million in the bank in a rainy-day fund that year, bringing its cash on hand up to a staggering $103 million, an unusually large amount, charity experts told CBS. That same year, the charity gave $5.2 million -- 11 cents on the dollar -- for grants to help cancer survivors.
Livestrong has already announced it has no plan to refund money to dissatisfied former donors, but that's not stopping Birdsong and other plaintiffs from going ahead with their legal actions.
"I gave you this money, and worked for you and asked other people for this money in good faith," he told CBS. "We were suckers. We got taken. That's the way sometimes we feel about it."