(CNN MONEY) Michael Phelps may be untouchable in the water, but even he can't out-swim the Tax Man.

America's Olympic medalists must pay state and federal taxes on the prize money they get for winning. The U.S. Olympic Committee awards $25,000 for gold medals, $15,000 for silver and $10,000 for bronze.

Gold and silver medals are made mostly of silver, while bronze medals are composed of mostly copper. Rio's medals are among the largest and heaviest ever and contain about 500 grams of either silver or copper.

The value of a gold medal is about $564; silver is worth about $305. Bronze is worth a negligible amount so it's not taxed.

Taxes are yet another burden for Olympians -- the majority of whom are already struggling to get by.

The U.S. is one of the only countries that doesn't provide government funding to its Olympians.

A handful of lucky athletes land lucrative endorsement deals. But most of them rely on small stipends from the USOC, support from local businesses or supplemental income from a day job.

Olympic medal winners may catch a break though.

Proposed federal legislation would make "the value of any medal or prize money" awarded during the Olympics or Paralympics exempt from income taxes.

The bill was passed by the Senate last month and is being considered by the House. It would apply to earnings from January 1, 2016 to January 1, 2021.

California is reviewing a similar proposal.

Dr. Steven Gill, a tax professor at San Diego State University, isn't convinced an exception for Olympians and Paralympians would change anything.

For one thing, the USOC might be tempted to reduce Olympians' prize money, Gill said.

He added that even tax free, American athletes get a fraction of the financial support that athletes in other countries get.

"When I think about why these prizes exist, it's to compete with state-supported athletes from other countries," he said. "Cutting taxes isn't going to fix the fact that these athletes don't get paid enough -- it's a short-term fix."

Gill also noted that other individuals who win prestigious awards are taxed on their winnings. Such is the case with Nobel prize winners although they receive more prize money -- around $1 million.