A federal judge surprised financial services industry and retailer advocates alike this week when he suggested that banks might be forced to rebate potentially billions of dollars to merchants in "overcharges" related to debit interchange fees.

But there appears to be little chance that institutions will ever have to cough up a portion of the swipe fees they've collected in the past two years, with legal experts saying the court lacks the authority to enact such an order. Instead, observers suggested the judge's comments were aimed at increasing pressure on the Federal Reserve Board to change its interchange rule.

"Assuming that the Durbin Amendment does not expressly confer authority to impose retroactive liability, the Fed cannot now adopt a rule that makes banks return what they charged for past transactions," said Ronald Levin, a law professor at Washington University.

The so-called overcharges by banks came up during a hearing Wednesday on U.S. District Judge Richard Leon's July ruling that the Fed set a cap on swipe fees too high under the Dodd-Frank Act's Durbin Amendment. Under a 2011 rule, payment networks can set fees at roughly 24 cents per transaction, but the judge said they should be 12 cents or less.

The Fed, which is facing pressure from banks and credit unions to appeal the ruling, has until Sept. 30 to decide whether it will do so. During the hearing, the judge also asked for briefs by Sept. 16 on whether and how banks should have to repay any "overcharges" collected in the two years the rule has been in effect.

"We're going to have to have a briefing schedule with regard to the proposed remedy as to those funds that have been collected that shouldn't have been collected," the judge said.

Even the suggestion that banks can be forced to repay merchants appears to be an overreach, observers said. The retailers, who first sued the Fed over the interchange rule two years ago, never even contemplated the idea in their original lawsuit. Instead, they sought that the rule be rewritten. It's not clear now that they plan to ask for repayment, which would almost certainly open up a new area of litigation.

"It's not even been 24 hours," said Mallory Duncan, general counsel at the National Retail Federation, one of the plaintiffs in the suit. "We're going to research it."

Financial services industry representatives, meanwhile, said the court couldn't punish banks because the suit was directed against the Fed, not financial institutions.

"This is a group of merchants and trade associations who were suing the Federal Reserve," said Carrie Hunt, general counsel of the National Association of Federal Credit Unions. "And they didn't directly sue banks or credit unions for anything. And so from our perspective, this is outside of the scope of the litigation."

Levin, who is an outside expert and not associated with either side of the litigation, pointed to a 1980s Supreme Court decision, Bowen v. Georgetown University Hospital, that established that agencies cannot impose retroactive liability unless they have specific authority from Congress to do so. The 2010 financial reform law, which included the Durbin Amendment, did not include any such backward-looking language.

"The Georgetown principle is likely to be a major hurdle for the retailers to surmount," Levin said.

Instead, most observers attributed the judge's comments on Wednesday to reflect his frustration with what he perceives to be the Fed's foot-dragging in implementing his decision to overturn the interchange rule. Leon spent most of the hearing lambasting the Fed for not knowing whether it plans to make changes to its rule.

"This has gone on long enough. And the banks are obviously pushing the Fed to appeal," said the National Retail Federation's Duncan. "But it's time for the governors to decide whether they should be adhering to public law, or bowing to the will of the banks."

A banking industry source said that Judge Leon "is just ticked. … He is just really frustrated."

This source speculates that the judge's hardline approach could lead the Fed to conclude that it should appeal his decision to another court.

"Is this the right forum to have these very complicated issues resolved?" the source asks. "Regulators have to think about that."

A Fed spokeswoman declined to comment.

During the hearing, a Fed attorney and a lawyer representing the banking industry also questioned the legality of requiring the reimbursement of swipe fees that, according to the judge, should not have been charged.

"I don't know where in the statute there would be an authorization for that," Katherine Wheatley, associate general counsel for the Fed, told the judge.

Seth Waxman, a lawyer with Wilmer Cutler Pickering Hale and Door who is representing financial industry trade groups, noted that merchants never asked to be reimbursed money they paid between 2011 and 2013.

"And there's no basis in the law for ordering us to pay money when we were complying with a rule," he added.