WASHINGTON — Despite tough talk from regulators that they are cracking down on mortgage servicers that do not follow modification guidelines, the agencies have so far done little to nothing to punish firms.

The Treasury Department and the Federal Housing Administration have said servicers have failed to comply with conditions of some government programs, but have mostly just ordered them to do better next time.

While regulators argue that they are taking necessary steps to ensure compliance, observers said the agencies are unable and unwilling to act in most cases.

"I don't think the tools have teeth," said Tim Rood, managing director at Collingwood Group LLC. "That's the problem. … It's a lot of frowning and finger-pointing but not a lot of culpability."

In an interview Friday, FHA Commissioner David Stevens defended his agency's record, and said he will punish egregious offenders.

"We're in the process now of communicating with servicers who have violations and if it ultimately should come to pass that they cannot resolve them, there will be claims that can be assessed," he said. "There's absolute confidence from my perspective if we find these servicers have not followed compliance we will assess penalties and they will be sizable."

But lawmakers are already crying for blood.

At a House Financial Services housing subcommittee hearing on Nov. 18, Rep. Maxine Waters, D-Calif., blasted the Treasury for failing to punish servicer noncompliance with its Home Affordable Modification Program. "There have been no monetary penalties, from what I'm hearing from you, and no sanctions," she said. The only thing the Treasury has done is "some work in instructing them that they have to change their practices and procedures."

Phyllis Caldwell, the Treasury's chief homeownership preservation officer, said the agency has been auditing servicers and ordering them to improve.

"As we go in and review the compliance, what we have found, that in less than 5% of the time servicers have not met those requirements," Caldwell said. "When they do, we have instructed them that they may not decline a homeowner from Hamp and that they must go back and fix the process."

But that response didn't satisfy Rep. Stephen Lynch, D-Mass. He cited a Nov. 15 Government Accountability Office report that found that the "Treasury has yet to fine any servicer for noncompliance or even establish any specific penalties or consequences for noncompliance."

"Ms. Caldwell … GAO says you're not penalizing. You're not."

The Treasury and the FHA have different policies in place for their modification programs.

Under Hamp, the Treasury has the power to withhold compensation for permanent modifications if it discovers errors, and even claw back incentives paid. It additionally has the power to order monetary penalties on wayward firms.

But despite acknowledging in September that the three largest banks — Bank of America Corp., JPMorgan Chase & Co. and Wells Fargo & Co. — have failed to properly solicit and consider homeowners for Hamp, the Treasury has only directed the companies to change their processes. It has yet to issue any fines or attempt to withhold compensation.

Consumer groups said it's a crucial reason why Hamp is not a success. "That puts the servicers in a position of no harm, no foul," said Julia Gordon, a senior policy counsel at the Center for Responsible Lending. "That's not likely to change behavior in a significant way. … Particularly in voluntary situations a program that doesn't have serious penalties never is going to have serious compliance."

Howard Glaser, a former Department of Housing and Urban Development official who now runs his own consulting firm, said regulators just don't have the will to hold servicers accountable.

"It's about the will of the regulator to bring a muscular approach to the banks and that will has not been there," he said. "The administration has treated the banks as business partners instead of entities to be regulated."

Some industry representatives said that's exactly the point. The Hamp program is dependent on servicers voluntarily participating, as opposed to the FHA's modification program, which is mandatory for all of the agency's lenders.

As a result, the Treasury is reluctant to crack down on firms.

"If they came out and slapped somebody with a huge penalty, would everyone notice? Yes, they would, but remember these are voluntary programs," said Keith Gumbinger, vice president at the mortgage research firm HSH Associates in Pompton Plains, N.J.

For the FHA, the situation is somewhat different. The agency began a review of servicer practices in May and aims to complete the review by next month and, unlike the Treasury, has taken actions against some servicers.

Under its program, it can ban servicers from being FHA lenders and fine them up to three times the total claim amount if companies do not properly follow their loan modification policy. So far, the FHA has withdrawn approval for more than 1,500 institutions and imposed $4.25 million in civil money penalties.

While critics claim it isn't enough, Stevens said the agency has to follow due process.

"I do resist a little bit the cries for guilt before proving the case and I think we have to be prudent in the process but that being said, that shouldn't be a sign of weakness," Stevens said. "We need to make sure we don't act recklessly but we do need to make sure the industry knows we will react to violations of policy as they occur."

But the FHA rarely suspends major servicers. Most often it has acted against small companies, with the notable exception of Taylor Bean & Whitaker Mortgage Corp. in August.

Some observers said that is because lenders will work to make the FHA happy, fearing the consequences if they do not. "Companies want to work with FHA," Gordon said. "It's not like Hamp where the only reason to sign up is for reputational reasons."

Michael Youngblood, principal at Five Bridges Advisors LLC, said the odds of the FHA suspending a large servicer are close to zero.

"It's unlikely to happen," he said. "If you have yanked their ability to originate and service FHA or Fannie and Freddie loans, you have to transfer the servicing to someone else. No one has the capacity to absorb Wells Fargo, B of A or JPMorgan's loans."

Other observers said the FHA and the Treasury are the wrong places to put the blame. They point the finger at the bank regulators for not taking action against servicers.

"This isn't something anyone would say is [the Treasury's] main job," said Douglas Landy, a partner at Allen & Overy LLP. "I guess they are doing it but isn't that something you would think the OCC would be doing. The bank regulators aren't. They vacated the territory on this."