It's like a night out on the town — it's great at first, but there's always a hangover the next morning.

The mortgage-refinance boom lifted banks' bottom lines when they needed it most. But once rates started rising, the refinance industry collapsed, knocking out a huge revenue stream and forcing banks to fire thousands of workers.

Now, the same scenario is about to play out in the area of prepayment penalty fees. The drop-off is expected to be most pronounced among banks with a big exposure to multifamily loans.

The situation is especially precarious in the New York metro area, home to hundreds upon hundreds of apartment buildings. The $13.8 billion-asset Investors Bancorp (ISBC), in Short Hills, N.J., posted a 125% increase in the amount of revenue it received from prepayment penalty income in the 12-month period that ended on Sept. 30, compared with the year-earlier period.

Several other New York-area banks have enjoyed big jumps in prepayment penalty income, including Signature Bank (SBNY); Provident Financial Services (PFS) in Jersey City; and ConnectOne Bancorp (CNOB) in Englewood Cliffs, N.J.

Now, with rates expected to eventually rise more, bankers are anticipating the end of the gravy train.

The $16 billion-asset Astoria Financial (AF), in Lake Success, N.Y., has already seen a slowdown and will likely see more slowing, David Hochstim, an analyst at Buckingham Research Group, wrote in an Oct. 17 research note. Astoria's prepayment penalty income in the third quarter was $1.5 million, down from $1.9 million in the previous quarter, Hochstim wrote.

Dime Community Bancshares (DCOM), in Brooklyn, N.Y., has reported a drop-off, too. Its prepayment penalty income fell $1.2 million from the second quarter to the third quarter.

Income from prepayment penalties has a direct effect on a bank's net interest margins. The quarter-over-quarter decline narrowed $4 billion-asset Dime's margin by 13 basis points, according to Mark Fitzgibbon, an analyst at Sandler O'Neill & Partners.

The holders of multifamily loans have been refinancing at rapid rates because the underlying properties have risen in value, says William Jacobs, chief financial officer of $2.8 billion-asset Northfield Bancorp (NFBK) in Woodbridge, N.J. The borrowers take cash out of the refinancing to fund the purchase of another apartment building, or to make upgrades to existing buildings.

"People are able to do cash-out refis and they're ending up with the same monthly payment that they had," Jacobs says.

Banks assess penalty fees on borrowers when they prepay to "protect" themselves and provide the bank with compensation if the loans are prepaid, Oritani Financial (ORIT) in Washington Township, N.J., said in its third-quarter earnings release.

Some bankers say they are prepared to weather the storm.

There is a bright side to losing the prepayment penalties, argues Kevin Lynch, the chairman and chief executive of Oritani.

"I'd rather keep the loans" on the bank's book, Lynch says.

The $2.8 billion-asset Oritani recorded $1.8 million of prepayment penalty fees in the third quarter, a 29% increase from a year ago.

New York Community Bancorp (NYCB) is an outlier when it comes to prepayment penalty income, says Ted Kovaleff, an analyst at Informed Sources Service Group. The $45.8 billion-asset thrift routinely books a large revenue stream from prepayment penalty income, and can expect to do so even as rates rise, he says. That's because New York Community has such a huge share of the multifamily loan market in its footprint.

"New York Community is always getting a boost from prepayment penalty income, no matter the rate environment," Kovaleff says.

Last quarter New York Community posted a 26% year-over-year rise in prepayment penalty income, to $39.6 million. It booked a record amount of prepayment penalty fees in the second quarter, $44.4 million, which contributed a whopping 47 basis points to the thrift's profit margin.

Joseph Ficalora, president and CEO of New York Community, shares the upbeat outlook.

"The contribution of prepays has been especially important, given the steady decline in mortgage banking income over the course of this year," Ficalora said in an Oct. 23 conference call with analysts.

"We've never tried to forecast prepay, because it's a very difficult thing to do," Ficalora said. "But we do believe … that there will be a great deal of prepay activity even in the period in front of us."

Andy Peters writes about regional banks and community banks for American Banker.