The population of U.S. banks has dwindled rapidly over the last three years — to below 7,000 — even though the pace of failures has abated and merger activity has slowed.

Since mid-2010 the total number of banks has fallen by 11.4%, by far the largest three-year drop since the mid-to-late 1990s, according to data from the Federal Deposit Insurance Corp.'s most recent Quarterly Banking Profile.

Failures, mergers and charter consolidations have all played a role in the decline, but less so than in the prior three-year cycle. Twenty-three fewer banks failed between June 2010 and June 2013 than in the three years prior, and there were 107 fewer mergers in the most recent three-year span.

What's really driven down the numbers of late is the lack of startups. No new banks have opened for business since mid-2011 and only 23 have come online since 2008, according to the FDIC. Contrast that with the five years leading up to the financial crisis, when an average of 156 new banks opened each year — or roughly one bank for every two that failed or were merged out of existence.

Industry observers say there's little reason to expect a return to those days. Investors have little interest in starting banks at a time when loan demand is weak, competition remains fierce and compliance costs continue to climb, says Robert Kafafian, the president and chief executive at the Kafafian Group, an industry consulting firm based in Parsippany, N.J. He says that while there are plenty of ex-bankers out there with money to invest, "they are more likely at this point to try to buy an existing franchise than start one from scratch."

Regulators, too, have been wary about granting new charters and inviting even more competition. They are well aware that many of the community banks that failed during the recession were startups that had loosened underwriting standards to win business from established rivals.

Of course, the banking industry has been shrinking steadily for decades, so it's not a huge surprise that the number of banks has dipped below 7,000, to 6,940 at June 30. What's noteworthy is how much the pace has quickened. It took more than five years for the population to fall from 9,000 to 8,000 but just slightly more than three for it to go from 8,000 to 7,000.

Eugene A. Ludwig, the CEO at Promontory Financial Group, says he fears the pace of decline will accelerate if Congress and regulators do not ease the regulatory burden on smaller institutions. Nearly all the banks that are disappearing have less than $1 billion of assets and it's that group that struggles most to comply with the Dodd-Frank Act and other new rules and regulations that have been added since the financial crisis.

Community banks did score a victory earlier in the summer when federal regulators made a number of key changes to Basel III capital rules that gave significant relief to smaller institutions.

In May, the Consumer Financial Protection Bureau also amended mortgage rules to allow small banks to continue making balloon mortgages for another two years. As initially proposed, the rules would have essentially banned such loans except in rural or underserved markets.

Ludwig, a former comptroller of the currency under President Clinton, says Congress and regulators "have tried to do the right thing and lighten the burden on small institutions." But, he adds, "They've got to do a lot more of it to ease up on these community banks and give them elbow room."

Kafafian says a reluctance to adopt new technologies could also drive more banks out of business. Offering mobile banking or sophisticated cash management services can help small banks better compete against large or regional banks and few will survive unless they make the investment, he says.

That can be easier said than done, Ludwig says, at a time when compliance costs are rising and low interest rates continue to squeeze net interest margins.

"If you're being hammered on your income because of the yield curve and you're being hammered by the cost of regulation, it's harder for you to invest in new technologies," he says.

All of these factors — tight margins, a weak economy, higher capital requirements, the cost of technology — explain why bank investors are reluctant to start new banks, says Paul Merski, the chief economist at the Independent Community Bankers of America. "There's a much higher hurdle rate today," Merski says. "It used to take several years for a [startup] to become profitable, and now it's likely to take even longer than that."

Still, Merski says that if the economy improves and policymakers continue to address small banks' concerns about regulation, new banks could start opening again and help stem the overall decline in the bank population.

Ludwig agrees.

"Will there be 6,000 [banks]? 4,000? Who can say? But without change, it will be a smaller number and, with sensible changes, it'll be a larger number," he says.