The nearly $4.5 billion deal is another case of consolidation in the industry, as firms grapple with falling fees and the rising challenge from managers of index-tracking funds.