Charlie Munger had thoughts about passive investing, and they weren't exactly comforting. At the 2022 shareholder meeting for Daily Journal Corporation (DJCO), the late Berkshire Hathaway vice chairman didn't hold back when discussing BlackRock (BLK) and the uncomfortable reality of who actually controls corporate America these days.

The Rise of the Index Fund Emperors

When asked whether index funds were affecting stock prices, Munger's response was immediate: "Oh, huge." But his concern went deeper than price distortions. The real issue was power.

Here's what happened: Index funds were supposed to democratize investing, letting regular people own a slice of the entire market without paying high fees. That part worked. But there was a side effect nobody quite anticipated. When millions of people invest through index funds, those funds vote the shares on their behalf. And when just three firms manage most of the index fund money, those three firms suddenly control a shocking amount of corporate voting power.

Munger was blunt about it. "We have a new bunch of emperors, and they're the people who vote the shares in the index funds," he said. Giants like BlackRock, Vanguard, and State Street (STT) now vote on behalf of millions of investors, giving them extraordinary influence over corporate boards and strategic decisions.

"Maybe we can make Larry Fink and the people at Vanguard Pope," Munger quipped, though the joke had an edge to it.

Then came the kicker: "I don't know what the consequences are going to be, but I predict it will not be good."

How BlackRock Got This Big

BlackRock manages over $13.46 trillion in assets, making it the world's largest asset manager by a wide margin. The firm owns significant stakes in virtually every major publicly traded company in America. That gives CEO Larry Fink a seat at every important table.

Fink hasn't been shy about using that influence either. In 2022, he sent a letter to CEOs warning that BlackRock might vote against management teams that didn't meet certain sustainability standards. That wasn't an empty threat. The previous year, BlackRock backed activist fund Engine No. 1 in a proxy fight against Exxon Mobil (XOM), successfully replacing three board members despite the activist fund owning just 0.02% of shares. BlackRock's support made the difference.

"I think the world of Larry Fink, but I'm not sure I want him to be my emperor," Munger said.

The ESG Retreat and What It Means

Since Munger's comments, BlackRock has pulled back considerably from its environmental and social governance push. The firm exited the Net Zero Asset Managers Initiative and scaled back support for climate-related shareholder proposals. That shift came after intense political pressure, particularly from Republican state officials who accused the firm of prioritizing ideology over returns.

But even with the ESG retreat, the fundamental problem Munger identified hasn't gone anywhere. BlackRock still manages $13.46 trillion. It still votes those shares. And Larry Fink still has more influence over American corporate governance than probably any single person should.

Munger believed deeply in capitalism and free markets, but he also understood that concentrating too much power in too few hands creates systemic risks. When three firms control the voting rights for a huge chunk of the American economy, you've essentially created a new power structure that sits outside the traditional checks and balances.

The investing world has moved on since 2022, but Munger's warning lingers. Passive investing isn't going anywhere—it's too cheap and too effective. But the governance question remains unresolved. Who should control corporate America: dispersed shareholders making individual decisions, or a handful of asset managers voting enormous blocks on behalf of millions of people who may not even realize it's happening?

Munger didn't have the answer, but he knew enough to be worried. And when someone that smart expresses concern, it's probably worth paying attention.