Do you care who in the government knew about the AIG bonuses, and when they knew it? Do you think the AIG executives who collected bonuses deserve opprobrium? I don’t. Although the $165 (or $218) million in bonuses have become a symbol of executive compensation gone wild, pointing fingers at Tim Geithner or Chris Dodd or some AIG executive who may or may not have done a good job misses the point. It’s a distraction from a more fundamental question, namely: What have the directors of companies like AIG, Citigroup and Merrill Lynch been doing and how can they be held accountable?

Can you name a director of AIG? I didn’t think so. (Hint: one former director has a prominent role in the Obama administration.) The Citi board has gotten a more attention, and thankfully several board members will be replaced. But, as shareholder advocate Nell Minow points out in this excellent column for CNN’s website, the directors of these companies have gotten off easy. They did a dismal job of managing risk and overseeing executive compensation.

Again, though, let’s be clear on the problem. Although I happen to think that most FORTUNE 500 CEOs are overpaid, the real issue is not the absolute size of their paychecks. It’s the way in which compensation has become disconnected from performance, the way in which pay practices create perverse, short-term incentives and, most of all, the fact that board of directors cannot be reigned in by share owners in a meaningful way.

To be clear: Rich people aren’t the problem. Capitalism can’t thrive without them. We don’t and shouldn’t begrudge people who have created great companies—the Bill Gates and Michael Dells of the world—their wealth because they have made a whole lot of money for other people, including their shareholders. They created jobs and products people want. (Why people would want Dell computers or Windows is a different question…) Nor, for the most part, should we be troubled by the vast sums of money paid to movie stars or athletes or musicians who put fannies in the seats or on front of TVs. The problem arises when CEOs get paid well for failure. Here the names of Carly Fiorina (Hewlett-Packard), Chuck Prince (Citi) and Dick Parsons (former CEO of Time Warner and lead director of Citi) spring to mind.

Here’s what’s worse: On Wall Street, compensation practices promoted dangerous risk taking. As best as I can tell, the people who packaged CDOs and MBSs got bonuses each year that were tied to the number of deals that they made and the amounts of dollars involved. (Just as mortgage brokers got paid when they wrote mortgages.) It didn’t matter whether those deals made money in the long run, or whether the mortgage loans would be paid. If you pay people to make deals, they will make deals, for better or worse. As Robert Weissman of CorpWatch wrote last fall:

Wall Street players knew they were speculating in a bubble economy. But the riches to be made while the bubble was growing were extraordinary. No one could know for sure when the bubble would pop. And Wall Street bonuses are paid on a yearly basis. If your firm does well, and you did well for the firm, you get an extravagant bonus. This is not an extra few thousand dollars to buy fancy Christmas gifts. Wall Street bonuses can be 10 or 20 times base salary, and commonly represent as much as four fifths of employees’ pay. In this context, it makes sense to take huge risks. The payoffs from benefiting from a bubble are dramatic, and there’s no reward for staying out.

So what is to be done? It’s unlikely that the Obama administration will be find a way to regulate executive compensation that doesn’t lead to unanticipated and unwelcome consequences. I favor a simple solution—giving shareholders access to proxy statements, so they can nominate candidates for the board and thereby seek to replace directors who have failed on the job. By giving shareholders proxy access, the SEC can end the self-perpetuating system that permits CEOs and incumbent boards to hand-pick director candidates. Here’s an AFL-CIO website devoted to proxy access.

As for the AIG board, the director alluded to above was Richard Holbrooke, who is the state department’s special envoy to Pakistan and Afghanistan and Friend of Hillary and Bill. Holbrooke served on the AIG board from February 2001 until last July and, according to Huffington Post, he “may have earned as much as $800,000 in cash and company stock.” (The stock’s not worth much now.) From 2001 to 2005, Holbrooke was on the board’s compensation committee.

It gets worse. Holbrooke was not only on the AIG board, he was one of the “friends of Angelo” who got special treatment from Countrywide Financial and its CEO, Angelo Mozillo. (Chris Dodd was another.) Holbrooke, his wife Kati Marton and his son David  got sweetheart deals from Countrywide, which is now out of business, as Portfolio reported last year:

Holbrooke’s wife, author Kati Marton, received loans totalling $1.4 million to refinance two properties in 2002. “Look for these,” one Countrywide manager wrote in a September 27, 2002, email, alluding to Marton’s loan applications. “These loans are incredibly important to Angelo and as such they are incredibly important to us.”

The next year, Holbrooke borrowed $1.2 million to refinance a vacation home in Telluride, Colorado. Countrywide waived at least 1.25 points, or $15,000. “Per Angelo, this loan is to be at zero points,” a Countrywide manager wrote in a February 20, 2003, email. Also in 2003, Holbrooke’s son, David, and daughter-in-law Sarah received a half-point discount on a $559,500 loan, or about $2,800, when they refinanced their Brooklyn high-rise co-op, and five-eighths of a point discount on a $428,000 loan, or about $2,600, when they bought the floor above it. Neither Holbrooke nor his wife and son returned messages.

Director of AIG? Borrower from Countrywide? If we have to point fingers, let’s point them at people like Mr. Holbrooke.