If there is one thing we can learn from the headlines of the past week or so – Market Plunge Baffles Wall Street, Size of Spill in Gulf of Mexico is Larger Than Thought, ‘Amateurish’ Bomb Defused in Times Square—it is that we cannot reliably forecast the future, that the world is bound to surprise us, frequently in unpleasant ways, and that, as the poet Robert Burns wrote, the best laid schemes of mice and men oft go awry.

Shit, as they say, happens.

And yet we keep on devising those well-laid schemes, don’t we? We extrapolate the future based on the past. We imagine that we can make useful economic forecasts (because now we have more data than we did before). We imagine that regulation will protect use from the meltdowns of markets (as well as off-shore oil drilling platforms and nuclear power plants). We imagine that the Department of Energy can lead us to a clean energy future, or that scientists can make geo-engineering safe. We imagine that we understand things better than we do. And we forget the words of that other poet, John Lennon, who wrote that “life is what happens to you while you’re busy making other plans.”

So the timing is excellent for this week’s updated version of The Black Swan: The Impact of the Highly Improbable by Nassim Nicholas Taleb, which includes a new essay called “On Robustness and Fragility.”  I’ve haven’t read the essay yet, but Taleb discussed the book and much more with my friend Russ Roberts, the Hayekian economics professor, at EconTalk. Their 67-minute conversation is never dull.

For those of you who aren’t familiar with The Black Swan, the title refers to the fact that the phrase “black swan” was used for hundreds of years in Europe to describe the impossible or near-impossible–until, that is, a Dutch explorer discovered black swans in western Australia in 1697, at which point the term morphed into the idea that the impossible sometimes happens. In Taleb’s account black swan events are rare, extreme, impossible to foresee and often world-changing. “History does not crawl, it jumps,” he writes. Brilliant and egotistical,  Taleb worried in the book, which was published in 2007, that

Financial Institutions have been merging into a smaller number of very large banks. Almost all banks are interrelated. So the financial ecology is swelling into gigantic, incestuous, bureaucratic banks – when one fails, they all fall.  The increased concentration among banks seems to have the effect of making financial crises less likely, but when they happen they are more global in scale and hit us very hard. We have moved from a diversified ecology of small banks, with varied lending policies, to a more homogeneous framework of firms that all resemble one another. True, we now have fewer failures, but when they occur ….I shiver at the thought.

You know what happened next.

And yet, as he tells Russ, Taleb recently spoke on a panel with a top official of the International Monetary Fund, who began his remarks by offering his economic forecast for the years 2011 through 2014–infuriating Taleb, who thought

Anyone in this room who’s listening to this forecast without asking what his forecasts for 2008 and 2009 were in 2001, 2, 3, 4, 5, 6 and 7 needs to blow up again…

How can people still listen to the No. 2 man of the IMF talking about his forecasts for the future when he never got anything right in the past?

And why are we building systems based on these forecasts?

Some people say having an imperfect forecast is better than having no forecast at all. But of course that’s not so, at least not if we make big decisions based on that imperfect forecast that can impact all of our lives. The banks, remember, bet their future on complex computer models saying that housing prices could never decline by 15 to 20% in a year. Their formulas, they said, reduced the amount of value at risk. Unhappily, the fact that they were so tragically mistaken does not seem to have diminished the supply of hubris at, say, Goldman Sachs.

Now, the problem with wrong-headed forecasts is not that they are wrong-headed; that’s inevitable. The problem comes when our collective well-being depends on them. This, Taleb says in his essay, is why we need to build robust systems and not fragile ones. The global banking system was and is fragile. So was BP’s oil-rig. So, we were reminded last Friday, are stock markets when a “glitch” can cause a 1,000 point drop. So is the economy of Greece, which in an interconnected world, has global repercussions.

Taleb says:

I want to live in a society in which human error doesn’t penalize the multitudes. This is pretty much my mission. To try to figure out how to build a society in which people can make mistakes and mistakes are inconsequential.

One way to do that is to reduce debt. Personal debt, corporate debt and government debt all make borrowers more fragile. Surplus cash makes us more resilient, better able to withstand the unforeseen. (Recall the pre-recession balance sheets of GM and Ford.) Debt is driven, at least in part, by hubris. The shoppers who flock to the mall despite their negative balance sheets, GM, Chrysler, Sam Zell when he bought the Chicago Tribune, Greece or the people in charge at the U.S. Treasury grow overconfident about the future; they make their plans, borrow money and then life gets in the way and the rest of us get stuck with the bill. This can’t go on forever. As Russ put it during the podcast: “We’ll bail out Greece” — he could as easily have said Citi or AIG — “and then we’ll get bailed out by Mars or Neptune.”

Interestingly, western religions have understood hubris and debt for a long time. “All Mediterranean religions had either an explicit or implicit ban on debt,” Taleb notes. “We’ve known about how debt fragilizes systems since the Babylonians.”

So beware of economists bearing forecasts, bankers offering teaser rates and governments when they take on obligations that our children and grandchildren will have to repay. In a fragile and interconnected world, we all pay for their mistakes.

How we do we live in a world we don’t understand? That question has been on my mind lately because I’m again under the sway of Nassim Nicholas Taleb, author of The Black Swan. Taleb did a podcast recently with the free-market (or, if you prefer) the Smith-Hayekian) economist Russell Roberts, available here at EconTalk and highly recommended.

They didn’t discuss climate policy but their conversation got me wondering what Taleb would say about the threat of climate change, particularly since I’ve been thinking about how to respond to a thoughtful commentary on climate and energy by Jesse Jenkins of The Breakthrough Institute, who blogs as Watthead, Jesse and I serve on the blogger board for The Energy Collective, a website that attracts people who are interested in the nitty-gritty of energy and climate policy. It’s a good place to keep up with people like Joe Romm and Robert Stavins.

In his post, Jesse argues in his post that a cap-and-trade policy or carbon tax designed to “put a price on carbon” – that is, to raise the cost of burning fossil fuels – won’t do nearly enough by itself to reduce greenhouse gas emissions and respond meaningfully to the threat of climate change. That’s because any policy that drives prices high enough to discourage people from burning coal and oil will, by definition, be politically unpopular. Jesse writes:

The ultimate effectiveness of a strategy premised centrally on an effort to make dirty energy more expensive will always be limited by this fundamental reality of the political economy of energy — which we at the Breakthrough Institute have dubbed “Global Warming’s Gordian Knot.” If the price of carbon must rise too high to drive emissions reductions, various cost containment mechanisms or public backlash will kick in — either of which effectively abrogates the emissions cap. Yet if we constrain the price of carbon, it will have very little impact on emissions absent a steady supply of low-cost emissions reductions opportunities.

Instead of trying to make dirty energy expensive, Jesse argues, we need to make clean energy cheap. This requires what he calls “a coordinated, well-funded and effective strategy to accelerate clean energy innovation and drive major improvements in the price and performance of clean energy technologies.” Yep, that means lots of government spending, perhaps $50 billion a year.

What does this have to do with Taleb? I’m wary of trying to summarize his worldview but Taleb essentially argues that we know a lot less than we think we know. “My major hobby,” he writes on his website, “is teasing people who take themselves & the quality of their knowledge too seriously & those who don’t have the courage to sometimes say: I don’t know….”  In essence, Taleb says we are not very good at understanding the past or present – his first book was called Fooled by Randomness–and that we are downright horrible at predicting the future. (Although he sort of predicted the global financial meltdown.)

I don’t know what Taleb thinks about climate policy or, for that matter, climate science, but I suspect that he would not have much enthusiasm for a federal government effort to spend $50 billion a year to research and commercialize technology to make clean energy cheap. None of us know  how to make clean energy cheap, and the government has a pretty poor track record of picking marketplace winners.

Here are several examples. In 1980, President Carter signed legislation to establish the U.S. Synthetic Fuels Corp., to find ways to create alternatives to petroleum and promote energy independence. It flopped, of course, and one reason why is that it got caught up in pork-barrel politics. “Fuel-cell projects under the SFC, for example, were allotted to each of the 50 states, regardless of economic viability,” according to a book called “The Government Role in Civilian Technology,” by the National Academies of Science and Engineering and the Institute of Medicine.

Not long ago, Congress gave us FutureGen, the “public-private partnership to design, build, and operate the world’s first coal-fueled, near-zero emissions power plant, at an estimated net project cost of US $1.5 billion.” Well, good luck with that. An environmentalist pal of mine likes to refer to FutureGen as NeverGen.

More recently, we had the biofuels mandate in the 2007 energy bill, which was a boon to Midwest farmers and the corn ethanol industry, at least until oil prices dropped last year and big ethanol refiners went bankrupt. The politics of biofuels are incredibly complicated – the mandate was opposed by environmental groups like Friends of the Earth and by oil-state Republicans – but figuring out which biofuels make economic and environmental sense and which do not is no job for Washington.

Only markets will do that.

Now let me be clear. I am not arguing that venture capitalists or energy startups or academics or big oil companies are any smarter or more capable than U.S. Senators or DOE researchers. What I am saying is more voices (i.e., the market) are better than a few (politicians and civil servants). The way to make clean energy cheap is to create a market that promotes as much tinkering and experimentation by as many people are possible (crowdsourcing, if you like) and not by giving the government $50 billion a year to spend. Nobody knows today how to make clean energy cheap. Together we may be able to figure it out.

Best as I can tell, the best way to unleash that experimentation is with cap-and-trade or a carbon tax, by making dirty energy expensive. Ideally, by making it very expensive. This is logical and just. So long as we allow the fossil fuel industry to dump global warming pollutants into the air at no cost, that’s what the industry will do, and future generations will pay a terrible price. Better to pay more for electricity and gasoline today, right?

The question remains, how can environmentalists and their political allies persuade people to pay more for fossil fuel energy in the short run? Americans haven’t been very good lately at making sacrifices today for the sake of future generations. But with the right leadership, that can change.

One way to begin is to get our metaphors right, as Steven Chu, the new energy secretary, has argued.

Some people have said the clean energy revolution will require a national effort like the Manhattan Project or the Apollo project to send a man to the moon. Wrong—those were government-funded efforts, involving small numbers of people, aimed at a very specific goals.

Here we have a much broader goal—cheap clean energy—but no clear path to get there. Will it be wind, solar, wave or geothermal power, or clean coal, or nuclear power, or all of the above? What we need—and all credit to Chu for this metaphor—is something more like the mobilization of the U.S. economy during World War II, which involved everything from Victory Gardens (local food!) to energy conservation (“Don’t Travel—Unless Your Trip Helps Win the War”).

Put simple, the best way to untie the Breakthrough Institute’s Gordian Knot is with politics. We need to persuade people that it’s worth paying more for dirty energy today to save the planet for our kids and grandkids.

I’ve nearly finished reading The Black Swan, Nassim Nicholas Taleb’s brilliant book about how we know less than we think we know, how we draw too many inferences from our own experiences and how we grossly misunderstand risk. (Yes, it’s a timely book. Taleb is a former Wall Street trader who evidently had little impact on his reckless peers, although the causes of the current meltdown go way beyond recklessness.) In any event, the metaphor of the Black Swan—that is, the sudden, unexpected, game-changing occurrence—is affecting the way I read the paper, listen to NPR and do what I’ve done for the last couple of days—go to conferences showcasing business and political leaders and other “experts.” Today’s list includes Newt Gingrich, Sheila Bair of the FDIC, Wall Street legend Pete Peterson and Richard Ward, CEO of Lloyd’s of London.

The question that keeps running through my head is this: Do these people really know what they are talking about? I suppose they may when describing the world as it is, with facts, figures and anecdotes. But my BS detector goes on alert when they try to explain why things happened, or what is likely to happen next—that is, when they impose a narrative structure on events or extrapolate the future based on what we think we know now. This is like the turkey whose opinion of the humans who feed him every day rises steadily until just before Thanksgiving.

Taleb argues persuasively that narratives distort more than they enlighten, that the world is more random that we realize, that we should avoid making predictions and avoid those who do. He writes:

Our minds are wonderful explanation machines, capable of making sense out of almost anything, capable of mounting explanations for all manner of phenomena and generally incapable of accepting the idea of unpredictability.

One of his chapters is subtitled: Nobody knows what’s going on.

I don’t know how many business people have read Taleb—his books are best sellers, and he’s garnered enormous amounts of publicity—but I’m noticing this week that many sound like they’ve read him. Most likely, they’ve been shaken a by the financial crisis and the recession and the sense that really smart people like Ben Bernanke and Hank Paulson may not know what to do about them. This is another upside of the meltdown: It has taken some of the swagger out of corporate America. (For the first upside of the meltdown, see this post.) People are more willing to admit they don’t understand what’s going on, or know what to do about it, and that’s refreshing.

In that context, here are some things that I heard today at the FORTUNE 500 Forum and at a Newsweek event called “The Future of Energy.” (Taleb would not approve.) So please take all that follows with a shaker full of salt:

Gingrich is smart, knowledgeable, funny and opinionated, and therefore a great guy to have on a panel. I wouldn’t go so far as to say that his ideological edges have softened but he had kind words for the Obama cabinet including the new secretary of state, Hillary Clinton. “The team he just assembled on national security is as solid and capable a team as we’ve seen,” Gingrich said. “It’s pretty hard to argue that General Jones represents the pacifist wing of the Marine Corps.” And: “Any place we can find common ground, I want to help President-elect Obama be successful.” They share, among other things, the belief that as a nation we need to deal swith the threat of climate change, although Gingrich opposes both a cap-and-trade program and a carbon tax because they would raise the price of energy and bring in more revenue for the government to spend. People who want to raise the price of oil to penalize the owners of gas guzzlers a lesson fail to see, in Gingrich’s view, that “it will crush the poor, it will crush senior citizens, it will crush rural communities.” (Newsweek’s Fareed Zakaria, who interviewed Newt, neglected to ask if he would support revenue-neutral controls on greenhouse gases.) Gingrich favors incentives over controls, carrots over sticks—a major increase in spending on basic research into clean energy, tax breaks for renewables, and series of well-funded by outsourced efforts to figure out ways to burn coal cleanly. He likes the idea of energy prizes that excite entrepreneurs. “Prizes are very powerful devices,” he said, “because they enable anyone to compete.” He doesn’t, however, believe that markets alone can get us where we need to go when it comes to curbing global warming: “I am for the Republican Party that built the transcontinental railroad with big incentives and opened the west with the Homestead Act.” Gingrich, it must be said, has lost none of his swagger.

Bair of the FDIC was more reserved and Taleb-like at our FORTUNE event. A Republican and former aide to Bob Dole, she has had the political courage to stand up to Paulson and Bernanke and argue that one way out of our economic mess would be to directly aid homeowners. She favors expanding a model that seems to have worked after the FDIC took over the failed IndyMac bank and gave homeowners facing foreclosure the chance to restructure their loans, on more favorable terms. She’d like to see Paulson release about $24 billion in so-called TARP funds to apply that protocol more broadly, keep more people in their homes, reduce the number of foreclosures and get us closer to the day when housing prices stop falling. “There are a lot of unnecessary foreclosures,” she said,  Things could get worse as the economy slumps and people lose their jobs, or have their hours cut back, or see their earnings shrink because they collect fewer sales commissions. “I don’t see the light at the end of the tunnel,” she said.

Peterson, too, is a worrier and a regulator. On a panel grandly called The Future of Capitalism, Peterson argued that vast markets in credit default swaps, interest rate swaps and currency trades need more rules and transparency. “When you have the volume of the financial securities that we’re talking about, I don’t see how anyone can argue that they shouldn’t be regulated,” Peterson said. He also argued, as he often does, that Americans need to save more, that we are now scarily dependent on foreign buyers of our debt, particularly the Chinese, to finance our government. “There are huge geo-economic and political risks that emerge from a country that doesn’t save,” he said.

Finally, I sat down with Richard Ward of Lloyd’s, the giant insurer that, it turns out, learned about Black Swans the hard way. The company got whacked when it insured companies against asbestos-related claims in the 1990s, and took another big hit after insuring the directors and officers of corrupt companies like Enron and Worldcom. Since then, Lloyd’s has become more prudent and conservative in its underwriting and in its investments. So, for example, Lloyd’s is writing more short-term policies “because the world is changing so rapidly and the risks can change rapidly,” Ward said. If you doubt it, recall last week’s terrorism attacks in Mumbai. Ward doesn’t lose sleep over gyrations in the global stock markets, either: About 45% of Lloyd’s assets are invested in government bonds, and only about 5% is invested in equities, he said.

Ward and I talked about climate change—not surprisingly, global warming worries him because of the potential damage from severe weather—after which the conversation turned, inevitably, to the meltdown. “There are lots of lessons,” he said. “First and foremost, don’t overcomplicate your business. If it becomes so complex that people can’t understand it, you’ve got to ask yourself why you are doing it. Second, understand your risk and spread it. Third, insure that the remuneration policies you have in place don’t reward staff for gambling with the company’s balance sheet.”

Humility—admitting that even the so-called experts don’t know what’s lurking around the corner—helped Lloyd’s avoid the fate of AIG.