Clay Shirky is giving a lunchtime talk at the Shorenstein Center, which may be a joint event with the Berkman Center.

NOTE: Live-blogging. Getting things wrong. Missing points. Omitting key information. Introducing artificial choppiness. Over-emphasizing small matters. Paraphrasing badly. Not running a spellpchecker. Mangling other people’s ideas and words. Posted without re-reading. You are warned, people.

The commercial structure of the newspaper industry means that it’s not enough for them to run at a profit. Advertisers had been forced to overpay because there weren’t other waysto reach people for display ads or coupons. This gave the newspaper enough capital to do long-term investigation; mere profitability wouldn’t have allowed this.

The advertisers were overcharged and under-served. That is, they couldn’t influence coverage.

Neither the overpaying and the underserving is true of the current market. The new market is efficient and so the price of advertising plummets. “We may be seeing advertising priced at its real value for the first time in history.” And if you want to sell a bike, you don’t go to the people who print news and crosswords; you go to Craigslist. As Bob Garfield says (says Clay), it turns out that people will go to sites that do nothing but post ads.

The news is now disaggregated and is re-forming itself around actual user desires. The aggregation is going from server-side to client-side. “The decision about what to bring together in a bundle” is made by the consumer, not the producer.

We should worry about echo chambers, although it turns out that people are interested in serendipity. But people are not interested in the omnibus approach. The number of people going to the NYTimes home page is going down because people go straight to the article. The bundle is put together more by other readers.

None of this will be reversed by increasing the commercial viability of printed newspapers.

“This doesn’t mean all newspapers go away. It does mean a lot of them go away.” “Newspapers will play a less significant role in accountability than they have in the past, which leaves us with a giant hole.” A big problem: Every town of 500,000 or less “sinks into endemic civic corruption” because no one is watching.

He refers to how a story in the Boston Globe went worldwide not because of the Globe but because its audience passed it around. “The public created itself.” “The penumbra of reuse around the article created an enormous amount of its value.” An article on a similar topic (priest pedophiles) in the early 1990s didn’t spread the same way, because the forward-and-recommend infrastructure didn’t exist.

If there were a pay wall around the later article, it would have forestall its effect and value. First, “We need the public good of accountability journalism.” Some newspapers are trying to get an anti-trust exemption to establish a pay wall for the sake of the public good. But that will destroy the village to save it. We should be looking at ways of balancing the cost of producing good journalism and the public good that comes from reuse.

There are three ways to create things accessible to the public. Private companies. NGOs. Social/peer production where people get together and do it. #3 had been confined to picnics, etc. Now it’s becoming a big part of the ecosystem. E.g., Pro Publica. Wikileaks. Open source. “The Internet makes all commercial models of journalism harder to sustain…and social models much, much easier to sustain.” “We’re seeing a re-balancing of the landscape” where all three of these modes of production will be operating. We want experiments across all three of these.

Also, we don’t want to replace newspapers. Newspapers have a single point of failure problem because they do 85% of accountability journalism. We don’t need a single point. We need someone who does 5% and then repeat that 15 times. “It’s a shift from one class of institutions to an ecosystem as a whole.”

Clay says he wants to distance himself from the utopians and optimists. “I think a bad thing is going to happen.” People don’t take seriously that things may get a lot worse for a while. He doesn’t think there’s any way to get out of the coming of public corruption. Between the printing press and the Treaty of Westphalia there were a long 100 years when people didn’t know what to think. “Our goal should be to minimize the depth of that trough … and hasten its end.” But there’s not simple and rapid alternative to 20th century newspapers, in part because what held papers together was “so crazily contingent.”

“I believe that newspapers are irreplaceable in their production of accountability journalism.” Some think we should therefore spend whatever we have to in order to replace them. Others say we should be “transferring our concern to the production of lots and lots of overlapping models of accountability journalism.” “The next step needs to be vast and varied experimentation.”

Q: Alex Jones: I don’t agree that newspapers are ready to be abandoned. In the priest pedophile story you cite, the Catholic Church was brought to heel by the viral information but also by the institutional power of the Globe. As you imagine this future, do you see in this array of smaller entity an institutional power that can bring institutions of power to heel?
A: Not in any simple way. That is the great weakness of the experimental trough: No one news org has that sort of power. Hard news is cross-subsidized by people who buy the paper for the coupons. But the front page has institutional power. The media has lost its force in almost all cases. The question is: Can news gather a public the way newspapers have done? The optimistic face is: We don’t know yet, but it’s there. The pessimistic: The ability of media to bring institutions is fading with the mass audience. I don’t know enough about the economics of converting newspapers to nonprofits.

Q: How about magazines?
A: They’re essentially non-profits. The New Yorker has operated at a lost throughout most of its history. The amount of journalism done by non-NPR radio is very small. Magazines are subsidized by billionaires. “The way to get around the problem with the media model is to have lots of models.”

Q: The revenue base is shrinking but it’s also much easier to acquire information.
A: That’s why we need lots of overlapping 5% reporting. The last time we had a big push for transparency — Watergate — it created K Street: You now knew how people voted, so lobbyists could get paid for effects. “It’s not enough to make the data available. We also have to make the public able to assemble and act on the data.”

Q: What’s the model for something like Pro Publica, which is not reaching the ordinary joe?
A: In the past, city hall news generally wasn’t front page. We think readers of newspapers read the whole thing. But it was cross-subsidized. It’s never been that all citizens care about all news. Pro Publica is reaching elites, and the question is whether it’s giving them what they need. “The real danger is that our political life is organized around geography, but the Web not so much.” The midpoint between nation and neighborhood is hard to do on the Web. Web stories are either hyperlocal or spill across all borders. Pro Publica isn’t well suited to regional reporting. The media markets and the political markets overlapped, but not any more. The trough will be worst at state and county levels.

Q: How does The Economist fit? They’re growing.
A: The one big exception is to the sharing model is financial news. A pay wall damages general news and benefits financial news, because people want to act on that news before they share. The Financial Times’ online audience is 1% of the Times. I don’t believe the Economist, FT and WSJ model is applicable to the general news.

Q [bill mitchell]: As you describe your three models — commercial, public, social — what in each of them really holds value for the public at large. What might they pay for, whether in donations, contributing their own journalism, etc.?
A: The core of the value is the set of the values accuracy and timeliness, but also shareability. General news has more value the more people know about it. People contribute unexpectedly. E.g., SETI. People donate not just because they wanted to help but because they got a cool screensaver. NPR tote bags say “I’m paying for your radio.” The power of that type of mockerhood is under-estimated. 6-8% of NPR listeners contribute, which might be enough to keep a newspaper alive, doing something (but not all of what it used to do).

Q: The story on Randy Cunningham required figuring out how to take the database of info and turn it into a story. Who’s going to do this?
A: Richard Hackman [sp?] says that groups are no good at writing. E.g., Wikipedia’s writing isn’t its strong point. Amanda Michel at OffTheBus found out that most people can’t be David Broder. Instead, she had hundreds of people crowd-sourcing data, and then gave it to a writer. She had a professional-amateur fusion. “No one is smart enough to get it right, which is why we need lots of experimentation.”

Q: The NYT says it has 800K readers who have been with them for 2 yrs, and they pay $700/year. Is that sustainable?
A: No. Someone suggested that newspaper rename their obit column as “Reader countdown.” Many newspapers pursuing a pay wall are only trying to stave off the Web.

Read the original blog entry...