Sunlight Journalism Through Code

ProPublica is still the cool new kid in investigative journalism— less than five years in the game, and they’ve got the best toys, plenty of talent, and everyone wants to be their friend. And that doesn’t just apply to journalists. The investigative journalism outfit’s big data projects draw interest from programmers too. That’s why the Engineering and Computer Science departments at UC Berkeley asked Jeff Larson from ProPublica’s News Applications team, to talk about the new ways that coding is helping tell stories. The most popular news app on the site, says Larson, is called Dollars for Doctors. This feature shows how money from pharmaceutical companies reaches your doctor. It’s a database of about 500,000 doctors collecting over $760 million dollars. The app can divide information by state, for instance, $90 million pouring into CA practices; and reason for the fee, like “research” or “speaking engagement.” Accepting money from a drug company is not illegal, nor does it imply, but as ProPublica puts it, “it can raise ethical issues.” The data challenge for Dollars for Doctors isn’t about accessing information. The Affordable Care Act says drug companies have to disclose when they pay doctors. But Larson tells us, publicly available information doesn’t necessarily mean legible; wrappers and poorly scanned documents obscure these transactions. So News App developers write code that follows the money trail through this murkiness. Still, vetting at the end of the day still falls to a journalist’s eye. Larson says “half is scraping, half is validating.” Which pretty much addresses the room he spoke to on Wednesday at Cal: half journalists, half programmers. And might be a pretty reasonable formula for the future of journalism. After the talk, I asked Jeff Larson a few questions about the editorial process at ProPublica. Here are his answers, by email:

For the crowd-sourced projects, like the one on political messaging, how do you determine that people wanted this information? Are investigation topics crowd-sourced or do they always come from an editor? A lot of it is editorial judgement, we work like a traditional newsroom where reporters pitch stories and editors assign thing that are interesting. We don’t crowd source stories, but we do have an suggestions email inbox that is routinely checked. How did you create user-engagement around that tool? A large portion of our success with our crowd sourcing is due to our top notch social media team. Who, unlike at other organizations, are reporters themselves and are committed to building community and engagement around our projects, and also are tremendously good at building source networks for people to rely on. Generally, how do you determine what’s a high impact story to justify investing resources. I suppose the mission statement kindof answers that– but can you give a sense of the editorial process? That is a difficult question and I really haven’t figured it out. Some stories do surprisingly well and others don’t quite have the impact we would have liked. At the end of the day though if we have exposed wrongdoing or highlighted abuses of power, then I will take it.