Metasciencey Stuff

I tried to report scientific misconduct. How did it go? It's exactly as tragicomic as you expect. Hilariously blatant fraud, recalcitrant journal editors (though some did their job properly), and of course "A month after that, Southwest University cleared Dr. Zhang of all charges."

The clearest consequence of my actions has been that Zhang has gotten better at publishing. Every time I reported an irregularity with his data, his next article would not feature that irregularity.

Elisabeth Bik twitter thread on which journals are worst at responding to image duplication problems. "An old favorite, @Oncotarget. I flagged 161 of their papers (most reported a year ago), but only 2 retractions. They have been sending annoyed responses though, that I should look at other journals, and what my conflicts of interests are."

I think that the vast majority of researchers think that they have strong theories, and they think their theories are “true” (whatever that means). It’s tricky, though: they think their theories are not only true but commonsensical (see item 2 of Rolf Zwaan’s satirical research guidelines), but at the same time they find themselves pleasantly stunned by the specifics of what they find. Yes, they do a lot of conscious p-hacking, but this does not seem like cheating to them. Rather, these researchers have the attitude that research methods and statistics are a bunch of hoops to jump through, some paperwork along the lines of the text formatting you need for a grant submission or the forms you need to submit to the IRB. From these researchers’ standpoint, they already know the truth; they just need to do the experiment to prove it. They’re acting in good faith on their terms but not on our terms.

On the Newton Hypothesis: is science done by a small elite? Plus good HN discussion.

Groundhog is a package that fixes one of the (many) great evils of R: the lack of reproducible scripts. It does so by getting old package versions from the time the script was written.

The influence of hidden researcher decisions in applied microeconomics: investigating the consequences of "researcher degrees of freedom" by doing seven independent replications of two studies. One of them ended up all over the place, the other relatively uniform. "The standard deviation of estimates across replications was 3-4 times the typical reported standard error." Twitter thread.

Ulrich Schimmack argues that psychology has improved in the past decade: Replicability Rankings 2010-2020. "...about 20% of [120 psychology journals] have responded to the crisis of confidence by publishing studies with higher power that are more likely to replicate." Plus some back and forth on the IAT.

Reactivation-Dependent Amnesia for Contextual Fear Memories: Evidence for Publication Bias. Look at these funnel plots:

Corona

Exploring the Supply Chain of the Pfizer/BioNTech and Moderna COVID-19 vaccines. An exhaustive look into how vaccines are produced, distributed, and used. "I’ll start with the bad news: Nobody will be making an mRNA vaccine in their garage any time soon."

Lemoine contra lockdowns. Argues that estimates of the effect of lockdowns are confounded by changes in behavior due to a rise in cases, and that the small effect of lockdowns does not pass a cost-benefit analysis. The studies are indeed weak but I'm not 100% convinced.

Youyang Gu of covid19-projections.com: One Year Later. "The most frequent question I receive is how I was able to do what I did with no background in infectious disease modeling. [...] You don’t need decades of experience to be able to think critically and adapt to new information."

Forecasting & Experts

There's been a lot of interesting writing on credentials, expertise, and social epistemology lately. Here's a couple of good pieces:

Scott Alexander: WebMD, And The Tragedy Of Legible Expertise

When the Director of the CDC asserts an opinion, she has to optimize for two things - being right, and keeping power. If she doesn't optimize for the second, she gets replaced as CDC Director by someone who does. That means she's trying to solve a harder problem than Zvi is, and it makes sense that sometimes, despite having more resources than Zvi, she does worse at it.

Then Zvi's response to Scott: Why I Am Not in Charge.

The idea of Zeroing Out is that you can do better by taking a market price or model output as a baseline, then taking into account a bias or missing consideration. Thus, you can be much worse at creating the right answer from scratch, and yet do better than the consensus answer. [...] The true beauty of Zeroing Out is that I can take all the analysis of the CDC as given, find one place where they are biased or making a mistake or missing a piece of information, and then reliably do better than the CDC by otherwise taking the CDC’s answers.

Being an expert in biology is one thing. Being an expert in leadership or politics, or the politics of biologists, is another.

When the CDC issues its guidelines it seems very clear that those guidelines are designed to let those involved tell stories about being safe and following proper procedure, and they have at best vague bearing on whether things are actually safe. They’re certainly not even attempts to create a correct physical model of anything. They’re giving people a chance to avoid blame and to show adherence to power, while themselves avoiding blame and showing their adherence to power.

Vitalik on betting in the election prediction markets, great stuff as usual. "The amazing promises of what prediction markets could bring if they work mathematically optimally will, of course, continue to collide with the limits of human reality, and hopefully, over time, we will get a much clearer view of exactly where this new social technology can provide the most value."

metaforecast.org allows you to search for forecasts from Metaculus, PolyMarket, and a bunch of other sources through a single convenient interface.

What makes good forecasters good? The Bias, Information, Noise model.

Using historical data on post-war financial crises around the world, we show that crises are substantially predictable. The combination of rapid credit and asset price growth over the prior three years, whether in the nonfinancial business or the household sector, is associated with about a 40% probability of entering a financial crisis within the next three years.

AI

An excellent new science fiction short from Sam Hughes: Lena. (Inspired by Age of Em?)

OpenAI's DALL·E creates images from text.

OpenAI: Multimodal Neurons in Artificial Neural Networks. Lots of cool stuff, including the Stroop effect in ML models.

The National Security Commission on Artificial Intelligence (chaired by Eric Schmidt) has produced its "Final Report". 756 pages long, do read the executive summary and skim though. Strikes me as rather fantastically idealistic.

The Rest

A whirlwind tour of Ethereum finance. "Why should I care about this? For one thing, it's the coolest, most cypherpunk thing going." Plus Wrecking sandwich traders for fun and profit: "Being a DeFi degenerate is a lot of fun, but be careful in your trading - because this game is highly adversarial and we play for keeps."

One man's experience with early retirement. Keeping up with the Joneses is an underrated aspect of psychology and economic decision-making. Selection happens on relative fitness—so much of human behavior is downstream from that.

A chart from The Recent Large Reduction in Space Launch Cost. Note the logarithmic y axis. Also check out The New Space Race: What Took So Long, for an explainer of why this wasn't possible a few decades ago.

Tariana is a language spoken by about 100 people in the Amazon rainforest, and it has evidentiality baked into it, with different suffixes depending on whether you saw something, inferred it, or learned it from someone else. Here's a paper that goes into more detail. And it's not the only language with evidentiality features!

Joe Carlsmith on objecting to moral claims because their implications are too demanding. "In general, I think that the abstract arguments for very “demanding” forms of moral behavior are very strong (even for non-consequentialists). But I think that the fuzzier, harder-to-articulate heuristics, evidence, and forms of wisdom encoded in our hesitations about such behavior are important sources of information as well." Plus Alienation and meta-ethics (or: is it possible you should maximize helium?).

Can You Ever Be Too Smart for Your Own Good? Comparing Linear and Nonlinear Effects of Cognitive Ability on Life Outcomes. "We found no support for any downside to higher ability and no evidence for a threshold beyond which greater scores cease to be beneficial. Thus, greater cognitive ability is generally advantageous—and virtually never detrimental."

bmgarfinkel reviews Renaissance polymath Gerolamo Cardano's memoir The Book of My Life (1575). "Cardano kicks off the chapter “Things of Worth Which I Have Achieved in Various Studies” by reminding the reader that “there is practically no new idea which one may bring forward.”"

A mouse embryo has been grown in an artificial womb—humans could be next. "Hanna says to make such experiments more acceptable, human embryos could be altered to limit their potential to develop fully. One possibility would be to install genetic mutations in a calcium channel so as to prevent the heart from ever beating." (What a sentence!)

A Group of Orca Outcasts Is Now Dominating an Entire Sea. "Granny and her kin are considered part of the same species as transient killer whales, Orcinus orca. But residents and transients have lived separate lives for at least a quarter-million years. They generally do their best to avoid each other, and they don’t even speak the same language—the patterns and sounds they use to communicate are completely different. Over time, each type has established cultural traditions that are passed from generation to generation."

On the SawStop and the table saw industry. "The CPSC says table saws result in about $4B in damage annually. The market for table saws is about$200-400M. This is a product that does almost 10x in damage as the market size."

There were a few hundred pieces written about the NYT/Scott thing, most of them incredibly tiresome. @yashkaf produced the best take by just writing about the features of the mainstream narrative and how one ought to interact with it. "There are three main goals of engaging in argument: for truth, for power, for lulz."

Apparently there are bacteria that do photosynthesis from the infrared glow of hydrothermal vents deep in the ocean.

A fan-made visual effects demo for the unfilmed Shane Carruth script A Topiary.