ICML 2025 Paper Advises Against Same-Family Model Code Review

On November 5, 2026, a DEV Community piece argued that AI code review should not be outsourced to the same model family that produced the code being reviewed. Citing an ICML 2025 paper by Goel and colleagues, the author pointed to a bias that shows up when models grade one another: models from the same family tend to favour answer styles that match their own, including the specific ways they get answers wrong.

The evidence cited comes from a grading experiment described in that paper. When Qwen2.5-72B scored free-form answers from two smaller models, it rated the 7B model from its own family as 71% correct, while Llama-3.1-70B received 67%. On a multiple-choice version of the same questions, the ranking flipped: the Qwen 7B answered 41% correctly, versus 51% for Llama. The author attributes these figures to Goel et al.'s study, which is referenced in the article.

Two practical recommendations follow from this. First, avoid letting a model review code that was written by models from its own maker, and instead deploy a review agent from a different vendor. Second, keep human review of the diff, since automated review alone is not considered sufficient.

As of the time covered by the available reporting, the proposal remains a single author's recommendation. No vendor or research group has publicly responded to it, and no follow-up experiments testing it in a real code review workflow have been reported. It is also worth noting that while Claude appears in the title as the cautionary example, no grading or code review data involving Claude is presented; the only models with verifiable figures in the material are Qwen2.5-72B, Qwen 7B, and Llama-3.1-70B.