Signal vs. Slop
2026-07-20 · 5 min read · Janaina Maia
This week, a research paper gave a name to something open-source maintainers have been feeling for months: AI-DDoS. The paper, titled "AI Slop is DDoSing Open Source," analysed 294 repositories with over 2 million pull requests and found that first-time contributors' merge rates dropped 18.18% after AI-generated contributions flooded in. The volume went up. The quality went down. And the humans who review those contributions — unpaid, overworked, doing it for love — are drowning.
That same week, an essay called "AI Mania Is Eviscerating Global Decision-Making" went viral on Hacker News. Its argument: organisations have been captured by people in the throes of AI excitement, and it has become dangerous to even raise the possibility that AI might not be the solution. Executives mandate AI adoption without using AI tools themselves. Employees fake AI usage to keep their jobs. Honest assessment has become a career risk.
These are the same story. One is about open-source maintainers drowning in AI-generated pull requests. The other is about organisations drowning in AI-generated decisions. In both cases, the problem is not that AI is bad. It is that AI makes volume cheap while judgment stays expensive, and the people who provide the judgment are being overwhelmed.
What the research actually says
The paper from University of California and University of Zurich researchers is rigorous. They used Bayesian Structural Time Series analysis — a statistical method that models what would have happened without the intervention and compares it to what actually happened — across 294 repositories. This is not a survey. It is a causal analysis.
The findings:
- Pull request volume increased in 2025 as AI coding tools made it trivial to generate contributions.
- Merge rates declined, especially for first-time contributors, whose merge rates dropped 18.18% relative to the counterfactual.
- The quality gap widened — not because AI-generated code is always bad, but because the flood of plausible-looking but shallow contributions makes it harder for reviewers to find the genuinely good ones.
- Maintainers are burning out — they are defaulting to "low-effort defensive strategies" that protect short-term review capacity but make the project harder to sustain long-term.
The paper calls this "AI-DDoS" — a denial-of-service attack where the flood of requests is not malicious, but the effect is the same: the system that processes those requests — the human reviewer — cannot keep up. The maintainers are the infrastructure, and the infrastructure is failing.
Why this is a product design problem
If you work in product, you might think this is an open-source problem. It is not. It is a preview of what happens when you make generation cheap and evaluation expensive.
Consider what is happening inside organisations right now. The "AI Mania" essay documents it clearly: executives who have never used ChatGPT are writing AI strategies. Employees are running AI tools in the background to hit usage quotas. Managers are mandating AI adoption without defining what success looks like. The volume of AI-generated output — code, documents, analysis, decisions — is going up. The capacity to evaluate that output is staying flat or declining.
This is the same dynamic as the open-source slop problem, but inside companies. The reviewers are managers, directors, VPs who now have to evaluate AI-generated strategy documents, AI-generated code reviews, AI-generated product specs, and AI-generated performance scores. The volume has increased. The judgment has not.
I wrote recently about what happens when AI decides who gets fired — Meta's internal scoring system that penalised workers on disability leave. That story is also about this pattern: AI-generated volume (activity metrics, keystroke counts, AI adoption scores) overwhelming human judgment (the actual quality of someone's work). The scoring system produced a flood of data. Nobody checked whether the data measured what mattered.
The signal problem
The fundamental issue is not that AI produces bad output. Sometimes it produces excellent output. The issue is that AI makes it essentially free to produce output that looks good, which means the cost of evaluating output has become the bottleneck.
Before AI, the cost of generation was a natural filter. If you wanted to submit a pull request, you had to understand the codebase, write working code, and test it. That investment weeded out most low-quality contributions before they reached a reviewer. The reviewer's job was to catch the remaining edge cases.
Now, the generation filter is gone. Anyone can generate a plausible-looking pull request in seconds. The entire quality burden has shifted to the reviewer. And reviewers — whether they are open-source maintainers, engineering managers, or product directors — are not ten times faster than they were before. They might be slightly faster with AI assistance, but the volume increase has vastly outpaced the evaluation increase.
This is not a temporary imbalance. It is a structural shift. The cost of generation has dropped by orders of magnitude. The cost of evaluation has barely moved. The gap will widen as generation gets cheaper and cheaper.
What product teams should do
- Design for evaluation cost, not just generation cost. Every AI feature that generates content, suggestions, or decisions should include a mechanism that makes that output easy to evaluate. If your AI produces 10 options, the interface should help the human compare them quickly — not dump them into a list and hope for the best. The cost of evaluating AI output is your user's most scarce resource. Design for it.
- Build curation into the system, not on top of it. The open-source research found that maintainers default to defensive strategies — closing PRs faster, lowering their standards for what they review, or simply giving up. These are coping mechanisms, not solutions. If your product generates volume, build the curation into the product. Rank, filter, cluster, and prioritise before the human sees the flood.
- Measure the signal-to-noise ratio of your AI outputs. If you are producing AI-generated content, code, or decisions, track what percentage of your output is actually used, accepted, or acted on. If the acceptance rate is dropping, you have a slop problem even if your users are not complaining yet. The complaints come later, after trust has eroded.
- Make "I don't know" a valid output. One of the more alarming findings from the research this week was that AI advice reduced "I don't know" answers from 44% to 3%. That sounds like improvement until you ask: were the people who said "I don't know" wrong to say it? Confidence is not the same as accuracy. If your AI makes people more confident without making them more correct, you have not reduced uncertainty. You have hidden it.
- Resist the mandate-then-measure trap. The "AI Mania" essay documents organisations that mandate AI usage, then measure adoption, then declare victory based on adoption numbers. This is the same feedback loop that penalised Meta employees on parental leave: measuring usage instead of outcome, counting activity instead of impact. If your AI strategy starts with "everyone must use AI" and ends with "look how many people are using AI," you have a measurement problem, not an AI strategy.
The broader context
This week also brought the Kimi K3 moment — a Chinese AI model that leapfrogged Western competitors on multiple benchmarks at a fraction of the cost. It brought LM Studio Bionic, a local AI agent that runs open models on your machine. And Mozilla published a sharp piece arguing that open models are ready for agents, but their APIs are not — the intelligence is there, but the platform layer (tool calls, file handling, streaming, caching) breaks the moment you swap a frontier model for an open one.
The connecting thread is the same: the AI world is producing more output, more models, more agents, more capability than ever before. But the interfaces that evaluate, curate, and channel that capability — the APIs, the review processes, the governance, the human judgment — are not keeping up. Kimi K3 proves you can build a competitive model. Mozilla proves that building the model is not enough. The open-source slop paper proves that volume without curation is a denial-of-service attack.
My take
The AI industry has spent the last three years obsessing over generation. More parameters. More benchmarks. More tokens per second. More capabilities. More, more, more.
The next three years will be about evaluation. About curation. About building interfaces and institutions that can handle volume without drowning in it. The product teams that win will not be the ones that generate the most output. They will be the ones that make it easiest for humans to find the signal in the noise.
Open-source maintainers are the canary. They are experiencing right now what every reviewer, every manager, every director, every user will experience as AI-generated volume increases: more to evaluate, less time to evaluate it, and a growing share of the flood that looks plausible but is not worth the attention it demands.
The answer is not to slow down AI generation. It is to speed up human evaluation — through better interfaces, better curation, better filtering, and better design that recognises the cost of judgment. The organisations that figure this out will have a real competitive advantage: not more output, but more signal.
The rest will just have more slop.