What happened
- Six authors, among them Andrew Rabinovich and Zhao Chen, published on arXiv on 3 November 2026 PAYWORTHY, a test of whether a language model can foresee if an AI agent will complete a paid job. The code is on Upwork’s GitHub account.
- The test uses 1,494 real freelance postings, sampled from 18.8 million made between 2020 and 2026 on a platform.
- An agent attempts each job in a single run and from the posting alone. A model acts as judge in the role of whoever commissioned the work and decides whether to accept the result.
- Then 18 models forecast the probability of success before any attempt, and the test measures how well they rank accepted jobs above rejected ones.
Why it matters
- Before assigning paid work to an agent, someone has to judge whether it will succeed. If that estimate fails, money or time is lost on impossible jobs.
- The result indicates that the key input is not the chosen model, but how honest the description of what the agent cannot do is.
- A generic description, the kind that usually comes with these products, can take away nearly all the value of the prediction.
The number
0.66 to 0.86 versus 0.45 to 0.56. That is the range of AUC, a measure in which 0.5 equals a random ranking. The first comes from telling the models what the agent cannot do. The second, from describing it only as a generic agent with a browser and code execution.
Context
The authors checked that the result holds with judges from four other model families: Claude, GLM, Kimi and Qwen. In addition, the informed forecasts also rank the verdicts of two human raters, a relevant check because the main judge is another model.
A caveat: accepting a job here is the decision of an automatic judge, not of a paying person. It is also a preprint without peer review, and the posting is all the agent receives, with no conversation with whoever commissioned the work.
At Mamífero we have followed how agents enter the workplace. We reported that Impression-R1 drafts radiology conclusions and that GitHub Copilot adds projects with several agents. Also that ThinkingBox rewards the agent for the database it leaves behind and that Legora raised US$ 550 million.
What’s next
- Work platforms could use forecasts like these to filter which jobs go to an agent.
- The open code allows the test to be repeated with other agents and descriptions.
- It has yet to be checked at larger scale against decisions by people who pay for the work.
Bottom line
Knowing what an agent cannot do turns out to be more useful than knowing which model judges it. The question for those who sell them is whether they are willing to publish that list.
Sources
- Rezaei, M. R., Wang, D., Levert, B. et al., “PAYWORTHY: Can LLMs Predict Which Freelance Jobs an AI Agent Will Complete?”, arXiv, 3 November 2026, 2611.05759. Preprint without peer review; this note is based on the abstract.
- Cover image: helloquence, CC0, via Wikimedia Commons
Edited by Rodrigo Cornejo. How we select and verify: who writes these notes.
