in a post-truth world where the internet is full of non-obvious ai-generated mostly-but-not-quite-true information, academic journals should be more important than ever; unfortunately, the companies that run them had already become complacent and negligent long before LLMs came on the scene.
> in a post-truth world where the internet is full of non-obvious ai-generated mostly-but-not-quite-true information, academic journals should be more important than ever; unfortunately, the companies that run them had already become complacent and negligent long before LLMs came on the scene.
There will have to be institutional adaptation to this, but it will be painful and I think the end-result will be worse that what we had pre-LLM. Things will be much more closed, inaccessible, and based on who you know. The openness of the last ~30 years will end.
The models don't know or need permission to make up references. References are just complicated things to predict in terms of next tokens -- Author names are found in many references, with many journals. Most hallucinated refs use researcher names in the field and credible article titles related to their research.
Agentic models now, however, can correctly search pubmed or OpenAlex and obtain valid refs, so that tell is mostly gone.
Wait until we get the first retractions of papers with AI generated micrographs or blots, and synthetic data plots. That's the bit that really scares me.
The opposite is more interesting, (and the author points it out too). SOTA agents can help run small experiments and help you do the writeup. Should you publish it? I'd personally want to keep a human at least partially in the loop, or have very solid checkers. But for one off personal research it's at the same point as one off personal code: make sure you have a way to test what's valid; but if you do, it often Just Works(tm) Now.
Part of the equation you’re missing is that a human is always in the loop, after you publish.
If you don’t rigorously verify everything by hand before then, you are telling the world that they must do what you should have done for you. Or more likely, you are telling the world that that you expect free intellectual labor and will be ignored and chastised- and cause more scrutiny to fall upon others who did the work properly.
This whole issue seems to hinge on a fundamental disagreement about what science (or mathematics) is. Papers aren't science, they're the receipts. But too many people apparently can't tell the difference, so we have the present blizzard of forged receipts (first from paper mills, now with new lemon scented AI++).
Mathematics and science represent two systems for sorting truth from BS. An exponentially increasing supply of BS makes that even more necessary.
> kinda scary to think about how many scientific papers are probably written using AI these days
Journals now routinely ask you to declare any use of AI in writing. Better authors are up front about its use, so peer review can take this into account.
You can't build trust over science, that's just an error of category. You can only build trust in some people on some topic. One can measure reliability in a process, but if no specific human is involved, there is no literal trust engaged.
Now of course, to each its definition and metaphors are useful tools.
But when one starts a discussion on truth, trust or reality as core topic, it's better to start with what is meant with these words and in which context to which scoped boundaries.
Because of course, science can back a contemporary etnologic account that asserts there is no (longer) different races in homo sapiens, just as much as it can serve as frame for phrenology and whatever supremacism claim.
So no, we didn't build trust in science over centuries, and not even in any scientific person because we don't have that much humans to hold the role of such a trustable entity over such a long period of time.
Social institutions can carry unchanged names over centuries, but that tells nothing about how trustable are the people that inhabit them over time.
LLMs certainly come with their own challenges, but the erosion or scientific institutions with its bizarre scheme of publication process have been there basically as far as I can remember.
Save yourself a slog. The automated checks found that the citations didn't exist and flagged the "papers". A better title would have been "Automated Systems seem to hold back the AI Slop papers. For now." Sweet writing studio, though.
In other news people prompted a model to pick a problem to solve, solve it and write a paper about it. And it did and the paper contains novel contributions.
This experiment was agreed with the managing editor of the multidisciplinary and open-access journal IEEE Access, and IEEE itself - it was red teaming, essentially.
in a post-truth world where the internet is full of non-obvious ai-generated mostly-but-not-quite-true information, academic journals should be more important than ever; unfortunately, the companies that run them had already become complacent and negligent long before LLMs came on the scene.
> in a post-truth world where the internet is full of non-obvious ai-generated mostly-but-not-quite-true information, academic journals should be more important than ever; unfortunately, the companies that run them had already become complacent and negligent long before LLMs came on the scene.
There will have to be institutional adaptation to this, but it will be painful and I think the end-result will be worse that what we had pre-LLM. Things will be much more closed, inaccessible, and based on who you know. The openness of the last ~30 years will end.
>When one model refuses or stalls, I switch models, rephrase prompts, and ask what the next step would look like “hypothetically.”
I wonder If the hypothetically criterion gives the models carte blanche to make up references.
The models don't know or need permission to make up references. References are just complicated things to predict in terms of next tokens -- Author names are found in many references, with many journals. Most hallucinated refs use researcher names in the field and credible article titles related to their research.
Agentic models now, however, can correctly search pubmed or OpenAlex and obtain valid refs, so that tell is mostly gone.
Wait until we get the first retractions of papers with AI generated micrographs or blots, and synthetic data plots. That's the bit that really scares me.
The opposite is more interesting, (and the author points it out too). SOTA agents can help run small experiments and help you do the writeup. Should you publish it? I'd personally want to keep a human at least partially in the loop, or have very solid checkers. But for one off personal research it's at the same point as one off personal code: make sure you have a way to test what's valid; but if you do, it often Just Works(tm) Now.
Part of the equation you’re missing is that a human is always in the loop, after you publish.
If you don’t rigorously verify everything by hand before then, you are telling the world that they must do what you should have done for you. Or more likely, you are telling the world that that you expect free intellectual labor and will be ignored and chastised- and cause more scrutiny to fall upon others who did the work properly.
You're not wrong. Up till recently I'd have agreed without reservation even.
Except in recent months, I've seen modern agents verify things more rigorously than I do now.
Could be a me thing ofc, but I've found that a human's eyes tend to glaze over while the agent just keeps on checking in more and more detail.
Quis custodiet ipsos custodes?
Sidenote: way to flex your "writing studio".
A shed, but a cool shed ig.
This whole issue seems to hinge on a fundamental disagreement about what science (or mathematics) is. Papers aren't science, they're the receipts. But too many people apparently can't tell the difference, so we have the present blizzard of forged receipts (first from paper mills, now with new lemon scented AI++).
Mathematics and science represent two systems for sorting truth from BS. An exponentially increasing supply of BS makes that even more necessary.
Even though these experiments got rejected, I feel like if he had used better AI versions, they might have worked.
Also, kinda scary to think about how many scientific papers are probably written using AI these days. The system might have to change.
> kinda scary to think about how many scientific papers are probably written using AI these days
Journals now routinely ask you to declare any use of AI in writing. Better authors are up front about its use, so peer review can take this into account.
Whenever I read "Here's" in a title, I immediately assume that AI wrote the content.
You can't build trust over science, that's just an error of category. You can only build trust in some people on some topic. One can measure reliability in a process, but if no specific human is involved, there is no literal trust engaged.
Now of course, to each its definition and metaphors are useful tools.
But when one starts a discussion on truth, trust or reality as core topic, it's better to start with what is meant with these words and in which context to which scoped boundaries.
Because of course, science can back a contemporary etnologic account that asserts there is no (longer) different races in homo sapiens, just as much as it can serve as frame for phrenology and whatever supremacism claim.
So no, we didn't build trust in science over centuries, and not even in any scientific person because we don't have that much humans to hold the role of such a trustable entity over such a long period of time.
Social institutions can carry unchanged names over centuries, but that tells nothing about how trustable are the people that inhabit them over time.
LLMs certainly come with their own challenges, but the erosion or scientific institutions with its bizarre scheme of publication process have been there basically as far as I can remember.
There is still no paper or notable result in gender studies published by an AI.
Save yourself a slog. The automated checks found that the citations didn't exist and flagged the "papers". A better title would have been "Automated Systems seem to hold back the AI Slop papers. For now." Sweet writing studio, though.
In other news people prompted a model to pick a problem to solve, solve it and write a paper about it. And it did and the paper contains novel contributions.
https://www.reddit.com/r/OpenAI/comments/1xgqv27/two_physici...
TLDR - both papers got flagged and rejected by automated integrity checks.
The first paper (free-tier chatbots) due to fake references, the second paper (a specialized AI-research tool) due to overreliance on preprints.
Please, even if it is an experiment, don't do this. We don't need more AI slop in the world.
This experiment was agreed with the managing editor of the multidisciplinary and open-access journal IEEE Access, and IEEE itself - it was red teaming, essentially.