"These are important questions because scientists are increasingly overwhelmed with the volume of new work posted on preprint servers and published in journals. As a result, traditional quality signals used for triaging papers, such as journal, conference venue, and institution, are becoming less reliable."
The diagnosis is right but the treat is dead wrong. Instead of silly scoring systems, please improve recommender systems for papers. In this space, also opt-in and query-based personalization would be okay.
Recommender systems for papers tend to be pretty bad, so there's a lot of room for improvement. I'll use Semantic Scholar as an example. I have a bunch of folders in what they call a "Library" with recommendations turned on. Semantic Scholar tends to recommend things that are in the same general area but not specific enough. So I guess that Semantic Scholar seems to interpret adding a paper to a folder as expanding the scope of the folder, but it could be narrowing. There's no way to distinguish between the two. Their recommender system is supposed to magically figure it out. Some way to add additional context like relevant keywords or a way to select which parts of the papers are relevant would be helpful. As it stands, I have to repeatedly thumbs down recommendations, and Semantic Scholar doesn't figure out what I mean from that vague signal and instead stops recommending much anything. It's not that there are no additional papers to go into these folders either as I've added more over time that I've found through other means.
"These are important questions because scientists are increasingly overwhelmed with the volume of new work posted on preprint servers and published in journals. As a result, traditional quality signals used for triaging papers, such as journal, conference venue, and institution, are becoming less reliable."
The diagnosis is right but the treat is dead wrong. Instead of silly scoring systems, please improve recommender systems for papers. In this space, also opt-in and query-based personalization would be okay.
Recommender systems for papers tend to be pretty bad, so there's a lot of room for improvement. I'll use Semantic Scholar as an example. I have a bunch of folders in what they call a "Library" with recommendations turned on. Semantic Scholar tends to recommend things that are in the same general area but not specific enough. So I guess that Semantic Scholar seems to interpret adding a paper to a folder as expanding the scope of the folder, but it could be narrowing. There's no way to distinguish between the two. Their recommender system is supposed to magically figure it out. Some way to add additional context like relevant keywords or a way to select which parts of the papers are relevant would be helpful. As it stands, I have to repeatedly thumbs down recommendations, and Semantic Scholar doesn't figure out what I mean from that vague signal and instead stops recommending much anything. It's not that there are no additional papers to go into these folders either as I've added more over time that I've found through other means.
To save people the bother:
An LLM read a paper and concluded it was a top 1% paper.
Everyone involved were happy.
It actually sounds like not everyone involved is happy. Maybe read the article, because it's a criticism, not an endorsement.