A new AI literature-search study: when were the chatbots actually tested?

The paper is new. The chatbot sessions aren't. Comparing AI chat tools and PubMed for medical literature searches appeared in the Journal of the Medical Library Association this October. In its methods, the authors date their searches to August and September 2024. They also explain that participants reported model versions inconsistently, and that many versions have since been superseded. That doesn't make the work useless. Librarians checked whether the suggested articles existed and were relevant, and found fabricated citations, repeated results and inconsistent lists. But I wouldn't use those results to rank the tools someone can open today. The authors explicitly say a definitive verdict on a specific product wasn't their goal. There's another boundary: they didn't assess the accuracy or recency of the information inside the articles. A real, relevant citation wasn't a certificate that its findings were sound. If you're choosing an AI tool to help find medical reading, ask for the test dates and exact versions before comparing scores. Use the paper's failure patterns to decide what to check in a current trial, not as a current buying table. Otherwise a careful study can leave someone choosing between tools that no longer exist in the form tested.