The importance of
high-quality content:
curation and reevaluation in Scopus
Scopus uniquely combines a comprehensive, curated abstract and citation database with
enriched data and linked scholarly content.
Users can quickly find relevant and trusted research, identify experts, and access reliable
data, metrics and analytical tools to support confident decisions around research strategy
— all from one database.
New content is added to Scopus after a rigorous evaluation process by the subject experts
of the Content Selection and Advisory Board (CSAB). The experts of the CSAB determine
whether a title is eligible for indexing in Scopus using the Scopus title selection criteria.
They also continuously re-evaluate titles indexed in Scopus to ensure the quality of the
existing content is maintained.
In this article we discuss the importance of a rigorous content curation mechanism to exclude
poor-quality and predatory publications from Scopus. We use the terms “poor-quality”
and “predatory” journals throughout this piece as they are not the same. Poor-quality
journals may not meet certain quality standards based on their publication performance
or bibliometric indicators. However, these journals do not necessarily engage in unethical
publication practice.
https://www.elsevier.com/__data/assets/pdf_file/0004/891058/ACADLIBSCARTImportance-of-high-quality-contentWEB.pdf
The importance of using research information that you can trust
Researchers need curated, high-quality content. Poor-quality journals may be easy to identify
from bibliometric indicators’ perspective, however, particularly journals in niche areas may
have limited readership or low citations, in which case bibliometrics might not necessarily
be an indicator of low quality. Predatory journals are a threat to the integrity of science.
Although usage of the term predatory publishing is widespread, at the practical level it
remains ill-defined and subject to personal interpretation and judgement. Specifically:
• Labeling a journal predatory may be controversial if there is no evidence. Publishers that
have, in someone’s subjective opinion, some predatory publications may also publish
high-quality journals.
• It is not always straightforward to identify predatory journals. Journals change their
editorial policies over time; a journal that did not start off as predatory may become so
over the years or vice versa.
There have been several attempts to define predatory publishing. Scopus has been using the
following guidelines and definitions.
• Organizations including the Committee on Publication Ethics (COPE) and the World
Association of Medical Editors (WAME) define global best editorial and publication
practices —predatory journals do not meet them.
• In 2019 a group of authors reached a consensus definition of predatory publications
which was subsequently published as Defining predatory journals and responding to the
threat they pose: a modified Delphi consensus process. In summary, this definition reads:
“Predatory journals and publishers are entities that prioritize self-interest at the expense of
scholarship and are characterized by false or misleading information, deviation from best
editorial and publication practices, a lack of transparency, and/or the use of aggressive and
indiscriminate solicitation practices.”
In recent years, the growth of global research output and the expansion of various
publishing business models have led to a rise in the number of newly launched journals,
including titles that could be considered predatory. Despite forming a small proportion of
all journals published globally, predatory journals are a significant challenge that concerns
all research publishing stakeholders: authors, editors, researchers, research institutions,
publishers, funding bodies and governments.
The rise of new predatory journals is partly driven by the increasing pressure on researchers
to publish, in order to secure funding and to advance their careers. In some instances, these
forces may encourage authors to be less scrupulous about where their work appears than
might previously have been the case. Many governments and institutions have sought to
establish expectations for their country’s research by issuing mandates for their researchers
to publish in Scopus or Web of Science (WoS) indexed titles; this has inevitably induced many
predatory journals and publishers to target these databases for their business practices.
Scopus and the CSAB seek to minimize the impact of poor-quality or predatory journals and
ensure that Scopus users get the best possible evidence for their work.
Evaluation and re-evaluation of content covered by Scopus
Only the most reliable journals and their content are available in Scopus as they are carefully
curated and ultimately selected by the independent CSAB, an international group of subject
experts across all fields of research. Year round, the CSAB are reviewing titles that are
submitted to Scopus.
In addition, a process of continuous monitoring and re-evaluation was developed ensuring
that the quality of the existing content is also maintained. Out of the full corpus of journals
covered by Scopus, journals considered predatory or that perform poorly are identified
for comprehensive re-evaluation by the CSAB. Four ways a journal can be flagged for
re-evaluation include:
1. The journal is underperforming as it does not meet any of the three metrics and
benchmarks for journals in the same subject area. The indicators have been developed
and agreed by the CSAB in partnership with the Scopus team.
2. Concerns about the publication standards of the journal or publisher have been raised
by formal complaints.
3. The journal shows outlier behavior based on its publishing performance in Scopus.
4. Continuous curation based on CSAB feedback.
Underperforming titles are detected using the three metrics shown in Table 1.
The CSAB
re-evaluates journals that fail to meet any of the three metrics and benchmarks for two
consecutive years.
Metric Benchmarks and Explanation
Self-citation rate The journal has a substantially higher self-citation rate, when
compared to peer journals in its subject field.
Total citation rate The journal received a substantially lower number of citations,
when compared to peer journals in its subject field.
CiteScore The journal has a substantially lower CiteScore, when compared to
peer journals in its subject field.
Table 1. The three metrics and benchmarks to identify underperforming journals for re-evaluation.
A journal can also be identified for re-evaluation if concerns about its publication standards
are raised, such as the quality or editorial practices of specific titles, or other issues that
impact their suitability for continued coverage.
Outlier behavior identified through statistical analysis is particularly effective in flagging
potential predatory journals. Scopus runs an algorithm that flags journals based on a
number of predictors, including sudden change in output volume, sudden change in
publishing country and/or affiliations and high journal/author self-citation rates.
Finally, the Scopus team collects and analyzes previous CSAB title evaluation feedback and
continuously flags titles for re-evaluation.
The CSAB re-evaluates all titles identified for underperformance, publication standard
concerns, outlier behavior, or continuous content curation. A CSAB subject expert
re-evaluates these titles with the same criteria used for the selection of new journals.
Because poor-performing journals may still be relevant for the communities they serve
and because predatory publishing is subject to personal interpretation, independent
review of individual journals by academic subject experts in each field is essential. Not all
journals that are initially identified as poor-quality or predatory are confirmed to be below
standards after re-evaluation by the CSAB, and they may decide to continue coverage.
If the CSAB determines that a publication is no longer meeting the standards for inclusion
in Scopus anymore, indexing of new content from that journal is discontinued. To make
sure that content from journals undergoing re-evaluation is not included in Scopus during
the re-evaluation process, the content flow is put on hold pending the review. For journals
that are discontinued or put on hold, the content already indexed in Scopus remains. This
as a matter of scientific record and to ensure the stability and consistency of research trend
analytics which is a core value of Scopus. In exceptional cases of proven severe unethical
publication practice, content already indexed in Scopus may be removed.
The Scopus team informs publishers of the titles about the re-evaluation procedure as well
as its outcome. All journals discontinued in Scopus are publicly available via the Scopus
discontinued sources list.
Since 2016, the CSAB has re-evaluated 990 titles published by 539 different publishers
leading to 536 titles discontinued for indexing. In Figure 1 below, the results are broken
down by the original reason why the journal was identified for re-evaluation.
“Journal hijacking” refers to cybercrimes that involve building fake websites that mimic the
names, editorial boards, ISSNs, similar URLs, or journal indexing information of reputable
journals. [1,2] This, for example, occurs in but is not exclusive to print-only journals or
regional-language journals, for which authentic websites are not always easily detectable.
In The guardians of Scopus, Prof. Sack mentions “Whether or not a journal is predatory is
not a binary decision; there is a broad spectrum of predatory journal behaviors.” Journal
hijacking similarly falls within this spectrum and stands on the deceptive side. We have
stringent procedures in place to detect hijacked journals as early as possible and to ensure
that the content indexed in Scopus is from the legitimate source.
When a journal is hijacked, it does not necessarily mean that the hijacked content is also
indexed in Scopus. Despite our continuous efforts to remain vigilant, organizations behind
this activity are clever and can be extremely effective in their endeavors, such that Scopus is
sometimes misled and content from a hijacked journal gets in. This can occur when journals
update URLs or content feeds. We have strict validations in place to manage these changes
and continue to improve our procedures. In those cases where there is evidence that the
articles in Scopus are not authentic, that is, are not sourced from the genuine journal, the
content will be removed from Scopus. We realize this may have huge implications for the
authors involved, therefore we only do this by exception and as soon as possible after the
content has been loaded into Scopus.
[1] Butler, D. Sham journals scam authors. Nature 495, 421–422 (2013). https://doi.org/10.1038/495421a
[2] COPE Council. COPE Discussion Document: Predatory Publishing. November 2019. https://doi.org/10.24318/
cope.2019.3.6.
Scopus responds to community concerns about predatory journals
It is not surprising that Scopus, as a leading abstract and citation database, receives questions
about possible predatory content. All journals within Scopus face re-evaluation based on the
criteria outlined above, and the Scopus team also responds to community input.
In February 2021, Nature published a news item on the publication of a research article
by authors from Charles University and the Economics Institute of the Czech Academy
of Sciences in the journal Scientometrics (now retracted). The article is based on a study
from 2017 looking into predatory publishing using Beall’s list as a definition for predatory
journals and analyzing the geographical origin of authors publishing in these journals.
Please note that since January 2017, Beall’s List is no longer being maintained. All of
the 137 suspicious titles mentioned in the paper have gone through the re-evaluation
process and as a result for 97 titles (71%), the decision was made to discontinue coverage
in Scopus. Also, all other journals listed by Beall that are mentioned in the paper have
gone through the re-evaluation process and as a result 65% of these titles were also
discontinued. Meanwhile, the Scientometrics article has been retracted because some of
the findings are found to be unreliable. It also emphasizes the risks of the continued use
of blacklists and its criteria for characterizing predatory journals or publishers.
Throughout 2020, the Scopus team was in continuous contact with national organizations
such as the Russian Academy of Sciences (RAS) and the Indian University Grants Commission
(UGC), both of which shared their investigation results on publication malpractices for over
100 journals of concern, some of which were included in both lists. All journals affected
were actively investigated and those journals for which publication concerns were validated
were sent for re-evaluation by the CSAB. As a result, approximately 45 journals were
discontinued or were no longer active and publicly announced in the Scopus discontinued
sources list. The UGC Consortium for Academic and Research Ethics (UGC-CARE) approved
list of journals includes all journals covered by Scopus.
In January 2019 the paper Is Scopus polluting its own database by indexing junk articles? was
posted on the institutional repository of the Ludwig Maximilian University of Munich. The
unpublished paper is a case study of five journals that are claimed to pollute the Scopus
database quality. Eventually, for all five journals mentioned, the CSAB made the decision to
discontinue coverage in Scopus.
Conclusion
Scopus and the CSAB recognize they have a shared responsibility for the ongoing curation
and re-evaluation of the database to ensure that only journals containing high quality
content remain actively indexed and those journals that do not adhere to ethical publication
standards are kept out.
With its independent expert title selection, re-evaluation and publication discontinuation
practices, Scopus puts forth a state-of-the-art defense against predatory publishing.
The decision to re-evaluate and potentially exclude a journal takes time and is complex, but
if done properly this process minimizes the risk of discontinuing legitimate sources. It is an
ongoing challenge to detect journals that begin to show signs of practices associated with
predatory journals. Along with their current rigor in curation and re-evaluation, Scopus and
the CSAB are committed to developing new approaches to ensure that the combined quality
and breadth of Scopus content continue to be unrivalled and highly trusted.
Useful sources
• Scopus Content Selection and Advisory Board
• Title selection criteria
• Committee on Publication Ethics (COPE)
• World Association of Medical Editors (WAME)
• Principles of Transparency and Best Practice in Scholarly Publishing
• Nature. 2019 Dec; 576(7786): 210–212. DOI: 10.1038/d41586-019-03759-y
• BMJ Open 2020 10(2) 10:e035561 DOI: 10.1136/bmjopen-2019-035561
• Scopus’ discontinued sources list
• Scopus Re-evaluation Process
• Scopus curation and re-evaluation webinar
• The importance of high-quality content in Scopus (Scopus blog)
Written by:
Karen Holland, CSAB Chair—Nursing; Health Professions
Prof. Peter Brimblecombe, CSAB Chair—Environmental Science
Dr. Wim Meester, Director of Product Management, Scopus
Tracy Chen, Product Manager, Scopus
