University of Basel

University Library

TARCiS Rationale & Explanation Texts

As compiled in a recent scoping review,1 the reporting of citation searching methods is frequently unclear and far from being standardized. For example, “citation searching”, “snowballing”, or “co-citation searching” are sometimes used as methodological umbrella terms but also to denote a specific method such as backward or forward citation searching.1 For clarity, standardized vocabulary is needed. The set of terms brought forward in this recommendation is consistent in itself as well as with the terminology used in PRISMA-S and PRISMA 2020 guidelines2,3 and hence well suited for uniform reporting of citation searching.
  1. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  2. Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. JMLA 2021;109(2):174-200. doi: 10.5195/jmla.2021.962
  3. Page MJ, McKenzie JE, Bossuyt PM, et al. The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. BMJ 2021;372:n71. doi: 10.1136/bmj.n71
Evidence indicates that the ability of citation searching as a supplementary search technique to find additional unique records in a systematic literature search varies between reviews.1 Searches for particular study designs (qualitative, mixed-method, observational, prognostic, or diagnostic test studies) or health science topics such as non-pharmacological, non-clinical, public health, policy making, service delivery, or alternative medicine have been linked with effective supplementary citation searching.2-5 The underlying reasons include poor transferability to text-based searching owing to poor conceptual clarity, inconsistent terminology, or vocabulary overlaps with unrelated topics.6 The ability of citation searching to find any publication type including unpublished or grey literature or literature that is not indexed in major databases (e.g., concerning a developing country) may also be relevant.7 However, a clear categorization of "difficult-to-search-for" topics is currently not possible and it remains for the review authors themselves to judge whether their review topic is likely to fall into this category.
We recommend that persons conducting the search who have difficulty assessing whether the topic is difficult- or easier-to-search-for always opt for citation searching or consult an experienced information specialist.8 If for whatever reason the search strategy does not exhaustively capture the topic, backward and forward citation searching may compensate for the potential loss of information to some extent.
  1. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  2. Levay P, Heath A, Tuvey D. Efficient searching for NICE public health guidelines: Would using fewer sources still find the evidence? Res Synth Methods 2022;13(6):760-89. doi: 10.1002/jrsm.1577
  3. Preston L, Carroll C, Gardois P, et al. Improving search efficiency for systematic reviews of diagnostic test accuracy: an exploratory study to assess the viability of limiting to MEDLINE, EMBASE and reference checking. Syst Rev 2015;4:82. doi: 10.1186/s13643-015-0074-7
  4. Lasda Bergman EM. Social Gerontology-Integrative and Territorial Aspects: A Citation Analysis of Subject Scatter and Database Coverage. Behav Soc Sci Lib 2011;30(3):154-75. doi: 10.1080/01639269.2011.592803
  5. Cooper C, Lovell R, Husk K, et al. Supplementary search methods were more effective and offered better value than bibliographic database searching: A case study from public health and environmental enhancement. Res Synth Methods 2018;9(2):195-223. doi: 10.1002/jrsm.1286
  6. Greenhalgh T, Peacock R. Effectiveness and efficiency of search methods in systematic reviews of complex evidence: audit of primary sources. BMJ 2005;331(7524):1064-65.
  7. Haddaway NR, Collins AM, Coughlin D, et al. The Role of Google Scholar in Evidence Reviews and Its Applicability to Grey Literature Searching. PLoS One 2015;10(9):e0138237. doi: 10.1371/journal.pone.0138237
  8. Rethlefsen ML, Murad MH, Livingston EH. Engaging medical librarians to improve the quality of review articles. JAMA 2014;312(10):999-1000. doi: 10.1001/jama.2014.9263
Evidence indicates that the ability of citation searching as a supplementary search technique to find additional unique references in a systematic literature search varies between reviews.1 Searches for clearly defined clinical interventions as part of Participant-Intervention-Comparison-Outcome (PICO)-questions have been linked with less effective supplementary citation searching, especially when the search strategies are sensitive and conducted in several databases. However, a clear categorization of "easier-to-search-for" topics is currently not possible and it remains for the review authors themselves to judge whether their review topic is likely to fall into this category.
By checking reference lists within the full-texts of seed references, review authors can test the sensitivity of their primary search strategy (i.e., electronic database search).2 Should no additional relevant, unique studies be found, the primary search may have been sensitive enough. Should additional relevant, unique studies be found, it may be an indication that the primary search was not sensitive enough.
We recommend that persons conducting the search who have difficulty assessing whether the topic is difficult- or easier-to-search-for opt for citation searching or consult an experienced information specialist.3 If for whatever reason the search strategy does not exhaustively capture the topic, backward and forward citation searching may compensate for the potential loss of information to some extent.
  1. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  2. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev 2011(8):MR000026. doi: 10.1002/14651858.MR000026.pub2
  3. Rethlefsen ML, Murad MH, Livingston EH. Engaging medical librarians to improve the quality of review articles. JAMA 2014;312(10):999-1000. doi: 10.1001/jama.2014.9263
The more seed references used, the better the chance that citation searching finds additional relevant unique records. While using only a sample of the included records as seed references may be enough, there is currently no evidence that could help decide how many seeds are needed or how to decide which may perform better. Hence, we recommend using all the records that meet the inclusion criteria of the review after full-text screening of the primary database search results.
However, review authors may deviate from this recommendation if they deal with a very small or large number of included records. A very small number of included records may not yield additional relevant records or only have limited value. In this case, review authors could use further records as seed references for citation searching (e.g., systematic reviews on the topic that were flagged during the screening phase).27 A very large number of included records could lead to too many records to screen. In this case, review authors may use a selected sample of included records as seed references for citation searching. In the event of such deviation, authors should describe their rationale and sampling method (e.g., random sample).
  1. Westphal A, Kriston L, Holzel LP, et al. Efficiency and contribution of strategies for finding randomized controlled trials: a case study from a systematic review on therapeutic interventions of chronic depression. J Pub Health Res 2014;3(2):177. doi: 10.4081/jphr.2014.177
Citation searching workflows encompass two consecutive steps: retrieval of records and screening of retrieved records for eligibility. When using an electronic citation index for citation searching, retrieval and screening are usually separated. While forward citation searching requires a citation index, backward citation searching can also be performed by manually checking the reference lists of the seed references. Reference list checking is sometimes part of an established workflow, e.g., done during eligibility assessment of the full-text record or during data extraction.1 Merging these two steps has the benefit that researchers know the context in which a reference was used and that all references can be screened. However, reference list checking has three disadvantages: (i) the retrieval and screening phases are no longer separated which makes reporting of the methods/results difficult and unclear, (ii) citations from reference list checking cannot be deduplicated against each other and/or against the primary search results which may add an unnecessarily high workload (see recommendation 7), and (iii) eligibility assessments are restricted to the titles (instead of titles and abstracts) which could lead to relevant records being overlooked due to uninformative titles mentioned in vague contexts.
In recent years, online citation searching options via citation indexes or free to access citation searching tools have become more readily available leading to faster and easier procedures.2-5 More and even better tools to facilitate this workflow are expected in the future. Combining citation searching via citation indexes with automated deduplication (free online tools available)6-8 makes this recommendation feasible. A caveat is that a search in a single citation index will in most cases fail to retrieve all the cited references.9,10 Thus, references to some documents (such as websites, registry entries or grey literature) that are less likely to be indexed in databases may only be retrievable by checking reference lists or only in some citation indexes.11
  1. Horsley T, Dingwall O, Sampson M. Checking reference lists to find additional studies for systematic reviews. Cochrane Database Syst Rev 2011(8):MR000026. doi: 10.1002/14651858.MR000026.pub2
  2. Janssens A, Gwinn M, Brockman JE, et al. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol 2020;20(1):25. doi: 10.1186/s12874-020-0907-5
  3. Haddaway NR, Grainger MJ, Gray CT. Citationchaser: A tool for transparent and efficient forward and backward citation chasing in systematic searching. Res Synth Methods 2022;13(4):533-45. doi: 10.1002/jrsm.1563
  4. Smalheiser NR, Schneider J, Torvik VI, et al. The Citation Cloud of a biomedical article: a free, public, web-based tool enabling citation analysis. JMLA 2022;110(1):103-08. doi: 10.5195/jmla.2022.1117
  5. Pallath A, Zhang Q. Paperfetcher: A tool to automate handsearching and citation searching for systematic reviews. Res Synth Methods 2023;14(2):323-35. doi: 10.1002/jrsm.1604
  6. Ouzzani M, Hammady H, Fedorowicz Z, et al. Rayyan-a web and mobile app for systematic reviews. Syst Rev 2016;5(1):210. doi: 10.1186/s13643-016-0384-4
  7. Zotero Version 6: 2023. Available from: www.zotero.org
  8. Clark J, Glasziou P, Del Mar C, et al. A full systematic review was completed in 2 weeks using automation tools: a case study. J Clin Epidemiol 2020;121:81-90. doi: 10.1016/j.jclinepi.2020.01.008
  9. Bramer WM. Reference checking for systematic reviews using Endnote. JMLA 2018;106(4):542-46. doi: 10.5195/jmla.2018.489
  10. ClarivateAnalytics. Cited Reference Search: 2023. Available from: https://webofscience.help.clarivate.com/en-us/Content/cited-reference-search.htm
  11. Cooper C, Booth A, Britten N, et al. A comparison of results of empirical studies of supplementary search techniques and recommendations in review methodology handbooks: a methodological review. Syst Rev 2017;6(1):234. doi: 10.1186/s13643-017-0625-1
A single citation index or citation analysis tool may not cover all seed references and very likely will not find all the citing and cited literature. The reasons for this are that citation indexes do not offer 100% coverage as some references are currently not indexed in one or several citation index(es)1 as well as data quality issues.2 Evidence indicates that when using more than one citation index for citation searching, the results of the different indexes can complement each other.3-5 Thus, retrieval of backward and forward citation searching results from more than one citation index or citation analysis tool (e.g., Lens.org via citationchaser, Scopus, citation indexes in Web of Science) followed by deduplication (see recommendation 6) can increase the sensitivity of citation searching. It is similar to the complementary effect of using multiple electronic databases for the primary database search, which is the gold-standard in systematic search workflows.6 In recent years, online citation searching options have increased and many open access tools make rapid electronic citation searching universally accessible.7-10
  1. Falagas ME, Pitsouni EI, Malietzis GA, et al. Comparison of PubMed, Scopus, Web of Science, and Google Scholar: Strengths and weaknesses. Faseb J 2008;22(2):338-42. doi: 10.1096/fj.07-9492LSF
  2. Franceschini F, Maisano D, Mastrogiacomo L. The museum of errors/horrors in Scopus. J Informetrics 2016;10(1):174-82. doi: 10.1016/j.joi.2015.11.006
  3. Levay P, Ainsworth N, Kettle R, et al. Identifying evidence for public health guidance: a comparison of citation searching with Web of Science and Google Scholar. Res Synth Methods 2016;7(1):34-45. doi: 10.1002/jrsm.1158
  4. Rogers M, Bethel A, Briscoe S. Resources for forwards citation searching for implementation studies in dementia care: A case study comparing Web of Science and Scopus. Res Synth Methods 2020;11(3):379-86. doi: 10.1002/jrsm.1400
  5. Wright K, Golder S, Rodriguez-Lopez R. Citation searching: a systematic review case study of multiple risk behaviour interventions. BMC Med Res Methodol 2014;14:73. doi: 10.1186/1471-2288-14-73
  6. Higgins J, Thomas J, Chandler J, et al. Cochrane Handbook for Systematic Reviews of Interventions version 6.3 (updated February 2022): Cochrane; 2022. Available from: www.training.cochrane.org/handbook
  7. Janssens A, Gwinn M, Brockman JE, et al. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol 2020;20(1):25. doi: 10.1186/s12874-020-0907-5
  8. Haddaway NR, Grainger MJ, Gray CT. Citationchaser: A tool for transparent and efficient forward and backward citation chasing in systematic searching. Res Synth Methods 2022;13(4):533-45. doi: 10.1002/jrsm.1563
  9. Smalheiser NR, Schneider J, Torvik VI, et al. The Citation Cloud of a biomedical article: a free, public, web-based tool enabling citation analysis. JMLA 2022;110(1):103-08. doi: 10.5195/jmla.2022.1117
  10. Pallath A, Zhang Q. Paperfetcher: A tool to automate handsearching and citation searching for systematic reviews. Res Synth Methods 2023;14(2):323-35. doi: 10.1002/jrsm.1604
The concept of citation searching as a supplementary search method relies on the notion that reference and cited-by lists of eligible references are topically related to these references.1 This implies a considerable degree of overlap within these lists leading to several duplicates. Furthermore, the overlap likely also extends to the results of the primary database search that was performed on the same topic. Based on these considerations and on the fact that the results of the primary database search have already been screened for eligibility, the screening load of citation searching results can be significantly cut by removing those references that have already been screened for eligibility (deduplication against the primary database search) and those that appear as duplicates during citation searching.2 Depending on the method of deduplication, this can be done in one go.
While deduplication can be conducted manually, nowadays standard bibliographic management software and specialized tools provide automated deduplication solutions, allowing for easier and faster processing.2-4
If citation searching leads to only a very small number of results, omission of the deduplication step can be considered to save time and administrative effort.
  1. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  2. Bramer WM. Reference checking for systematic reviews using Endnote. JMLA 2018;106(4):542-46. doi: 10.5195/jmla.2018.489
  3. Borissov N, Haas Q, Minder B, et al. Reducing systematic review burden using Deduklick: a novel, automated, reliable, and explainable deduplication algorithm to foster medical research. Syst Rev 2022;11(1):172. doi: 10.1186/s13643-022-02045-9
  4. Chai KEK, Lines RLJ, Gucciardi DF, et al. Research Screener: a machine learning tool to semi-automate abstract screening for systematic reviews. Syst Rev 2021;10(1):93. doi: 10.1186/s13643-021-01635-3
Citation searching methods can be conducted over one or more iterations, a process we refer to as iterative citation searching.1 The first iteration is based on the original seed references (see recommendation 4). If eligibility screening of the results of this first iteration leads to the inclusion of further eligible records, these records serve as new seed references for the second iteration and so forth. There is evidence that conducting iterative citation searching can contribute to the identification of more eligible records.1-4
Since iterations beyond the first round of citation searching require additional time and effort and may interrupt the ongoing review process, the decision in favor of or against further iterations should be guided by an informal cost-benefit assessment. Relevant factors to be assessed include the review topic (difficult- or easier-to-search-for), sensitivity of the primary search, aim for completeness of the literature search, and the estimated potential benefit of the iteration(s) (e.g., based on the number/percentage of included records found with the previous citation searching iteration).
Review authors should report the number of iterations and possibly the reason for stopping if the last iteration still retrieved additional eligible records.
Please note that stating "citation searching was done on all included records" can lead to confusion. Most authors may mean all records included after full-text screening of the primary search results. However, strictly speaking, "all included records" also includes the records retrieved via citation searching. The latter interpretation implies that iterative citation searching is required until the last iteration leads to no further identification of eligible records.
As outlined in the rationale of recommendation 7, results of citation searching iterations can be deduplicated against all previously retrieved records to reduce the screening load.
  1. Hinde S, Spackman E. Bidirectional citation searching to completion: an exploration of literature searching methods. Pharmacoeconomics 2015;33(1):5-11. doi: 10.1007/s40273-014-0205-3
  2. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  3. Janssens AC. Updating systematic reviews and meta-analyses, the easy way: 2021. Available from: https://cecilejanssens.medium.com/updating-systematic-reviews-and-meta-analyses-the-easy-way-cbb2e23b48b9
  4. Kleijnen J, Knipschild P. The comprehensiveness of Medline and Embase computer searches - Searches for controlled trials of homoeopathy, ascorbic acid for common cold and ginkgo biloba for cerebral insufficiency and intermittent claudication. Pharm Weekbl Sci Ed 1992;14(5):316-20. doi: 10.1007/BF01977620
We refer to stand-alone citation searching when any form of citation searching is used as the primary search method without extensive prior database searching.1 This is contrary to citation searching as a supplementary search method to a primary database search. Seed references for stand-alone citation searching could, for example, be records from researchers’ personal collections or retrieved from less sensitive literature searches. Stand-alone citation searching can be based on a broad set of seed references. It can comprise backward and forward citation searching as well as indirect methods that collect co-citing and co-cited references.
When study authors have replicated published systematic reviews with stand-alone citation searching, they mostly missed literature that was included in the systematic review.2-5 Since search methods for systematic reviews and scoping reviews should aim at completeness of recall, stand-alone citation searching is not a suitable method for these types of literature review.
  1. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  2. Janssens AC, Gwinn M, Brockman JE, et al. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol 2020;20(1):25. doi: 10.1186/s12874-020-0907-5
  3. Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol 2015;15:84. doi: 10.1186/s12874-015-0077-z
  4. Belter CW. Citation analysis as a literature search method for systematic reviews. JASIS&T 2016;67(11):2766-77. doi: 10.1002/asi.23605
  5. Belter CW. A relevance ranking method for citation-based search results. Scientometrics 2017;112(2):731-46. doi: 10.1007/s11192-017-2406-y
Relevant guidance for researchers conducting citation searching in systematic literature searching can be found in item 5 of PRISMA-S.1 Accordingly, required reporting items are the directionality of citation searching (examination of cited or citing references), methods and resources used for citation searching (bibliographies in full text articles or citation indexes), and the seed references that citation searching was performed upon.1 Additional information for the reporting of citation searching can be found in PRISMA-S items 1 (database name), 13 (dates of searches), and 16 (deduplication).1 While PRISMA-S can be seen as the minimum reporting standard for citation searching as a supplementary search technique, other important elements that emerged from our scoping review2 need to be reported to achieve full transparency and/or reproducibility. These elements are listed in recommendation 10 as a supplement to PRISMA-S to comprehensively guide the reporting of supplementary citation searching in systematic literature searching.
Concerning reporting of citation searching results in the PRISMA 2020 flow diagram,3 two variants are possible: (i) reporting of deduplicated records only which are additional to the primary search results or (ii) reporting of all retrieved records followed by insertion of an additional box where the number of deduplicated records is reported.
Please note that the detail of the citation searching methods do not have to be reported in the main methods of a study. Detailed search information can be provided in an appendix or an online public data repository.

 

Example 1 for good reporting:"As supplementary search methods, we performed […] direct forward and backward CT of included studies and pertinent review articles that were flagged during the screening of search results (on February 10, 2021). For forward CT, we used Scopus, Web of Science [core collection as provided by the University of Basel; Editions = SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, BKCI-S, BKCI-SSH, ESCI, CCR-EXPANDED, IC], and Google Scholar. For backward CT, we used Scopus and, if seed references were not indexed in Scopus, we manually extracted the seed references' reference list. We iteratively repeated forward and backward CT on newly identified eligible references until no further eligible references or pertinent reviews could be identified (three iterations; the last iteration on May 5, 2021)."2
Example 2 for good reporting:"To supplement the database searches, we performed a forward (citing) and backwards (cited) citation analysis on 2 August 2022 using SpiderCite (https://sr-accelerator.com/#/spidercite)."4
Example 3 for good reporting:"Reference lists of any included studies and retrieved relevant SRs published in the last five years were checked for any eligible studies that might have been missed by the database searches."5

 

  1. Rethlefsen ML, Kirtley S, Waffenschmidt S, et al. PRISMA-S: an extension to the PRISMA statement for reporting literature searches in systematic reviews. JMLA 2021;109(2):174-200. doi: 10.5195/jmla.2021.962
  2. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
  3. Rethlefsen ML, Page MJ. PRISMA 2020 and PRISMA-S: common questions on tracking records and the flow diagram. JMLA 2022;110(2):253-57. doi: 10.5195/jmla.2022.1449
  4. Nightingale R, Yadav K, Hamill L, et al. Misdiagnosis of Uncomplicated Cellulitis: a Systematic Review and Meta-analysis. J Gen Intern Med 2023 doi: 10.1007/s11606-023-08229-w
  5. Fautrel B, Patterson J, Bowe C, et al. Systematic review on the use of biologics in adult-onset still's disease. Sem Arthritis Rheum 2023;58:152139. doi: 10.1016/j.semarthrit.2022.152139
Indirect citation searching involves the collection and screening for eligibility of records that share references in their bibliography or citations with one of the seed references (i.e., co-citing or co-cited references).1 Indirect citation searching typically retrieves a large volume of records to be screened.2,3 Therefore, prioritization algorithms for the screening of records and cut-offs that may discriminate between potentially relevant and non-relevant records have been proposed that aim at reducing the workload of eligibility screening.4,5 The methodological studies that have pioneered indirect citation searching methods for health-related topics have so far exclusively focused on stand-alone citation searching.6 It is currently unclear whether the added workload and resources for searching and screening warrant indirect citation searching methods as supplementary search techniques in systematic reviews of any type (qualitative/quantitative, difficult/easier-to-search-for).
  1. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Using citation tracking for systematic literature searching - study protocol for a scoping review of methodological studies and a Delphi study. F1000Research 2020;9(1386):1386.
  2. Belter CW. Citation analysis as a literature search method for systematic reviews. JASIS&T 2016;67(11):2766-77. doi: 10.1002/asi.23605
  3. Janssens AC, Gwinn M. Novel citation-based search method for scientific literature: application to meta-analyses. BMC Med Res Methodol 2015;15:84. doi: 10.1186/s12874-015-0077-z
  4. Belter CW. A relevance ranking method for citation-based search results. Scientometrics 2017;112(2):731-46. doi: 10.1007/s11192-017-2406-y
  5. Janssens A, Gwinn M, Brockman JE, et al. Novel citation-based search method for scientific literature: a validation study. BMC Med Res Methodol 2020;20(1):25. doi: 10.1186/s12874-020-0907-5
  6. Hirt J, Nordhausen T, Appenzeller-Herzog C, et al. Citation tracking for systematic literature searching: A scoping review. Res Synth Methods 2023;14(3):563-79. doi: 10.1002/jrsm.1635
To top

Quick Links