Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Expert searching - what are we missing? Sarah Hayman
Similar to Expert searching - what are we missing? Sarah Hayman (20)
Recently uploaded
Recently uploaded (20)
Expert searching - what are we missing? Sarah Hayman
- 1. Expert searching: what are we missing? Sarah Hayman, Research Fellow (Evidence) HLA PD Development Day 10 July 2012
- 2. Introduction and overview Where I work: CareSearch What I will discuss today: – overview of search in systematic reviews – analysis of sources – terms, testing, filters – checking your retrievals – Grey literature
- 3. Search in systematic reviews Importance of the search Comprehensiveness Health professionals use them in providing care Review question and PICO Population (participants, patients) Interventions Comparisons Outcomes Example: Does physical rehabilitation benefit older people in long-term care? P - older people in long-term care I - physical rehabilitation C - no intervention (or alternative intervention) O - improved physical function
- 4. Search in systematic reviews (ctd.) Types of systematic review 3 Cochrane types: Intervention reviews, Diagnostic test accuracy reviews, Methodology reviews Others often mentioned: Effectiveness studies, Qualitative studies, Economic evaluations, Meta-analyses Record-keeping - for standards compliance - for transparency and reproducibility - for iterative development of your searches - for validation of your searches Record (as you go, from the start) • which databases were searched and when • the list of search terms used and the combinations in which they were applied • the search results
- 5. Analysing bibliographic database sources Identify all electronic database sources for your review Medline, EMBASE, CENTRAL, PubMed Subject-specific databases (e.g. CINAHL, PEDro, AgeLine, PsycINFO) Grey literature databases Understand the sources Published search strategies in systematic reviews Help sections for each database Papers published in library journals Health librarians’ discussion lists Health librarians’ blogs and wikis (eg Laika’s MedLibLog and KSS)
- 6. Analysing terminology Identify synonyms, including different terms in different countries Check history and relationships of MeSH terms Consult with experts Develop a term identification set of references Use textual analysis tools if necessary (e.g. Concordance, Writewords))
- 8. What can this approach offer? It is: • evidence-based • iterative We examine terminology in records that we know are relevant We can identify and test which terms are likely to retrieve the most results We can analyse where terms are occurring in the records and how indexing has been done We can develop search strategies to use for free-text searching in non- indexed sources and sources indexed without MeSH headings We can adjust searches for greater sensitivity or specificity Note: free online alternatives exist (e.g. Writewords)
- 9. Using search filters Definition Search filters are predetermined literature search strategies, developed using an explicit methodology and tested using a gold standard test comparison study design (http://www.flinders.edu.au/clinical-change/research/search-filters.cfm ) Types: Methodological and subject-based Further reading: http://www.york.ac.uk/inst/crd/intertasc/ https://sites.google.com/site/healthliteraturesearchers/tools/search-filter http://pubmedsearches.blogspot.com.au/ http://www.ncbi.nlm.nih.gov/pubmed/clinical http://www.caresearch.com.au/caresearch/tabid/371/Default.aspx
- 10. Analysing your results and validating your search Use analysis to: 1. Improve your search for this particular review 2. Provide evidence of the validity of your search for this and future reviews Consult with your subject experts Analyse “missed” items Consider an evaluation instrument such as PRESS (Peer Review of Electronic Search Strategies) see suggestions at: https://sites.google.com/site/healthliteraturesearchers/tools/search-strategy-peer-review-to
- 12. Grey literature Not formally published Not accessible through databases such as Medline Can include: – Conference papers – Government reports – Theses and dissertations Can be an important source of studies for inclusion; prevent publication bias in intervention studies May need many different strategies to identify
- 13. Grey Literature: some sources Grey Matters: A Practical Search Tool for Evidence-Based Medicine (general) http://libguides.newcastle.edu.au/content.php?pid=259537&sid=22 (excellent set of advice and links on grey literature) Google scholar ProQuest Dissertations & Theses Database
- 14. Other sources and methods Handsearching Fulltext journal searching Table of contents services Reference lists in other reviews Guidelines Trials registers Web searching Contacting study authors, experts, manufacturers, and other organisations Using a project Internet site to canvas for studies The Cochrane handbook has excellent detailed and well set-out advice about these and more. http://www.cochrane.org/handbook/62-sources-search
- 15. Summary and key points Collaborate with the clinical experts at every stage Do research: investigate other systematic review reports, Cochrane guidelines and standards for systematic review searching Analyse: – Your sources – Your search terms – Your search strategy – Your search results Adjust your searching in response to the analysis: test and redo if necessary Be thorough and transparent: document and record your steps as you go Consider using: – Gold standard set – Textual analysis tools
- 16. Further reading Fehrmann P, Thomas J. Comprehensive computer searches and reporting in systematic reviews. Research Synthesis Methods. 2011;2(1):15-32. Hausner E, Waffenschmidt S, Kaiser T, Simon M. Routine development of objectively derived search strategies. Syst Rev. 2012;1(1):19. Lasserre K. Expert searching in health librarianship: a literature review to identify international issues and Australian concerns. Health Info Libr J. 2012 Mar;29(1):3-1 McGowan J, Sampson M. Systematic reviews need systematic searchers. J Med Libr Assoc. 2005 Jan;93(1):74-80. Niederstadt C, Droste S. Reporting and presenting information retrieval processes: the need for optimizing common practice in health technology assessment. International Journal of Technology Assessment in Health Care. 2010;26(04):450-7. Ogilvie D, Hamilton V, Egan M, Petticrew M. Systematic reviews of health effects of social interventions: 1. Finding the evidence: how far should you go? Journal of Epidemiology and Community Health. 2005 September 1, 2005;59(9):804-8. Papaioannou D, Sutton A, Carroll C, Booth A, Wong R. Literature searching for social science systematic reviews: consideration of a range of search techniques. Health Info Libr J. 2010 Jun;27(2):114-22. Sampson M, McGowan J, Cogo E, Grimshaw J, Moher D, Lefebvre C. An evidence-based practice guideline for the peer review of electronic search strategies. J Clin Epidemiol. 2009 Sep;62(9):944-52. Sampson M, Tetzlaff J, Urquhart C. Precision of healthcare systematic review searches in a cross-sectional sample. Research Synthesis Methods. 2011;2(2):119-25. Cochrane Handbook for Systematic Reviews of Interventions Systematic reviews: CRD’s guidance for undertaking reviews in health care http://www.diigo.com/user/sarahhayman/systematicsearching
- 17. Thank you! CareSearch would like to thank the many people who contribute their time and expertise to the project, including members of the National Advisory Group and the Knowledge Network Management Group. CareSearch is funded by the Australian Government Department of Health and Ageing. www.caresearch.com.au
Editor's Notes
- CareSearch is f unded by Australian Government Dept of Health and Ageing and it is based at Flinders University. It has a website and online resources for those providing palliative care and also those needing palliative care and their families, carers and friends It is built on a Knowledge Translation framework, to facilitate access to, and use of, evidence in practice My role at CareSearch involves development of search filters which I will be talking about more later, but essentially, e vidence-based search filters use a detailed and intensive research methodology to enable automated real-time retrieval of relevant literature, and can result in more comprehensive and routine retrieval, for health librarians and for clinicians, academics and researchers. I will assume that you all have a good level of searching skill already. Other presentations today will talk about other aspects of searching for systematic reviews. This session will provide a general overview but will then focus on some strategies that we use at CareSearch to develop search filters that we believe can be applied more generally to expert searching and will help you examine and evaluate your searches. Why would we want to do this? Of course to improve our search results To provide evidence of the quality of our searches. I also want to get you to think about your searches as testable and verifiable – you can apply an experimental approach to the process in order to get an improved outcome. Test, analyse, adjust and redo if necessary.
- Importance of the search In undertaking the searching for a systematic review of an intervention it is of vital importance not to miss anything. Omitting one relevant study can potentially result in a clinical outcome that will affect patients. Systematic review searching requires an explicit and tested search methodology. Systematic reviews aim to identify, evaluate and summarise the findings of all relevant individual studies. The search should be both comprehensive and appropriate to the review question. Screening later in the process will determine whether a study matches the inclusion criteria; the original search should be broader so as not to miss any possibly relevant item and affect the findings of the review. The aim of the search is to generate a bibliography of candidate studies that includes titles and abstracts of all potentially relevant studies. To reduce the risk of bias (e.g. only reviewing the most accessible research) systematic research synthesis requires searches for literature to be both comprehensive and systematic. The search aims to produce a comprehensive and unbiased set of research relevant to the review question. Need while being comprehensive to consider the trade-off between sensitivity (retrieving all relevant items) and specificity (retrieving no irrelevant items). A systematic review search will tend towards sensitivity in order to be as comprehensive as possible. This may be different from other literature searches librarians are familiar with, where users do not wade through irrelevant items therefore want a more specific search. However you may still need to balance sensitivity and specificity so you do not end up with too great a proportion of irrelevant items for the screening stage. You may need to analyse the results you are getting and redo searches. The review question is the heart of the systematic review. It should be clear, focused and closely define the boundaries of the topic. A typical systematic review of effectiveness of an intervention usually contains the 4 PICO elements. (Comparison may be implied but may need to be searched for). Each of these elements provides a concept or concepts that will need to be accounted for by your search terms. However, note that it is often not recommended to search for outcomes as this will restrict the search too much (if outcome is not stated) and you may miss relevant items. To be comprehensive, focus on population and intervention. Discuss with the review team and sample your results. The review team will develop the review protocol which will set out the parameters of the search. Ideally the librarian will work as part of this team; if not, you will need to develop a good working relationship with key reviewers in order to provide the best search to fit the review question. Will come back to this – one of the most important points – good communication with the review team. This is one of key moments for that. Like the idea of a reference interview – need to understand well what the question is and what is being searched for. Example SR question and its PICO elements is shown.
- Want to briefly mention that there are different types of systematic review and the type of review is likely to have an impact on the search. Don’t have time to go into much detail here but the review protocol and discussion with the review team should help inform you as a searcher if particular types of studies are being sought for a particular type of review, e.g. randomised controlled trials. Record-keeping It is vital to keep records of your searches AS YOU GO, right from the start. This is not only important as part of the SR methodology, for transparency and reproducibility, but will also help you to analyse your search strategies and results and understand and improve them Keep a search log to record: which databases were searched and when the list of search terms used and the combinations in which they were applied the search results. Keeping a record of all this information will facilitate the process of making the review methods explicit and transparent in the final report, so judgements can be made about the quality of the review’s findings.
- A comprehensive SR search will involve going beyond the electronic databases and we’ll talk about other sources and methods later. But the electronic search strategy which is the main focus of the presentation is key; it is likely to be the starting off point for retrieving the initial set of citations which can then themselves provide links or references to other material. It is very important that the electronic search strategy be of high quality. Cochrane handbook recommends (requires_ always searching Medline, EMBASE and CENTRAL. PubMed of course searches all of Medline but also has material that is not yet indexed for Medline. Studies have been done showing that a comprehensive search requires searching beyond Medline and that not doing so can affect the outcome of the systematic review. Need to understand the databases that you are using. What do they cover? Do they index any key journals that have been identified for this research question? Systematic reviewers are usually experts in the subject of the review. Work with them to identify major studies as well as the leading journals and conferences. Then you can verify that these are indexed in the selected electronic sources. How do they work? Are search terms automatically exploded or mapped to a thesaurus? When and how to use truncation and wildcard features? Is there a thesaurus? Is adjacency searching possible? Take the time for each database to learn the optimum way of searching it. How are numbers and phrases searched? Be wary of stop words within phrases – always check with and without quotes to see if your results are different. Be careful searching for methodological studies – some databases do not index these well and research has shown that restricting to methodological type may miss relevant material. You may need to analyse what is happening by careful examination of your results and the contents of the fields. We’ll talk more about that later on. Do your own research on the databases. Look up what others have written and read the evaluation and comparison studies that are available in the library literature. Previous systematic reviews on similar questions may have published good details and reports on how they did their searching which will offer some good starting points. There are some excellent websites offering advice for health librarians on searching and monitoring changes in services like PubMed. You can send messages to the mailing lists and you can also contact databases directly with questions. Two links here that I think are good – there will be others I am sure various of you know.
- It is important not to overlook this stage in the search process. Time spent identifying all possible synonyms and related terms for each of your PICO elements or concepts will ensure that your search retrieves as many relevant records as possible. You’ll be well aware of this, as experienced searchers. How to be comprehensive? Some strategies: - Think laterally about how others may describe the same concept - What terminology is used internationally? - Are there spelling differences in UK English and US English words? - Are there any colloquial terms or phrases used ? - Check the search terms used in other papers or systematic reviews - other terms may be suggested from these. It might be useful to check relevant dictionaries, encyclopedias and key texts for alternative terms. Remember to look at the history information in MeSH – remember you may need to consider an earlier form of a heading or know that a particular heading has only been used since a certain date. Consult with experts for advice on terminology and also advice on key references, including journals, which will give you a guide to the terminology used in the area. Consider a term identification set. How to develop: You can ask your expert advisors to suggest a set of references that they would expect to be retrieved. You can develop your own from a set of studies that have been validated by your experts as meeting the inclusion criteria; from some key journals or key authors if they have been identified. The set can be used both as a source of terminology and also as a validation set to show that your search has been as comprehensive as possible. We’ll come to that later. Strict methodology that we would use when building a search filter would require you to split the gold standard set into separate subsets, one of which you use to develop your search terms and the other which you use as a validation set to show that your search has been correct. Even if you don’t have the time and resources to go as far as that, you will still be enhancing your search with terminology that has been objectively derived which will result in a better search. If you become interested in this methodological approach, there is literature available on how search filters are developed and research has been done into different methods, different types of filter and their validity. The CareSearch website has links to several papers. For textual analysis: Store your known relevant references in a ref management system, export abstracts and titles, then feed that text into a textual analysis program such as Writewords (free online go to phrase search) or Concordance – needs to be purchased but gives much more functionality. It might sound complex but can be quite easy to do and it is very illuminating about important language and terms to use. You know that you are using terms that occur in the articles that are key to your topic. Demonstrate an approach.
- This screen shows Concordance which provides other ways of scrutinising the data and different views. As a purchased product it does give you more functionality. In this case we have taken residents as the headword and looked at the occurrences of the terms within the records and the surrounding text. You can see phrases such as long term care residents and residents with dementia occurring.
- What can this approach do for our search? You can actually test the performance of each of the search terms in retrieving the records in the other subset you have created, if you have done that. You can at the very least use the terms you have found through the textual analysis as synonyms in your database searches knowing that they were terms that had a high occurrence in known relevant citations.
- Also known as hedges. Haven’t got time to go into much detail about search filters here but am happy to talk to anyone about CareSearch’s work on filters later on. It is important to touch on them though in this context of systematic searching. The CareSearch project team in association with the Flinders Filters team, has developed several search filters and associated research expertise. Current filters include: palliative care, heart failure, lung cancer and primary health care, with future filters to include Indigenous health, residential aged care facilities and bereavement. Search filter use and development are of considerable current interest to librarians and others in the health knowledge and evidence field. It is worth considering the use of search filters for your search, when appropriate. You should not rely exclusively on a search filter even if there is one available for your topic, because it may have been weighted for specificity when you are wanting a highly sensitive search; but it has been objectively derived and validated and therefore will give reliable results and be an excellent starting point. Cochrane handbook recommends using search filters to identify randomised trials. The handbook states: Existing highly sensitive search strategies (filters) to identify randomized trials should be used Types: Methodological and subject-based Search filters are mostly methodological although there are some subject based ones. CareSearch has developed several which can be found on our website. Methodological filters Method filters target certain published studies which use certain designs, for example, randomised controlled trials or systematic reviews. Subject filters Subject filters are far less common and target a particular subject field. The palliative care filter was developed to help people find relevant palliative care articles in PubMed. Note that there is a difference between the validated search filters developed and tested following established methodology and what are sometimes presented as search filters that may have had less testing. They may still be useful but you should use with caution and you can still apply your own validation to the search results by testing if you have retrieved the items you have identified in your gold standard or sentinel set. Note also that the new PubMed interface now uses search filter terminology for what were previously called limits. They are really presenting a faceted search based on the fields in their metadata; this is not the same as a filter that has been built by testing the behaviour of search terms across all fields and analysing the results. The systematic reviews filter at the side does however use the same search filter as the clinical queries systematic review link – the results are the same. PubMed uses the term filters for the side bar options and points out that the systematic reviews filter was produced using a search strategy . For some good sources of more information, see http://www.york.ac.uk/inst/crd/intertasc/ https://sites.google.com/site/healthliteraturesearchers/tools/search-filter http://pubmedsearches.blogspot.com.au/ http://www.ncbi.nlm.nih.gov/pubmed/clinical
- Depending on how much time you have available, you can apply the same analytical techniques to the results you are getting to test the quality of your search. This is useful for two reasons: Improving your search for this particular review. Providing evidence of the validity of your search for the review and for planning and resource allocation for future reviews. It is a way to demonstrate the skills librarians can bring to the systematic review process and to add strength to the argument that librarians should be involved as key members of systematic review project teams. Make sure you include in your report of your searching how you derived and validated your search strategy. If you have identified a gold standard set of references which your subject experts confirm should be included studies in the review, you can verify that your search strategy is retrieving these studies. It is possible to create a set of references within PubMed or Medline as a saved search and demonstrate that your search will retrieve these items. Best practice, as we mentioned before, is to use a subset of the gold standard set for term identification and a different subset for validating your search, in order to minimise bias. Otherwise you risk creating a search to find a set of items that you have already found and will not be demonstrating that your search is capable of finding other relevant items. If you do not have the time or resources for this, at the very least ensure that you are able to send an early set of references through to your clinical subject experts for validation; you can then use their feedback to determine if your search strategy has been on the right track and improve it if needed. If they can suggest references that you should have retrieved and didn’t, that can be very valuable information. You can analyse the terms that would have retrieved those items. You can consider an evaluation instrument such as PRESS, and use it for peer review or self review Results: can demonstrate that terms used performed better against a known set (sensitivity) And/or that expert advisory group (or individual experts) verified that your results were relevant (specificity) https://sites.google.com/site/healthliteraturesearchers/tools/search-strategy-peer-review-tools
- McGowan J, Sampson M, Lefebvre C. Evidence based checklist for the peer review of electronic search strategies (PRESS EBC). Evidence Based Library and Information Practice 2010; 5(1):149–154. Could be a useful checklist to use – peer review or self review Can suggest ways to improve your search as well as provide a checklist
- Grey literature is a big topic in itself and I haven’t got time to go into much detail about it, but it is important to mention it. The textual analysis approach can certainly be used in searching grey literature databases as well and may be very useful if you are needing to go further afield in databases you are not as familiar with and that don’t use MeSh headings. A systematic review conducted in 2008 by members of the Cochrane methodologies team found that often the results from grey literature significantly effect the outcome of a review. It is therefore very important to treat grey literature as another potential source of studies for inclusion. Different countries will mean different databases for government information Different topic areas require different databases
- Some examples only, far from comprehensive. Look within the top two for further links. Grey Matters is Canadian but does have sections on other countries including Australia http://research.fraserhealth.ca/media/Grey%20Matters%20%20A%20Practical%20Search%20Tool%20for%20EBM%20%20CADTH.pdf Wanted to show the CareSearch one because it is an example of an organisation collecting and publishing links to quality grey literature to save users the time. There will be other topic based examples, also many beyond medicine, like Australian policy online.
- Other sources and methods Not comprehensive Don’t have time to go into these in detail in this session. I suggest working through carefully the suggestions in the Cochrane Handbook for all these different ways and places to search – it is well set out and very thorough. Evidence suggests that hand searching is still necessary for a comprehensive review, although it is time-consuming. It must also be documented like the other searches. Hand-searching is the process of manually screening a defined set of peer-reviewed biomedical journals, conference proceedings and other publications on the web for relevant studies. it: locates relevant articles poorly or inaccurately indexed or unindexed; allows researchers to scan content quickly for relevant studies from the high-impact journals, and ensures that relevant studies are not overlooked backward chaining, footnote chasing, pearl growing, reference harvesting, reference searching, and so on The general points about analysing your search strategy, testing and checking and re-doing the search to capture more widely, or more effectively, will also apply to searching these sources.
- In general, the aim is to examine and analyse your results and use what you learn from that to inform the next iteration of your search. This can make documentation cumbersome but it is the best way to ensure a comprehensive search. Reiterate that what I have given are suggestions of elements you could apply to your searching to assist with thinking “what might I have missed” and “how can I aim to minimise errors and bias to produce a comprehensive search?” Remember with a systematic review search we are looking for comprehensiveness. We understand everyone has time constraints and resourcing constraints and some methodologies I have mentioned can be very time-consuming; but the benefit of applying even a minor aspect - such as sending your first set of results to a subject/clinical expert for feedback - can be enormous. Treat your search like an experiment – testable and verifiable – and document all the way.
- Selection only of some documents that might be useful. Look for more articles by these same authors and follow their references. I have also put a link to where I have bookmarked some of the freely available online information that’s relevant – and will aim to add new items to it. It is possible of course on diigo to create a group where others can add and share items.