2011 Mining Unique Information Sources & Deep-Invisible-Hidden-Opaque Web

August 2, 2011 recap of
Anna F. Shallenberger SLA Annual Conference Session 6.14.2011
President & “Chief Archer”
Shallenberger Intelligence Services

anna@targetedknowledge.com
203.258.2383 Thank you generous sponsor, for enabling
917.591.6732 fax us to offer this free of charge recap
www.targetedknowledge.com
http://twitter.com/ClosetLibrarian
http://www.slideshare.net/ClosetLibrarian

http://closetlibrarian.blogspot.com
www.linkedin.com/in/annafayshallenberger
www.ci2020.com/profile/AnnaFShallenberger
http://tinyurl.com/AIIP-AFS
Anna F Shallenberger, All Rights Reserved, for educational use only, not for redistribution or commercial re-use

 Pearl Grow …

 ID Leakage Points [factoring in copyright & other IP concerns]

 Non Central Hosts of Content
 E. G. Content not controlled by “HQ”

 Surgical Manual Browsing

 Dark Web Browsers [also use pathfinders]

Anna F. Shallenberger
President & “Chief Archer”  Leverage Your {on & offline} Networks ….

203.258.2383
917.591.6732 fax

Anna F Shallenberger, All Rights Reserved, for educational use only, not for redistribution or commercial re-use 2

Conduct
Search

Evaluate What
You Find…

Mine the most on
point for more
ideas …

Anna F. Shallenberger Refine your search strategy,
President & “Chief Archer” Continue your investigation,
Shallenberger Intelligence Services Repeat process as needed…
203.258.2383
917.591.6732 fax


Look For…
 Concepts/Terms/Catch Phrases etc – the more unique, the better…
 Names – Experts/Reports/Publications

 URL Roots , e.g. are the most relevant loaded on the same part of site

 Revisit strategy adding incremental terms and/or
re-weighting / editing Boolean linkages

 Don’t forget the reverse
 What are key terms repeating in the “false drops”

203.258.2383
917.591.6732 fax



203.258.2383
917.591.6732 fax

Example of expansion search strategy,
Results may/may not have been “hidden”

Example of expansion
search strategy,
Results may/may not
have been “hidden”


203.258.2383
917.591.6732 fax



anna@targetedknowledge.com www.pearltrees.com
203.258.2383
917.591.6732 fax


[factoring in copyright & other IP concerns]

 Analogy: Similar to emotional conversations,
where “speaker” may or may not
 Intend for public [or so much of them] to “hear” [have access to] it
 Fully comprehend others’ valuation of the information

 Understand originators perspective – “But I only told that to…
And they promised not to tell anyone…”

 Information Leakage Points , lag time similar to embargos…
President & “Chief Archer”  Reuse by others – clients, ex employees

 Conference Presentations
203.258.2383
917.591.6732 fax  Case Studies or other Sales & Marketing Collateral
 Continuing Education [especially MBA classes]
http://closetlibrarian.blogspot.com  Social Media

[factoring in copyright & other IP concerns]


203.258.2383 Example of market research vendor with many English language sites.
917.591.6732 fax
Found reports covering US on non-US sites , not available on US sites.
Spottier now, company may have better process oversight

Illustrates content recycling…
Authoritative sources…
Pearl grow them….
Example dated though…


203.258.2383
917.591.6732 fax


Don’t be surprised if you receive conflicting advice from experts …

Consult with your legal dept – rulings/interpretations evolve . May vary geographically….

 SLA 2011 www.slideshare.net/cavlec/i-own-copyright-so-i-pwn-you

 Nice overview from UMN https://www.lib.umn.edu/pdf/CanIUseThat-Handout-03_2011.pdf

 Public Domain Sherpa www.publicdomainsherpa.com/10-misconceptions-about-the-public-domain.html

 Fair Use Podcast www.sla.org/podcasts/Replays/2010/2010Feb_AsktheCopyrightExperts.wmv

 Ask CR Experts www.sla.org/content/learn/members/webinars/webreplays/podcast2011/2011replay03.cfm

 Copyright Clarity www.slideshare.net/reneehobbs/copyright-clarity-using-copyrighted-materials-for-digital-learning

Brooklyn PL’s great link list www.brooklynpubliclibrary.org/pdf/copyright_faq.pdf
 ALA/ARL www.arl.org/bm~doc/mm10fall-butler-jaszi.pdf
 Center For Study of Public Domain www.law.duke.edu/cspd/index
203.258.2383
917.591.6732 fax


Non Central Content Hosts, e. g. content not “HQ” controlled

 Branch offices of consultants, research firms, usually non-US

 Biz units migrating tech platforms [generally post-merger]

 Satellite campuses, larger academic institutions

 Event-driven sites – conferences, product introductions, etc…
President & “Chief Archer”  Non-merger partnerships / joint initiatives

203.258.2383  “Relationships” NEC–specialized social networks , non-profits..
917.591.6732 fax
[sometimes exec bios NOT pasted verbatim from corp site]


Dark Web / Specialized Browsers, etc


203.258.2383
917.591.6732 fax


Surgical Browsing
 Identify Potential High-Value Sites
 Navigate Manually
 Create Your Own Site Index Using a Browser

It can include
 Downloading {smaller} sites into Adobe to browse offline
 Looking for cross-linking to site, especially several layers in
 Locating historical content in caches or archiving sites
Be Careful
203.258.2383  While limiting searches by doc type [pdf etc] is effective
917.591.6732 fax
www.targetedknowledge.com Searchable layers can mask them behind other file types


Leveraging Who You Know, Digitally & Offline

 What do people in that field read, on & offline ?
What would they consider a waste of time?

A large part of the challenge is indexing…
 And you need to ID what they “miss”

 Sometimes there’s no GPS, must already know
where you’re going, or at least a mid-point…

203.258.2383
917.591.6732 fax


Needles in Haystacks aren’t invisible,
but they can be more work to locate
Some even hide in plain sight

Have a plan, but flex it as needed
Take good notes, bookmark good leads, save best hits
Might not find them again or they change

ALWAYS Consider the Source

Manage time spent, don’t get lost
Anna F. Shallenberger Be Flexible, but still…
Plan Ahead
203.258.2383
917.591.6732 fax


Planning Ahead

*AFS example based on model designed by KnowledgeInforrm

Based On Questions You
Are Seeking To Answer
ID Potential Sources, &
“Pearl Grow” From There


203.258.2383
917.591.6732 fax


ID/Target Sources
{categorization subjective – many fit multiple}

Influencers
Consultants / Think Tanks
Pollsters / Market Researchers
Academia
Governmental
NGOs & Advocacy Groups
Trade/Professional Associations
Other Niche Organizations
Businesses & Publishers, NEC
Aggregators/Re-packagers/Peer-Sharing , NEC
PEOPLE NEC

203.258.2383
917.591.6732 fax


 Deep Web: Surfacing Hidden Value
www.brightplanet.com/images/uploads/DeepWebWhitePaper_20091015.pdf

 Marcus Zillman: Deep Web Research
www.llrx.com/features/deepweb2011.htm or www.deepwebresearch.info

 Chris Sherman @ Information Online
www.docstoc.com/Docs/Document-Detail-14.aspx?doc_id=84592274

 August Jackson [for SCIP] Getting Most ….
http://homepage.mac.com/cornfed/internetdeepweb.pdf

 Using Web Investigative Reporting Tool www.slideshare.net/tccj/web-as-investigative-tool or
http://campuscoverage.org/sites/default/files/Docs/Presentations/CCPInternet.ppt
Model & Analyze Deep Web
www.scribd.com/doc/59496007/Modeling-and-Analyze-the-Deep-Web-Surfacing-Hidden-Value
President & “Chief Archer”  Accurate & Efficient Crawling Deep Web
Shallenberger Intelligence Services www.scribd.com/doc/57147960/Accurate-And-Efficient-Crawling-The-Deep-Web-Surfacing-Hidden-Value

203.258.2383  Web & Twitter Archiving @ Library of Congress
917.591.6732 fax www.slideshare.net/nullhandle/web-and-twitter-archiving-at-the-library-of-congress
http://twitter.com/ClosetLibrarian CRS report to Congress www.docstoc.com/docs/84024621/CRS-Report-for-Congress

http://closetlibrarian.blogspot.com  How Much Information [UC Berkeley]
www.linkedin.com/in/annafayshallenberger http://www2.sims.berkeley.edu/research/projects/how-much-info-2003/printable_report.pdf

 Definitions vary as to what it is / is not

 Many names – deep, invisible, hidden, opaque etc
Surface web is “visible” portion

 Baseline Research, particularly re size, dated
Term coined by Michael Bergman…


203.258.2383
917.591.6732 fax


 Pages lower ranked due to Search Engine Optimization [SEO]

 Sites coded to exclude bots

 Dynamic content generated by page search – stats, etc

 Search engine chooses not cover whole site due volume of context

 Format – video/image w/o text/tags, or they’re incomplete

 Site/pages not connected with pages browser(s)
President & “Chief Archer”  Password protected pages

203.258.2383
917.591.6732 fax


Figure 1. Search Engines: Dragging a Net Across the Web's Surface


203.258.2383
917.591.6732 fax



203.258.2383
917.591.6732 fax
http://www.slideshare.net/ClosetLibrarian Q4 NPD Search &Portal Site Study, reported by Search Engine Watch


203.258.2383
917.591.6732 fax


Information Not elsewhere classified

Customers


203.258.2383
917.591.6732 fax


Thank you generous sponsor, for enabling
us to offer this free of charge recap

Thanks & Best Regards,

203.258.2383 cell
917.591.6732 fax

An experienced researcher, educator, author, blogger, strategist & consultant,
Anna Shallenberger, aka the ClosetLibrarian, was recently recognized in Best of
the Business Web & featured on SlideShare’s home page.
President & “Chief Archer” At SLA 2011 , Anna was a panelist for “Integrating with Sales & Marketing to
Capture & Deliver Intelligence” & led an "Intelligence Café“ discussion regarding
203.258.2383 Unique Information Sources & the Deep Web. Anna was also a spotlight panelist
917.591.6732 fax @ SLA 2010 & served as conference planner for the CI Division.
No member of a crew is praised for the rugged individuality of his rowing.
~Ralph Waldo Emerson


Client
Special
Projects

Actionable Deliverables:
 Analysis
5/9 Forces
SWOT/PEST
Landscapes- Cos, Mkts, Prods
Win/Loss, Cust Sat [B2B, B2C, B2B2C]
multiple other approaches
 Human Source Interviews
 Due Diligence / Fact-Checking
 Benchmarking / Best Practices
 Counter Intelligence

Process/Basics:
Early Warning Systems, Communities of Practice [internal & external], Content Collection –
Anna F. Shallenberger Aggregation – Synthesis, Document Retrieval, Human Source Identification / Qualification /
President & “Chief Archer” Prioritization / Bios, Published Source ID & Contract Negotiation, Taxonomy Development,
Shallenberger Intelligence Services Digitization, Patent Searching, Ethics/Standards Advice, Templates/Forms

203.258.2383
Foundation:
917.591.6732 fax Secondary Research, Primary Research, Knowledge Management, HUMINT, 6Sigma/Organizational
www.targetedknowledge.com Effectiveness, Project Management, Moderation, Coaching / Advisory Services, Training / Instruction

www.ci2020.com/profile/AnnaFShallenberger Intel Types include: competitive, market, sales, defensive, trade show, etc

2011 Mining Unique Information Sources & Deep-Invisible-Hidden-Opaque Web

Recomendados

Recomendados

Mais conteúdo relacionado

Mais de Anna Shallenberger

Mais de Anna Shallenberger (20)

Último

Último (20)

2011 Mining Unique Information Sources & Deep-Invisible-Hidden-Opaque Web