The Information Discovery, Verification & Knowledge Navigation Guide

1.1 What Information Is

Data is raw, uninterpreted signal: a number, a timestamp, a pixel value, a sensor reading. It has no meaning on its own.

Information is data given context and structure — data that answers a question ("the temperature was 21°C at 14:02 on the roof sensor"). Information is what most search and retrieval systems traffic in.

Knowledge is information that has been integrated into a model of the world — connected to other information, tested, and made actionable ("this sensor consistently reads 2°C high, so 21°C actually means ~19°C, and that matters because the greenhouse threshold is 20°C"). Knowledge implies understanding of relationships, causes, and limits.

Wisdom (a fourth, often-omitted layer) is knowing when and how to apply knowledge — judgment under uncertainty, including knowing what you don't know.

        ▲
        │      WISDOM      "when/whether to act on it, and what could be wrong"
        │  ────────────
        │    KNOWLEDGE     "how it connects, why it's true, what it predicts"
        │  ────────────
        │    INFORMATION   "data + context: what, who, when, where"
        │  ────────────
        │       DATA       raw signal, no interpretation
        └───────────────────────────────────────────────►
                 increasing interpretation & structure

This matters practically: a search engine gives you information. Your job as a researcher is to turn it into knowledge — which requires provenance, corroboration, and context that the raw hit rarely carries with it.

1.2 Primary, Secondary, and Tertiary Sources

Practical rule: tertiary sources are a map, not a destination. Use Wikipedia, an encyclopedia entry, or an AI summary to locate the primary/secondary sources in its citation list, then verify the claim against those.

A single artifact can be primary for one question and secondary for another — a newspaper article is a primary source about media coverage of an event but a secondary source about the event itself.

1.3 Authority and Credibility

Authority and credibility are not the same thing and must be evaluated separately:

• Authority = institutional standing, credentials, position (a government agency, a peer-reviewed journal, a named expert).
• Credibility = track record of being correct, transparent about method, and honest about uncertainty — independent of title.

A source can have high authority and low credibility (an official body shown to misreport), or low authority and high credibility (an anonymous but rigorously-sourced dataset that has been independently reproduced). Evaluate both axes, never substitute one for the other.

CRAAP-style evaluation (a widely used checklist, extended here):

1.4 Scientific Evidence Hierarchy

Not all evidence carries equal weight, even within legitimate science:

STRONGEST  ┌──────────────────────────────────────────────┐
           │ Systematic reviews & meta-analyses            │
           │ (of high-quality studies)                     │
           ├──────────────────────────────────────────────┤
           │ Randomized controlled trials (RCTs)           │
           ├──────────────────────────────────────────────┤
           │ Cohort studies (prospective)                  │
           ├──────────────────────────────────────────────┤
           │ Case-control studies                          │
           ├──────────────────────────────────────────────┤
           │ Cross-sectional studies                       │
           ├──────────────────────────────────────────────┤
           │ Case series / case reports                    │
           ├──────────────────────────────────────────────┤
           │ Expert opinion / editorials                   │
           ├──────────────────────────────────────────────┤
           │ Anecdote, testimonial                         │
WEAKEST    └──────────────────────────────────────────────┘

This hierarchy is domain-specific (clinical/epidemiological research) but the underlying logic generalizes: designs that control for confounding and are replicated independently outrank designs that don't, regardless of field. A single preprint, however dramatic, sits far below a replicated, peer-reviewed, meta-analyzed result — and pre-registration and open data/code push a study up in practical trustworthiness independent of where it sits on this ladder.

Also weigh: sample size, effect size, replication status, pre-registration, conflicts of interest, and whether the result has been independently reproduced. A "significant" result in one small, non-replicated study is evidence, not proof.

1.5 Reliability vs. Popularity

Search rankings, social shares, view counts, and "trending" status measure engagement, not accuracy. These frequently diverge:

• Content optimized for engagement (novelty, outrage, confirmation of existing belief) outcompetes accurate-but-boring content in ranking/virality systems.
• A highly-cited paper is not automatically correct — citation can mean "built upon," "refuted by," or even "cited as an example of bad practice."
• Domain age, backlink count, and follower count are reputation proxies, not truth proxies.

Treat "top result" and "most shared" as candidates to investigate, never as conclusions.

1.6 Truth, Uncertainty, and Confidence Levels

Rarely does research yield binary true/false. Adopt an explicit confidence vocabulary and use it in your notes:

Write the confidence level next to every claim in a research dossier. An unlabeled claim is treated by anyone reading your work as more certain than you intended.

1.7 Cognitive Biases That Distort Research

1.8 Misinformation, Disinformation, Propaganda & Information Warfare

1.9 Algorithmic Influence

Feed ranking, recommendation, and personalization systems are themselves an information-shaping force, independent of any single piece of content:

• Filter bubbles / echo chambers — personalization narrows the range of what you see over time based on past engagement.
• Engagement optimization — systems trained to maximize watch-time/clicks systematically favor emotionally activating content, which correlates with — but is not identical to — misinformation.
• Search-as-a-battleground (SEO / "answer engine optimization") — content is increasingly produced to rank, not to inform, including at scale via generative AI.
• Zero-click answers — when a search engine or assistant answers directly, the user often never evaluates the underlying source. Deliberately click through when the claim matters.

2.1 The Core Pipeline

┌───────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐     ┌───────────┐
│  CRAWL    │ ──► │  PARSE /  │ ──► │   INDEX   │ ──► │   RANK    │ ──► │  SERVE     │
│ (spiders/ │     │  EXTRACT  │     │ (inverted │     │ (query-   │     │ (results,  │
│  bots)    │     │  content  │     │  index +  │     │  time     │     │  snippets, │
│           │     │  & links) │     │  vectors) │     │  scoring) │     │  answers)  │
└───────────┘     └───────────┘     └───────────┘     └───────────┘     └───────────┘

Crawling (spiders / bots)

Automated programs start from seed URLs and known sitemaps, fetch each page, extract outbound links, and queue those for future crawling. Crawl behavior is governed (voluntarily) by robots.txt and <meta name="robots"> directives, crawl-rate limits, and sitemap.xml files. Crawlers prioritize by estimated importance and change frequency.

Parsing / extraction

Fetched HTML (and increasingly rendered JavaScript, PDFs, OCR'd images, video transcripts) is parsed into visible text, title/meta tags, structured data (schema.org/JSON-LD), the link graph, and — for image/video — extracted metadata and learned embeddings.

Indexing

Text becomes an inverted index: term → documents containing it, with position/frequency/field data. Modern systems layer a vector index alongside this — every document and query becomes a dense numeric embedding, enabling semantic matching without exact keyword overlap.

Ranking

Candidate documents are scored on hundreds of signals:

• Relevance — term match, proximity, field weighting, embedding similarity.
• Authority — link-based measures, domain trust/history, E-E-A-T signals for health/finance/legal queries.
• Quality/spam — thin/duplicate content detection, ad density, manipulated-link detection.
• User signals — click-through rate, dwell time, reformulation rate.
• Freshness — recency weighted heavily for time-sensitive queries.
• Personalization — location, language, device, history.

Link analysis and PageRank

PageRank (Google's original algorithm, now one signal among many) models the web as a graph and computes each page's importance from how many pages link to it and the importance of those linking pages, recursively. A link from a high-authority page counts far more than one from an obscure page — resistant, though not immune, to simple link-count spam.

Semantic search, vector search, and embeddings

Keyword search fails when a query and a document use different words for the same concept ("heart attack" vs. "myocardial infarction"). Embeddings map text to a vector such that similar meaning lands nearby in vector space; vector search finds nearest neighbors there; modern engines blend lexical and vector scores as "hybrid search," since each catches failure modes the other misses.

AI-powered search / generative answers

Newer search UIs retrieve candidate documents, then feed them to a language model to synthesize a direct answer with inline citations — a form of Retrieval-Augmented Generation (see Part 10). New failure modes: synthesis errors, citation mismatch, and disagreement between sources getting masked by a single confident answer.

Search quality evaluation

Engines are tuned using human relevance raters against published guidelines, live A/B experiments, and offline metrics like nDCG and precision/recall at k. You can informally replicate this: run the same query across engines and score each top-10 for relevance, authority, and source diversity — the fastest way to learn an engine's blind spots.

2.2 Comparing Major Search Engines

Practical rule: never rely on a single engine. Run the same query across at least one of {Google, Bing} and one independent index {Mojeek, Brave, Yandex if regionally relevant}, plus a metasearch pass. Divergence between engines is itself a signal — censorship, SEO manipulation, regional blocking, or an indexing gap.

2.3 Search Operators and Advanced Syntax

These vary by engine — test each on the engine you're using, support changes over time:

Combine operators for precision: site:.gov filetype:pdf intitle:"environmental impact" 2023 — a core OSINT technique sometimes called "Google dorking" when applied systematically. Legitimate use: finding public documents that are indexed but not linked from a site's navigation. Never for accessing anything not intended to be public — see Part 12.

2.4 Reverse Image and Video Search

Reverse image search finds where else an image (or a visually similar one) appears online, revealing its earliest known appearance, original context, and manipulated reuses.

1. Extract the image — get the highest-resolution original you can, not a screenshot of a screenshot.
2. Search it across multiple engines: Google Images, Bing Visual Search, Yandex Images (strong at faces/landscapes), TinEye (exact/near-duplicate match with earliest-found date).
3. Compare all appearances by date to find the earliest, and read the original caption/context.
4. Check manipulation cues: cloned regions, inconsistent shadows/reflections, compression-artifact mismatches, inconsistent metadata.

Reverse video search is harder — indexes are shallower:

• Extract representative key frames and reverse-image-search those individually.
• Search the exact caption text and any burned-in text/timestamps.
• Check the platform's own "similar videos" / re-upload detection.
• Use frame-level or hash-based provenance tools where you have legitimate access.

2.5 Metadata Analysis

• Images — EXIF can include camera model, timestamp, GPS, software used, but is frequently stripped on upload; absence isn't itself suspicious.
• Documents — author name, organization, creation/edit software and dates, revision history, or leftover hidden text — a classic technique for finding an author's real name behind an "anonymized" PDF.
• Web pages — HTTP headers, source comments, and structured data can reveal CMS, hosting, or publication workflow.
• Caveat — metadata is easy to spoof or strip; treat it as corroborating, never as sole proof.

3.1 Surface, Deep, and Dark Web

 ┌───────────────────────────────────────────────────────────┐
 │  SURFACE WEB  — publicly indexed by general search engines │
 │  news sites, public wikis, public social posts, blogs      │
 └───────────────────────────────────────────────────────────┘
                              │
 ┌───────────────────────────────────────────────────────────┐
 │  DEEP WEB — online, but not indexed by general crawlers    │
 │  paywalled journals, logged-in social content, private     │
 │  cloud docs, most databases behind a search form, most     │
 │  corporate intranets, academic subscription content,       │
 │  medical records, banking portals                          │
 └───────────────────────────────────────────────────────────┘
                              │
 ┌───────────────────────────────────────────────────────────┐
 │  DARK WEB — requires special software/protocols (e.g. Tor  │
 │  hidden services); a small subset of the deep web,         │
 │  intentionally not discoverable via normal DNS/browsing    │
 └───────────────────────────────────────────────────────────┘

Two clarifications researchers routinely get wrong:

1. "Deep web" ≠ "dark web." The deep web is overwhelmingly mundane — simply not crawled, usually because it sits behind a login, a paywall, or a search form a crawler can't fill in. The dark web is a small, specific subset requiring specific software (most commonly Tor) to reach .onion addresses that don't resolve via ordinary DNS.
2. Size estimates from the 2000s ("500× larger") are folklore at this point — the web's structure has changed too much (APIs, SPAs, mobile apps with no crawlable equivalent) for that number to mean anything precise today.

3.2 Public vs. Private Systems

What is publicly searchable: content a site owner has exposed without authentication, not blocked via robots.txt/noindex, and reached by a crawler.

What is not indexed: content blocked by robots.txt, marked noindex, session-generated, behind a search form with no linkable URL, or simply never discovered.

What is inaccessible without authorization: anything behind a login, paywall, VPN, or network boundary requiring credentials you have not been legitimately granted.

4.1 Timeline of Internet Indexing & Discovery Tools

4.2 Why This History Matters to a Researcher

• Directories vs. crawlers reflect two discovery philosophies — curated-by-humans vs. discovered-by-machine — a tension that recurs today in AI-curated vs. crawler-ranked results.
• Pre-Google relevance relied on keyword frequency/density, which is why early-2000s SEO was keyword stuffing — a useful dating signal for outdated pages.
• Gopher still technically exists and is occasionally revived by hobbyists; dedicated clients/gateways exist to browse a gopher:// reference.

4.3 How Historical Internet Research Is Performed

1. Establish the object of study — a specific page, a site's evolution, a defunct service, a claim about "what the internet looked like."
2. Triangulate across archive systems (Part 5) — no single archive has full coverage.
3. Use secondary historical sources — contemporaneous news, Usenet archives, digitized old magazines, academic papers on internet history.
4. Corroborate with technical artifacts — WHOIS history, DNS history, SSL certificate-transparency logs, archived robots.txt/sitemap.xml.
5. Respect the limits of memory-by-proxy — personal recollection is primary evidence about perception, not proof of fact.

5.1 Major Web Archives

5.2 How Wayback Machine Snapshots Work

• A crawl or manual "Save Page Now" request fetches the page and stores it as a snapshot, keyed by exact URL and timestamp.
• The calendar view shows all captures over time — dot density means crawl frequency, not change frequency.
• Captures may have broken sub-resources even when the text content was faithfully captured.
• Only what's linked (or submitted) gets crawled — deep, unlinked, or login-gated pages typically have no snapshot.

5.3 Archive Limitations

• robots.txt impact — historically a site could retroactively hide already-captured history; policy has shifted since — check current behavior.
• Missing content — JS-heavy SPAs and anything behind authentication are frequently incompletely captured.
• Snapshot gaps — absence of a snapshot is not evidence the page didn't exist.
• Redirect/takedown artifacts — some captures reflect a legal request, not the page's actual historical content.

5.4 Recovering Historical Pages — Practical Method

1. Try the exact URL in the Wayback calendar; if empty, try variants (http/https, trailing slash, www).
2. Try Archive.today's own search — independent capture history.
3. If the page moved, search the site broadly in Wayback for the content under a new path.
4. Check for lingering cached-text remnants in search snippets.
5. Search for mirrors and reposts — forum quotes, RSS aggregators, email newsletters.
6. For images/PDFs, check whether the file itself was separately crawled.

5.5 Tracking a Website's Evolution Over Time

• Pull the full snapshot calendar and sample at meaningful intervals rather than reading every capture.
• Diff structure (navigation, headings) separately from content (body text) — structural change often marks a strategic shift.
• Cross-reference against historical domain ownership to see whether content shifts correlate with ownership changes.
• Historical ownership — WHOIS has grown less transparent (privacy-proxy registration, GDPR redaction); corroborate with certificate-transparency logs, DNS-history aggregators, and archived WHOIS snapshots.

6.1 General Science / Cross-Disciplinary

Coverage

Broadest cross-disciplinary index of scholarly literature, including preprints, theses, patents, grey literature.

Strengths

Very high recall, free, "Cited by" counts, clusters preprint + published versions, full-text search inside PDFs.

Weaknesses

No quality filter — indexes predatory journals alongside rigorous work; citation counts can be inflated; no bulk API.

Search strategy

Quotes for exact phrases, the author: operator, year-range filter; sort by date or relevance.

Citation tracking

"Cited by N" opens the full citing-literature list; "Related articles" for near-neighbors.

Verification

Click through to the actual publisher/repository page; check the venue independently for legitimacy.

Coverage

AI-enhanced index, originally CS/biomedical-focused, now broader.

Strengths

Distinguishes citation context (background vs. building-block vs. dispute), TLDR summaries, good API.

Weaknesses

AI summaries can misrepresent nuance — verify against the full text.

Verification

Use the citation-context feature to see how a paper was cited, not just that it was.

Coverage

Fully open scholarly graph — works, authors, institutions, venues, concepts.

Strengths

Free open API and full data dumps, transparent methodology, good institutional/funding metadata.

Weaknesses

Entity disambiguation is imperfect, as with any large automated graph.

Verification

Cross-check author/institution disambiguation manually for high-stakes bibliometric claims.

Coverage

Scholarly literature and global patent data in one graph.

Strengths

Strong for "who patented what, citing which papers" workflows; institutional/funder analytics.

Weaknesses

Less commonly used than Scholar/OpenAlex; smaller community corrections.

6.2 Medicine and Life Sciences

Coverage

The authoritative biomedical literature index (MEDLINE); PMC holds full-text open-access articles.

Strengths

Rigorous MeSH controlled-vocabulary tagging (see Part 9), authoritative, free.

Weaknesses

Abstract-only for many entries; MeSH search has a learning curve.

Search strategy

Use MeSH terms ("Myocardial Infarction"[MeSH]) for recall; combine with field tags.

Citation tracking

NCBI "Similar articles" and external tools — limited native citation browsing.

Verification

Check whether a claim has been picked up in a systematic review before treating a single study as settled.

Coverage

Europe's biomedical aggregator, overlapping PubMed with broader preprint coverage (bioRxiv/medRxiv integrated).

Strengths

Strong full-text search, preprint-to-published-version linking, text-mined annotations.

Verification

Check whether a preprint has since been peer-reviewed, and whether conclusions changed.

6.3 Physics

Coverage

The dominant preprint server for physics (also math, CS, q-bio, econ, stats).

Strengths

Immediate free access; full version history preserved (v1, v2…).

Weaknesses

Not peer-reviewed at posting — treat as provisional until reviewed or replicated.

Verification

Check version history for corrections, whether a later version was published, and citation trackers for critical responses.

Coverage

Specialized literature/citation database for high-energy physics.

Strengths

Deep physics-specific citation network and author-disambiguation; tracks conference proceedings.

Verification

Citation-graph view shows whether a result was built upon or contradicted by its own sub-community.

6.4 Mathematics

Same platform, math.* categories — the provisional-until-reviewed caveat applies strongly, since subtle proof errors can take years to surface.

Coverage

Comprehensive reviewing/abstracting database for mathematics.

Strengths

Expert-written reviews — a critical comment can flag a flawed proof long before retraction.

Weaknesses

Review coverage lags new publications by months.

Coverage

The AMS's parallel review/abstract database.

Weaknesses

Subscription-based; institutional access typically required.

Verification (both)

Read the review, not just the abstract — a terse or hedging review is itself a signal.

6.5 Computer Science

The dominant preprint venue for ML/AI — much of the most-cited recent AI research appears here first, often never in a traditional journal (conferences are the CS norm instead).

Coverage

Near-exhaustive bibliography of CS publications, journals + conference proceedings.

Strengths

Extremely reliable metadata; authoritative venue lists for spotting a pay-to-present mill.

Weaknesses

No abstracts/full text — link out to arXiv/publisher.

Verification

Cross-check a claimed venue name against DBLP's venue list to detect fabricated conferences.

6.6 Engineering

Coverage

Major index/full-text repository for EE, electronics, computer engineering.

Strengths

Authoritative for standards-adjacent and applied-engineering literature.

Weaknesses

Subscription/paywall for full text; institutional access usually required.

6.7 Life Sciences Preprints

Coverage

Dominant preprint servers for biology and clinical/health research (Cold Spring Harbor Laboratory).

Strengths

Fast dissemination — notably valuable during infectious-disease outbreaks.

Weaknesses

medRxiv is explicitly not peer-reviewed — never use it to guide clinical practice directly.

Verification

Check for a published, peer-reviewed version before treating a health claim as actionable.

6.8 Chemistry

The chemistry preprint server (ACS-partnered). Same provisional-status caveats as other preprints — check for the peer-reviewed version.

6.9 Earth Sciences

Preprint server for earth/planetary/environmental sciences — community-run, transparent moderation, useful for fast-moving events. Same provisional-preprint caveat.

6.10 Social Sciences

Coverage

Huge preprint/working-paper repository across economics, law, management, social sciences.

Strengths

Economics and legal scholarship circulate here well before journal publication.

Weaknesses

No peer review at posting; heterogeneous quality.

Verification

Check author affiliation and whether it's part of a recognized series (e.g. NBER); search for the eventual publication.

6.11 Humanities

Coverage

Massive digitized archive of humanities/social-science journals, books, primary sources.

Strengths

Excellent historical journal runs; strong full-text search.

Weaknesses

Paywalled; a "moving wall" delays the most recent issues entirely.

Coverage

Full-text humanities/social-science journals and books, university-press focused.

Strengths

Strong current-issue coverage, high curation quality.

Weaknesses

Paywalled; smaller than JSTOR.

6.12 Spotting Predatory or Low-Quality Venues

Checklist for any journal/conference you haven't heard of:

• Is it indexed in DBLP / MEDLINE / a recognized field-specific index — not just claimed on its own website?
• Does it have a real, checkable editorial board?
• Is peer-review turnaround implausibly fast for the claimed rigor?
• Does the publisher appear on current, actively-maintained "predatory publisher" watch-lists?
• Does it solicit submissions via unsolicited mass email with flattering, generic language?
• Is the claimed "impact factor" independently verifiable, or self-reported?

6.13 General Citation-Tracking Method

1. Forward citation search — who cited this paper since (Scholar, Semantic Scholar, Web of Science, Scopus).
2. Backward citation search — what it's built on; check whether claims trace to primary data or to "citation laundering."
3. Look for a systematic review or meta-analysis already aggregating the sub-field.
4. Check retraction status (Retraction Watch and the journal's own site).
5. Check for an independent replication.

7.1 Government Open-Data Portals

Treat these as examples of a category — nearly every national government and many major cities run some form of open-data portal.

7.2 Census & Demographic Data

National statistical agencies publish population, economic, and household data with methodology documentation — always read the definitions page, since terms like "unemployment" or "poverty" are defined differently across agencies and time periods.

7.3 Geographic Data & Satellite Imagery

• OpenStreetMap — crowd-sourced map data with an auditable edit history.
• National mapping agencies (USGS, Ordnance Survey, etc.) — authoritative topographic data.
• Satellite imagery (Sentinel/Copernicus, Landsat) supports change-detection research; OSINT geolocation combines imagery with landmarks, shadow-angle analysis, signage, and terrain cues.

7.4 Economic Databases

World Bank Open Data, IMF, OECD.Stat, national central banks and statistical agencies. Always check the base-year/methodology vintage — rebasing changes historical series retroactively.

7.5 Legislative Databases

National legislature sites (congress.gov, UK Parliament's Hansard, the EU's EUR-Lex) — primary sources for what a law actually says, versus news paraphrase.

7.6 Court Records

Public court-record systems (PACER for U.S. federal courts, national equivalents) — treat docket entries as primary but remember filings are allegations, not established fact, until adjudicated.

7.7 Corporate Registries

National/regional company registries (UK Companies House, EU business registers, U.S. Secretary-of-State systems) — foundational for ownership research, though beneficial-ownership transparency varies enormously by jurisdiction.

7.8 Patent & Trademark Databases

Patents — national offices (USPTO, EPO), WIPO's PATENTSCOPE, Google Patents, Lens.org. Trademarks — national databases (USPTO TESS/TSDR), WIPO's Global Brand Database. Useful for corroborating a company's real activity independent of its own marketing.

8.1 Cross-Source Verification & Triangulation

Triangulation: establishing a fact via at least two (ideally three) independent sources with different methods, incentives, or vantage points. Independence matters more than count — three outlets re-reporting one press release are one source, not three.

             Source A (primary data)
                    \
                     \___ Do they converge
             Source B (independent          on the same fact?
             methodology) __/               │
                    /                       YES → well-supported
             Source C (different            NO  → contested;
             vantage point/incentive)              investigate the
                                                    discrepancy itself

8.2 Citation Tracing

Follow a claim backward through its citation chain until reaching the primary data, or the point where the trail breaks — an assertion with no source, or a citation that on inspection doesn't say what's claimed. Actually opening the cited source catches a large share of circulating misinformation.

8.3 Network Analysis

Map relationships (who links to/cites/funds/employs/co-authors with whom) as a graph to reveal structure invisible from any single node — coordinated networks, funding clusters, or the small number of hub sources a whole narrative traces back to. Manual diagrams for small cases; graph software (e.g. Gephi) for larger ones.

8.4 Timeline & Event Reconstruction

1. Collect every dated artifact touching the event.
2. Normalize all timestamps to one timezone/format.
3. Order them; look explicitly for gaps (flag, don't fill with assumption) and contradictions.
4. Distinguish event time, report time, and index time — conflating these is a common error.

8.5 Historical Reconstruction & Source Mapping

For "what really happened" questions spanning years: build a source map listing every source, its type, its independence from every other source, and its confidence contribution to each specific sub-claim — the discipline that separates rigorous history from a well-written but under-sourced narrative.

8.6 How Professionals Verify Claims — Synthesis

1. State the specific, falsifiable claim.
2. Identify what would confirm or refute it, before searching.
3. Search broadly across source types.
4. Separate primary from secondary/tertiary; prioritize primary.
5. Actively seek disconfirming evidence.
6. Assess independence, authority, and credibility separately.
7. Assign an explicit confidence level and document why.
8. Record the full source trail so the work is reproducible.

9.1 Taxonomies

A hierarchical classification scheme (broader → narrower), like a library's subject tree or species classification. Simple and readable, but rigid — an item often genuinely belongs in more than one branch.

9.2 Ontologies

Formally define not just categories but the relationships and constraints between them (e.g. "a Person worksFor an Organization"), typically machine-readable (RDF/OWL) — letting software infer new facts from stated ones.

9.3 Knowledge Graphs

A large, populated instance of an ontology-like structure: real entities as nodes, real relationships as edges. Search engines' Knowledge Panels are a public-facing view onto an internal graph, increasingly the retrieval layer behind AI assistants.

9.4 Wikidata

The largest fully open, collaboratively edited knowledge graph — structured facts with cited sources per statement, each item carrying a stable Q-number independent of language. Verification check: does a given statement actually have a reference attached? Many don't.

9.5 Linked Open Data

The practice of publishing structured data with shared identifiers (URIs) so datasets from unrelated publishers can be automatically cross-referenced — the infrastructure behind "everything is connected" in modern digital libraries.

9.6 Controlled Vocabularies & Metadata Systems

• Controlled vocabularies (MeSH, LC Subject Headings) enforce one standard term so search doesn't miss synonyms — why PubMed's MeSH search beats free-text keywords.
• Metadata standards (Dublin Core, MARC, schema.org, EXIF/IPTC) define what descriptive fields an item carries, so systems can interoperate.

10.1 LLMs as a Discovery Layer

LLMs generate fluent text by predicting likely continuations — they do not, by default, "look anything up" at answer time. This is the source of their central research risk: a fluent, confident-sounding answer is not evidence that it's correct.

10.2 Retrieval Systems and RAG

Retrieval-Augmented Generation pairs an LLM with a live retrieval step, feeding retrieved documents into the model's context so it can ground its answer in checkable sources. RAG substantially reduces — but does not eliminate — fabrication.

   User question
        │
        ▼
  ┌───────────┐     retrieve top-k     ┌──────────────┐
  │  Query →  │ ─────────────────────► │  Document /   │
  │ embedding │                        │  vector index │
  └───────────┘ ◄───────────────────── └──────────────┘
        │           relevant passages
        ▼
  ┌────────────────────────────┐
  │  LLM generates answer using │
  │  retrieved passages as      │
  │  grounding context          │
  └────────────────────────────┘
        │
        ▼
  Answer + citations  ← you must still open and check these

10.3 Embeddings, Revisited

The same embedding technology underlies RAG's retrieval, semantic deduplication, clustering, and recommendation — why an AI tool can retrieve an on-topic document sharing no keywords with your query, and, conversely, a superficially similar but substantively wrong one.

10.4 Knowledge Graphs in AI Systems

Some systems query a structured knowledge graph directly for entity facts — generally more reliable for discrete facts (dates, relationships, numbers) than free-text generation, because it's a lookup, not a guess.

10.5 AI Hallucinations

A fluent, confident output that is factually wrong or fabricated. Risk factors: very recent events, obscure topics, requests for precise numbers/quotations, and long reasoning chains where an early error compounds.

10.6 Verification Workflows for AI-Assisted Research

1. Use AI to generate leads and structure, not as a final citation.
2. Demand inline sources for every load-bearing claim.
3. Open every cited source — confirm it exists, says what's claimed, and is characterized fairly.
4. Cross-check any specific number, date, quote, or name independently.
5. Re-ask in a different phrasing/tool; treat inconsistency as a flag to dig deeper.

10.7 Human-in-the-Loop Validation

Treat AI output as a first-pass draft requiring verification proportional to stakes: a quick sanity check for low-stakes questions; the full workflow above — with a named accountable human — for anything health, legal, financial, safety-related, or destined for publication.

11.9 Building a Research Dossier

A reproducible dossier should contain, at minimum:

This structure is what makes a dossier reproducible: someone else, given only this document, should be able to retrace your path and reach the same — or a specifically disputable — conclusion.

12.1 Responsible Research — Core Principles

• Minimize harm — weigh public-interest value against the harm of compiling/publishing.
• Necessity and proportionality — collect only what the question actually requires.
• Transparency of method — a reproducible method is also an accountable one.
• Right of reply — seek a subject's response before publication where feasible.

12.2 Privacy

A legal ability to find information is distinct from an ethical justification to compile, publish, or act on it. Aggregating many individually-public facts about a private individual can itself constitute a privacy harm — the "mosaic effect" — even where no single fact was secret.

12.3 Copyright & Fair Use

• Archiving, quoting, and citing for research/criticism generally falls within fair use/fair dealing, but specifics vary by jurisdiction — this is not legal advice.
• A personal archival copy for research is on safer ground than republishing/redistributing it.
• Respect explicit licensing terms on datasets/APIs — "technically accessible" isn't "licensed for your use case."

12.4 Data Protection

Jurisdiction-specific law (GDPR and analogues) can impose obligations even on researchers who didn't collect data directly from the subject — around retention, further processing, and republication. Route sensitive-data research through institutional ethics review where applicable.

12.5 Public vs. Private Information

"Publicly accessible" and "intended for public consumption" are not identical. When in doubt about whether information was meant to be found and used this way, treat it as more sensitive, not less.

12.6 Ethical OSINT — Boundaries

• Legitimate OSINT uses only overtly available information and authorized access.
• It does not involve credential theft, pretexting, exploiting misconfigurations, bypassing paywalls, or impersonation.
• When access is genuinely required but unavailable, the answer is a legitimate request path — never a technical workaround.

12.7 Research Boundaries — When to Stop and Ask

A.5 Confidence-Level Quick Reference

Confirmed > Well-supported > Plausible > Contested > Unverified > Disputed/Debunked

Always state which one, and why.