Research Scrape Admin
View archive →

Research Sources

The provider sites research_scraper.py scrapes weekly, each summarized by Ollama and emailed as a digest of new thematic research. Adding a source here only makes the weekly job actually scrape it once a developer has registered a matching parser for its key in research_scraper.py's PARSER_REGISTRY — these sites are too different from each other for one generic scraper, so this page manages the catalog (URL, method, on/off), not the scraping logic itself.

Add research source

Lowercase, no spaces. Treat as permanent once items have been scraped under it.
The two generic methods are config-driven — fill in Config (JSON) below, no Python needed. Pick Custom only for something a config truly can't express (so far: just DB Research's PDF-viewer article step).
Leave empty ({}) for method=Custom. See research_sources' DDL comment in db.py and research_scraper.py's parse_generic_json_api_listing / parse_generic_html_listing for the full field reference (next_text, links_text, item_filter, url_prefix, ...). Use "View last fetch" on an existing row below to see real payload shapes to write selectors/paths against.

All sources