marcocosta10000 5697cd0c40 perf(agent): cache scraped pages and retry failed fetches ...

Adds a disk-backed scrape cache under workspace/web_cache keyed by
the SHA-256 of the URL, honoring WEB_SEARCH_SCRAPE_CACHE_TTL so
repeated lookups within the window skip the network entirely.

Fetches now retry twice with exponential backoff before giving up,
and URLs are screened against SCRAPING_BLOCKED_EXTENSIONS and
SCRAPING_BLOCKED_DOMAINS to avoid downloading binaries or
denylisted hosts.

New scraping knobs (enable flag, max chars, cache TTL, blocklists)
are exposed via .env, alongside a provider switch to stepfun with
the step-3.7-flash default model.

2026-10-26 17:01:50 -03:00

347 KiB

Raw History

View Raw