perf(agent): cache scraped pages and retry failed fetches

Browse Source

Adds a disk-backed scrape cache under workspace/web_cache keyed by
the SHA-256 of the URL, honoring WEB_SEARCH_SCRAPE_CACHE_TTL so
repeated lookups within the window skip the network entirely.

Fetches now retry twice with exponential backoff before giving up,
and URLs are screened against SCRAPING_BLOCKED_EXTENSIONS and
SCRAPING_BLOCKED_DOMAINS to avoid downloading binaries or
denylisted hosts.

New scraping knobs (enable flag, max chars, cache TTL, blocklists)
are exposed via .env, alongside a provider switch to stepfun with
the step-3.7-flash default model.

...

This commit is contained in:

marcocosta10000

2026-10-26 17:01:50 -03:00

parent 723ca1506b

commit 5697cd0c40

4 changed files with 216 additions and 81 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

__pycache__/psagentai_agent.cpython-311.pyc

BIN

Binary file not shown.