fix(fetch): improve SPA rendering and URL normalization in browser scraper

Browse Source

Increase hydration wait to 5s and scroll the page three times to
trigger lazy-loaded content before extracting text. Prefer body text
when it yields more content than main-content selectors.

Normalize URLs to https and strip the www. prefix for consistent
deduplication, and allow bare domains with no trailing path to match
the extraction pattern.

Also update agent/user profiles with site reputation and price
comparison guidance, and append conversation logs for 2026-10-27.

...

This commit is contained in:

marcocosta10000

2026-10-27 20:14:50 -03:00

parent aa70c4f6f6

commit 2323a2d1ab

5 changed files with 259 additions and 104 deletions

Show all changes Ignore whitespace when comparing lines Ignore changes in amount of whitespace Ignore changes in whitespace at EOL

Download Patch File Download Diff File Expand all files Collapse all files

__pycache__/psagentai_agent.cpython-311.pyc

BIN

Binary file not shown.