vddf

DEPLOY.md

@@ -91,6 +91,9 @@ nano .env
 | `MIN_MESSAGE_LINES` | `3` | Minimum lines per agent message |
 | `WEB_SEARCH_ENABLED` | `true` | Enable automatic web search |
 | `WEB_SEARCH_MAX_RESULTS` | `5` | Max web search results |
+| `WEB_SEARCH_SCRAPE_ENABLED` | `true` | Fetch page content for search results (uses headless browser fallback) |
+| `WEB_SEARCH_SCRAPE_CACHE_TTL` | `3600` | Cache TTL for scraped pages in seconds |
+| `WEB_SEARCH_SCRAPE_MAX_CHARS` | `15000` | Max characters per scraped page |
 | `VISION_MODEL` | `meta-llama/llama-4-maverick-17b-128b-instruct` | Vision model |
 | `VISION_PROVIDER` | auto | Vision provider |
 | `AUTO_ENRICH_EVERY` | `2` | Profile auto-enrichment frequency (`0` = disabled) |
@@ -302,6 +305,9 @@ Recommended naming convention to avoid conflicts:
 | `UPLOAD_CONTEXT_MAX_CHARS` | `8000` | Max chars from uploads sent to AI (`0` = unlimited) |
 | `WEB_SEARCH_ENABLED` | `true` | Enable automatic web search |
 | `WEB_SEARCH_MAX_RESULTS` | `5` | Maximum search results returned |
+| `WEB_SEARCH_SCRAPE_ENABLED` | `true` | Fetch page content for search results (uses headless browser fallback for JS-rendered sites) |
+| `WEB_SEARCH_SCRAPE_CACHE_TTL` | `3600` | Cache TTL for scraped pages in seconds |
+| `WEB_SEARCH_SCRAPE_MAX_CHARS` | `15000` | Max characters per scraped page |
 | `AUTO_ENRICH_EVERY` | `2` | Auto-enrich profiles every N messages (`0` = disabled) |
 | `AUTO_IMPROVE_EVERY` | `20` | Auto-improve system every N messages (`0` = disabled) |
 

README.md

@@ -1,13 +1,14 @@
 # PsAgentAI - Personal Conversational AI Agent
 
-A Telegram-based conversational AI agent that acts as a personal companion. It maintains long-term memory through user/agent profiles, manages daily routines, supports file uploads/workspace editing, performs web searches, analyzes images, and sends proactive messages based on configured schedules.
+A Telegram-based conversational AI agent that acts as a personal companion. It maintains long-term memory through user/agent profiles, manages daily routines, supports file uploads/workspace editing, performs web searches with headless browser fallback, analyzes images, visits websites directly, and sends proactive messages based on configured schedules.
 
 ## Key Features
 
 - **Long-term memory**: User and agent profiles with auto-enrichment from conversations
 - **Daily routine management**: Schedule activities, meals, work/study times
 - **File workspace**: Create, edit, list and download files via natural language
-- **Web search**: Automatic or on-demand search for up-to-date information with hallucination validation
+- **Web search**: Automatic or on-demand search with headless browser fallback for JS-rendered e-commerce sites
+- **Direct website access**: Visit and analyze any website by name — the agent can enter sites and extract their content
 - **Anti-hallucination**: Post-search validation ensures responses cite sources and don't invent facts
 - **Vision**: Image analysis with configurable vision models
 - **Voice**: Voice message transcription and response
@@ -31,7 +32,8 @@ app/
 ├── contexts/                   # Conversation history by date
 │   └── YYYY-MM-DD.json
 ├── workspace/                  # Created and edited files
-│   └── projects/
+│   ├── projects/
+│   └── web_cache/              # Cached webpage content (auto-created)
 ├── README.md
 ├── DEPLOY.md
 ├── RULES.md
@@ -76,6 +78,9 @@ nano .env
 | `UPLOAD_CONTEXT_MAX_CHARS` | `8000` | Max chars from uploads sent to AI (`0` = unlimited) |
 | `WEB_SEARCH_ENABLED` | `true` | Enable automatic web search |
 | `WEB_SEARCH_MAX_RESULTS` | `5` | Max search results returned |
+| `WEB_SEARCH_SCRAPE_ENABLED` | `true` | Fetch page content for search results (uses headless browser fallback) |
+| `WEB_SEARCH_SCRAPE_CACHE_TTL` | `3600` | Cache TTL for scraped pages in seconds |
+| `WEB_SEARCH_SCRAPE_MAX_CHARS` | `15000` | Max characters per scraped page |
 | `VISION_MODEL` | `meta-llama/llama-4-maverick-17b-128b-instruct` | Model for image analysis |
 | `VISION_PROVIDER` | auto | Provider for vision model |
 | `AUTO_ENRICH_EVERY` | `2` | Update profiles every N messages (`0` = disabled) |
@@ -194,9 +199,44 @@ During conversation, you can update profiles using phrases:
 3. **Profiles**: User and agent profiles are persisted as JSON and enriched automatically based on conversations
 4. **AI Calls**: The agent uses the configured provider/model to generate responses with full context
 5. **Proactive Messages**: Based on your schedule and routine, the agent may initiate conversations
-6. **Web Search**: When enabled, the agent searches the web for questions requiring current information
+6. **Web Search**: When enabled, the agent searches the web for questions requiring current information. If HTTP scraping fails (JS-rendered sites), it falls back to a headless Chromium browser via Playwright.
-7. **Vision**: Images are analyzed using the configured vision model
+7. **Direct Website Access**: You can ask the agent to visit any website by name or URL. It fetches the page content using a headless browser (or HTTP scrape for static sites) and extracts the readable text.
-8. **Auto-improvement**: The system periodically analyzes conversations and suggests improvements
+8. **Vision**: Images are analyzed using the configured vision model
+9. **Auto-improvement**: The system periodically analyzes conversations and suggests improvements
+
+## Web Search & Website Access
+
+### How Web Search Works
+
+When you ask a question that requires current information, the agent:
+1. Runs a DuckDuckGo search to find relevant results
+2. Fetches page content from each result using HTTP scraping + BeautifulSoup
+3. For JS-rendered sites (e-commerce, SPAs), automatically falls back to a headless Chromium browser via Playwright
+4. Extracts readable text, strips scripts/styles, and passes it to the AI for grounded responses
+5. Validates the response against source data to prevent hallucination
+
+The system caches scraped pages for 1 hour (configurable via `WEB_SEARCH_SCRAPE_CACHE_TTL`).
+
+### Direct Website Visit
+
+You can ask the agent to enter or analyze any website:
+
+- `Entre no site syspanel.com.br e me diga do que se trata`
+- `Analisa https://www.google.com para mim`
+- `Visite este site e verifique os preços`
+
+The agent detects URLs + action keywords (entra, acesse, analise, visite, pesquise, etc.) and automatically scrapes the page. It uses:
+1. Headless Chromium (Playwright) — best for JS-rendered/spa sites like KaBuM, MercadoLivre, Buscape
+2. HTTP scraping (BeautifulSoup) — fallback for static sites
+3. Cache — previously fetched pages are reused from `workspace/web_cache/`
+
+**Supported websites**: Most informational sites, blogs, news portals, e-commerce product listing pages. Sites with heavy bot protection (Cloudflare CAPTCHA) may not work.
+
+### Example Use Cases
+
+- **Price comparison**: `Encontre os SSD de 120GB mais baratos e me informe detalhes e link de compra` → agent searches and fetches real prices from KaBuM/Zoom/Buscape
+- **Website analysis**: `Entre no site syspanel.com.br e me diga do que se trata` → agent visits and summarizes the site
+- **Current events**: `Pesquisa na internet as notícias sobre tecnologia` → agent searches and reads article content
 
 ## File Upload and Analysis
 
@@ -263,6 +303,26 @@ tar czf /root/backups/psagentai-$(date +%F).tar.gz -C /opt/psagentai/app user_pr
 - **AI errors**: Check API keys and provider configuration with `/viewai`
 - **Voice not working**: Ensure `ffmpeg` is installed
 - **Image analysis not working**: Check vision configuration with `/viewvision`
+- **Web search results empty**: Check that web scraping works — e-commerce sites (KaBuM, MercadoLivre) require Playwright + Chromium. Reinstall with `./iniciar.sh`. Sites with CAPTCHA protection may fail regardless.
+- **Website visit fails**: Some sites block headless browsers or show CAPTCHA. Try simpler informational sites first. The agent shows `[URL_VISIT] failed to fetch url=...` in console output if fetching fails.
+
+## Dependencies
+
+### Python packages
+
+All listed in `requirements.txt`:
+- `python-telegram-bot` — Telegram bot framework
+- `requests` + `beautifulsoup4` + `lxml` — HTTP scraping for static pages
+- `playwright` — Headless Chromium for JS-rendered/spa sites (e-commerce, dynamic content)
+- `openai` — OpenAI-compatible AI API client
+- `PyMuPDF`, `python-docx`, `Pillow`, `SpeechRecognition`, `pydub` — File processing
+
+### System packages
+
+- `ffmpeg` — Voice message processing
+- `chromium-browser` — Installed automatically by Playwright (`playwright install chromium`)
+
+Run `./iniciar.sh` to set up all dependencies including browser installation.
 
 ## Development