feat(agent): scrape linked pages to ground answers in real content

Adds `_fetch_page_content()`, which downloads a URL with a browser-like User-Agent and uses BeautifulSoup with the lxml parser to strip scripts, navigation, and other non-content markup, returning up to 3000 characters of readable text. The response validator now also compares generated replies against this scraped text, flagging named entities and claims that do not appear in the retrieved page content. Dependencies: adds beautifulsoup4==4.12.3 and pins lxml==5.3.0 for parser compatibility.
2026-10-26 16:48:29 -03:00
parent 9eed02645b
commit 723ca1506b
231 changed files with 20508 additions and 5330 deletions
@@ -0,0 +1 @@
+pip
@@ -0,0 +1,111 @@
+Metadata-Version: 2.5
+Name: soupsieve
+Version: 2.10
+Summary: A modern CSS selector implementation for Beautiful Soup.
+Project-URL: Homepage, https://github.com/facelessuser/soupsieve
+Author-email: Isaac Muse <Isaac.Muse@gmail.com>
+License-Expression: MIT
+License-File: LICENSE.md
+Keywords: CSS,HTML,XML,filter,query,selector,soup
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Internet :: WWW/HTTP :: Dynamic Content
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+[![Donate via PayPal][donate-image]][donate-link]
+[![Coverage Status][codecov-image]][codecov-link]
+[![PyPI Version][pypi-image]][pypi-link]
+[![PyPI Downloads][pypi-down]][pypi-link]
+[![PyPI - Python Version][python-image]][pypi-link]
+[![License][license-image-mit]][license-link]
+
+# Soup Sieve
+
+## Overview
+
+Soup Sieve is a CSS selector library designed to be used with [Beautiful Soup 4][bs4]. It aims to provide selecting,
+matching, and filtering using modern CSS selectors. Soup Sieve currently provides selectors from the CSS level 1
+specifications up through the latest CSS level 4 drafts and beyond (though some are not yet implemented).
+
+Soup Sieve was written with the intent to replace Beautiful Soup's builtin select feature, and as of Beautiful Soup
+version 4.7.0, it now is :confetti_ball:. Soup Sieve can also be imported in order to use its API directly for
+more controlled, specialized parsing.
+
+Soup Sieve has implemented most of the CSS selectors up through the latest CSS draft specifications, though there are a
+number that don't make sense in a non-browser environment. Selectors that cannot provide meaningful functionality simply
+do not match anything. Some of the supported selectors are:
+
+- `.classes`
+- `#ids`
+- `[attributes=value]`
+- `parent child`
+- `parent > child`
+- `sibling ~ sibling`
+- `sibling + sibling`
+- `:not(element.class, element2.class)`
+- `:is(element.class, element2.class)`
+- `parent:has(> child)`
+- and [many more](https://facelessuser.github.io/soupsieve/selectors/)
+
+
+## Installation
+
+You must have Beautiful Soup already installed:
+
+```console
+$ pip install beautifulsoup4
+```
+
+In most cases, assuming you've installed version 4.7.0, that should be all you need to do, but if you've installed via
+some alternative method, and Soup Sieve is not automatically installed, you can install it directly:
+
+```console
+$ pip install soupsieve
+```
+
+If you want to manually install it from source, first ensure that [`build`](https://pypi.org/project/build/) is
+installed:
+
+```console
+$ pip install build
+```
+
+Then navigate to the root of the project and build the wheel and install (replacing `<ver>` with the current version):
+
+```console
+$ python -m build -w
+$ pip install dist/soupsieve-<ver>-py3-none-any.whl
+```
+
+## Documentation
+
+Documentation is found here: https://facelessuser.github.io/soupsieve/.
+
+## License
+
+MIT
+
+[bs4]: https://beautiful-soup-4.readthedocs.io/en/latest/#
+[codecov-image]: https://img.shields.io/codecov/c/github/facelessuser/soupsieve/master.svg?logo=codecov&logoColor=aaaaaa&labelColor=333333
+[codecov-link]: https://codecov.io/github/facelessuser/soupsieve
+[pypi-image]: https://img.shields.io/pypi/v/soupsieve.svg?logo=pypi&logoColor=aaaaaa&labelColor=333333
+[pypi-down]: https://img.shields.io/pypi/dm/soupsieve.svg?logo=pypi&logoColor=aaaaaa&labelColor=333333
+[pypi-link]: https://pypi.python.org/pypi/soupsieve
+[python-image]: https://img.shields.io/pypi/pyversions/soupsieve?logo=python&logoColor=aaaaaa&labelColor=333333
+[license-image-mit]: https://img.shields.io/badge/license-MIT-blue.svg?labelColor=333333
+[license-link]: https://github.com/facelessuser/soupsieve/blob/main/LICENSE.md
+[donate-image]: https://img.shields.io/badge/Donate-PayPal-3fabd1?logo=paypal
+[donate-link]: https://www.paypal.me/facelessuser
@@ -0,0 +1,20 @@
+soupsieve-2.10.dist-info/INSTALLER,sha256=zuuue4knoyJ-UwPPXg8fezS7VCrXJQrAP7zeNuwvFQg,4
+soupsieve-2.10.dist-info/METADATA,sha256=AuzWxy0voJEb7f2Z1TshmOG79Vb0YGEIgZ6MGf3zKIA,4437
+soupsieve-2.10.dist-info/RECORD,,
+soupsieve-2.10.dist-info/WHEEL,sha256=W3fkpkm7-wf9vBI5Z-7s0eWkeM-spu78I8Neb98DeEg,87
+soupsieve-2.10.dist-info/licenses/LICENSE.md,sha256=dqCs25jDpjaB4gtvCWaZ74ytX7NdVu74OH4Ppx15epo,1096
+soupsieve/__init__.py,sha256=1L-rgrX3dQKWxvglaVuoAcZk4ImaY_jrUA8VkEqIh2o,5145
+soupsieve/__meta__.py,sha256=uid8eA6uT5j9RV3Zkkpr4JFzJ59iu38sKjzJdwcDEkg,6767
+soupsieve/__pycache__/__init__.cpython-311.pyc,,
+soupsieve/__pycache__/__meta__.cpython-311.pyc,,
+soupsieve/__pycache__/css_match.cpython-311.pyc,,
+soupsieve/__pycache__/css_parser.cpython-311.pyc,,
+soupsieve/__pycache__/css_types.cpython-311.pyc,,
+soupsieve/__pycache__/pretty.cpython-311.pyc,,
+soupsieve/__pycache__/util.cpython-311.pyc,,
+soupsieve/css_match.py,sha256=3l7inf4tZEUk6GKtkeVwd8jDu8-kN2n4Rlo-OX0zj2g,63105
+soupsieve/css_parser.py,sha256=_p7p-UCWZtuAQDFbSpygLQSekipKQJNjq03RIRALKh8,48962
+soupsieve/css_types.py,sha256=yboTb2zl0CDwVI2k221FIhT8BDsCDOW9BMYeZp9Dxu8,9226
+soupsieve/pretty.py,sha256=iSM-NKCzruJQCwEriQT72LrF4slxsib7TB552H2bsqA,4362
+soupsieve/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+soupsieve/util.py,sha256=M6bE3QSBfbzPhUDJ9UqwO4ldzYVC44GmVxpqgqpm_fs,3370
@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.32.4
+Root-Is-Purelib: true
+Tag: py3-none-any
@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 - 2026 Isaac Muse <isaacmuse@gmail.com>
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.