feat(agent): scrape linked pages to ground answers in real content

Adds `_fetch_page_content()`, which downloads a URL with a browser-like User-Agent and uses BeautifulSoup with the lxml parser to strip scripts, navigation, and other non-content markup, returning up to 3000 characters of readable text. The response validator now also compares generated replies against this scraped text, flagging named entities and claims that do not appear in the retrieved page content. Dependencies: adds beautifulsoup4==4.12.3 and pins lxml==5.3.0 for parser compatibility.
2026-10-26 16:48:29 -03:00
parent 9eed02645b
commit 723ca1506b
231 changed files with 20508 additions and 5330 deletions
@@ -0,0 +1,103 @@
+Metadata-Version: 2.1
+Name: lxml
+Version: 5.3.0
+Summary: Powerful and Pythonic XML processing library combining libxml2/libxslt with the ElementTree API.
+Home-page: https://lxml.de/
+Author: lxml dev team
+Author-email: lxml-dev@lxml.de
+Maintainer: lxml dev team
+Maintainer-email: lxml-dev@lxml.de
+License: BSD-3-Clause
+Project-URL: Source, https://github.com/lxml/lxml
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: License :: OSI Approved :: BSD License
+Classifier: Programming Language :: Cython
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.6
+Classifier: Programming Language :: Python :: 3.7
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: C
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing :: Markup :: HTML
+Classifier: Topic :: Text Processing :: Markup :: XML
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.6
+License-File: LICENSE.txt
+License-File: LICENSES.txt
+Provides-Extra: cssselect
+Requires-Dist: cssselect>=0.7; extra == "cssselect"
+Provides-Extra: html5
+Requires-Dist: html5lib; extra == "html5"
+Provides-Extra: html_clean
+Requires-Dist: lxml-html-clean; extra == "html-clean"
+Provides-Extra: htmlsoup
+Requires-Dist: BeautifulSoup4; extra == "htmlsoup"
+Provides-Extra: source
+Requires-Dist: Cython>=3.0.11; extra == "source"
+
+lxml is a Pythonic, mature binding for the libxml2 and libxslt libraries.  It
+provides safe and convenient access to these libraries using the ElementTree
+API.
+
+It extends the ElementTree API significantly to offer support for XPath,
+RelaxNG, XML Schema, XSLT, C14N and much more.
+
+To contact the project, go to the `project home page
+<https://lxml.de/>`_ or see our bug tracker at
+https://launchpad.net/lxml
+
+In case you want to use the current in-development version of lxml,
+you can get it from the github repository at
+https://github.com/lxml/lxml .  Note that this requires Cython to
+build the sources, see the build instructions on the project home
+page.  To the same end, running ``easy_install lxml==dev`` will
+install lxml from
+https://github.com/lxml/lxml/tarball/master#egg=lxml-dev if you have
+an appropriate version of Cython installed.
+
+
+After an official release of a new stable series, bug fixes may become
+available at
+https://github.com/lxml/lxml/tree/lxml-5.3 .
+Running ``easy_install lxml==5.3bugfix`` will install
+the unreleased branch state from
+https://github.com/lxml/lxml/tarball/lxml-5.3#egg=lxml-5.3bugfix
+as soon as a maintenance branch has been established.  Note that this
+requires Cython to be installed at an appropriate version for the build.
+
+5.3.0 (2024-08-10)
+==================
+
+Features added
+--------------
+
+* GH#421: Nested ``CDATA`` sections are no longer rejected but split on output
+  to represent ``]]>`` correctly.
+  Patch by Gertjan Klein.
+
+Bugs fixed
+----------
+
+* LP#2060160: Attribute values serialised differently in ``xmlfile.element()`` and ``xmlfile.write()``.
+
+* LP#2058177: The ISO-Schematron implementation could fail on unknown prefixes.
+  Patch by David Lakin.
+
+Other changes
+-------------
+
+* LP#2067707: The ``strip_cdata`` option in ``HTMLParser()`` turned out to be useless and is now deprecated.
+
+* Binary wheels use the library versions libxml2 2.12.9 and libxslt 1.1.42.
+
+* Windows binary wheels use the library versions libxml2 2.11.8 and libxslt 1.1.39.
+
+* Built with Cython 3.0.11.
+
+