feat(agent): scrape linked pages to ground answers in real content
Adds `_fetch_page_content()`, which downloads a URL with a browser-like User-Agent and uses BeautifulSoup with the lxml parser to strip scripts, navigation, and other non-content markup, returning up to 3000 characters of readable text. The response validator now also compares generated replies against this scraped text, flagging named entities and claims that do not appear in the retrieved page content. Dependencies: adds beautifulsoup4==4.12.3 and pins lxml==5.3.0 for parser compatibility.
This commit is contained in:
@@ -0,0 +1,29 @@
|
||||
lxml is copyright Infrae and distributed under the BSD license (see
|
||||
doc/licenses/BSD.txt), with the following exceptions:
|
||||
|
||||
Some code, such a selftest.py, selftest2.py and
|
||||
src/lxml/_elementpath.py are derived from ElementTree and
|
||||
cElementTree. See doc/licenses/elementtree.txt for the license text.
|
||||
|
||||
lxml.cssselect and lxml.html are copyright Ian Bicking and distributed
|
||||
under the BSD license (see doc/licenses/BSD.txt).
|
||||
|
||||
test.py, the test-runner script, is GPL and copyright Shuttleworth
|
||||
Foundation. See doc/licenses/GPL.txt. It is believed the unchanged
|
||||
inclusion of test.py to run the unit test suite falls under the
|
||||
"aggregation" clause of the GPL and thus does not affect the license
|
||||
of the rest of the package.
|
||||
|
||||
The isoschematron implementation uses several XSL and RelaxNG resources:
|
||||
* The (XML syntax) RelaxNG schema for schematron, copyright International
|
||||
Organization for Standardization (see
|
||||
src/lxml/isoschematron/resources/rng/iso-schematron.rng for the license
|
||||
text)
|
||||
* The skeleton iso-schematron-xlt1 pure-xslt schematron implementation
|
||||
xsl stylesheets, copyright Rick Jelliffe and Academia Sinica Computing
|
||||
Center, Taiwan (see the xsl files here for the license text:
|
||||
src/lxml/isoschematron/resources/xsl/iso-schematron-xslt1/)
|
||||
* The xsd/rng schema schematron extraction xsl transformations are unlicensed
|
||||
and copyright the respective authors as noted (see
|
||||
src/lxml/isoschematron/resources/xsl/RNG2Schtrn.xsl and
|
||||
src/lxml/isoschematron/resources/xsl/XSD2Schtrn.xsl)
|
||||
Reference in New Issue
Block a user