Repository navigation
Boilerpipe fails to extract certain urls with 406 Error #24
Description
I'm trying to extract content for website but boilerpipe fail with this error ..
File "/local/lib/python2.7/site-packages/boilerpipe/extract/init.py", line 36, in init
connection = urllib2.urlopen(request)
File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
return _opener.open(url, data, timeout)
File "/usr/lib/python2.7/urllib2.py", line 406, in open
response = meth(req, response)
File "/usr/lib/python2.7/urllib2.py", line 519, in http_response
'http', request, response, code, msg, hdrs)
File "/usr/lib/python2.7/urllib2.py", line 444, in error
return self._call_chain(_args)
File "/usr/lib/python2.7/urllib2.py", line 378, in _call_chain
result = func(_args)
File "/usr/lib/python2.7/urllib2.py", line 527, in http_error_default
raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 406: Not Acceptable
First i thought error is coming from urllib2 . So i tried it with urllib2 independently but it was working fine.
These are the few url where its failing ..
http://mlmgod.com/onlinedeals/20-off-on-max-3000-ebay-rs-0-0-other-free-deals-coupons/
http://swapmyapp.com/how-to/use-microsoft-office-for-iphone/
http://www.techeyetech.com/micromax-canvas-tube-specifications-features-price-canvas-tube-palm-theater.html
Activity
[-]Boilerpipe fails to extractor for certain urls with 406 Error[/-][+]Boilerpipe fails to extractor certain urls with 406 Error[/+][-]Boilerpipe fails to extractor certain urls with 406 Error[/-][+]Boilerpipe fails to extract certain urls with 406 Error[/+]rshiva commentedon Aug 6, 2014
i traced back and found the errors is coming from urllib2 in headers
/local/lib/python2.7/site-packages/boilerpipe/extract/init.py
headers = {'User-Agent': 'Mozilla/5.0'}
request = urllib2.Request(kwargs['url'], headers=self.headers)
Instead if just pass headers = {'User-Agent': 'Mozilla'}
it worked fine. But couldn't understand why its breaking when we pass Mozilla/5.0 in headers
tuxdna commentedon Sep 12, 2016
@rshiva This seems like an issue outside of
python-boilerpipe. Can you reproduce this issue now ?jonjrodriguez commentedon Apr 9, 2017
I just started using python-boilerpipe on a project and am running into the same issue. Is there no update on this? Thanks.
tuxdna commentedon Apr 9, 2017
@thoughts1053 Can you provide with a sample script that can help replicate this issue?
krishnaupadhyay3 commentedon Jul 21, 2021
fixed by https://github.com/misja/python-boilerpipe/pull/59