Skip to content

Encoding Issues - UnicodeDecodeError: 'utf8' codec can't decode byte #19

Open

Description

@jimishjoban

Hey guys,

First of all thanks for python-boilerpipe

Trying to use Boilerpipe but can't extract properly some documents...

from boilerpipe.extract import Extractor
extractorType="DefaultExtractor"
sourceUrl = 'http://www.indiatimes.com/news/india/arvind-kejriwal-to-seek-political-sanyas-127620.html'
extractor = Extractor(extractor=extractorType, url=sourceUrl)
Traceback (most recent call last):
File "", line 1, in
File "/Library/Python/2.7/site-packages/boilerpipe/extract/init.py", line 41, in init
self.data = unicode(self.data, encoding)
UnicodeDecodeError: 'utf8' codec can't decode byte 0x91 in position 53647: invalid start byte

The document seems to be having some non-utf8 characters... which do not seem to parse well... Any workaround for the problem?

Activity

  1. Caimany commented on Jun 1, 2015

    @Caimany

    I solved UnicodeDecodeError ,you can see what I modified in init.py
    https://github.com/Caimany/python-boilerpipe/blob/master/src/boilerpipe/extract/__init__.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions