A python wrapper for Boilerpipe, an excellent Java library for boilerplate removal and fulltext extraction from HTML pages.

Dependencies:

• jpype
• chardet

The boilerpipe jar files will get fetched and included automatically when building the package.

Checkout the code:

git clone https://github.com/misja/python-boilerpipe.git cd python-boilerpipe 

virtualenv

virtualenv env source env/bin/activate pip install -r requirements.txt python setup.py install 

Fedora

sudo dnf install -y python2-jpype sudo python setup.py install 

Be sure to have set JAVA_HOME properly since jpype depends on this setting.

The constructor takes a keyword argument extractor, being one of the available boilerpipe extractor types:

• DefaultExtractor
• ArticleExtractor
• ArticleSentencesExtractor
• KeepEverythingExtractor
• KeepEverythingWithMinKWordsExtractor
• LargestContentExtractor
• NumWordsRulesExtractor
• CanolaExtractor

If no extractor is passed the DefaultExtractor will be used by default. Additional keyword arguments are either html for HTML text or url.

from boilerpipe.extract import Extractor extractor = Extractor(extractor='ArticleExtractor', url=your_url) 

Then, to extract relevant content:

extracted_text = extractor.getText() extracted_html = extractor.getHTML() 

For KeepEverythingWithMinKWordsExtractor we have to specify kMin parameter, which defaults to 1 for now:

extractor = Extractor(extractor='KeepEverythingWithMinKWordsExtractor', url=your_url, kMin=20)