webLyzard Publications

TextSweeper - A System for Content Extraction and Overview Page Detection

Lang, Heinz-Peter and Wohlgenannt, Gerhard and Weichselbraun, Albert (2012) TextSweeper - A System for Content Extraction and Overview Page Detection. In: International Conference on Information Resources Management (Conf-IRM), 21-23 May 2012, Vienna, Austria.

[thumbnail of TextSweeper - A System for Content Extraction and Overview Page Detection]
PDF (TextSweeper - A System for Content Extraction and Overview Page Detection) - Accepted Version


Web pages not only contain main content, but also other elements such as navigation panels, advertisements and links to related documents. Furthermore, overview pages (summarization pages and entry points) duplicate and aggregate parts of articles and thereby create redundancies. The noise elements in Web pages as well as overview pages affect the performance of downstream processes such as Web-based Information Retrieval. Context Extraction's task is identifying and extracting the main content from a Web page. In this research-in-progress paper we present an approach which not only identifies and extracts the main content, but also detects overview pages and thereby allows skipping them. The content extraction part of the system is an extension of existing Text-to-Tag ratio methods, overview page detection is accomplished with the net text length heuristic. Preliminary results and ad-hoc evaluation indicate a promising system performance. A formal evaluation and comparison to other state-of-the-art approaches is part of future work.

Item Type:Conference or Workshop Item (Paper)
Uncontrolled Keywords:content extraction, overview pages, text filtering, natural language processing, contextualized information spaces, Web-based information retrieval
Subjects:Q Science > QA Mathematics > QA75 Electronic computers. Computer science
Divisions:Faculty of Engineering, Science and Mathematics > School of Electronics and Computer Science
ID Code:55
Deposited By: Dr Albert Weichselbraun
Deposited On:28 Sep 2012 12:57
Last Modified:28 Sep 2012 12:57

Repository Staff Only: item control page