图 书 馆
在压缩文本Web上On Compressing the Textual Web |
|
| 课程网址: | http://videolectures.net/wsdm2010_ferragina_octt/ |
| 主讲教师: | Paolo Ferragina |
| 开课单位: | 比萨大学 |
| 开课时间: | 2010-10-07 |
| 课程语种: | 英语 |
| 中文简介: |  |
| 课程简介: | Nowadays we know how to effectively compress most basic components of any modern search engine, such as, the graphs arising from the Web structure and/or its usage, the posting lists, and the dictionary of terms. But we are not aware of any study which has deeply addressed the issue of compressing the raw Web pages. Many Web applications use simple compression algorithms— e.g. gzip, or word-based Move-to-Front or Huffman coders— and conclude that, even compressed, raw data take more space than Inverted Lists. In this paper we investigate two typical scenarios of use of data compression for large Web collections. In the first scenario, the compressed pages are stored on disk and we only need to support the fast scanning of large parts of the com- pressed collection (such as for map-reduce paradigms). In the second scenario, we consider the fast access to individual pages of the compressed collection that is distributed among the RAMs of many PCs (such as for search engines and miners). For the first scenario, we provide a thorough experimental comparison among state-of-the-art compressors thus indicating pros and cons of the available solutions. For the second scenario, we compare compressed-storage solutions with the new technology of compressed self-indexes. Our results show that Web pages are more compressible than expected and, consequently, that some common beliefs in this area should be reconsidered. Our results are novel for the large spectrum of tested approaches and the size of datasets, and provide a threefold contribution: a non- trivial baseline for designing new compressed-storage solutions, a guide for software developers faced with Web-page storage, and a natural complement to the recent figures on InvertedList-compression. |
| 关 键 词: | 计算机科学; 语义Web; 搜索引擎; 压缩算法 |
| 课程来源: | 视频讲座网 |
| 最后编审: | 2019-10-28：lxf |
| 阅读次数: | 28 |