Shingleprinting code for estimating document similarity

This package impelements the Broder et al method for approximating the Jaccard metric for comparing two text files. The Jaccard metric measures the similarity of two sets of features by computing the cardinality of the intersection of the two sets divided by the cardinality of the union. The Broder method (first published in the 1997 World-Wide Web Conference) for approximating this uses random one-to-one functions from the space of features to a bounded set of integers to select a single feature from each document for each function. The probablity of selecting the same feature from two documents is (when averaged over all choices of random functions) equal to the Jaccard metric. This package implements shingling, and a specific family of hash functions for converting text documents into sets of features. Many of the techniques used here, and most applications of this code, are covered by patents owned by Yahoo, which purchased Overture, which purchased Alta Vista, so only investigative use, consistent with applicable patent laws, is to be pursued using this code without negotiation of appropriate licenses. Nonetheless, we find this to be of great value as a research tool, and release it in that spirit. The HTML parsing included herein is pretty kludgy; I recommend changing it. Version 1.1.1 fixes a warning message, and compiles correctly. Version 1.1 differs from the 1.0 in that 1) The random number generator is now seeded, badly. 2) The big-endian code is now correct.