NAME HTML::Detergent - Clean the gunk off an HTML document VERSION Version 0.03 SYNOPSIS use HTML::Detergent; my $scrubber = HTML::Detergent->new($config); # $input can be a string, GLOB reference, or XML::LibXML::Document my $doc = $scrubber->process($input, $uri); DESCRIPTION HTML::Detergent is for isolating the main content of an HTML page, stripping it of navigation, visual design, and other ancillary content. The main purpose of this module is to aid in the migration of web content from one content management system to another. It is also useful for preparing HTML resources for automated content inventories. The module currently has no heuristics for determining the main content of a page. It works instead by assuming prior knowledge of the layout, given in the configuration by an XPath expression that uniquely isolates the container node. That node is then lifted into a new document, along with the contents of the "", and returned by the "process" method. To accommodate multiple layouts on a site, the module can be initialized to match multiple XPath expressions. If further processing is necessary, an expression can be associated with an XSLT stylesheet, which is assumed to produce an entire document, thus overriding the default behaviour. After the new document is generated and before it is returned by "process", it is possible to inject "" and "" elements into the "". This enables the inclusion of metadata and the re-association of the main content with links that represent aspects of the page which have been removed (e.g. navigation, copyright statement, etc.). In addition, if the page's URI is supplied to the "process" method, the "" element is either added or rewritten to reflect it, and the URI attributes in the body are rewritten relative to the base. Otherwise they are left alone. The document returned is an XML::LibXML::Document object using the XHTML namespace, "http://www.w3.org/1999/xhtml", but does not profess to validate against any particular schema. If DTD declarations (including the empty "" recommended in HTML5) are desired, they can be added on afterward. Likewise, the object can be converted from XML into HTML using "toStringHTML" in XML::LibXML::Document. METHODS new %CONFIG | \%CONFIG | $CONFIG Initialize the processor, either with a list of configuration parameters, a HASH reference thereof, or an HTML::Detergent::Config object. Below are the valid parameters: match This is an ARRAY reference of XPath expressions to try against the document, in order of preference. Entries optionally may be two-element ARRAY references themselves, the second element being a URL where an XSLT stylesheet may be found. match => [ '/some/xpath/expression', [ '/other/expr', '/url/of/transform.xsl' ], ], link This is a HASH reference where the keys correspond to "rel" attributes and the values to "href" attributes of "" elements. If the values are ARRAY references, they will be processed in document order. "rel" attributes will be sorted lexically. If a callback is supplied instead, the caller expects a result of the same form. link => { rel1 => 'href1', rel2 => [ qw(href2 href3) ] }, # or link => \&_link_cb, meta This is a HASH reference where the keys correspond to "name" attributes and the values to "content" attributes of "" elements. If the values are ARRAY references, they will be processed in document order. "name" attributes will be sorted lexically. If a callback is supplied instead, the caller expects a result of the same form. meta => { name1 => 'content1', name2 => [ qw(content2 content3) ] }, # or meta => \&_meta_cb, callback These callbacks will be passed into the internal XML::LibXSLT processor. See XML::LibXML::InputCallback for details. callback => [ \&_match_cb, \&_open_cb, \&_read_cb, \&_close_cb ], # or callback => $icb, # isa XML::LibXML::InputCallback process $INPUT [, $URI, $CONFIG ] Processes $INPUT, which may be a string, GLOB reference, or XML::LibXML::Document object. Returns an XML::LibXML::Document object with the changes mentioned in the "DESCRIPTION". AUTHOR Dorian Taylor, "" BUGS Please report any bugs or feature requests to "bug-html-detergent at rt.cpan.org", or through the web interface at . I will be notified, and then you'll automatically be notified of progress on your bug as I make changes. SUPPORT You can find documentation for this module with the perldoc command. perldoc HTML::Detergent You can also look for information at: * RT: CPAN's request tracker (report bugs here) * AnnoCPAN: Annotated CPAN documentation * CPAN Ratings * Search CPAN SEE ALSO XML::LibXML XML::LibXSLT HTML::HTML5::Parser LICENSE AND COPYRIGHT Copyright 2013 Dorian Taylor. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.