Skip to content

Latest commit

 

History

History
21 lines (17 loc) · 1019 Bytes

README.md

File metadata and controls

21 lines (17 loc) · 1019 Bytes

Kit

Build Status Crate Coverage

Crates:

  • countio: The wrapper struct to enable byte counting for std::io::Read and std::io::Write and its asynchronous variants from futures and tokio.
  • robotxt: The implementation of the Robots.txt (or URL exclusion) protocol with the support of crawl-delay, sitemap and universal match extensions.
  • sitemapo: The implementation of the Sitemap (or URL inclusion) protocol with the support of txt, xml formats and video, image, and news extensions.