Table of Contents File Suggestion

[brbucher]

Since the allowed amounts and in network rates files will be coming from different sources and in some cases be generated by separate entities, implementing the table of contents file introduces dependencies between the two other files that otherwise don't exist.
To solve this dependency issue, could we separate the table of contents file for the allowed amounts and in network rates files so there would be a table of contents for each?

The naming convention from here
Could be updated to:
_in_net_index.
and
_allowed_index.

[NigelEllisBSC]

The schema notes say it should be a fully qualified domain name for the file and gives an example of a JSON file. However if we do that we lose our protection of ensuring the downloader is a human and not a Bot - i.e. they can mine the URLs from the ToC file. Can we supply a page URL for another page that then requires the user to validate they are human again before downloading the related JSON file? Sorry if this is in the wrong conversation.

[rajkumarsowmy]

whats your thoughts on

Robots.txt
To allow for search engine discoverability, neither a robots.txt file nor meta tag on the page where the files are hosted will have rules such that give instructions to web crawlers to not index the page.

This is typically follows the format of or for a robots.txt file using the Disallow directive.

[NigelEllisBSC]

That works for web crawlers but once a user has discovered the URL by downloading the ToC file what is to stop them from using an automated solution to download the files in the future?