can URLs in index/table of contents files expire?

[nmaylevc]

It is OK to have URLs in the table of contents file that specify in_network_files, but that expire?

For example, a URL that looked like: ..in-network-rates.json.gz?&Expires=1661104969&Signature=DX8ihY...
and that could only be read for some period of time after downloading the table of contents file.

Various CDNs and can produce temporary URLs that only last a few minutes or a few hours. I would hope that any URL that is included in a TiC file should be able to be read at some later date within the 30 day window.

Expiring URLs would seem to be in conflict with the stated goal of public discoverability: "These machine-readable files must be made available to the public without restrictions that would impede the re-use of that information."

[shaselton-usds]

Producers of the files can add whatever querystrings they wish if they think it'll help both themselves and consumers for any reason.

Those wishing to consume the files should do so in real time as they are updated. Leveraging basic approaches such as ETags and expire headers is advised (not just with this project, but with most anything).

Expiring URLs is not in conflict with the files being freely available -- if that was the case then a URL could never expire if they followed that logic. There is a mismatch in understanding of accessibility and timing here.

[nmayle]

While I understand what you are saying, the table-of-contents files are not the two files mandated by the rule; they are just part of the data that should be in the files. As I understand, the TOC files were intended to help the insurers by sharing information across plans. If we go back to the rule, the in-network and out-of-network data files are the ones that need to be available.

The data files are supposed to be discoverable as per the specification on GitHub ("These machine-readable files must be made available to the public”) and "made available to the public without restrictions that would impede the re-use of that information.” Making the actual files only linked via the TOC file is already a barrier to the data files being discovered. Adding expiration to the urls in the TOC file is clearly a barrier and clearly is a “restriction that would impeded the reuse”; if I can’t readily download the files, then how can I use them.

If the links in the table-of-contents files are only live (valid) for a few minutes, is that really accessible?
How about if they are live for so short a time one can’t even download the whole file, say one minute?
Somewhere between permanent and enough time to download all of the linked files would seem to be accessible and shorter would not be.

[nmaylevc]

Scott, While I understand what you are saying, the table-of-contents files are not the two files mandated by the rule; they are just part of the data that should be in the files. As I understand, the TOC files were intended to help the insurers by sharing information across plans. If we got back to the rule, the in-network and out-of-network data files are the ones that need to be available. The data files are supposed to be discoverable as per the specification on GitHub ("These machine-readable files must be made available to the public”) and "made available to the public without restrictions that would impede the re-use of that information.” Making the actual files only linked via the TOC file is already a barrier to the data files being discovered. Adding expiration to the urls in the TOC file is clearly a barrier and clearly is a “restriction that would impeded the reuse”; if I can’t readily download the files, then how can I use them. If the links in the table-of-contents files are only live (valid) for a few minutes, is that really accessible? How about if they are live for so short a time one can’t even download the whole file, say one minute? Somewhere between permanent and enough time to download all of the linked files would seem to be accessible and shorter would not be. All the best, Neil "on their website where the machine-readable files would be readily accessible by the intended users"

…

On Jun 21, 2022, at 5:11 PM, scott haselton ***@***.***> wrote: Producers of the files can add whatever querystrings they wish if they think it'll help both themselves and consumers for any reason. Those wishing to consume the files should do so in real time as they are updated. Leveraging basic approaches such as ETags and expire headers is advised (not just with this project, but with most anything). Expiring URLs is not in conflict with the files being freely available -- if that was the case then a URL could never expire if they followed that logic. There is a mismatch in understanding of accessibility and timing here. — Reply to this email directly, view it on GitHub <#508 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AV3U2BMZCV6MQDUFJ7UTXB3VQIVZRANCNFSM5YVXRHMQ>. You are receiving this because you authored the thread.

[shaselton-usds]

If you have complaints, they can be submitted here.