append terminating forward slash ' / ' to CR_URL

[pmusser]

Expected Behavior

In the ocw_oer_export.csv file, entries in CR_URL end in a slash (e.g. https://ocw.mit.edu/courses/11-165-infrastructure-and-energy-technology-challenges-fall-2011/ ) since that's the canonical URL.

Current Behavior

In the ocw_oer_export.csv file, entries in CR_URL do not end in a slash (e.g. https://ocw.mit.edu/courses/11-165-infrastructure-and-energy-technology-challenges-fall-2011 ). As a result, when our servers check these URLs' HTTP status code they return a 301 redirecting to the canonical URL in 'Expected Behavior.'

We have adjusted the URL's on OER Commons to point to the canonical url that terminates with the slash to address this. However, when importing resources into OER Commons using a CSV, the URL is also used as a primary key to determine if the URL already exists. Because there is a mismatch between the csv URL and the canonical URL, OER Commons would create a duplicate record (which is less than ideal).

Steps to Reproduce

1. ocw_oer_export.csv is generated
2. user downloads the csv and reviews entries in CR_URL, finding that the urls do not end with a forward slash /

Possible Solution

Not sure if it's an issue in the code or how MIT Open Learning API returns results. If code issue, maybe do a quick check to see if a / is at the end of the entry for CR_URL. If not, append one.

If an artefact from the API response, amend the code at the API side?

Additional Details

For what it's worth, on OER Commons side we're currently checking to see if we can refactor our automated URL checker so that it normalizes URL's when they're added using CSV import. Currently, our system does check URL's for redirects and other HTTP status codes when added individually by users through the front end interface (https://oercommons.org/courses/add) and will automatically repopulate the CR_URL field with the results of any redirects. This has not been implemented for the CSV checker yet though, likely because it would make our server appear as if it was trying to DOS any domains that show up lots of times.

[pdpinch]

@ibrahimjaved12 should we fix this here, or do you think it should be addressed upstream, in the learn API?

[ibrahimjaved12]

@pdpinch I think this should be adjusted in the Learn API. Left a message here: https://mitodl.slack.com/archives/C03K5HYGPT9/p1733138501817499