[Bug] Error when uploading files with & in filename

[atdservicebot]

See also:

• [Bug] Error when uploading files from G drive #12000

What application are you using?

Moped

Describe the problem.

Error: Cannot retrieve file signature for file 'William Cannon & Bannockburn 3-2-23.pdf'. Please reload your page and try again or contact the Data & Technology Services department. Feedback message: 403 -

Website Address

https://mobility.austin.gov/moped/projects/218?tab=files

Internet Browser: Chrome

Requested By
Billy B.

Attachment (65.28kb)

Request ID: DTS23-106905

[ChristinaTremel]

@johnclary assigning this to you because this is an error Billy is receiving in Moped!

[johnclary]

thanks @ChristinaTremel! I was unable to reproduce the issue on Friday. I'm waiting to hear back from Billy.

[johnclary]

this is caused by the & character in the filename. i can reproduce this in staging and prod, but not by running the Moped API file name validations locally. my suspicion is that some layer between the inbound moped request and flask is not handling the character encoding correctly. i'm going to open a PR to get better logging out of the Moped api.

[johnclary]

this appears to be a known issue with Zappa (#1227).

to recap exactly what's happening:

1. In Moped, try to upload a file with a & in the filename. e.g.: 1131_Howard & Cantarra Plan Sheets.pdf
2. You should get the red alert/error message as shown here.
3. Through your browser console, observe the GET request query string sent to the Moped API:
   file=1131_Howard%20%26%20Cantarra%20Plan%20Sheets.pdf&project_id=1769
   The decoded query string:

file: 1131_Howard & Cantarra Plan Sheets.pdf
project_id: 1769`

4. And observer the GET response payload

{
    "filename": "1131_Howard ",
    "message": "Invalid file name",
    "status": "error"
}

As you can see, the filename that was handled by the API (1131_Howard) does not match the filename that was sent, and is in turn causing this is_valid_filename test to fail.

I see two workarounds:

1. Strip out the & from filenames in Moped Editor before sending them to the API. As is, we already transform the filenames before saving the files in S3, so I don't think users should notice this or have a problem with it. I estimate that as a 2.

2. In the Moped API, change the files/request-signature endpoint to receive a POST request instead of a GET request. That way we can send a JSON request body that does does not need to be URL encoded. I estimate this as a...5?