Application Error for bigger files

[sylvanr]

Hey,

This might be a duplicate of #11 but with less behind-the-scenes insight:
This validator is blisteringly fast for smaller datasets compared to the http://validator.iatistandard.org/! From 41.28 seconds down to less than one second. Kudos!

However when uploading a larger file (43MB - "USAID Communications, Utilities, and Rental Payments"), there's an issue. Understandably, the upload process takes longer (~17s).

But alas, as mentioned in #11 , we are faced with:

Application error
An error occurred in the application and your page could not be served. If you are the application owner, check your logs for details. You can do this from the Heroku CLI with the command
heroku logs --tail

Is this something that could be solved?
This is partially related to #34, as "live validation" could be a good reason to consider using this validator over the official version.

Note: We can also move this into a comment for the mentioned ticket. Created a new one for clarity.

[andylolz]

Thanks for this, @sylvanr! No problem re. creating new ticket. If in doubt, new ticket is always best!

However when uploading a larger file (43MB - "USAID Communications, Utilities, and Rental Payments")

I’m not able to find this dataset… Could you provide a link to its registry page? I’d just like to recreate the problem before I go further, just so I can check the logs and see if this looks like something easily fixable, or something that might require a bit more investigation.

[sylvanr]

Yes of course, see https://iatiregistry.org/publisher/usaid?q=worldwide&sort=metadata_modified+desc. For reference, this is the dataset with the second-most activities on the registry (as far as I can see from the datastore).

[andylolz]

Nice, okay. So I tested with that dataset and checked the logs, and I’m seeing:

2021-06-10T14:42:44.006911+00:00 heroku[web.1]: Process running mem=805M(157.4%)
2021-06-10T14:42:44.024259+00:00 heroku[web.1]: Error R14 (Memory quota exceeded)
2021-06-10T14:42:55.799707+00:00 heroku[router]: at=error code=H12 desc="Request timeout" method=GET path="/validate/6de875a6-f3d8-4853-9191-4a0cbfeb5864" host=iati-validator.herokuapp.com request_id=c01c9d5f-d76d-4bd8-bdbe-383789c635b8 fwd="213.205.198.161" dyno=web.1 connect=0ms service=30000ms status=503 bytes=0 protocol=https
2021-06-10T14:42:56.166409+00:00 app[web.1]: [2021-06-10 14:42:56 +0000] [4] [CRITICAL] WORKER TIMEOUT (pid:15)
2021-06-10T14:42:56.253109+00:00 app[web.1]: [2021-06-10 14:42:56 +0000] [15] [INFO] Worker exiting (pid: 15)

So it looks the same as #11. I’ll have to take a look into the cause of this!