[Feature]: how to scale the GUI usage of > 100 harvests (definitions/crawls) and > 10000 jobs

[tuehlarsen]

What change would you like to see?

In the Current NetarchiveSuite GUI we have a well working and scalable GUI to handle >100 crawls/harvestdefinitions and >100K jobs for the last 20 years - actually the GUI handles >100 (incl. historical inactive and active) definitions and more 450K jobs. The Browsertrix GUI is nightmare when you have more than 10K jobs and >50 crawldefinitions.
Just to give you an idea how it could be made. Here is some screendumps from NetarchiveSuite:

Context

The current Browsertrix GUI is not very usable when you have a lot of Harvests/crawls and 1000's of jobs.
It is very diffecult to get an overview and to find a harvest or a given job and it requires many, many clicks and paging forward and backward e.g we need to fuzzy search fast by e.g. Crawl Name, Crawl_id, wacz file name and not only in Archived Items but also in Crawl workflows

e.g. In our local installation without any ongoing crawls having more than 500 (archived) crawl list "Crawl workflows" as well as "Archived Items" lags a few sec's and if you try to search for "tv2" in "Crawl workflows" you get no results but if you do the same in Archived Items and press enter then you get some matches after some sec's. We have tried to give the Mongo dbs more RAM and CPU but it still lags some secs.