Add Filesystem Storage Status in UI

[ypatios]

As an OC admin, I want to have the overview of the available storage capacity of the system. I want to be visibly warned, if it becomes critically low.

Suggested implementation (mockup):

---

Note for possible implementation:

The filesystem status is already available through the files/storage REST endpoint:
https://<your-oc-server>/files/storage

{
  "summary": "Usable space 220 Gb out of 1024 Gb (21%)",
  "usable": 236265144320,
  "size": 1099511627776,
  "used": 9374792153
}

[mtneug]

There is currently opencast/opencast#7376 which will add storage usage data to the Prometheus metrics endpoint. As you can see there, Opencast has various storage locations. So it's not as simple as a single percentage value.

[ypatios]

There is currently opencast/opencast#7376 which will add storage usage data to the Prometheus metrics endpoint. As you can see there, Opencast has various storage locations. So it's not as simple as a single percentage value.

oh thanks for that @mtneug :)
2 quick points:

1. on our system where we use cephFS for shared storage (w/ oc-nodes, wowza...), files/storage seems to return the overall capacity (not only eg /workspace).
2. in case you meant that one should use the prometheus metrics feature in order to get the overview of the system storage, I have to disagree. I think that it should be a basic feature that's always there per default — simple and easy as described above :)

but maybe I'm missing something idk... ?

[mtneug]

1. on our system where we use cephFS for shared storage (w/ oc-nodes, wowza...), files/storage seems to return the overall capacity (not only eg /workspace).
2. in case you meant that one should use the prometheus metrics feature in order to get the overview of the system storage, I have to disagree. I think that it should be a basic feature that's always there per default — simple and easy as described above :)

I'm not against including it in the UIs. However, adopters sometimes mount different disks/volumes for different Opencast storage locations. Heck, there is even the possibility to split the asset manager over multiple disks/volumes (never liked that idea). Thus, a single percentage value is not always possible.

[ypatios]

I'm not against including it in the UIs. However, adopters sometimes mount different disks/volumes for different Opencast storage locations. Heck, there is even the possibility to split the asset manager over multiple disks/volumes (never liked that idea). Thus, a single percentage value is not always possible.

OK. I understand that getting the correct data here is potentially not as simple as one GET. This should be taken into account when implementing this feature, for sure.
Do you see any conflicts/overlaps/redundancies between the 2 issues (possible merge?) or can they be kept (and implemented) separately in your opinion @mtneug ?

[mtneug]

These should be separate in the exposed APIs, i.e. Admin UI API and Prometheus metrics API, but could use the same internal Java APIs.

[TurRil]

Just for information, we have a fairly simple setup:
/opt/opencast - Application (19.x)
/data/opencast/archive - Storage for snapshots (~194 000)
/data/opencast/distribution - Storage for publications

And the https://<your-oc-server>/files/storage endpoint timeouts before it actually returns any values (after 15 min). The fetching of the size of the various folders might need some reviewing.

[ypatios]

Just for information, we have a fairly simple setup: /opt/opencast - Application (19.x) /data/opencast/archive - Storage for snapshots (~194 000) /data/opencast/distribution - Storage for publications

And the https://<your-oc-server>/files/storage endpoint timeouts before it actually returns any values (after 15 min). The fetching of the size of the various folders might need some reviewing.

in our case, prod took about 1' to respond (~35k events). def not ideal.
point noted! thanks @TurRil

[JamesUoM]

This is complicated by cloud storage being essentially infinite.
S3 is object storage, not a filesystem
Archive, distribution could be directories, not separate disks
Running the equivalent of du -s could be resource-intensive.

[mtneug]

Oh boy. Yes, we shouldn't run du -s or equivalent. That could take ages. Also, not sure if that's very helpful when you are concerned with filling up the disk. You want to know how full the disk is and not how much different folders contribute to the fullness 😅