Admin commands for gloss videos #1398

susanodd · 2024-11-26T14:25:35Z

COMMANDS AND FILTERS FOR GLOSS VIDEO ADMIN

FILTERS

- By NME Video

All
True
False

By Perspective Video

All
True
False

By Filename Correct

All
True
False

By Backup Video

All
True
False

COMMANDS (on selection)

- Rename video files to match type

-- The code checks selected backup and normal video files and renames them accordingly
-- This takes care of old "bak bak" sequences and special 7-char Django characters appearing between the ID and extension.
-- It also makes sure the video extension matches the type of the video file. Previously, backup files with bak bak sequences did not have any video extension.

Set incorrect NME/Perspective filenames to empty string
-- The code effectively erases the selected objects filename from the video object
-- The empty filename shows up in the admin table
-- Once set to empty, it is possible to delete the incorrect objects without deleting the normal video file
-- The correct normal gloss video has not been altered, because the command merely erased the string
-- other code has been modified to handle the possibility that a videofile object may have no path [str(videofile) may be empty]
Move selected backups to DELETED_FILES_FOLDER location
Renumber selected backups

Original setup NOVEMBER 2024 is below. The code has evolved from what it originally was.

Goal: from a selected set of GlossVideo objects in the admin, delete the video file and the object if the version is a backup.
Reorder the remaining backup videos for the glosses in the set.

@vanlummelhuizen if you're working today on Signbank, the new "video admin" command merely prints what it is doing, it does not actually change anything.
I added a rename to it to get the correct video file extension on the backup file.

Perhaps the command is a bit weird now. Or perhaps it should be split into multiple commands.

The command needs to operate on a selection of GlossVideo objects.
So it does what it's supposed to (although commented out and doesn't), and for the "remaining" objects that were not deleted, it renames them and fixes the version number.

Since the command currently "doesn't do anything other than print stuff" it is harmless to try on the real server.

after selection was deleted, version numbers adjusted for new set. Change extension of backup file if it has the wrong video extension for its format.

vanlummelhuizen · 2024-11-28T08:11:09Z

The glosses need to be retrieved from the selected GlossVideo objects before anything is deleted.

You are right.

change file extension command, match file type

plus filter for wrongly named files that do not match the type of object.

susanodd · 2025-02-03T15:00:43Z

I added high priority to this issue because there are still lots of NME and Perspective videos that point to the normal video, even after they were re-uploaded.

This issue has a filter on Perspective video, NME video, Backup video, as well as "wrong filename".

It's not possible to "witness" whether the videos are fixed or not without filters such as given here.

It seems the "package" is being used to obtain the video paths to see if the path is correct.

But the API and also the Signbank interface cannot delete the wrong things because that deletes the normal videos that are pointed to. (#1502)

In the Admin, some of the normal video objects (on the production server) indeed do not point to any file (no timestamp in the admin means no file exists). Most likely that file was deleted by deleting a perspective or NME video that pointed to it.

Woseseltops

I like this initiative, although it's not 100% clear to me how the commands should be used. Perhaps you can create a page on the wiki for it, along with some examples of what will happen? Let me know if you have something, I will try to organize it for you :)

About code style, I have quite a few smaller stylistic things this time. Don't get discouraged by this! It's all small stuff.

Woseseltops · 2025-02-14T15:36:24Z

signbank/video/convertvideo.py

+    if file_extension not in ACCEPTABLE_VIDEO_EXTENSIONS:
+        # some other extension is present in the filename
+        file_extension = '.mp4'


Wouldn't returning None and handling that outside this function be cleaner? I feel it's a bit weird that you cannot trust that an mp4 really is an mp4 . Or perhaps even throwing an error?

THE FUNCTION HAS BEEN REWRITTEN WITH COMMENTS

For this, no it's not true. Because the original "bak bak" format of backup videos did not include any video type file extension, it was not maintained whether this was an mp4 file. We assumed it was mp4 because the code used "ensure_mp4". But that was turned off at the request of UvA. The newer "bakNNN" includes the video file type as an extension before the backup extension, like .mp4.bakNNN.
Context of video file extension maybe being wrong

The "video type" filter was added because some are not actually mp4, but instead something else.
The CNGT script that converts the videos was broken. Jetske discovered that the frame rate of the original video and the converted video did not match. For a while it was hard-coded as mp4 in the filename, because it was supposedly being coverted.

The older m4v files are already mp4 format, so lots were merely just renamed as mp4.

Woseseltops · 2025-02-14T15:38:21Z

signbank/video/convertvideo.py

+    if not video_file_full_path:
+        return ''
+    if 'glossvideo' not in video_file_full_path:
+        return ''


Wouldn't it be cleaner to return None or throwing an error?

For this issue, this function is being called inside either a queryset loop, or as a filter in the Admin.

The caller can test whether it's empty, instead catching an error.

Woseseltops · 2025-02-14T15:42:05Z

signbank/video/models.py

        except (OSError, PermissionError):
            if settings.DEBUG_VIDEOS:
-                print('delete_files exception GlossVideo OSError, PermissionError: ', str(self.videofile))
+                print('delete_files exception GlossVideoNME OSError, PermissionError: ', file_system_path)


Perhaps this function can return a boolean to indicate success? Seems weird to me that it just returns silently or puts something in the log if something goes wrong, but the code where you call this function wouldn't know.

Woseseltops · 2025-02-14T15:43:37Z

signbank/video/models.py

@@ -1036,7 +1051,7 @@ def delete_files(self):
            return
        try:
            os.unlink(file_system_path)
-        except OSError:
+        except (PermissionError, OSError):


Perhaps this function can return a boolean to indicate success? Seems weird to me that it just returns silently or puts something in the log if something goes wrong, but the code where you call this function wouldn't know.

THIS HAS BEEN MADE TO RETURN A BOOLEAN

Woseseltops · 2025-02-14T15:45:57Z

signbank/video/admin.py

+    try:
+        if m := re.search(r".+-(\d+)_(nme_\d+|nme_\d+_left|nme_\d+_right)$", filename_without_extension):
+            return True
+        return False


return m := re.search(r".+-(\d+)_(nme_\d+|nme_\d+_left|nme_\d+_right)$", filename_without_extension):

Same for the functions below this one.

Comments about code style and Django implementation

I'll go ahead and change it for readability. But I think that violates type checking and implementation efficiently (see below)

Nope, m := is a syntax error.

It returns None if there isn't a match, or a match object if there is.
But that's not of type Boolean.

Isn't what you suggest violating "type checking" guidelines?
Also, do we want to use memory for storing an object in the return value when it's not necessary?

I understand the comment, but is this wise regarding the compiler?

It can just throw away the search result instead of send it to the caller. (Original code)

Just FYI: if you want to fully match a string, you could use re.fullmatch (see also https://docs.python.org/3/library/re.html#search-vs-match). And if you want True if there is a match and False there is none, cast it to a boolean with bool(). Then it becomes:

return bool(re.fullmatch(r".+-(\d+)_(nme_\d+|nme_\d+_left|nme_\d+_right)$", filename_without_extension))

Again just FYI, the changes @susanodd made per review are clean enough.

Woseseltops · 2025-02-14T15:57:46Z

signbank/video/admin.py

+            destination = os.path.join(WRITABLE_FOLDER, GLOSS_VIDEO_DIRECTORY,
+                                       dataset_dir, two_letter_dir, desired_filename)
+            desired_relative_path = os.path.join(GLOSS_VIDEO_DIRECTORY,
+                                                 dataset_dir, two_letter_dir, desired_filename)


Wouldn't it make more sense to first determine the relative path, and then use that as the latter part of the absolute path?

I feel like we have similar code in other places... perhaps better to reuse other code that determines where videos should be stored?

Woseseltops · 2025-02-14T16:00:59Z

signbank/video/admin.py

+
+
+@admin.action(description="Remove selected backups")
+def remove_backups(modeladmin, request, queryset):


Alternative suggestion: instead of deleting, move them to an archive location that we can manually empty later.

Woseseltops · 2025-02-14T16:01:45Z

signbank/video/admin.py

+        relative_path = str(obj.videofile)
+        video_file_full_path = os.path.join(WRITABLE_FOLDER, str(obj.videofile))
+        if os.path.exists(video_file_full_path):
+            # remove the video file so the GlossVideo object can be deleted


I'd prefer this as guard clause

Woseseltops · 2025-02-14T16:07:05Z

signbank/video/admin.py

+    # allow to erase the filename from an object if it has the wrong format and it is for a subclass video
+    # this is a patch for repairing doubly linked files


Let's make this a doscstring.

Woseseltops · 2025-02-14T16:08:15Z

signbank/video/admin.py

+    def video_type(self, obj=None):
+        # if the file exists, this will display its timestamp in the list view
+        if obj is None or not str(obj.videofile):
+            return ""
+        import os
+        video_file_full_path = os.path.join(WRITABLE_FOLDER, str(obj.videofile))
+        if os.path.exists(video_file_full_path):
+            return video_file_type_extension(video_file_full_path)
+        else:
+            return ""
+


Here too I feel it would make more sense to return None or raise an error

This function is a column "VIDEO TYPE" in the Admin. It makes no sense to return an error.

Modifed caller to check not None since it's not a Boolean anymore.

…ideo Amin.

…mmands

susanodd · 2025-02-18T11:29:15Z

Bullets

All of the code modifications have been made, major changes summarised below

Backup files moved to a garbage location DELETED_FILES_FOLDER; filenames are flattened to keep the location flat (this also works for non-Latin languages)
I revised as requested the "delete_files" function for NME videos and for Perspective videos so it returns a Boolean (this function wasn't from this issue, but suggest for in review.)
-- Because it now returns True or False, I made it so when the name of the file is not correct, it merely erases the name (by the by, that resolves issue Domino effect on video deletion #1502) It returns True in this case, but merely erased the file name from the object. The caller of this function, "reversion" or "pre_delete signal, removes the object itself.
Text for Video Admin Wiki

Changed typo in name of function Made delete_files return Boolean, print OS exception text in log Moved functions from admin.py to models.py so they can be shared by models Check on correct filename before deleting doubly linked file in normal delete

for symmetry. Also takes care of #1502 by checking if filename is correct before deleting

Renamed, added patterns, added comments, mp4 still last resort value

…dmin

…trash location Files renamed to include dataset and 2char folder at front of filename to keep trash flat.

…_path function.

Stored as a text file so it gets maintained in GitHub.

…n wiki. Added text describing filenames and storage structure. This provides context for the commands and filters.

#1397: Command to delete excess backup videos and reorder versions

591d97a

susanodd added the DRAFT label Nov 26, 2024

susanodd requested a review from vanlummelhuizen November 27, 2024 12:02

#1397, #1398: Added renaming of backup files to those that remain

c9608be

after selection was deleted, version numbers adjusted for new set. Change extension of backup file if it has the wrong video extension for its format.

This comment was marked as resolved.

Sign in to view

#1397, #1398: Split into two commands, refined code as per review

aa2ec41

susanodd changed the title ~~#1397: Command to delete excess backup videos and reorder versions~~ #1397: Commands to delete excess backup videos and reorder versions Nov 29, 2024

Merge branch 'master' into video_storage_commands

a87dd85

This comment was marked as outdated.

Sign in to view

#1397, #1398: Added filter on video file type mp4 true false

cae023a

change file extension command, match file type

susanodd mentioned this pull request Nov 30, 2024

Manage Datasets -> Manage Video Storage (functionality enhancements) #1406

Open

susanodd changed the title ~~#1397: Commands to delete excess backup videos and reorder versions~~ Commands to delete excess backup videos and reorder versions Nov 30, 2024

susanodd added 2 commits December 2, 2024 06:42

#1390: Import get two char dir from tools instead of defining.

f59a97f

Merge branch 'master' into video_storage_commands

19f2bbd

susanodd changed the title ~~Commands to delete excess backup videos and reorder versions~~ Admin commands to delete excess backup videos and reorder versions Dec 10, 2024

susanodd mentioned this pull request Dec 10, 2024

Reduce the number of identical get_two_letter_dir functions to one #1390

Closed

susanodd added 3 commits December 19, 2024 13:21

Merge branch 'master' into video_storage_commands

87e3ec8

Merge branch 'master' into video_storage_commands

260852d

#1398: Added video admin filters for NME, Perspective videos

c080da7

plus filter for wrongly named files that do not match the type of object.

susanodd added signbank-susan and removed DRAFT labels Jan 7, 2025

This comment was marked as outdated.

Sign in to view

susanodd requested a review from Woseseltops January 7, 2025 13:04

#1398: Expanded filters for video admin. Renaming files implemented.

ae58875

susanodd mentioned this pull request Jan 8, 2025

Videos randomly disappearing #1453

Open

susanodd changed the title ~~Admin commands to delete excess backup videos and reorder versions~~ Admin commands for gloss videos Jan 8, 2025

#1398: Video admin filter for backups, commands renumber, remove

e8e1569

Merge branch 'master' into video_storage_commands

eabba1f

susanodd added the high priority label Feb 3, 2025

susanodd added 3 commits February 6, 2025 11:25

Merge branch 'master' into video_storage_commands

1fba673

Merge branch 'master' into video_storage_commands

aca9782

Merge branch 'master' into video_storage_commands

8241c53

susanodd mentioned this pull request Feb 12, 2025

Domino effect on video deletion #1502

Open

susanodd added 2 commits February 12, 2025 14:06

Merge branch 'master' into video_storage_commands

fe7c47a

Merge branch 'master' into video_storage_commands

19be23f

Woseseltops requested changes Feb 14, 2025

View reviewed changes

susanodd added 6 commits February 17, 2025 10:27

1398: Per review, moved all imports to top.

722cbc8

#1398: Per review simplied function code.

b8bffb1

Modifed caller to check not None since it's not a Boolean anymore.

#1398: Per review: Added docstring documentation to Filters of GlossV…

0f2a0e4

…ideo Amin.

#1398: Per review: filename strings are f-strings. Doc-strings for co…

2b983e9

…mmands

#1398: Per review, added exceptions caught (not this issue)

e1aea91

#1398: Per review, guarded clauses (also fixed other places in file)

5174027

susanodd added 11 commits February 19, 2025 13:41

#1398: Per review - made other delete_files also return Boolean

3492a1f

for symmetry. Also takes care of #1502 by checking if filename is correct before deleting

Merge branch 'master' into video_storage_commands

89a4571

Merge branch 'master' into video_storage_commands

350a333

#1398: Per review: Rewrote video file extension helper function.

97d4861

Renamed, added patterns, added comments, mp4 still last resort value

Merge branch 'master' into video_storage_commands

e1cc632

#1398: Per review, added doc-strings to column methods of GlossVideoA…

3c68049

…dmin

#1398: Per review: Move admin-command selected backup video files to …

cac7166

…trash location Files renamed to include dataset and 2char folder at front of filename to keep trash flat.

#1398: Per review: Added docstring and fstring to new flattened_video…

7bbf95b

…_path function.

#1398: Per review: Gloss Video Admin Wiki Text File

c9e2a82

Stored as a text file so it gets maintained in GitHub.

#1398: Per review: Enhanced documentation in text file for video admi…

677a610

…n wiki. Added text describing filenames and storage structure. This provides context for the commands and filters.

susanodd requested a review from Woseseltops February 27, 2025 07:43

Merge branch 'master' into video_storage_commands

880cbfb

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Admin commands for gloss videos #1398

Admin commands for gloss videos #1398

susanodd commented Nov 26, 2024 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

vanlummelhuizen commented Nov 28, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

susanodd commented Feb 3, 2025 •

edited

Loading

Woseseltops left a comment •

edited

Loading

Woseseltops Feb 14, 2025

susanodd Feb 17, 2025 •

edited

Loading

Woseseltops Feb 14, 2025

susanodd Feb 17, 2025 •

edited

Loading

Woseseltops Feb 14, 2025

Woseseltops Feb 14, 2025

susanodd Feb 17, 2025 •

edited

Loading

Woseseltops Feb 14, 2025

susanodd Feb 17, 2025 •

edited

Loading

vanlummelhuizen Feb 17, 2025

Woseseltops Feb 14, 2025

Woseseltops Feb 14, 2025

Woseseltops Feb 14, 2025

Woseseltops Feb 14, 2025

Woseseltops Feb 14, 2025

susanodd Feb 17, 2025 •

edited

Loading

susanodd commented Feb 18, 2025 •

edited

Loading



		@admin.action(description="Remove selected backups")
		def remove_backups(modeladmin, request, queryset):

		# allow to erase the filename from an object if it has the wrong format and it is for a subclass video
		# this is a patch for repairing doubly linked files

Admin commands for gloss videos #1398

Are you sure you want to change the base?

Admin commands for gloss videos #1398

Conversation

susanodd commented Nov 26, 2024 • edited Loading

This comment was marked as resolved.

This comment was marked as resolved.

vanlummelhuizen commented Nov 28, 2024

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

susanodd commented Feb 3, 2025 • edited Loading

Woseseltops left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

susanodd Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

susanodd Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

susanodd Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

susanodd Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

susanodd Feb 17, 2025 • edited Loading

Choose a reason for hiding this comment

susanodd commented Feb 18, 2025 • edited Loading

susanodd commented Nov 26, 2024 •

edited

Loading

susanodd commented Feb 3, 2025 •

edited

Loading

Woseseltops left a comment •

edited

Loading

susanodd Feb 17, 2025 •

edited

Loading

susanodd Feb 17, 2025 •

edited

Loading

susanodd Feb 17, 2025 •

edited

Loading

susanodd Feb 17, 2025 •

edited

Loading

susanodd Feb 17, 2025 •

edited

Loading

susanodd commented Feb 18, 2025 •

edited

Loading