Hardlinks for Duplicates, to Keep the Duplicates

[eduncan911]

As mentioned in Discord, this feature is to provide another option in the duplicate-in-other-playlists functionality. Currently, the logic is to skip over the video if it exists in another playlist (of the same channel).

This feature is to add the ability to create HARDLINKs instead, pointing to the original video. The idea is to keep the video showing up in both the all "Videos" standard playlist, as well as all other playlists that video is listed in (within the same channel, for now).

In this proposal:

• all videos would be pulled into a single S00.Videos playlist normally.
• for each Sxx.Playlist, we would create a HARDLINK for the video/audio, back to the original S00.Videos video. No videos will ever exist in the playlists in this configuration (see below as to why).
• the HARDLINK would be renamed to match the Sxx numbering concept, at a minimal.
• for subscriptions that point straight to a playlist URL only, ignore the Hardlink option if set.

To figure out:

1. How do we want to handle videos listed in a playlist, that belong to another channel?
2. What if we aren't pulling in that other channel?
3. What if we are pulling in that other channel?
4. I don't know if it makes sense to implement this Hardlink Duplicates feature, without the example in Add unique_indexer plugin to create indices for unique values #767 that shows how to possibly pull in all playlists automagically. Need others to flush out use-cases for "By Date" and what this would do?

Implementation:

• new `HARDLINK_DUPLICATES_IN_PLAYLISTS config option (defaults to False for backwards compatibility).
• (for @jmbannon to point us to the relevant duplicating/skipping code).
• possibly requires Add unique_indexer plugin to create indices for unique values #767 example plugin for automating playlist downloads, with a new Output format.

FAQs:

Why a hardlink, and not a symbolic link? Because Plex and Emby (and possibly others) do not follow symlinks. Whereas with a hardlink, they have no idea! :)

Why only hardlinks in all playlists for a channel? The reason is because channel authors have free reign to organize, format, rename, and manipulate playlists at will - and they do. I have several subscriptions of authors that spend hours keeping an extremely clean Playlist organizations... but there are also several channels that the author is just, well, sloppy in the playlist edits. By hard-coding a rule in this setup of hardlink, it means one could simply wipe-out the playlists and let them re-populate to pull in all the updates. Because it's nearly impossible to track and update playlists, as I've seem them completely deleted and new playlists created in the same name (different IDs) and all sorts of changes over the years.

Windows doesn't support links! Actually, Windows has supported Hardlinks since the Vista era (and I've used them plenty).

[eduncan911]

Do we keep a list of IDs and where they are written? something that could be looked up easily to reference, even for another channel?

Because for 1, 2 and 3 above, we could only keep 1 copy, the first one it pulls down, and hardlink all others we run across. Tagging and other metadata would be off. but I think that's the price to pay.

[jmbannon]

Yes, ytdl-sub maintains a basic archive for each subscription (see the hidden file prefixed with . in each subscription's directory). The rough idea I have in my head in how this could work is 1) see if it exists in the archive, and 2) make a 'stub' which will only run particular plugins (NFO, file naming, etc).

One issue I see with hardlinks is, for Plex specifically, it reads metadata from the video file's container. Hardlink'ing a video for multiple seasons would outright not work for Plex.

The second issue that needs testing is, with Jellyfin/Kodi presets, we write metadata to both the video container and NFO file. I'm not sure which takes priority when ingesting into the player. Worst case, we would need to disable video container metadata to force NFO scraping. The con here is it introduces incompatibility with Plex, which does not support NFOs.

Not trying to shoot down this idea -- I think it would be a great addition to the TV Show Collection presets. Want to get all of the incompatibles/risks out in the open first to see how we could approach it.

[eduncan911]

Want to get all of the incompatibles/risks out in the open first to see how we could approach it.

And that's why it'll remain an optional plugin, with caveats (we no longer use my lifetime Plex, after i found the goodness of Emby). :)