An easy solution could be to have the following file structure:

storage/plugins/latest (symlink)
storage/plugins/[checksum of package.json]

So the latest folder is always stable, and the symlink will change after update. Any opinions, @Chocobozzz ?

I'm sorry but I'm not sure I want to maintain this particular use case. Plugin installation/updates should be quite fast (a few seconds). If it takes 15-60 minutes on your system, I don't think s3fs is the appropriate tool to store PeerTube server files (including plugins).

I know s3fs is a bad solution for such case, and we will switch do a disk. But even a few seconds is bad for us, especially if something goes wrong with the update and the plugin will be unavailable for a longer time.

The use case is that we've a plugin that exposes an API that is critical in some apps that we're developing.

We've left s3fs now, but still it's not ideal that the plugin gets uninstalled during updates. With the solution below we should be able to skip the decache functionality used today.

An easy solution could be to have the following file structure:

storage/plugins/latest (symlink)
storage/plugins/[checksum of package.json]

So the latest folder is always stable, and the symlink will change after update.

#6443 is a bit hacky and also leads to some kind of memory leak.

What about implementing plugins as micro services?

Pros:

• Plugins won't be able to interrupt Peertube core functionality.
• Updates without downtime.
• Update failures won't leave Peertube in a state where the plugin is uninstalled.
• Better performance by not needing I/O operations after installation.
• More secure since it can be ran in an isolated environment.
• No need to clear require cache.
• Easy to do smoke tests on plugins by running something like npx @peertube/plugin-tools test to just start the plugin server and verify nothing breaks.

Cons:

• Will break eventual plugins that relies on either manipulating object references or calling incoming parameter methods (i.e. express.Request/Response).

Plugin server side implementation

import { registerPlugin } from '@peertube/plugin-tools'

const register = (options: RegisterServerOptions) => {
   ...
}

const unregister = () => {
  ...
}

registerPlugin({ register, unregister })

export { register, unregister } // Backward compatibility

registerPlugin will start a server that communicates with Peertube. This server will also serve the client files to Peertube.

Plugin client side implementation

Same as today.

Peertube implementation

A plugin is installed by download it's npm package and then spawn a process that will run the plugin (i.e. start the plugin server). The plugin servers port and hostname (if plugins are ran remotely) is stored in the database. When it's up and running it will communicate to Peertube its ready to accept connections.

Most of the current plugin functionality should work as today, I'm a bit unsure of some parts though ("Video transcoding manager API" for example).

gRPC, HTTP or quic could be used as protocol. Neither node, bun or deno has HTTP/3- or Quic-support though.

@Chocobozzz Any thoughts? Out of scope or something worth digging deeper into?

It's a good idea, and seems to be the way Ghost "integrations" work where plugins are not loaded in core but uses the ghost REST API: https://ghost.org/help/integrations/

I think external integration is a better choice than internal plugins because they are safer, can be used by a horizontally scaled server, cannot crash the main server etc. But they require more work (integration side and on peertube side) and are less "powerful", it's the reason why we chose to develop internal hooks.

I'm not against to develop webhooks and external JS library that would use the REST API so the server can spawn remote integrations (in a process or maybe a worker thread?). But unfortunately we clearly won't have time to develop, maintain and document such system in the near future