Merge branch 'master' into Documentation

Author: Brijeshthummar02

.github/workflows/generate_catalog_templates.yml

@@ -35,7 +35,7 @@ jobs:
           github-token: "${{ secrets.GITHUB_TOKEN }}"
           pull-request-number: ${{ steps.cpr.outputs.pull-request-number }}
 
-      - uses: pascalgn/[email protected].3
+      - uses: pascalgn/[email protected].4
         if: steps.cpr.outputs.pull-request-operation == 'created'
         env:
           GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

.github/workflows/sphinxbuild.yml

@@ -23,7 +23,7 @@ jobs:
       shell: bash
       run: tar czf /tmp/documentation.tar.gz -C user_manual/_build/html .
     - name: Upload static documentation
-      uses: actions/[email protected].0
+      uses: actions/[email protected].3
       with:
         name: User manual.zip
         path: "/tmp/documentation.tar.gz"
@@ -55,7 +55,7 @@ jobs:
       shell: bash
       run: tar czf /tmp/documentation.tar.gz -C developer_manual/_build/html/com .
     - name: Upload static documentation
-      uses: actions/[email protected].0
+      uses: actions/[email protected].3
       with:
         name: Developer manual.zip
         path: "/tmp/documentation.tar.gz"
@@ -75,7 +75,7 @@ jobs:
       shell: bash
       run: tar czf /tmp/documentation.tar.gz -C admin_manual/_build/html/com .
     - name: Upload static documentation
-      uses: actions/[email protected].0
+      uses: actions/[email protected].3
       with:
         name: Administration manual.zip
         path: "/tmp/documentation.tar.gz"

.github/workflows/transifex.yml

@@ -17,7 +17,7 @@ jobs:
     name: Auto-merge
     needs: approve
     steps:
-      - uses: pascalgn/[email protected].3
+      - uses: pascalgn/[email protected].4
         if: github.actor == 'transifex-integration[bot]'
         env:
           GITHUB_TOKEN: "${{ secrets.GITHUB_TOKEN }}"

admin_manual/ai/ai_as_a_service.rst

@@ -11,8 +11,16 @@ Installation
 
 In order to use these providers you will need to install the respective app from the app store:
 
-* ``integration_openai`` (With this application, you can also connect to a self-hosted LocalAI instance or to any service that implements an API similar to OpenAI, for example Plusserver or MistralAI.)
+* ``integration_openai``
 
 * ``integration_replicate``
 
 You can then add your API token and rate limits in the administration settings and set the providers live in the "Artificial intelligence" section of the admins settings.
+
+
+OpenAI integration
+------------------
+
+With this application, you can also connect to a self-hosted LocalAI or Ollama instance or to any service that implements an API similar enough to the OpenAI API, for example Plusserver or MistralAI.
+
+Do note however, that we test the Assistant tasks that this app implements only with OpenAI models and only against the OpenAI API, we thus cannot guarantee other models and APIs will work.

admin_manual/ai/app_assistant.rst

@@ -63,6 +63,20 @@ In order to make use of text processing features in the assistant, you will need
 * :ref:`llm2<ai-app-llm2>` - Runs open source AI language models locally on your own server hardware (Customer support available upon request)
 * *integration_openai* - Integrates with the OpenAI API to provide AI functionality from OpenAI servers  (Customer support available upon request; see :ref:`AI as a Service<ai-ai_as_a_service>`)
 
+These apps currently implement the following Assistant Tasks:
+
+* *Generate text* (Tested with OpenAI GPT-3.5 and Llama 3.1 8B)
+* *Summarize* (Tested with OpenAI GPT-3.5 and Llama 3.1 8B)
+* *Generate headline* (Tested with OpenAI GPT-3.5 and Llama 3.1 8B)
+* *Extract topics* (Tested with OpenAI GPT-3.5 and Llama 3.1 8B)
+
+Additionally, *integration_openai* also implements the following Assistant Tasks:
+
+* *Context write* (Tested with OpenAI GPT-3.5)
+* *Reformulate text* (Tested with OpenAI GPT-3.5)
+
+These tasks may work with other models, but we can give no guarantees.
+
 Text-To-Image
 ~~~~~~~~~~~~~
 
@@ -79,6 +93,7 @@ In order to make use of our special Context Chat feature, offering in-context in
 
 * :ref:`context_chat + context_chat_backend<ai-app-context_chat>` -  (Customer support available upon request)
 
+You will also need a text processing provider as specified above (ie. llm2 or integration_openai).
 
 Configuration
 -------------
@@ -161,16 +176,7 @@ This field is appended to the block of chat messages, i.e. attached after the me
 The number of latest messages to consider for generating the next message. This does not include the user instructions, which is always considered in addition to this. This value should be adjusted in case you are hitting the token limit in your conversations too often.
 The AI text generation provider should ideally handle the max token limit case.
 
-Improve AI processing throughput
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Most AI tasks will be run as part of the background job system in Nextcloud which only runs jobs every 5 minutes by default.
-To pick up scheduled jobs faster you can set up background job workers that process AI tasks as soon as they are scheduled:
-
-run the following occ commands a daemon (you can also spawn multiple, for parallel processing):
-
-.. code-block::
-
-   occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob'
+Improve AI task pickup speed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-Make sure to restart these daemons regularly, for example once a day, to make sure the daemon runs the latest code.
+See :ref:`the relevant section in AI Overview<ai-overview_improve-ai-task-pickup-speed>` for more information.

admin_manual/ai/app_context_chat.rst

@@ -47,18 +47,32 @@ Installation
 
 0. Make sure the :ref:`Nextcloud Assistant app<ai-app-assistant>` is installed
 1. :ref:`Install AppAPI and setup a Deploy Demon<ai-app_api>`
-2. Install the *context_chat_backend* ExApp via the "External Apps" admin page in Nextcloud
+2. Install the *context_chat_backend* ExApp via the "External Apps" admin page in Nextcloud, or by executing
+
+.. code-block::
+
+   occ app_api:app:register context_chat_backend
+
 3. Install the *context_chat* app via the "Apps" page in Nextcloud, or by executing
 
 .. code-block::
 
    occ app:enable context_chat
 
-4. Optionally, run two instances of this occ command for faster processing of requests:
+4. Install a text generation backend like *llm2* (via the "External Apps" page in Nextcloud) or *integration_openai* (via the "Apps" page in Nextcloud), or by executing
+
+.. code-block::
+
+   occ app_api:app:register llm2
+
+or
 
 .. code-block::
 
-   occ background-job:worker 'OC\TaskProcessing\SynchronousBackgroundJob'
+   occ app:enable integration_openai
+
+
+5. Optionally but recommended, setup background workers for faster pickup of tasks. See :ref:`the relevant section in AI Overview<ai-overview_improve-ai-task-pickup-speed>` for more information.
 
 **Note**: Both apps need to be installed and both major version and minor version of the two apps must match for the functionality to work (ie. "v1.3.4" and "v1.3.1"; but not "v1.3.4" and "v2.1.6"; and not "v1.3.4" and "v1.4.5"). Keep this in mind when updating.
 
@@ -69,15 +83,15 @@ Context chat will automatically load user data into the Vector DB using backgrou
 
 .. code-block::
 
-   occ background-job:worker 'OCA\ContextChat\BackgroundJobs\StorageCrawlJob'
+   set -e; while true; do sudo -u www-data occ background-job:worker -v -t 60 "OCA\ContextChat\BackgroundJobs\StorageCrawlJob"; done
 
 .. code-block::
 
-   occ background-job:worker 'OCA\ContextChat\BackgroundJobs\IndexerJob'
+   set -e; while true; do sudo -u www-data occ background-job:worker -v -t 60 "OCA\ContextChat\BackgroundJobs\IndexerJob"; done
 
 This will ensure that the necessary background jobs are run as often as possible: ``StorageCrawlJob`` will crawl Nextcloud storages and put files that it finds into a queue and ``IndexerJob`` will iterate over the queue and load the file content into the Vector DB.
 
-Make sure to restart these daemons regularly. For example once a day.
+See :ref:`the task speedup section in AI Overview<ai-overview_improve-ai-task-pickup-speed>` to know better ways to run these jobs.
 
 Scaling
 -------

admin_manual/ai/app_llm2.rst

@@ -6,10 +6,13 @@ App: Local large language model (llm2)
 
 The *llm2* app is one of the apps that provide text processing functionality using Large language models in Nextcloud and act as a text processing backend for the :ref:`Nextcloud Assistant app<ai-app-assistant>`, the *mail* app and :ref:`other apps making use of the core Text Processing API<tp-consumer-apps>`. The *llm2* app specifically runs only open source models and does so entirely on-premises. Nextcloud can provide customer support upon request, please talk to your account manager for the possibilities.
 
-This app uses `ctransformers <https://github.com/marella/ctransformers>`_ under the hood and is thus compatible with any model in *gguf* format. Output quality will differ depending on which model you use, we recommend the following models:
+This app uses `llama.cpp <https://github.com/abetlen/llama-cpp-python>`_ under the hood and is thus compatible with any model in *gguf* format.
 
-* `Llama3 8b Instruct <https://huggingface.co/QuantFactory/Meta-Llama-3-8B-Instruct-GGUF>`_ (reasonable quality; fast; good acclaim; multilingual output may not be optimal)
-* `Llama3 70B Instruct <https://huggingface.co/QuantFactory/Meta-Llama-3-70B-Instruct-GGUF>`_ (good quality; good acclaim; good multilingual output)
+However, we only test with Llama 3.1. Output quality will differ depending on which model you use and downstream tasks like summarization or Context Chat may not work on other models.
+We thus recommend the following models:
+
+* `Llama3.1 8b Instruct <https://huggingface.co/QuantFactory/Meta-Llama-3.1-8B-Instruct-GGUF>`_ (reasonable quality; fast; good acclaim; comes shipped with the app)
+* `Llama3.1 70B Instruct <https://huggingface.co/bartowski/Meta-Llama-3.1-70B-Instruct-GGUF>`_ (good quality; good acclaim)
 
 Multilinguality
 ---------------
@@ -27,6 +30,8 @@ Llama 3.1 `supports the following languages: <https://huggingface.co/meta-llama/
 * Hindi
 * Thai
 
+Note, that other languages may work as well, but only the above languages are guaranteed to work.
+
 Requirements
 ------------
 

admin_manual/ai/overview.rst

@@ -175,6 +175,82 @@ Apps can integrate their content with Context Chat to make it available for quer
 * *files*
 * `Analytics <https://apps.nextcloud.com/apps/analytics>`_
 
+.. _ai-overview_improve-ai-task-pickup-speed:
+
+Improve AI task pickup speed
+----------------------------
+
+Most AI tasks will be run as part of the background job system in Nextcloud which only runs jobs every 5 minutes by default.
+To pick up scheduled jobs faster you can set up background job workers that process AI tasks as soon as they are scheduled.
+If the PHP code or the Nextcloud settings values are changed while a worker is running, those changes won't be effective inside the runner. For that reason, the worker needs to be restarted regularly. It is done with a timeout of N seconds which means any changes to the settings or the code will be picked up after N seconds (worst case scenario). This timeout does not, in any way, affect the processing or the timeout of the AI tasks.
+
+Screen or tmux session
+^^^^^^^^^^^^^^^^^^^^^^
+
+Run the following occ command inside a screen or a tmux session, preferably 4 or more times for parallel processing of multiple requests by different or the same user (and as a requirement for some apps like context_chat).
+It would be best to run one command per screen session or per tmux window/pane to keep the logs visible and the worker easily restartable.
+
+.. code-block::
+
+   set -e; while true; do sudo -u www-data occ background-job:worker -v -t 60 "OC\TaskProcessing\SynchronousBackgroundJob"; done
+
+You may want to adjust the number of workers and the timeout (in seconds) to your needs.
+The logs of the worker can be checked by attaching to the screen or tmux session.
+
+Systemd service
+^^^^^^^^^^^^^^^
+
+1. Create a systemd service file in ``/etc/systemd/system/[email protected]`` with the following content:
+
+.. code-block::
+
+   [Unit]
+   Description=Nextcloud AI worker %i
+   After=network.target
+
+   [Service]
+   ExecStart=/opt/nextcloud-ai-worker/taskprocessing.sh %i
+   Restart=always
+
+   [Install]
+   WantedBy=multi-user.target
+
+2. Create a shell script in ``/opt/nextcloud-ai-worker/taskprocessing.sh`` with the following content and make sure to make it executable:
+
+.. code-block::
+
+   #!/bin/sh
+   echo "Starting Nextcloud AI Worker $1"
+   cd /path/to/nextcloud
+   sudo -u www-data php occ background-job:worker -t 60 'OC\TaskProcessing\SynchronousBackgroundJob'
+
+You may want to adjust the timeout to your needs (in seconds).
+
+3. Enable and start the service 4 or more times:
+
+.. code-block::
+
+   for i in {1..4}; do systemctl enable --now nextcloud-ai-worker@$i.service; done
+
+The status of the workers can be checked with (replace 1 with the worker number):
+
+.. code-block::
+
+   systemctl status [email protected]
+
+The list of workers can be checked with:
+
+.. code-block::
+
+   systemctl list-units --type=service | grep nextcloud-ai-worker
+
+The complete logs of the workers can be checked with (replace 1 with the worker number):
+
+.. code-block::
+
+   journalctl -xeu [email protected] -f
+
+
 Frequently Asked Questions
 --------------------------
 

admin_manual/installation/nginx-root.conf.sample

@@ -184,7 +184,7 @@ server {
         access_log off;     # Optional: Don't log access to assets
     }
 
-    location ~ \.woff2?$ {
+    location ~ \.(otf|woff2?)$ {
         try_files $uri /index.php$request_uri;
         expires 7d;         # Cache-Control policy borrowed from `.htaccess`
         access_log off;     # Optional: Don't log access to assets

admin_manual/installation/nginx-subdir.conf.sample

@@ -181,7 +181,7 @@ server {
             access_log off;     # Optional: Don't log access to assets
         }
 
-        location ~ \.woff2?$ {
+        location ~ \.(otf|woff2?)$ {
             try_files $uri /nextcloud/index.php$request_uri;
             expires 7d;         # Cache-Control policy borrowed from `.htaccess`
             access_log off;     # Optional: Don't log access to assets

admin_manual/occ_command.rst

@@ -514,7 +514,9 @@ A set of commands to create and manage addressbooks and calendars::
 
  dav
   dav:create-addressbook                 Create a dav addressbook
+  dav:list-addressbooks                  List all addressbooks of a user
   dav:create-calendar                    Create a dav calendar
+  dav:create-subscription                Create a dav calendar subscription
   dav:delete-calendar                    Delete a dav calendar
   dav:fix-missing-caldav-changes         Insert missing calendarchanges rows for existing events
   dav:list-calendars                     List all calendars of a user
@@ -536,6 +538,20 @@ This example creates a new calendar for molly::
 
 Molly will immediately see these in the Calendar and Contacts apps.
 
+The syntax for ``dav:create-subscription`` is 
+``dav:create-subscription [user] [name] [url] [optional color]``. This example creates the subscription subscription for the lunar
+calendar ``Lunar Calendar`` for the user molly::
+
+ sudo -u www-data php occ dav:create-subscription molly "Lunar Calendar" webcal://cantonbecker.com/astronomy-calendar/astrocal.ics
+
+Molly will immediately see this new subscription calendar in the Calendar app.
+
+Optionally, a color for the new subscription calendar can be passed as a HEX color code::
+  
+ sudo -u www-data php occ dav:create-subscription molly "Lunar Calendar" calendar webcal://cantonbecker.com/astronomy-calendar/astrocal.ics "#ff5733"
+
+If not set, the theming default color will be used.
+
 ``dav:delete-calendar [--birthday] [-f|--force] <uid> [<name>]`` deletes the
 calendar named ``name`` (or the birthday calendar if ``--birthday`` is
 specified) of the user ``uid``. You can use the force option ``-f`` or
@@ -549,11 +565,17 @@ This example will delete the birthday calendar of user molly::
 
  sudo -u www-data php occ dav:delete-calendar --birthday molly
 
-``dav:lists-calendars [user]`` will display a table listing the calendars for a given user.
+``dav:list-calendars [user]`` and ``dav:list-addressbooks [user]`` will display a
+table listing the calendars or addressbooks for a given user.
+
 This example will list all calendars for user annie::
 
  sudo -u www-data php occ dav:list-calendars annie
 
+This example will list all addressbooks for user annie::
+
+ sudo -u www-data php occ dav:list-addressbooks annie
+
 ``dav:dav:fix-missing-caldav-changes [user]`` tries to restore calendar sync changes when data in the calendarchanges table has been lost. If the user ID is omitted, the command runs for all users. This can take a while.
 
 ``dav::move-calendar [name] [sourceuid] [destinationuid]`` allows the admin

admin_manual/release_notes/upgrade_to_30.rst

@@ -8,6 +8,7 @@ System requirements
 * PHP 8.1 is now deprecated but still supported.
 * PHP 8.0 is no longer supported.
 * PostgreSQL 9.4 is no longer supported.
+* MariaDB 10.3 and 10.5 are no longer supported.
 
 Web server configuration
 ------------------------

admin_manual/windmill_workflows/index.rst

@@ -9,7 +9,7 @@ Installation
 
  * Install Windmill
 
-   * Either as a standalone install or via the Windmill External App in Nextcloud (see :ref:`External Apps<ai-app_api>`)
+   * Either as a standalone install or via the External App "Flow" in Nextcloud (see :ref:`External Apps<ai-app_api>`)
 
  * Enable the ``webhook_listeners`` app that comes with Nextcloud
 
@@ -41,6 +41,13 @@ The magic listener script
 
 The first script (after the "Input" block) in any workflow you build that should listen to a Nextcloud webhook must be ``CORE:LISTEN_TO_EVENT``. It must be an empty script with two parameters that you should fill statically: ``events``, which is a list of event IDs to listen to and ``filters`` a filter condition that allows more fine grained filtering for which events should be used. The filter condition as well as the available events with their payloads is documented in :ref:`the webhook_listeners documentation<webhook_listeners>`.
 
+You can copy the following Deno script for this:
+
+.. code-block:: typescript
+
+   export async function main(events: string[], filters: object) { }
+
+
 Nextcloud Scripts
 -----------------