Merge branch 'develop' into 11129-send-feedback-to-contacts

.github/workflows/container_app_pr.yml

@@ -35,14 +35,14 @@ jobs:
 
             # Note: Accessing, pushing tags etc. to GHCR will only succeed in upstream because secrets.
             - name: Login to Github Container Registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   registry: ghcr.io
                   username: ${{ secrets.GHCR_USERNAME }}
                   password: ${{ secrets.GHCR_TOKEN }}
 
             - name: Set up QEMU for multi-arch builds
-              uses: docker/setup-qemu-action@v2
+              uses: docker/setup-qemu-action@v3
 
             # Get the image tag from either the command or default to branch name (Not used for now)
             #-   name: Get the target tag name

.github/workflows/container_app_push.yml

@@ -126,20 +126,20 @@ jobs:
             # Depending on context, we push to different targets. Login accordingly.
             - if: github.event_name != 'pull_request'
               name: Log in to Docker Hub registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   username: ${{ secrets.DOCKERHUB_USERNAME }}
                   password: ${{ secrets.DOCKERHUB_TOKEN }}
             - if: ${{ github.event_name == 'pull_request' }}
               name: Login to Github Container Registry
-              uses: docker/login-action@v2
+              uses: docker/login-action@v3
               with:
                   registry: ghcr.io
                   username: ${{ secrets.GHCR_USERNAME }}
                   password: ${{ secrets.GHCR_TOKEN }}
 
             - name: Set up QEMU for multi-arch builds
-              uses: docker/setup-qemu-action@v2
+              uses: docker/setup-qemu-action@v3
 
             - name: Re-set image tag based on branch (if master)
               if: ${{ github.ref_name == 'master' }}

.github/workflows/deploy_beta_testing.yml

@@ -68,7 +68,7 @@ jobs:
           overwrite: true
 
       - name: Execute payara war deployment remotely
-        uses: appleboy/ssh-action@v1.0.0
+        uses: appleboy/ssh-action@v1.2.0
         env:
           INPUT_WAR_FILE: ${{ env.war_file }}
         with:

README.md

@@ -1,20 +1,81 @@
 Dataverse®  
 ===============
 
-Dataverse is an [open source][] software platform for sharing, finding, citing, and preserving research data (developed by the [Dataverse team](https://dataverse.org/about) at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][]).
+![Dataverse-logo](https://github.com/IQSS/dataverse-frontend/assets/7512607/6c4d79e4-7be5-4102-88bd-dfa167dc79d3)
 
-[dataverse.org][] is our home on the web and shows a map of Dataverse installations around the world, a list of [features][], [integrations][] that have been made possible through [REST APIs][], our [project board][], our development [roadmap][], and more.
+## Table of Contents
 
-We maintain a demo site at [demo.dataverse.org][] which you are welcome to use for testing and evaluating Dataverse.
+1. [❓ What is Dataverse?](#what-is-dataverse)
+2. [✔ Try Dataverse](#try-dataverse)
+3. [🌐 Features, Integrations, Roadmaps, and More](#website)
+4. [📥 Installation](#installation)
+5. [🏘 Community and Support](#community-and-support)
+6. [🧑‍💻️ Contributing](#contributing)
+7. [⚖️ Legal Information](#legal-informations)
 
-To install Dataverse, please see our [Installation Guide][] which will prompt you to download our [latest release][]. Docker users should consult the [Container Guide][].
+<a name="what-is-dataverse"></a>
 
-To discuss Dataverse with the community, please join our [mailing list][], participate in a [community call][], chat with us at [chat.dataverse.org][], or attend our annual [Dataverse Community Meeting][].
+## ❓  What is Dataverse?
 
-We love contributors! Please see our [Contributing Guide][] for ways you can help.
+Welcome to Dataverse®, the [open source][] software platform designed for sharing, finding, citing, and preserving research data. Developed by the Dataverse team at the [Institute for Quantitative Social Science](https://iq.harvard.edu/) and the [Dataverse community][], our platform makes it easy for research organizations to host, manage, and share their data with the world.
+
+<a name="try-dataverse"></a>
+
+## ✔  Try Dataverse
+
+We invite you to explore our demo site at [demo.dataverse.org][]. This site is ideal for testing and evaluating Dataverse in a risk-free environment.
+
+<a name="website"></a>
+
+## 🌐 Features, Integrations, Roadmaps, and More
+
+Visit [dataverse.org][], our home on the web, for a comprehensive overview of Dataverse. Here, you will find:
+
+- An interactive map showcasing Dataverse installations worldwide.
+- A detailed list of [features][].
+- Information on [integrations][] that have been made possible through our [REST APIs][].
+- Our [project board][] and development [roadmap][].
+- News, events, and more.
+
+<a name="installation"></a>
+
+## 📥 Installation
+
+Ready to get started? Follow our [Installation Guide][] to download and install the latest release of Dataverse.
+
+If you are using Docker, please refer to our [Container Guide][] for detailed instructions.
+
+<a name="community-and-support"></a>
+
+## 🏘 Community and Support
+
+Engage with the vibrant Dataverse community through various channels:
+
+- **[Mailing List][]**: Join the conversation on our [mailing list][].
+- **[Community Calls][]**: Participate in our regular [community calls][] to discuss new features, ask questions, and share your experiences.
+- **[Chat][]**: Connect with us and other users in real-time at [dataverse.zulipchat.com][].
+- **[Dataverse Community Meeting][]**: Attend our annual [Dataverse Community Meeting][] to network, learn, and collaborate with peers and experts.
+- **[DataverseTV][]**: Watch the video content from the Dataverse community on [DataverseTV][] and on [Harvard's IQSS YouTube channel][].
+
+<a name="contributing"></a>
+## 🧑‍💻️ Contribute to Dataverse
+
+We love contributors! Whether you are a developer, researcher, or enthusiast, there are many ways you can help.
+
+Visit our [Contributing Guide][] to learn how you can get involved.
+
+Join us in building and enhancing Dataverse to make research data more accessible and impactful. Your support and participation are crucial to our success!
+
+<a name="legal-informations"></a>
+## ⚖️ Legal Information
 
 Dataverse is a trademark of President and Fellows of Harvard College and is registered in the United States.
 
+---
+For more detailed information, visit our website at [dataverse.org][].
+
+Feel free to [reach out] with any questions or feedback. Happy researching!
+
 [![Dataverse Project logo](src/main/webapp/resources/images/dataverseproject_logo.jpg "Dataverse Project")](http://dataverse.org)
 
 [![API Test Status](https://jenkins.dataverse.org/buildStatus/icon?job=IQSS-dataverse-develop&subject=API%20Test%20Status)](https://jenkins.dataverse.org/job/IQSS-dataverse-develop/)
@@ -37,6 +98,11 @@ Dataverse is a trademark of President and Fellows of Harvard College and is regi
 [Contributing Guide]: CONTRIBUTING.md
 [mailing list]: https://groups.google.com/group/dataverse-community
 [community call]: https://dataverse.org/community-calls
-[chat.dataverse.org]: https://chat.dataverse.org
+[Chat]: https://dataverse.zulipchat.com
+[dataverse.zulipchat.com]: https://dataverse.zulipchat.com
 [Dataverse Community Meeting]: https://dataverse.org/events
 [open source]: LICENSE.md
+[community calls]: https://dataverse.org/community-calls
+[DataverseTV]: https://dataverse.org/dataversetv
+[Harvard's IQSS YouTube channel]: https://www.youtube.com/@iqssatharvarduniversity8672
+[reach out]: https://dataverse.org/contact

doc/release-notes/10241-new-solr-client.md

@@ -0,0 +1,9 @@
+[HttpSolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html) is deprecated as of Solr 9, and which will be removed in a future major release of Solr. It's recommended to use [Http2SolrClient](https://solr.apache.org/docs/9_4_1/solrj/org/apache/solr/client/solrj/impl/Http2SolrClient.html) instead.
+
+[Solr documentation](https://solr.apache.org/guide/solr/latest/deployment-guide/solrj.html#types-of-solrclients) describe it as a _async, non-blocking and general-purpose client that leverage HTTP/2 using the Jetty Http library_.
+
+With Solr 9.4.1, the Http2SolrClient is indicate as experimental. But since the 9.6 version of Solr, this mention is no longer maintained.
+
+The ConcurrentUpdateHttp2SolrClient is now also used in some cases, which is supposed to be more efficient for indexing.
+
+For more information, see issue [#10161](https://github.com/IQSS/dataverse/issues/10161) and pull request [#10241](https://github.com/IQSS/dataverse/pull/10241)

doc/release-notes/10304-add-move-dataverse-collection.md

@@ -0,0 +1,5 @@
+### Move a collection from the dashboard
+
+In addition to the api [Move a Dataverse Collection](https://guides.dataverse.org/en/latest/admin/dataverses-datasets.html#move-a-dataverse-collection), it is now possible for a Dataverse administrator to move a collection from the Dataverse dashboard.
+
+For more information, see #10304.

doc/release-notes/10340-forbidden.md

@@ -0,0 +1,3 @@
+### Backward Incompatible Changes
+
+The [Show Role](https://dataverse-guide--11116.org.readthedocs.build/en/11116/api/native-api.html#show-role) API endpoint was returning 401 Unauthorized when a permission check failed. This has been corrected to return 403 Forbidden instead. That is, the API token is known to be good (401 otherwise) but the user lacks permission (403 is now sent). See also the [API Changelog](https://dataverse-guide--11116.org.readthedocs.build/en/11116/api/changelog.html), #10340, and #11116.

doc/release-notes/10714-access-requests-missing-since-upgrade-v6-0.md

@@ -0,0 +1,6 @@
+### Flyway Script added to Fix File Access Requests when upgrading from Dataverse 6.0
+
+Database update script added to prevent duplicate keys when upgrading from V6.0
+This script will delete access requests made after the initial request and will set the initial request to "Created"
+
+See: https://github.com/IQSS/dataverse/issues/10714

doc/release-notes/10887-solr-field-types.md

@@ -73,6 +73,26 @@ Now start Solr.
 
 8\. Reindex Solr
 
+Note: these instructions are a little different than usual because we observed a strange error about `DOCS_AND_FREQS_AND_POSITIONS` when testing upgrades (see #11139 for details). Extra steps about explicitly clearing the index and reloading the core are included. If you run into trouble, as a last resort, you could reinstall Solr completely and then reindex.
+
+Clear the Solr index:
+
+```shell
+curl http://localhost:8080/api/admin/index/clear
+```
+
+Make sure the Solr index is empty:
+
+```shell
+curl "http://localhost:8983/solr/collection1/select?rows=1000000&wt=json&indent=true&q=*%3A*"
+```
+
+Reload the Solr core:
+
+```shell
+curl "http://localhost:8983/solr/admin/cores?action=RELOAD&core=collection1"
+```
+
 Below is the simplest way to reindex Solr:
 
 ```shell

doc/release-notes/10943-featured-items.md

@@ -0,0 +1,15 @@
+CRUD endpoints for Collection Featured Items have been implemented. In particular, the following endpoints have been implemented:
+
+- Create a feature item (POST /api/dataverses/<dataverse_id>/featuredItems)
+- Update a feature item (PUT /api/dataverseFeaturedItems/<item_id>)
+- Delete a feature item (DELETE /api/dataverseFeaturedItems/<item_id>)
+- List all featured items in a collection (GET /api/dataverses/<dataverse_id>/featuredItems)
+- Delete all featured items in a collection (DELETE /api/dataverses/<dataverse_id>/featuredItems)
+- Update all featured items in a collection (PUT /api/dataverses/<dataverse_id>/featuredItems)
+
+New settings:
+
+- dataverse.files.featured-items.image-maxsize - It sets the maximum allowed size of the image that can be added to a featured item.
+- dataverse.files.featured-items.image-uploads - It specifies the name of the subdirectory for saving featured item images within the docroot directory.
+
+See also #10943 and #11124.

doc/release-notes/11075-ror.md

@@ -0,0 +1,15 @@
+### ROR (Research Organization Registry) as Author Identifier Type 
+
+ROR (Research Organization Registry) has been added as an Author Identifier Type (alongside ORCID, etc.) for when the author is an organization rather than a person. As with all author identifiers, be sure to select the proper identifier type (e.g. "ROR") and enter just the unique identifier (e.g. "03vek6s52") rather than the full URL (e.g. "https://ror.org/03vek6s52"). Like ORCID, ROR will appear in the "Datacite" metadata export format. See also the [ROR](https://ror.org) website, a new [note](https://dataverse-guide--11118.org.readthedocs.build/en/11118/user/dataset-management.html#adding-a-new-dataset) in a guides about entering author identifiers, #11075, and #11118.
+
+## Upgrade Instructions
+
+6\. Update metadata blocks
+
+These changes reflect incremental improvements made to the handling of core metadata fields. Expect the loading of the citation block to take several seconds because of its size (especially due to the number of languages).
+
+```shell
+wget https://raw.githubusercontent.com/IQSS/dataverse/v6.6/scripts/api/data/metadatablocks/citation.tsv
+
+curl http://localhost:8080/api/admin/datasetfield/load -H "Content-type: text/tab-separated-values" -X POST --upload-file citation.tsv
+```

doc/release-notes/11095-fix-extcvoc-indexing.md

@@ -0,0 +1,7 @@
+Some External Controlled Vocabulary scripts/configurations, when used on a metadata field that is single-valued could result 
+in indexing failure for the dataset (e.g. when the script tried to index both the identifier and name of the identified entity for indexing).
+Dataverse has been updated to correctly indicate the need for a multi-valued Solr field in these cases in the call to /api/admin/index/solr/schema.
+Configuring the Solr schema and the update-fields.sh script as usually recommended when using custom metadata blocks will resolve the issue.
+
+The overall release notes should include a Solr update (which hopefully is required by an update to 9.7.0 anyway) and our standard instructions 
+should change to recommending use of the update-fields.sh script when using custom metadatablocks *and/or external vocabulary scripts*.

doc/release-notes/8739-publisher-during-harvesting.md

@@ -0,0 +1 @@
+The publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. This improves the citation associated with these datasets, but the change only affects newly harvested datasets. All datasets should be re-harvested if you wish to pick up this change on already harvested datasets. For more information, see [the guides](https://dataverse-guide--9013.org.readthedocs.build/en/9013/admin/harvestclients.html#harvesting-client-changelog), #8739, and #9013.

doc/sphinx-guides/source/_static/api/dataset-add-single-compound-field-metadata.json

@@ -0,0 +1,13 @@
+{
+    "fields": [
+        {
+            "typeName": "targetSampleSize",
+            "value": {
+                "targetSampleActualSize": {
+                    "typeName": "targetSampleSizeFormula",
+                    "value": "n = N*X / (X + N – 1)"
+                }
+            }
+        }
+    ]
+}

doc/sphinx-guides/source/_static/api/dataset-add-single-cvoc-field-metadata.json

@@ -0,0 +1,4 @@
+{
+    "typeName": "journalArticleType",
+    "value": "abstract"
+}

doc/sphinx-guides/source/admin/harvestclients.rst

@@ -48,6 +48,11 @@ Each harvesting client run logs a separate file per run to the app server's defa
 
 Note that you'll want to run a minimum of Dataverse Software 4.6, optimally 4.18 or beyond, for the best OAI-PMH interoperability.
 
+Harvesting Client Changelog
+---------------------------
+
+- As of Dataverse 6.5, the publisher value of harvested datasets is now attributed to the dataset's distributor instead of its producer. This change affects all newly harvested datasets. For more information, see https://github.com/IQSS/dataverse/pull/9013
+
 Harvesting Non-OAI-PMH
 ~~~~~~~~~~~~~~~~~~~~~~
 

doc/sphinx-guides/source/admin/metadatacustomization.rst

@@ -272,7 +272,11 @@ Each of the three main sections own sets of properties:
 |              | “Value” field is used as the identifier.   |                                         |
 +--------------+--------------------------------------------+-----------------------------------------+
 | displayOrder | Control the order in which the enumerated  | Non-negative integer.                   |
-|              | values are displayed for selection.        |                                         |
+|              | values are displayed for selection. When   |                                         |
+|              | adding new values, you don't have to add   |                                         |
+|              | them at the end. You can renumber existing |                                         |
+|              | values to update the order in which they   |                                         |
+|              | appear.                                    |                                         |
 +--------------+--------------------------------------------+-----------------------------------------+
 
 FieldType definitions
@@ -544,7 +548,7 @@ a necessary re-index, but for your custom metadata you will need to keep track o
 
 Please note also that if you are going to make a pull request updating ``conf/solr/schema.xml`` with fields you have
 added, you should first load all the custom metadata blocks in ``scripts/api/data/metadatablocks`` (including ones you
-don't care about) to create a complete list of fields. (This might change in the future.)
+don't care about) to create a complete list of fields. (This might change in the future.) Please see :ref:`update-solr-schema-dev` in the Developer Guide.
 
 Reloading a Metadata Block
 --------------------------
@@ -564,8 +568,7 @@ Using External Vocabulary Services
 
 The Dataverse software has a mechanism to associate specific fields defined in metadata blocks with a vocabulary(ies) managed by external services. The mechanism relies on trusted third-party Javascripts. The mapping from field type to external vocabulary(ies) is managed via the :ref:`:CVocConf <:CVocConf>` setting.
 
-*This functionality is considered 'experimental'. It may require significant effort to configure and is likely to evolve in subsequent Dataverse software releases.*
-
+*This functionality may require significant effort to configure and is likely to evolve in subsequent Dataverse software releases.*
 
 The effect of configuring this mechanism is similar to that of defining a field in a metadata block with 'allowControlledVocabulary=true':
 
@@ -590,6 +593,9 @@ Configuration involves specifying which fields are to be mapped, to which Solr f
 These are all defined in the :ref:`:CVocConf <:CVocConf>` setting as a JSON array. Details about the required elements as well as example JSON arrays are available at https://github.com/gdcc/dataverse-external-vocab-support, along with an example metadata block that can be used for testing.
 The scripts required can be hosted locally or retrieved dynamically from https://gdcc.github.io/ (similar to how dataverse-previewers work).
 
+Since external vocabulary scripts can change how fields are indexed (storing an identifier and name and/or values in different languages),
+updating the Solr schema as described in :ref:`update-solr-schema` should be done after adding new scripts to your configuration.
+
 Please note that in addition to the :ref:`:CVocConf` described above, an alternative is the :ref:`:ControlledVocabularyCustomJavaScript` setting.
 
 Protecting MetadataBlocks