feat(send_queue): Implement sending of MSC4274 galleries (#4977)

This was broken out of
#4838 and is a step
towards implementing
[MSC4274](matrix-org/matrix-spec-proposals#4274).

* The entry point for sending galleries via the send queue is a new
method
[`RoomSendQueue::send_gallery`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-8752e86459c22cff470d7ca617240dcbdf222c3c6c98be2af2e43ddec071154cR362)
which is a generalization of `RoomSendQueue::send_attachment`.
* `send_gallery` takes as input parameters a
[`GalleryConfig`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d38d74cec6159cf769c32bed496199146175f05428fc23ab13bc2c629900da3eR283)
(containing info about the gallery itself, such as its caption) and a
vector of
[`GalleryItemInfo`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d38d74cec6159cf769c32bed496199146175f05428fc23ab13bc2c629900da3eR355)s
(containing info about each image / file / etc. in the gallery).
* `send_gallery` creates the gallery event content via
[`Room:make_message_event`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-474f20e47fdcf60feac3f839a81f82fbb2c1fd0bb406388b0262adb60216ce3bR2281)
which was renamed from `make_attachment_event` to reflect the fact that
it creates general `msgtype` events now.
* `send_gallery` maps the passed item infos into
[`GalleryItemQueueInfo`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d569f54c7901b5cd660aae77045c80dabbd3c251f0fa5891530469f35941327aR2008)s.
This additional struct allows grouping all the metadata for a single
gallery item together.
* Finally `send_gallery` invokes
[`QueueStorage::push_gallery`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-d569f54c7901b5cd660aae77045c80dabbd3c251f0fa5891530469f35941327aR1294)
which is a generalization of `QueueStorage::push_media`.
* `send_gallery` pushes upload requests for the media and thumbnails to
the queue in a "daisy chain" manner. The first thumbnail (or media if no
thumbnail exists) is pushed as a `QueuedRequestKind::MediaUpload`. The
remaining thumbnails and media are pushed as
`DependentQueuedRequestKind::UploadFileOrThumbnail`s while chaining each
request to the previous one.
* Finally a
[`DependentQueuedRequestKind::FinishGallery`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-42a1a7ebe2446d916c9e013293bafc5eb16fd89bfe89248e3877ef031cc83ef2R265)
is pushed to finalize the gallery upload (analogous to the existing
`FinishUpload` for single media uploads).
* The `FinishGallery` request is handled in
[`QueueStorage::handle_dependent_finish_gallery_upload`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-8752e86459c22cff470d7ca617240dcbdf222c3c6c98be2af2e43ddec071154cR628)
which was modeled after `handle_dependent_finish_upload`.
* To be able to map the temporary event source for each gallery item to
the final `AccumulatedSentMediaInfo`, a hash map is used.
* Using the hash map, the gallery event is then updated to use the
actual media sources via
[`update_gallery_event_after_upload`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-8752e86459c22cff470d7ca617240dcbdf222c3c6c98be2af2e43ddec071154cR104)
and a new method
[`Room::make_gallery_item_type`](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-474f20e47fdcf60feac3f839a81f82fbb2c1fd0bb406388b0262adb60216ce3bR2303)
* An [integration
test](https://github.com/matrix-org/matrix-rust-sdk/pull/4977/files#diff-21532ad5467a69489d7913b8da7afd4d618b7e357ce94f769e6e60e395b58055R2048)
has been added to demonstrate the functionality.

This is relatively large, unfortunately, but including everything needed
to actually send the event made it possible to also add a test for it.
It would be nice if the amount of new code could be reduced but I'm
struggling a bit to find ways to integrate galleries with the existing
media uploads further.

- [x] Public API changes documented in changelogs (optional)

---------

Signed-off-by: Johannes Marbach <[email protected]>

Author: Johennes

crates/matrix-sdk-base/src/store/mod.rs

@@ -72,7 +72,7 @@ mod send_queue;
 #[cfg(any(test, feature = "testing"))]
 pub use self::integration_tests::StateStoreIntegrationTests;
 #[cfg(feature = "unstable-msc4274")]
-pub use self::send_queue::AccumulatedSentMediaInfo;
+pub use self::send_queue::{AccumulatedSentMediaInfo, FinishGalleryItemInfo};
 pub use self::{
     memory_store::MemoryStore,
     send_queue::{

crates/matrix-sdk-base/src/store/send_queue.rs

@@ -258,6 +258,19 @@ pub enum DependentQueuedRequestKind {
         /// Information about the thumbnail, if present.
         thumbnail_info: Option<FinishUploadThumbnailInfo>,
     },
+
+    /// Finish a gallery upload by updating references to the media cache and
+    /// sending the final gallery event with the remote MXC URIs.
+    #[cfg(feature = "unstable-msc4274")]
+    FinishGallery {
+        /// Local echo for the event (containing the local MXC URIs).
+        ///
+        /// `Box` the local echo so that it reduces the size of the whole enum.
+        local_echo: Box<RoomMessageEventContent>,
+
+        /// Metadata about the gallery items.
+        item_infos: Vec<FinishGalleryItemInfo>,
+    },
 }
 
 /// If parent_is_thumbnail_upload is missing, we assume the request is for a
@@ -285,6 +298,18 @@ pub struct FinishUploadThumbnailInfo {
     pub height: Option<UInt>,
 }
 
+/// Detailed record about a file and thumbnail. When finishing a gallery
+/// upload, one [`FinishGalleryItemInfo`] will be used for each media in the
+/// gallery.
+#[cfg(feature = "unstable-msc4274")]
+#[derive(Clone, Debug, Serialize, Deserialize)]
+pub struct FinishGalleryItemInfo {
+    /// Transaction id for the file upload.
+    pub file_upload: OwnedTransactionId,
+    /// Information about the thumbnail, if present.
+    pub thumbnail_info: Option<FinishUploadThumbnailInfo>,
+}
+
 /// A transaction id identifying a [`DependentQueuedRequest`] rather than its
 /// parent [`QueuedRequest`].
 ///
@@ -369,6 +394,13 @@ pub struct AccumulatedSentMediaInfo {
     pub thumbnail: Option<MediaSource>,
 }
 
+#[cfg(feature = "unstable-msc4274")]
+impl From<AccumulatedSentMediaInfo> for SentMediaInfo {
+    fn from(value: AccumulatedSentMediaInfo) -> Self {
+        Self { file: value.file, thumbnail: value.thumbnail, accumulated: vec![] }
+    }
+}
+
 /// A unique key (identifier) indicating that a transaction has been
 /// successfully sent to the server.
 ///
@@ -442,6 +474,11 @@ impl DependentQueuedRequest {
                 // This one graduates into a new media event.
                 true
             }
+            #[cfg(feature = "unstable-msc4274")]
+            DependentQueuedRequestKind::FinishGallery { .. } => {
+                // This one graduates into a new gallery event.
+                true
+            }
         }
     }
 }

crates/matrix-sdk/CHANGELOG.md

@@ -34,6 +34,9 @@ All notable changes to this project will be documented in this file.
   flag of the room if an unthreaded read receipt is sent.
   ([#5055](https://github.com/matrix-org/matrix-rust-sdk/pull/5055))
 - `Client::is_user_ignored(&UserId)` can be used to check if a user is currently ignored. ([#5081](https://github.com/matrix-org/matrix-rust-sdk/pull/5081))
+- `RoomSendQueue::send_gallery` has been added to allow sending MSC4274-style media galleries
+  via the send queue under the `unstable-msc4274` feature.
+  ([#4977](https://github.com/matrix-org/matrix-rust-sdk/pull/4977))
 
 ### Bug fixes
 

crates/matrix-sdk/Cargo.toml

@@ -61,7 +61,7 @@ docsrs = ["e2e-encryption", "sqlite", "indexeddb", "sso-login", "qrcode"]
 experimental-share-history-on-invite = []
 
 # Add support for inline media galleries via msgtypes
-unstable-msc4274 = ["matrix-sdk-base/unstable-msc4274"]
+unstable-msc4274 = ["ruma/unstable-msc4274", "matrix-sdk-base/unstable-msc4274"]
 
 [dependencies]
 anyhow = { workspace = true, optional = true }

crates/matrix-sdk/src/attachment.rs

@@ -25,7 +25,7 @@ use ruma::{
         },
         Mentions,
     },
-    OwnedTransactionId, TransactionId, UInt,
+    OwnedTransactionId, UInt,
 };
 
 use crate::room::reply::Reply;
@@ -219,8 +219,8 @@ impl AttachmentConfig {
     ///   in its unsigned field as `transaction_id`. If not given, one is
     ///   created for the message.
     #[must_use]
-    pub fn txn_id(mut self, txn_id: &TransactionId) -> Self {
-        self.txn_id = Some(txn_id.to_owned());
+    pub fn txn_id(mut self, txn_id: OwnedTransactionId) -> Self {
+        self.txn_id = Some(txn_id);
         self
     }
 
@@ -236,21 +236,21 @@ impl AttachmentConfig {
         self
     }
 
-    /// Set the optional caption
+    /// Set the optional caption.
     ///
     /// # Arguments
     ///
-    /// * `caption` - The optional caption
+    /// * `caption` - The optional caption.
     pub fn caption(mut self, caption: Option<String>) -> Self {
         self.caption = caption;
         self
     }
 
-    /// Set the optional formatted caption
+    /// Set the optional formatted caption.
     ///
     /// # Arguments
     ///
-    /// * `formatted_caption` - The optional formatted caption
+    /// * `formatted_caption` - The optional formatted caption.
     pub fn formatted_caption(mut self, formatted_caption: Option<FormattedBody>) -> Self {
         self.formatted_caption = formatted_caption;
         self
@@ -260,7 +260,7 @@ impl AttachmentConfig {
     ///
     /// # Arguments
     ///
-    /// * `mentions` - The mentions of the message
+    /// * `mentions` - The mentions of the message.
     pub fn mentions(mut self, mentions: Option<Mentions>) -> Self {
         self.mentions = mentions;
         self
@@ -270,9 +270,123 @@ impl AttachmentConfig {
     ///
     /// # Arguments
     ///
-    /// * `reply` - The reply information of the message
+    /// * `reply` - The reply information of the message.
     pub fn reply(mut self, reply: Option<Reply>) -> Self {
         self.reply = reply;
         self
     }
 }
+
+/// Configuration for sending a gallery.
+#[cfg(feature = "unstable-msc4274")]
+#[derive(Debug, Default)]
+pub struct GalleryConfig {
+    pub(crate) txn_id: Option<OwnedTransactionId>,
+    pub(crate) items: Vec<GalleryItemInfo>,
+    pub(crate) caption: Option<String>,
+    pub(crate) formatted_caption: Option<FormattedBody>,
+    pub(crate) mentions: Option<Mentions>,
+    pub(crate) reply: Option<Reply>,
+}
+
+#[cfg(feature = "unstable-msc4274")]
+impl GalleryConfig {
+    /// Create a new empty `GalleryConfig`.
+    pub fn new() -> Self {
+        Self::default()
+    }
+
+    /// Set the transaction ID to send.
+    ///
+    /// # Arguments
+    ///
+    /// * `txn_id` - A unique ID that can be attached to a `MessageEvent` held
+    ///   in its unsigned field as `transaction_id`. If not given, one is
+    ///   created for the message.
+    #[must_use]
+    pub fn txn_id(mut self, txn_id: OwnedTransactionId) -> Self {
+        self.txn_id = Some(txn_id);
+        self
+    }
+
+    /// Adds a media item to the gallery.
+    ///
+    /// # Arguments
+    ///
+    /// * `item` - Information about the item to be added.
+    #[must_use]
+    pub fn add_item(mut self, item: GalleryItemInfo) -> Self {
+        self.items.push(item);
+        self
+    }
+
+    /// Set the optional caption.
+    ///
+    /// # Arguments
+    ///
+    /// * `caption` - The optional caption.
+    pub fn caption(mut self, caption: Option<String>) -> Self {
+        self.caption = caption;
+        self
+    }
+
+    /// Set the optional formatted caption.
+    ///
+    /// # Arguments
+    ///
+    /// * `formatted_caption` - The optional formatted caption.
+    pub fn formatted_caption(mut self, formatted_caption: Option<FormattedBody>) -> Self {
+        self.formatted_caption = formatted_caption;
+        self
+    }
+
+    /// Set the mentions of the message.
+    ///
+    /// # Arguments
+    ///
+    /// * `mentions` - The mentions of the message.
+    pub fn mentions(mut self, mentions: Option<Mentions>) -> Self {
+        self.mentions = mentions;
+        self
+    }
+
+    /// Set the reply information of the message.
+    ///
+    /// # Arguments
+    ///
+    /// * `reply` - The reply information of the message.
+    pub fn reply(mut self, reply: Option<Reply>) -> Self {
+        self.reply = reply;
+        self
+    }
+
+    /// Returns the number of media items in the gallery.
+    pub fn len(&self) -> usize {
+        self.items.len()
+    }
+
+    /// Checks whether the gallery contains any media items or not.
+    pub fn is_empty(&self) -> bool {
+        self.items.is_empty()
+    }
+}
+
+#[cfg(feature = "unstable-msc4274")]
+#[derive(Debug)]
+/// Metadata for a gallery item
+pub struct GalleryItemInfo {
+    /// The filename.
+    pub filename: String,
+    /// The mime type.
+    pub content_type: mime::Mime,
+    /// The binary data.
+    pub data: Vec<u8>,
+    /// The attachment info.
+    pub attachment_info: AttachmentInfo,
+    /// The caption.
+    pub caption: Option<String>,
+    /// The formatted caption.
+    pub formatted_caption: Option<FormattedBody>,
+    /// The thumbnail.
+    pub thumbnail: Option<Thumbnail>,
+}

crates/matrix-sdk/src/room/edit.rs

@@ -269,6 +269,12 @@ pub(crate) fn update_media_caption(
             event.formatted = formatted_caption;
             true
         }
+        #[cfg(feature = "unstable-msc4274")]
+        MessageType::Gallery(event) => {
+            event.body = caption.unwrap_or_default();
+            event.formatted = formatted_caption;
+            true
+        }
         MessageType::Image(event) => {
             set_caption!(event, caption);
             event.formatted = formatted_caption;