Regression with MQTT clients - Cross User QoS 1 delivery

[persinac]

Observed behavior

Problem Description

When two different MQTT users (mapped via mTLS certificate CN) attempt pub/sub communication:

1. Publisher sends QoS 1 message and receives PUBACK (message stored in JetStream)
2. Subscriber's JetStream consumer receives the message (consumer sequence increments)
3. However the subscriber's MQTT client never receives the message
4. Messages accumulate as outstanding acks and get redelivered indefinitely

The internal $MQTT.sub.* delivery subject binding appears broken for cross-user scenarios.

Initial though: a Wildcard Translation Bug

Fun fact - it's not a wildcard translation issue.

Initial investigation suggested MQTT # wildcards weren't being translated to NATS .>. This was incorrect. Further testing revealed:

1. Consumer filters ARE correct - Both .> and exact-match consumers are created
2. Messages ARE delivered to consumers - Consumer sequence increments
3. But acknowledgments never come - Ack floor stays at 0
4. The bug is in delivery subject binding - Messages go to $MQTT.sub.xxx but the client isn't receiving on that subject

Diagnostics

Consumer State Shows Delivery Without Acknowledgment

$ nats consumer info '$MQTT_msgs' '111SUQEl_UO4iyguny1Jz4clB1xwhjy'

Configuration:
    Delivery Subject: $MQTT.sub.UO4iyguny1Jz4clB1xwhgO
      Filter Subject: $MQTT.msgs.bridge.alex-garage-1.down.>   # <-- Filter IS correct!

State:
   Last Delivered Message: Consumer sequence: 9 Stream sequence: 289769
     Acknowledgment floor: Consumer sequence: 0 Stream sequence: 0   # <-- NOTHING acknowledged!
         Outstanding Acks: 3 out of maximum 1024                      # <-- Messages stuck
     Redelivered Messages: 3                                          # <-- Being redelivered
     Unprocessed Messages: 0

This proves:

1. The consumer filter is correct (includes .>)
2. Messages ARE being delivered (consumer sequence = 9)
3. But the client NEVER acknowledges (ack floor = 0)
4. Messages accumulate and get redelivered indefinitely

Consumer Lifecycle is Correct

Testing confirmed that consumers are properly cleaned up on disconnect and recreated on connect:

# Before reconnect:
0EuYuueO_5epcT7g45uz49HkuBdAjBh  (Delivery: $MQTT.sub.5epcT7g45uz49HkuBdAj8M)

# After reconnect - old consumer deleted, new one created:
0EuYuueO_UO4iyguny1Jz4clB1xwds4  (Delivery: $MQTT.sub.UO4iyguny1Jz4clB1xwdoU)

MQTT Wildcard Translation is Correct

NATS correctly creates two consumers for # wildcard subscriptions:

Consumer	Filter Subject	Purpose
...hjy	$MQTT.msgs.bridge.alex-garage-1.down.>	Matches subtopics
...hyI	$MQTT.msgs.bridge.alex-garage-1.down	Matches exact topic

This is correct behavior since MQTT # matches zero or more levels, but NATS .> matches one or more.

NATS Debug Logs

With -DV flags enabled:

# Device subscribes
[TRC] "device@example.com" - <<- [SUBSCRIBE [bridge/alex-garage-1/down/# QoS=1] pi=53284]
[TRC] "device@example.com" - ->> [SUBACK pi=53284]

# Service publishes
[TRC] "service@example.com" - <<- [PUBLISH bridge/alex-garage-1/down/keys/response QoS=1 size=152 pi=4]
[TRC] "service@example.com" - ->> [PUBACK pi=4]

# NOTE: No message forwarded to device!

Theory

I'm not the most privy to NATs jetstream user/subject bindings, however, something smells with how NATS 2.12 binds the JetStream consumer's delivery subject to the MQTT client's session in cross-user scenarios.

1. Device subscribes to bridge/alex-garage-1/down/#
2. NATS creates JetStream consumer with delivery subject $MQTT.sub.xxx
3. In same-user scenarios, the internal subscription on $MQTT.sub.xxx is properly connected
4. In cross-user scenarios, the binding is broken - messages are delivered to $MQTT.sub.xxx but the MQTT session isn't receiving from that subject
5. Messages accumulate as outstanding acks since the client can't acknowledge what it never received

Additional Observations

1. Cross-user delivery fails: Messages between different MQTT users (mapped via mTLS certs) are not delivered
2. Same-user delivery works: When publisher and subscriber use the same certificate, delivery works perfectly
3. Messages are stored: The $MQTT_msgs stream receives and stores the messages correctly
4. Consumer filters are correct: The .> wildcard is properly added to filter subjects
5. Consumer lifecycle is correct: Consumers are properly created/deleted on connect/disconnect
6. Delivery attempts happen: Consumer sequence increments, showing NATS tries to deliver
7. Acknowledgments never come: Ack floor stays at 0, messages redelivered indefinitely

Workaround

None known. Downgrading to NATS 2.11.11 restores correct behavior

Expected behavior

Test Scenario

Subscriber (Device):

• Connects via MQTT with mTLS (certificate CN: device@example.com)
• Subscribes to: bridge/alex-garage-1/down/# with QoS 1

Publisher (Service):

• Connects via MQTT with mTLS (certificate CN: service@example.com)
• Publishes to: bridge/alex-garage-1/down/keys/response with QoS 1

Result:

• Publisher receives PUBACK (message stored in JetStream)
• Subscriber receives the message
• Consumer shows messages delivered and acknowledged

Server and client version

Environment

• NATS Server Version: 2.12.3-alpine
• Previous Working Version: 2.11.x (issue appeared after upgrade)
• Protocol: MQTT over TLS (port 8883)
• Authentication: mTLS with verify_and_map: true
• JetStream: Enabled
• Accounts: Using multi-account setup (SYS + APP accounts)

Host environment

Running in a docker container on an ec2 instance running Amazon Linux 2023

Steps to reproduce

Reproduction Steps

1. NATS Configuration

# nats.conf
server_name: nats-mqtt

port: 4222
http: 8222

jetstream: {
  store_dir: "/data/jetstream"
}

include "/includes/users.inc"

mqtt {
  port: 1883
}

mqtt {
  host: 0.0.0.0
  port: 8883
  tls {
    cert_file: "/etc/nats/certs/server-cert.pem"
    key_file:  "/etc/nats/certs/server-key.pem"
    ca_file:   "/etc/nats/certs/root-ca.pem"
    verify_and_map: true
  }
}

2. Users Configuration (users.inc)

accounts {
  SYS {
    users = [
      { user: "admin", password: "...", permissions: { publish: [">"], subscribe: [">"] } }
    ]
  }

  APP {
    jetstream { max_file: 25Gb }
    users = [
      { user: "service@example.com", permissions: {"publish": [">"], "subscribe": [">", "$MQTT.sub.>"]}, allowed_connection_types: ["MQTT"] },
      { user: "device@example.com", permissions: {"publish": [">"], "subscribe": [">", "$MQTT.sub.>"]}, allowed_connection_types: ["MQTT"] }
    ]
  }
}

system_account: SYS

3. Test Scenario

Subscriber (Device):

• Connects via MQTT with mTLS (certificate CN: device@example.com)
• Subscribes to: bridge/alex-garage-1/down/# with QoS 1

Publisher (Service):

• Connects via MQTT with mTLS (certificate CN: service@example.com)
• Publishes to: bridge/alex-garage-1/down/keys/response with QoS 1

Result:

• Publisher receives PUBACK (message stored in JetStream)
• Subscriber NEVER receives the message
• Consumer shows messages delivered but never acknowledged

4. Same-User Test (Works)

When both publisher and subscriber use the same certificate/user, message delivery works correctly. This rules out permission issues and confirms the bug is specific to cross-user scenarios.

[kozlovic]

The users are part of the same account, so that should not be an issue. I can't reproduce with simple authentication (that is, without TLS and just use a username and password in your provided config file). I tried with both 2.11, 2.12.3 and main branch. I will have to try with mTLS and verify_and_map: true to see if it is related to that.

Note that with your current config file, you would not be able to use the NATS CLI to see the streams/consumers. If I use "admin" then it says that "JetStream not enabled for account (10039)", which is not surprising, and any user from "APP" account would fail since they are only for MQTT clients (allowed_connection_types: ["MQTT"]). I added a new user without any perms/allowed_connection_types to verify that the consumer is properly receiving/acking messages.

Again, I will try with mTLS/verify_and_map and update this ticket.

[kozlovic]

I can't reproduce the issue, even with mTLS. Here is the output with -D of the nats-server:

ik@MacBookAirSynadia nats-server % rm -fr ~/dev/datastore/ 
ik@MacBookAirSynadia nats-server % nats-server -c c.conf -D
[37257] 2026/01/05 13:15:23.659061 [INF] Starting nats-server
[37257] 2026/01/05 13:15:23.659173 [INF]   Version:  2.12.3
[37257] 2026/01/05 13:15:23.659176 [INF]   Git:      [450a519]
[37257] 2026/01/05 13:15:23.659195 [DBG]   Go build: go1.24.11
[37257] 2026/01/05 13:15:23.659197 [INF]   Name:     nats-mqtt
[37257] 2026/01/05 13:15:23.659198 [INF]   Node:     BMfiqAIw
[37257] 2026/01/05 13:15:23.659200 [INF]   ID:       NCJ75KHZGR43EJCBGD3SH5Z64VP5TDZ3P56P2UVAUNML2I77VJWYPGRL
[37257] 2026/01/05 13:15:23.659235 [WRN] Plaintext passwords detected, use nkeys or bcrypt
[37257] 2026/01/05 13:15:23.659258 [INF] Using configuration file: c.conf (sha256:6d2c7d21dbe9b94259f91aeb2477fea0c0b6677cb28fe6504e1563b09b5f718d)
[37257] 2026/01/05 13:15:23.660656 [INF] Starting http monitor on 0.0.0.0:8222
[37257] 2026/01/05 13:15:23.660964 [INF] Starting JetStream
[37257] 2026/01/05 13:15:23.661182 [DBG] JetStream creating dynamic configuration - 12.00 GB memory, 1.16 TB disk
[37257] 2026/01/05 13:15:23.661401 [INF]     _ ___ _____ ___ _____ ___ ___   _   __  __
[37257] 2026/01/05 13:15:23.661404 [INF]  _ | | __|_   _/ __|_   _| _ \ __| /_\ |  \/  |
[37257] 2026/01/05 13:15:23.661406 [INF] | || | _|  | | \__ \ | | |   / _| / _ \| |\/| |
[37257] 2026/01/05 13:15:23.661408 [INF]  \__/|___| |_| |___/ |_| |_|_\___/_/ \_\_|  |_|
[37257] 2026/01/05 13:15:23.661409 [INF] 
[37257] 2026/01/05 13:15:23.661411 [INF]          https://docs.nats.io/jetstream
[37257] 2026/01/05 13:15:23.661412 [INF] 
[37257] 2026/01/05 13:15:23.661413 [INF] ---------------- JETSTREAM ----------------
[37257] 2026/01/05 13:15:23.661415 [INF]   Strict:          true
[37257] 2026/01/05 13:15:23.661418 [INF]   Max Memory:      12.00 GB
[37257] 2026/01/05 13:15:23.661420 [INF]   Max Storage:     1.16 TB
[37257] 2026/01/05 13:15:23.661439 [INF]   Store Directory: "<..>/datastore/jetstream"
[37257] 2026/01/05 13:15:23.661441 [INF]   API Level:       2
[37257] 2026/01/05 13:15:23.661443 [INF] -------------------------------------------
[37257] 2026/01/05 13:15:23.661513 [DBG]   Exports:
[37257] 2026/01/05 13:15:23.661517 [DBG]      $JS.API.>
[37257] 2026/01/05 13:15:23.661817 [DBG] Enabled JetStream for account "APP"
[37257] 2026/01/05 13:15:23.661822 [DBG]   Max Memory:      -1 B
[37257] 2026/01/05 13:15:23.661825 [DBG]   Max Storage:     25.00 GB
[37257] 2026/01/05 13:15:23.662088 [DBG] JetStream state for account "APP" recovered
[37257] 2026/01/05 13:15:23.662135 [INF] Took 1.166458ms to start JetStream
[37257] 2026/01/05 13:15:23.662220 [INF] Listening for MQTT clients on tls://0.0.0.0:1883
[37257] 2026/01/05 13:15:23.662536 [INF] Listening for client connections on 0.0.0.0:4222
[37257] 2026/01/05 13:15:23.662540 [DBG] Get non local IPs for "0.0.0.0"
[37257] 2026/01/05 13:15:23.662661 [DBG]   ip=<..>
[37257] 2026/01/05 13:15:23.662667 [DBG]   ip=<..>
[37257] 2026/01/05 13:15:23.662669 [DBG]   ip=<..>
[37257] 2026/01/05 13:15:23.662670 [DBG]   ip=<..>
[37257] 2026/01/05 13:15:23.662723 [INF] Server is ready
[37257] 2026/01/05 13:15:23.662728 [DBG] maxprocs: Leaving GOMAXPROCS=8: CPU quota undefined
[37257] 2026/01/05 13:15:28.210309 [DBG] 127.0.0.1:62812 - mid:6 - Client connection created
[37257] 2026/01/05 13:15:28.210365 [DBG] 127.0.0.1:62812 - mid:6 - Starting TLS client connection handshake
[37257] 2026/01/05 13:15:28.220476 [DBG] 127.0.0.1:62812 - mid:6 - TLS handshake complete
[37257] 2026/01/05 13:15:28.220490 [DBG] 127.0.0.1:62812 - mid:6 - TLS version 1.3, cipher suite TLS_AES_128_GCM_SHA256
[37257] 2026/01/05 13:15:28.220821 [DBG] 127.0.0.1:62812 - mid:6 - "my_dur" - Multiple peer certificates found, selecting first
[37257] 2026/01/05 13:15:28.220839 [DBG] 127.0.0.1:62812 - mid:6 - "my_dur" - Using email found in cert for auth ["derek@nats.io"]
[37257] 2026/01/05 13:15:28.220925 [INF] Creating MQTT streams/consumers with replicas 1 for account "APP"
[37257] 2026/01/05 13:15:28.231498 [DBG] 127.0.0.1:62812 - mid:6 - "my_dur" - Client connected
[37257] 2026/01/05 13:15:38.536875 [DBG] 127.0.0.1:62813 - mid:29 - Client connection created
[37257] 2026/01/05 13:15:38.536887 [DBG] 127.0.0.1:62813 - mid:29 - Starting TLS client connection handshake
[37257] 2026/01/05 13:15:38.545222 [DBG] 127.0.0.1:62813 - mid:29 - TLS handshake complete
[37257] 2026/01/05 13:15:38.545237 [DBG] 127.0.0.1:62813 - mid:29 - TLS version 1.3, cipher suite TLS_AES_128_GCM_SHA256
[37257] 2026/01/05 13:15:38.545267 [DBG] 127.0.0.1:62813 - mid:29 - "mqttx_40aa46c0" - Multiple peer certificates found, selecting first
[37257] 2026/01/05 13:15:38.545277 [DBG] 127.0.0.1:62813 - mid:29 - "mqttx_40aa46c0" - Using email found in cert for auth ["ivan@nats.io"]
[37257] 2026/01/05 13:15:38.546142 [DBG] 127.0.0.1:62813 - mid:29 - "mqttx_40aa46c0" - Client connected
[37257] 2026/01/05 13:15:38.548899 [DBG] 127.0.0.1:62813 - mid:29 - "mqttx_40aa46c0" - Client connection closed: Client Closed
[37257] 2026/01/05 13:15:46.155327 [DBG] 127.0.0.1:62814 - cid:30 - Client connection created
[37257] 2026/01/05 13:15:47.019089 [DBG] 127.0.0.1:62814 - cid:30 - "v1.37.0:go:NATS CLI Version development" - "APP/user:a" - Client connection closed: Client Closed

This is how the consumer was started:

% mqttx sub -V 3.1.1 -h 127.0.0.1 -p 1883 --topic 'bridge/alex-garage-1/down/#' -q 1 --no-clean -i my_dur --cert client-cert.pem --key client-key.pem --ca ca.pem
✔ Connected
✔ Subscribed to bridge/alex-garage-1/down/#
topic: bridge/alex-garage-1/down/keys/response, qos: 1
something

The publisher:

% mqttx  pub -V 3.1.1 -h 127.0.0.1 -p 1883 --topic 'bridge/alex-garage-1/down/keys/response' --message "something" -q 1 --cert client-cert2.pem --key client-key2.pem --ca ca.pem 
✔ Connected
✔ Message published

The output from consumer info:

% nats -s nats://a:x@127.0.0.1:4222 consumer info '$MQTT_msgs'                                  
? Select a Consumer H5qS24fo_9gB7zAyq0sL7kwSAmUL9J1
Information for Consumer $MQTT_msgs > H5qS24fo_9gB7zAyq0sL7kwSAmUL9J1 created 2026-01-05T13:15:28-07:00

Configuration:

            Durable Name: H5qS24fo_9gB7zAyq0sL7kwSAmUL9J1
        Delivery Subject: $MQTT.sub.9gB7zAyq0sL7kwSAmUL9Dl
          Filter Subject: $MQTT.msgs.bridge.alex-garage-1.down.>
          Deliver Policy: New
              Ack Policy: Explicit
                Ack Wait: 30.00s
           Replay Policy: Instant
         Max Ack Pending: 1,024
            Flow Control: false

State:

  Last Delivered Message: Consumer sequence: 1 Stream sequence: 1 Last delivery: 8.46s ago
    Acknowledgment Floor: Consumer sequence: 1 Stream sequence: 1 Last Ack: 8.46s ago
        Outstanding Acks: 0 out of maximum 1,024
    Redelivered Messages: 0
    Unprocessed Messages: 0
         Active Interest: Active

And here is my version of the configuration file:

accounts {
  SYS {
    users = [
      { user: "sys", password: "x", permissions: { publish: [">"], subscribe: [">"] } }
    ]
  }

  APP {
    jetstream { max_file: 25Gb }
    users = [
      { user: "ivan@nats.io", permissions: {"publish": [">"], "subscribe": [">", "$MQTT.sub.>"]}, allowed_connection_types: ["MQTT"] }
      { user: "derek@nats.io", permissions: {"publish": [">"], "subscribe": [">", "$MQTT.sub.>"]}, allowed_connection_types: ["MQTT"] }
      { user: "a", password: "x"}
    ]
  }
}

system_account: SYS

server_name: nats-mqtt

port: 4222
http: 8222

jetstream: {
  store_dir: "<...>/datastore"
}

mqtt {
  port: 1883
  tls: {
    cert_file: "server-cert.pem"
    key_file:  "server-key.pem"
    ca_file:   "ca.pem"
    verify_and_map: true
  }
}

I also tried with current main branch and same result, that is, it works: message is received and not showing as un'acked.

[kozlovic]

I noticed that you used -DV, so restarted the test with -DV and here is the output from the server (this time it is main branch):

[37432] 2026/01/05 13:23:55.934157 [DBG] 127.0.0.1:62831 - mid:6 - Client connection created
[37432] 2026/01/05 13:23:55.934175 [DBG] 127.0.0.1:62831 - mid:6 - Starting TLS client connection handshake
[37432] 2026/01/05 13:23:55.942624 [DBG] 127.0.0.1:62831 - mid:6 - TLS handshake complete
[37432] 2026/01/05 13:23:55.942645 [DBG] 127.0.0.1:62831 - mid:6 - TLS version 1.3, cipher suite TLS_AES_128_GCM_SHA256
[37432] 2026/01/05 13:23:55.942726 [TRC] 127.0.0.1:62831 - mid:6 - "my_dur" - <<- [CONNECT clientID=my_dur keepAlive=1m30s]
[37432] 2026/01/05 13:23:55.942747 [DBG] 127.0.0.1:62831 - mid:6 - "my_dur" - Multiple peer certificates found, selecting first
[37432] 2026/01/05 13:23:55.942758 [DBG] 127.0.0.1:62831 - mid:6 - "my_dur" - Using email found in cert for auth ["derek@nats.io"]
[37432] 2026/01/05 13:23:55.942811 [INF] Creating MQTT streams/consumers with replicas 1 for account "APP"
[37432] 2026/01/05 13:23:55.950221 [TRC] JETSTREAM - <<- [SUB $JSC.CI.APP.$MQTT_rmsgs.$MQTT_rmsgs_W6X73ytIGaqKw5UynbJDes  59]
[37432] 2026/01/05 13:23:55.950602 [TRC] 127.0.0.1:62831 - mid:6 - "my_dur" - ->> [CONNACK sp=false rc=0]
[37432] 2026/01/05 13:23:55.950608 [DBG] 127.0.0.1:62831 - mid:6 - "my_dur" - Client connected
[37432] 2026/01/05 13:23:55.952660 [TRC] 127.0.0.1:62831 - mid:6 - "my_dur" - <<- [SUBSCRIBE [bridge/alex-garage-1/down/# (bridge.alex-garage-1.down.>) QoS=1] pi=49516]
[37432] 2026/01/05 13:23:55.953447 [TRC] JETSTREAM - <<- [SUB $JSC.CI.APP.$MQTT_msgs.H5qS24fo_W6X73ytIGaqKw5UynbJDkU  60]
[37432] 2026/01/05 13:23:55.954206 [TRC] JETSTREAM - <<- [SUB $JSC.CI.APP.$MQTT_msgs.H5qS24fo_W6X73ytIGaqKw5UynbJDoE  61]
[37432] 2026/01/05 13:23:55.954374 [TRC] 127.0.0.1:62831 - mid:6 - "my_dur" - ->> [SUBACK pi=49516]
[37432] 2026/01/05 13:24:05.314613 [DBG] 127.0.0.1:62832 - mid:29 - Client connection created
[37432] 2026/01/05 13:24:05.314628 [DBG] 127.0.0.1:62832 - mid:29 - Starting TLS client connection handshake
[37432] 2026/01/05 13:24:05.322970 [DBG] 127.0.0.1:62832 - mid:29 - TLS handshake complete
[37432] 2026/01/05 13:24:05.322986 [DBG] 127.0.0.1:62832 - mid:29 - TLS version 1.3, cipher suite TLS_AES_128_GCM_SHA256
[37432] 2026/01/05 13:24:05.323016 [TRC] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - <<- [CONNECT clientID=mqttx_d74bb5a2 keepAlive=45s clean]
[37432] 2026/01/05 13:24:05.323024 [DBG] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - Multiple peer certificates found, selecting first
[37432] 2026/01/05 13:24:05.323033 [DBG] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - Using email found in cert for auth ["ivan@nats.io"]
[37432] 2026/01/05 13:24:05.323381 [TRC] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - ->> [CONNACK sp=false rc=0]
[37432] 2026/01/05 13:24:05.323387 [DBG] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - Client connected
[37432] 2026/01/05 13:24:05.325270 [TRC] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - <<- [PUBLISH bridge/alex-garage-1/down/keys/response dup=false QoS=1 retain=false size=9 pi=51213]
[37432] 2026/01/05 13:24:05.325277 [TRC] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - <<- MSG_PAYLOAD: ["something"]
[37432] 2026/01/05 13:24:05.325442 [TRC] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - ->> [PUBACK pi=51213]
[37432] 2026/01/05 13:24:05.325507 [TRC] 127.0.0.1:62831 - mid:6 - "my_dur" - ->> [PUBLISH bridge/alex-garage-1/down/keys/response dup=false QoS=1 retain=false size=9 pi=1]
[37432] 2026/01/05 13:24:05.326078 [TRC] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - <<- [DISCONNECT]
[37432] 2026/01/05 13:24:05.326107 [DBG] 127.0.0.1:62832 - mid:29 - "mqttx_d74bb5a2" - Client connection closed: Client Closed
[37432] 2026/01/05 13:24:05.326519 [TRC] 127.0.0.1:62831 - mid:6 - "my_dur" - <<- [PUBACK pi=1]

[persinac]

What in world... Those logs and configs are nearly identical from what I was seeing. Ok, my service(s) are working without any changes on 2.11.11. I'll tear the container down and spin up 2.12.3 without any config changes just to double verify.

After that, I'll spin up locally with your config and then look for deltas.

I'll post updates here, but it may take a few days. Would appreciate it if you could keep this open for a bit.

[kozlovic]

@persinac I even changed now to use the include as you have, even the mqtt{} block defined twice and the last with tls{} block on port 8883. I have also tested running the v2.11.3 version and then upgrade to v2.12.3 version without removing the datastore (keep the data created in v2.11.3) and again, everything works fine.

If you reproduce, you may want to check the monitor port for connz and see if you see the connection and if it has subscriptions. Here is the output from my latest run:

{
  "server_id": "NDVAYMWQQE63XMRQHMQFYVQPOQ3PZU7ATW55QGVAJ2T6DBQ6RAOSAKTT",
  "now": "2026-01-06T00:12:07.679017Z",
  "num_connections": 1,
  "total": 1,
  "offset": 0,
  "limit": 1024,
  "connections": [
    {
      "cid": 27,
      "kind": "Client",
      "type": "mqtt",
      "ip": "127.0.0.1",
      "port": 63666,
      "start": "2026-01-05T17:12:00.559191-07:00",
      "last_activity": "2026-01-05T17:12:03.888578-07:00",
      "rtt": "0s",
      "uptime": "7s",
      "idle": "3s",
      "pending_bytes": 0,
      "in_msgs": 0,
      "out_msgs": 0,
      "in_bytes": 0,
      "out_bytes": 0,
      "subscriptions": 4,
      "tls_version": "1.3",
      "tls_cipher_suite": "TLS_AES_128_GCM_SHA256",
      "subscriptions_list": [
        "bridge.alex-garage-1.down.\u003e",
        "$MQTT.sub.vn5cnI7ginh8mB0gz3IlMI",
        "bridge.alex-garage-1.down",
        "$MQTT.sub.vn5cnI7ginh8mB0gz3Ilgk"
      ],
      "mqtt_client": "device@example.com"
    }
  ]
}

As you can see from the http://localhost:8222/connz?subs=1 capture, I have used the mqtt client ID device@example.com and you see the subscriptions in the topic/subject (with wildcard and the one without - as you understand why) and the two $MQTT.sub.xxx subscriptions for the JetStream consumers (since this is a QoS1 subscription).

I can't explain why it works for you on v2.11.x and not on v2.12.3, but I can tell you that on bare metal (on a macOS laptop), it works well.

I will of course leave this ticket open until you are able to run more tests.