Multi-word reads and writes (#29)

Add multi-word register type support: uint32, int32, uint64 and int64.
Co-authored-by: Travis Howse <tjhowse@gmail.com>

Author: icypete

.github/workflows/hacs_action.yml

@@ -55,6 +55,7 @@ update_rate: 5
 address_offset: 0
 variant: sungrow
 scan_batching: 100
+word_order: highlow
 ```
 | Field name | Required | Default | Description |
 | ---------- | -------- | ------- | ----------- |
@@ -64,6 +65,7 @@ scan_batching: 100
 | address_offset | Optional | 0 | This offset is applied to every register address to accommodate different Modbus addressing systems. In many Modbus devices the first register is enumerated as 1, other times 0. See section 4.4 of the Modbus spec. |
 | variant | Optional | N/A | Allows variants of the ModbusTcpClient library to be used. Setting this to 'sungrow' enables support of SungrowModbusTcpClient. This library transparently decrypts the modbus comms with sungrow SH inverters running newer firmware versions. |
 | scan_batching | Optional | 100 | Must be between 1 and 100 inclusive. Modbus read operations are more efficient in bigger batches of contiguous registers, but different devices have different limits on the size of the batched reads. This setting can also be helpful when building a modbus register map for an uncharted device. In some modbus devices a single invalid register in a read range will fail the entire read operation. By setting `scan_batching` to `1` each register will be scanned individually. This will be very inefficient and should not be used in production as it will saturate the link with many read operations. |
+| word_order | Optional | 'highlow' | Must be either `highlow` or `lowhigh`. This determines how multi-word values are interpreted. `highlow` means a 32-bit number at address 1 will have its high two bytes stored in register 1, and its low two bytes stored in register 2. The default is typically correct, as modbus has a big-endian memory structure, but this is not universal. |
 
 ### Register settings
 ```yaml
@@ -97,6 +99,9 @@ registers:
   - pub_topic: "external_temperature"
     address: 13015
     type: int16
+  - pub_topic: "minutes_online"
+    address: 13016
+    type: uint32
 ```
 
 This section of the YAML lists all the modbus registers that you consider interesting.
@@ -113,4 +118,4 @@ This section of the YAML lists all the modbus registers that you consider intere
 | scale | Optional | 1 | After reading a value from the Modbus register it will be multiplied by this scalar before being published to MQTT. Values published on this register's `set_topic` will be divided by this scalar before being written to Modbus. |
 | mask | Optional | 0xFFFF | This is a 16-bit number that can be used to select a part of a Modbus register to be referenced by this register. For example a mask of `0xFF00` will map to the most significant byte of the 16-bit Modbus register at `address`. A mask of `0x0001` will reference only the least significant bit of this register. |
 | json_key | Optional | N/A | The value of this register will be published to its pub_topic in JSON format. E.G. `{ key: value }` Registers with a json_key specified can share a pub_topic. All registers with shared pub_topics must have a json_key specified. In this way, multiple registers can be published to the same topic in a single JSON message. If any of the registers that share a pub_topic have the retain field set that will affect the published JSON message. Conflicting retain settings are invalid. The keys will be alphabetically sorted. |
-| type | Optional | uint16 | The type of the value stored at the modbus address provided. Only uint16 (unsigned 16-bit integer) and int16 (signed 16-bit integer) are currently supported. |
+| type | Optional | uint16 | The type of the value stored at the modbus address provided. Only uint16 (unsigned 16-bit integer), int16 (signed 16-bit integer), uint32, int32, uint64 and int64 are currently supported. |

.github/workflows/hassfest.yml

@@ -0,0 +1,61 @@
+ip: 192.168.0.xxx
+port: 502
+update_rate: 5
+address_offset: 0
+variant: sungrow
+scan_batching: 1
+registers:
+  - pub_topic: "energy_meter_power" # Feed-in power is negative  and taken-back power is positive (W)
+    address: 5082
+    table: 'input'
+    type: int32
+  - pub_topic: "output_power" #total output power kWh
+    type: uint16
+    address: 5000
+    table: 'input'
+    scale: 0.1
+  - pub_topic: "daily_yield"  #daily yield kWh
+    address: 5002
+    table: 'input'
+    scale: 0.1
+  - pub_topic: "total_yield"  #Total yield kWh
+    address: 5003
+    table: 'input'
+  - pub_topic: "total_running_time"  #Total running time (h)
+    address: 5005
+    table: 'input'
+  - pub_topic: "internal_temperature" #inverter internal temperature 0.1C
+    address: 5007
+    table: 'input'
+    scale: 0.1
+  - pub_topic: "dc_output"  #dc output power (W)
+    address: 5016
+    table: 'input'
+  - pub_topic: "phase_a_voltage" #Phase A Voltage (0.1V)
+    address: 5018
+    table: 'input'
+    scale: 0.1
+  - pub_topic: "phase_a_current" #Phase A Current (0.1A)
+    address: 5021
+    table: 'input'
+    scale: 0.1
+  - pub_topic: "ac_output" #AC output power, total active power (W)
+    address: 5030
+    table: 'input'
+  - pub_topic: "power_factor" #Power factor (0.001)
+    address: 5034
+    table: 'input'
+    scale: 0.001
+  - pub_topic: "grid_frequency" #Grid Frequency (0.1Hz)
+    address: 5035
+    table: 'input'
+    scale: 0.1
+  - pub_topic: "device_state" #Device State  (see comments below for states)
+    address: 5037
+    table: 'input'
+  - pub_topic: "daily_running_time" #Daily running time (1m)
+    address: 5112
+    table: 'input'
+
+    #see https://github.com/tjhowse/modbus4mqtt/files/5732710/TI_20190704_Communication.Protocol.for.Residential.Single-phase.Grid-Connected.Inverters_V10_EN.pdf for full list of registers and details
+

README.md

@@ -42,11 +42,17 @@ def connect(self):
         self.connect_mqtt()
 
     def connect_modbus(self):
+        if self.config.get('word_order', 'highlow').lower() == 'lowhigh':
+            word_order = modbus_interface.WordOrder.LowHigh
+        else:
+            word_order = modbus_interface.WordOrder.HighLow
+
         self._mb = modbus_interface.modbus_interface(self.config['ip'],
                                                      self.config.get('port', 502),
                                                      self.config.get('update_rate', 5),
                                                      variant=self.config.get('variant', None),
-                                                     scan_batching=self.config.get('scan_batching', None))
+                                                     scan_batching=self.config.get('scan_batching', None),
+                                                     word_order=word_order)
         failed_attempts = 1
         while self._mb.connect():
             logging.warning("Modbus connection attempt {} failed. Retrying...".format(failed_attempts))
@@ -59,7 +65,7 @@ def connect_modbus(self):
             sleep(self.modbus_reconnect_sleep_interval)
         # Tells the modbus interface about the registers we consider interesting.
         for register in self.registers:
-            self._mb.add_monitor_register(register.get('table', 'holding'), register['address'])
+            self._mb.add_monitor_register(register.get('table', 'holding'), register['address'], register.get('type', 'uint16'))
             register['value'] = None
 
     def modbus_connection_failed(self):
@@ -96,16 +102,18 @@ def poll(self):
 
         for register in self._get_registers_with('pub_topic'):
             try:
-                value = self._mb.get_value(register.get('table', 'holding'), register['address'])
+                value = self._mb.get_value( register.get('table', 'holding'),
+                                            register['address'],
+                                            register.get('type', 'uint16'))
             except Exception:
                 logging.warning("Couldn't get value from register {} in table {}".format(register['address'],
                                 register.get('table', 'holding')))
                 continue
             # Filter the value through the mask, if present.
-            value &= register.get('mask', 0xFFFF)
-            # Tweak the value according to the type.
-            type = register.get('type', 'uint16')
-            value = modbus_interface._convert_from_uint16_to_type(value, type)
+            if 'mask' in register:
+                # masks only make sense for uint
+                if register.get('type', 'uint16') in ['uint16', 'uint32', 'uint64']:
+                    value &= register.get('mask')
             # Scale the value, if required.
             value *= register.get('scale', 1)
             # Clamp the number of decimal points
@@ -187,9 +195,8 @@ def _on_message(self, client, userdata, msg):
                               "Bad/missing value_map? Topic: {}, Value: {}".format(topic, value))
                 continue
             type = register.get('type', 'uint16')
-            value = modbus_interface._convert_from_type_to_uint16(value, type)
             self._mb.set_value(register.get('table', 'holding'), register['address'], int(value),
-                               register.get('mask', 0xFFFF))
+                               register.get('mask', 0xFFFF), type)
 
     # This throws ValueError exceptions if the imported registers are invalid
     @staticmethod
@@ -200,7 +207,7 @@ def _validate_registers(registers):
         duplicate_json_keys = {}
         # Key: shared pub_topics, value: set of retain values (true/false)
         retain_setting = {}
-        valid_types = ['uint16', 'int16']
+        valid_types = ['uint16', 'int16', 'uint32', 'int32', 'uint64', 'int64']
 
         # Look for duplicate pub_topics
         for register in registers:

modbus4mqtt/SG8K-D.yaml

@@ -1,4 +1,5 @@
 from time import time, sleep
+from enum import Enum
 import logging
 from queue import Queue
 from pymodbus.client.sync import ModbusTcpClient, ModbusSocketFramer
@@ -13,9 +14,13 @@
 DEFAULT_WRITE_SLEEP_S = 0.05
 DEFAULT_READ_SLEEP_S = 0.05
 
+class WordOrder(Enum):
+    HighLow = 1
+    LowHigh = 2
+
 class modbus_interface():
 
-    def __init__(self, ip, port=502, update_rate_s=DEFAULT_SCAN_RATE_S, variant=None, scan_batching=None):
+    def __init__(self, ip, port=502, update_rate_s=DEFAULT_SCAN_RATE_S, variant=None, scan_batching=None, word_order=WordOrder.HighLow):
         self._ip = ip
         self._port = port
         # This is a dict of sets. Each key represents one table of modbus registers.
@@ -29,6 +34,7 @@ def __init__(self, ip, port=502, update_rate_s=DEFAULT_SCAN_RATE_S, variant=None
         self._writing = False
         self._variant = variant
         self._scan_batching = DEFAULT_SCAN_BATCHING
+        self._word_order = word_order
         if scan_batching is not None:
             if scan_batching < MIN_SCAN_BATCHING:
                 logging.warning("Bad value for scan_batching: {}. Enforcing minimum value of {}".format(scan_batching, MIN_SCAN_BATCHING))
@@ -53,11 +59,14 @@ def connect(self):
                                        framer=ModbusSocketFramer, timeout=1,
                                        RetryOnEmpty=True, retries=1)
 
-    def add_monitor_register(self, table, addr):
+    def add_monitor_register(self, table, addr, type='uint16'):
         # Accepts a modbus register and table to monitor
         if table not in self._tables:
             raise ValueError("Unsupported table type. Please only use: {}".format(self._tables.keys()))
-        self._tables[table].add(addr)
+        # Register enough sequential addresses to fill the size of the register type.
+        # Note: Each address provides 2 bytes of data.
+        for i in range(type_length(type)):
+            self._tables[table].add(addr+i)
 
     def poll(self):
         # Polls for the values marked as interesting in self._tables.
@@ -79,19 +88,42 @@ def poll(self):
                     start = group + self._scan_batching-1
         self._process_writes()
 
-    def get_value(self, table, addr):
+    def get_value(self, table, addr, type='uint16'):
         if table not in self._values:
             raise ValueError("Unsupported table type. Please only use: {}".format(self._values.keys()))
         if addr not in self._values[table]:
             raise ValueError("Unpolled address. Use add_monitor_register(addr, table) to add a register to the polled list.")
-        return self._values[table][addr]
+        # Read sequential addresses to get enough bytes to satisfy the type of this register.
+        # Note: Each address provides 2 bytes of data.
+        value = bytes(0)
+        # TODO Make HighLow LowHigh word ordering configurable for multi-register reads.
+        type_len = type_length(type)
+        for i in range(type_len):
+            if self._word_order == WordOrder.HighLow:
+                data = self._values[table][addr + i]
+            else:
+                data = self._values[table][addr + (type_len-i-1)]
+            value += data.to_bytes(2,'big')
+        value = _convert_from_bytes_to_type(value, type)
+        return value
 
-    def set_value(self, table, addr, value, mask=0xFFFF):
+    def set_value(self, table, addr, value, mask=0xFFFF, type='uint16'):
         if table != 'holding':
             # I'm not sure if this is true for all devices. I might support writing to coils later,
             # so leave this door open.
             raise ValueError("Can only set values in the holding table.")
-        self._planned_writes.put((addr, value, mask))
+
+        bytes_to_write = _convert_from_type_to_bytes(value, type)
+        # Put the bytes into _planned_writes stitched into two-byte pairs
+
+        type_len = type_length(type)
+        for i in range(type_len):
+            if self._word_order == WordOrder.HighLow:
+                value = _convert_from_bytes_to_type(bytes_to_write[i*2:i*2+2], 'uint16')
+            else:
+                value = _convert_from_bytes_to_type(bytes_to_write[(type_len-i-1)*2:(type_len-i-1)*2+2], 'uint16')
+            self._planned_writes.put((addr+i, value, mask))
+
         self._process_writes()
 
     def _process_writes(self, max_block_s=DEFAULT_WRITE_BLOCK_INTERVAL_S):
@@ -144,22 +176,33 @@ def _scan_value_range(self, table, start, count):
             # The result doesn't have a registers attribute, something has gone wrong!
             raise ValueError("Failed to read {} {} table registers starting from {}: {}".format(count, table, start, result))
 
-def _convert_from_uint16_to_type(value, type):
+def type_length(type):
+    # Return the number of addresses needed for the type.
+    # Note: Each address provides 2 bytes of data.
+    if type in ['int16', 'uint16']:
+        return 1
+    elif type in ['int32', 'uint32']:
+        return 2
+    elif type in ['int64', 'uint64']:
+        return 4
+    raise ValueError ("Unsupported type {}".format(type))
+
+def type_signed(type):
+    # Returns whether the provided type is signed
+    if type in ['uint16', 'uint32', 'uint64']:
+        return False
+    elif type in ['int16', 'int32', 'int64']:
+        return True
+    raise ValueError ("Unsupported type {}".format(type))
+
+def _convert_from_bytes_to_type(value, type):
     type = type.strip().lower()
-    if type == 'uint16':
-        return value
-    elif type == 'int16':
-        if value >= 2**15:
-            return value - 2**16
-        return value
-    raise ValueError("Unrecognised type conversion attempted: uint16 to {}".format(type))
+    signed = type_signed(type)
+    return int.from_bytes(value,byteorder='big',signed=signed)
 
-def _convert_from_type_to_uint16(value, type):
+def _convert_from_type_to_bytes(value, type):
     type = type.strip().lower()
-    if type == 'uint16':
-        return value
-    elif type == 'int16':
-        if value < 0:
-            return value + 2**16
-        return value
-    raise ValueError("Unrecognised type conversion attempted: {} to uint16".format(type))
+    signed = type_signed(type)
+    # This can throw an OverflowError in various conditons. This will usually
+    # percolate upwards and spit out an exception from on_message.
+    return int(value).to_bytes(type_length(type)*2,byteorder='big',signed=signed)

modbus4mqtt/modbus4mqtt.py

@@ -1 +1 @@
-version = "0.5.1"
+version = "0.6.0"