AqualinkD: Nothing read on blocking serial port

[matthewhadley]

I have been using AqualinkD successfully for a few years - installed on a Raspberry Pi 4 using the suggested USB to RS485 Serial Hardware.

At some point at least a year ago I noticed that the pool pump would randomly turn itself off. This happened enough that I wrote a Home Assistant automation to restart the pump when it happened. I recently correlated instances of this happening to seeing this message in the aqualinkd logs:

Jan 11 11:39:51 raspberrypi aqualinkd[15514]: AqualinkD: Nothing read on blocking serial port
Jan 11 11:39:51 raspberrypi aqualinkd[15514]: AqualinkD: Aqualink daemon looks like serial error, resetting.

When this reset happens, any running equipment is shut off.

I have managed to capture log output of this happening while logging in DEBUG_SERIAL. The attached log excerpt has the error at line 4275, timestamp Jan 11 11:39:51
syslog_serial_debug.txt

Also attached is my aqualinkd.conf
aqualinkd.conf

In terms of box setup, I used to have an iaqualink 2.0 attached to the system, but have removed that and only use AqualinkD to control the system

The usb to serial is connected to the top left connector shown in the image (yellow and orange wires) and the Pi is the enclosure at the bottom of the box:

I used to have the aqualinkD connected on the right initially (with the iAqaulink 2.0 on the left). The system has a variable speed pump, with the speed values controlled by the relays. The system also has a SWG and similar to the pump that is just controlled as ON or OFF via a relay.

If it's useful, here's the output of serial_logger

./release/serial_logger /dev/ttyUSB0
AqualinkD serial_logger V2.7
Notice:  RS Serial: Port /dev/ttyUSB0 low latency mode is set
Notice:  Serial Log:Logging serial information!
Please wait.|

Notice:  Serial Log:RS485 interface received 800 packets in 61 seconds ( 13.11 Msg/Sec)
Notice:  Serial Log:Jandy Control Panel Model   :  RS-8 Combo
Notice:  Serial Log:Jandy Control Panel Version :  REV T.2
Notice:  Serial Log:Jandy ID's found
Notice:  Serial Log:ID 0x60 is not used  <-- can use for Aqualinkd (PDA mode only)
Notice:  Serial Log:ID 0x08 is not used  <-- can use for Aqualinkd
Notice:  Serial Log:ID 0x30 is not used  <-- can use for Aqualinkd (Prefered Extended Device ID)
Notice:  Serial Log:ID 0x31 is not used  <-- can use for Aqualinkd (Prefered Extended Device ID)
Notice:  Serial Log:ID 0x32 is not used  <-- can use for Aqualinkd (Prefered Extended Device ID)
Notice:  Serial Log:ID 0x33 is not used  <-- can use for Aqualinkd (Prefered Extended Device ID)
Notice:  Serial Log:ID 0x48 is not used  <-- can use for Aqualinkd (RSSA ID)
Notice:  Serial Log:ID 0x09 is not used  <-- can use for Aqualinkd
Notice:  Serial Log:ID 0x0a is not used  <-- can use for Aqualinkd
Notice:  Serial Log:ID 0x0b is not used  <-- can use for Aqualinkd
Notice:  Serial Log:ID 0x40 is not used  <-- can use for Aqualinkd (Extended Device ID)
Notice:  Serial Log:ID 0x41 is not used  <-- can use for Aqualinkd (Extended Device ID)
Notice:  Serial Log:ID 0x42 is not used  <-- can use for Aqualinkd (Extended Device ID)
Notice:  Serial Log:ID 0x43 is not used  <-- can use for Aqualinkd (Extended Device ID)
Notice:  Serial Log:

Notice:  Serial Log:Suggested aqualinkd.conf values
Notice:  Serial Log:-------------------------
Notice:  Serial Log:panel_type =  RS-8 Combo
Notice:  Serial Log:device_id = 0x08
Notice:  Serial Log:rssa_device_id = 0x48
Notice:  Serial Log:extended_device_id = 0x30
Notice:  Serial Log:-------------------------

I am looking for any thoughts/help on how to fix the serial problem that is causing the resets. It can happen a few times a day, or sometimes not at all. I have a backup USB to RS485 Serial device that I can try switching in...

Thanks for any help!

[sfeakes]

I can think of a few reasons why this would be happening. (From least lightly to most lightly)

• 1. The USB adapter has a issue / or OS is resetting it for some reason
• 2. The RS485 connection / wiring
• 3. The Pi I/O is overloaded
• 4. The Pi has seen an undervolt issue and disconnected the USB

First check if the OS is disconnecting / reconnecting the USB for some reason. There should be some messages in syslog about USB/TTY disconnect / reconnect. Depending on the OS version is where they will be written /var/log/message, /var/log/syslog, or journald

Their is a script in the extras directory of AqualinkD repo called pi_health.sh, if you run that it'll tell you if the Pi is currently in / or has seen an undervolt, throttled, frequency capped issue. If you get anything other than OK for any of those states, then check you PSU, as that's probably the issue.

Pi's 1,2,3 & zero all use 1 chip for all I/O, so USB / Network / CF etc, and it's very easy to hit the limits of that IO. So if you have anything else running on the Pi, or are noticing any other form of latency, start turning services off.

On the RS485 wiring, I would expect to also see a few checksum errors if this was the root cause, and I didn't notice that in the logs, so I'm not sure it's that. BUT, what's interesting is that a device will turn off when you get this problem, that usually means AqualinkD (or something) has sent a message out of sequence and the control panel is confused. So that means that AqualinkD may have been trying to send a message when the issue happened. Since RS485 is half duplex that means that the USB RS485 adapter & the OS has to switch from write to read as it can't do both at the same time. Some cheaper chips are not good at this, so again may be the adapter or something to do with IO being busy.

I'll go over the logs in greater detail later today if I get time and port an update if I notice anything different.

BTW, since you don't have iaqualink connected, you can enable the below in AqualinkD.conf, this will increase the RS485 messages, it would be interesting to see if the problem get's worse or stays about the same.

extended_device_id=0x31
enable_iaqualink=yes

[matthewhadley]

Thanks for the suggestions @sfeakes

I have the Pi 4 running with a read only disk. When the logger is in DEBUG_SERIAL that memory only disk fills up fast and I end up rebooting to start over. This just to say I need to wait for the issue to re-trigger and get some new logs. I will make the config changes you recommend and monitor.

I have previously run the health check in extras and everything looked good. The load on the machine is low - just aqualinkd running and a health check software.

[matthewhadley]

Something I quickly noticed with these values set:

extended_device_id=0x31
enable_iaqualink=yes

Is that the pump controls weren't doing what I expected - with the LOW relay on and trying to turn to MEDIUM (so the operation is turn the MEDIUM relay on and the LOW off) was instead toggling both of them off (which results in HIGH speed). I didn't fully monitor it before I decided to just deactivate those settings in the conf. It seemed like there was some competing commands to control the relays. I can't fully explain it. I did capture some logs:

syslog.txt

It could be somehow to do with how I program the MQTT commands in home assistant, but it's definitely a different behavior with those settings enabled.