New firmware breaks MQTT

Projects related to Solar Power
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

New firmware breaks MQTT

Post by JohnH »

Discovered the unit updated automatically sometime between 1/22/23 - 1/25/23, and the mqtt settings had been reset to blank entries. I put back in the topic and server address, and data started flowing again after a reboot of the device, but at 8:02 this morning it stopped updating again. Deleting the MQTT settings, rebooting, setting it back up for MQTT, then rebooting again does not correct the issue. The dashboard looks to be functioning normally.

I changed the server from hostname to IP address and did not resolve. I also changed the topic name from coop, to either /coop, coop/solar, and /coop/solar, and also could not resolve. There are over 10 different devices that are publishing to the same MQTT server without issue.

Version: 5.0-0dc546-2023-01-22-20-34
Update Channel: Stable
Hardware: v4 w/external WiFi antenna (2021)

Here is the log snippet where the failure starts:

Code: Select all

[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 13:58:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:58:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:58:55
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:59:05
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:59:15
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:59:25
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:59:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 13:59:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:59:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 13:59:55
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:00:05
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:00:15
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:00:25
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:00:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 14:00:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:00:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:00:55
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:01:05
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:01:15
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:01:25
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:01:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 14:01:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:01:45
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:01:55
[mqtt_client]: publishing data to topic: coop  OK 26-01-2023 14:02:05
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:02:15
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:02:25
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:02:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 14:02:45
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:02:45
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:02:55
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:03:05
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:03:15
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:03:25
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:03:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 14:03:45
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:03:45
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:03:55
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:04:05
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:04:15
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:04:25
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:04:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 14:04:45
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:04:45
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:04:55
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:05:05
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:05:15
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:05:25
[mqtt_client]: publishing data to topic: coop  Failed 26-01-2023 14:05:35
[sd_card]: Wrote Sample: 2023/01/26/data.txt 26-01-2023 14:05:45
snekTek.png
snekTek.png (22.09 KiB) Viewed 41533 times
Last edited by JohnH on Sat Jan 28, 2023 9:32 am, edited 1 time in total.
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

Re: New firmware breaks MQTT

Post by JohnH »

Seems to be failing at the same time every day since the firmware upgrade.

Here is the snippet from today:

Code: Select all

[sd_card]: Wrote Sample: 2023/01/27/data.txt 27-01-2023 14:00:20
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:00:20
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:00:30
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:00:40
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:00:50
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:01:00
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:01:10
[sd_card]: Wrote Sample: 2023/01/27/data.txt 27-01-2023 14:01:20
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:01:20
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:01:30
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:01:40
[mqtt_client]: publishing data to topic: coop  OK 27-01-2023 14:01:50
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:02:00
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:02:10
[sd_card]: Wrote Sample: 2023/01/27/data.txt 27-01-2023 14:02:20
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:02:20
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:02:30
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:02:40
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:02:50
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:03:00
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:03:10
[sd_card]: Wrote Sample: 2023/01/27/data.txt 27-01-2023 14:03:20
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:03:20
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:03:30
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:03:40
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:03:50
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:04:00
[mqtt_client]: publishing data to topic: coop  Failed 27-01-2023 14:04:10
[sd_card]: Wrote Sample: 2023/01/27/data.txt 27-01-2023 14:04:20
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

Re: New firmware breaks MQTT

Post by JohnH »

Two days in a row tracking the issue; it is failing to publish mqtt at the same time both days, and automatically resumes the same time both days...

Code: Select all

[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:02:28
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:02:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:02:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:02:48
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:02:58
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:03:08
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:03:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:03:28
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:03:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:03:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:03:48
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:03:58
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:04:08
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:04:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:04:28
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:04:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:04:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:04:48
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:04:58
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:05:08
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:05:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:05:28
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:05:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:05:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:05:48
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:05:58
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:06:08
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:06:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:06:28
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:06:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:06:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:06:48
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:06:58
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:07:08
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:07:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:07:28
[clock]: RTC epoch: 1674864448 28-01-2023 00:07:28
[clock]: NTP epoch 1674864449 28-01-2023 00:07:29
[clock]: NTP epoch 1674864449 28-01-2023 00:07:29
[clock]: NTP updated RTC 28-01-2023 00:07:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:07:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:07:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:07:48
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:07:58
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:08:08
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:08:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:08:28
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:08:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:08:38
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:08:48
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:08:58
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:09:08
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:09:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:09:28
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:09:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:09:38
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:09:48
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:09:58
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:10:08
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:10:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:10:28
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:10:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:10:38
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:10:48
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 00:10:58
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:11:08
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:11:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:11:28
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:11:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:11:38
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:11:48
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:11:58
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:12:08
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 00:12:18
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 00:12:28
This is what it looks like working.
snekTek.png
snekTek.png (10.58 KiB) Viewed 41520 times
Off work today so I will monitor it live until it stops publishing to MQTT.
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

Re: New firmware breaks MQTT

Post by JohnH »

Caught it live. 3rd day in a row failing at exactly the same time.

Last measured MQTT values:
Solar: 14v
Battery: 12.6v
Load W: 1w
Load A: 0.08a
SoC: 74%

Other data from dashboard @ time of failure:
Internal Voltage: 3.486v
Controller Temp: 55.400000000000006F (lol)
SoC: 100%
Battery: 13.8v
Battery C: 0.77a
Load W: 1w
Load A: 0.09a

Code: Select all

[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 14:00:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:00:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:00:39
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:00:49
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:00:59
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:01:09
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:01:19
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 14:01:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:01:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:01:39
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:01:49
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:01:59
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:02:09
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:02:19
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 14:02:29
[mqtt_client]: publishing data to topic: coop  OK 28-01-2023 14:02:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:02:39
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:02:49
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:02:59
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:03:09
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:03:19
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 14:03:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:03:29
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:03:39
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:03:49
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:03:59
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:04:09
[mqtt_client]: publishing data to topic: coop  Failed 28-01-2023 14:04:19
[sd_card]: Wrote Sample: 2023/01/28/data.txt 28-01-2023 14:04:29
Dashboard is still updating properly while MQTT is failing.

TLDR;
MQTT is failing every day from 8:02am until 6:03pm every day, while the dashboard continues to update properly. The battery SoC is not in the "discharge danger" range at the time of failure and continues to charge properly per the dashboard.
My guess is if there isn't a coding bug that causes a failure based on the clock, it must be caused by the fact that the Sun is just starting to provide enough energy to charge the battery, and at ~14v pV and/or ~13v bV is triggering a bug. That must be why it is failing when the voltages get to that point, and when it automatically resumes when they fall below that point. Attached is the image from Grafana showing the failure/recovery trend. Disregard the horizontal red line as it is my low battery voltage alarm threshold. Also reference the purple line. That is the W/m2 from my weather station and you can see that it seems to support my theory stated above.
Attachments
snekTek.png
jonathan
Posts: 237
Joined: Thu Feb 01, 2018 1:12 am

Re: New firmware breaks MQTT

Post by jonathan »

Hi John thanks for looking into this and putting together all the information, I think this is a side effect of another bug that I discovered in epever models, more to come

.. this bug causes negative values to be recorded when solar watts goes above around 320W. Does this line up with your data?
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

Re: New firmware breaks MQTT

Post by JohnH »

Matching the failure start and the dashboard, it happened today at 14v pV and 0.93w pW. MQTT stops right when the snekTek dashboard shows the charge controller starts charging the battery. It is a 100w panel connected to a Renogy Wanderer.

Attached are the Grafana/MQTT and snekTek dashboards for today:
Attachments
snekTek.png
jonathan
Posts: 237
Joined: Thu Feb 01, 2018 1:12 am

Re: New firmware breaks MQTT

Post by jonathan »

JohnH wrote:
Mon Jan 30, 2023 8:18 pm
Matching the failure start and the dashboard, it happened today at 14v pV and 0.93w pW. MQTT stops right when the snekTek dashboard shows the charge controller starts charging the battery. It is a 100w panel connected to a Renogy Wanderer.

Attached are the Grafana/MQTT and snekTek dashboards for today:
Thanks John. I am testing a firmware that fixes the problem. The issue was related to a buffer that was big enough for low numbers but not big enough to contain many double digit numbers.
jonathan
Posts: 237
Joined: Thu Feb 01, 2018 1:12 am

Re: New firmware breaks MQTT

Post by jonathan »

Hi John, if you'd like to try the new firmware, change your update channel to edge. Only do it if you're comfortable reflashing the board if something goes wrong.
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

Re: New firmware breaks MQTT

Post by JohnH »

I changed it
JohnH
Posts: 20
Joined: Sat Jan 09, 2021 1:14 pm

Re: New firmware breaks MQTT

Post by JohnH »

Although the edge version shows

Code: Select all

4.0-dev-2020-01-06-02-38
, the dashboard has all of the newer graphical elements. MQTT is working again though. Thank you.
Post Reply