Trigger emails not arriving

Thank you for taking the time to submit your bug/issue! Please use the points below as a guide when submitting.

  • Device & model you are using (Ex: Pi 2 Model B or Arduino Uno with W5100 ethernet shield)

WeMos D1 R2 with RS485 interface probing Hiking electricity meter.

  • What dashboard are you using? (Web, iOS, Android)

Web.

  • Please describe the bug / issue as detailed as possible. Attaching the code and any relevant screenshots would be very helpful!

This project has been running since April 2018 capturing mains voltage, current, frequency and accumulated kWh and relaying those metrics to a Cayenne dashboard. I set a trigger on the voltage metric to send an email whenever voltage exceeds 253V. Those emails have been arriving without hassle since I set this up (also some time in the first half of 2018).

Recently (days) I realised that I had not been getting alerts even though the dashboard chart shows plenty of occasions when voltage has exceeded the threshold:

I have clicked on one of the observations as the chart gets into the >253V range both to show that metrics are being sent into Cayenne and that some of those are over the trigger threshold.

The trigger is still there and claims to be sending emails:

but no emails have been arriving.

I sent a test message to the target email account (from outside the domain of the target account so that it was a fair test). The test message arrived within seconds, so the target email account is “reachable” from the wider Internet.

The logical conclusion seems to be that emails from Cayenne are getting “stuck” somewhere. Maybe Cayenne’s MTA has fallen over? Maybe Cayenne is on a block list somewhere.

Further to my earlier report, I have done some more digging and I think I have been able to narrow down a small time-window when whatever-it-is that is causing trigger emails to not arrive started happening.

The last notification email I successfully received was timestamped by my ISP’s mail server as “8 October 2019 09:52:51 PM CDT”.

My understanding is CDT is UTC-5 so, unless I’ve got my calculations wrong, that is 2019-10-09T02:52:51Z, which is 13:52:51 my time (AEDT UTC+11) on 9 Oct. I have been able to find that period in the dashboard:

Scrolling forward through the timeline reveals that the next over-voltage incident occurred at 14:18 (my time), which is 03:18UTC:

No email arrived for that incident so that’s the first missing email.

To summarise: It looks like whatever is causing this problem started on 9 October 2019 sometime between 02:52 and 03:18 UTC. I hope this helps you track it down.

can you try deleting and setting up a new trigger. Also are you sending data in the rate limit Release Notes September 13, 2016 (Android app update). it is not applicable for email, but a better approach is send data only once.

Well, I read through that link but nothing seems particularly relevant to the problem at hand. Are you sure it’s the right link?

But, to try to answer the questions I think you might be asking, I am not using an Android app for anything so if the question is Android-specific, the answer is “no”.

If the question is about sending too much data too fast, the data is being sent via the CayenneMQTTESP8266.h API in the WeMos D1 R2 module, so the update frequency is whatever that API implements. I just provide the data when CAYENNE_OUT_DEFAULT is invoked. I’d also make the point that this part of the sketch has been unchanged since I first set it up 18 months ago. That well and truly post-dates the date on your release notes link so, if the API had been sending data too fast circa 2016, I suspect that would have been fixed by the time I started using the API in 2018 (yes/no?). More to the point, if my sketch sending too much data too fast was the cause of the current problem with email notifications then surely it would’ve surfaced long before now? Basically, whichever way I consider this data-rate issue, I can’t wrap my brain around why it would suddenly present itself after all this time.

The answer to the “what’s changed?” question really is “nothing on my side”. I also haven’t done anything on the Cayenne server side except login to view the charts.

The situation is exactly as stated in my bug report. Alerts have been working, day in, day out since April 2018, give or take. A few days ago I notice that I’m not getting any alerts - but I can see evidence in the graphs that alerts should be being sent. Delving into the trigger shows it has been firing. Logically, that leaves three possibilities: (1) the trigger is firing but the email is not being produced (a back-end API problem), (2) the email is being produced but is being queued somewhere (a sender-side MTA issue), (3) the email is being sent but is not being received at my end (a receiver-side MTA or mailbox issue). I have been able to send test messages to the target mailbox from a third domain (ie two MTAs are involved in different parts of the Internet). To my way of thinking that eliminates the receiving MTA and the target mailbox as likely causes. So we’re back to (1) or (2) as the logical candidates.

I can try deleting and re-instantiating the trigger. If that solves the problem, it’s great from my perspective. The reason I did not try that at the get-go is because clobbering the old trigger is likely to remove the evidence that will be needed to nail down the cause and prevent it from happening again.

Here’s an updated screen shot from the trigger display. The count shows that another 17 alerts have gone walkabout since my original report.

sorry for sending the link, but not sure how that happened. i wanted you to go through this topic Sending MQTT messages within notification limit

can you give a try deleting and readding a new trigger.

My actual code is:

CAYENNE_OUT_DEFAULT() {

    // has the current request achieved a value state?
    if (meterRequests[cayenneMeterRequestIndex].hasCurrentValue()) {

        // retrieve the channel (-1 is a sentinel meaning "do not log")
        int cayenneChannel = meterRequests[cayenneMeterRequestIndex].cayenneChannel();

        // should this metric be sent to Cayenne?
        if (cayenneChannel > 0) {

            // yes! 
            const char* cayenneDataType = meterRequests[cayenneMeterRequestIndex].cayenneDataType();
            const char* cayenneUnits = meterRequests[cayenneMeterRequestIndex].cayenneUnits();
    
            // this is the value to log
            float digestValue = meterRequests[cayenneMeterRequestIndex].referenceValue();
    
            // log the value
            Cayenne.virtualWrite(cayenneChannel, digestValue, cayenneDataType, cayenneUnits);

        }

    }

    // advance to the next request (unconditional)
    cayenneMeterRequestIndex = (cayenneMeterRequestIndex + 1) % numberOfMeterRequests;

}

hasCurrentValue() guards against CAYENNE_OUT_DEFAULT being invoked before the sketch has had time to acquire metrics from the electricity meter. There are four items in the metrics array (voltage, current, frequency and accumulated kWh) and each invocation of CAYENNE_OUT_DEFAULT sends exactly one of those at a time. In other words, if CAYENNE_OUT_DEFAULT sends a voltage measurement for time period T(n) then another voltage measurement will not be sent until time period T(n+4).

On the topic of trigger values and, more specifically, the sendTriggerValue() API call mentioned in the link you provided, I am not using that. I set up the trigger in the Cayenne web interface, loosely paraphrased as “if voltage exceeds 253 then send an email”.

Now I’d like to turn to the idea of a 90-per-day notification limit. I do not recall reading about such a limit before but now that I am aware of it, I’m quite happy with the idea. If grid voltage in my part of the world in any given 24 hour period gets so bad that Cayenne needs to stop after 90 emails then the first 90 will already have given me plenty of notice that my electricity provider is up to no good. If I then have to wait until either midnight or +24 hours before emails start coming in again, so be it.

But let’s test the hypothesis that hitting the 90-per-day notification limit is the cause of the emails going walkabout. The first screen shot in this thread that shows the trigger details records the trigger having fired 10172 times since it was created with a “last run” of “10/13/19 1:33pm”. The corresponding values from the second screen shot are 10189 trigger events with a last run of 10/14/19 3:12pm. That’s 17 in just over a day. Nowhere near the limit of 90. Now let’s drill into the detail:

Ignoring, for the moment, that the last successfully received email was on 8 October, let’s just assume that the Oct 13 1:33:14 trigger was the 90th of a sequence and caused the notification limit to come into effect. The question becomes, does the notification limit reset at midnight or is it a rolling 24 hours?

If it resets at midnight then all 17 triggers on Oct 14 should have resulted in emails. If it is a rolling 24 hour period then that expired at 1:33:14 on Oct 14 so the three triggers after that should have resulted in emails.

It seems to me that, no matter how this particular cake is cut, it leads back to the conclusion that somewhere between 3 and 17 trigger firings should have resulted in emails arriving. They didn’t, so I don’t see that there is any evidence to support the hypothesis that the 90-a-day limit is the explanation. I think it is far more likely that the problem will be traced to the email delivery system (which is why I keep banging-on about it).

Having said all that, I have created a new trigger via the web UI. Grid voltage has not exceeded 253 since I did that so I will have to wait until my electricity distributor starts sending me “dirty power” again before I will be able to say whether (a) the new trigger has fired and, if so (b) whether an email has arrived.

Watch this space…

it would be better if you share the entire code. Let me know whether the new trigger is working.

Well, the sketch is 3300 lines spread across 12 files so, no, the whole thing ain’t gonna get posted here.

Meanwhile, I have had to wait for my electricity supplier to send me some “dirty power”. Murphy’s Law pretty much guaranteed that that wasn’t going to happen on cue. But it just has.

Rather than use the Cayenne time-series chart, I thought I’d show you my Grafana chart because it’s a more spectacular demonstration of a voltage spike:

The sketch probes the electricity meter every second, sequencing through the same four metrics (voltage, current, frequency and accumulated kWh). At ten-second intervals it logs the most-recent set of four metrics to a local Influx database. This is subtly different to Cayenne where only one metric of the four is sent every time CAYENNE_OUT_DEFAULT is invoked. Or, to put it another way, Influx has better granularity:

> select voltage from hiking where time > '2019-10-15T13:00:00.000Z' and voltage > 253.0;
name: hiking
time                        voltage
----                        -------
2019-10-15T20:18:24.61879Z  253.1
2019-10-15T20:18:34.837778Z 253.2
2019-10-15T20:18:45.014596Z 253.5
2019-10-15T20:18:55.202915Z 253.4
2019-10-15T20:19:05.357465Z 253.4
> 

Because the timespan is under a minute I would only expect one of those would be guaranteed to reach Cayenne. It turns out that two did because the NEW trigger fired twice:

If the trigger was “working end-to-end” in the sense of that producing an email, it didn’t. The inbox is:

The first two lines are my test messages when I was making sure the mailbox could receive from another domain. The last four lines are copies of your replies to this ticket.

Apart from anything else, that the inbox contains copies of your thread replies also demonstrates that the receiving MTA and clients are working. It’s yet more evidence that the true problem is on the Cayenne server side between the trigger firing and email generation. I just have no means of figuring out whether the trigger is producing the email but the email is getting stuck somewhere, or whether the trigger is firing but just not producing the email.

Bottom line: Brand new trigger. Trigger condition met twice. Trigger fired twice. No emails.

can you private message me your email id used for cayenne acoount. i will check what is the issue.

Done.

Not really an ideal solution but a decent workaround. You can send email through your client (WeMos). I found this method much reliable than depending on the cayenne server to send an email. You will need email library for this purpose though. If interested have a look at these links,

SMS notification: You can add your phone number to get notified along with email notification. Edit your trigger, add more recipients → then add your phone number starting with 011+CountryCode+PhoneNumber (Example 01161420615180)

Note: I have checked, mydevices.com is not on any blacklist. In fact, cayenne is using Amazon Email server that has really a good reputation.

I could do that. I also have a local Node-Red server where I could implement my own email and “push” notifications. I’m already doing that for some other projects and I will eventually get around to rolling my own “over voltage” notifications too.

My motivation, though, has less to do with getting notifications going again “by any means” than it is helping Cayenne’s administrators figure out why a process that has been stable for so long (18 months) has suddenly stopped working. In this I’m assuming that other Cayenne users are also affected but perhaps simply have not yet realised that their triggers are no longer working properly.

Anyway, to the state of play right now. Grafana shows plenty of over-253:

Influx confirms 88 events (10-second data):

> select count(voltage) from hiking where time > '2019-10-16T23:00:00.000Z' and voltage > 253.0;
name: hiking
time                           count
----                           -----
2019-10-16T23:00:00.000000001Z 88

and the NEW trigger confirms a bunch of firings (30-2=28) since the previous report:

23

while not a single one of those has made it to the inbox:

Still just test messages (first two lines) and copies of postings. No triggers.

we will look into the issue of email notification.

For the benefit of those who may be following this thread, there has been a reasonable amount of offline investigation. The situation at the moment is as follows:

Three mailboxes:

  1. original@mydomain.com
  2. existing@anotherdomain.com
  3. different@mydomain.com

Three classes of email:

  1. Generated by triggers.
  2. Confirming copies from the Cayenne forum.
  3. Test messages.

Observed behaviour falls into two categories:

A. Failing case:

Triggers sending to original@mydomain.com.

B. Succeeding cases:

Triggers sending to existing@anotherdomain.com.

Triggers sending to different@mydomain.com.

Cayenne forum messages to original@mydomain.com.

Test messages to original@mydomain.com.

Test messages to different@mydomain.com.

The problem appears to be confined to one specific class of email (produced by Cayenne triggers) being sent to one specific mailbox (original@mydomain.com).

The trigger behaviour is the same, irrespective of whether the trigger existed before the problem manifested, or was since as part of the investigation. In all cases, the triggers appear to fire correctly when the condition is met (voltage exceeds 253V). If the trigger is aimed at original@mydomain.com, there is no email delivery. If the trigger is aimed anywhere else, email delivery occurs promptly.

The original@mydomain.com mailbox has been checked. There are no spam or blacklist filters configured and, were the mailbox “full”, which it isn’t, that would also block Cayenne forum and test messages, which aren’t being blocked.

The DNS for mydomain.com contains exactly one MX record resolving to exactly one A record so the problem is unlikely to be a round-robin style misconfiguration malfunction. If the DNS was wrong, nothing should be working.

As yet, Cayenne administrators have not reported finding either queued mail or evidence of the mydomain.com MTA refusing delivery.

The obvious “workaround” is to deprecate original@mydomain.com in favour of different@mydomain.com but that just kicks the can down the road. It would be preferable to get to the bottom of this in case it turns out that triggers on other Cayenne projects are not behaving as expected, with their users unaware of the problem.

i guess we may conclude that the issue is that your original@mydomain.com is been blocked because it has hit rate limit of 300 messages. Will check it further what other things might be affecting here and try to find a solution on it.

Have you checked your email logs to see if you are getting any connections from the Cayenne email server around the time the email should be sent?

Unfortunately, my hosting provider (HostGator) doesn’t provide access to the email logs. I suspect this is because they front-end their hosted domains with “websitewelcome.com” (one of their own domains) and this probably implies that the logs for all hosted domains would be coalesced, so “Privacy!” (said with appropriate tones of shock and horror) would be the answer to any “can I get access” question.

All I can see is what has arrived in the mailbox, and what I can deduce from the headers in messages that have arrived, or headers in “bounced” messages I send myself that don’t arrive. Sub-optimal, I know, but not much I can do about it.

Yes, that is unfortunate. You may be able to ask them if they have seen any emails from a specific email address to your email though. It’s the only way to know for sure if your email server is for some reason blocking the emails or if they are indeed never being sent.

Wouldn’t it be much better if cayenne let users use their own smtp server?