Delayed Email Notifications

(Josh Cohen) #1

Is anyone noticing delayed email notifications? I’m not sure if it’s just me.

It may be that the email notification queue is getting so long that the server can’t process them fast enough and I will have to make some adjustments.


Yes, I’ve noticed for the past couple weeks, I always get the emails at one time. I don’t get them when a new comment is on a post, they just all come at once. I have it set on “immediately” though.

I actually haven’t received any in a week or so, I’m not sure whether it’s because I read everything within a couple hours, or I might have somehow unsubscribed to everything. After checking, I still have 198 active subscriptions, so I’m not sure what that is.

I don’t actually check my e-mail for responses to posts and stuff, I just check the “unread” tab. But I have noticed delayed messages that come in bulk.


Edit: Looking at my e-mail address, I last received messages from this site March 31st, before that March 24th, and March 18th. Seems suspiciously close to a once a week schedule, even though I’m pretty sure I’ve always had it on a “immediately” schedule. Also haven’t received any April 6th, so it doesn’t seem to fit…

(Josh Cohen) #3

Thanks… that helps answer my question.

Originally, the site would send the email notifications at the same time someone posted a new comment. On really long threads, this would cause the server to slow down which would encourage people to click the submit button again. I had a suspicion that that was causing the double posts, so I changed the settings so that email notifications are put in a queue and sent once per hour.

I think what is happening is that the email queue tends to grow much larger than can be processed in that narrow window of time once per hour.

I will see if I can set up a cron job to run the email delivery every 15 minutes or so, but there is a delicate balance between processing the emails and overwhelming the server. :slight_smile:

(Josh Cohen) #4

A quick check of the logs makes this seem likely.

“Reached processing limit on queue processing:”

(Josh Cohen) #5

I just found the settings, and there were 628 queued emails. It only had four seconds per hour to work on the queue, so was falling behind. I’m trying a couple of things now that I hope will fix it…

(Josh Cohen) #6

My apologies to anyone who just got a lot of email from the site.

I’m hoping that it’s fixed now. :slight_smile:


Glad it’s fixed. Is it going to send 4 times an hour now? Every 15 minutes?

Why did it only have 4 seconds an hour?


(Josh Cohen) #8

Sorry about that email overload. :confused:

Every hour, the website runs a process called “cron” which performs maintenance tasks like sending the notification emails. Whichever user hits the site during that process the triggered the cron gets a delay. To reduce the length of that delay, I limited the settings to four seconds for sending the emails.

I fixed things so that the site now runs cron independently of user visits and spends eight seconds to send the emails.

Since this afternoon, the email queue is still at zero, so it looks like it’s now keeping up with the number of new posts. I’ll keep an eye on it over the next few days.

Currently, it should send about every 30 minutes, just to keep load off the server, but if that isn’t fast enough, I’ll try increasing it and see if the server can still handle it without going offline.


Thanks, I was wondering about that. At first I thought I must be saying nothing that anybody was interested in, then all of a sudden everybody was caring.

(Josh Cohen) #10

Sorry about that. The current settings (15 minutes now) are keeping the queue cleared, so the problem appears to be fixed for now.

(Josh Cohen) #11

When I woke up this morning, the queue was backed up again with 162 emails. I think that the site can handle new threads, but when people post to the longer threads, the site prepares a couple hundred emails to send out.

I’m processing the emails this morning. If anyone gets too many at once, let me know and I’ll see what else I can do to fix it.


Are the default settings set to subscribe people to a thread? Perhaps changing the default will mean only those who truly want a notification have to worry about it. Just a thought anyway :slight_smile: Could always offload email processing to a separate server as well

(Josh Cohen) #13

People choose the settings when they sign up, but I think it defaults to yes. I will double check that.

Moving the email notifications to a separate server is something that I would like to do.

It looks like the current settings are working though. If anyone notices too much email, just let me know and I will try to optimize it a bit more. :slight_smile:

(Josh Cohen) #14

This problem may have just temporarily returned due to the server change. I forgot to setup the cron job on the new server, so the queue got backed up again.

Sorry in advance if anyone just received a flood of emails. :confused: