Thursday, September 13, 2007

Alerts vs. Time of Day

A long time ago I worked at a company called DTN. We had several systems that worked 24/7 processing news, weather, and market feeds. One of my early jobs was to create a system to alert us when something went wrong.

All our data-gathering systems could send text messages to a pager via an API I created. And the central paging system itself would poll data-gathering systems to confirm they were up. Anyone could write plug-in modules for the paging system and expand it. You could telnet into the paging system to administrate it. Today, people use programs like Nagios, but back then we didn't have all those cool open-source tools.

In the spirit of eating your own dog food, I first tried this paging system I created myself. Everytime something went wrong with the system, I got a text page on what the error was. Then I could log into the system and control the alerts. That system was cool.

Soon after, I got paged 100 times in just a few minutes. So the paging program got updated to have the additional ability to limit pages to 1 per 15 minutes. It was 'cool' again.

Later, the system woke me up multiple nights in a row for errors that were minor, and corrected themselves. After another round of development, the paging system learned that between 10 pm and 7 am, something had to be really, really, wrong before it would page.

(One day, all our data feeds stopped. It turns the satellite that they used failed. So did our pages go off? No. The satellite those data feeds used was also used by our paging system.)

This was an evolution in knowing how to send alerts 'smart' depending on type of alert, time of day, and quantity. I wish many other devices would go through it. My cell phone will beep merrily at 3 am letting me know it has a low battery. Come on, do you really think I care at that time of night? Worse yet is the smoke alarm that tells me it has a low battery in the middle of the night. Many of our smoke detectors require a ladder to access. Just what I want to be doing.

So the moral of the story: If you are working on a system that makes causes an audible alert to be made, allow for different 'priorites' to be made depending on the hour of the day. It will improve the quality of life of your customers.