The Communications Hub and a very annoying SL bug

Today, one of my Hubs stopped working after a sim restart.   Mesmerizers and Communicators just wouldn't connect to it when the sim came back.  Luckily, I recognized this as the re-appearance of an SL bug that dates back for at least 15 years.

CHAOS devices (Mesmerizers and Communicators) talk to the Communications Hub using HTTP - the same protocol that a web browser uses to retrieve web pages.   To do that, they have to know the URL of the Hub they want to talk to.  In Second Life, URLs for objects are allocated on-demand, and when a sim is restarted the Hub needs to ask for a new URL, and it will be allocated a different URL each time it asks.  So how do Mesmerizers and Communicators find out the current URL of their Hub?  Well, they ask the Hub. But obviously they can't use HTTP to do that, since they don't know the URL.  Instead they use email.

Every object in SL has the potential to receive email.   The object's key becomes part of its email address, and since keys of rezzed objects don't change on a sim restart, a Hub always keeps the same email address, so long as you keep it rezzed.   If you take it back into inventory and then re-rez it, it will have a different key, and therefore a different email address, so the Mesmerizers it is supposed to serve will no longer be able to find it.   That's why you have to keep Communications Hubs rezzed - so they keep the email address that their connected devices know.  When a device needs to talk to a Hub for the first time, it sends it an email asking it for its current URL, and the Hub replies with its URL (over HTTP).

So what's the bug?  Well, every so often, it appears that some in-world objects suddenly stop receiving email.  And that's what happened to my Hub.  The Hub goes to check email every few minutes, and it was doing that, but finding no new emails.  As a result, the devices trying to connect couldn't get the Hub's new URL, so they couldn't communicate with it.

As I said, this bug has been around for over fifteen years, but it seemed to stop happening around 7 years ago, although there was no indication that a fix had been made.  And it appears that indeed, no fix has been made, since it happened again today.  Anyway, there is a workaround, if it happens to one of your Hubs.  In fact there are three.  But before you try any of these, click on the offending Hub and choose "Restart" - if the issue is something other than the bug under discussion here, a restart of the Hub may fix it, and will allow your users to connect once more.   If that doesn't work, you can try one of the following:

• If you own your sim, or are on friendly terms with the actual owner, you can request that the sim be restarted.   This method may permit the queued emails to start flowing once again, but it doesn't always seem to clear the condition.  But it has the lowest impact on your users, and as people are used to weekly sim restarts, it's worth trying first if the simple Hub restart above doesn't help.

• If you don't have many users on your Hub, you could simply take the Hub back into inventory (save the user database to the Users notecard first) and then rez a fresh instance.  Since re-rezzing will give the new instance a fresh key, the users will have to reconnect by physically coming to the Hub, clicking it and choosing "Connect".

• If you'd rather not bother your users, there is an alternative.  First, pick up the Hub.  This doesn't mean take it into inventory - instead you're going to actually attach it to your avatar.   To do that, right-click the Hub and choose "Attach" from the pie-menu.  You will have to select an attachment point, and the Hub will attach to the point you select.  Picking up an object this way doesn't change its key.  Once you have the Hub attached, teleport to a different sim.  Teleporting doesn't change the key of worn attachments, so the Hub will retain its email address through all of this.  Then teleport back to the original sim - again the key and email address of the Hub will be preserved.  Finally, right click the (worn) Hub and choose "drop" to drop it back onto the ground.   Then click on the Hub and choose "Restart".

It appears that bringing an object into a sim has a side-effect of clearing whatever condition is preventing it receiving the emails.  In fact, after performing this operation, when the Hub first checks for email it will retrieve all the emails that had been sent, and it may take a while to work through those and get to current requests.  The Restart at the end is necessary to force the Hub to obtain fresh HTTP URLs once it's back in its home sim.

None of these workarounds is particularly straightforward, but until the Lindens deal with this bug, they are the only options I know of.   Given that I've been running six Hubs in my own sim for years, and haven't encountered this issue for at least 7 years, it seems to be a pretty rare bug.  However, it is possible that its re-emergence today may be related to some change the Lab has made, in which case it might recur more frequently going forward.  The possible good news is that, unless it happens soon after a sim restart, there's a good chance that CHAOS devices will have already obtained the current URL of their Hub by the time that the email path stops working, so if your users regularly log in to SL, they may not be affected by this bug.

The Enforcer 1.1 and CHAOS Scheduler Preview

 In a previous post, I mentioned my intent to support the scheduling of events within The Enforcer.  This is something that's been requested, and there are several interesting use-cases for such a feature.  So I had started work on building a scheduler into The Enforcer that could generate events on a predefined schedule.  However, when quite a way along with this effort, I had an idea if how to do it rather more elegantly.

There are two very distinct types of scheduled events we might want to support: ones that occur on a schedule (e.g. "every day at 10am"), and ones that are one-shot and occur after a delay (e.g. "in ten minutes from now").  The latter could be used to initiate a trance every so often, or to check whether the sub is making progress with some task;  the former is suitable for "housekeeping" tasks, particularly in situations where you don't really care which sub(s) the event runs for.

The Enforcer already has a crude way of doing the second type of event - the !delay command.  While this doesn't generate a new event but merely suspends the current event for the specified interval, that may often be sufficient, and if a separate event really is needed, the event could be generated immediately via the !event command, and then the notecard that processes that event could begin with a !delay command.  The only real problem with this is that the context that represents the event within The Enforcer hangs around for the duration of the delay, taking up resources.  So a variant of the !event command that fires after a delay (rather than immediately) would be useful.  This can be used both for one-shot events, or for repeating events (the event would schedule another future event each time it runs), and would typically run on behalf of the sub who caused the original event.  

The first type of event - scheduled for a specific time and then possibly repeating - is much more complicated.   There's a lot of complexity around handling date-times, that really has no place in The Enforcer.  And this type of event, being independent of an particular sub logging in, would presumably want to continue to run after a reset of The Enforcer.

I have also been thinking about teaching The Enforcer to run Mesmerizer trance scripts.  The Enforcer can be used for trances today, but it's clumsy in that you need to do a manual delay after every piece of text, and it's easy to forget that trance text needs to be sent within a ^text or ^text2 command.  The Mesmerizer script format is designed to make creating trances as simple as possible, and it would be very useful if The Enforcer could play them back unchanged.  But doing this within The Enforcer itself would mean that it would now have to support two scripting languages, which would very likely introduce bugs and increase memory use.

So, instead of adding these new features directly into The Enforcer, I have decided to add a remote control interface to The Enforcer, so that functionality like this - generation of events, and playback of trances - can be done by new individual products that connect to The Enforcer.  This means that only those people who want to make use of these functions will be impacted by them.  In addition, it opens the door to various custom add-ons to The Enforcer.  I plan to make available a simple toolkit that will permit LSL scripts to easily be connected to The Enforcer to allow for the easy construction of custom controllers.

So in addition to the "future event" capability, The Enforcer 1.1 will add the concept of "External Controllers".  The Enforcer will be able to scan the sim for new controllers and, if found, can connect to them.  Once connected, a controller can send commands to The Enforcer to generate events, or to be forwarded to a logged-in user's Mesmerizer.  At the same time, I will launch the CHAOS Scheduler as a new product.  The trance reader will come somewhat later. 

The CHAOS Scheduler offers the ability to generate Enforcer events according to a timetable set out in a notecard in the Scheduler's inventory.  Events can be repeating or one-shot, and since they're defined in a notecard, they will survive script resets, and will continue to operate if the Scheduler is taken back into inventory and then re-rezzed (although the new instance will have to be manually connected to The Enforcer before the events will have anywhere to go).