Jul 31, 2018 - Malfunctioning bots and how we deal with them

Bots, no matter how robust they are, are sometimes prone to errors and malfunctions. There’s a chance that any bot can go rogue by sending multiple chat messages at once, or adding multiple automatic flags on the same post. It doesn’t matter if you’re a room owner, bot developer, or a regular user of the room - we all need to be geared up and prepared for these emergency situations.

Does the situation warrant any action?

First, you must decide whether action really needs to be taken:

  • Are the reports all (or nearly all) unhelpful or redundant, such as being false-positives or duplicates? If the reports are useful, they’re probably OK.
  • Does the bot appear to have a developer working on it at the moment? If someone’s developing, the report spam is probably just a bug or a test, and the developer likely knows about it and is working to fix it.
  • Is the bot posting a truly large volume of reports, such that chatting and interacting with other bots becomes difficult?
    • Posting the same message 3 times in a row (like Smokey’s “conflicting feedback” reports) does not require drastic action; it’s just a minor glitch that maybe is worth talking to the bot developer or filing a GH issue over.
    • Posting the same message 30 times in a row or reporting nearly every post on the site makes chatting and interacting with other bots difficult, and requires intervention.

How do I act, if it warrants action?

If action is required, the first priority is to stop the reports in the short-term. If the quickest way to do that is to disable the bot entirely, so be it — the end result is the bot being offline until a developer can fix the problem and re-enable it.

Don’t perform any of the below steps unless you’re confident you know what you’re doing and you’re sure no one will get angry at you for doing it. If you’re not 100% confident in what you’re about to do, delegate the problem to someone else or try a different step.

If you’re not a RO, bot developer, or moderator, ping someone who is (if anyone’s around). If no one is available, proceed with caution.

If a developer of the bot is around, ping them as they likely have more knowledge of the bot’s internal workings.

If the bot stops malfunctioning, stop trying to fix it (even if you don’t think you did anything to fix it). Sometimes the problem will eventually resolve itself, such as the time someone posted 27 identical answers and Guttenberg reported every pair it found. If you know the problem will go away shortly, it’s probably best to just wait it out.

With the above disclaimers in mind, proceed through the following checklist:

  • First, find the bot’s documentation.
    • Different bots function differently and accept different commands, so knowing your available options is critical.
  • If there’s an obvious reason for the malfunction and an easy way to fix it, do so immediately.
    • For example, if a recent blacklist change is over-zealous, try to revert that blacklist change.
  • Try a reboot. More often than not, rebooting fixes the problem entirely.
  • If the bot is not responding to a reboot, it may be overloaded processing commands or sending chat messages. Some bots such as SwiftChatSE-based bots have a kill command that will immediately crash the bot, bypassing any work that’s queued up or cleanup tasks that are usually performed during a shutdown.
    • If the bot auto-restarts after crashes, it may come back up after a kill command.
  • If you have access to the bot on Redunda, try a failover to another instance.
  • If the bot responds to a reboot or kill, but comes back online and starts malfunctioning again, use the shutdown command.
    • If the bot is behaving, a shutdown should cause that instance to cease operating.
    • If multiple instances are available, Redunda should launch another instance, which may or may not experience the same problem. If the new instance works, problem solved; otherwise, shut down this instance too.
  • If all else fails, use Stack Exchanges moderation tools to silence the bot. The least drastic way to do this is to put the room in gallery mode and remove the bot’s write access. An alternative is to kick the bot, which will temporarily ban the bot from the chat room, however if the bot is kicked enough times automatic moderator flags will be raised and restrictions will be applied to the bot’s account. If you’re not a RO or mod, ping a RO immediately. If there is no only available to ping raise a moderator flag by clicking the message dropdown and choosing “flag for moderator.”

What to do after the bot has been silenced?

At this point, either the bot has been fixed or disabled. If you’re a RO or mod, clean up the chat transcript by moving the erroneous reports to a trash room. If you’re not a RO, ping someone who is.

The developer of the bot needs to be informed of the malfunction, as well as what you did and why. If you had to disable the bot, a developer is needed to fix and re-enable the bot. If you were able to fix it yourself or the problem went away on its own, explaining what happened is courteous and helpful so that the developer is aware and can prevent it from happening again in the future.

Jul 13, 2018 - Stack Exchange is removing OpenID. How does this affect us, and what do we have to do about it?

As you might have heard, Stack Exchange will be removing OpenID login on July 25, 2018. Because our bots depend on OpenID to log in to chat, we had to reverse engineer the new login.

Let’s have a look at our Java library, ChatExchange, to show you how the new login works:

Step 0: Prepare to store cookies

We’ll need to store some of the cookies we receive when logging in. In our case, we’re using a HashMap<String, String>. We pass it on to our own HttpClient. If you implement this on your own, make sure tha you store all the cookies and send them along with your requests!

Step 1: Get the login form

Every site in the Stack Exchange network (except for stackexchange.com itself - we’ll discuss that later) should now have this new login form located at /users/login:

login screen

Along with the fields for email and password, it has a hidden field called fkey, which is filled with a server generated value. We need to post this key along with the credentials. In order to be able to get this key, we first need to send a GET request to /users/login and read the fkey:

Response response = httpClient.get("https://" + host + "/users/login", cookies);
String fkey = response.parse().select("input[name='fkey']").val();

Step 2: Submit the form

Now we just need to post the credentials and the fkey to /users/login:

response = httpClient.post("https://" + host + "/users/login", cookies, "email", email, "password", password, "fkey", fkey);

Step 3: Check if you’re now logged in

To check if the login worked, we’re sending a GET-request to /users/current, which redirects to your profile when you’re logged in. If we can find a HTML element with the class js-inbox-button in the response, we’re logged in. Make sure that you send the cookies you’ve previously saved.

Response checkResponse = httpClient.get("https://" + host + "/users/current", cookies);
if (checkResponse.parse().getElementsByClass("js-inbox-button").first() == null) {
	throw new IllegalStateException("Unable to login to Stack Exchange.");
}

And now the edge cases…

Up until now, the implementation was quite easy and worked well for the Stack Overflow chat. And then there was chat.stackexchange.com…

Login to chat.stackexchange.com

As mentioned earlier, stackexchange.com is a special case. It does not have the login form that other sites use. To solve this problem, thesecretmaster ♦ had an idea. (actually two, but I prefer explaining the easy way ;-) )

meta.stackexchange.com has the same login as stackoverflow.com and the other sites. thesecretmaster ♦ figured out that we can simply use the cookies from meta.stackexchange.com and send them to chat.stackexchange.com. To implement this, we just needed to send steps 1 and 2 to meta.SE and step 3 to stackexchange.com.

What happens if the user does not have an account on the site they try to use?

With some accounts, our code just didn’t work on chat.stackexchange.com. The problem was that since we now take a little detour, the user account for the bot has to have an account on meta.SE. My bot didn’t have one. Creating an account is quite easy. The POST-request in step 2 will return a message and a button, if the user does not have an account yet. If we don’t click that button, the user won’t be logged in.

Identifying that we received that message is quite easy, although the ID of the <form>-element is not intuitive: logout-user

The bigger issue is in actually sending that form. Since we can’t just click the element in Java and didn’t know which of the hidden fields in that form is acutally being used, we had to read them all and post the contents to the action-attribute of the element:

Element formElement = response.parse().getElementById("logout-user");
if (formElement != null) {
  Elements formInputs = formElement.getElementsByTag("input");
  List<String> formData = new ArrayList<>();

  for (Element input : formInputs) {
    String key = input.attr("name");
    String value = input.val();

    if (key == null || key.isEmpty())
      continue;

    formData.add(key);
    formData.add(value);
  } // for formInputs

  String[] formDataArray = formData.toArray(new String[formData.size()]);

  String formUrl = "https://" + host + formElement.attr("action");

  Response formResponse = httpClient.post(formUrl, cookies, formDataArray);
  
  if (formResponse.parse().getElementsByClass("js-inbox-button").first() == null) {
    throw new IllegalStateException("Unable to create account on " + host + "! Please create the account manually.");
  } // if
} // if

If you have further questions about implementing this, feel free to join us in our chatroom.

Jul 1, 2018 - Connecting my bot to Higgs

Higgs

Higgs is a generic dashboard for viewing and providing feedback to bots found in SOBotics. It originated alongside Boson - a framework for creating SOBotics bots. This blog post will run you through the process of setting up your own bot and integrating it with Higgs.

Setting up a dashboard

Bot setup is now entirely self-service! If you’re not already a bot owner, you’ll need to ask a Higgs admin to grant your account bot privileges. Once that’s done, you’ll be able to create a new bot yourself.

  1. You should now see an ‘Admin’ dropdown in the top left of Higgs:

    enter image description here

  2. Clicking ‘Bots’ will bring you to the bot management page. Here, you’ll see all bots that you own. Admins are able to see and edit every bot. From here, you’re also able to create a new bot:

    enter image description here

  3. When creating a new bot, you’ll be presented with the following page:

    enter image description here

    The following fields are required:

    1. The name of the bot
    2. The dashboard name
    3. The description of the bot
    4. The secret
    5. Optionally, you may also enter:
    • A link to the homepage of the bot.
    • A link to the bot’s logo.
    • A favicon. This will change the page’s favicon when viewing one of your bots reports. Must point to a URL.
    • Tab Title. This will change the page’s title when viewing one of your bots.
  4. Once you’ve submitted your bot, you’ll need to set up the feedback types. You can manage this from the /admin/bots page (as shown above).

  5. You’ll be presented with the following page:

    enter image description here

    • Name: The name of the feedback. This is what will be rendered on buttons for reviewers.
    • Colour: A browser-supported colour string. Hex or named colours are supported. Styles the colour of the feedback button, as well as the icon.
    • Icon: An icon to represent already cast feedback. It’s simply a string, and can technically be anything, but we recommend using a Unicode character.
    • Actionable: Whether or not this type of feedback is counted when putting reports into the review queue.
    • Enabled: Whether or not this type of feedback is allowed to be cast by a review.

Getting started in code

Higgs uses swagger to document its API. A benefit of this is that the boilerplate API code can be automatically generated. Here’s an example script being used to generate the API structure for Higgs’ frontend.

Note that the above is entirely optional. Generating the boilerplate is a nice-to-have, and not required. If you’d like to implement the API calls yourself, you can see the available endpoints here, as well as what security is required for each call.

Here’s an example of a bot authenticating itself, and then sending feedback for a user:

// Point the API to Higgs
const string basePath = "http://45.77.238.226";

var botApi = new BotApi(basePath);

// Here, 1 is the ID of your bot. This makes a POST request to /bot/AquireToken
var tokenResponse = botApi.BotAquireTokenPost(1, "THIS IS MY SECRET KEY");

// Now we've got the bot's access token, we need to configure our API to add it to future requests
// If you're not using swagger gen, you need to ensure future api requests add the following header:
// Authorization: Bearer [token]
botApi = new BotApi(new Configuration
{
    BasePath = basePath,
    AccessToken = tokenResponse.Token
});

// Example request after the bot has been authenticated
botApi.BotRegisterUserFeedbackPost(new RegisterUserFeedbackRequest(5, 1, "False Positive"));

The access token is a JWT token, and is currently valid for 7 days. Bots will be able to decode the token to inspect what scopes (permissions) it has been granted, and when the token expires.