plainblack.com
Username Password
search
Bookmark and Share

Troubleshooting

If you’re reading this chapter then you’re either being a good administrator, and preparing for the future, or something bad has just happened and you’re trying to figure out what you can do about it. Either way, this chapter should give you some good tips about how to proceed.

Having said that though, there’s no way to give you a comprehensive guide that would cover all things. If you really get stuck, don’t spin your wheels, give Plain Black support a shot, and get your site back online sooner rather than later.

 

Don’t Panic

The worst thing you can do in an emergency situation is panic. This is true of any emergency, whether it be a down server, you’re trapped on a deserted island, or your wife is having a baby.

 

Collect Your Thoughts And Gather Intel

Breathe. Now that you’re breathing again, find out what the symptoms of the problem are. Not just “we’re down”, but perhaps when we went down, what was going on with the server at that time (perhaps backups or web stats generation).

 

Make Notes Not Assumptions

As you make discoveries, make notes of those discoveries so they can help with your diagnosis of the problem, and your post-mortem to stop the problem from happening again in the future. Do not make assumptions at this point. Assumptions will only likely take you down the wrong path and make you waste hours of valuable time, and more importantly, make you miss the actual cause when it passes in front of you.

 

Reading Logs

If you’re having a problem, the first place you should go is directly to your log files. Look at your WebGUI logs, your Apache error logs, your MySQL logs, and sometimes even your operating system logs will have some insights for you. Sometimes the logs will tell you exactly what’s wrong “Your out of disk space”, “Can’t connect to the mail server”, etc. But most of the time, the log file will just be giving you your first round of symptoms.

Don’t assume the symptom is the problem. For example, if the log tells you that it can’t connect to the mail server. Does that mean the mail server is down? Does that mean that WebGUI has a bug? Does that mean someone made a firewall change? Does that mean your router has failed? Maybe all of the above? If you get into this mindset, you’re doomed to failure. After you have your list of symptoms, you need to ask yourself only one question:

What has changed?

Everything was working until it went down, right? So what has changed? Maybe it’s none of the above things, but rather a user went into the settings and changed the mail server from yours to some other mail server and that’s the mail server it can’t contact. But if you made the assumption that it was your mail server, you could spend decades trying to troubleshoot a problem that’s not there.

If you find that the WebGUI error log is reporting an error, but it’s not enough for you to go on, perhaps you should increase your log level. Doing so may just give you that added bit of detail that you need to see what’s going on. See the chapter called “Logging” for details.

 

WebGUI Debug Mode

WebGUI debug mode is mostly useful for developers, but sometimes it can also help in diagnosing problems with WebGUI. To turn it on, follow these steps:

  1. Log in and turn admin on.

  2. Go to the Admin Console tab of the Admin Bar and then choose Settings.

  3. Flip to the “Miscellaneous” tab.

  4. Fill in your IP address or your entire subnet in CIDR notation in the “Debug IP” field.

  5. Then set “Show debugging?” to yes.

  6. Click on the “save” button.

 

Now as you browse pages you will see debugging output at the bottom of every page as long as your IP address matches the Debug IP field in the settings. However, all your users will just see the pages as normal.

This can be especially useful if you believe that certain actions, pages, assets, or macros are causing errors on your site. Then all you have to do is visit the problematic page, or perform the problematic action and see if there are any error messages in the debug output that can help you resolve the problem.

 

Spectre Debug Mode

Spectre has a debug mode that can help you work through connectivity issues between it and WebGUI. To start it running in debug mode do the following:

 

cd /data/WebGUI/sbin


perl spectre.pl --run --debug

 

Now Spectre will inform you of everything it’s doing when it does it. You’ll be able to see if it’s able to connect to WebGUI, if it’s getting error messages, how many messages are in it’s queue, and a whole host of other information.

 

Spectre’s Automated Tests

Spectre also has a test mode, that can be used to test whether it can successfully connect to WebGUI. While Spectre is running (whether in debug mode or not), run the following commands:

 

cd /data/WebGUI/sbin


perl spectre.pl --test

 

It should report that it can connect just fine to everything like this:

 

Running connectivity tests.

Testing www.example.com.conf

Tests completed.

 

However, if you get back something like this:

 

Running connectivity tests.

Testing www.example.com.conf

ERROR: Couldn't connect to WebGUI site www.example.com.conf

Tests completed.

 

Then you’ve got a connectivity problem. It could be that either Spectre or WebGUI is offline, or it could be that one or the other is misconfigured.

One of the most common mistakes people make is in setting the spectreSubnets directive in each of their WebGUI config files. It needs to contain all the IP addresses that Spectre may use to communicate to the server. Here’s an example where the server has two IP addresses assigned to it:

 

“spectreSubnets” : [“127.0.0.1/32”, “111.111.111.111/32”, “222.222.222.222/32”],

 

Notice that the loopback IP address is also included. This is important. Note that it’s not possible to simply set the list to 0.0.0.0/0 (ie: the whole Internet) for security reasons.

Spectre’s Runtime Status

Spectre has a runtime status which can go a long way toward troubleshooting problems with workflows. To view status of workflows inside of Spectre run the following commands:

 

cd /data/WebGUI/sbin


perl spectre.pl --status

 

If Spectre has no workflows waiting to run, then the output will look like this:

 

Suspended Workflows 0

Waiting Workflows 9

Running Workflows 0

Total Workflows 9



WebGUI / Spectre Communication

If you're having WebGUI / Spectre communication problems, you can manually run the transactions from Spectre to WebGUI to see if there's anything wrong.

You'll need a program called cURL to do this. It comes with most Unix, Linux, and BSD operating systems, and the Windows WRE. You'll also need to do this from the command line of the machine that runs Spectre.

The first transaction you can try is the one Spectre calls on each site to get that site's configuration data.

 

curl http://www.example.com/?op=spectreGetSiteData

 

It should return a JSON object with all the Workflow Instances and Scheduler Tasks that need to be run.

 

{"workflow":[],"cron":[{"sitename":"www.example.com","taskId":"pbcron0000000000000001","dayOfMonth":"*","dayOfWeek":"*","gateway":"/","minuteOfHour":"30","hourOfDay":"23","cookieName":"wgSession","runOnce":"0","monthOfYear":"*"},{"sitename":"dev.localhost.localdomain","taskId":"pbcron0000000000000002","dayOfMonth":"*","dayOfWeek":"0","gateway":"/","minuteOfHour":"30","hourOfDay":"1","cookieName":"wgSession","runOnce":"0","monthOfYear":"*"},{"sitename":"dev.localhost.localdomain","taskId":"pbcron0000000000000003","dayOfMonth":"*","dayOfWeek":"*","gateway":"/","minuteOfHour":"15","hourOfDay":"*","cookieName":"wgSession","runOnce":"0","monthOfYear":"*"},{"sitename":"dev.localhost.localdomain","taskId":"pbcron0000000000000004","dayOfMonth":"*","dayOfWeek":"*","gateway":"/","minuteOfHour":"*/5","hourOfDay":"*","cookieName":"wgSession","runOnce":"0","monthOfYear":"*"}]}

 

The next transaction you can try is the one that runs a workflow activity. Note the \ before the ; in the following command. That's because on most systems the semi-colon separates multiple commands on a single line. By escaping it with a backslash we're saying we want to pass the semi-colon along through the URL.

 

curl http://www.example.com/?op=runWorkflow\;instanceId=xxxxxxxxxxxxxxxxxxxxxx

 

It should return a one word status.

 

complete

 

The final transaction you can try is the one that runs a scheduler task. Again we escape the semi-colon with a backslash.

 

curl http://www.example.com/?op=runCronJob\;taskId=xxxxxxxxxxxxxxxxxxxxxx

 

Again we're expecting a one word status to be returned.

done

 

WebGUI Advisories

If you’re having a problem and you can’t figure out how to fix it, perhaps it’s a bug and it’s been fixed already. As a WebGUI administrator, it’s your job to stay on top of the releases of WebGUI, and upgrade to get bugs fixed, as well as apply patches for security problems. Luckily, we make this easy for you. You can subscribe to our advisories list at www.getwebgui.com. This is a very low traffic list, and is only posted to when a new release of WebGUI or the WRE comes out, or if a serious security problem was found. We strongly encourage all WebGUI administrators to be subscribed to this list, or at the very least check it out once a week.

 

Reporting Bugs

So you’ve tried everything and you’re sure you have a bug on your hands? Before you report it, create a demo out on http://demo.webgui.org and see if you can reproduce the bug there. If you can you probably have a bug, otherwise you probably have a local configuration issue of some sort. If you think it’s a bug, read on.

Stop! Before you report a bug, you need to know what the difference between a bug, and a request for enhancement (RFE) is. Many people confuse the two.

A bug is a functional problem with something not working as advertised. For example, if you see a “save” button, and click on it, and it does not save the data in the form, then that is a bug. However, if the save button is saving the data, but you wish that after saving it would take you to some other page, then that’s an RFE. The reason is that the save button is working, you just want it to do something beyond what it’s designed to do. If this is the case, go to the “Making a Request For Enhancement” section later in this chapter.

Reporting a bug is very easy. You can simply go to http://webgui.org/bugs and submit your bug. However, reporting a bug correctly, is not as easy. First, you need to check if the bug you’re reporting has already been reported. Second, if you want the bug fixed expediently, and to your satisfaction, help us help you. You can do that by including lots of information in your bug report. The following things should be in every bug report:

  1. Your WebGUI version number.

  2. Your WRE version number (or if you’re not using the WRE, then provide us with information about what platform you are using).

  3. A step by step account of how to reproduce the bug.

  4. A description (or screenshot) of what is happening.

  5. A description of what you think should be happening.

Also, with each bug report, include anything else that you think may help us diagnose the bug. And if you’re the sort that can fix bugs, feel free to include a suggested patch to resolve the problem.

 

Making a Request For Enhancement

A Request For Enhancement (RFE) is any change in functionality you want made for WebGUI. It could be that you want a new asset written that will handle the publishing of __insert__some__custom__thing__here__, or maybe you want to change some existing functionality to work slightly differently. Whatever the case, you should publish an RFE.

To publish an RFE, simply go to http://webgui.org/rfe and type out what you want done.

Before you publish your RFE though, see if anyone else has already published something similar. If they have, you should publish your comments as a reply to theirs, and then apply your karma to the post to move it up to the top of the list.

When you do finally start publishing your RFE, be sure to be detailed. If you have an idea about how something should look, consider attaching a screenshot. If you know how to write HTML, CSS, Javascript, Perl, SQL, or any other code/script that may provide additional insight into your request, include that as well. And most importantly, include the thing that everyone forgets, include why this is such an important feature.

Keywords: bugs debug logs request for enhancement rfe spectre

Search | Most Popular | Recent Changes | Wiki Home
© 2018 Plain Black Corporation | All Rights Reserved