plainblack.com
Username Password
search
Bookmark and Share

Utility Scripts

A utility script is anything outside of the main web interface that uses WebGUI's API. With most other extensible places controlled by WebGUI from the outside, utility scripts control the flow of execution for themselves.

 

Why Create Utility Scripts?

Utility scripts are primarily useful for maintenance. WebGUI is designed so basically everything can be controlled or changed through the web interface, but there are some tasks that lend themselves to the command line. Sometimes you want to interact with the local filesystem, which isn't safe to expose to the web. You may also want to perform an action across multiple sites hosted on the same server. Other actions are somewhat dangerous and should only be done by someone who knows the risks and can perform backups and be ready to repair damage. Writing your own script is often useful because creating a command line interface is simpler than making a web interface. The same tasks can probably be performed through SQL and shell commands, but by using WebGUI's API your scripts are protected against changes to the underlying structure of the database or file organization. And even when using SQL statements directly, the WebGUI API allows you to connect simply and use various quick methods to simplify your code.

 

Existing Utility Scripts

Here are some examples of utility scripts and their uses.

fileImport.pl

If you have a number of files to add to a server, it can be a slow and inconvenient process to upload them, especially if they are large. If you already have the files on the server, there is an easier way. The fileImport script can load all of the files from a directory (recursively if desired) on the server into the appropriate uploads location. This is much faster and bypasses any limits you might have on uploading. Like most utility scripts, the first parameter needed is the site's WebGUI config file. This lets the script find the database, file locations, and other information about the site. Additionally, it needs the location of the files to import, and the asset to place them under.

 

$ sudo Perl fileImport.pl –configFile=dev.localhost.localdomain.conf \

--pathToFiles=/data/import --parentAssetId=PBasset000000000000002

Starting...OK

End of the childs detection

Building file list.Found file newimage.png.

Found file newpage.html.

File list complete.

Adding files...

Adding /data/import/newimage.png to the database.

Create the new asset.

Setting filesystem privilege. Privileges set.

Adding /data/import/newpage.html to the database.

Create the new asset.

Setting filesystem privilege. Privileges set.

Finished adding.

Committing version tag...Done.

Cleaning up...OK



rebuildLineage.pl

While rare, it is possible for the lineage information on assets to get corrupted. Lineage is what allows assets to quickly find their parent or descendant assets, without crawling through every parent to child to child relationship. If something went wrong with this script, it would cause large problems with the site, so it is good to have a backup before doing so.

 

$ Perl rebuildLineage.pl –configFile=dev.localhost.localdomain.conf

Starting...OK

Looking for descendant replationships...

Got the relationships.



Looking for orphans...

No orphans found.



Disabling constraints on lineage...

Rebuilding lineage...

Asset ID Old Lineage New Lineage

PBasset000000000000001 000001 000001

PBasset000000000000002 000001000001 000001000001

...

IWFxZDyGhQ3-SLZhELa3qw 000001000002000010 000001000002000008

PBasset000000000000003 000001000003 000001000003

tempspace0000000000000 000001000004 000001000004



Re-enabling constraints on lineage...



Repairing search index...

Cleaning up...OK



Don't forget to clear your cache.



 

search.pl

This script has two functions. The first is to rebuild the search index. It can also be used to search the index for matching assets. When indexing, it can be told to act on all sites on the server instead of only one. It may be useful to reindex a search if your database search limits are changed. Also, it is possible to, either programmatically or through SQL, modify assets, and this will allow you to create the index so they have to be found through search.

 

$ Perl search.pl --configFile=dev.localhost.localdomain.conf --search Features

4Yfz9hqBqM8OYMGuQK8oLw Get Features

OhdaFLE7sXOzo_SIP2ZUgA Welcome



Search took 0.002149 seconds.

 

Useful API Functions

Included here are examples of useful API functions.

 

WebGUI::Session

This is the core of any utility script. The WebGUI session contains information about configuration files, the database connection, the current user, and more. The first step in any utility script is usually to initialize a WebGUI session.

 

open method

This initializes a WebGUI session. The needed parameters are the WebGUI root directory, and the name of the config file for the site. There are other optional parameters which are used only when run as a web service.

 

config method

This gets an instance of the configuration object, and allows you to get or set any option in the configuration file. You may want to add or remove macros from the list. Other options, like the location of the uploads, are also available here.

 

settings method

This provides access to all of WebGUI's settings, which are normally set in the Settings screen of the Admin Console.

 

db method

While it is usually safer and easier to use the other API's to interact with WebGUI's database, sometimes you need the power of raw SQL. The $session->db method gives you a WebGUI::SQL object, with easy to use methods like quickArray or quickHash to make database interactions fast and simple.

 

Most other object creation methods require a session object as their first parameter.

 

WebGUI::Asset

Here are some method examples for WebGUI::Asset.

 

new method

If you already know the ID of an asset, you can use the WebGUI::Asset->new method to retrieve an object instance of it, and then use its display or modification methods to perform whatever actions you need. These ID's may have come from an SQL statement, or a getLineage call.

 

newByUrl method

Similar to new, but given a URL instead of an asset ID.

 

getRoot method

Sometimes, you don't have any particular starting point, but want to operate on the entire site. All other assets are descendants of the Root asset, so it is a good starting point.

 

getLineage

getLineage is one of the most powerful functions in WebGUI's API. You can use it to find related assets by defining a set of rules for which to select.

 

WebGUI::User

Here are some method examples for WebGUI::User.

new, newByUsername, newByEmail methods

These methods get a user object given either a user ID, login name, or email address.

 

profileField method

This method gets or sets the value of a specific profile field for a user. With a single parameter, the value of that profile field is returned. If a second parameter is given, the profile field is set to that value.

 

A Simple Utility Script : settings.pl

WebGUI has a good number of global settings, dealing with things like the commit process and user authentication. Usually, these are adjusted through the Setting screen in the Admin Console. It can be useful to control them through the command line or as part of an automated process. While you can connect to the database and use SQL to set these, looking up the needed information and typing out the commands can be time consuming. You're better off creating a small script to do the work for you. While you could start from scratch every time you want to create a script, a utility script skeleton is included with the WebGUI distribution. The first step when creating a utility script is to create a copy of this skeleton, then open it in an editor.

 

$ cd /data/WebGUI/sbin

$ cp _utility.skeleton setting.pl

$ vim setting.pl



use lib "../lib";

use strict;

use Getopt::Long;

use WebGUI::Session;



my $session = start();

# Do your work here

finish($session);



#-------------------------------------------------

# Your sub here



#-------------------------------------------------

sub start {

my $configFile;

$| = 1; #disable output buffering

GetOptions(

'configFile=s' => \$configFile,

);

...

 

There are some subs predefined in the script, but to start with, look at the line near the top saying # Do your work here. At this point the default script has already initialized a WebGUI session, which is what is required to make changes. This example starts simple with the script: make it report the current value of a setting. Below the call to start(), add:

 

my $name = shift @ARGV;

my $value = $session->setting->get($name);

print $value, “\n”;

 

Now you can save the script and run it on the command line, giving it the additional parameter of companyName.

 

$ Perl setting.pl –configFile=dev.localhost.localdomain.conf companyName

My Company

 

In addition to the script name and the setting to retrieve, you need to provide the site's configuration file. Since there can be multiple sites on one server, this is a necessary command for most scripts. The configFile parameter is handled in the start sub, which you can adjust soon. Of the code added, the part that is doing the important work is $session->setting->get($name), which uses the setting object to retrieve the setting.

 

Just retrieving one setting from the database isn't very useful, though. You can adjust the script to work with multiple settings and set them as well as retrieve. Use a loop for parameters, and allow use = to indicate a new value to store in the database.

 

# Loop through parameters

for my $setting (@ARGV) {

# Check for an '=' in the parameter

if ($setting =~ /^(\w+)=(.*)/) {

# If there was, our regular expression matched the name and requested value

my ($name, $value) = ($1, $2);

# Report the old value to help the user detect any mistakes

print $name . " - old: " . $session->setting->get($name);

# Set the new value

$session->setting->set($name, $value);

# Report the new value as well for easy comparison

print " new: " . $session->setting->get($name) . "\n";

}

else {

# If there was no '=', we just have a name.

# Report the value of that setting

print $setting . " - " . $session->setting->get($setting) . "\n";

}

}

 

In this case, there are multiple calls on the setting object, both to set and get values. Now run:

 

$ Perl setting.pl –configFile=dev.localhost.localdomain.conf companyName companyEmail=webmaster@example.com

companyName - My Company

companyEmail - old: info@mycompany.com new: webmaster@example.com

 

Now one setting has been retrieved, and another set in one line.

 

These setting names aren't always obvious though. You may only know, or be able to guess, part of the name. Unfortunately, the settings API doesn't provide a way to list WebGUI's settings. For this task, you'll have to go directly to the database. The first change you'll need to make is to allow you to request a list of settings. You'll want to add a variable to store this flag:

 

use Getopt::Long;

use WebGUI::Session;



my $optList;

my $session = start();

 

Next, add the parameter to the command line parser. The start sub uses the standard Perl module Getopt::Long to parse the –configFile parameter. You don't need to get a value like that option; you only need a simple flag. So add an entry to use the --list parameter, and save the flag in your variable:

 

GetOptions(

'configFile=s' => \$configFile,

'list' => \$optList,

);



 

Now you just need to operate on this flag. The database table 'settings' is what contains your settings, so select all the names that match your parameters. You can adjust your existing setting loop:

 

for my $setting (@ARGV) {

# If we are in list mode

if ($optList) {

# buildArray selects all the results into an array

my @settings = $session->db->buildArray('SELECT name FROM settings WHERE name LIKE ?', ['%' . $setting . '%']);

# Loop through our list of settings

for my $name (@settings) {

# and output their names

print $name, "\n";

}

}

# Otherwise, operate like before.

elsif ($setting =~ /^(\w+)=(.*)/) {

 

Now, if you don't specify the list option, it operates exactly like before, but with the list option you can see all the settings related to email:

 

$ Perl setting.pl –configFile=dev.localhost.localdomain.conf --list email

companyEmail

userInvitationsEmailExists

userInvitationsEmailTemplateId

webguiRecoverPasswordEmail

webguiValidateEmail

 

Package Maintenance: Short but Powerful

WebGUI packages are a convenient way to share content you've created on your site with other WebGUI sites. You may also run a number of sites that share certain assets. You may want to use packages to synchronize these assets, but the process of exporting and importing can be time consuming when done manually. One option is to create a script so you can automate the process. Then it can be a single command to update your sites, or could even be done as a scheduled task. Start again with the utility skeleton.

 

$ cd /data/WebGUI/sbin

$ cp _utility.skeleton package.pl

$ vim package.pl

 

There are several WebGUI modules you'll need to do your work, as well as some regular Perl file modules. There are also three specific pieces of information you'll need from the user:

 

  1. the directory to export to

  2. the package to import

  3. the location to import to

 

use lib "../lib";

use strict;

use Getopt::Long;

use WebGUI::Session;

use WebGUI::Asset;

use WebGUI::Storage;

use File::Copy qw(copy);

use File::Spec;



my $outputDir = '.';

my $importPackage;

my $importRoot;

my $session = start();

 

Next, you need to set the script up to parse these parameters. In the start sub, you'll need to add the three parameters. Additionally, the process of importing packages will create a version tag. Unless you specifically handle this version tag, it will be left uncommitted. There is sample code provided in the start and finish subs to create and commit a version tag, so uncomment it and choose an appropriate name.

 

sub start {

my $configFile;

$| = 1; #disable output buffering

GetOptions(

'configFile=s' => \$configFile,

'outputDir=s' => \$outputDir,

'import=s' => \$importPackage,

'root=s' => \$importRoot,

);

my $session = WebGUI::Session->open("..",$configFile);

$session->user({userId=>3});



my $versionTag = WebGUI::VersionTag->getWorking($session);

$versionTag->set({name => 'Package Import'});

 

return $session;

}



#-------------------------------------------------

sub finish {

my $session = shift;



my $versionTag = WebGUI::VersionTag->getWorking($session);

$versionTag->commit;



$session->var->end;

$session->close;

}

 

You now have the appropriate information gathered and can start doing the actual work. After the call to start(), you'll want to choose between two different modes, importing and exporting. If you're importing, you need to create a storage location for your package, add the package file to it, and call the package importing method on your root asset. To export, you'll need to get your asset object from a URL, export it as a package, and copy the package file to the requested location.

 



# First, check if we are importing or exporting

if ($importPackage) {

# Importing requires a storage location, create a temporary one

my $storage = WebGUI::Storage->createTemp($session);

# Add out package file to that storage location

$storage->addFileFromFilesystem($importPackage);

# Now we need a base asset to import into. The user can specify the URL

# to use as a parent, or we'll default to the import node.

my $asset = $importRoot ? WebGUI::Asset->newByUrl($session, $importRoot) : WebGUI::Asset->getImportNode($session);

# The import process itself is easy, just a single call returning the new asset

my $newAsset = $asset->importPackage($storage);

# Report out success

print "imported $importPackage to " . $newAsset->get('url') . "\n";

}

# Otherwise, we are exporting

else {

# Calculate full path to output files to first

my $fullOutputDir = File::Spec->rel2abs($outputDir);

# Multiple URLs can be specified to export, loop through them

for my $url (@ARGV) {

# Find the corresponding asset

my $asset = WebGUI::Asset->newByUrl($session, $url);

# If there isn't a matching URL on the site, warn

# the user and try the next one

if (!$asset) {

warn "No asset with URL: $url\n";

next;

}

# Exporting creates a storage location with the package file

my $storage = $asset->exportPackage;

# There will only be one file, the package. Get its filename

my $file = $storage->getFiles->[0];

# Calculate the full path we'll be outputing this file to

my $outputFile = "$fullOutputDir/$file";

# Copy our package from its storage location to the user specified location

copy($storage->getPath($file), $outputFile);

# Report our success

print "created package $outputDir/$file for $url\n";

}

}

 

Now exporting is as simple as running the script and giving it a list of URL's.

 

$ Perl package.pl --configFile=dev.localhost.localdomain.conf home

created package ./home.wgpkg for home

 

You can then import this package just as easily.

 

$ Perl package.pl --configFile=localhost.conf --import=home.wgpkg

imported home.wgpkg to home

 

Apache Authentication: When a Script isn't a Script

While writing scripts to run on the command line is a useful way to interact with the WebGUI API, the same techniques can be applied anywhere Perl is used. One powerful place to use Perl, as WebGUI is evidence of, is inside Apache using mod_Perl. With mod_Perl, you have full access to Apache's request cycle and can implement handlers for URL parsing, authentication and access control, content handling, and even logging. WebGUI is implemented as a series of Apache handlers, but takes control of the entire request process.

 

You may have files available that you want to be handled by Apache's normal request mechanisms, but controlling access to these files would normally mean creating files with user and password information for everyone you want to be able to access them. If you already have users and groups in WebGUI for these people, this is a duplication of effort. With mod_Perl, though, you authenticate against WebGUI's database, leaving the rest of the request to be handled as normal.

 

This script is a large departure from your previous command line based scripts, so starting with the skeleton wouldn't provide you with anything. In this case, start with a blank file.

 

$ vim apacheAuth.pl

First, you have to import all the required modules from WebGUI and Apache. You''ll also want to prevent the script from being accidentally run on the command line.

 

BEGIN { # Prevent command line execution

if (!$ENV{MOD_Perl}) { # This variable is always defined by mod_Perl

die "This script must be run by Apache 2\n";

}

}



package Apache2::AuthWebGUI;

use strict;

use warnings;

use WebGUI::Session; # Load the parts of WebGUI that are needed

use WebGUI::Utility;



use Apache2::Access (); # Load the Apache modules that are used

use Apache2::RequestUtil ();

use Apache2::Const -compile => qw(OK DECLINED HTTP_UNAUTHORIZED SERVER_ERROR);

 

Apache has separate phases for authentication and authorization, so your first sub is just to identify the user. Start with the basics of a mod_Perl handler and get the request object and the server object. Next, initialize a WebGUI session. WebGUI sessions know how to handle those objects, so you'll provide them to simplify parts of your code.

 

# Authentication sub

# Add to httpd.conf as PerlAuthenHandler Apache2::AuthWebGUI::authen

# All we are doing is identifying the user

sub authen {

my $r = shift; # Apache Request handle

my $s = Apache2::ServerUtil->server; # Server handle



# Create a WebGUI session, use the same configuration settings as WebGUI

my $session = WebGUI::Session->open($s->dir_config('WebguiRoot'), $r->dir_config('WebguiConfig'), $r, $s);

 

This code handles two types of authentication: basic auth and cookie based. Basic auth is part of the HTTP standard, with the user information being provided in the HTTP headers. The interface for entering this information is determined by the client, though, and cannot be integrated into a website's pages. This makes it less suited for a media rich website, but ideal when working with a command line client, such as wget. Cookie based auth is what WebGUI normally uses. By reusing that cookie, users are allowed to log into WebGUI, then be automatically authenticated to a file store protected by this handler. When using cookie based auth, the session object does all the work for you. It will determine what user is logged in, and you can save that information with Apache.

 

# Two auth modes, by default we allow both to be used

my $mode = $r->dir_config('WebguiAuthMode') || 'both';



# First is cookie mode, uses the session cookie from webgui.

if ($mode eq 'cookie' || $mode eq 'both') {

# The session will pick out the cookie and find the user it is logged in as

my $userId = $session->user->userId;

# If it isn't visitor, they are authenticated

if ($userId ne '1') {

# This is a shortcut for the authz handler so we don't have to

# create a new session or keep track of one across multiple handlers

checkGroupAccess($session);

# set the username for the request object

$r->user($session->user->username);

$session->close;

return Apache2::Const::OK;

}

}

 

For basic auth, you first have to request the authentication information from the client. After doing that, use WebGUI's authentication system to verify the username and password.

 

# Next try basic auth, part of the HTTP standard

if ($mode eq 'basic' || $mode eq 'both') {

# Check if the browser sent auth info

my ($status, $password) = $r->get_basic_auth_pw;

# It didn't or something else went wrong.

# Tell the browser we need login information.

if ($status != Apache2::Const::OK) {

$session->var->end;

$session->close;

return $status;

}

 

# The request has a user, find the auth settings for that user

my $user = $r->user;

my $auth = getAuth($session, $user);

if (!$auth) {

$session->var->end;

$session->close;

return Apache2::Const::SERVER_ERROR;

}



# Check the the password, if it checks out, they are logged in

if ($auth->authenticate($user, $password)) {

# Our shortcut again for the authz handler

checkGroupAccess($session, $user);

$session->var->end;

$session->close;

return Apache2::Const::OK;

}



# Otherwise, note it for logs

$r->note_basic_auth_failure;

}

# We didn't success, so clean up and return the failure

$session->var->end;

$session->close;

 

return Apache2::Const::HTTP_UNAUTHORIZED;

}

 

In order to check the identity of a user given its username and password, you need to get a WebGUI authorization object to check it with. The sub getAuth retrieves this for a given user.

 

sub getAuth {

my $session = shift;

my $user = shift;

# need to find out the authentication method, and only use it if valid

(my $method) = $session->db->quickArray("select authMethod from users where username=?", [$user]);

# Valid methods are listed in our config file

if (!isIn($method, @{$session->config->get("authMethods")})) {

$method = $session->setting->get("authMethod");

}

# make sure the module is loaded

my $module = "WebGUI::Auth::$method";

(my $modFile = "$module.pm") =~ s{::}{/}g;

if (!eval { require $modFile; 1 }) {

$session->errorHandler->error("Authentication module failed to compile: $module: $@");

return;

}

# create a new instance of the auth module, or log a failure

my $auth = eval { $module->new($session, $method) };

if (!$auth) {

$session->errorHandler->error("Couldn't instantiate authentication module: $method. Root cause: $@");

return;

}

return $auth;

}

 

After running this handler, Apache will have verified the identity of the user, and will be able to use that username for any of its normal authorization controls.

 

Then, configure this authorization handler in your Apache config.

 

# Uses the same configuration as normal WebGUI

PerlSetVar WebguiConfig dev.localhost.localdomain.conf

# Load the file with the code in it

PerlPostConfigRequire /data/WebGUI/sbin/apacheAuth.pl

# Register it to handle authentication

PerlAuthenHandler Apache2::AuthWebGUI::authen



<Location />

PerlSetVar WebguiAuthMode both # Allow session cookie or basic auth

AuthType Basic # Setup for basic auth

AuthName "WebGUI Data Area"

Require valid-user # Require a valid login to get access

</Location>

 

This requires that the user can log in in some form, but doesn't check any group memberships. What if you have different users with different access levels, and want to restrict different files to different groups? For that, you need an authorization handler. Authorization takes place after the user is identified, to check if that user should have access to a given resource. For this handler, you need to take the username and check if the user is a member of the groups you are allowing access. When configuring the directory access, you'll specify a whitespace separated list of group ID's that have access.

 

# Check authorization settings to see if user has access

sub checkGroupAccess {

my $session = shift;

my $username = shift;

my $r = $session->request;

# Session's user if no user specified

my $user = $username ? WebGUI::User->newByUsername($session, $username) : $session->user;

# Read config, split required groups into a list

my $groupsToAccess = $session->request->dir_config('WebguiAuthRequireGroup');

return

unless $groupsToAccess;

if (!ref $groupsToAccess) {

$groupsToAccess = [split /\s+/, $groupsToAccess];

}

# if the user is in any of the groups, make as success

for my $groupId (@$groupsToAccess) {

if ($user->isInGroup($groupId)) {

$r->pnotes(WebGUIAuthGroupAccess => 1);

return;

}

}

return;

}

 

This example cheats a bit. This sub is called in the authentication handler, not the authorization handler. This simplifies the session handling. The caveat is that you won't be able to authorize using WebGUI groups without first authenticating using WebGUI, but the odds of wanting that are very low. When this is processed, it saves a simple true or false in the Apache request object. The actual authorization handler, then, only has to check this value.

 

# would be expensive to reinitialize session, so we use the data we saved earlier

sub authz {

my $r = shift; # Apache Request handle

if ($r->pnotes('WebGUIAuthGroupAccess')) { # Our saved value

return Apache2::Const::OK;

}

return Apache2::Const::HTTP_UNAUTHORIZED;

}

 

Configuring this only requires a small change to the Apache config.

 

# Uses the same configuration as normal WebGUI

PerlSetVar WebguiConfig dev.localhost.localdomain.conf

# Load the file with the code in it

PerlPostConfigRequire /data/WebGUI/sbin/apacheAuth.pl

# Register it to handle authentication

PerlAuthenHandler Apache2::AuthWebGUI::authen

# Register it to handle authorization

PerlAuthzHandler Apache2::AuthWebGUI::authz



<Location />

PerlSetVar WebguiAuthMode both # Allow session cookie or basic auth

PerlSetVar WebguiAuthRequireGroup 11 # Only allow access to secondary admins

AuthType Basic # Setup for basic auth

AuthName "WebGUI Data Area"

Require valid-user # Require a valid login to get access

</Location>

 

You can add additional location sections with their own WebguiAuthRequireGroup directives to have separate restrictions for different paths.

 

While the main point of WebGUI is to display assets, you can see many places where interacting with the API outside of a web interface is useful. You may want to perform maintenance on your sites, or you might want to use local files on your server. Or, you can use one of WebGUI's subsystems to perform tasks as part of a different program. Any time you need to direct the entire flow of execution, and not have it controlled for you, utility scripts will answer this need.

Keywords: apache API assets config file maintenance utilities

Search | Most Popular | Recent Changes | Wiki Home
© 2018 Plain Black Corporation | All Rights Reserved