plainblack.com
Username Password
search
Bookmark and Share
Subscribe

Deploy Cloudfront for WebGUI

NOTE: These notes are based on some internal deployment documents at donor.com and may not be 100% perfect or applicable for your situation - please revise and update as you find improvements!

Overview

* ensure you are using the latest version of WebGUI from svn - 7.7.6 at the time of this writing
* signup for an Amazon Web Service account
* create an S3 bucket, publish it as a Cloudfront distribution
* install s3tools on your WebGUI server
* Update WebGUI site config with new CDN section
* create the queuePath dir under /data
* enable the new CDN section in the WebGUI config
* Run synctocdn to migrate your uploads to Cloudfront
* Optionally, migrate your /extras to Cloudfront
* set up cron job for syncing new uploads (only way new uploads are synced out to the cloudfront)



Instructions

* create a bucket in amazon
    - try the Firefox S3 organizer (https://addons.mozilla.org/en-US/firefox/addon/3247)
* publish bucket as a Cloudfront distribution
    - you can do this in the S3 organizer as well
    - wait for the distribution to be deployed - may take 5 to 10 minutes
*Create a CNAME dns entry as an alias for the Cloudfront distribution URL
    - see http://docs.amazonwebservices.com/AmazonCloudFront/latest/DeveloperGuide/index.html?CNAMEs.html
* install s3tools from s3tools.org on your WebGUI server
    - requires python 2.4+
    - a Perl alternative may be available if someone wants to replicate the s3tools functionality in Perl
*create the queuePath dir in /data/domains/<sitename>/cdn (chown webgui:webgui on these dirs)
*WebGUI config updates: (see example configs below)

*update webgui config with bucket name,
*extras needs to be changed to extras version for cdn in webgui.config
*update ssl variables in webgui config

*Set up cron entry on webgui server for syncToCdn.pl using the following as a "sample" cron (run-syncToCdn.sh shell script to loop through configs)
  -One script per server setup:
  -Need to add each server to cfglist in script (see sample cron script below)
*each bucket requires a <bucketname>.s3cfg file in /data/WebGUI/etc (see example below)
*restart apache and webgui (the webgui instance must be restarted to have the cdn enabled for this site before next step)
*setenvironment.sh and run : sudo perl syncToCdn.pl --configfile=test2.dwvpn.net.conf --migrate # migrate the data to CDN
*test

Example webgui.conf file changes

For each site that will use CDN, you need to update that site's WebGUI config file. 

    "cdn" : {
       "enabled" : 1,
       "url" : "http://cdn.example.com",
       "sslAlt" : 0,
       "queuePath" : "/data/domains/example.com/cdn",
       "syncProgram" : "/data/WebGUI/utils/cdn-s3.py 'yourbucket' '%s'",
       "deleteProgram" : "/data/dw/bin/s3del 'yourbucket' '%s'"
    },


Cloudfront does not currently support SSL.  So you need to either server your SSL connections from your WebGUI server or from the Amazon S3 bucket (which is not as fast or distributed as the Cloudfront edge servers).  If you want SSL versions served off Amazon S3 instead of your WebGUI server, set your config like this:

    "cdn" : {
       "enabled" : 1,
       "url" : "http://cdn.example.com",
       "sslAlt" : 1,
       "sslUrl" : "https://yourbucket.s3.amazonaws.com/",

       "queuePath" : "/data/domains/example.com/cdn",
       "syncProgram" : "/data/WebGUI/utils/cdn-s3.py 'yourbucket' '%s'",
       "deleteProgram" : "/data/dw/bin/s3del 'yourbucket' '%s'"
    },

If you want your /extras folder to be served by CDN, you need to additionally add these configs:

    "cdn" : {
       "enabled" : 1,
       "url" : "http://cdn.example.com",
       "sslAlt" : 1,
       "sslUrl" : "https://yourbucket.s3.amazonaws.com/",
       "extrasUrl" : "http://cdn.example.com/extras",
       "extrasSsl" : "https://yourbucket.s3.amazonaws.com/extras",

       "extrasExclude": ["tinymce", "^blah$"],
       "queuePath" : "/data/domains/example.com/cdn",
       "syncProgram" : "/data/WebGUI/utils/cdn-s3.py 'yourbucket' '%s'",
       "deleteProgram" : "/data/dw/bin/s3del 'yourbucket' '%s'"
    },

Since TinyMCE basically uses some AJAX, we need to avoid attempting any cross-site scripting, so the extrasExclude variable allows us to make sure content matching those regexes will be served from the WebGUI server, and not the CDN server, and therefore will not be a XSS issue.  You may find other Javascripts that need to be included in this exclusion variable.

Example cdn-s3.py script

#!/data/wre/prereqs/bin/python

# Configurations (if any)
log = '/dev/null'   # or e.g. /tmp/cdn.log
DEBUG = 0
# end config

from datetime import datetime, timedelta
import locale
import sys
import subprocess

def futureDate():
    """
    futureDate: a date in the pseudo-infinite future, for expiring pseudo-never
    RFC-2616-14.21 says one year max, but prominent examples use 10 years, e.g.
    http://developer.yahoo.com/performance/rules.html
    """
    # stackoverflow.com/questions/225086/rfc-1123-date-representation-in-python
    locale.setlocale(locale.LC_TIME, 'en_US')
    in10yr = datetime.utcnow() + timedelta(days=3650)
    return in10yr.strftime('%a, %d %b %Y %H:%M:%S GMT')

try:
    bucket = sys.argv[1]
    storageLocation = sys.argv[2]
except:
    print 'Usage: %s <bucket> <storageLocation>' % sys.argv[0]
    raise SystemExit
forever = futureDate()
cmd = "/data/wre/prereqs/bin/s3cmd --config=/data/WebGUI/etc/%s.s3cfg put --recursive --acl-public --add-header " % ( bucket )
cmd += "'Expires: %s' '%s' s3://%s/" % (forever, storageLocation, bucket)
cmd += " >> %s 2>&1" % log
if DEBUG > 2:
    print cmd
subprocess.call(cmd, shell=True)

Example s3del script

#!/bin/sh
echo $0 $* >> /tmp/cdn-$1.log
echo "s3cmd del --recursive s3://$1/$2" >> /tmp/cdn-$1.log
/data/wre/prereqs/bin/s3cmd --config=/data/WebGUI/etc/$1.s3cfg del --recursive "s3://$1/$2" >> /tmp/cdn-$1.log
echo status $? >> /tmp/cdn-$1.log

Example cron script for run-syncToCdn.sh (/data/WebGUI/sbin):

If you have multiple sites, this wrapper for syncToCdn.pl can be used to sync content for multiple sites back to their various target Amazon S3 buckets.

#!/bin/bash
. /data/wre/sbin/setenvironment.sh

cd /data/WebGUI/sbin

# next variable needs to be space delimited with no line breaks"
CFG_LST="yourdomain1.conf yourdomain2.conf yourdomain3.conf"

for file in ` echo $CFG_LST`
    do
        perl ./syncToCdn.pl --quiet --configFile=$file
    done
    ret=$?
    if [ $ret -eq 0 ]; then
        exit $ret
        else
        echo "A error occured on SyncToCDN Cronjob from this host"
        exit $ret
        fi
exit 0

Example cron entry for run-syncToCdn.sh:

* * * * * /data/WebGUI/sbin/run-syncToCdn.sh

Example s3cfg file:

If you are using the s3tools, you can use it to configure itself the first time (http://s3tools.org/s3cmd-cloudfront) to setup your s3cfg file, or you can copy this file, put it in your /data/WebGUI/etc/ directory with a filename of <bucket>.s3cfg where bucket is the name of the Amazon S3 bucket used as the source for your Cloudfront CDN.

[default]
access_key = YOURAWSACCESSKEY
acl_public = False
bucket_location = US
cloudfront_host = cloudfront.amazonaws.com
cloudfront_resource = /2008-06-30/distribution
default_mime_type = binary/octet-stream
delete_removed = False
dry_run = False
encoding = UTF-8
encrypt = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = s3.amazonaws.com
host_bucket = %(bucket)s.s3.amazonaws.com
human_readable_sizes = False
list_md5 = False
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
recursive = False
recv_chunk = 4096
secret_key = YOURAWSSECRETKEY
send_chunk = 4096
simpledb_host = sdb.amazonaws.com
skip_existing = False
use_https = False
verbosity = WARNING

Keywords: Cloudfront Amazon S3 CDN Content Deliver Network

Search | Most Popular | Recent Changes | Wiki Home
© 2019 Plain Black Corporation | All Rights Reserved