plainblack.com
Username Password
search
Bookmark and Share
Subscribe

FilePump

FilePump

Introduction

FilePump is a tool for speeding up your site. It is a WebGUI core feature since WebGUI 7.7.7 (stable). By bundling multiple external files (either stylesheets or scripts) into a single file, the number of requests during the loading of a page can be reduced. Moreover, FilePump will reduce the size of the files by removing whitespace and comments. Depending on the amount and the size of external files, using FilePump can reduce page load time significantly.

 

Quickstart Guide

FilePump can be set up in a few simple steps. Before you start, make sure the FilePump macro is on (you can check this by looking it up in "Macros, List of Available" in the help).

1. Make a bundle

You can find the FilePump configuration screen by clicking on "FilePump" in the Admin Console. In the menu on the right, click on the option "Add a bundle". Add a name for the bundle and click "save". The bundle will appear in the list.

2. Add files

Click on the Bundle and add some of the Javascript, CSS, and collateral (images etc..) files that you are using on your site. These are the files that you want FilePump to streamline.

Files are added as "URI"s. A URI is just a more general form of URLs where you specify the resource type, followed by a colon (:), followed by the path. For example,

  • For a local file inside your uploads folder, enter: file:uploads/myfile.js
  • For a remote file, enter: http://site.com/remotefile.js
  • For local WebGUI assets, enter: asset:/path/to/webgui/asset
  • You can also use absolute paths to your own site, e.g. http://mysite.com/myfile.js

 

By using URIs we make it possible to extend FilePump in the future to support additional types of resources.

3. Build the bundle

Before you can use the bundle, you have to build it by clicking the "Build this bundle link" in the right hand menu. You can see when the bundle is last built by looking at the "last modified" date next to the added files (or in the Bundle list). FilePump will warn you if any of the resources you have specified can't be found.

4. Add CSS with the FilePump macro

You can add a bundle to your site by adding the FilePump macro to the style template. In the head section, you can add the CSS bundle like this:

^FilePump(MyBundleName,css);

The first parameter is the bundle name (you gave the bundle a name in step 1, remember?), the second one determines the type of file you want to add: "js" (or "javascript") or "css". This is case insensitive.

Optionally, you can add extra attributes. For example if you have a bundle with print stylesheets:

^FilePump('MyPrintBundle','css','media="print"');

5. Add scripts with the FilePump macro

If you have added your CSS, why not bundle your scripts as well? Add them to the bundle, rebuild it and add this to your style template:

^FilePump(MyBundleName,js);

Best practice is to add this at the bottom of you template, right before the closing body tag.

That's it! The files you have added to your bundle will now be served by FilePump as optimised minified, concatenated and auto-revved) my-bundle.css and my-bundle.js files. You can now go ahead and remove any manual references to them from your template.

Troubleshooting

Files are concatenated in the order you list them in the bundle, so make sure you maintain the same order that you would normally use when listing the files in your html.

When you view your site with Admin Mode on, FilePump will automatically display your original files rather than "filepumped" files. This makes life easier when debugging your code.

For more information, read on!

 

The Full Details

FilePump is all about maximizing front-end performance, in other words, YSlow score. 

Any non-trivial WebGUI site is going to have a custom look and feel, which will invariably involve a bunch of design files which are referred to in custom WebGUI templates. By design files I mean all the usual suspects such as Javascript, CSS, CSS images, xml, etc.. The number and size of these files is only going to increase in the future, especially as people start embracing the YUI Widget framework which encourages breaking up your javascript into reusable modules.

Design Files

I can think of four different ways that people manage design files.

1. Upload them into WebGUI via FilePile and refer to them in Templates via the FileUrl macro.

Pros: All the benefits of WebGUI Assets (Security, Version Tags, AssetProxy macro, etc..).

Cons: The edit-view feedback loop isn't great (you need to edit files locally and re-upload, then view), plus, most of the Pros mentioned aren't really all that useful for design files.

 

2. Paste them into WebGUI as Snippets.

Pros: Same as (1)

Cons: The edit-view feedback loop is better, but you still need to do a lot of navigation gymnastics. And editing via a codearea is probably not as nice as via your own text editor/IDE.

 

3. Drop the site-specific files into a custom folder in the uploads folder (preferable to using the extras folder because uploads can only be accessed on a per-site basis), and edit them directly on the file-system (assuming you have ssh access). 

Pros: Shortest edit-view feedback loop. Allows you to keep files under an external version control system such as svn or git, which, among other things, makes it easier to sync changes between development, staging and production servers.

Cons: Requires direct file access to server and technical know-how.

 

4. Keep the files onto an external CDN, and refer to them via fully-qualified URLs. For example, people who chose to serve their YUI files off the Yahoo or Google CDN are following this approach.

Pros: Fastest delivery

Cons: Costs $$$

 

FilePump supports all four approaches (although if you're paying for a CDN you probably want to keep on serving your files via it rather than using FilePump).

 

The Static Folder

FilePump requires a new folder be added to uploads called filepump. You should add the following mod_expires rule to /data/wre/etc/site.modproxy so that modproxy serves up all content in filepump with a far-future expiry header:

       <Location /uploads/filepump>

          ExpiresActive On

          ExpiresDefault "access plus 10 years"

       </Location>

 

The physical location of this folder is /data/domains/site.com/public/uploads/filepump but of course it can be a symlink to somewhere more convenient for designers to access. Probably the "correct" place to put it would be under public, and of course Apache is flexible enough to allow us to do that, but inside uploads seems like the best place to put it to avoid conflicts with existing URLs on deployed sites. Other suggestions welcome.

FilePump automatically bundles up your design files (see definition above), minifying, concatenating and revving them in the process, and drops them into a build dir in the filepump folder so that they can be served up in the most efficient manner possible. All with minimal inconvenience to designers. And with some nice extra features to boot.

 

Bundles

Imagine you have a simple website with 3 javascript files:

  • a WebGUI Asset located at /my-asset.js
  • a WebGUI Snippet located at /my-snippet.js
  • a file located at /uploads/static/my-file.js

..and 3  CSS files:

  • a WebGUI Asset located at /my-asset.css
  • a WebGUI Snippet located at /my-snippet.css
  • a file located at /uploads/static/my-file.css

And 3 CSS images:

  • a WebGUI asset located at /my-asset.jpg
  • a file located at /uploads/static/my-file.jpg
  • a remote image located at http://images.com/remote.jpg

All pages make use of the same 9 design files, so you decide to throw them all at the user on the first page view and then let the browser permanently cache it after that. To do this, you create a single bundle called "my bundle". You do this by going to the FilePump section of the Admin Console and clicking "New Bundle". Inside the bundle you add some Javascript files by clicking a button that says "Add Javascript File(s)" and then specifying one or more URLs. Later, FilePump will examine each URL and intelligently resolve it as an Asset, a local file access, a local web request or remote web request. For local files, you can use a file glob to specify multiple files (handy when you have a lot of images).

CSS files are added via the "Add CSS File(s)" button in the same fashion.

All other files are referred to as Collateral, are added via the "Add Collateral" button in the same fashion. Mostly thesewill be CSS images, but in general they can be anything you like. The point here is that FilePump does special things for Javascript and CSS, whereas "Other" files are treated generically.

 

FilePump Macro and Build Process

FilePump contains a ^FilePump(bundle name, [ javascript | css ]); macro that is used to insert bundles (by name) wherever you want them (in this case into your Style template). You can tell FilePump to build a bundle via the FilePump admin interface, by clicking the "Build" button. As we will see later, FilePump tells you when you might want to do this.

The build process works as follows:

  • A timestamped, bundle-specific build folder is created inside the filepump folder, e.g. filepump/my-bundle.1235012562
  • The contents of all Javascript files in the bundle are resolved, concatenated (in order) and minified to <bundle name>.js in the build folder
  • The contents of all CSS files in the bundle are resolved, concatenated (in order) and minified to <bundle name>.css in the build folder
  • All collateral files (in our case the css images) are copied as-is into the new folder.
  • If the URL resolves to a directory instead of a file, the entire directory is copied to the build folder (this is useful for CSS images, as explained later).
  • Any build dirs lying around from previous builds are removed

Then, with your newly defined and built bundle, you update your style template to look like this:

       <html>
         <head>
         <!-- css at the top -->
         ^FilePump(my bundle, css);
         </head>
         <body>
         ...
         <!-- js at the bottom -->
         ^FilePump(my bundle, javascript);
         </body>
       </html>

When users view your site, they see the following:

       <html>
         <head>
           <link rel="stylesheet" type="text/css" href="/uploads/filepump/my-bundle.1235012562/my-bundle.css">
         </head>
         <body>
         ...
         <!-- js at the bottom -->
           <script type="text/javascript" src="/uploads/filepump/my-bundle.1235012562/my-bundle.js">
        </body>
       </html>

Both of those files (and any images referred to in the CSS file) are permanently cached in the users' browser so that they are only requested once (without subsequent conditional-get requests). The bundle was pre-built, so WebGUI didn't have to do any work at the time of the page request. And if you ever change any files and rebuild the bundle, the src path will contain a different timestamped folder so caching does not prevent users from immediately seeing the new files.

 

Admin Mode

Now, as a designer you'd much rather see the original unminified, fully-commented source files. FilePump makes this easy. Whenever you view your site in Admin Mode, you instead see:

       <html>
        <head>
          <link rel="stylesheet" type="text/css" href="/my-asset.css">
          <link rel="stylesheet" type="text/css" href="/my-snippet.css">
          <link rel="stylesheet" type="text/css" href="/uploads/static/my-file.css">
        </head>
        <body>
        ...
        <!-- js at the bottom -->
          <script type="text/javascript" src="/my-asset.js">
          <script type="text/javascript" src="/my-snippet.js">
          <script type="text/javascript" src="/uploads/static/my-file.js">
        </body>
       </html>

that is, all the design files in their original, un-bundled location. Handy, right?

 

Variations

In a more complex, real-world website, it's much more efficient to create multiple bundles. For example, you might have a base-level bundle that everyone gets on the front page. After signing in you might want to then add a second bundle, containing design files for content only accessible to registered users. In this way you give both visitors and registered users a highly optimized set of files, without forcing them to download files they don't need.

That's easy to do too. You just create as many bundles as you like in the FilePump admin console. A single Template can call as many different bundles as you like, and conversely, you can use FilePump to call the same bundle from different Templates. This is a great way of ensuring that when you add/remove/rename a new design file, all relevant templates are updated accordingly. You can also have the same design file bundled up in multiple different bundles if you want to do that too.

Aside from making online debugging easier, Admin Mode has one big plus: the files are being linked from their original, un-bundled location, which means that as soon as the designer edits them, they immediately see the effect of their changes. Meanwhile, regular users keep on seeing the pre-built version. Just remember to rebuild if you want users to see your changes! FilePump will indicate which bundles are out of date, but currently you only see that notification inside the FilePump admin area. This will hopefully improve in the future.

When building an entirely new skin some designers like to do so under a new Version Tag. In this case, they can create an entirely new bundle for the new skin, and refer to the new bundle name in the Template inside the new version tag. There are lots of ways we could integrate FilePile with Version Tags, such as having per-version tag builds of bundles, but for now I thought it best to keep things as simple as possible.

 

Implementation Details

This section lists some of the thoughts behind FilePump's design.

In a previous version, we used unique query strings to rev filenames. However we discovered that AdBlock likes to block images with query params (I guess it thinks they're being used to track users). Timestamped build folders negates the need for query strings.

Obviously it's really important that no extra overhead is introduced on regular page requests. This is why bundles are prebuilt. All the macro has to do is look up the last build timestamp for the bundle and spit out the script tags. Minification can take over a second, even with Javascript::Minify::XS and CSS::Minify::XS, particularly if you're bundling up YUI files with your own js. In a previous version we had the bundle automatically rebuilt on every page request for Admin Mode, but that annoyed designers because it slowed down the edit-view feedback loop too much. And besides, designers much prefer seeing their individual files for online debugging via Firebug etc..

 

It's tempting to symlink "Other" design files rather than copying them. However you need to make copies so that designers can edit-view the original files privately.

 

CSS images required a lot of thought. The problem is that CSS images are usually referenced in CSS rules with relative paths. When you concatenate a bunch of CSS files from different locations, you need to make sure that the images remain in the same place relative to the generated <bundle name>.css file. Unless of course you want to go down the route of dynamically rewriting CSS files (which is what we were doing in the previous version). That is why FilePump lets you copy entire directories to the build folder. For example, if your css assumes that all images are inside a relative sub-folder called "my-images", you add the "my-images" folder to the bundle and FilePump copies it into a sub-dir of the build folder. FilePump will warn if two files with the same name have been added to the same bundle (N.B. this feature has not been implemented yet), to make sure you don't end up with name collisions. In the worst case you might have to rename some of your design files. This is really the only sacrifice you have to make to use FilePump. And really, it's a small sacrifice to make for ultimate YSlow glory. 

Future Improvements

The list of available template attachments should include all FilePump bundles. When chosen, the FilePump macro should be automatically inserted into the template, eliminating the need for users to insert the FilePump macro by hand into their templates.

FilePump should prepend the following javascript to the start of each js bundle:

       var WEBGUI_FILEPUMP_PATH = <bundle build dir>;

This would allow Javascript in the bundle to refer to design files such as images in the same bundle. Each new bundle on the page redefines the global variable, but each bundle gets a chance to store a local copy of the path before the next bundle is loaded. e.g.

       var myImagesPath = WEBGUI_FILEPUMP_PATH ? WEBGUI_FILEPUMP_PATH +
       '/img' : '/original/path/to/img';

In the code above, myImagesPath will resolve to a sub-dir of the build folder for users who see the bundled version, and /original/path/to/img for designers viewing the site in Design Mode. Javascript in one bundle should not need to refer to design files in other bundles. If it does, you should add those design files to both bundles.

One thought that came up during this design process, as an entirely separate RFE, is that WebGUI Asset Files might be able to do far-future caching & revving quite easily, as an option on the Edit tab. Without working through the details, it appears all you'd really have to do is make the filename dependent on the version tag (to rev them) and then serve them up with a far-future expires header. It'd be nice to embrace the /uploads/static folder as the officially recommended way of delivering site-specific design files (for more tech savvy users). An ^Uploads() macro, or better yet a ^Static() macro would encourage people to stop putting hard-coded URLs into their templates to refer to these files. FilePump could keep a cached copy of the unmodified original files (perhaps via a git backend). Then it could show you not only which bundles are out of date, but a full diff of changes. This would be pretty awesome.

Speaking of file system version control, it would be nice if we added an extension to FilePump that gave designers an "Update from repository" button, that would automatically run "git pull" or "svn update" in the /uploads/static folder (which for my sites is usually under version control). This would allow designers to edit files on a dev server, check them in, and then pull them down on the live server without needing SSH access to the production server.

We could also look at making bundles dynamic. For example, you could have a single named bundle in your style template that delivers different pre-built files depending on user group, flux rule, etc.. RFE #10971 discusses one possible use of dynamic FilePump bundles (involving default wG templates).

The holy grail would be for FilePump to become aware of YUI module dependencies. One day I'd love to be able to declare in an Assets or in a Templates what YUI modules the Asset requires, using the same module names and dependency tree used by YUI Loader, and then have WebGUI automatically figure out what minimal set of YUI files to load on the page, dynamically based on what assets are being displayed. I don't want to even start thinking about this before YUI 3.x since the YUI Loader mechanism is still evolving, but one day I'm hopeful that WebGUI will be smart enough to know that I need menu-min.js because my Nav menu uses it, and wrap that up with whatever other YUI modules are needed, taking into account the presence of already-cached bundles. Custom YUI widgets can use the same dependency mechanism as official YUI widgets, which means that people could take advantage of these smarts for their own Javascript code too. But I'll leave that for FilePump version 2.

Other References

Keywords: filepump file pump yslow performance

Search | Most Popular | Recent Changes | Wiki Home
© 2018 Plain Black Corporation | All Rights Reserved