| Previous · Next | |
| User | Message |
|
cap10morgan
|
Date: 6/3/2009 2:55 pm · Subject: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
I have added a few features to the webguiupdate.pl script that comes with the WRE. I've attached that and the other files I had to modify / create in order to get it all working.The new features are: 1. automated handling of intermediate upgrades (i.e. when you need to install a particular version's code and run that upgrade before continuing on towards your ultimate destination version) 2. the ability to automatically install new Perl module dependencies into the WRE when a new version you're upgrading to requires them3. the ability to use the SVN repo for grabbing new versions (so you only download the deltas between each release, as well as maintaining other nice aspects of having an SVN working copy rather than an extracted tarball) 4. the ability to run it non-interactively so you can script upgrades on machines where you don't necessarily have console access at the time (like when launching new EC2 instances via Rightscale) I have used this script to successfully and non-interactively upgrade a WebGUI 7.5.40 site running on WRE 0.9.1 to WebGUI 7.7.8, including all intermediate required upgrades and all new Perl modules. However, I wanted to post about it here before creating an RFE because actually adding this to WebGUI core would entail some new responsibilities on the part of Plain Black and/or the community. So I would love to get everyone's feedback on how I've implemented this. I modified webguiupdate.pl heavily, added some new functionality to WRE::WebguiUpdate, and created WRE::WebguiVersion so I could do things like this: my $currentVersion = new WRE::WebguiVersion(version => '7.5.40'); my $newVersion = new WRE::WebguiVersion(version => '7.7.8');if ($currentVersion < $newVersion) { print "Let's upgrade!";} I also created a new file in WebGUI/docs/upgrades called upgrades.json. This is in some ways a machine-readable gotchas file. It obviously doesn't replace gotchas.txt or the need to read that before every upgrade you install, but this JSON file does specify when you need to do an intermediate upgrade before continuing on and when a new release has additional Perl module dependencies. Here's a very short example of what this file looks like (I attached a more complete version as well): { "webgui" : { "modules" : { ...earlier versions go here... "7.7.6" : [ "Business::Tax::VAT::Validation", "Crypt::SSLeay >= 0.57", "Scope::Guard >= 0.03" ], "7.7.7" : [ "Digest::SHA", "JavaScript::Minifier::XS >= 0.05", "CSS::Minifier::XS >= 0.03" ] }, "upgrades" : { "7.5.24" : "7.6.0", "7.5.40" : "7.6.10", "7.6.14" : "7.7.0", "7.7.2" : "7.7.3", "7.7.3" : "7.7.4", "7.7.4" : "7.7.5", "7.7.5" : "7.7.6" }}} So as you can see, you put new module dependencies (optionally including version information) in the "modules" section and you put required intermediate upgrades in the "upgrades" section. The key is the version you need to upgrade to first, and the value is the minimum next version you can upgrade to after that. So you need to specify *all* major version jump points (such as 7.5.24 -> 7.6.0, 7.5.40 -> 7.6.10, and 7.6.14 -> 7.7.0 above) and any other required intermediate releases (such as 7.7.2 -> 7.7.3 above). The script is smart enough to know that the lack of a point release being specified in this list means that the upgrade *is* allowed (i.e. it's fine to jump from 7.7.5 -> 7.7.8 because 7.7.5 says that the minimum next version is 7.7.6, there are no other intermediate upgrades listed, and it's all still within the 7.7.x series), but that a missing major version leap means that the upgrade *is not* supported (i.e. it's NOT OK to upgrade from 7.6.25 to 7.7.8 because there is no way to get from 7.6.25+ to any 7.7.x version specified in the list; once 7.7.x-stable is released, then that will change). So Plain Black would probably have to keep this file up-to-date if this were to be accepted into core, but it's not that difficult and uses information they're already putting into gotchas.txt. JT, what do you think? The automatic Perl module installation works exactly like testEnvironment.pl's similar feature. It tries to require the module, then checks its version, then calls CPAN::Shell->install if either of those checks fail. The SVN repo option requires that your /data/WebGUI is already a working copy of WebGUI from https://svn.webgui.org/plainblack and it runs `svn switch https://svn.webgui.org/plainblack/releases/WebGUI_x.x.x-[stable|beta] .` using whichever version of WebGUI code it needs for that upgrade step. This is very handy for me because I like to have SVN working copies of WebGUI anyway for various reasons. It is also nice in that it only transfers the delta between your currently installed codebase and the release tag that you're upgrading to. It also needed the ability to lookup the version status (i.e. stable or beta) for any given version number. So, for example, if you specify that you want to upgrade to 7.7.8 and you have 7.5.40 installed, it figures out that it will need to do intermediate upgrades to 7.6.14, 7.7.2, 7.7.3, 7.7.4, and 7.7.5 before installing 7.7.8. However, since I need it to be 100% non-interactive, it can't prompt for each of these versions to ask if they're beta or stable. So instead it accesses update.webgui.org and looks at the directory index to grab the status from the filename itself. This is similar to the getLatestVersion check that's already in WRE::WebguiUpdate. I had it send "http://webgui.install.getversionstatus/".$host->getHostname as the referer header because I assumed Plain Black uses that header to track / count / otherwise detect these automated requests on the update.webgui.org server. Note that I changed it to "getversionstatus" so these requests can be differentiated from the getLatestVersion requests, if necessary. To try it out, do this: Put webguiupdate.pl in /data/wre/sbin/Put WebguiUpdate.pm in /data/wre/lib/WRE/Put WebguiVersion.pm in /data/wre/lib/WRE/ Put upgrades.json in /data/WebGUI/docs/upgrades/ Then run: sudo /data/wre/sbin/webguiupdate.pl --nonInteractive --upgradeToVersion=x.x.x-[beta|stable] --useSubversion (or --getFromMirror=[plainblack|sourceforge] or --path=/my/webgui/tarball.gz) Oh, I also fixed the bug where --skipBackup wasn't actually being propagated to the upgrade.pl script. Is this all RFE-worthy? Or do folks want to discuss how I've implemented this before I submit it? Wes -- "Small acts of humanity amid the chaos of inhumanity provide hope. But small acts are insufficient." - Paul Rusesabagina, Rwandan and former hotel manager whose actions inspired the movie Hotel Rwanda Attached Files |
| Back to Top |
Rate [ | ]
|
|
cap10morgan
|
Date: 6/3/2009 3:08 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
Ack! Sorry for the formatting mess in the previous message. Gmail decided to insert some HTML that turned it all into a line-break free wall of text. :/ I can't seem to fix it in the collab system on the site, either. |
| Back to Top |
Rate [ | ]
|
|
JT
|
Date: 6/3/2009 3:29 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
Having not looked a the code, on principal I like the idea. The SVN integration part bothers me for two reasons. 1) We're not going to be using SVN as of July. 2) It seems like it would encourage people to run out of SVN, which isn't ever a good idea. I haven't figured out exactly where the json comes into play, but I suspect that will be clear once I look at the code. My gut feeling is that should be published as a web accessible file or a web service. I don't think it's that big of a deal for us to maintain. It's just a matter of creating a policy and a procedure for how it is maintained. I'll review the code at a later date (today I'm swamped) and then we'll determine where to go from there. Perhaps others have feedback as well. |
| Back to Top |
Rate [ | ]
|
|
cap10morgan
|
Date: 6/3/2009 3:35 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
On Wed, Jun 3, 2009 at 2:29 PM, <jt@plainblack.com> wrote: JT wrote:I was planning to switch it to --useGit at that point. :) Oh? Even if you're running a working copy of a release tag? That would make a lot of sense, yes. Thanks, JT.
-- "Small acts of humanity amid the chaos of inhumanity provide hope. But small acts are insufficient." - Paul Rusesabagina, Rwandan and former hotel manager whose actions inspired the movie Hotel Rwanda |
| Back to Top |
Rate [ | ]
|
|
JT
|
Date: 6/3/2009 4:50 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
> Oh? Even if you're running a working copy of a release tag?Unless it's a dev box that you don't care about it getting trashed, no. And if you're doing that, you shouldn't need WRE to manage it for you. JT Smith ph: 703-286-2525 x810 fx: 312-264-5382 Create like a god. Command like a king. Work like a slave. |
| Back to Top |
Rate [ | ]
|
|
cap10morgan
|
Date: 6/3/2009 4:55 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
On Wed, Jun 3, 2009 at 3:50 PM, <jt@plainblack.com> wrote: JT wrote: Why is that? What will "trash" it if it's an svn working copy? I ask because I have a WebGUI app server template in Rightscale that checks out the requested version from SVN and runs from that. It's not in production yet, so I'd like to understand why this is a bad idea before it is.
-- "Small acts of humanity amid the chaos of inhumanity provide hope. But small acts are insufficient." - Paul Rusesabagina, Rwandan and former hotel manager whose actions inspired the movie Hotel Rwanda |
| Back to Top |
Rate [ | ]
|
|
JT
|
Date: 6/3/2009 7:05 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
> Why is that? What will "trash" it if it's an svn working copy? I ask > because I have a WebGUI app server template in Rightscale that > checks out the requested version from SVN and runs from that. It's > not in production yet, so I'd like to understand why this is a bad > idea before it is. There are two things that can/will cause problems. 1) Though very rare, we sometimes make changes to the codebase after we've tagged the release in SVN, but before it gets out to the general public. Out of the hundreds of releases we've had over the years, it's probably only happened 5 or 6 times, but it has happened. When it does happen, it's usually due to a serious bug that was discovered and fixed at the last second. 2) We have no official policy about what is or is not in version control. What I mean is, to date we keep things like TinyMCE and YUI in version control. We also commit back most of our build changes (like create.sql) back to subversion. However, that's just tradition, not policy. We've been talking a lot lately about what we may change as we move to git, and start work on WebGUI 8. Fairly large changes in these areas are on the list of things we're discussing seriously. Both of these things lead to one big bottom line: what is in version control is *not necessarily* the same as what is in a release tarball. I'm entirely uncomfortable recommending anybody run a site out of SVN/ git. If I can't recommend it, then why should I distribute tools that make it easy to do? Ultimately, these things may not destroy your site. But I'm not willing to guarantee it, or even suggest that it's a good idea. If you're an experienced developer/admin, like yourself, feel free to do what you want. I'm not going to tell you how to run your own servers. But I also don't want to support all the people out there who are not experienced and then get themselves in trouble because I distributed a tool that does something I don't think they should do. Does that make sense? JT Smith ph: 703-286-2525 x810 fx: 312-264-5382 Create like a god. Command like a king. Work like a slave. |
| Back to Top |
Rate [ | ]
|
|
cap10morgan
|
Date: 6/3/2009 7:15 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
On Wed, Jun 3, 2009 at 6:05 PM, <jt@plainblack.com> wrote: JT wrote: Yes it does. I'm going to stop using SVN checkouts, and I now agree that we shouldn't release this new webguiupdate.pl script in any official capacity with SVN integration. Thanks for the explanation.
-- "Small acts of humanity amid the chaos of inhumanity provide hope. But small acts are insufficient." - Paul Rusesabagina, Rwandan and former hotel manager whose actions inspired the movie Hotel Rwanda |
| Back to Top |
Rate [ | ]
|
|
patspam
|
Date: 7/7/2009 7:15 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
2) We have no official policy about what is or is not in version Hey JT - if you guys do decide to change things in this area it sounds like there would be a flow-on effect to people like me who maintain (and run sites off) custom branches of the WebGUI codebase? Not necessarily for the worse of course, it just might change our workflow a little? Patrick |
| Back to Top |
Rate [ | ]
|
|
JT
|
Date: 7/7/2009 8:25 pm · Subject: Re: increased automation of WebGUI upgrades (warning: long w/ attachments) · Rating: 0
> Hey JT - if you guys do decide to change things in this area it > sounds like there would be a flow-on effect to people like me who > maintain (and run sites off) custom branches of the WebGUI codebase? > Not necessarily for the worse of course, it just might change our > workflow a little? It might. But what you're doing is also unsupported and unsupportable. You're a bright guy, I'm sure that no matter what changes we make it make make a change in your process, but won't cause a dead stop. |
| Back to Top |
Rate [ | ]
|
