Back in the early days at PeopleSoft we used to ship software on floppy disks. A whole boatload of floppy disks. There was even this really cool floppy disk duplicator that we had where you would load in a stack of floppies, with the source in one stack, and blanks in the other. I think it could handle something like 50 disks at a time. Jay Hann would know for sure. Anytime the disk duplicator broke (basically whenever you looked at it), he was the one to get it going again.

Then this magical thing called a "CD" started getting popular enough that we could ship all of our software on 1 CD, and have room to spare. Wow! It wasn't quite as good as regular backup tapes, but still a huge leap forward.

A quaint blast from the past, eh? Well, I felt like I was transported back to that same era earlier today while looking at various escrow services out there (that's "escrow", as in code escrow, not escort services Mr. Spitzer). You see, we ship a lot of source with our applications, but we do have some parts that we do not ship the source to (and yes, just shipping all the source would make the question of code escrow go away). We had a customer ask about source escrow recently so we started looking into it.

The various escrow services all seem to be focused on the idea that once in a blue moon you will have a software release, send them physical copies of the software, and they will provide announcements that you have done so to the customers that you have deemed as beneficiaries. All this for several hundred dollars per customer per year.

Even worse, it doesn't appear that those entry level prices include much in the way of validating the code. A recent CIO magazine article about source code escrow mentions an incident with Radisson Hotels where they ended up having to get code out of escrow, but couldn't use it, because it was missing objects, etc.

The article does have several comments that complain that the authors should have mentioned that best practice is to "audit" the code escrow. But, none of the commenters, nor the escrow sites, has any good details about this audit would be performed. I'm sure that people with regular code escrow experience know a few tips and tricks about this, but I don't see too many of them blogging :-)

In a perfect world what would happen is that when we signed up, they'd give us a Subversion post-commit hook that we could use to mirror our version control commits to the escrow service. Then we'd give them the configuration for our Continuous Integration server (we use and love Hudson), so that the escrow service could run the same builds that we do. Heck, they could even use Hudson's email notification services to send the build logs and unit test results to customers as proof that they are actually receiving code and it is at least in semi-decent shape.

For the customer that then wants to do even more auditing, the escrow service could make the generated binaries available for the customer to test with. If those still work properly, then that's a pretty good sign that the escrow service is working properly.

The crazy thing is that this wouldn't be all that hard to do, or even that expensive. Maybe you wouldn't want to actually send all of your commits over, but it would be easy enough to limit what gets sent to release branches and not current development work.

The biggest challenges would be things like customer A is only entitled to version 1 plus maintenance releases, while customer B gets everything, etc. But figuring out those details is the value add that an escort (whoops - escrow!) service can provide. Most of the rest of it would just a bunch of hosted servers and some fairly standard configuration.

So will someone go build that so that we can toss out the CD burner. Please.

Labels: 2008, Business, VersionControl

Steve Chan has a link to an iSeminar from Cliff Godwin with some real meat about details on how the technical upgrade to Fusion is planned to work.

The screenshot from the presentation on Steve's website nearly made me spit out my drink though. The actual title of the tool is called "Upgrade Assistant for Fusion". One of the bullet points says "Leveraging the best ideas from PeopleSoft Change Assistant".

Don't get me wrong; the PeopleSoft Upgrade Assistant (which became Change Assistant after it learned how to deal with maintenance as well as upgrades) is a whole heck of a lot better than some of the old manual processes in PeopleSoft upgrades, but most PeopleSoft customers aren't huge fans of Change Assistant. Change Assistant has been around for awhile now¹, so a lot of folks have forgotten how much of an improvement that it really was.

In fact, when we first started Grey Sparling, we considered doing product packaging as Change Assistant Change Packages (A "Change Package" is essentially just a .zip file that obeys Change Assistant's structural conventions about what files/directories are in it; similar to the Java .jar file format), but we got a lot of pushback. Customers told us that they didn't use Change Assistant for anything beyond just standard PeopleSoft maintenance, and therefore didn't have it up and running in demo and test environments, which is typically where people install our evaluation versions.

What Change Assistant Needs

What does Change Assistant really need, even before Fusion? Two things.

One is to beef up the logic for dealing with large volumes of patches and fixes. There's one bug in particular that rears it's head regularly where there will be pauses of several minutes between each file being copied. I've seen this in action at customer sites and it came up as a question during one of the OpenWorld sessions as well. It's not a slow file copy; each file gets copied quickly. It's more like some sort of "don't swamp the network" logic swung the pendulum too far on the conservative side. People really hate this.

The other is a bit more focus on using Change Assistant as part of the regular customization process. Application Designer actually has support for PeopleSoft customers to create their own Change Packages when doing custom development, but it's not well documented or supported. This forces customers to require other procedures in place for moving customizations around (since even to this day, there are almost zero PeopleSoft shops that don't have any customizations). Since customers end up dealing with this, learning (and understanding; see item 1) Change Assistant is viewed as an extra cost.

In keeping with that second item of better integration with customer development processes, we here at Grey Sparling will have some Change Assistant integration for our version control product. Since we're already dealing with all of the pieces of a Change Package anyways (App Designer projects, SQRs, Crystals, etc.), it makes sense to go ahead and add knowledge of what a Change Package is to the product so that you can version your PeopleSoft Change Packages just like anything else and have that more deeply integrated into your development processes.

1) The number one hit on Google for "Change Assistant" is a link to the original Change Assistant Flash demo from back in early 2004. If you watch the actual demo and pay close attention you can actually see one of the demo environments labeled APOGONOS.

Andrew "Pogs" Pogonoski was the original product manager for a bunch of the work that went on to actually have PeopleSoft be able to deal with all of the Customer Connection integration and hosting the Web Services that provide Change Assistant with it's data. All of the actual demo that you see in the movie is him working away.

It's safe to say that without Pogs' diligence at the large amount of cat herding involved that Change Assistant never would have gotten off of the ground.

Labels: 2008, Fusion, Sysadmin, Upgrade, VersionControl