Source Configuration Management

For over 14 years, I've been a ClearCase administrator, and ClearCase has been very good to me. However, lately I've been telling all of my friends that ClearCase is quickly becoming a dead end version control system, and that they should start learning Perforce. I believe that Perforce will become the most important version control system, and it is all thanks to Subversion. To understand why ClearCase is in trouble, you have to understand Cado Computers.

Cado Systems built business computers way back in the late 1970s and early 1980s. The basic Cado 20/24 system was based upon the eight bit Intel 8085A and came with 48K (that's Kilobytes, not Megabytes) of memory. Despite this, Cado Computers could handle up to four users simultaneously. While similar computers like the IBM Datamaster were over $100,000, a typical Cado system could be had for around $20,000. That was an amazing price breakthrough. A typical small business could finally afford a real computer! They could easily save the cost of a Cado Computer in just a single year by having the computer handle much of the accounting and ledger keeping.

A Cado Computer can track invoices, and help keep money flowing into your organization. It could print your invoices, checks, and bills automatically. The closest competitors couldn't produce a computer with this feature set for five times the price! So, how was Cado able to produce a computer that could do so much for so little money? Simple, Cado realized that hardware was much more expensive than software. By optimizing the hardware, Cado pushed everything off to the people who handled the software. For example, Cado had no file system. If the developer wasn't careful, they could easily place two files over the same area of the disk, or put a program on top of a file. The developer actually wrote down on a piece of paper where each file and program was stored on the disk.

Cado saved hardware costs by allocating only 2K for each user for program space. In order to fit programs into that tight amount of memory, developers were given few resources in order to program. Developers only had 26 six byte numeric registers, nine alpha registers that ranged in size from 20 bytes to 40 bytes, and five 255 byte long buffers for data storage. All these resource were shared by the developer and the OS, so the OS might decide to use a particular buffer which meant that any data the developer saved there would be overwritten.

Sure development was slow and difficult, but as long as hardware costs were high, it made sense to push these tasks onto the developer. In fact, the whole architecture of a Cado Computer was built upon the premise the Hardware was always going to be more expensive than software. You could say that Cado Computers bet the company on this fact.

By the mid 1980s, that assumption broke down, and Cado quickly saw its sales plunge. PCs were dirt cheap and plentiful. Software tools like Dbase made developing complex business software much easier. When PCs prices really dropped, and millions of PCs were out on the market, PC accounting software could be written that was more flexible and more powerful than could fit on the anemic Cado hardware. The cost of developing that software could be spread over millions of PCs. By the late 1980s, you could by a PC based accounting system for under $10,000 that could outperform anything found on a $20,000 Cado. Cado simply disappeared in a wave of sales and mergers.

ClearCase got its start almost 20 years ago by a little company called Atria. Most version control systems are fairly similar: The user creates a working directory on their local computer, and then retrieves the version of the software they want from the version control system. The user has to keep retrieving the software, and has to make sure they have enough disk space to get all of the files they need. Compiling the files into a software package was a long process since most computers had slow processors.

ClearCase changed all of that. Unlike most version control systems, ClearCase used views that could be stored on a large and very fast server instead of a slow and small desktop system. To a typical ClearCase developer, once you set up your view, it appeared that you directly accessed the software from the source archive. The slow process of copying all of the files to your local workspace was no more. Better yet, you really didn't need a local workspace since your view and the smoke and mirrors of the VOB handled what versions of the software you would see.

And, ClearCase also had another magic trick. Why bother compiling code if someone else already compiled it for you? What if the version control system looked at what you wanted to build, said to itself, "Hey Ralph has already built this piece of code!, why not simply give you what Ralph built instead of you wasting your time compiling it yourself?". This was called winking in and was one of ClearCase's most highly touted features.

In the past 20 years, things have really changed. That slow and small desktop system contains a really fast and powerful processor (maybe even two), and now you've got at least 40 or more gigabytes of space sitting on your desktop. Meanwhile, networks have become more ubiquitous and more crowded. Now, ClearCase isn't the only thing that wants to use your network, and the heavy traffic used by ClearCase bogs everything down.

To handle the change in environments, ClearCase introduced what are known as static views. Static views download all needed files to your local system where you can do your development. Of course, this simply makes ClearCase just like any other version control system, except that ClearCase costs more, takes more resources, and is slower. Besides using snapshot views means you can no longer do winking in already built objects.

Except that ClearCase's winkin feature isn't as important as it once was. You see, that powerful piece of iron on your desktop compiles code in no time flat. By the time ClearCase searches its database, finds a view that has the same code you're compiling, and then transfers that to your view, you've already built that code. What takes long now is linking all those little pieces of code together, and that takes quite a while thanks to the much more complex structure of the software and underlying API. And, linking over a network is much slower than linking on a local machine.

So, you see that like Cado, ClearCase made basic assumptions about its environment that are no longer true. However, Rational and later IBM have found a way to keep ClearCase sales up. Developers may hate ClearCase, but IT departments love it.

In a large corporation IT departments want to control development, and ClearCase's centralized structure makes it easy to do. ClearCase is very complex to setup and use and local expertise is expensive. However, if you have a centralized IT organization, that cost of expertise can be shared among all development organizations. Rational Software (and later IBM when it bought Rational) understood this, and was able to encourage ClearCase sales by making ClearCase part of a larger IT strategy. Fortunately for Perforce, Subversion was released.

Subversion was released about three years ago as an open source version control system. It was pretty much everything ClearCase was not. Where ClearCase was extremely expensive, Subversion was free. And, where ClearCase was hard to setup and understand, Subversion was easy to use. Although Subversion was not as feature rich than ClearCase, it was much easier to use, and being free meant that anyone could down load it and use it. And, that's just what happened.

As I said, developers hated ClearCase as much as IT loved it. And, what developers really hated was the way IT could control the use and workflow of ClearCase. Want to start a new project? You'd had to contact IT. Add a new user to your development group? Contact IT. Change something around in your development? Create a new branch? Add new files? IT needed to approve it. And, IT's approval could take days. However, what if the development organization could bypass IT by downloading their own version control system. A system that is freely available, and easy to setup and install? Thousands of development departments broke their corporate policy, bypassed IT, and setup their own Subversion development environment.

When IT found out, their typical reaction was try to stomp it out. Unfortunately, you couldn't. How can you prevent someone from downloading and using a freely available piece of software? And, when push came to shove, local managers would side with their development departments. After all, their department needed the software, and bypassing IT allowed more flexible development and less downtime.

In the last year or two, IT is recognizing that you cannot force developers to use ClearCase anymore. However, there are legitimate concerns with local development shops using Subversion. First of all, most of these Subversion archives are sitting on people's desktop machines and not getting backed up. Not something you want for important commercial assets. Second of all, Subversion really was missing some important features. For example, Subversion does not really have built in account protection. It depends upon the operating system or Apache server to provide it. Subversion also didn't allow users to delete individual versions of a file (or entire versions of a file). Normally, this was a good thing, but what if a user checked something into an archive that shouldn't be there such as proprietary information?

There were also problems scaling Subversion, and with Subversion's ability to handle more complex development strategies. Plus, who is IT going to call if they have a problem with Subversion? This is where Perforce comes into play. Perforce is fairly straight forward to use and liked by most developers. Perforce is also more feature rich than Subversion. Plus, unlike Subversion, there's a company you can call for support. Although Perforce is more expensive than Subversion, Perforce is much, much cheaper than ClearCase.

The results is what I call the great compromise. IT gives up their ability to control every single aspect of CM. In return, local developers allow IT to take care of the iron, training, and other tasks that local development shops never liked in the first place. Developers have a fast and efficient system for development, and IT has secured the corporate software assets from disasters. All in all, not really a bad compromise.

Ever since the first version control system, SCCS, came out from AT&T's unix distribution over 30 years ago, version control systems allowed one to branch their code. Merging, however, was a completely different story.

The biggest problem with merging is that it can be quite difficult. In order to merge two different files, you not only need the two versions being merged, but you also have to identify the base to use for the merge. This base selection process is very difficult because the results of the merge depends heavily upon the selection of the base. Plus, any merging algorithm has to take into account the chunk size that will be used for the merge process, whether or not the merging algorithm will recognize line movements, white space issues, and even the syntax of the code.

Whitespace means things like tabs, spaces, and line endings. Windows and Unix systems use two different line ending characters, so if one version of a program was done on Windows, and another on Unix, each line could be different even though not a single line was changed in the code. Another problem is that Python scripts and Makefiles are whitespace specific file formats, but most programming languages like C++, Java, and Perl are not. Most merging algorithms get around this issue by allowing you to decide whether or not to ignore whitespace.

Then, there is the chunk size of the code to consider. What if only a single character on the line changes? Do you consider that as a single change, or do you do the whole line. How about non-line oriented languages like HTML or XML. Should your merging algorithm recognize various language syntaxes? Most merging algorithms simply look for line changes since that will work with almost all files. However, ClearCase does not only do regular text files , but also does XML diffs too.

So, most version control systems let you branch, but avoid merging. Enter ClearCase to change all of this. When Atria created ClearCase, they spent a lot of time and effort making sure merging two versions of a file is a smooth and mostly automatic operation. It is such a rare feat, that it takes developers quite a long time to get use to the idea of allowing a program to automatically handle their merging. ClearCase does such a great job, that branches proliferate in places that use ClearCase. In most places, each developer has their own branch off of the main line of code. They merge the main line of code to their branch, and then when they are ready for the world to see their efforts, merge this code back into the main line. A single development shop could have hundreds of active branches at once, and it isn't too unusually for a single developer to use multiple branches at once.

Perforce also worked hard at merging. You have to if you want your software to be considered a world class version control system. However, Perforce looked at branches in a somewhat different light than ClearCase. In Perforce's view, developers work directly on the main branch, and not on development branches as one wold do in ClearCase. Branching is mainly done to prevent code freezes.

A code freeze happens when you are almost ready to deliver a piece of software. You send the release candidate to your Quality Assurance (QA) organization. QA tests the release and sends you back that list of defects. Your developers fix the defects, then send a new release to QA for testing. Adding new features at this point may add new defects, so all new development is forbidden and only defects are fixed. Meanwhile, you are now paying your developers to twiddle their thumbs while they wait for QA to finish testing.

In order to more efficiently use your development workforce, you create a release branch. All new development continues on the main line while only defects are fixed on the release branch. Now, most of your developers can continue working on without worrying that they are possibly adding new defects to the release candidate. Defects that are found on the release branch can be fixed by only one or two members of your development team while the rest continue their work on the main line.

If a defect is discovered on the release branch, Perforce allows you to fix that defect on the release branch, and merge that change back to the main line development branch.

Although Perforce merging algorithm works great on release branches, it doesn't work very well when developers use branching like they do in ClearCase. Most developers have discovered that Perforce does not handle the back and forth merging required by the rebase/deliver method used in most ClearCase branches, and ask "Why doesn't Perforce just do what ClearCase does?".

At the Perforce European Users Conference, Laura Wingerd gave an excellent presentation about branching strategies and explained about the two different types of branching and why Perforce and ClearCase tend to handle branching differently.

ClearCase handles what she called Convergent Branching. That is, the differences between the two branches tend to converge over time. A developer creates a branch from the main line development, and uses it to do their development. Every once in a while, they rebase by merging any new changes on the main line branch to their development branch. After a while, they do one final rebase, and then deliver their changes by merging their development branch back to the main line branch. ClearCase is perfectly tuned for this rebase/deliver algorithm.

This type of branching is called convergent because, as they merge the code back and forth between the branches, the two branches will remain pretty much in sync. After all, the developer wants to start off with a copy of what the main line development looks like. And, when the developer delivers their code, they pretty much want the main line branch to match what is on their branch. Although the branches can differ, the differences will generally disappear over time.

Perforce, on the other hand, handles what Laura Wingerd calls divergent branching. That is, the differences between the two branches tend to diverge over time. In my above example, the code on the release branch is generally frozen while the main line keeps developing. Within six months, the differences between the main line and the release branch will be fairly large. In two years, they'll be even larger.

However, not all divergent branching involves release branches. For example, you might have a Unix product that you port over to Windows. This involves quite a few changes in the code because of the way the OSs handle internal calls, and probably differences in the API calls. Later on, you want features developed for one OS will be merged onto the other. However, the differences between the two branches will never be 100% reconciled and may grow during time.

Perforce merges were designed to handle divergent branches. ClearCase always picks for the base the version of the file that was the most recent common ancestor of the two branches being merged. Perforce, on the other hand, finds the ancestor with the most rich history in common between the two versions. Perforce also tracks not only that a merge was done, but whether it was a copy or a merge, and whether the merge or copy operation was done without modifications, or if the developer had to edit the results. And, Perforce tracks exactly which versions of a file were considered for a merge operation. ClearCase can only track that a merge took place between two versions of a file.

This results in ClearCase having difficulties with divergent merging and most developers simply perform divergent merges in ClearCase manually. However, for all of its smarts, Perforce has problems when it comes to convergent merges. Most of the time, Perforce simply picks the wrong base to use for convergent merges because it attempts to over analyzes the situation and doesn't realize that in convergent branches, most of the differences between the branches have already been accounted for in the last merge.

To get around its problems with convergent merges, Perforce recommends what they call a Merge Down/Copy Up strategy. That is, you merge changes from the parent branch to the child branch for rebase operations, but you copy the files from the child branch back to your parent branch for delivery. Although the rebase step in Perforce is fairly straight forward (you do the merge the standard way), the delivery operation becomes a complex seven step dance which involves running the merge operation twice.

Perforce recognizes this issue and promises that they will simplify the process of thecopy down operation, and maybe completely automating it in future versions. Meanwhile, Laura Wingerd's presentation has helped clarify the question on when it comes to merging, "why doesn't Perforce just do what ClearCase does?" in merge operations.

There are two schools of thought on this subject:

• Why save something when you can just rebuild it? According to this school of thought, you should never version the binary results of a build. It simply waste space.

  With a line oriented text file, that is pretty easy to do. Imagine you have a 100 line program, you modified three lines, deleted two lines, and added four new lines. If I had the original 100 line version, all I need to store are the instructions on how to get from the older version to the newer version - just nine changes. If I store two versions of a binary file, I must store a complete copy of each version. If I produce 500 megabytes of binaries with every build, and I am doing five builds per week, I am adding 2 1/2 gigabytes of storage per week or about 125 gigabytes of storage per year. I can quickly overwhelm any version control system if I don't have a policy that helps me reclaim obsolete binaries. This becomes another management headache.

  Plus saving build binaries can lead to bad build practices or covers up bad build practices. If I know I have to rebuild my compiled code each and every time, I make sure that my build practices allow me to do just that.

  The Subversion development team does not believe in storing products of the build process, and this shows up in the design philosophy behind Subversion: Subversion has no command or easy mechanism for removing versions of files.

  In fact, much of the open source community is against storing products of the build process which is why most open source software is distributed strictly as source.

• Why build something when you can save it under version control? I personally lean in this camp, and the reason maybe because I work mainly in non-open source environments with very large teams of developers and multiple products. Our developers may depend upon the pre-built libraries created by their fellow developers. As part of my build process, I compile these libraries and allow other developers to use them. Yes, these developers could build their own libraries, but why should I assume that the developer will select the right code and version of the files to build?

  By distributing the binaries of the libraries, I can ensure that each developer is using the same set of files. Another advantage is that storing the output of the build process means everyone knows where the official copy of the release is stored. Plus, I can use the power of my version control system to keep up with the binaries.

  This is one of the areas where ClearCase excels. In ClearCase, I can take a built binary, and ClearCase will give me the names of the files used in the build, but also the versions, build scripts, and environmental settings. But. that is only true if the file never leaves ClearCase's storage area.

  Another advantage is that I am pretty sure my version control area is being backed up. A storage area for the built files may not be backed up.

  The big disadvantage is that you have to keep cleaning out obsolete and unimportant versions of your builds. For example, you might not be interested in built files older than two weeks old as long as those files haven't been sent to clients or to QA. This means determining which versions you want to keep and which to throw out. Again, ClearCase makes this very easy. Under ClearCase, the rmver (Remove version) command won't by default delete a version of a file if it is labeled. If you're using ClearCase, all you have to do is delete the labels that are no longer important to you, and run the rmver command.

  Perforce allows you to remove obsolete versions of a file via the obsolete command. The problem is that Perforce will delete interesting and uninteresting versions of files. The best way to handle this is to use branches for environments. Any version that is QA'd should be moved onto the QA branch. Any built version that is ready for customer usage should be placed on the distribution branch. This way, you can remove old builds from the development versions without worrying that you might delete version that is sitting on 353 customer sites.

  Another more philosophical question is how can you send something out to production when a different version of the file is in Ohio. If you store your build output, and QA tests what is stored and likes it, you know exactly what was tested. If you have to rebuild your output for production, how can you be sure that is what QA really tested.

In an era of cheap disk space, storing the results of a build is not a terrible waste. Yes, you have to have one more management headache -- finding the obsolete revisions and removing them on a regular basis. But, it isn't that difficult to implement such a task.

Introduction

About half the places I worked tend to be financial institutions who do what is known as internal development. That is, instead of producing software for customers, they are producing software for their own use. However, this article can also be for any organization that runs their own website or uses scripts written in Perl, Visual Basic, Python, etc. for system administration.

The Power of Version Control Systems

Version Control Systems are great ways of tracking the history of files. You can easily see the changes that have taken place from one version to another. When combined with good defect tracking practices (or when tightly integrated with a defect tracking system which is used in a correct manner), you can even track why changes took place. Most version control systems can help you see who wrote what line and when (see Perforce's p4 annotate command or Subversion's svn blame command).

Yes, with a good version control system, you can track any change, and answer the questions of who made a change, what change was made, when the change was made, and why the change was made. Too bad this great tracking falls completely apart when you deploy the change. At that point, especially for non-compiled code like scripts and websites, it becomes extremely difficult to know what was changed and why on deployed systems. Fortunately, you can use your version control system to do automatic deployment for you. By using your version control system for deployment, you can extend the powerful tracking mechanisms offered by your version control systems to your deployment. This includes deployment to your QA systems, production systems, and to all your various environments.

Perforce

The examples given are done via Perforce, and Perforce is probably one of the best tools for this particular task. Perforce's sync mechanism is rapid and takes very little overhead, so it will not slow down your system. Perforec allows you to use all of the power of the version control system inside the production directory, so you could do diffs, examine the history of files, and answer any questions without first creating a separate workspace.

Perforce also allows for complex remapping of the source archive directory structure to the production directory structure. After all, there are a lot of times when the file layout in development might not match your file layout in production. Perforce even allows you to merge two different source directories into a single production directory. For example, I have a development directory with my webpage source code. I also have separate directories for files that differ between each environment. These can be configuration files, but might also include webpages that differ slightly between different environments.

Another advantage of Perforce is that all of your files are set automatically to read-only. This prevents the temptation of editing the files that are in production without going through version control. Plus, Perforce permissioning scheme will even prevent someone from marking the file as editable and therefore, writeable.

And, whether you use Unix or Windows, you can use the P4V Perforce Visual Client GUI.

Subversion and CVS

Both Subversion and CVS can be used for this task, but there are a few caviets. Subversion and CVS add tracking directories to their checkouts. (The CVS and .svn directories). This could be solved by doing an import instead of a checkout. Unfortuately, that means you lose much of the version control tracking power in these directories. You still have the automatic deployment, but you lose most of the other functionality that makes using automatic version control deployment so nice. Just configure your application to ignore these tracking directories.

Subversion and CVS also don't allow you to easily remap your source directory layout to your production layout. You can do it, but it might take multiple checkout commands. And, you do not have the ability to merge two separate source directories into a single production directory.

Another problem is that the standard checkout in Subversion and in CVS sets the files as read/write. This means someone could manually edit a file without going through the version control system. The best thing to do is to set the files to read only once the checkout is complete.

Again, nothing prevents you from using Subversion nor CVS. It'll take a bit more work, but it is possible, and if you're doing this on a Windows system, using CVS Tortoise or SVN Tortoise give you a nice powerful GUI front end.

ClearCase

ClearCase is the only version control system that actually has the built in ability to handle this type of work. ClearCase dynamic views can be treated as read-only NFS mounts. When the ClearCase view changes, the NFS mount changes. The synchronization between is complete, automatic, and instantanious. There are just a few problems.

First of all, ClearCase doesn't allow the remapping of the ClearCase directory layout. You can setup symbolic links inside the ClearCase view to replicate the production structure, but that can be tedious. Instead, you'll probably have to settle to have your development structure match your deployment structure.

Also ClearCase NFS mounts are truely read only. Your program can't create temporary files inside of the mount, so you may have to modify your program to get around this issue. And, although the NFS mount is live, it isn't a version control view. In order to answer questions about what is in the directory, you have to startup a separate ClearCase instance with the mapped view. Finally, this is an NFS mount which means that if you are using Windows, you have to get an NFS package like Hummingbird Maestro in order to get everything to work. And, forget about using the C:\Inetpub\wwwroot directory.

The Configuration

You should have a separate Perforce user for each instance and each environment. For example, you have a web page for your New York and London offices. You also have a QA group that tests this page. For this one web page, it is best to have three different users. One for QA production, one for New York production, and one for London production. These users will have read access only to the directories and branches that they have permission to see. They should not be able to do a p4 edit on the files in their directory. And, these users should not have access to any other files in Perforce. You don't want to open up any security holes.

In fact, you might even consider setting up a separate Perforce server for these users, and using remote depots to export the files you want these users to see. That will prevent the user from perusing changelists, user lists, and other information that you want to keep only in your development environment.

In your development environment, you want to setup separate branches for each environment, for example, I might have this directory setup for Project Foo's setup:

//Depot/Foo/MAIN/... #Main Development Environment
//Depot/Foo/QA/... #QA's Testing Environment
//Depot/Foo/NEW_YORK/... #New York's Production Environment
//Depot/Foo/LONDON/... #London's Production Environment

When I want to deploy something to QA, I can run this command:

$ p4 integrate Depot/Foo/MAIN/... Depot/Foo/QA/...
$ p4 resolve -at //...
$ p4 submit

The p4 integrate copies all the developer's files to the QA branch. The -at flag on the resolve command tells Perforce to accept whatever is on the source branch and not to do an actual merge. This is usually what you want. The p4 submit command checks in the changes onto the QA branch. Your QA website will automatically update itself with the latest changes.

Setting Up the Perforce Client

You want to setup a Perforce client in the root of the directory tree that you want to use for your production files. This client should be owned by the Perforce user who has access to this directory and branch. This does not necessarily have to be the same user as the owner of the directory where the files will actually be deployed. In fact, there is probably an advantage to not having these as the same user.

For example, I want to setup my webpage in the /usr/local/apache/htdocs directory. I need to put the files from Depot/Foo/QA/... into this directory, plus the special environment files located in Depot/Foo/Special/QA/... My Perforce Client would look something like this:

Client: qa-web-client
Root: /usr/local/apache/htdocs
Options: noallwrite clobber nocompress unlocked nomodtime rmdir
View:
Depot/Foo/QA/... qa-web-client/...
+Depot/Foo/Special/QA/... qa-web-client/...


Now, a simple p4 sync will automatically update my installation to the latest version on QA.

The Sync Mechanism

Now that I have my client setup, and my Perforce depot setup, I need a way of automatically sychronizing my Perforce depot. How does this work?

First problem is logging into Perforce. You could run the Perforce commands that do the sync to tell Perforce which client, user, server, and password to use. However, Perforce provides a simple mechanism for this: The P4CONFIG environment variable. Set this variable to .p4rc. If you have a Windows system, you can do this in your System Control Panel. In Unix, you can do this either in /etc/profile, or in the user's .profile file.

Now use this to set P4USER, P4CLIENT, and P4PORT. Whenever you're in this directory, yor Perforce user, client, and port will be automatically set. The only other question is how to send your password. You could set your password using the P4PASSWD in the P4CONFIG file, but it will be visible to everyone.

I recommend that you create a Perforce group and put all of your autosync users in this group. You won't use this group in your Perforce tables, but you could setup the Timeout value to zero. This means, once you're logged into Perforce, you will create a Perforce login ticket that will never expire. This ticket can be set, so it is only valid on a particular machine. This is the most secure way of setting up this user. You can use the P4TICKETS environment variable in your P4CONFIG file to point to this ticket. This ticket should be read only, and only visible to the user who will be running the p4 sync command on that machine.

Okay, client is setup, the Perforce ticket is set, you've got the Perforce depot set correctly and the permissioning is set. Now, what is the Perforce mechanism that automatically does my sync'ing?

1. Well, there isn't one built into Perforce itself. Fortunately, it isn't too difficult to create one. The easiest way is setting up a Unix cronjob or a Windows Schedule Task that runs every minute. This task will automatically change into the directory in question, have the P4CONFIG environment variable set, and do a p4 sync.

   It isn't the most elegent of systems, but it is fairly efficient. Perforce syncs have very low overhead, and that is especially true if there is nothing to sync. Although the synchronization isn't instantanious, it does happen pretty fast after the commit that you probably won't notice. Plus, if you want to only do changes after business hours, you can set your cronjob to only run after business hours.

2. A more elegent solution and works under Apache's http server is to user Perforce's Webkeeper. This Apache module will automatically synchronize your webpage whenever the directory in your view changes. In fact, it is a fairly simple mechanism to use even if you're not using a webpage. Just setup an Apache httpd process that doesn't serve any webpages. Then you can use this same mechanism for SQL and server maintenance scripts.
3. Another choice is to use a product like ICmanage. ICmanage is an entire Perforce server setup. It includes hardware, hot backups, fail over, and autosychronization software. It may be a bit of an overkill for simple auto synchronization, but if you really don't have a strong Perforce setup (Our Perforce server is Bob's desktop system. I think Bob backs it up to CDs every once in a while.), it might be worth considering.
4. Finally, there's the roll your own approach. I wrote a simple Perl test synchronization manager in a few hours. The heart of the system was a server that took two different clients. Client #1 was a Perforce trigger whenever someone does a submit. It simply sent a one line message that said "Hey! Someone submitted a change.". Client #2 were all the machines that needed to synchronize to the Perforce server. All this sync-server did was relay the message to the various clients that they should do a sync. After playing around with it for a few minutes, I've decided to go with the cronjob approach.

So, What Does This Buy You?

Imagine you have this entire automatic syncronization system setup. Next time a new release is ready, it will automatically be put on your server. No more worrying if the release was or was not installed, and whether it was installed correctly. You can also see who did the install by looking at the integration history in Perforce.

Imagine that the new release has a bug in it. You go to your web page directory and do a p4 diff. Now you see what files were changed from the last release. You can also see who authorized the change, and why this change took place. Once you track down the file with the defect, you can find out which developer changed it, and why.

Is this a quick fix? If it is, you can log into Perforce as yourself, make the fix, on the development branch, and integrate it on your environment branch. With in a minute, it is now on your system.

Can't be fixed, and want to revert to your previous release? Several things you can do. Change your Perforce client to use a new branch. Then integrate your previous release to this new branch. Once the defect is straighten out, you can switch back to your original environment branch. Meanwhile, other changes from QA can still be integrated into your new release branch.

Is your organization ISO 9000 certified? I bet it is, and I bet management is pretty darn proud of that fact. Every few months, your whole organization is thrown into a full panic because the "certification people" are mucking around making sure all of your I's are dotted and T's crossed. You've probably been to a couple of meetings and you're given a review of what to say to these "certification people". It's a great achievement and certainly you must have an excellent CM process. But, if that's true, why does our software stink?

Next Passover, go down the aisle with all the Passover products. Go ahead, I won't tell anyone. Find the white box of Yehuda branch Matzoh, and take a look at the side of the box. There on the side of the box is their ISO 9000 certification. That's right, your fancy shmancy up-to-date-and-modernly-with-it company is following the same certification process they do when they bake this cardboard-like substance. The only difference is that Yehuda successfully got their product to market and it does what it is suppose to do. Chances are you can't make the same claim.

My sons go to a private school, and every year, they take something called the TerraNova test. The school loves to brag about their students' performance. My kids always score in the top 5% of all of their subjects. Both the school and I as a parent should be proud. I hate the whole thing.

I hate it because I am paying the school for two weeks teaching the kids how to do well on this test. I am paying the school, so that their curriculum matches what is taught on this test. I am not against the test itself. Achievement tests can help the school understand whether or not they are teaching their students. It can help them evaluate their methods and their teachers. But, this is not what happens.

The purpose of the school is to teach the students a wide variety of subjects. The students should be trained to think and solve problems. They should learn to communicate and write. They need to learn to learn and to love learning. And, achievement tests like the TerraNova can do this. Unfortunately, instead of the TerraNova test helping the school become better, it has subverted the school, so the main purpose of the school is to get their students to do well on the TerraNova test.

There is nothing intrinsically wrong with following a process. After all, a lot of CM is establishing well known processes, so people know what they are suppose to be doing and how to communicate with each other. And, there is nothing wrong with adopting an outside process for your needs. It might take you years to figure out how to make sure that a change in your code gets tracked. Or, you may believe you have a repeatable process for doing development. Then, discover you don't when something goes wrong. A well tested and developed process like ISO and CMMI can really help there.

The problem is when our business becomes following the process and not producing software. Sure, gather requirements and track them. Take metrics. Use a $100,000 defect tracking to squash your bugs. But, keep your eyes on the prize: You're in the software business. If your software doesn't do what your users need, all the certifications in the world won't help.

The best CM process I ever was involved in was a company called ExecuFlow Systems. They produced medical software. We had no defect tracking system. We didn't even have a version control system. Instead, we logged all of our changes defects in a book. What made the process so good? The president of the company really believed in producing a quality product, and this trickled down into the management and the workers.

Nothing got out the door until it was thoroughly tested. We had user groups to find out what people liked and didn't like about our software. We tracked each change on paper, and anyone who messed up would hear from the President of the company himself. We did six to eight releases per year (and this was back at a time when a company may have made only one revision every two years).

We would have never been ISO 9000 certified. We probably didn't even make CMMI Level 1. But, we did produce excellent software. Because of that, we had an excellent reputation, and were one of the biggest medical billing systems in the Northeastern United States.

I left right before the company was sold to a software conglomerate. This company was going to blend our software into their development methodology. After all, they were ISO 9000 certified and we weren't. I have no idea what happened after that. The local office was closed, and all of the people, the developers, the trainers, the QA people, the engineers, and all of the people who dedicated their lives to producing the best software they could moved onto other companies.

The doctors who use to use our product moved on too. All that's left is a piece of paper gathering dust in some forgotten filing cabinet stating that we were once ISO 9000 certified.