| Prev | Next | ||
ClearCase provides a software build environment closely resembling that of the “traditional” make(1) program. make
was originally developed for UNIX systems, and has since been ported to
other operating systems. When building software systems with
ClearCase-controlled data, developers can use their hosts' native make programs, their organization's home-grown shell scripts, or third-party build utilities.
ClearCase includes its own build utility, clearmake, with many features not supported by other build programs:
ClearCase includes its own build utility, clearmake, with many features not supported by other build programs:
- build auditing, with automatic detection of source dependencies, including header file dependencies
- automatic creation of permanent “bill-of-materials” documentation of the build process and its results
- sophisticated build-avoidance algorithms, guaranteeing correct results when building in a parallel development environment
- sharing of binaries among users' views, saving both time and disk storage
- parallel and distributed building, applying the resources of
multiple processors and/or multiple hosts to builds of large software
systems
Users perform builds (along with all other ClearCase-related work) in views.
Typically, users work in separate, private views; sometimes, a group
shares a single view — for example, during a software integration
period.
As described in Chapter 4, “ClearCase Views,” each view provides a complete environment for building software, including:
As a build environment, each view is “semi-isolated” from other views. Building software in one view can never disturb the work in another view — even another build of the same program taking place at the same time. But a user can examine and benefit
from work previously performed in another view. A new build
automatically shares files created by past builds, when appropriate.
This saves the time and disk space involved in building new objects
that duplicate existing ones.
The user can (but need not) determine what other builds have taken place in a directory, across all views. ClearCase includes tools for listing and comparing past builds.
The key to this scheme is the fact that the development team's VOBs constitute a globally-accessible repository for files created by builds — the same role they play for the source files that go into builds. A sharable file produced by a software build is termed a derived object (DO). Associated with each derived object is a configuration record (CR), which clearmake can use during subsequent builds to decide whether or not the DO can be reused or shared.
Figure 5-1 illustrates the ClearCase software build scheme.
The next section describes the way in which ClearCase keeps track of the objects produced by software builds. Later, in “Build Avoidance”, we describe the mechanism that enables such objects to be shared among views.
As described in Chapter 4, “ClearCase Views,” each view provides a complete environment for building software, including:
- a particular configuration of source versions
- a private workspace in which the user(s) can modify source
files, and can use build tools to create object modules, executables,
and so on
The user can (but need not) determine what other builds have taken place in a directory, across all views. ClearCase includes tools for listing and comparing past builds.
The key to this scheme is the fact that the development team's VOBs constitute a globally-accessible repository for files created by builds — the same role they play for the source files that go into builds. A sharable file produced by a software build is termed a derived object (DO). Associated with each derived object is a configuration record (CR), which clearmake can use during subsequent builds to decide whether or not the DO can be reused or shared.
Figure 5-1 illustrates the ClearCase software build scheme.
The next section describes the way in which ClearCase keeps track of the objects produced by software builds. Later, in “Build Avoidance”, we describe the mechanism that enables such objects to be shared among views.
During build script execution, a host's MVFS (multiversion file system) automatically audits low-level system calls performed on ClearCase data: create, open, read, and so on. Calls involving these objects are monitored:
Note that some of these objects are actually located in VOB storage, and others in view storage. The view combines them into a virtual workspace, where they all appear to be located in VOB directories. They are termed MVFS files and MVFS directories, because they are accessed through the MVFS.
- versions of elements used as build input
- view-private files used as build input — for example, the checked-out version of a file element
- files created within VOB directories during the build
Since auditing of MVFS files is completely automatic, users need not
keep track of exactly which files are being used in builds — that's
ClearCase's job. For example, ClearCase determines exactly which
C-language source files referenced with #include directives
are used in a build. This eliminates the need to declare such files in
the makefile, and eliminates the need for dependency-detection tools,
such as makedepend.
A build can also involve files that are not accessed through VOB directories. Such non-MVFS files
are not audited automatically, but are tracked “on request” (by being
declared as dependencies in a makefile). This enables auditing of build
tools that are not stored as ClearCase elements (for example, a
C-language compiler), flag files in the user's home directory, and so
on. Tracking information on a non-MVFS file includes its time-modified
stamp, size, and checksum.
When it finishes executing a build script, ClearCase records the
results, including build audit information, in the form of derived
objects and configuration records.
For each new MVFS file (pathname within a VOB) created during a
build, ClearCase creates a corresponding object in the VOB database.
Together, the file and the database object constitute a derived object, or DO.
All the derived objects created by execution of a build script have
equal status, even though some of them might be explicit build targets,
and others might be created as “side effects” (for example, compiler
listing files). The term siblings
describes a group of DOs created by the same script. Siblings “travel
together” — whenever a DO becomes shared among views, all its siblings
become shared, too.
A derived object is automatically assigned a unique identifier, its DO-ID; this enables users to access any derived object with an extended pathname (Figure 5-2).
A derived object's DO-ID is analogous to a version's version-ID: it is unique, globally accessible, and view-independent; it distinguishes a particular object in the VOB database from others that share the same file system pathname.
As development progresses, many DOs will be created at the same pathname, by builds performed in different views, and by rebuilds within a given view. But no name collisions occur, because each derived object has its own DO-ID.
For the most part, users need not be concerned with DO-IDs; they can use a derived object's standard pathname when working in the view where it was built (or any other view that shares it):
Thisis another instance of ClearCase's transparency feature (see “Transparency and Its Alternatives”): a standard pathname accesses one of many different variants of a file system object. Note this distinction, however:
A derived object's DO-ID is analogous to a version's version-ID: it is unique, globally accessible, and view-independent; it distinguishes a particular object in the VOB database from others that share the same file system pathname.
As development progresses, many DOs will be created at the same pathname, by builds performed in different views, and by rebuilds within a given view. But no name collisions occur, because each derived object has its own DO-ID.
For the most part, users need not be concerned with DO-IDs; they can use a derived object's standard pathname when working in the view where it was built (or any other view that shares it):
- To standard software — for example, linkers and debuggers — the standard pathname of a derived object (util.o) references the data file.
- To ClearCase commands, the same standard pathname references the
VOB database object — for example, to access the contents of the
associated configuration record.
- A version of an element appears in a view by virtue of having been selected by a config spec rule.
- A particular derived object appears in a view as the result of a ClearCase build.
For each build script it executes, clearmake creates a configuration record, or CR,
that is permanently associated with the newly-built derived objects.
Each CR documents both the “bill of materials” and the “assembly
procedure” for a set of sibling DOs.
Users can examine CRs to see a wealth of information:
Users can examine CRs to see a wealth of information:
- Input files — Identifiers for objects used as build input: version-IDs (for versions of elements); DO-IDs
(for derived objects created as subtargets); timestamps (for
view-private files and for each non-MVFS file explicitly declared as a
dependency in the makefile)
- Output files — The pathname and DO-ID of each derived object created during the build (all the siblings)
- Build procedure — The text of the makefile's build script, including expansions of all make macros
- General data
— The user who performed the build; the host used to execute the build
script; the view in which the build took place; the date and time at
which build script execution started
ClearCase includes commands for working with derived objects and configuration records:
ClearCase can also display an entire CR hierarchy, which mirrors the multiple-level structure of a makefile-based
build (building a top-level target requires building of several
second-level targets, which requires building of some third-level
targets, and so on).
- DO listing — A user can list all the DOs created at a particular pathname, across all views:
% cleartool lsdo -long util.o 14-Mar-94.17:50:49 Allison K. Pak (akp.users@neptune) create derived object "util.o@@14-Mar.17:50.331" references: 1 => neptune:/usr/akp/tut/tut.vws 14-Mar-94.17:48:40 Allison K. Pak (akp.users@neptune) create derived object "util.o@@14-Mar.17:48.260" references: 1 => neptune:/usr/akp/tut/fix.vws 14-Mar-94.17:45:42 Allison K. Pak (akp.users@neptune) create derived object "util.o@@14-Mar.17:45.215" references: 1 => neptune:/usr/akp/tut/old.vws
- CR display — A user can display the contents of an individual CR:
% cleartool catcr util.o@@14-Mar.17:48.260 Target util.o built by akp.users Host "neptune" running HP-UX A.09.01 (9000/715) Reference Time 14-Mar-94.17:48:35, this audit started 14-Mar-94.17:48:37 View was neptune:/usr/akp/tut/fix.vws Initial working directory was /tmp/akp_neptune_hw/src ---------------------------- MVFS objects: ---------------------------- /tmp/akp_neptune_hw/src/hello.h@@/main/1 <20-May-93.14:46:00> /tmp/akp_neptune_hw/src/util.c@@/main/rel2_bugfix/1 <14-Mar-94.17:48:32> /tmp/akp_neptune_hw/src/util.o@@14-Mar.17:48.260 ---------------------------- Variables and Options: ---------------------------- MKTUT_CC=cc ---------------------------- Build Script: ---------------------------- c -c util.c ----------------------------
- CR comparison — ClearCase can use CRs to compare different builds of the same target, showing differences in how the objects were built:
% cleartool diffcr util.o@@14-Mar.17:48.260 util.o@@14-Mar.17:45.215 < Target util.o built by akp.users > Target util.o built by akp.users < Reference Time 14-Mar-94.17:48:35, this audit started 14-Mar-94.17:48:37 > Reference Time 14-Mar-94.17:45:33, this audit started 14-Mar-94.17:45:40 < View was neptune:/usr/akp/tut/fix.vws [uuid 7033a691.3f8011cd.ae42.08:00:09:41:e6:03] > View was neptune:/usr/akp/tut/old.vws [uuid d4a3a27f.3f7f11cd.adce.08:00:09:41:e6:03] ---------------------------- MVFS objects: ---------------------------- < /tmp/akp_neptune_hw/src/util.c@@/main/rel2_bugfix/1 <14-Mar-94.17:48:32> > /tmp/akp_neptune_hw/src/util.c@@/main/1 <20-May-93.17:05:00> ---------------------------- < /tmp/akp_neptune_hw/src/util.o@@14-Mar.17:48.260 > /tmp/akp_neptune_hw/src/util.o@@14-Mar.17:45.215 - Version annotation
— A user can annotate some or all the versions listed in a CR (or an
entire CR hierarchy) with a particular version label or attribute.
- View configuration — A user can configure a view to select exactly the versions that were used to build a particular DO, or set of DOs.
- Build management — ClearCase uses DOs and CRs to implement its build-avoidance and derived-object-sharing capabilities.
An essential aspect of makefile-based software building is avoiding unnecessary builds. clearmake's build-avoidance algorithm, based on the CRs produced by build audits, is more sophisticated than that of other make variants. The standard make
timestamp-based algorithm cannot guarantee correct results in a
parallel-development environment. An object module's being newer than a
particular version of its source file does not guarantee that it was
built using that version. In fact, reusing recently-built object
modules and executables is likely to be incorrect when a previous
release of an application is to be rebuilt from “old” sources.
clearmake's build-avoidance algorithm does guarantee correct results. Moreover, clearmake can avoid building a new derived object by reusing an existing one built in any view, not just the user's own view. Thus, a derived object can be shared by any number of views.
clearmake's build-avoidance algorithm does guarantee correct results. Moreover, clearmake can avoid building a new derived object by reusing an existing one built in any view, not just the user's own view. Thus, a derived object can be shared by any number of views.
When requested to update a build target, clearmake first determines whether an existing derived object in the current view qualifies for reuse. If not, clearmake evaluates other existing derived objects that were built at the same pathname.
The process of “qualifying” a candidate DO is termed configuration lookup. It involves matching the DO's configuration record against the user's current build configuration (Figure 5-3):
The search ends as soon as clearmake finds a DO
whose configuration exactly matches the user's current build
configuration. In general, a configuration lookup can have these
outcomes:
Reuse and wink-in take place only if clearmake
determines that a newly-built derived object would be identical to the
existing derived object. In general, wink-in takes place when two or
more views select some or all of the same versions of source elements.
For example, a user might create a “clone” view, with exactly the same
configuration as an existing view. Initially, the new view sees all the
sources, but contains no derived objects. Executing clearmake causes a wink-in of many derived objects from the existing view.
The process of “qualifying” a candidate DO is termed configuration lookup. It involves matching the DO's configuration record against the user's current build configuration (Figure 5-3):
- Files
— The version of an element listed in the CR must match the version
selected by the user's view; any view-private files or non-MVFS files
listed in the CR must also match.
- Build procedure — The build
script listed in the CR must match the script that will be executed if
the target is rebuilt. The scripts are compared with all make macros
expanded; thus, a match occurs only if the same build options apply
(for example, “compile for debugging”).
- Reuse — If the DO (and its siblings) already in the user's view match the build configuration, clearmake simply keeps them.
- Wink-in — If another, previously-built DO matches the build configuration, clearmake causes the DO and its siblings to appear in this view. This operation is termed wink-in.
- Rebuild — If configuration lookup fails to find any DO that matches the build configuration, clearmake executes the target's build script. This creates one or more new DOs, and a new CR.
Some organizations, or some individual users, may want to use ClearCase build auditing without using the clearmake program. Others may want to audit development activities that do not involve makefiles at all. These users can do their work in an audited shell, a standard shell with build auditing enabled.
For example, a technical writer might produce formatted manual page files by running a shell script that invokes nroff(1). When the script is executed in a shell created by clearaudit, ClearCase creates a single configuration record, recording all the source versions used. All MVFS files read during execution of the audited shell are listed as inputs to the “build”. All MVFS files created become derived objects, associated with the single configuration record.
For example, a technical writer might produce formatted manual page files by running a shell script that invokes nroff(1). When the script is executed in a shell created by clearaudit, ClearCase creates a single configuration record, recording all the source versions used. All MVFS files read during execution of the audited shell are listed as inputs to the “build”. All MVFS files created become derived objects, associated with the single configuration record.
This section discusses the physical storage of derived objects (DOs) and configuration records (CRs). As first discussed in “Derived Objects and Configuration Records”:
When a DO is first built, its data container and CR are placed in the private storage area of the user's view (Figure 5-4). When the DO becomes shared, through wink-in,
its data container and CR migrate to VOB storage. This implements a
“localize the cost” disk-space allocation scheme: the cost of a derived
object used by just one view is borne by that view; the cost of a
shared derived object is borne by the VOB, which is a shared resource.
- A derived object is a compound entity, consisting of both a data file (its data container) and a corresponding VOB database object (which has a unique DO-ID).
- Each derived object has an associated configuration record.
Many make utilities are available in the multiple-architecture, multiple-vendor world of open systems. ClearCase's clearmake utility shares features with many of them, and has some unique features of its own.
Users can adjust clearmake's level of compatibility with other make programs:
Users can adjust clearmake's level of compatibility with other make programs:
- Suppressing clearmake's special features
— Command options can selectively turn off such features as wink-in,
comparison of build scripts, comparison of automatically-detected
dependencies, and creation of DOs and CRs. Configuration lookup can be
turned off altogether, so that the standard timestamp-based algorithm
is used for build avoidance.
- Enabling features of other make programs — clearmake has several compatibility modes, which provide for partial emulations of popular make programs, such as Gnu Make and Sun Make.
Users can achieve absolute compatibility with other make programs by actually using those programs to perform builds. But in a build with a “standard” make,
there is no build auditing, no configuration lookup, and no sharing of
DOs. The MVFS files created by the build are simply view-private files,
not derived objects (unless the make program is executed in a clearaudit shell).
clearmake includes support for parallel building (concurrent execution of a set of build scripts), and for distributed
building (use of other hosts to execute build scripts). A command
option specifies the number of hosts to use; hostnames to use are read
from a build hosts file (Figure 5-5).
For example, a user might perform a three-way build, all of whose processes execute on a single multiprocessor compute server; an overnight build might be distributed across all the workstations in the local network.
For example, a user might perform a three-way build, all of whose processes execute on a single multiprocessor compute server; an overnight build might be distributed across all the workstations in the local network.
Before starting a parallel build, clearmake
considers each target, determining whether an existing DO can be
reused, or an actual build is required. Then, it organizes all the
actual build work into a hierarchical master schedule.
clearmake then dispatches the build scripts to some or all the hosts listed in the build hosts file. It uses a dynamic load-balancing algorithm that makes best use of the available hosts, preferring lightly-loaded hosts to ones that are relatively busy. Access to a “build server” host can be restricted to particular times, to particular users working on particular remote hosts, and so on.
Execution of build scripts is managed on each remote host by the ClearCase audited build executor (abe), which is invoked through standard remote-shell facilities.
clearmake then dispatches the build scripts to some or all the hosts listed in the build hosts file. It uses a dynamic load-balancing algorithm that makes best use of the available hosts, preferring lightly-loaded hosts to ones that are relatively busy. Access to a “build server” host can be restricted to particular times, to particular users working on particular remote hosts, and so on.
Execution of build scripts is managed on each remote host by the ClearCase audited build executor (abe), which is invoked through standard remote-shell facilities.
Many organizations develop multiple variants of their products,
targeted at different platforms. But ClearCase may not currently be
available for some of the platforms. How, then, can users build all the
required product variants?
One solution is cross-development — for example, perform a build on a SunOS host that produces executables for the unsupported host. ClearCase provides another solution — hosts on which ClearCase is not installed can still access ClearCase data, using standard network file-sharing services (such as NFS). This capability is termed non-ClearCase access.
Non-ClearCase access takes advantage of view transparency. Through automatic version-selection, a view makes any VOB appear to be a standard directory tree. Any such “VOB image” can be “exported” to a non-ClearCase host (Figure 5-6).
Users on the non-ClearCase host can perform builds within these “VOB image” directory trees, using native make utilities or other local build tools. The object modules and executables produced by such builds are stored in the view that instantiates the VOB image(s).
One solution is cross-development — for example, perform a build on a SunOS host that produces executables for the unsupported host. ClearCase provides another solution — hosts on which ClearCase is not installed can still access ClearCase data, using standard network file-sharing services (such as NFS). This capability is termed non-ClearCase access.
Non-ClearCase access takes advantage of view transparency. Through automatic version-selection, a view makes any VOB appear to be a standard directory tree. Any such “VOB image” can be “exported” to a non-ClearCase host (Figure 5-6).
Users on the non-ClearCase host can perform builds within these “VOB image” directory trees, using native make utilities or other local build tools. The object modules and executables produced by such builds are stored in the view that instantiates the VOB image(s).
Since ClearCase software is not running on a non-ClearCase host,
non-ClearCase access has some limitations. In many cases, there are
simple workarounds to the limitations.
- ClearCase build auditing, configuration lookup, and wink-in are not performed during builds on such hosts.
- Files produced by such builds are not derived objects, and have
no configuration records. (Users can employ remote-shell techniques to
overcome this limitation, so that the files built on a non-ClearCase
host do become derived objects.)
- Users cannot execute ClearCase development commands, such as checkout and checkin, on non-ClearCase hosts. (But they can probably remote-login to a ClearCase host for such purposes.)
- Users cannot change the VOB image, by reconfiguring the view,
from non-ClearCase hosts. (Again, a remote-login capability addresses
this issue.)
A typical software system is constructed hierarchically. For
example, executables are built only after programming libraries have
been built. With ClearCase, stable versions of libraries and other
subtargets can be stored as versions of elements. At build time, the
subtarget can be used essentially as a source file, instead of
(potentially) being rebuilt. Derived objects that have become versions
of elements are termed DO versions.
A development group can use DO versions to make its “product” (for example, a library) available to other groups. Typically, the group establishes a “release area” in a separate VOB. For example:
A development group can use DO versions to make its “product” (for example, a library) available to other groups. Typically, the group establishes a “release area” in a separate VOB. For example:
- A library is built by its development group in one location — perhaps /vobs/monet/lib/libmonet.a.
- The group periodically “releases” the library by creating a new version of a publicly-accessible element — perhaps /vobs/publib/libmonet.a.
It is easy to generalize the idea of maintaining a development release area to maintaining a product
release area. For example, a Release Engineering group might maintain
one or more “release tree” VOBs. The directory structure of the tree(s)
mirrors the hierarchy of files to be created on the release medium.
(Since a release tree involves directory elements, it is easy to change
its structure from release to release.) A release tree might be used to
organize “Release 2.4.3” as follows:
- When an executable or
other file is ready to be released, a release engineer checks it in as
a version of an element in the release tree.
- An appropriate version label (for example, REL2.4.3) is attached to that version, either manually by the engineer or automatically with a trigger.
- When all files to be shipped have been released in this way, a release engineer configures a view to select only versions with that version label. As seen through this view, the release tree contains exactly the set of files to be released.
- To cut a release tape, the engineer issues a command to copy the appropriately configured release tree.
| Prev | Table of Contents | Next |
| Chapter 4. ClearCase Views | Chapter 6. ClearCase Process Control |
No comments:
Post a Comment