PointCast: A Case Study in Incident Management

Project-orientation, scalability, and ease of use make Microsoft Visual SourceSafe and Elsinore Technologies' Visual Intercept a unified system for managing change in PointCast's software development.

April, 1998

Change is so rapid in the digital age that just keeping up with the news is a major chore. PointCast Incorporated–privately held and based in Sunnyvale, California–revolutionized the news business in 1996 by creating the first Internet-based news network that gathers exactly the information the viewer wants and beams it straight to the viewer's desktop–for free. The PointCast® Network redefined the Internet media paradigm while becoming a primary source of daily news for millions of viewers.

The Challenge

To support their unique broadcast media, PointCast engineers manage an enormous amount of software development. In addition to supporting desktop and web-based client software for the PointCast Network, the developers are also producing intranet and extranet broadcasting tools, content creation tools, and infrastructure for PointCast's distribution. With more than a million lines of code to manage, and 1000+ feature requests and bug reports to handle per month, PointCast needs to use development tools that leverage programmer productivity and the company's investment in Microsoft technology.

As the manager of release engineering for PointCast, Mark Andrews found himself on the front lines of coordinating these efforts – he had to meet deadlines without losing code or good ideas. How did he do it? With a winning combination of Microsoft's Visual SourceSafe™ and Elsinore Technologies' Visual Intercept™.

Why Visual SourceSafe?

Initially PointCast used primitive command-line tools for their version control. "It just wasn't enough," says Andrews. "It was fine for a single project, but once we were supporting several different versions and several different code bases, we needed something more flexible, something that would let us organize code better."

The power of project-orientation

Andrews and his team chose Visual SourceSafe because its project-orientation let them organize their different products, versions, and code modules in an intuitive, graphically displayed hierarchy. "We really didn't want to struggle with keeping our source code sorted out. We easily cut in half our time spent on source code management by going to Visual SourceSafe. We've also seen an average 25% increase in time savings with each new version of Visual SourceSafe."

PointCast's rapid development pace mandated the use of Visual SourceSafe's branching feature; programmers could take a given project and instantly recreate new sets of code for parallel development. "With every major milestone, we split off new branches so we could develop new features while still maintaining the original code base," said Andrews. Branches for beta releases, localized versions, and different platform support were all easily generated from the same project. "Instead of everyone using '#ifdefs' to work around each other in the same code base, we gave everybody their own branch to use. The level of sophistication we put in our software went up exponentially with Visual SourceSafe, because we could put ten or fifteen people to work on a given set of code instead of two or three."

Cloaking and archiving let PointCast developers set aside old versions and projects without losing access to the code. "Just because we had moved on to a new version didn't mean we were finished with the old code. We kept older code in a separate project, clearing the workspace for more current work," Andrews notes.

Ease of Use

Visual SourceSafe's ease of use also made it a natural choice for controlling documentation and web pages. "A viewer's manual changes and adapts as fast as the source code does," says Andrews. The PointCast technical writers worked in their own database while creating all the pieces of documentation, and occasionally passed assembled versions of the complete files over to the developers to keep in their own system. "It's great to keep all items related to a given release–source code, release notes, helpfiles–all in the same project," says Andrews. Similarly, PointCast's Web developers began using Visual SourceSafe for managing the development of the corporate website.

Scalability

As PointCast's development team grew, Visual SourceSafe kept pace. Even as PointCast's engineering division added multiple platforms, Visual Source allowed them to work together in the same version control system. Macintosh® and UNIX versions of Visual SourceSafe were available for projects where PointCast did cross-platform development, and the SourceSafe system placed no inherent limitations on the size of the database, its files, or the number of users it could support. For instance, PointCast's Visual SourceSafe database for Engineering was over 5 GB in size and held over 10,000 files in over 200 projects and subprojects. "SourceSafe worked just as well for a hundred developers as it did for ten," said Andrews.

Flexible security

PointCast developers were especially pleased with how Visual SourceSafe let them selectively control access to the source code, according to different circumstances. "Early in a project's development, we enable the multiple checkout feature, so everyone has easy access to the code. As we draw close to a milestone or a release, we restrict code to a single checkout, so the engineers can watch every change going into the final product. And when we finally have the product ready for release, we lock everybody out of the project so we can proceed with final builds and distribution–without worrying about additional changes creeping in."

Automating builds

In addition to using the GUI interface, Andrews often uses Visual SourceSafe's programmatic interfaces (OLE and command-line) to automate complex processes like managing builds. "Our batch-file process for creating builds from Visual SourceSafe is now a company-wide standard. We can automatically check out header files for a given project, rev the build numbers, do the build, check the files back in, and apply a label to the code for the new build.

"We've seen a 30% increase in the reproducibility of our builds by using Visual SourceSafe. And since the whole process is scripted, we have daily builds ready for testing in the morning, with little human intervention," says Andrews.

Incident Management: Knowing the "Why" as well as the "What"

Visual SourceSafe did a great job of controlling the changes to PointCast's source code, but an equally important issue remained–incident management, or what is more commonly know as "bug-tracking." PointCast developers knew what changes were made to the code, but they still needed to organize information about feature requests and bug reports – and to document why the changes were being made in the first place.

Like many companies, PointCast initially tracked bugs with its own home-made system. "I put together an MS Access™-based application," said Andrews. "We used it for about a year, and put over 3000 records in it. We got a lot of mileage out of it, but we knew from the beginning that we would need something more as we grew." As the PointCast development team grew from 15 developers in 1994 to over a hundred developers in 1997, scalability proved to be a critical requirement for their bug-tracking system. "We needed more flexibility in creating new projects, assigning different leads, and reporting on incidents based on priority," Andrews notes.

Andrews also realized that PointCast's bug-tracking needed to integrate tightly with its version control. "SourceSafe let us track every change to the code™there was no reason why we shouldn't also be able to clearly track the issues that motivated the changes."

As longtime Visual SourceSafe users (since 1994) PointCast was among the first companies to learn of Visual Intercept, a new bug-tracking system specifically designed for teams using Microsoft development tools–especially Visual SourceSafe. After thorough involvement in the product's beta cycle in 1997, PointCast standardized on Visual Intercept as its primary system for tracking and assigning engineering issues.

Catching all the issues

Visual Intercept uses the same project-oriented organization that made Visual SourceSafe so useful to PointCast. Andrews quickly created a project structure in Visual Intercept for categorizing bugs by program and component. This allowed PointCast developers to track incidents with the same sort of intuitive project hierarchy they had been using to track their code. Visual Intercept includes a Wizard program for importing Visual SourceSafe project hierarchies directly into Visual Intercept, but Andrews opted to create his project structure manually so he could tailor it to exactly what was required for PointCast's bug-tracking. Within two weeks of initial installation, nearly a hundred developers, QA engineers, and product managers were using Visual Intercept in their everyday work.

The most immediate benefit of using Visual Intercept was making it as easy as possible for developers and QA staff to enter incident data into a central database. "As soon as we know about something that's an engineering issue, it goes into Visual Intercept. We have strict policy on that. Our development moves so fast that we have to catch all the issues and address them in a timely fashion. Even brand-new features go into the system, as soon as they become buildable units of code," says Andrews. Visual Intercept's tab-based, multiple document interface was straightforward and easy to use, allowing users to record issues quickly without interfering with their work.

At the end of seven months, PointCast had logged over 8,000 incidents in their primary engineering Visual Intercept database. Despite the enormous load of data and number of users, Visual Intercept readily kept up with the company's needs. Because Visual Intercept is an ODBC application, data was kept on a single machine running Microsoft SQL Server™. "Visual Intercept is the most resource-efficient bug-tracking tool we've ever reviewed. The only limit to its speed is the hardware itself."

A part of the environment

PointCast now keeps its source code changes closely associated with incident reports. Andrews: "When we're getting close to a milestone, we require that every check-in of code to the Visual SourceSafe database has a specific incident record associated with it. " PointCast developers could even directly check files in and out of SourceSafe from within Visual Intercept, and make associations between code changes and the bugs they address. "The synergy between Visual SourceSafe and Visual Intercept greatly magnifies the value of both tools," said Andrews.

At the center of development

All that information going into Visual Intercept allowed PointCast engineers to stay focused on the most important development issues.

"We relied on Visual Intercept to track our progress as we developed PointCast 2.0," said Greg Hassett, PointCast co-founder. "It was at the center of our development process. We would start each day with a 'war team' meeting which focused on the reports and information that we pulled from Visual Intercept."

When engineers met with quality assurance staff and the product managers in their morning meetings, they reviewed the lists of outstanding issues efficiently. "Because we were all reading from the same page, it was easy to see what needed doing and discuss relative priorities. We could quickly figure out which items could go into the next release, and which could be pushed back," says Samir Shah, the manager of quality assurance at PointCast. Members of Shah's staff immediately update Visual Intercept with the results of these meetings, so that daily development assignments reflected the agreed-upon priorities.

Staying focused

For the developers at PointCast, Visual Intercept provided exact information on what issues they were assigned to resolve and the state of affairs for each assignment. "Using Visual Intercept has become second nature for our developers. Practically the first thing everybody does after checking email is log into Visual Intercept and see what assignments are pending," Andrews says. Visual Intercept's Query Builder and Filter features let the developers immediately get all the information they needed on their own assignments or a particular project without getting swamped with other data that doesn't relate to their work. Developers pull up the issues that concern them in seconds.

The ability to see all the issues facing a development team, prioritize them, make assignments, and keep everybody clearly focused on the issues of greatest importance provided the biggest payoff to PointCast's engineering efforts. "Visual Intercept is really a load-balancing tool – an efficient way to know what we're doing and what still needs to be done," said Andrews. "You simply have to know what you're doing as a group if you're going to get product to market on time. The process of logging and monitoring bugs slows down development minutely, but it's increased our productivity and workflow tremendously."

Monitoring Progress

Visual Intercept bug-tracking data also provided an excellent way to track the progress of development. In addition to using Visual Intercept's built-in reporting functions, PointCast managers used Seagate's Crystal Reports™ and Microsoft Excel™ graphing features to see trends in their development process: the absolute number of open bugs, the rate at which new bugs were being logged, and the rate at which bugs were being closed. The development teams set goals based on these numbers, striving to bring the bug count down to zero as they approached a milestone or release date. "That feedback is crucial to planning our development schedules–it's the primary yardstick for gauging the success of any given version," says Shah.

Evolving Visual SourceSafe / Visual Intercept solutions

The data that Visual SourceSafe and Visual Intercept generate has proven valuable to PointCast – so valuable that Mark Andrews is looking for more ways to integrate the information into the development process.

Andrews recently began beta testing Visual Intercept's Software Developer's Kit, a programmatic interface to Visual Intercept via ActiveX™ components. Andrews sees the potential in the SDK, combined with Visual SourceSafe's command-line interface, for some powerful reporting functions. "I envision creating a Web-based database using Microsoft Visual InterDev that is automatically updated with data from SourceSafe and Intercept every time we do a build. That way, users could quickly see the specification for any given build – what code versions went into the build? Which new feature requests were addressed in this build? That would save us a ton of time."

The Bottom Line

PointCast engineers manage an enormous amount of change–thousands of issues that need to be addressed by the entire development staff working under tight deadlines.

Together, Visual SourceSafe and Visual Intercept provide PointCast with a single, comprehensive system for tracking development. The system has unified the development process, allowing PointCast to set a single, company-wide standard for project progress and productivity. And the Visual SourceSafe/Visual Intercept system's project-orientation, scalability and ease of use promotes wide-spread use throughout the company – by developers, QA, and management – no matter how large PointCast grows. Looking to the future, the open APIs of both systems leaves the door wide open to even greater synergy between the two tools.

"For an enterprise of this size and sophistication, we have to have a standard, reliable, scalable process for managing development," said Andrews. "Visual Intercept and Visual SourceSafe do that and more. Together they have a greater impact on our overall development productivity than any other tools we use."