Patch Management Testing Procedures
Computer Patch Management Procedures Approved by Sr. Staff 06/29/10 Patch updates to College workstations and servers for low, moderate, important, and critical.
By Jason Chan, stake
Introduction
A few years ago, patch management was barely a blip on the radar screens of most security
and IT personnel. Install and forget was a fairly common practice; once deployed,
many systems were infrequently or never updated. Obviously, for a number of reasons,
this approach is no longer an option. The rise of widespread worms and malicious code
targeting known vulnerabilities on unpatched systems, and the resultant downtime and
expense they bring, is probably the biggest reason so many organizations are focusing
on patch management. Along with these threats, increasing concern around governance
and regulatory compliance e.g. HIPAA, Sarbanes-Oxley has pushed enterprises to gain
better control and oversight of their information assets. Add in increasingly interconnected
partners and customers and the rise of broadband connections and remote workers, and
you have the perfect storm that has thrust patch management to the forefront of many
organizations list of security priorities.
It s obvious that patch management is a critical issue. What is also clear is the
main objective of a patch management program: to create a consistently configured
environment that is secure against known vulnerabilities in operating system and application
software. Unfortunately, as with many technology-based problems, good, practical solutions
aren t as apparent. Managing updates for all the applications and operating system
versions used in a small company is fairly complicated, and the situation only becomes
more complex when additional platforms, availability requirements, and remote offices
and workers are factored in.
Just as each organization has unique technology needs, successful patch management
programs will vary in design and implementation. However, there are some key issues
that should be addressed and included in all patch management efforts. This paper
provides a technology-neutral look at these basic requirements. The tips and suggestions
provided are rooted in best practice, so a given patch management program shouldn t
be considered a failure if all items haven t been accounted for. Instead, use this
overview as a means of assessing your current patch management efforts or as a framework
for designing a new program from the ground up.
Security and Patch Information Sources
A key component of patch management is the intake and vetting of information regarding
both security issues and patch release - you must know which security issues and software
updates are relevant to your environment. An organization needs a point person or
team that is responsible for keeping up to date on newly released patches and security
issues that affect the systems and applications deployed in its environment. This
team can also take the lead in alerting administrators and users of security issues
or updates to the applications and systems they support and use. A comprehensive and
accurate asset management system can help determine whether all existing systems are
accounted for when researching and processing information on patches and updates.
An organization should also have relationships with their key operating system, network
device, and application vendors that facilitate the timely release and distribution
of information on product security issues and patches. These relationships can range
from weekly or monthly calls with the account manager to simple subscriptions to the
vendor s security announcement list. In addition, public web sites and mailing lists
should be regularly monitored. Such information sources include Bugtraq, the various
SecurityFocus Focus lists, and patchmanagement.org.
Patch Prioritization and Scheduling
Several scheduling guidelines and plans should exist in a comprehensive patch management
program. First, a patch cycle must exist that guides the normal application of patches
and updates to systems. This cycle does not specifically target security or other
critical updates. Instead, this patch cycle is meant to facilitate the application
of standard patch releases and updates. This cycle can be time or event based; for
example, the schedule can mandate that system updates occur quarterly, or a cycle
may be driven by the release of service packs or maintenance releases. In either instance,
modifications and customizations can and should be made based on availability requirements,
system criticality, and available resources.
The second scheduling plan deals more with critical security and functionality patches
and updates. This plan helps the organization deal with the prioritization and scheduling
of updates that, by their nature, must be deployed in a more immediate fashion. A
number of factors are routinely considered when determining patch priority and scheduling
urgency. Vendor-reported criticality e.g. high, medium, low is a key input for calculating
a patch s significance and priority, as is the existence of a known exploit or other
malicious code that uses the vulnerability being patched as an attack vector. Other
factors that should be taken into account when scheduling and prioritizing patches
are system criticality e.g. the relative importance of the applications and data
the system supports to the overall business and system exposure e.g. DMZ systems
vs. internal file servers vs. client workstations.
Patch Testing
Ideally, the breadth and detail of an organization s patch testing will relate directly
to the criticality of systems and data handled and the complexity of the environment
e.g. number of supported platforms and applications, number of remote offices. The
patch testing process begins with the acquisition of the software updates and continues
through acceptance testing after production deployment. The first component of patch
testing will thus be the verification of the patch s source and integrity. This step
helps ensure that the update is valid and has not been maliciously or accidentally
altered. Digital signatures or some form of checksum or integrity verification should
be a component of patch validation. This signature should be regularly verified, especially
as an update is passed through an organization s technology operations e.g. on the
update server, in build images, in software repositories.
Once a patch has been determined valid, it is typically placed in a test environment.
While the perfect test environment will mirror production as closely as possible,
it is important to at least account for the majority of critical applications and
supported operating platforms in your patch testing infrastructure. Many organizations
will use a subset of production systems as an ad hoc test environment; department-level
servers and IT employee systems are typically used in these cases. Regardless of the
available test equipment and systems, exposing the update to as many variations of
production-like systems as possible will help ensure a smooth and predictable rollout.
The actual mechanics of testing a patch vary widely by organization. This testing
could be simply installing a patch and making sure the system reboots, or the test
procedure could involve the execution of a battery of detailed and elaborate test
scripts that validate continued system and application functionality. In the end,
a suitable approach toward detailed patch testing will be dictated by system criticality
and availability requirements, available resources, and patch severity.
The initial phases of production rollout can be considered an additional component
of the testing process. Rollouts are often done in tiers, with the initial tiers often
involving less critical systems. Based on the performance of these stages of the patch
deployment process, the entire environment will be updated, and the testing process
can be considered finished with the completion of final acceptance testing.
Change Management
Change management is vital to every stage of the patch management process. As with
all system modifications, patches and updates must be performed and tracked through
the change management system. It is highly unlikely that an enterprise-scale patch
management program can be successful without proper integration with the change management
system and organization.
Like any environmental changes, patch application plans submitted through change management
must have associated contingency and backout plans. What are the recovery plans if
something goes wrong during or as a result of the application of a patch or update.
Also, information on risk mitigation should be included in the change management solution.
For example, how are desktop patches going to be phased and scheduled to prevent mass
outages and support desk overload. Monitoring and acceptance plans should also be
included in the change management process. How will updates be certified as successful.
There should be specific milestones and acceptance criteria to guide the verification
of the patches success and to allow for the closure of the update in the change management
system e.g. no reported issues within a week of patch application.
Patch Installation and Deployment
The deployment phase of the patch management process tends to be where administrators
and engineers have the most experience. Installation and deployment is where the actual
work of applying patches and updates to production systems occurs. And, while this
stage is the most visible to the organization as a whole, the effort expended throughout
the entire patch management process is what dictates the overall success of a given
deployment and the patch management program in total.
The most important technical factor affecting patch deployment is likely the choice
of tools used. One key distinction between patch tools is a common system development
issue - to buy or to build. Historically, many organizations have created custom solutions
using scripting languages combined with available platform tools to distribute and
apply patches. As the industry has matured and the need for comprehensive and automated
updates has increased, many tools have become available to help manage the patch application
process. These tools are often classified as being either agent-based or agentless
systems, depending on whether they rely on software being installed on the target
systems that are to be patched. Additionally, many existing system management tools
have the capability to perform software and system updates. The correct choice of
patch management tools for any organization depends on a number of issues, including:
the number of platforms supported, the number of systems to be patched, existing expertise
and personnel involved, and the availability of existing system management tools.
While applying patches, and especially security updates, in a timely manner is critical,
these updates must be made in a controlled and predictable fashion. Without an organized
and controlled patch application process, system state will tend to drift rather quickly
from the norm and compliance with mandated patch and update levels will diminish.
In general, users and even administrators should not be permitted to apply patches
arbitrarily. While this should be addressed initially at a policy and procedure level
e.g. with acceptable use policies, change management, and established maintenance
windows, it may also be appropriate to apply additional technical controls to limit
when and by whom patches can be applied. The type of controls enforced will vary by
organization and requirement, but include items such as restricted user rights the
user does not have sufficient permissions to update the system and network-based
access controls the system cannot access the resources needed to perform an update,
for example Windows Update or RedHat Network. In smaller organizations, automated,
user-driven tools such as Windows Update may be acceptable. However, groups that use
these update methods will likely need to rely heavily on policy guidance and enforcement
along with regular assessment to ensure that organizational goals for patch and configuration
compliance are met.
Audit and Assessment
Regular audit and assessment helps gauge the success and extent of patch management
efforts. In this phase of the patch management program, you are essentially trying
to answer two questions:
What systems need to be patched for any given vulnerability or bug.
Are the systems that are supposed to be updated actually patched.
The audit and assessment component will help answer these questions, but there are
dependencies. Two critical success factors are accurate and effective asset and host
management. Often, these related goals of asset and host management are addressed
by a single product, such as with Tivoli, Unicenter, or SMS. The major requirement
for any asset management system is the ability to accurately track deployed hardware
and software throughout the enterprise, including remote users and office locations.
Ideally, host management software will allow the administrator to generate reports
e.g. all clients without a given hot fix, all versions of particular applications
that will be used to drive the effort toward consistent installation of patches and
updates across the organization.
System discovery and auditing are also components of the audit and assessment process.
While asset and host management systems can help you administer and report on known
systems, there are likely a number of systems that have been either unknowingly or
intentionally excluded from inventory databases and management infrastructures. System
discovery tools can help uncover these systems and assist in bringing them under the
umbrella of formal system management and patch compliance. Organizations typically
use either their own discovery and assessment mechanisms or one of the various managed
vulnerability assessment tools. Regardless of the tools used, the goal is to discover
unknown systems within your environment and assess their compliance with organization
update and configuration guidelines.
Consistency and Compliance
While the audit and assessment element of your patch management program will help
identify systems that are out of compliance with your organizational guidelines, additional
work is required to reduce non-compliance. Your audit and assessment efforts can be
considered after the fact evaluation of compliance, since the systems being evaluated
will typically be already deployed into production. To supplement post-implementation
assessment, controls should be in place to ensure that newly deployed and rebuilt
systems are up to spec with regard to patch levels.
System build tools and guidelines are the primary enforcement means of ensuring compliance
with patch requirements at installation time. As new patches are approved and deployed,
build images and scripts should be updated so that all newly built systems are appropriately
patched, and associated build documentation should be updated to reflect these changes.
In addition to updates to build tools and documentation, operational procedures must
exist to facilitate ongoing compliance of newly built systems. If an engineering team
typically builds servers e.g. with the base operating system and applications and
a separate operations team then assumes management of the system, a process must exist
to funnel operational changes back to the build and engineering stage of the system
lifecycle. These modifications are most ideally and suitably handled via an enterprise-wide
change management system. Any new patches and updates that are approved and installed
by operations should also be integrated by the engineering team into new builds, with
the change management system providing both an appropriate audit trail and suitable
procedural guidelines for this implementation.
Conclusion
While the issue of patch management has technology at its core, it s clear that focusing
only on technology to solve the problem is not the answer. Installing patch management
software or vulnerability assessment tools without supporting guidelines, requirements,
and oversight will be a wasted effort that will further complicate the situation.
Instead, solid patch management programs will team technological solutions with policy
and operationally-based components that work together to address each organization s
unique needs.
Learn the best Windows patch management procedure, including the ideal patch testing process and plan. We show you how to address the risk of security.
Patch Testing Ideally, the breadth While this should be addressed initially at a policy and procedure level Installing patch management software or.
Given the current state of security, patch management can easily become overwhelming, which is why it s a good idea to establish a patch management policy.
SANS Institute InfoSec Reading Room Patch management is a subset of the overall this situation it is vital that proper change management procedures be.
Patch management is an area of systems management that involves acquiring, testing, and installing appropriate patches to administered systems.
