Expect the unexpected - sound software change management practices can help minimize impact when the unforeseen occurs.

For quite some time, increasingly complex software systems on offshore drilling and production facilities have been neglected. Because it is invisible, software is often the forgotten stepchild, until an incident occurs that causes expensive downtime, or worse — a serious accident.

From drilling equipment to dynamic positioning systems to fire and gas detection and alarming, failing to plan for software change management puts these mission-critical systems at risk. All it takes is for one vendor to board a rig to update his company’s software, telling no one what he has done or where the two different versions of the software are stored. If he unknowingly introduces a bug and then leaves the rig, the result may be system shutdown until he can fly back to repair the problem.

It’s not only human error that can cause problems. When lightning destroyed a programmable logic controller (PLC) on a modern rig, the rig crew was able to replace the PLC with a spare, but there was no current backup of the software available, rendering the PLC useless until the correct software could be identified and dispatched to the rig. In such a case, if the vendor hasn’t kept proper records, it may be impossible to know what version of the software should be installed. You need to think twice about reinstalling from that CD found in a desk drawer — software in an unknown state can cause equipment to perform unexpectedly, often dangerously.

Version control

Drilling and production facilities often don’t have enough experience to know that there are comprehensive and systematic procedures to control software versioning and integration, allowing quick recovery from failures related to version control.

One management team, responsible for operating and maintaining multiple, highly automated drilling rigs, knew there had to be a better way, and decided to take a proactive approach. Closely integrated, custom-built and custom-configured software is required to operate the equipment on these drilling rigs. The unique nature of this type of software creates a need to track the latest versions and their configuration, as well as manage the changes required to maintain the integrity of the systems over time.

The management team knew that maintaining the software on the rigs was crucial for the platforms’ success and safety, so they asked the software control management experts at Athens Group to assess the situation. They knew that the standard practices that vendors used to control software versions varied widely. Some have good updating and tracking procedures in place but others make changes to software without recording their actions, and offer no record of the latest version installed. In order to circumvent the situation, the experts helped identify what could be key problems and ranked them in severity. They then created a list of detailed requirements to make sure the status of the software systems was continuously updated and made an action plan to directly address each identified risk. Together with the rig operations and maintenance teams, they developed new procedures for controlling software changes on the rigs, versioning and documenting those changes to avoid future ambiguity and maintaining software backups both on the rig and onshore.

Action plan

Initial efforts were focused on only one rig. The software experts worked with the personnel onshore and offshore to review the initial state of the software backups, hardware and procedures. They created and reviewed a prioritized plan of action to make sure that everyone was in agreement. Elements of the plan included:

  • Cataloging existing versions of installed software for all mission-critical drilling systems;
  • Specifying, purchasing and installing a dedicated server to host complete software backups and change management records on the rig;
  • Backing up all software from the drilling systems and documenting backup procedures in detail for future reference;
  • Developing and documenting a process for creation and ongoing update of an off-site, onshore backup, tied in to existing corporate disaster recovery procedures;
  • Developing and documenting a detailed process for controlling updates performed on the rig, outlining responsibilities for vendor technicians and rig operations and support personnel, including electronic technicians, the chief electrician, rig drilling superintendent, as well as onshore management and support personnel;
  • Coordinating the vendor tracking processes with the existing Management of Change procedures; and
  • Reviewing the plan with rig operations and support personnel to ensure that everyone was on board and understood their responsibilities.

Once the basic change management processes and procedures were developed and documented on the first platform, rolling them over to the next was straightforward. Differences in equipment required different detailed backup procedures but the basic processes for management and review of changes remained the same.

The result is a Software Change Management and Backup Procedure for multiple rigs that is based on common processes and software best practices. Upon completion of this project, the operations support staff were confident that they had resolved any software version ambiguity issues and had an action plan with detailed procedures applicable to multiple rigs. Rig personnel have been trained to ensure that they have the knowledge and ability to employ software versioning and control throughout the life of the rigs. This has significantly reduced the risk of costly rig downtime due to unintended or poorly understood software changes or missing backups.

Seven principles of software change management

  1. Require that the vendor technicians whose responsibilities include software follow a check-in process. Inspect their laptop computers for malware and viruses before allowing them to plug into control networks. Require them to notify you before installing any software upgrades, and to document all changes in system function that are expected as a result of those upgrades.
  2. Ensure that these software changes are properly reviewed and understood before being approved, and schedule upgrades to occur when the risk of operations impact is minimized.
  3. Use software update notifications. Develop a standard format for documenting software changes before they occur. Make the document available to all crews who may be affected, so that they can see what software changes are being made and when, and how they might affect the operations of the rig.
  4. Make sure vendors properly test all affected systems once the software modifications have been implemented, ensuring that the modifications did not have an adverse affect on system functions. Many of today’s control systems are highly integrated, so even if the upgraded system works correctly the changes could still introduce problems in interfaces to other systems. If deemed necessary, rig management can specify additional function tests be performed to ensure the integrity of systems before putting equipment back into operation.
  5. Require the vendor to demonstrate that the previous version of the software can be re-installed in case an upgrade fails integration tests or otherwise causes problems.
  6. Ensure that backup copies of the software are being created and stored on the rig and that they get refreshed when software is updated. This is useful for two reasons:
    1. When a vendor arrives on the rig to make software changes, it can be verified that he or she is working with the proper version of the software.
    2. It provides disaster recovery capability in the event of a hardware failure.
  7. Employ a backup application for Windows computers that works without requiring software to be installed on the vendor’s machine because installing unauthorized software on vendor-supplied machines can have serious, unpredictable effects.