Posted on

What can ITSM learn from Hurricane Irma

Article first published 21st September 2017

I recently had the opportunity to take a short break between contacts and booked a holiday in Cuba. Unknown to me this was about to coincided with Hurricane Irma (Sept 2017) moving from the Mid Atlantic across the range of islands. Out of every life experience I try to draw some learnings, especially in the ITSM space and this once in a lifetime event certainly taught me a lot. So, let’s look at what happened and draw the comparison.

We started our break in the Cayo Coco region, which quickly became known to be in the flight path. Once this was confirmed we were moved West, 9 hours across the island to an area with hotel capacity to take the 1200 people and outside of the known path. Once established there, the hurricane changed again and our hotel was put on hurricane watch and then full blown emergency procedures implemented as the eye was due to pass close by.

Whilst this is predominantly an ITSM reflection it would be wrong not to acknowledge both the Melia Hotel Managers at Cayo Coco and Valadero and our Thomas Cook holiday rep, people who demonstrated incredible professionalism in their operational ranks .

So, once it was evident that our evacuation location was now in the direct path, all residents were briefed and taken to a safe area under the hotel. This was a large area under the hotel which actually housed the kitchen prep and inbound delivery area. The Hotel Manager assumed the crisis manager role and introduced his deputy and the crisis operating model (in three languages) and then the carrier reps provided specific client updates through the duration.

Putting the crisis management element to one side, part of the focus of this article is on service. We were in the throes of a natural disaster with over 500 “paying” guests. It would have been easy to drop the level of service and yet throughout the incident food was continually provided, the toilets were cleaned and the holiday makers endured over 18 hours in basically an access corridor without any cross words and no alcohol! In fact, the ratio of staff to guests was around 1:5 (an observation I am planning to cover in a later article).

So, what compelled me to write this article? Well its quite simple, in ITSM we regularly talk about 3 things, service design, DR and Service / SLA’s but too regularly organisations do not take this seriously or do this well and I challenged myself to ask why? What is the under-riding factor?

If I reflect on the example of Irma it was clear on a number of things

1) There was a plan. This was not created a few days before. The level of service maintained and the level of execution could only be put down to strong planning and regular modelling / simulation

2) There was a clear mandate – protect life and continue to deliver outstanding service

3) The outcome (for point 2) was never in doubt. There was no hesitancy, management crisis or shortfall in delivery. Everything “just worked”

Now in reflection I would suspect two things ensured this happened.

Firstly, the event was life protecting dependent. Both the hotel chain and the holiday carrier had a duty of care. That was clear in the approach and a 99.95% outcome in this area would just not be acceptable. I would suggest that the service aspect was a secondary consideration but in our ITSM terms, all they decided was that the Vital Business Functions were set with a very high bar.

Secondly, they were used to the natural phenomenon. Whilst this was a Cat 4-5 Hurricane, lower categories are not uncommon, as are tropical storms, therefore at some point there would always be expectation to initiate the plan. I would suspect this was not a procedure given lip service in a management meeting or taken out of scope due to a cost or time over-run.

So why the reflection?

Each year, the UK media seems to report significant IT failures and I am sure on a global scale they occur just as regularly and yet in most cases the true root cause never gets reported. We, as ITSM professionals can speculate on it and I am sure our experiences of working in different organisations can cite system failures that should never have happened or should have been recovered much more quicker with a better level of service. In simple terms, feedback regularly tells us that we let our customers or users down.

Watching the recovery process during Hurricane Irma peaked my interest in a number of areas:

1) Do we approach any service with a mindset that our common “natural disasters” happen? In our world, this could be a cryptolocker type attack, a significant database corruption or a major datacentre / infrastructure outage. As a service function, we have a right to be the voice of doom and approach service with a mindset that our “Irma” is always out there

2) As part of service design do we challenge the architects to put the potential points of failure on the table and then have clear options to mitigate them. More than this, do we actually turn these scenarios into service metrics?

3) Once those are clearly identified and solutioned we then put our service hat on and take up our position of “is the contractual SLA good enough or as a collective is a higher-level outcome our goal”

4) Regardless of the answer to the above question, once that is agreed that becomes the recovery objective. In essence we reset the bar, if a 4hr recovery is our agreed outcome then failure to achieve this should be taken in the same mindset as the mandate demonstrated by the crisis team in Cuba (ie not an option).

Acceptance of point 4 above is not uncommon and I am sure that a lot of organisations go through the 4 steps above during major projects and implementations, the problem is they fail to turn them into an executable outcome that only has a guaranteed conclusion of success. By that I mean truly exploring the “what ifs”, getting the recovery process clearly documented, testing them and then throwing in a curve ball at the last minute! (on a complete side note, I recently watched the film Sully and it was interesting to see the “simulated outcome” turned around significantly once the “human factor” was added in, but that’s for another day).

Maybe it needs a change of perspective? Maybe if a life depended on it or if service was truly king then the focus on service design, recovery, continuity and ensuring that the customer continues to receive the same level of service in the face of adversity regardless of the cost would be more prevalent in the ITSM service cycle.

In closing, I suspect the real benefits of V3 (if you hang your hat on this model) as still not being realised and as an industry we sitting in a hybrid world of acknowledging V3 as a framework but working in a V2 comfortable manner, otherwise the elements of Service Strategy and Service Design in their new guise would be ensuring that services designed and implemented (certainly in the last 6 years since the 2011 refresh of V3) would be closer to the service excellence I recently experienced.

If you have enjoyed reading this article and would like to discuss in more detail, I would welcome your thoughts on this.

Posted on

Has the SLA had its day.

Article first published 16th July 2017

I was first introduced to IT Service Management back in 2002. The concept of the SLA and availability targets was fairly straightforward with the service components having a supplier given availability figure and the overall service target being a result of multiplying the service elements out. A recent assignment has significantly challenged my view of this. If you are interested in my viewpoint, please read on.

Now for me, this was probably a late to the game “eureka” moment but when you are elbows deep in the day to day delivery of service it is sometimes difficult to step back and see the changes around you. My assignment involved a request to create a new set of SLA’s as a new IT Director wanted to quickly understand the portfolio of services.

The scope was agreed as a full end to end study. Taking the commercial agreement to the end client, mapping the internal operational systems and identifying the service elements then carrying out the traditional mapping of support and support hours against the operational hours. All of that was straight forward but the first step of reviewing the client contracts raised an interest observation which to be honest when reviewed in the cold light of day was obvious and did pose to me the basic question of the validity of the system availability driven SLA.

What was the new variable that challenged the foundation of my ITSM compass? It was quite simple. The majority of the commercial contracts had limited reference to systems, system uptime or availability. Quite simply the majority of the contracts now referenced “outcomes”. Two examples are as follows:

  • All orders received into the suppliers system prior to 17:00 where flagged as a next day delivery to be processed in time for the final planned transport pick up in order to fulfill the next day delivery criteria
  • Order confirmation and order dispatch messages to be received back into the customers EDI gateway no longer than 15 minutes after the corresponding action is taken in the suppliers WMS system

So what has changed? The basic premise of the old availability approach to the SLA was purely about the fact that the system was “available”. I am sure when the concept was drawn up it was “good enough” to give both a level of re-assurance that IT was taking the internal customer seriously and could “nail its colours to the mast” but also give a point of reference to conduct a service review. But as technology has moved forward and if for example in the retail and logistics world, the service delivery to the end user has become close to real time (who would of thought at the change over to the millennium the likes of Amazon would soon be offering a service proposition whereby you pay a fixed fee and can order a wide range of product whereby if you order by 5pm can be delivered to you the next day at no cost), a basic availability target no longer is sufficient.

Why is that? Well quite simple a standard availability figure does not allow for the constraints of time bound activities. Taking the first example (above) of this we can clearly prove how this is no longer suitable as follows:

  1. An operation works Monday to Saturday 06:00 – 23:00 (but the final transport pick up is at 21:00). Therefore the service window is 60 x 15 x 6 = 5400 mins per week
  2. System availability target is calculated as 98.4% (measured from example server target = 99.5%, network target = 99.75%, application support target based on P1 (90 min fix across 24 x 7) = 99.1%)
  3. 1.6% allowed unplanned downtime against 5400 minutes gives 86 minutes
  4. With an order cut off of 17:00 and a last order pick up of 21:00, the 4 hour window to pick, pack and dispatch is now reduced to just over 2 1/2 hours. The risk and potential penalty is now moved from IT to the operation. Unless a sliding application fix SLA is provided that reduces the P1 fix time to 30 minutes during that 17:00 – 21:00,the availability driven SLA no longer favours the outcome driven contract clause

In a similar way, if we take the second contract criteria of messages being delivered back to the customer system within 15 minutes, the traditional availability target SLA which allows 84 minutes of unplanned downtime clearly does not support that requirement.

The challenge then comes that if you accept the observations above as a basic principle, what are the alternatives?

The obvious move is to realign the IT SLA’s to the business outcomes but in doing so a number of factors such as those below may need to be considered;

  • IT need to have a change of mindset to align them closer to the operational contract. In doing so their is an inevitable risk that differences in operation culture could create natural obstacles
  • In order to prevent future challenges, IT need to be engaged as early as possible in any client contract negotiations
  • Traditional supplier SLAs may need to be aligned to deliver the outcome based approach
  • Concept of fix time to P1’s has to be removed as the measure is no longer time bound but measured in the success of “outcomes”
  • Systems may need to be designed with a greater level of resilience / availability, considering true high availability during change activates to support business outcomes
  • Service processes such as Major Incident Management and service report will need to be aligned

Whilst the SLA may not have seen its day, certainly in order to keep up with the increasing demands, I would suggest that the traditional service measurements may need to be revisited and replaced with an outcome based expectation.

Posted on

Ten top tips for Major Incident Management

Article first published 12th June 2017

When I was looking to have the website redeveloped, it was a good opportunity to look at the old blog articles. A great deal of these were written with a different focus and technologies had changed significantly, but every now and then I came across one which I felt was still pertinent. This article contributed to by a good friend and colleague of mine still (in my opinion) hits all of the spots.

Our top 10 tips:
1. Information is key – The last thing you want to be doing in the middle of a major system failure is scrabbling around for information. Worse still is sat that knowing that all the information you need is saved on the hard drive of the system that has just failed. To manage a major system failure you usually need the following things: o Key contact names and number o Copies of any support agreements o Technical information and if possible, system diagrams These items alone will not guarantee you a fix, but they do provide you with all of the raw information you need to get the right people together and to make the right decisions. Make sure all of this information is held in a paper format, ideally in a file that can be accessed easily. Assign an owner and if anything changes make sure it is updated.
2. Deadlines help focus the mind – One of the first questions we always ask at the start of a major system failure is “when do we need to get the service back?” Now there are always two times’, the time the customer would like it back (which is normally in the next hour!) and the time the customer actually needs it back. Working to an unrealistic and unnecessary deadline causes corners to be cut and poor decisions to be made. This can result in only a partial success which just comes back to catch you out later on. Deadlines should be driven around key business activities and if flexibility exists always take the latest window. Experience has told us that if a technical person tells you how long it will take to fix, always multiply it by 3.
3. When to have a plan b – Having clearly understood deadlines also allows you to re-plan when things are not going well. If you have been working on a strategy for 12 hours and you only have 4 hours left, it may be prudent to start considering other options. It is not normally efficient to approaching a problem from various angles. Firstly, resource and costs do not normally allow it; secondly, it pays to have all of your knowledge working in the same direction. BUT occasions do occur when no matter how long you have been working on it or what resource has been thrown at it, the break through has not been made. The hardest (and sometimes bravest) decision is to ask all of the support people to down tools for a short period of time and to start consider contingency plans or other avenues of investigation. Normally they will want to continue to explore their current theory and that’s fine, but at some point you have got to start putting other eggs into the basket and getting the framework of these into place.
4. If needed, allow one to direct whilst another one writes – I had only been in the job for 6 weeks, when I received a phone call from my boss telling me our main computer room was on fire. This resulted in multiple systems failing and having to juggle a lot of deadlines and support people. The approach was quite simple, he directed the operations and I recorded everything, no matter how trivial or insignificant. From the times that people arrived on site, to decision we made (and who signed them off) down to things we needed to check and key deadlines, it was my responsibility to capture everything and then remind my boss of any key points or key times. This is only really relevant for big system failures or when you lose a few systems at the same time, but by clearly define the roles of the “manager” and the “scribe/reminder” the problem can be managed much tighter with key issues not being missed. Also when you come to trace your steps later on (to prevent it happening again or to review what worked and what did not), you will be surprised the small details that are recorded.
5. Conference calls – Most system failures normally involve 4 groups of people:
a) Those controlling it
b) Those trying to fix it
c) Those who are directly affected
d) Those who are indirectly affected but just want to be involved
The most time consuming part of managing a system failure is normally around the areas gathering information, making decision and keeping people updated. Depending on the size of the groups involved, the most efficient way to do this is to utilise one of the many telephone conference call services advertised on the internet. With rates of around 5ppm billed directly to the caller bill, these can be a low cost method of updating large groups quickly as well as discussing possible options without people feeling left out. Be aware that these non face to face meetings need a strong person in the chair and the expectations of meeting need to be outlined at the start. Due to people’s attention span tending to drift quickly, the chair should take the opportunity to recap and clarify points on a regular basis.
6. Don’t change anything without … – Recording it and making sure, everyone who needs to agree to it. It is so easy in the heat of a major system failure to stumble across a possible fix and to make changes in quick succession with no real rationale of why you have done it. Worse still, if the changes do not work (and notice the emphasis on “changes” as it normally results in several), it is very difficult without referring back to records, to remember exactly what was changed and why. This is especially relevant if several different individuals have provided the support over a prolonged period (as at the point of regression, some of them may be catching up on their sleep). Agreement to change to important for two reasons; Firstly, most changes come with a risk. The risk that you may extend the outage or the risk that it may do more damage that the current situation you find yourself in are two which spring to mind very quickly. Therefore, it is essential that in controlling the system failure, you set up a small group of people who will validate any decisions and where necessary question the reason for them. These do not have to be people from the end user community, but it is benefitial to have someone who understands the business and can explain the impact if the change causes further issues. Secondly, some changes need a level of technical understanding. A change to “system x” to resolve a problem may seem straight forward but in doing so may cause a new problem for “system y”. It is essential that all changes are considered within the scope of all of your computer systems, not just the one that is broke.
7. Trade off full service for vital business functionality – The common approach when dealing with a system failure is normally to want to restore the full service but when deadlines are short or the availability of technical support is limited consideration should be given to restoring vital business functionality only. Vital business functionality as the name suggests, looks at restoring only the key aspects of a system that the business needs to continue its operations to maintain a state of profitability or customer satisfaction. It is not normal for companies to have this documented for each system (although that would be beneficial), but generally is discussed near the start of a system failure to establish requirements early on. This gives the technical support the option of restoring a partial service to keep the business going or to try to fix the full problem.
8. 18 hours is enough! – Whether it is the support staff or the person managing the system failure, productivity and accuracy start to get impaired at around 12 hours into the fault. Once you hit 18 hours, your effectiveness is significantly diminished. Strangely, this does not seem to be influenced by when you last woke up from a nights sleep, so whether the problem starts at 9AM or 9PM the 12 & 18 hour rule seems to be the same (the difference is in your personal recovery time after getting the next good nights sleep!) Whenever a system failure is drawn out and due to its impact on you business you are working into the late hours, you should always aim to have a handover point somewhere between the 12–18 hour mark. Don’t forget, that this handover should include a full briefing of all the decisions that have been made and any courses of action that are currently being considered along with a summary of key milestones that are due to appear.
9. Blame has no place in restoring the service – It is so easy in the heat of dealing with a system failure to start looking for someone to blame. This is normally linked with the question why? Why did it happen? Why did we do it at that time? Why did we listen to that advice? Why didn’t we take a backup? All of these whys are great questions and actually add to the prevention next time, but during the failure they can encourage negative feeling or create barriers. No matter how much you want to pin the blame on someone or something, it should always be avoided, your energy is best invested in finding out why the failure has happened and what you need to do to fix it.
10. Restoration is only half of the job – Once you have returned your service back to a useable state, it may be time to take a deep breath and pat both yourself and any people who have assisted you firmly on the back. But good problem management does not stop there. The next steps involve fully understanding and documenting both the reason for the failure and what actions were taken to restore the service back to normal. Following this, actions to prevent it happening again need to be identified and where costs allow, implemented.

Posted on

People and Change Management

Article first published 22nd May 2017

Process re-engineering is never an easy thing. Studies have been carried out which map peoples reaction when subjected to significant life changing events and this is no more apparent in the work place when reviewing some of the core ITSM processes. A key one I have observed having the most significant people impact is Change Management.

Since starting out as a contractor in 2014, I have had the opportunity to reshape the ITSM process on a few occasions. They resulted in a significant shift in the approach towards change management, the underpinning processes and use of the toolset.

Lets be honest, Change Management as a process is very well defined. The common text books have it documented to a good level of maturity and as a framework the key pillars such as the presence of the CAB, the easily recognised change types of normal, standard and emergency and a logical process flow allow a maturity review to be carried out quite easily.

But this article is not about that, its introduces the key observations which I have found to be common in a number of reviews and they are based around people not process.

I am going to start this short example list with “Lead Time” as from an emotional viewpoint it tends to be up there at the top. The Change Manager (in administering the process) tends to want the maximum length and it is not uncommon for this to be between 7 to 14 days. This allows a good review of the change record to ensure it has been written to a good standard, a period of time to allow the change to progress through a technical approval cycle (and where end user or client approval is needed, this also to be sought) then in the event of it needing a CAB, a number of days to prepare and execute prior to the required implementation date.

Unfortunately this extended lead time tends to push against the needs of the developers and project managers who as an average I have found seem to sit in the 3 – 4 days camp. In a large number of cases the requirement for a shortened lead time can be clearly tracked back to a level of poor planning but with project timeliness, the risk of escalation and “the change process” being cited as reason for late delivery, any transformation of the Change Management process should really approach any communication of pushing the lead time out with a clear communication plan being able to clearly state the reason why the lead time is as it is, the benefits and most importantly have a clear process for expedited changes which does not become the norm but gives a level of reassurance that where a shortened lead time is required, the process can accommodate it without adding risk into the deployment.

The second issue that has been common in the transformations has been to approve or not to approve before cab. Let me explain. With the general lifecycle for a normal change following some form a peer review and then a decision gate based on a criteria to create the requirement to be presented to a CAB or to immediately go to Change Manager for final approval, one debate I regularly find myself facilitating is that of pre-CAB approval. Now in this case there are generally two routes if the approver has a query and wants the matter discussed in detail at the CAB.

Option 1 is to refine the process to allow the change to progress to CAB “part approved”. In this model, if a change had for example 5 pre CAB approvers and only 3 approved it, there may be a pre-defined CAB date in the change record which automatically progresses the change into the CAB but provides clear visibility that the change record has not completed its approval cycle and a level of questioning would take place. The problem I find with this model is that the CAB can become log jammed and extended with part approved changes, especially if the discussion is quite in-depth. On a people note, it is also a convenient way to defer the decision making process and can become the norm making the Change Manager role much more complex.

The second option is to have a criteria that all changes must be approved by the peer review cycle before presentation at the CAB. This creates accountability on the approvers to really own the action of approving the change and ensuring that any issues they have are resolved prior to the CAB. In this scenario I regularly get challenges with “well if we have all approved it what is the purpose of the CAB?”. My reply to this is fairly standard in that I see the CAB as a collaborative review process which moves past what tends to be a silo’d approval process in the peer review stage to a more collective review of the change, its back out, the testing evidence etc. By forcing the individual peer reviewers to get down to the detail of their level of accountability prior to the CAB it also has had a tendency to drive up the quality of the change approval.

You will probably get a feel for the fact that the second approach is my preferred one although this tends to need more people input in achieving buy-in when compared to the first option.

Where the nature of the application and infrastructure drives a regular need to engage with the end users or clients to seek approval, the decision to include any back out time in or out of change window is an interesting debate. Take a simple change that will take 3 hours to execute excluding back out and then a 1 hour window to back out. The question is whether the last hour to back out should be included in the change window. If you do it appears that you are planning to fail and in some cases can extend the change window significantly and not give a true picture of the “happy path” service handover point. By excluding the back out, you give a more realistic point of service handover but if the change fails and needs to be rolled back you may breach the SLA.

The people element here for me, really does come down to the approach and mindset towards failure. I have found where the consensus falls in including the regression time, there can be an underlying culture that finds failed changes acceptable. By forcing the change to be approved without the regression time included (and the penalty being an imposed SLA breach), the senses are sharpened and accountability for the rigor around the change is elevated.

The approach in the three examples above ultimately come down to engagement. Any process review and certainly where Change Management is concerned can only succeed with a strong engagement plan that gives early visibility of the options available and creates an environment where the options can be debated to a satisfactory conclusion. The days of reviewing the process and then presenting a “fait accompli” are long gone but I would close by stating that this is not process design by committee. Ultimately someone has to take the lead and carry the accountability of the final process but by recognising the people triggers and considering item such as the three highlighted above, a smoother approach can be achieved.