Just call me a Mess Manager

Researching for the first chapter of Open University course #TU811, I came across the following quote by Russell Ackoff:

Manager’s are not confronted with problems that are independent of each other, but with dynamic situations that consist of complex systems of changing problems that interact with each other. I call such situations Messes. Problems are extracted from messes by analysis. Managers do not solve problems, they manage messes.

RUSSELL L. ACKOFF

 

While initially this made me chuckle – a mess is something I normally correspond with my son, a pot of Nutella, and limited adult supervision – I took a bit of time to think this through and its led me to the conclusion that maybe my job description should be changed to Environments Mess Manager

There are (primarily) two considerations when considering the complexity of an issue: number of variables, and number of perspectives.

Thus the levels can be mapped out on a table:

Single Perspective Multiple Perspectives
Few Variables A Difficulty A Mess
Many Variables  A Complicated Difficulty A Complex Mess

A mind-map may be a easier way to visualise the variables and perspectives to consider when deciding whether something is a Mess – here is one I knocked up in about 10 minutes using Mind42 (note that the term ‘Wicked’ Problem is another term that could be used in lieu of Mess. This term was coined by Rittel and Webber in 1973.)

The term ‘Complex Mess‘ could easily be applied in this context to such global issues as Climate Change, Deforestation, Terrorism or War. There are multiple perspectives and many variables which have coalesced to create a tangle of problems. Pull one thread and other knots appear. There is no single solution that will satisfy everyone – and indeed any solution that is taken could make the situation worse later as the variables are dynamic.

As I mapped this out though, I did find the the term ‘Mess’ is something that I could easily label to scenarios that I have seen in work – and to a lesser degree even within my own extended family at times!

As a Environments Manager, I have dealt with fragile legacy infrastructure (situation), teams working in silo’s (people), projects which don’t always seem to know what they are doing (thinking), companies resistant to change or with very set cultural norms (people), projects which span multiple departments or companies (situation, people and thinking)a mish-mish of methodologies (thinking) , lack of funding (situation)… the list can go on and on. The more of these type of variables I see in a project, the more I get that sinking feeling of unease that ‘something is probably going to go wrong’ and the more I tend to find I start unconsciously contingency planning.

Obviously variables and perspectives are not going to be the only factors at play in a mess; change and uncertainty are also big ticket items – but I would argue that these are categories of variables – albeit very big ones with giant flashing warning lights on them.

A question to ponder – are messes avoidable?

Take-away thoughts for DevOps / Agile practices

I attended the Computing DevOps Summit last Wednesday in London. It was a rather polished affair at the Grosvenor Square Marriott in Mayfair; well attended, well presented and with a broad lineup of Keynotes all roundly advocating DevOps.

It was a valuable day overall, but one speaker in particular resonated with me, as his choice of topic – ‘Unicorns and Elephants’ – is something quite familiar. The speaker, Rick Allan, Head of Delivery Capability at Zurich Insurance, spent his time waxing lyrical on the myriad challenges encountered by complex, multi-faceted organisations that have gone through significant organic growth. and the unexpected barriers this can create when adopting agile practices.

This is a scenario that I have seen in several organisations over the years – where do you even start when you have a estate that is riddled with legacy infrastructure and applications that have, over time, been ‘bolted’ onto each other, integrated with third parties and heavily customised. Automation in this landscape is a veritable minefield of problems.

My main take-aways from the event were:

  • Know what you actually have got, what it does, and what it communicates with.
  • Agree a vision with the business – implementing the changes needed to achieve that vision could have some serious cost and usability implications and so their buy-in and support is critical.
  • Pick off simpler applications first. Chipping small, tactical chunks from the monolith is going to be easier at first than trying to brute force it. Prove your method works.
  • Move away from project and start thinking by product and value stream (again – something the business really needs to get on-board with to enable this to work as this has budgeting implications.) This helps to break down velocity-sapping silos and move away from a transaction based culture.
  • Stand firm in the face of large, expensive programmes of change – they need to follow the rules too.
  • Implement a default tool-bench – a set of approved tools that are authorised and licensed for use. Migrate services over to them and shut off unauthorised tools behind you. Make sure your developers and engineers actually know about them, know what they are for and how to use them.
  • Use metrics to understand and prove where bottlenecks exist. This information can then be leveraged to determine a strategy to overcome the problem. Whatever the final option, the business case will need to be proven, and those metrics can help to establish that mandate.

In order to enable true continuous delivery, the ultimate goal needs to be consistent, converged and scalable architectures, ideally managed by smart software – and that is a long, and probably quite expensive, path for organisations with significant amounts of legacy applications and architecture. This type of change requires buy-in from the very top, and the investment, time and willpower to carry it through. Good intentions can only go so far.

What does Environments Management and a spiderweb have in common?

There is a look that you learn to identify in the early stages of taking on an Environments Management role. It’s somewhere between bashful, hopeful and slightly desperate. Occasionally it is preceded by a slightly nervous cough. And without fail it means a project has underestimated their requirements and they need you to magic something up… ideally within the next hour. Tomorrow will do.

One of the biggest challenges facing Environment Managers today is the speed that project environment requirements change. As projects become more agile, so the assumption is, should the environments that support them. However, no cash-conscious IT department will run extra environments ‘just in case’ – it doesn’t matter whether the applications are based on-premise, in the cloud or a hybrid of the two – this is just burning money. Also not every type of environment can be spun up at the drop of a hat – those with multiple integrations and specific connectivity requirements will take time if they are not part of a standard automated rig. At any given time an Environment Manager has to run the leanest ship possible, but remain flexible enough to deal with a certain ‘unspecified amount of changing requirements.’

So what is the best way I have found to handle this?

I will use the analogy of a  spiders web. In order to use the web to best effect the spider has to sit in a place where it can feel any single string vibrating at any time. The same goes for an Environment Manager. Except in our case the strings are radiating lines of communication.

Communication is probably the single biggest asset in an Environment Managers predictive arsenal. If you can build solid working relationships with people at every stage of a Release lifecycle – from the Business Analyst that shapes the projects, to the Project Managers that will run them, to the Technical resources that build them, the Testers that check them and to the Release team that puts the whole package live, you will have a far better chance of hearing or seeing about new or changing requirements before they are formally asked for. Nothing ever comes completely out of the blue.

It does take time to  build up your spiderweb (and you may get some odd looks at first from people who have never even heard of a Environment Manager, let alone spoken to one)- but I assure you -its worth it. The only downside is that after a while you will probably get a bit of a reputation with your Project teams for predicting the future.

My take on 3rd Annual DevOps Virtual Summit #dovs17

I have never attended a virtual summit before. It was an interesting experience – like a more impressive webinar – but it is probably not a format for the easily distracted. I am on GMT and the event was EDT – so it is in the evening for me (childrens dinnertime + software methodologies = a mess!)

Other than the sweetcorn on the floor though, I was pleasantly surprised.

I stumbled across a invitation for this event on Twitter about a week ago – it is hosted by CA Technologies. It was free to sign up, and they had some interesting keynotes from businesses that have trodden the path of moving from Waterfall to Agile and beyond United Airlines, CNN, GM Financial and a few other (the full list is on the CA website here.)

The most interesting keynote for me was GM Financial as this one showed a real work in progress transformation – quite unusual for this sort of event – in my experience delegates would only see the polished final article and not the journey to get there.

The best speaker of the event was Silvia Prickel, Managing Director at United Airlines. She brought a refreshing no-nonsense, senior business stakeholder attitude to the table – again a really useful insight into how differently senior managers think to the developers and testers for a IT project of this size.

Delegates that signed up came away with some useful whitepapers and a copy of a DevOps book that I have yet to have a really good look at (so I cannot really comment on it yet). 

Here is the link to the recorded sessions if you want to check them out yourself.

Would I attend another one? Yes… I probably would so long as I could could guarantee some quiet time to do so. You don’t get the Networking experience that you would with a ‘in-person’ event – you don’t really see or hear from other ‘attendees’ bar the Q&A chat box, but if you can give it your undivided attention I think you can focus more on the information being given by the keynotes. 

That and standing on sweetcorn in socks is just nasty.

Alright then. What is a IT Environment?

Leading on from my first post, the next question to clear up is ‘well….what actually IS a environment?’ To paraphrase IF4IT:

A…repeatable Configuration or set of Configurations that…act as a contained, bordered or surrounding operational context and that allow one or more Entities such as Resources or Systems to perform one or more controlled functions or activities.

…phew!… Lets try putting that into something a bit more manageable shall we?

Imagine a big car insurance company – pick any you want. Lets pick a system they might hypothetically use – lets imagine a website to sell their policies. What does a website need to run? A database, some applications, a server or two? OK. But is that ALL it needs? No. It probably needs to connect to a bank for payments right? Do they connect to other insurers to verify No Claims Discount? Who is doing the premium calculations? How about that extra legal cover they are offering you? What looks simple and seamless from the outside can actually be made up of tens – or even hundreds – of configurations, components and integrations. To the business – this whole entity is the Production environment for this website.*

But wait! The insurance company wants to add or change something on their website – maybe they want to offer Home Insurance? No matter how much faith they have in their technical teams, no business is just going to let them tinker about with the Production environment.. What if they break it? No website = no new policies being bought = no money = someone is getting fired!

So you need to make a copy of all (or at least the relevant parts) of the functionality of the Production environment so that the Project team tasked with adding Home Insurance functionality to the website can construct, verify, and promote controlled changes through the various stages of readiness before they are considered ready for the Production environment. How big should those copies be? What data will they use? What other systems do they need to push/pull information to/from?

In  businesses that have a broad or complex IT systems or are running significant IT change programs,  an Environment Manager (or a team of them) are needed to keep a handle on building these ‘copies,’keeping track of what they are used for, what is in them, who is using them, and what to do with them once a Project has finished.

*For any technical bods out there that are shuddering in horror at my vast generalisations I apologise. I find it easier to ‘tell a story’ when trying to explain concepts, (and I would assume that you all would know what a Non Production Environment is anyway!)

My kind of Environment Management has no Trees

Its a fairly standard question – so whats do you do as a job?

Outside of very specific circles, the answer I give usually has one of two responses.

          “Oh what – like trees and climate change?” (resigned sigh)

or if I am lucky,

          “Oh that’s something to do with IT right?” (brief start of surprise from me).

Yes that right! I manage test environments for a major retailer… oh you’ve gone.

For those that stick around long enough my favourite analogy is ‘an Air Traffic Controller for IT environments.’ It was a term I first saw mentioned over on the Plutora blog:  The Definitive Hiring Guide for Test Environment Managers 

Think of a test environment manager as an “air traffic controller” for environments and databases required to test and qualify software for release to production. This job is one focused primarily on tracking and scheduling, but it also involves integrating a number of conflicting inputs to support testing across multiple generations of interconnected systems. A test environment manager balances budgets with timelines and other constraints to give developers the systems they need to ensure that software works as designed in production.

NB: This analogy is not intended to belittle actual Air Traffic Controllers (they stop planes crashing into each other and killing a lot of people after all). There are just certain similarities when it comes to the complex interdependencies that both roles have to juggle.

The thing that draws me to the job is the thing that makes it very niche – the sheer breadth of the role. It the ideal job for a candidate that thrives on doing a bit of everything. Project, Release, Change, Configuration, Strategy, Access, Quality and Data Management are all elements of my role. You need to be able to talk to stakeholders at the right levels – enough technical knowledge to talk to an engineer and business savvy enough to be able de-jargonise for business leaders. It’s a role that requires both reductionist (going down to the details) and holistic (the big picture) thought processes. You need to be able to replan on the fly and also plan far into the future – often in the same conversation.