You may have heard that Equifax, a vast international company trusted with millions of people’s private data has been hacked. In this regard they can join many other organisations that have suffered massive data leaks in recent months. As they hang their heads in shame and consider the massive costs incurred, I’m sure there is something they can take solace in. I bet their patching policy is now first rate.
Patching is something organisations claim they do diligently, but rarely do completely. Tools such as Puppet, Ansible and Chef have ensured that patching operating systems and devices is both automated and easily audited. And the tendency is to leave things there. After all, if the network equipment, servers and desktops are patched things are OK
This is a mistake. These days the OS runs a slender attack surface and is not generally facing the wider internet. The biggest assault on OS’s comes from inside the network. Despite many warnings and the occasional bludgeoning, users still insist on opening Email attachments sent from people they’ve never heard of. In this case, OS patching is vitally important, as noted belatedly by the NHS in the aftermath of the wannacry outbreak.
However, for organisations that make a living by producing web facing apps, the largest attack surface is still the applications themselves. So it’s curious that development teams are still not taking patching seriously.
There are numerous reasons how this came to be. First of all, from a developer perspective updating dependent libraries can be painful. Even outside of major releases, some library updates have a nasty habit of breaking existing code. If it is a major release, then all bets are off, you’re almost certainly going to be re-writing the code that used this library. Developers are forced to wade through and rewrite existing code that is probably perfectly adequate for the task.
If it’s a pervasive library such as a date library then you could be in for a major piece of work as you visit each and every element that deals with dates and correct it for the new version of the library.
From a business owner’s point of view, application patching is costly and generally offers nothing visible to the end user. In complex applications, it can take up considerable time, time that isn’t being spent creating features for customers. This is a hard sell. Budget is finite and competition is fierce, and for some companies razor thin margins dictate that the Tech Debt credit card takes a battering, with patching being another line item on the ever growing list of tech niggles.
This is also a mistake. As many organisations will tell you, putting security on the back burner is a fundamental and potentially business ending mistake. However, it’s certain that every single CEO in the those organisations believed that systems were adequately patched. This may even extend to the CTO and project owners. Why is it then that there is such a disconnect? Partly, it comes down to outdated expectations and performance indicators.
Operations departments have always had patching at the core. Feature releases for operations departments are generally expressed in terms of rolling out new applications or operating systems, which are in themselves patches. A good operations department patches early, patches fast and patches completely. Conversely, developers are judged on visible and revenue generating features and are seen as being ‘fussy’ or ‘perfectionists’ when they suggest cleaning up old code. By allowing one group to proactively pay down tech debt and punish the other, the message is sent that dependency patching is to be avoided.
“By allowing one group to proactively pay down tech debt and punish the other, the message is sent that dependency patching is to be avoided.”
Modern development practices also play their part. These days, development generally takes a Lego style approach, with the bulk of the application made up of a slew of third party libraries. Everything from dealing with dates through to low level elements such as multi threading are dealt with in an imported library. This makes sense, and is generally a good approach; business features pay the bills, so why re-invent the wheel on the dull stuff?
The problem then comes from managing and patching these libraries. Some languages revel in this approach, with Node in particular not blinking at having scores of dependent libraries for even the smallest app.
We should also examine the role that outsourcing plays. Many software companies outsource some, if not most, of their development to third parties. These third parties will generally deliver exactly what you asked for, and tend not to look into the wider needs of the product; that’s what the business analysts are for. Patching of libraries is rarely proactively offered by the outsourced developers and, without a strong technical lead on the product side, it is not asked for.
So how can we extend the patching culture to all corners of an organisation?
Managing dependent libraries has historically been painful, requiring developers to list, research and act on potentially hundreds of dependencies manually. This is no longer the case. Tools such as Black Duck or Nexus Lifecycle will tie into most continuous integration systems and allow automated checking of dependencies.
With the advent of containers it is now possible to create a complete bill of materials that make up a deployment: from the operating system user space, right through to app libraries. Tools such as Anchore and Docker’s own security scan will then give a complete picture of which potential vulnerabilities exist within that deployment. Using this data, all levels of the organisation have an instant view of the current risk exposure within the application stack. This in turn can feed into corporate policy, setting out how much risk is allowable before deploying to the internet at large. This gives the development teams the much needed remit to improve, patch and secure code.
There is no excuse for not patching your applications. If you are a CEO/CTO then you need to be aware of the effort and impact of library patching and plan accordingly. If you are a product owner or scrum master, then you need to be pushing your team to patch early and patch often, and to be aware of the impact it’s going to have on productivity. And if you are a developer or DevOps engineer, it’s down to you to push from below, ensuring those above are aware of the risks.
After all, no one wants to be the next Equifax anymore.
Featured Photo by Matheus Ferrero on Unsplash
Hack Photo by Markus Spiske on Unsplash