On September 24, 2014 someone posted on the oss-sec mailing list about a
bash vulnerability that likely affects several decades of
bash versions (something like
4.3!). The vulnerability — aptly named "Shellshock" — can lead to remote code execution on un-patched hosts, for example web servers parsing HTTP environment variables via CGI GET requests, sshd configurations using
ForceCommand, DHCP clients, etc.
Anyways, I'll leave the infosec community to expound on attack vectors. The point of this post is really to illustrate that you should be using an infrastructure orchestration tool like Ansible to manage your servers.
Painless Patching With Ansible
Patching your systems is painlessly easy if you manage your server infrastructure with something like Ansible. Using a one-off command you can easily update all "web" servers, for example:
$ ansible web -m apt -a "name=bash state=latest update_cache=yes" -K -s
That's great, but what if you have both Ubuntu and CentOS hosts in the "web" group? CentOS doesn't use
apt for package management, so this has effectively only updated hosts running Debian-family GNU/Linux distros.
Playbooks: The Power of Ansible
When you have more than a handful of servers, the combinations of DNS names, IP addresses, roles, and distros becomes overwhelming. With Ansible you define your inventory of hosts, allocate them into groups, and then write "playbooks" to mold your servers into functional roles, ie web, database, compute, proxy, etc servers; the personal relationship between sysadmin and server is gone.
Here's a simple playbook I wrote which takes into account the different OS families in our infrastructure and updates the
bash package on each host.
--- # To update hosts for "Shellshock" bash vulnerability # See: https://en.wikipedia.org/wiki/Shellshock_(software_bug) - hosts: all sudo: yes tasks: - name: Update on Debian-based distros apt: name=bash state=latest update_cache=yes when: ansible_os_family == "Debian" - name: Update on RedHat-based distros yum: name=bash state=latest when: ansible_os_family == "RedHat" # vim: set sw=2 ts=2:
And then run the playbook with:
$ ansible-playbook shellshock.yml -K -s
In our case we patched twenty-five CentOS 6.x, Debian 6, Debian 7, Ubuntu 12.04, and Ubuntu 14.04 hosts living locally, in Amazon EC2, and in Linode. With one command. In less than five minutes!
Vendors started pushing patched versions of
bash on September 26th, two days after the initial disclosure. Two days after those patched versions were released there were new variations of this bug discovered, and new packages issued (and we patched our systems again!).
As of now, five days after initial disclosure, there exist five CVE identifiers for this bug! So keep an eye on social media (#shellshock?), Hacker News, and sites monitoring this bug, because more new vectors may emerge!
This was originally posted on my personal blog; re-posted here for posterity.