Pantry Post-mortem

It is been almost a year since I finished working for my previous company. Now I think I can tell what happened there.

I was hired to implement the system to start instances automatically. Company needed this urgently. People waited for several weeks to get new virtual machine up and running. Everything was installed manually. At that point it was impossible to get list of software. And people needed tons of software.

What we dreamed of was 5-minute start. We spent at least 6-human-year.

We configured (or mostly half-configured) chef, wsus, sensu, AD. And implemented/ improved cookbooks for configuring clients for this services. As a result we started about 1000 instances. I believe we had to ssh/rdp to at least 400 instances to finish broken installation.

People needed admin access and we gave it. It is hard to tell now, but I think it was the bad decision. We don’t know what is installed on instances. Because People needed additional software, but they installed it on their own, and we have never find out. They promised to create the ticket, but tickets obviously were ignored.

Our dream was – install everything in automated way. But we forgot our customers – it is better to do MVP with manual step than not to provide working solution at all. We never reached 100% automated installation. We created hacks to be able to install software and we never removed those.

Our goal were self-serving teams. But teams never tried to install something in automated way. They were ready to install something manually instead of waiting. People didn’t care about future.

Looking back – we spend more time on implementing system, than system saved.

The result is all here: