In my two years at Flatiron, I've been fortunate enough to play a role on several very different teams. I started out on our Provider Applications team, where our mission was to continue to improve Flatiron's electronic healthcare record software, OncoEMR. My six months on the team were incredibly fast paced and dynamic, but one thing remained constant: every decision that we made was directly informed by the needs of our customers. Our product management team was obsessed with improving the experience of the doctors, nurses and practice administrators who use OncoEMR on a daily basis. This is a fairly standard best practice: build for your customer.
One of my fondest memories over the past two years was being in the audience for the keynote speech at our annual customer conference—OncoCloud—in September 2016. Our head of clinical oncology, Dr. Bobby Green, and our VP of product management, Keith Cowing, demoed "Refresh," the project that my team had been working on feverishly for months. We're no Apple, but there were audible oohs and ahhs from the audience during the big reveal! It was an incredibly gratifying moment, witnessing the excitement that our customers showed for something that we'd worked so hard on.
Before we could build the new features that drove the crowd wild at our conference (only slightly exaggerating), we had to ask our customers to place an enormous amount of faith in us and accept that we wouldn't release anything new for a year while we worked to restructure the innards of OncoEMR. Amazingly, they understood and agreed to bear with us while we rejiggered the foundation so that we could iterate faster, and thankfully, the trust was repaid with interest. I give a lot of credit to our product managers—one role that they take on is as the buffer between engineers and customers to bridge the gap in empathy, and they had clearly pulled off something special with Refresh.
My next tenure at Flatiron was on our DevOps team. This meant transitioning from being on a product-oriented role with external customers to a focus on infrastructure team whose client base was most of Flatiron's engineers. On DevOps, one of the major projects we undertook was to deprecate our identity/DNS server in favor of something better.
This project was daunting in scope and an altogether new experience for me. It was also viewed as the white whale of the team—something that we'd been trying to achieve for a few years, but which seemed like it could easily extend far into the horizon. The reason that this was such an arduous project was because the server was the central identity and DNS source for all of our Linux fleet, which meant that most of our products relied on it for their uptime. So, making any changes to the cluster meant getting most of engineering onboard with some potential downtime.
This was a hard ask for two main reasons: engineers would not necessarily see the benefits that they would reap, as this was underlying infrastructure; secondly, they wouldn't be able to migrate at their own pace, this migration would be a hard-cutover. Because of these reasons, the deprecation kept getting pushed back over and over, leading to a state where no one was really happy. Engineers hated relying on a service that would have at least a couple of outages a year, and Ops was extremely frustrated with being saddled with a service that couldn't be improved.
Eventually, after one outage too many, a line was drawn in the sand, and we set a hard deadline for deprecation which was agreed upon by our engineering leadership team. The weeks leading up to the deadline were hard on everyone. The Ops folks set up camp in a meeting room for two entire weeks, where they were on-hand as needed to help engineers test their services against the new hotness, and make sure that everything was kosher. We set two days apart for doing the final migrations, at the end of which the old server was deprecated and decommissioned for good. As someone who had to deal with managing this service for so long with no light at the end of the tunnel, it was surreal that we actually managed to kill it. It felt like having a giant weight lifted off of our shoulders—a feeling shared by everyone in ops and engineering.
Me bashing a pinata to celebrate our win after deprecating our old identity and DNS server.
While we had project managers helping bridge the "empathy gap" when we were working on Refresh for our external customers, what I learned during my time in DevOps, though, is that engineers aren't always the best at showing empathy for other engineers. Every team wants to deliver to their customers, but when those customers are your peers, suddenly, people are willing to place a much higher burden of disruption and understanding on their user base. The attitude can easily shift from "Let's fix these bugs before building newer things" to "We know we have to fix this now, but the other teams will understand how this new platform will fix the problem when it's done". The more technical your users are, the more willing you are to defer on their needs.
Of course, this attitude can go both ways, as evidenced by the story of our unkillable auth server. Our team was so bound by the needs of our engineers—our internal customers—that we were continuously deferring on critical work, which was fine in the short-term, but left no one pleased the longer it continued. When your customers are technical folks, they might call your five-alarm fire a flash in the pan and be unwilling to evacuate the building. In the end, everyone gets burned, and there's no joy in saying "I told you so".
The key in engineering (as in life) is to treat others how you'd like to be treated. Developers and users have to maintain an equilibrium of empathy. If you move the slider all the way left to apathy, you might churn out the most beautiful code of your life with all the features and testing that you could ever hope for, only to find that your customers abandoned you a long time ago for a potentially buggier but more needs-tailored product. Likewise, if the slider is all the way right to subservience, your customers might be glad that you're catering to their every demand as they make them but get equally as disgruntled as you over time when the tape starts to fall off and the cracks get bigger. Just as developers need to understand why responding to customer needs must sometimes be prioritized above else, the latter group has to be willing to put up with delays or even a little pain periodically if the developers claim a need to make repairs or ask for patience when they're laying foundations for something new.
In the end, you must always listen to your engineering customers, but make sure to get a word in yourself when needed.
