July 01, 2020
Today sucked. Being Canada day it was supposed to be a Stat holiday but instead I spent the day work on a production degredation outage.
Some history… The backend of an application I do the frontend for went through 3 different devleopers and is now a “state”. This is a side effect of the lead developer switches and not so much of the ability of the developers themselves. What makes the situation even more complicated is that I as the front end developer was brought on 50% of the way through the project and I have had zero input/review of how things were developed in the backend other than the REST endpoints my React application calls. All of this has led to some pretty horrendous performance as the client scales up usage of the application.
In hindsight, the application would have been developed with a single lead developer and constant code standards and pull requests to ensure a more consistant code base and likely a lot less performance issues. So what happened? Disconnects happened. There was a disconnect between each of the developers working on the project. Disconnects between the frontend and backend teams to ensure code written lined up and built with the correct read/write priorities. Disconnects happen so fast in fast pace environments and can have catastrophic consequences for a project.
Fortunately for us, the current backend developer, project manager and I were all available to work through the performance issue in front of us and reduce the speed of the specific backend call by more than 50% and get it deployed for the customer. We are a long way from out of the woods but we will get there, one endpoint, one function at a time.
In closing, never underestimate how effective pull requests and code review can be in catching dumpster fire code before it gets into production…eg doing 1900 database queries every time a record is updated in a db.
Happy Canada Day…
Did you know I also send a weekly newsletter called The Eggbeater about tech, manufacturing, society and the problems I'm working on.
Follow me on Twitter at @colereid