Lesson 11: Custom-built is usually the cheapest option for development teams

Background

During one of my stints as a consultant, I was working for a client in the water management space. The client hoped to integrate their SharePoint environment with a third-party product they were using for authentication and membership called Personify, and my company was pulled in specifically to assist with that implementation.

Outcomes

My particular part was to allow SharePoint to have users authenticate using Personify. In other words, when a user enters their username and password in SharePoint, the user should be authenticated via the Personify credentials. I did extensive research into both SharePoint’s and Personify’s available APIs. While Personify had an “out of the box” solution for this, this solution didn’t meet the needs of our client. Instead, I discovered that I was able to build a custom Security Token Service in SharePoint which redirected authentication calls to Personify. After quite a bit of trial and error (and a lot of testing), we built a solution which fit the client’s needs perfectly.

Mistakes

The project was wildly successful, and we finished the integration work in just under 2 weeks. The rest of the time on the project was spent adding important functionality to the platform. Jumping ahead in my career, during several other projects for other clients and companies, I experienced struggles and failures where the choices were made to “buy” solutions rather than build exactly what was needed. I continually fell for the CIC Illusion: COTS (Commercial-off-the-shelf) Is Cheaper. This is the mistaken belief that a COTS product will be less expensive and/or faster to implement than a custom-built solution.

Lessons

I didn’t learn the lesson on this project, but many failed integrations later, I learned what I should have learned at this client: the reason we were so successful is that we spent the time to learn how everything works and built a custom solution that was exactly what was needed. I’m not suggesting that COTS products are bad, or even that they should be avoided. A COTS product has its place; however, they always have hidden costs about which very few people speak candidly.

In particular, there are three. First, many COTS products work as frameworks. By framework, I mean that the flow of control is inverted by these products so that developers are building the equivalent of plug-ins for the COTS products rather than tying the COTS product in via simple APIs or services. While frameworks aren’t inherently bad, the reality is that most development teams are already using frameworks for their development. When too many frameworks are introduced, flow of control is so inverted and twisted that it becomes very difficult for developers to reason about the flow of logic in a system. This complexity results in increased maintenance costs, longer delivery times, lower overall quality, increased ramp-up time for new team members, and a general inability to define the system using a metaphor.

Second, without careful design patterns, COTS products can result in something known as vendor lock-in. Vendor lock-in occurs when a third-party product, library, or API is not properly isolated in a code base. Effectively, teams and solutions are locked into a vendor because the cost of change is so high that it becomes prohibitively expensive to make the code changes necessary to change vendors. Framework-oriented COTS products, in particular, generally increase vendor lock-in because of the framework aspect; however, development teams can also increase vendor lock-in for any product, library, or API by not appropriately encapsulating them behind abstraction layers. For example, vendor lock-in commonly occurs with logging libraries because the “1-line of code it takes to log” becomes riddled inside production code. Developers, this is bad.

Finally, COTS products (like any managed software products) have unexpected management costs associated with them. These costs take two forms. The first form is management expertise for these products. This management expertise is usually siloed in many organizations, which has its own communication overhead and cost, but the expertise can also be distributed throughout the organization and the team members who need to manage the products on a day-to-day basis to get their work done. In either case, the management expertise takes time (and money) to create, to maintain, and to leverage or share when needed. The second cost form is increased complexity for work. In general, the simplest, and therefore the fastest, way for developers to create functionality is to simply write code for that functionality. When a product provides functionality which developers need to leverage, they have to think about where that functionality fits into the overall flow of a story or feature as well as the appropriate ways to test, encapsulate, and design ways to integrate with that functionality. All of this mental exercise requires time, often from multiple people.

In short, don’t fool yourself in believing in the great reusability myth. Given all of the costs associated with a COTS product, if it economically makes sense to use one, then use it! Just don’t be surprised by the hidden costs associated with these products.

Agile Reinforcement

The Manifesto for Agile Software development states that “the best architectures, requirements, and designs emerge from self-organizing teams.” If there is a COTS product you would like evaluated for use by a team, my recommendation is to suggest that product to the self-organizing team (your teams are self-organizing, aren’t they?), and then trust their judgment on whether to use that COTS product or not. I’m not even going to get into the morale impact of an outsider making a decision that the team is stuck with. Teams make bad decisions when they are politically motivated or when they are not provided enough business-level context behind their work; these are teams that are not self-organizing because they are necessarily directed by others. On the flipside, self-organizing teams are in the best position to evaluate various options because these teams will be burdened with all of the costs associated with a particular choice, including the hidden costs. Trust their judgment!

SAFe Reinforcement

The Scaled Agile Framework reminds us to “take an economic view.” When looking at something like a COTS product, decision-makers are often laser-focused on the up-front costs, the recurring licensing costs, and the maintenance costs, and then comparing these against the perceived efficiency gains and the assumed economies of scale the company will experience. To be clear the efficiency gains and economies of scale almost never bear out, and because the decision-makers assumed they would, this turns into a game of blaming the people that are responsible for implementing the tools rather than blaming those that made the decision to use the tools in the first place.

A more appropriate way to look at these choices is to compare the overall impact to delivery timelines associated with the various choices. In other words, implement several pieces of functionality with a COTS product and without it. Then, when comparing your options, in addition to considering monetary costs (which are important), consider what this COTS product does to your overall delivery timeline. If it speeds up your timeline, then it might be a no-brainer. If it slows down your timeline, though, don’t fall for the trap of believing that your people will get faster with it over time because they won’t. Between promotions, employee terminations, ramp-up for new hires, and general team turnover, your organization will never be the experts you hope they will be, because your company’s purpose is not to manage a COTS product, it is to serve your particular customers. That is the knowledge and expertise you want embedded in your people.

I hope my struggles and failures can help teach you this important lesson: Custom-built is usually the cheapest option for development teams.