The cost of software developed too soon

The cost of software development is critical for business to ensure a return on investment (ROI) of delivered projects, and great decisions can lead to the perfect timing in a market opportunity window. Various metrics can be used to measure that depending of the management style driving your company.

Making architecture decisions about the way a feature or a project must be oriented can be difficult and can create frictions between people striving for the same goal, particularly when the YAGNI principle comes into play. I learned about this principle in university and when I started to work, a mentality of trying to anticipate the future, or trying to build blocks that will be leveraged later, settled in. The YAGNI principle started to be considered a cheap and bad way to implement software with only short-term vision. Complex, generic and future proof design was the motto; it was well seen.

One thing I learned during those years is to tell the difference between building a block that will be use later in an on-going project and a block that will be used in a future project. During on-going projects, hypotheses can be quickly verified, requirements and constraints are known (most of the time 🙂) and the code evolves during the learning process of the domain you try to implement. Predicting a couple of steps ahead can be done with a relatively good success ratio. This is not the case with future projects. At least, this is not what I personally observed.

The following story is 100% fictional. The usage of real company names is purely to immerse you in the story, I completely invented their API functionalities and limitations for the purpose of this text.

Suppose you are developing the web site of a business located in Canada selling custom made running shoes using people’s unique foot morphology. You built a great cart, a great mobile app for foot scanning and you integrated a payment gateway. Business is going well, lots of sales in Canada. The real pain point is the shipping process. Everything is done manually. Displayed shipping costs are based on hardcoded data provided by Canada Post that needs to be updated each year and is sometimes outdated. Tracking numbers are sent manually by email to the customer after Brigitte received the shipping label with the number printed on it when she goes to the post office. Shipping to the US results in a poor user experience since you can only display an estimated shipping cost. But since sales in US are marginal, this is not critical.

After a couple years, the project to integrate a shipping provider API to the platform kicks in. Because the business had a great experience with Canada Post, and because their API exposes functionalities to know the shipping price depending on the dimension and weight of the package, to receive the tracking number automatically when sending a pre-authorization request and to fetch tracking details of an order, the business decided that Canada Post will be the shipping provider in Canada. Unfortunately, Canada Post is not competitive for US shipping, so the company decided to continue the manual process for the marginal sales in the US.

The current project targets the Canada Post API integration. But you know that one day, other providers will be integrated allowing the business to expand its business in the US and Europe. Upper management is not so willing to tax the current project to develop a generic framework that will accommodate future providers. But the development cost estimations show that the cost of doing it right now is attractive, since the tax is not that high. Finally, upper management gives the green light to do it.

During the project, the following situations occurred:

• Naming the generic concepts took longer than expected. You even renamed them a couple of times, and modified SQL schemas. Everybody has their own opinion about the ubiquitous language of the shipping domain because they read about it 20 minutes and because they know someone at UPS. In fact, there are no real shipping domain experts in the team.
• Designing the building blocks of the shipping tracking was kind of hard. Of course, you did not want to influenced too much by the Canada Post terminology and the data they exposed so you took great care to organize your class in a generic way that will be reusable. Lots of white board to finally achieve this!
• Since the integrated shipping provider will be only available for Canada, you designed a system that can classify providers depending of the country. This way, future providers will be easily integrated. But you had this feeling that you will need a way to rank providers available in the same country, or at least, to allow the choice of provider when many are available, like UPS and FedEx in the US. You even took time to imagine a system that could query multiple providers for submission and choose the cheaper one. But that was kind of far-fetched, so you decided not to implement it right now. But you kept that in mind and brought that to the table as an argument when design decisions are more or less related to that.
• You built a modular system that allowed authorization on multiple APIs with a testing framework in which you can easily switch the implementation for real integrated testing or mocked unit testing. You made sure to have a way to store the credentials in a generic way for any provider. That way, it will be easy to integrate the next provider.

That kind of list can be very long depending on the project, the number of people implicated in the project, and the kind of people themselves! After several weeks of delay, the project is finally in production with a couple features that have been scoped out. Nevertheless, the management is happy because they know they accepted the risk of building software for the future.

Two years later, the business decided to enhance the shipping process for the US by integrating another shipping provider. That should be easy and quick because the business decided to invest in the platform two years ago!

During development, some people realized that Canada Post had become competitive in the northeast states of the USA because there was good business there. At the moment of designing the code, market studies show that multiple providers could be wisely used for a single country, but not across multiple countries. Oops.

Another surprise arises; it seems that Canada Post API was way better than those American providers because it allows a price submission using the dimension and weight of a package while FedEx and UPS are so big that they only care about the weight. It felt so right two years ago to specify the package dimension as a mandatory parameter everywhere in the code.

And a last one! Even if you took care to model the customer address in a way that supports US and Europe standard format, you realize that none of the US shipping providers handle a two-lines address which you deliberately did because that’s how it works! It’s not your fault!

All those kinds of surprises can be fixed by some refactors, by test adjustments, by SQL schema modifications and data migrations, and by UX modifications. Sometimes it is easy, sometimes it can be very tricky. In all cases, it typically adds unanticipated delays in the execution of a phase 2 project and it becomes a broken promise to the stakeholders that initially approved the framework implementation, expecting faster new developments. Unfortunately, those delays directly caused by the initial design are rarely measured, and never opposed to the past decisions as a retrospective. It is often seen as a negligible collateral damage that is inherent to the software development process, backed up by the comfortable idea that without those anticipations here and there, the later cost of a future initiative will skyrocket at the point where that initiative won’t start at all. And even when the additional delays are not so important, it is not an indication that the initial building blocks were profitable and is unfortunately often considered a good move.

In that scenario, the company finally decided to integrate a second shipping provider. But what if it never did? The time spent to develop the code in the past is not the only cost to consider. All refactors and all new features added to the existing use cases need to live along the unnecessary code. They also need to match the unused functionalities or the future enablers, they need to be built on it! All the tests must also be maintained, and fixed, over time. We also need to consider the time required for new people getting onboard with a source code to analyze and understand those future-not-yet-ready-and-implemented-diamond-architecture-patterns-and-features-that-the-business-will-surely-need-one-day.

This is the cost of software developed too soon.

When the early enablers live in the code for too long period before actually being used, it is a matter of time before the initial design falls apart because someone will finally decide one day that supporting a previous architecture decision enabling an unused feature costs way too much to implement a new requirement. Anticipating a requirement today does not guarantee that it will be a requirement in the future. There are good chances that the concepts that have been imagined and implemented early, based on hypotheses, will need to be reworked anyway when a task force will address the domain problem for real. Ever heard the expression “bite off more than you can chew”? This is why it is important to tell the difference between a building block used in an ongoing project versus a future project.

People get excited to anticipate the future and to establish and place the puzzle pieces in advance. We developers are great at thinking we understand the business better than the stakeholders. Surely it is a stimulating exercise to do, but the further the execution of a project is in a roadmap, the more hypotheses that must be taken. Even the most talented people tend to fail at resolving these hypotheses. And all that extra time required pushes back the moment the code will be alive in production and used by your clients. The cost of software developed too soon also includes the cost of not learning from your client, having your feedback later in the life of your product, scoping out features because the unexpected delays tighten the schedule and deliver output that gives no business value to your clients.

Writing code is easy. Writing great code is hard, and you only know whether your code is great a couple years later after it has gone through multiple development lifecycles, when many people have used it, enhanced it and fixed it, when you have measured what was the cost of producing the code at each lifecycle and what was the gain for your business at each lifecycle. Measuring the return of investment should not only be done to justify the execution of a project, but also to determine its success years later. And that must include the cost of software developed too soon.