Invented here syndrome

Are you afraid to write code? Does the thought linger in your brain that somewhere out there somebody has already done this? Do you find yourself trapped in an analysis cycle where nothing is getting done? Is your product mutating to accommodate third party components? If yes, then perhaps you are suffering from invented-here syndrome.

Most of use are aware of not-invented-here syndrome, but the opposite problem is perhaps equally troublesome. We can get stuck in the mindset that there must be a product, library, or code sample, that already does what we want. Instead of just writing the code we need a lot of effort is spent testing out modules and trying to accommodate our own code. At some point we need to just say, “stop!”, and write the code ourselves.

Varying levels of quality

As a general rule it makes sense to not write code that already exists. Often the choice is easy. We just pick up a standard product, lump it in with our code, and it just works as promised. These situations are great, and do happen often enough.

Past this come the less than ideal choices. Perhaps a library covers only 95% of the requirements, configuration is a bit more involved, or the purpose has to be twisted a little bit. It’ll work, but not without some effort, or rethinking a few requirements.

At some level we enter the realm of crappy products. These are things advertised to do what we need, but fail utterly at that task. Or perhaps they just do something in a completely different way than expected. Maybe the integration is extremely troublesome. Perhaps the product is just too bug ridden to be trusted, or the documentation so bad that proper usage is a mystery.

From what I’ve seen, the vast majority of coding products, snippets, or libraries fall into this category of crappy software. Being available in a package manager, being downloaded by thousand of people, or having a fancy web page, are no indications of a good product. It’s trivial to publish programming products nowadays. Chances are for any obscure requirement there is already some matching product. Just because it’s there doesn’t mean it should be used.

I should point out that the vast majority of any project is comprised of standard products. Just consider the operating system, the compiler, file system, shell and the build system. Add to this the excessively common, or even builtin, libraries like SSL, HTTP, or simply using the browser to render HTML. The fear that a project is not using enough standard components is often misguided.

It’s about the time

It shouldn’t take long to discover when a required module doesn’t exist. Sure, there are products but either they are far less than ideal or just too crappy. New search terms are either coming up empty or returning the same results. Lists of popular, seemingly suitable products, just don’t include any that really fit. That’s it. The search is exhausted.

I’m not saying we should immediately jump from here to writing our own module. No, now is the point where evaluation becomes important. Time is usually the critical factor here. How long will it take to write a custom component? How long will it take to adapt one of the existing products?

I feel that the time savings from using a less-than-suitable third party product must be an order of magnitude higher than writing it myself to consider it worthwhile. Third party code has a lot of open questions, regardless of how well it has been evaluated. This uncertainty must be factored in to the consideration.

Getting stuck in analysis paralysis is very bad. I have no problem writing code prior to completing the analysis. I consider this a valid form of evaluation. Often I’m not really certain what I even need until I’ve attempted an implementation. Stalling a project can often be disastrous. Perhaps my quick code is just enough for now and let’s us defer the decision to later.

It’s about the features

One trap that’s easy to fall into is looking at the full feature set of a product during evaluation. Who cares about everything a product can do. Our project has a specific list of requirements, and those are the only requirements we should care about.

This is where a lot of popular products falter. They offer a complete package, but we simply aren’t looking for a complete package. Sure, they have the feature we want, but there’s no way to efficiently extract it. We need to view the individual features on their own. This is relevant to the time consideration. Clearly product Xyz would take millions of man-hours to recreate, but perhaps module Q will only take a few days.

A second aspect here is how vital a requirement is to our own product. Compromising our key selling points is going to doom the project to failure. There’s no value in saving time if it doesn’t result in the intended product. We just have to face reality sometimes: getting the desired feature set may involve writing a lot of code. We’re programmers though so that shouldn’t scare us.

Warning! Being delusional here is not helpful, and can often lead to genuine not-invented-here syndrome. Most features can be realized in many different ways without changing the substance of them. The time invested to write a feature exactly as we want has to be related to how critical that feature really is. It’s best to involve marketing, or product management, in such decisions. It can be easy at times to lose sight of what is truly important.

Write some code

Fretting over a selection of inadequate products is not productive. While it’s entirely reasonable to avoid not-invented-here syndrome, becoming overly frightened of writing code can land a project in a production quagmire. It shouldn’t be surprising that software development involves actual coding.

A great mass of libraries, modules, code, and other software products, available on the net are either crappy in their own right, or simply not suitable to our project. Forcing things to work together can be costlier than simply writing the required code on our own.

And what if coding turns out to be wrong choice? Well, that’s part of development. It’s quite possible attempting our own component actually leads to the correct third party product. Since we’re following good design practices it won’t be a huge problem to swap components in and out as desired.

In my programming journeys I’ve seen oddities and curiosities of all sorts. Follow me on Twitter to hear more such tales. I’m also available for presentations and seminars, just contact me.

19 replies »

  1. In theory I can see how this could be a problem, but in my real-world programming experience I’ve never seen it happen, except in the sense of management or architecture insisting we use a particular (unsuitable) framework. All the developers I’ve worked with have been more than happy to write things themselves; I’ve never seen anyone err by using a library when they should have done a custom implementation.

    • It’s possible that you’ve been lucky in this regards. I’ve had this happen to me on multiple unrelated projects. Are you sure you’ve considered the whole tool chain and deployment of the software? From framworks, to packaging tools, to deployers, to monitors, etc.?

      The symptom doesn’t always result in actually using the third party code, but often just ends up spending excessive time invested in testing products. I would say attempts to follow methods and best practices also falls in this category: forcing your code into an architecture just to meet some popular practice.

      It’s possible the degree is also domain specific. In the WebApp world their is probably at least 3-4 libraries that do everything you could ever want, but do it poorly. I often encounter the frustration of looking each time I do such an app. In the embedded medical market I suspect there’s less of a glut. More personal to me, while writing Leaf I don’t encounter an overdose of compiler products. Domain popularity and accessibility likely play a significant role here.

    • Consider yourself lucky. I’ve worked at places (Microsoft shops, mostly) that would sooner misuse and abuse a product offering than write new code. I still have nightmares from working with SQL-Server Integration Services, “because it’s just drag-and-drop, so there’s less code to potentially get wrong!”

      Yeah, sure, except for the reams of code we stuffed in SSIS to work around the fact that SSIS is an ETL tool, not a fucking medium for expressing business models. That’s what, you know, real programming languages are for. And here’s another idea: we’re programmers, so perhaps we should be paid to program, instead of gluing a hodgepodge of enterprise software into a festering abomination that would make their inventors shit their pants in horror.

      Trust me: you’re a very fortunate person if you’ve never had to deal with Invented Here Syndrome.

  2. People like to go doing opposite stuff.

    I prefer to say the “We will not used something here, if is Not Invented Here” Syndrome, to be more clear about the issue.

    But, I agree with your post.

    There is also the “we dont require stuff to get Invented Here …” Syndrome.

    Because “… there must be already invented somewhere, so we dont have to, Reinvent The Wheel, wasting time and other resources” .

    I had a rockstar coworker, that supported its own software by the “not welcome, if, Not Invented Here”, and applied the “Invented Here” / “Dont Reinvent the Wheel” syndorme to others coworkers’ software.

    I eventually quit.

    Good Post. Cheers.

    • There’ll obviously be people going in both directions, shielding themselves behind catch phrases. Anybody who actually uses the term “not invented here” or “invented here” syndrome to justify something needs to understand the issue deeper. These are just labels to a concept, meant to bring attention to underlying problems.

  3. Part of the problem is the terminology.

    When one states they are “reinventing the wheel” it implies they are wasting time.

    Had “refining the wheel” been stated it implies some useful work is being done; some existing solutions are square-pegs to a round-hole problem.

    There’s an extra benefit I frequently tell myself when writing software from scratch — the knowledge how all the pieces work together is better gained/understood when working at the lower levels of abstraction. It takes longer, but in the end the why and how a wheel turns is better understood.

    • I agree that phrasing can easily skew the issue. The moment a developer says “I’ll just role my own file store” somebody starts to think they are about to build their own relational database system, when in fact they might just be serializing a list of strings to a file.

      It should also be easier for us to write a piece of functionality than was for an author of existing software. It’d be silly to think we’re actually starting from scratch. Instead we can look at the existing software, various options, algorithms, and code and quickly see how to implement the piece we need. The burden of research is a lot less than when inventing truly new components.

  4. Hi,
    I’m a French teacher in computer sciences and I do agree, frameworks kill the students creativity and ability to write their own algorithms. I shall have your article studied by my students ! May be I’ll translate it, would you mind ? (I did not find any licence for your posts).
    If you read French, I’ve just written an article which deal nearly with the same subject ; if not and are still interested, I ought to translate at least an abstract of it…

    • Feel free to translate the article. Just cite me as author and I’m fine with you using it.

      I think at a teaching level it’s espeically important to avoid using too much third party code. The students need that opportunity to truly get into the material and code things. It’s the only way they’ll be able to properly evaluate third-party code later while working.

      My French is limited but I think I can figure out the gist of the article.

    • Stéphane, have you considered that you’re trying to promote the wrong kind of creativity? How many times do you have to invent a binary tree lookup? The whole point of frameworks is to eliminate the constant churn of reimplementing things we all use daily. It should free people up to be creative about the bigger picture.

      In a way, it’s like arguing that a painter who no longer grinds his own pigments and mixes his own paints is somehow less creative than one who goes and buys tubes of paint at the art store. In a sense, it’s true – but is it a useful kind of truth?

      I’m not sure it is.

    • Hi Jeff, are you commenting my comment or the article I pointed ? Anyway, my point, and I think Motoray’s too, is not to say that you need to rewrite anything all the time when you are coding, but :
      1) You need to know what is a B-Tree and associated algorithms whether you write one or use an already written one ; and, from a pedagogical point of view, in order to understand it, to grab it, you need to write it, several times… using frameworks too early prevent the student from that experience, and finally that knowledge ; I’m not a painter but I guess having basics in pigmentation is more or less the same point (and I think some artists will work at the pigment level).
      2) As Motoray wrote, it is not always more efficient to reuse comparing to recoding ; but if you are not able to code, you are doomed to use frameworks, you loose a possibility of choice.
      3) This is underlined by the French philosopher B. Stiegler (I added references and quotes in my article) as “prolétarisation” (proletarianization ?), extending Marx concept to the loose of knowledge and know-how we face as consumers, particularly with Internet constant proposals of “services”.
      So I surely wouldn’t say not to use frameworks neither not to use the web when looking for an information – while publishing myself on the web, teaching and developing digital document oriented software :) – but to be aware not to become a simple user, an remaining a contributor (which requires fundamental knowledge and know-how).
      Thanks for the discussion.

  5. “savings from using a less-than-suitable third party product must be an order of magnitude higher than writing it myself”

    I don’t know about an order of magnitude, but your point is well taken. It really comes down to balancing the effort of searching, evaluating, integrating, and making up for shortcomings with the effort of building it.

    Several years ago I was lead on a project that required a complex workflow with a multitude of rules that changed on a regular basis. I was convinced that there must be a workflow or business rules engine that could satisfy all or at least most of our requirements and I spent quite some time searching and evaluating existing products and libraries.

    When nothing fit quite right I began building it myself, which in the end turned out to be a good decision — our solution met all of the requirements, it was completed under budget, and it has easily handled ongoing changes ever since.

    But even during development and to this day I haven’t been able to get over the nagging feeling that I should have been able to buy rather than build more of the system. It’s time to just get over it.

    • Yes, that feeling is the worse when you are doing something that you just know hundreds of other companies must already have. There are just so many things that one assumes must already be written, which simply aren’t available in a usable form.

      In a away this makes sense. Most projects are to build a product for users, not to build libraries. So even if a project as a really good workflow engine, it would never be packaged up and made available to others. Often library products just don’t have the same level of man-power invested as these specific purpose in house projects, thus fall short on requirements. It’s also far harder to develop a general purpose library than a single-purpose one.

      But I always have this nagging feeling when I have to code a common module. Especially when I’ve coded the same thing on previous projects.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s