Philosophy

Invented here syndrome

Are you afraid to write code? Does the thought linger in your brain that somewhere out there somebody has already done this? Do you find yourself trapped in an analysis cycle where nothing is getting done? Is your product mutating to accommodate third-party components? If yes, then perhaps you are suffering from invented-here syndrome.

Most of us are aware of not-invented-here syndrome, but the opposite problem is perhaps equally troublesome. We can get stuck in the mindset that there must be a product, library, or code sample, that already does what we want. We spend a lot of time testing out modules and trying to jam them into our system. At some point, we need to say, “stop!”, and write the code ourselves.

Upon rereading this article, I’m wondering how much of this problem is attributed to imposter syndrome. I talk about this in my my book, and also look at all the skills of a good programmer. Perhaps, it’s easy to look at other people’s code and assume they have it all figured out, while we ourselves don’t? This would push us away from developing our own solution.

Varying levels of quality

As a general rule, we shouldn’t write code that already exists. Often the choice is easy. We pick up a standard product, lump it in with our system, and it works as promised. These situations are delightful and do happen often enough.

Past this come the less than ideal choices. Perhaps a library covers only 95% of the requirements, the configuration is a bit more involved, or the purpose has to be twisted a little bit. It’ll work, but not without some effort, or rethinking a few requirements.

At some level, we enter the realm of crappy products. These are things advertised to do what we need but fail utterly at that task. Or perhaps they do something in a completely different way than expected. Maybe the integration is extremely troublesome. Perhaps the product is too bug ridden to be trusted, or the documentation so bad that proper usage is a mystery.

From what I’ve seen, the vast majority of coding products, snippets, and libraries fall into this category of crappy software. Being available in a package manager, being downloaded by thousands of people, or having a fancy web page, are no indications of a good product. It’s trivial to publish programming products. Chances are, for any obscure requirement there is already some matching product. Just because it’s there doesn’t mean it should be used.

I should point out that the vast majority of any project is comprised of standard products. Consider the operating system, the compiler, file system, shell and the build system. Add to this the built-in libraries like SSL, HTTP, or even the browser to render HTML. It’s generally an unfounded fear that a project is not using enough standard components.

It’s about the time

When we search for suitable products, it shouldn’t take long to discover when none exist. We will find modules, but either they are less than ideal or just crappy. As we add new search terms, we are either coming up empty or getting the same results. Lists of popular, seemingly suitable products, don’t include any that really fit. That’s it. The search is exhausted.

I’m not saying we should immediately jump from here to writing our own module. No, now is the point where evaluation becomes essential. Time is usually the critical factor here. How long will it take to write a custom component? How long will it take to adapt one of the existing products?

I feel that the time savings from using a less-than-suitable third-party product must be an order of magnitude higher than writing it myself to consider it worthwhile. Third party code has a lot of open questions, regardless of how well it has been evaluated. This uncertainty must be factored into our consideration.

Getting stuck in analysis paralysis is very bad. I have no problem writing code before completing the analysis. I consider this a valid form of evaluation. Often I’m not confident about what I need until I’ve attempted an implementation. Stalling a project can be disastrous. Perhaps my quick code is enough for now and let’s defer the decision to later.

It’s about the features

It’s easy to fall into the trap of features. The full feature set of a product can seem impressive and alluring. But who cares about everything a product can do. Our project has a specific list of requirements, and those are the only requirements we should care about.

This is where a lot of popular products falter. They offer a complete package, but we aren’t looking for a complete package. They have the feature we want, but there’s no way to extract it efficiently. We need to view the individual features on their own. This is relevant to the time consideration. Clearly, product Xyz would take millions of man-hours to recreate, but perhaps module Q will only take a few days.

As a second aspect, how vital is a requirement is to our own product. Compromising our key selling points is going to doom the project to failure. There’s no value in saving time if it doesn’t result in the intended product. We have to face reality sometimes: getting the desired feature set may involve writing a lot of code. We’re programmers though so that shouldn’t scare us.

Warning! Being delusional here is not helpful, and can often lead to genuine not-invented-here syndrome. While features can be realized in many different ways, do we need it exactly as we want? The time invested to write our version has to be related to how critical that feature really is. It’s best to involve marketing, or product management, in such decisions. It can be easy at times to lose sight of what is truly important.

Write some code

Fretting over a selection of inadequate products is not productive. While it’s entirely reasonable to avoid not-invented-here syndrome, becoming overly frightened of writing code can land a project in a production quagmire. It shouldn’t be surprising that software development involves coding.

A great mass of libraries, modules, code, and other software products, available on the net are either terrible in their own right or simply not suitable to our project. Forcing things to work together can be costlier than writing the required code on our own.

And what if coding turns out to be the wrong choice? Well, that’s part of development. It’s entirely possible that attempting our own component leads us to find the correct third-party product. Since we’re following good design practices, it won’t be a huge problem to swap parts in and out as desired.

This is an updated version of an article I originally wrote on 2015-02-25.

21 replies »

  1. In theory I can see how this could be a problem, but in my real-world programming experience I’ve never seen it happen, except in the sense of management or architecture insisting we use a particular (unsuitable) framework. All the developers I’ve worked with have been more than happy to write things themselves; I’ve never seen anyone err by using a library when they should have done a custom implementation.

    • It’s possible that you’ve been lucky in this regards. I’ve had this happen to me on multiple unrelated projects. Are you sure you’ve considered the whole tool chain and deployment of the software? From framworks, to packaging tools, to deployers, to monitors, etc.?

      The symptom doesn’t always result in actually using the third party code, but often just ends up spending excessive time invested in testing products. I would say attempts to follow methods and best practices also falls in this category: forcing your code into an architecture just to meet some popular practice.

      It’s possible the degree is also domain specific. In the WebApp world their is probably at least 3-4 libraries that do everything you could ever want, but do it poorly. I often encounter the frustration of looking each time I do such an app. In the embedded medical market I suspect there’s less of a glut. More personal to me, while writing Leaf I don’t encounter an overdose of compiler products. Domain popularity and accessibility likely play a significant role here.

    • Consider yourself lucky. I’ve worked at places (Microsoft shops, mostly) that would sooner misuse and abuse a product offering than write new code. I still have nightmares from working with SQL-Server Integration Services, “because it’s just drag-and-drop, so there’s less code to potentially get wrong!”

      Yeah, sure, except for the reams of code we stuffed in SSIS to work around the fact that SSIS is an ETL tool, not a fucking medium for expressing business models. That’s what, you know, real programming languages are for. And here’s another idea: we’re programmers, so perhaps we should be paid to program, instead of gluing a hodgepodge of enterprise software into a festering abomination that would make their inventors shit their pants in horror.

      Trust me: you’re a very fortunate person if you’ve never had to deal with Invented Here Syndrome.

  2. People like to go doing opposite stuff.

    I prefer to say the “We will not used something here, if is Not Invented Here” Syndrome, to be more clear about the issue.

    But, I agree with your post.

    There is also the “we dont require stuff to get Invented Here …” Syndrome.

    Because “… there must be already invented somewhere, so we dont have to, Reinvent The Wheel, wasting time and other resources” .

    I had a rockstar coworker, that supported its own software by the “not welcome, if, Not Invented Here”, and applied the “Invented Here” / “Dont Reinvent the Wheel” syndorme to others coworkers’ software.

    I eventually quit.

    Good Post. Cheers.

    • There’ll obviously be people going in both directions, shielding themselves behind catch phrases. Anybody who actually uses the term “not invented here” or “invented here” syndrome to justify something needs to understand the issue deeper. These are just labels to a concept, meant to bring attention to underlying problems.

  3. Part of the problem is the terminology.

    When one states they are “reinventing the wheel” it implies they are wasting time.

    Had “refining the wheel” been stated it implies some useful work is being done; some existing solutions are square-pegs to a round-hole problem.

    There’s an extra benefit I frequently tell myself when writing software from scratch — the knowledge how all the pieces work together is better gained/understood when working at the lower levels of abstraction. It takes longer, but in the end the why and how a wheel turns is better understood.

    • I agree that phrasing can easily skew the issue. The moment a developer says “I’ll just role my own file store” somebody starts to think they are about to build their own relational database system, when in fact they might just be serializing a list of strings to a file.

      It should also be easier for us to write a piece of functionality than was for an author of existing software. It’d be silly to think we’re actually starting from scratch. Instead we can look at the existing software, various options, algorithms, and code and quickly see how to implement the piece we need. The burden of research is a lot less than when inventing truly new components.

  4. Hi,
    I’m a French teacher in computer sciences and I do agree, frameworks kill the students creativity and ability to write their own algorithms. I shall have your article studied by my students ! May be I’ll translate it, would you mind ? (I did not find any licence for your posts).
    If you read French, I’ve just written an article which deal nearly with the same subject ; if not and are still interested, I ought to translate at least an abstract of it…
    http://aswemay.fr/co/040002.html
    Stéphane.

    • Feel free to translate the article. Just cite me as author and I’m fine with you using it.

      I think at a teaching level it’s espeically important to avoid using too much third party code. The students need that opportunity to truly get into the material and code things. It’s the only way they’ll be able to properly evaluate third-party code later while working.

      My French is limited but I think I can figure out the gist of the article.

    • Stéphane, have you considered that you’re trying to promote the wrong kind of creativity? How many times do you have to invent a binary tree lookup? The whole point of frameworks is to eliminate the constant churn of reimplementing things we all use daily. It should free people up to be creative about the bigger picture.

      In a way, it’s like arguing that a painter who no longer grinds his own pigments and mixes his own paints is somehow less creative than one who goes and buys tubes of paint at the art store. In a sense, it’s true – but is it a useful kind of truth?

      I’m not sure it is.

    • Hi Jeff, are you commenting my comment or the article I pointed ? Anyway, my point, and I think Motoray’s too, is not to say that you need to rewrite anything all the time when you are coding, but :
      1) You need to know what is a B-Tree and associated algorithms whether you write one or use an already written one ; and, from a pedagogical point of view, in order to understand it, to grab it, you need to write it, several times… using frameworks too early prevent the student from that experience, and finally that knowledge ; I’m not a painter but I guess having basics in pigmentation is more or less the same point (and I think some artists will work at the pigment level).
      2) As Motoray wrote, it is not always more efficient to reuse comparing to recoding ; but if you are not able to code, you are doomed to use frameworks, you loose a possibility of choice.
      3) This is underlined by the French philosopher B. Stiegler (I added references and quotes in my article) as “prolétarisation” (proletarianization ?), extending Marx concept to the loose of knowledge and know-how we face as consumers, particularly with Internet constant proposals of “services”.
      So I surely wouldn’t say not to use frameworks neither not to use the web when looking for an information – while publishing myself on the web, teaching and developing digital document oriented software :) – but to be aware not to become a simple user, an remaining a contributor (which requires fundamental knowledge and know-how).
      Thanks for the discussion.
      Stéphane.

  5. “savings from using a less-than-suitable third party product must be an order of magnitude higher than writing it myself”

    I don’t know about an order of magnitude, but your point is well taken. It really comes down to balancing the effort of searching, evaluating, integrating, and making up for shortcomings with the effort of building it.

    Several years ago I was lead on a project that required a complex workflow with a multitude of rules that changed on a regular basis. I was convinced that there must be a workflow or business rules engine that could satisfy all or at least most of our requirements and I spent quite some time searching and evaluating existing products and libraries.

    When nothing fit quite right I began building it myself, which in the end turned out to be a good decision — our solution met all of the requirements, it was completed under budget, and it has easily handled ongoing changes ever since.

    But even during development and to this day I haven’t been able to get over the nagging feeling that I should have been able to buy rather than build more of the system. It’s time to just get over it.

    • Yes, that feeling is the worse when you are doing something that you just know hundreds of other companies must already have. There are just so many things that one assumes must already be written, which simply aren’t available in a usable form.

      In a away this makes sense. Most projects are to build a product for users, not to build libraries. So even if a project as a really good workflow engine, it would never be packaged up and made available to others. Often library products just don’t have the same level of man-power invested as these specific purpose in house projects, thus fall short on requirements. It’s also far harder to develop a general purpose library than a single-purpose one.

      But I always have this nagging feeling when I have to code a common module. Especially when I’ve coded the same thing on previous projects.

  6. Interesting article. I don’t think I’ve seen this presented quite in this light before so I feel like this is a unique take on this problem. I have personally been troubled with code writers block in the past and I feel like this fits what I was experiencing better than anything I’ve seen described. Three factors were involved that I found interesting, one that I had some major success in the past using third party libraries to shortcut development by a huge degree and two that as you grow more experienced you are more aware of what else is out there. So if I contrast with my early career, I was kind of in this naive bubble where I had to write everything myself because I wasn’t aware of another option. I was inexperienced and very ambitious, the Internet wasn’t available. I’ll skip past the success I’ve had with using third party code and libraries and focus on some reason big failures where I was let down by a library in a major way where we had no recourse than a huge custom expensive bespoke replacement. These few colossal failures have tempered my enthusiasm for third party libraries and now I’m somewhere in the middle. At the very least, if I am to use a third party library I would desire the source code as well. I’m back to being more likely to write something first now. I agree that for this to affect someone they probably don’t have a narcissist personality and maybe even lack confidence at times.

    • Perhaps you’ve not gone back to wanting to write the module on your own, more that you recognize it’s a real option. We should always consider what is available first (except for trivial things).

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s