Each new project, whether a standalone program, or a component for an existing program, faces a choice of programming language. Just using what is most popular, or what has been done before, is a poor way to select. You should always evaluate your options and be constantly looking for better ways to get the job done.
While evaluating a language you should also be considering the overall architecture of your project. Not all part of a project need to be, or even should be, written in the same language. The language process will actually be a vital part of your initial design process. How you split your modules and connect the pieces is highly influenced by the answers to the questions posed here.
I’m going to avoid giving specific language examples in this article; I don’t wish to bias your selection with my experience. While some projects will clearly have winners and losers, you should be able to arrive at those conclusions on your own. Languages also change over time, so what was best two years ago may no longer be so, and what was previously excluded could now be a good candidate.
What experience does your team have?
Though it may seem obvious, it bears saying, you should try to choose a language which you programmers know well. While it is great to try out new languages, a non-research project is not the place to start. If you need to make estimates and plan the timeline of a project, you want to avoid any massive unknown variables. You also don’t want a complete lack of language familiarity interfering with the overall design of the project.
This isn’t to say everybody on the team must be an expert in the language. You could even allow that a fraction of the team is new to the language provided you have others who are strong in that particular language. It’d be silly to exclude quality programmers from your project just because they are new to a particular language. Good programmers can adapt. There are also situations, such as after you work through all the questions here, that force you to use an unfamiliar language.
Do you have computationally expensive operations?
Applications such as video processing, graphic rendering, cryptography, statistical analysis, signal processing, and many others have a huge demand on raw processing power. How long they take to execute is tied directly to how efficiently the chips on the computer can be used.
For these modules you will almost certainly want a statically-typed and compiled language. Or more generically, these are the domains where you want a fast language. It is however rare where the entire project has such needs. Usually these performance intensive parts are limited and can be easily modularized and glued together by some other language.
Are many sub-processes and file management involved?
A lot of software primarily exists to automate a series of repeated, or otherwise manual tasks. Each step in the process already has a program well suited for that stage and you really only need to connect them together. This is the primary focus of software development by system administrators, but also includes most make systems and a lot of high-level business processing.
Here you will be best suited with what are often called scripting languages. You need something which can execute other processes and where file management is an innate feature. Above all else flexibility and simplicity are the key criteria.
Do you have tight resource limitations?
In the commodity computer world memory and storage have become relatively unbound. To a degree even processors have become excessive. There are however many applications where such resources are extremely limited. Embedded applications are the most apparent but by no means the only domain. Not all languages are suitable to these limited environments.
When memory is extremely limited you’ll need a language where your code, and the runtime, can fit within that memory and still leave enough working memory. Sometimes loading is more the issue: perhaps you need to load from a small EEPROM or initialize over the network. This will likely require small statically linked and/or pruned libraries. It doesn’t exclude VM based languages, but you will need a small VM.
Are your requirements roughly defined or rapidly changing?
While a well designed program in any language can be easily refactored and adjusted, some languages lend themselves somewhat better to rapid prototyping an experimentation. In the business process world there are particularly a lot of projects which completely lack a specification, or it is extremely rough. Here the client really doesn’t know what they want until they see it. You will find yourself experimenting with many variations until everyone is satisfied.
Often you’ll find it extremely helpful to change something on the fly at a meeting, or perhaps you just need to make continual changes to a report and see what it looks like. An interpreted language provides this ability. Dynamic languages can also be a benefit here as it makes combining several unrelated libraries somewhat easier. Being hidden from the fine details of programming, such as memory management, is also a bonus to rapid prototyping.
What’s the required lifetime of the product?
Not all languages are equal when it comes to stability of the language itself. Many of the newer dynamic languages continue to make backwards incompatible changes or deprecate parts of their core API. Projects which evolve rapidly will never really have a problem with these changes. In fact many of them will actually benefit from such changes. Projects with a short lifespan will also be fine since by the time backwards compatibility becomes a problem, nobody will really care about the project anymore.
If your product however needs to last five, ten, or even twenty or more years, backwards incompatible changes can become a real nightmare. You don’t want to have to keep old versions of your compiler and support tools around, especially if it also means keeping old hardware. Inevitably the project is also living and could benefit from upgrades, or installation in new products. For all this you want a language which has a rigid standard backed by a committee with a goal of long-life and backwards compatibility.
What are your target platforms?
Not all languages are available for all platforms. If your favourite language is not supported on your target device, then it really doesn’t make sense to use that language. You should also avoid experimental support; just because your favourite language is actually written in C and the target OS has a C compiler doesn’t mean the language will work properly there. Specialized chips, even something like a GPU, only have some many languages which can produce the required binaries.
This question doesn’t just apply to chipsets, but also to the other software with which it must work. If you need to have code running in the user’s browser then you don’t have a lot of language choices. Certain consumer device vendors have also been known to create arbitrary lists of allowed languages. Service providers will often specialize in some languages and frameworks at the expense of others. If you intend on writing a device driver for linux you’ll also only find the kernel team supports only one language. You can argue the merit of any of this all you want, but if you wish to support a certain platform, you’ll have little choice but to abide by the wishes of that platform.
Do you need to a lot of bit twiddling?
Working with file formats and protocols often requires manipulation of individual bytes and bits. You will need to convert to higher level formats, and then serialize back to a more compact format. Some algorithms will also explicitly require certain bit operations to be performed on the data. At the lowest levels wire protocols will require you to work with an actual bit stream.
To do such work you need a language that readily exposes these bit operations and provides suitable data types (such as unsigned integral values). Not all binary work requires this however. Many binary structures are simple enough to be handled by high-level packing functions. Look at your exact binary needs and find a language that will be able to do this without too much trouble.
Does you project fall into a speciality domain?
Not all problems are best solved by a general purpose language. There are a lot of speciality languages which exist for very specific problem domains. Artificial intelligence, text parsing, data transformation, expert systems, mathematics, financial analysis, are but a few of these domains.
Domain specific languages tend to save a lot of coding effort and prevent a phenomenal level of defects from ever happening. Use them whenever possible. Given the possible advantage here you may wish to even overrule the rule about choosing a language your team already knows.
To some degree the use of a domain specific language also restricts the other languages you can use in the project. Some are translated into another language while others are available as callable modules. Either way, you will need a way to integrate.
The same type of argument can also be made for excellent libraries. If a library so perfectly matches your needs it may make sense to adopt whatever language is needed to make use of that library.
To make an informed choice you will need to know multiple languages. If you’ve rigidly stuck with a single language you’ll find your decisions to be highly biased towards that language and its methodology. More important than the number of languages is the style of those languages. A good mix of static and dynamic languages, functional and imperative styles, high/low level coding, and domain specific experience is the only way you can properly investigate and evaluate languages. Even beyond language choice, you need a breadth of experience to know how to best make use of whatever language you end up choosing.
The above list of questions is by no means exhaustive, but gives a rough framework of the types of questions you should be asking. Unfortunately the very first question, about what your team knows, will often dominate the final result. In this case look for ways to split the project and use more appropriate languages for some components. I’ve never encountered a project where multiple languages has not been beneficial.
You might be thinking the above questions don’t always lead to a single language choice. In fact, most of the questions tend to prune away inappropriate choices rather than introduce new ones. You should also be planning the modularization of your project and considering languages for each module. You can be increasingly strict in your criteria for a component until only a couple of options remain. Unfortunately you could also end up with no appropriate languages. You may have no choice but to just choose the lesser of two evils, though splitting the project into more components, each with their own language, is often the preferred option.