My take on a software development success philosophy

Below I have compiled a list of  6 personal (overall) viewpoints about software development philosophy that I find particularly important when successfully running software projects. Some of them might be obvious (?), but then maybe not as I have observed that many projects don’t really adhere to them.  In fact, in many projects the opposite is done intentionally or not thus resulting in significant delays, cost overruns, unsatisfied users/customers and missed opportunities [reader: What is your take?] :

• Quality pays – Time and resources spent on quality gives positive return on investment on development time, development costs, team and customer satisfaction etc . The earlier in development process resources are spent on quality related tasks, the better ROI. For example, is well known that finding bugs in requirements or design can be 100-1000 times less costly then in a shipped solution. Unfortunately, many projects are not managed with this fact in mind and many developers don’t have a good enough grasp of quality issues.
• Careful risk-management is needed. – Too many projects are delayed, are over budget or completely fail to deliver. Managing the risks that can cause projects to fail is therefore essential. In particular, I find it important to limit scope of the solution/version worked on (not trying to do too many – possibly half baked – things at the same time), involve users from the beginning, consistent focus on writing well-crafted maintainable software (avoiding technical debt) and to address technical risks early on (like for instance scalability, performance or integration-issues).
• People, development process, technology and clear goals are all important for success. Even great developers fail at projects if process is wrong, technology has major problems or goals are unclear. In particular, the value of a good R&D process is often underestimated. For larger projects, average developers all following a good software process will perform far better then great developers following a bad process.
• Long-term priorities must be balanced with short time priorities. Quick workarounds, hacks and half-baked solutions may help to accomplish some business goals on the short term. However, unless addressed and reworked afterwards they risk permanently lowering the quality of the solution and imposing a technical debt that will effectively tax all future development. After many such short-term solutions, it is likely that development productivity will decrease substantially to the point where new features or bug fixes that should take hours will consistently take days or longer.
• Use modern efficient technology as long as they are not bleeding cutting-edge. Be open to the use of technologies before they are mainstream if (and only if) they offer substantial benefits, are actively maintained, have a significant growing community, reasonable tool support and have books and training/support organisations behind them, Note that besides obvious technical benefits and savings, use of non/pre-mainstream technologies also helps attract the very best developers.
• Investment in automation is key for efficient (agile) software development. Partly to save costs in the long run, partly to be more agile and responsive. In particular automate the build pipeline, deployment tasks and (most) technical tests used for regression. Beware that some automation efforts like for example GUI test automation requires special skills and tools to do right (high risk of failure or low ROI if done wrong).

Do’s and Dont’s for exception handling (moved posting)

… Or what every developer should know about the implementation of exception handling (in your compiler/os):

Modern Exception Handling (EH) in C++, JAVA, Ruby, Modular-3, C# and other modern programming languages is a great tool for handling errors but unfortunately it is sometimes abused by software developers that do not quite get what exceptions are really for or are just ignorant of possible implementations.

The handling of exceptions must be done at runtime by your compiler/os/virtual machine, since it is generally not possible to predict in advance which handler to transfer control to, identify which exception has been raised, where to perform object cleanup during the propagation of an exception and how much memory to pre-allocate for exception storage. Sometimes the EH runtime mechanism is compiler-specific and a part of compiler’s runtime library. In other cases, support for EH is provided through the operating system or virtual machine.

Common abuses of EH that affect performance include using exceptions as an alternative flow control mechanism (think sophisticated “goto’s” and you got the basic idea of this antipattern”)……. Don’t do that. It will only make the code harder to read. It will also make your code slower to execute since throwing exceptions are generally very expensive operations.

Another less apparent misuse of EH is usage of try-catch(-finally), or similar constructions your language may offer, inside the control flow of hotspots (such as inside time critical loops). Don’t do that, as a the try-catch-finally construction may have overhead even when you won’t expect it.

So why are throwing exceptions expensive and why may the try-catch-finally constructions (or similar) have overhead ? Well, it all depends on the language, the implementation of your VM or compiler (and sometimes on whether you use native code or not if your language allows it). Depending on your environment, just raising one exception can be from 10-100.000 times as slow as alternatively returning a simple return code from the method (in particular depending on if a dynamic registration approach or  a static table approach is used to support control flow transfer). And even if you don’t raise any exception, just having a try-catch-finally in your control flow can also be moderately expensive (but usually only enough to be a real problem inside hotspots).

Specifically, the case of overhead of try-catch-finally constructions when no exceptions occur is difficult to get rid of by compiler & virtual-machine implementers. Few implementations on selected virtual machine and chip architecture got it right and have 100% overhead-free implementations but many impose a overhead just for placing try/catch/finally constructions in your control flow. Basically this is because something like a “linked list” may have to be maintained internally by the compiler or VM each time the control flow enters or exits a try-catch-finally (unless you are luck to run om a fully static table driven implementation of EH).

For much more details about various possible implementations of exception handling and the impact on performance refer to this old thesis of mine here (not up-to-date for virtual machines though).

In conclusion, the morale of the story is:
* Do use exception handling for error handling only (not for control flow).
* Don’t use try-catch-finally constructs inside hot-spots (i.e. loops and such) if it can be avoided. Do the try-catch at a higher level that is called less often.
* If your particular java, c++, ruby, clr … implementation of exception handling on one chip architecture yields excellent performance even when you break the above rules you are just plain lucky. Change the version, vendor or chip architecture and you luck may desert you. Therefore don’t do it

P.S. This post is not about good use EH in general…. But of cause, don’t forget to use exception handling when it is necessary and in particular do NOT ignore exceptions. I have also seen quite a few mistakes in code where exceptions are catched and then ignored. I have even seen senior developers do this in difficult places like event-handlers where one have to go the extra mile to actually handle the error (one way to do this is to create have the main thread manage errors and transfer exceptions to that thread in the event handlers ; but that is long story so I will reserve that for another blog posting)

Notice:

This post is slightly updated version of a posting originally from my old blog at “http://www.mortench.net/blog” which I will shortly retire for good.

Building a Ruby on Rails application in a week

This post will (and perhaps some followups) describe how I did a web application in a week using the web framework Ruby on Rails

My company 41concepts is currently developing a rather large application using Ruby on Rails and during this I had a week where I had to wait for something being delivered by another developer. The application wasn’t really supposed to grow this large, but the feature list just seemed to grow the more we developed.

Because I had this week, I wanted to try to make a small Ruby on Rails application implementing an idea I’ve had for some time now.

I really like my Playstation 3 and if you ask my girlfriend this is also a bit too much sometimes. I generally like adventure and roleplaying games such as the Final Fantasy series and because I hate wasting my time, I usually follow a game walkthrough.

With this application I really wanted to give something back to the game walkthrough community, so I decided to create the site/application “My Walkthroughs“. The main idea is to let users of the site create their game walkthroughs and then share them in different format such as pdf, plain text and html. The idea comes from the fact that almost all walkthroughs are being done in plain ASCII text.

So the application should enable users to build a walkthrough through a web interface and publish it in various formats.

Ok… I had a week for this, so what did I do?

Planning

Well.. I must admit I went straight to implementation. This was a small site and really just a pet project.

Implementation

Design

I’m not really a Photoshop shark (I don’t even have Photoshop), so I found a free template:
http://www.opensourcetemplates.org/templates/preview/1303028527/

For icons I used the FamFamFam silk icon set that is simply astonishing:
http://www.famfamfam.com/

Finally I decided that I wanted some kind of Logo, so I invested in one from Logo Samurai:
http://www.logosamurai.com/

Cool css scalable buttons:
http://monc.se/kitchen/stew/buttons/btn.html

Took the confirmation boxes from:
http://www.templamatic.com/blog.asp?BlogID=18

Got the cool form hints from here
http://www.askthecssguy.com/2007/03/form_field_hints_with_css_and.html

Code

I used a lot of different plug-ins and gems. The fastest code to implement, is code you don’t write :). They are listed below:

Plug-ins

acts_as_bookmarkable
Suddenly I thought it would be a great idea if the user could bookmark other user’s walkthroughs

acts_as_commentable
Of course users should be able to comment each others walkthroughs, it wouldn’t quite get the web 2.0 feel without 🙂

acts_as_list (tutorial)
This is used for lists that should be sortable

acts_as_rated
This was added late, it turned out really nice with this tutorial, adding nice css/ajax support. Also note that all the averages are calculated when the entity is saved and not at render time

attachment_fu (tutorial)
This is used for all uploads and thumbnail generation. It works perfectly together with Amazon S3

auto_complete
Used for completing the game’s name when creating a new walkthrough

restful_authentication (basic use, great screencast)
The standard on authentication

simple_captcha
Used for registrations and other places where spam bots like to date

synch_s3_asset_host
This was used in the Capistrano recipe for synchronizing the public folder with the Amazon S3 asset hosts

Gems

htmldoc (tutorial)
I had to modify it though, when certain PDFs was generated, the gem reported false as return value but left nothing in the error collection. After some investigation I found out there was a bug in how the result from the command-line was parsed and changed line 186 from “case line” to “case line.strip” since there was some \r characters that the code didn’t quite expect. I’m not sure if this is only applicable to me, but now it works

mini_magick
Image manipulation for use with the attachment_fu plugin. This is not as heavy as the full rmagick since it uses the commandline. I wanted to use this with the simple_captcha plugin as well, but it can only modify existing images, not create new ones.

rmagick
Image manipulation for use with the simple_captcha plugin

aws-s3
Provides a nice api for Amazon S3

uuidtools
Used for generating GUIDs. Notice that it couldn’t find the MAC address on my virtual server at Slicehost, so I had to manually give it a MAC address

will_paginate
Great for pagination. If you want to use ajax paging – something we use in another application – this works perfectly

In total the model folder in the application consists of 11 models, 3 observers and 2 mailers. A small and simple app 🙂

The application use REST and search engine optimized urls everywhere which was something I really wanted from the start. Take e.g. the url for a walkthrough:

/walkthroughs/123456-resident_evil_5_ps3/summary

On this page the user gets a summary of the walkthrough and can download the walkthrough in three different formats using the URLs:

/walkthroughs/123456-resident_evil_5_ps3 (no extension => html)
/walkthroughs/123456-resident_evil_5_ps3.txt (plain text – ascii)
/walkthroughs/123456-resident_evil_5_ps3.pdf (well… pdf :))

The code looks at the last edited time of the walkthrough and generates a new one if needed. The output is placed on Amazon S3 and the user is redirected to this. It would be a bit nicer if the application streamed the file from S3 and then directly on to the user, but in order to save server resources, the user is redirected directly to the S3 url (which is currently not that nice).

Deployment

We are currently trying to do deployment on Amazon EC2 for our larger application, but with this one I really wanted to try out the new Passenger (mod_rails) Apache plugin, since it promised real easy rails deployment.

So I bought a slice over at Slicehost and pretty much followed this tutorial to the letter. It worked like a charm.

After this I set up Capistrano according to Slicehost’s own Capistrano documentation.

I also signed up for the Amazon S3 storage service for the walkthrough pdf and txt files, as well as the user avatars (from attachment_fu plugin). Furthermore I used the synch_s3_asset_host to synch the public folders with asset hosts in Amazon S3 during the capistrano deployment.

For tracking I added the site to our Google Analytics account.

Aftermatch

Well, I must admit that not everything was actually done in this one week. The following was added after week one:

• The domain name (I forgot to start this process in the beginning of the week, and it takes a couple of days to process)
• The logo from LogoSamurai took some days with revisions and all
• The bookmarking functionality
• The asset hosts in the capistrano recipe
• Forgot password functionality (don’t know why this slipped in the first place)
• Some minor bugfixes to the pdf and plain text rendering (I will probably keep improving this)
• Caching (this is something I will need to monitor as the site gets traffic)
• I normally write alot of tests, I must admit that I could do better with this app 🙂
• I’ll probably optimize some database indexes along the way
• A nicer system admin area using Active Scaffold or similiar

Price

So what did all this cost me!?

• Hosting @ Slicehost $20 a month first bill was $60
• Logo (LogoSamurai 3 designs unlimited revisions) $147
• A bunch of great rails plugins – FREE (thanks…)

$207 and one week of work (plus some follow up described above).

Now I only need to get some users on the site (the hard part). I might write a bit about the bootstrapping process later.

So go to http://www.mywalkthroughs.com and give it a spin.

Man I love rails…

Shooting yourself in the leg with a bazooka

The colorful title reflects that this posting is about common mistakes done by good but relatively inexperienced software developers. Big mistakes that a developer actually need quite some skills to make. Mistakes that we unfortunately see all too often and we would rather not see much of again (hopefully this blog entry can aid a bit towards that goal):

1. Overly complicated design – Instead of a simple design for a simple project some developers insist on using a impracticable mix of all the newest, fanciest abstractions, design patterns, techniques and advanced language constructs that can possibly be combined in one software solution. All design elements have benefits and drawbacks. Great (experienced) developers know when a particular benefit of an abstraction, pattern, technique or feature outweighs the drawbacks. There is no silver bullet. No design element works well in all situations (just as no rules that you learned in “programming school” are in fact absolute). Developers that get way too eager ends up with one big unmaintainable design mess with the combined drawbacks of all decisions but few if any real benefits remaining…. Remember, a simple design is a beautiful design!
2. Writing too much code – Writing lengthily code with a high maintenance cost by hand when writing such code can be avoided. Much code can be avoided by choosing a more suitable design/architecture, using standard framework/library features (xml serialization is a common example), using techniques such as dynamic reflection or specialized code generation and aspect oriented tools (be careful though).
3. Reinventing the wheel – Some developers think they can write their own code much faster than learning to understand the underlying framework and available libraries. This might in some cases even be true, but when considering overall quality and maintainability (which developers seldom do) reinventing the wheel is almost always a bad idea.
4. Incorrect use of advanced concepts such as multithreading – Some developers use multithreading without the discipline and deep understanding that writing correct, safe multi-threaded code requires. Multithreading can improve the user experience immensely. It is also the answer to scalability nowadays. However, before even considering to use multithreading in your design, make sure to know the theory and features in your environment well. A vague recollection of mutexes and semaphores from school is not good enough. You should also realize that by deciding to use multithreading, you generally need to upgrade on testing, documentation, reviews and quality insurance.