Sunday, April 8, 2007

Offshoring Pitfalls in the Trenches

If you are like me (corporate architect/software dev. manager working for a large company), then you probably could not avoid to be sucked into the "outsourcing black hole". Let me share my experiences on the subject.

In general, the higher level management of a corporation gets this "we need to be as efficient as X" bug, and then goes on to setup corporate policies for engagement with outsourcing partners. Normally (at least in my experience), the decisions come down to people like me to work with a team of developers in another country. This team would be pre-selected by management of the vendor itself, and often times, the qualifications of the offshore team members might not exactly match your projects needs. In addition to that, there could be a whole host of other impediments that you do not normally have when working with a local group:
  • Pre-selected team
  • Cultural barriers
  • Language barriers (not the same as cultural)
  • Time difference
  • Lack of good communications (IM, screen sharing, phone, VPN, Wiki, etc.)
  • Lack of a development ecosystem (VCS, bug tracking system, continuous build, integration env.)
  • Wrong contracting agreement (time boxing)
My first account with outsourcing was around 2003, when I was asked to deliver an internal project, and was given a team of four developers and an on-shore liaison from an approved vendor in India. I bravely charged ahead without thinking too much that the project actually was not setup right. This first project went exactly according to the Murphy's Law: "If anything can go wrong, it will..". The bullet list above is not a coincidence, as it came directly from my experience on this project. Every aspect of this project has gone wrong. To begin with, I probably should provide a technology overview of the project. The task was to build an internal database of UML models. This was a web-based J2EE application, which stored UML models in a native XML database. The architecture was not that complicated, but there was a lot of work with new technologies (native XML database, XPath, XQuery, EJBs, generated JavaScript, etc.).

Let's start describing every problem we faced. First off, we were assigned a pre-selected team for this project. Well, this is a strange world. When we hire full time employees, or even consultants to do the job, we go far and above in the interviewing process to ensure that the candidate is indeed qualified to do the job. On the other hand, when we work off-shoring vendors, somehow we are handed down teams that were put together without any input from the actual architecture and management of the project - just because they were provided by an "approved vendor". This, of course allows vendors abuse the situation and allocate junior team members to projects in order to cross-train them. When the project started, and I was able to interview some of the developers in the off-shore team, I was shocked to find out that they were all in their early twenties (nothing wrong with that, as long as there is experience) and only one team member had a cursory knowledge of UML(heard of it). None of the developers possessed any of the knowledge of technologies necessary to complete this project. In other words, there was a great mismatch between the experience of the team members and the project demands, so we started with the introduction to UML :).

Second, I quickly realized that there were cultural differences between our working style and that of the off-shore team. It came in a form of full agreements during the design and architecture discussions. When we started to receive deliverables, we realized that they had no resemblance to what was discussed and "agreed" upon.

Time difference. The time difference between Chicago and Bangalore (location of the off-shore team) is 11 hours, which pretty much prevents any live communications within the normal work ours. This translated into a lot of extra hours spent by US team members as well as the vendors'.

Lack of good communications. This is an area where our own management (either due to lack of understanding, or lack of attention) failed miserably. Software development is a highly social process, and requires people to be in constant communications. The only communication channels we had on this project were phone and e-mail. The following communication channels are a must for every project and especially
for an off-shoring one: instant messaging, screen sharing, phone, VPN, Wiki. Unfortunately, we had none of these, except for mail and phone. This resulted in extremely inefficient communications often delayed by days, with screen shots included. A simple e-mail "question - answer - confirmation" could take up to four days (did management factor this into the bottom line of off-shoring? ). We did not get IM, VPN, and shared Wiki because of our own managements' security concerns (which were superficial of course, as all code was sent back and fourth by e-mail!).

The off shore team used their local VSS as a source control system, while we used CVS. Each morning we would have an e-mail code dump, and quite often we had a very difficult time checking it in, merging, compiling, and running. Sometimes this activity alone would devour a good half a day.

The last but not the least, the contract agreement. I'm a firm believer that an agreement with any vendor must be based on a actual deliverables, and never just time - boxed, as it was on this project. The entire project was scheduled for nine months, but the off-shore contract (for reasons unknown to me) was only for fife. This meant that the off-shore team was not in any position to be responsible for just about anything they delivered. They would not be on the project long enough to see it go live.

All of the above resulted in the fact that about of 90% of code delivered by the off-shore team was discarded soon after the off-shore team rolled of the project, and the system was completed by a local team via a heroic effort. Of course, the management deemed it a successful use of the off-shore resources - how ironic!

Hey, you have been patient enough to read up until this point – my hat is off to you :). You might ask what is the moral of this story? The bottom line is that modern software development is a very hard process, and it should be approached very seriously. Working with off-shore vendors makes it much more difficult and sometimes creates problems you might not even think of when working with a local team. In addition to that, off-shoring work creates so many inefficiencies, which need to be factored in when deciding whether it is going to be good for your organization.
In other words, before an organization embarks on an off-shoring initiative, it needs to get it's ducks in a row, by providing a more productive environment, all possible better communication channels, better structured contracting agreement, etc - I think you get the point.

So, is the outlook so bleak? Not really, if it is approached properly. I will tell the success story in the next posting.
Any thoughts?

No comments: