Benefits of Agile methodologies have been realized in software product development as this approach attempts to break the complexity of software development by frequently(regularly) releasing working software.
As this model aligns itself with the ever changing software requirements, these methodologies are increasingly adopted in software engineering. There are various methodologies which fall under Agile umbrella, such as, DSDM, ASD, SCRUM, Extreme Programming. This article will refer to Scrum while discussing ETL development in conjuntion with Agile.
The challenge for Scrum master and product owners is to define the scope of a sprint, which produces a working model at the end of it. ETL development comprises of logical pieces of components (ETL jobs), which makes it convenient to plan a number of ETL jobs in a sprint.
A sprint is normally of two or three weeks of duration. On the basis of the number of resources available at hand, the scrum master can plan a sprint which is realistic and achievable. This is not being suggested here that Scrum or Agile can totally replace all phases, e.g. requirements for ETL normally flows from the “upstream” systems, but build phase is an area which can definitely be exploited.
Many would see this as a challenge that how one can follow the various approaches(SDLC and Agile) at the same time. What is being advocated here is that these methodologies are not intended to replace another, rather they should be used judiciously to leverage the pluses available with them.
A scenario: A software vendor wins a contract (may be Fixed price) to deliver an ETL project. They follow the traditional approach for requirements and high level design phases. During the build phase, they can internally use the Agile principles to drive the ETL development. The advantages are:
1. better planning as planning for less.
2. better estimation.
3. better use of lessons learnt, as lessons learnt from one sprint can be applied to another.
4. minimizing the risks as the working software is rolled out from early stages, providing an opportunity to early identification of errors.
5. A chance to get customer feedback early.
The planning for scheduling software can also be left to later sprints as there would be useful information available in form of jobs and dependencies which would again help in planning more effectively.
in a T&M model, using Agile for ETL development is even better as the customer can find it easy to relate with number of jobs available in a sprint. This in turn will help in early feedback, confidence building of customer as he sees the product much earlier and can provide invaluable feebback.
The Agile methodologies bring the team together and provide a platform to enagage the customer. Combine with the ease of use in ETL development and you can develop a project management model which is effective.