About Agility

Last week, the OutSystems team attended the Enterprise Software Development Conference (ESDC) in San Mateo California. This is the first year for this show and, as Alan Zeichick notes, it takes up where the old SD West conference left off.  As gold sponsors of the show, we got to both attend the sessions and talk to the conference attendees at the OutSystems booth. I just wanted to share a few highlights & take-aways from the show:

On Monday afternoon I was pleased to take part in the MythBlasters session and offered the crowd two myths to choose from : #1 "You Cannot Scale Agile in Enterprise IT shops" or #2 "It is less risky and more cost effective to buy a package than to do custom development."

The audience chose to 'blast' the Buy vs. Build myth.  My premise for blasting this myth is based on the success of our customers in building new custom applications for things like customer relationship management, call center management, various HR applications like recruiting, expense management, company portals, etc.  I shared the top three ingredients for successfully building vs. buying:  Speed to develop, rapid change and business involvement at each step.  Of course, our customers accomplish this by using the Agile Platform and an agile methodology.

I think I'll keep Myth #1 for another blog post - but what are you seeing on Buy vs. Build myth? Do you agree that building is a better way to go for many apps where there is a package?
 
Another highlight: Kent Beck's keynote on "Responsive Design: Efficiency Through Safety."  This was the first time I had heard Kent speak.  He started off by referencing Ed Yourdon's work on Systems Design and how it led him to try and distill his own working process for design.  This was the premise for his presentation.  My take-away was that no matter what you do, your design will change. I think we all accept this as fact - especially for application software.  Kent then explained his techniques to reduce the risk when making design changes.  For each of his examples I found myself thinking 'This is not really a problem with the Agile Platform because the TrueChange™ engine will keep you from breaking stuff you did not intend to break, allowing you to move very fast with little risk."  If you are hand-coding, then Kent's four techniques (as described here by Alan Zeichick) to reduce risk when making change is great advice, but why do that if you don't have to?  BTW, I think Kent would love the Agile Platform.

Since this was a new conference we've been asked who attended and why. Here are some thoughts from the OutSystems team:

• The crowd we met included CIOs, IT directors, Enterprise Architects, project managers, analysts and software engineers. Interestingly, a large proportion of the conference attendees represented Enterprise IT (rather than just the ISV folks who usually attend Agile conferences) ...I'd say Agile is definitely going (or has gone) mainstream!
• Many people were there to learn more about Agile and find out if it fit into their world.  They realize that Agile really makes sense, and that they should implement it, but were trying to learn the best ways to do this - both with or without tools to help them.
• The general feeling was that Agile is the growing wave of the present and the future - and many felt they needed to add Agile to their resumes, because it is rapidly growing in popularity.  One attendee stated that he felt Agile was the only "fad" methodology that would last because it actually works and makes sense. 
• A few folks told us they were already using some Agile practices, but just had never labeled them as such and would like to get their organizations to officially adopt Agile.

 

As a follow-up to our previous post; here is the third and final interview question from our conversation with Lawrence Wilkes, Director and Principal Consultant with Everware-CBDi, on how IT departments are addressing their legacy systems: 

Q.3 What are the top three mistakes people make when addressing their legacy?

LW: I might have hoped that post Y2K, organizations would have more actively sought to avoid building yet more legacy systems. However, time and again we see organizations doing the following:

1. Creating another legacy. They fail to understand why they have legacy systems in the first place, and just rush into creating another one. They don't understand the requirement for on-going change, nor does the business encourage them to think about it. This could be due to:  a lack of process - i.e. they don't understand how to facilitate change at each stage of the life-cycle; or, stakeholders unwillingness or inability to express requirements for change.
   
   
2. Not designing for agility. Agility isn't just a process - as in agile development - agility is also an analysis and design requirement. Ensuring a new solution is better able to respond to change than the legacy systems it is replacing requires appropriate techniques throughout the life cycle. It isn't just a matter of creating a more agile implementation built with finer-grained components and services, but also understanding the business's requirements for agility and identifying those components and services in a way that aligns with that. Conversely, you don't need to over-compensate and waste time providing too much agility where it is not needed - although, that is always going to be difficult to predict...
   
   
3. Not cleanly separating the legacy from the new solution. The two become entwined storing up an even bigger legacy problem for the future. For example, a proper service architecture is not developed that is truly independent of the legacy systems, and so the new solution remains tightly coupled to the legacy. Even though they may use loosely coupled technology such as web services, their content and behavior is just a direct reflection of the legacy system.

What do you think? Do you have any do's and don'ts on addressing legacy that you'd like to share with the community?

Lawrence Wilkes is a Director and Principal Consultant at Everware-CBDI. Lawrence is a frequent speaker, author and consultant on best practice in SOA, Application Modernization and Enterprise Architecture. Via CBDI Forum, the Everware-CBDI research capability and portal, Lawrence has led the development of the CBDI-SAE methodology and supporting Knowledgebase, which is used by both end-user organizations and system integrators around the world. Lawrence has an extensive background both within end-user and vendor organizations having worked both in the business and IT side, which brings particular insight into business/IT convergence.

As a follow-up to our previous post; here is the second interview question from our conversation with Lawrence Wilkes, Director and Principal Consultant with Everware-CBDi, on how IT departments are addressing their legacy systems: 

Q.2  Is there a role for model-based tools like the Agile Platform in legacy modernization?

LW: Although a large part of the functionality may be based on those that already exist in legacy systems, there is still a need to design and assemble the new solution that is typically focused on supporting a new business process, channel or product.  This requires a layer of new development that 'wraps' the legacy systems, and the capabilities created out of the legacy still need to be integrated into that new solution. For example, capabilities in the legacy systems might become service providers to the new solutions.

At the same time, new capabilities are likely to be required since legacy modernization usually entails more than the need to re-skin, or re-process the legacy system. There will be new business requirements that the legacy system didn't address.  Hence, model-based tools can be very useful in terms of:

a.      Designing and implementing the new process and UI layers
b.      Creating the services that wrap the legacy capabilities
c.      Creating both new business functionality, and new components & services to support them
d.      And finally, assembling it all together into the new solution.

Have you used a model-based tool to help with your legacy? What have been the benefits?

In the next and final question, Lawrence offers the top three mistakes people make with their legacy modernization efforts.  

Lawrence Wilkes is a Director and Principal Consultant at Everware-CBDI. Lawrence is a frequent speaker, author and consultant on best practice in SOA, Application Modernization and Enterprise Architecture. Via CBDI Forum, the Everware-CBDI research capability and portal, Lawrence has led the development of the CBDI-SAE methodology and supporting Knowledgebase, which is used by both end-user organizations and system integrators around the world. Lawrence has an extensive background both within end-user and vendor organizations having worked both in the business and IT side, which brings particular insight into business/IT convergence.

We recently met with Lawrence Wilkes, Director and Principal Consultant with Everware-CBDi, to talk about how IT departments are addressing their legacy systems. 

We will post Lawrence's response to the following three questions over our next few blog posts:

• What are the top three things you've seen people do to successfully address the competing demands of new build vs. legacy modernization with limited resources?

• Is there a role for model-based tools like the Agile Platform in legacy modernization?

• What are the top three mistakes people make when addressing their legacy?

#1   What are the top 3 things you've seen IT do to successfully address the competing demands of new build vs. legacy modernization when faced with limited or shrinking resources?

LW: My experience is enterprises that are really successful tend to buck the trend. They are more radical and don't just "follow the crowd". Rather, there are usually one or two key people who have an architectural vision that breaks new ground - at least from their organization's perspective. Typically they will:
 

1. Combine application modernization and business improvement programs. They get business stakeholders involved in understanding the options for modernizing, both in terms of business needs and the applications that allow a more radical approach to be taken to requirements.  They avoid the knee-jerk reaction of buying a package or simply outsourcing the entire problem because they are seen as expedient options that, however, often results in something that is no better able to respond to change than the legacy it is replacing. Instead, they are keen to search out economies of scale and cost; capitalizing on existing investments in applications where possible, rather than just throwing them away, and break-down organizational and application silos to create solutions that are genuinely more flexible.
   
    
2. Keep business design and architecture in-house; create strong architecture and Centers of Excellence (CoE) roles that are able to exert strong architectural governance over outsourcing and delivery. They do not allow the outsource parties to gain control, nor do they lose knowledge of their business, applications and environments.
   
   
3. They use agile approaches but are not slavishly following the conventional "agile" manifesto - agile needs to be used within an architected, governed framework, not ad hoc. They establish a formal reference model and framework, which allows rapid delivery and governance of products, delivered by an agile process.
   

What steps has your organization taken to successfully address legacy systems while delivering new systems?

In our next post we will address Q.2 and the role of model-based development tools play with legacy modernization.

Lawrence Wilkes is a Director and Principal Consultant at Everware-CBDI. Lawrence is a frequent speaker, author and consultant on best practice in SOA, Application Modernization and Enterprise Architecture. Via CBDI Forum, the Everware-CBDI research capability and portal, Lawrence has led the development of the CBDI-SAE methodology and supporting Knowledgebase, which is used by both end-user organizations and system integrators around the world. Lawrence has an extensive background both within end-user and vendor organizations having worked both in the business and IT side, which brings particular insight into business/IT convergence.

There's a lot of blogging about agile estimating and planning but most, if not all, that I've come across only pertain to after the project has been started.  Further, what is being estimated and planned is for the iterations.  So how does one define budget for the agile project itself, specifically a web business application project?  I mean, the cost of the hardware, 3rd party software, facilities, etc. everything except the actual software development cost can be fairly straight forward to budget. 

But what about the software development cost?  Using conventional methods or techniques is certainly an option but they do not account for the fact that agile projects tend to cost less when compared to similar projects.  So, perhaps this is not the way to go. Besides, business owners want to know what they're getting for their money and how much it is going to cost them. 

At OutSystems, we solved this by taking the following 3 steps:

1. First, we work with the business to gather and transform high-level requirements into user stories.  Stories highlight features that detail what the user wants to do (e.g., create, list, search) to specific objects (e.g.customer, store, loan). 
 
2. These features are patterns of implementation; it is the patterns that are estimated in hours based on years of experience delivering agile projects successfully and this experience is encapsulated in our OutSystems Agile Network Sizing (ANS) tool (BTW - you can see a demo video here).  The tool allows us to set various influencing variables such as number of developers, iteration size, number of user profiles, etc. that enable us to adjust the estimates based on complexity levels.  There are always unknowns during funding and these variables help provide a means to shore up those that may require more attention during the project.
3. Given the resources, we are able to calculate what we call the ideal project timeline, complete with sprints and a target go-live date.

You can probably use story points to achieve the same thing assuming you have a common baseline and have equated points to some level of effort which, in turn, equates to dollars. 

What if you get funded without this process?

Let's say your project gets funded but you did not use patterns or story points to determine your budget.  You start the project and gather your user stories, prioritize them and the size them.  You conduct your sprints and over the course of say 3 iterations, calculate your average velocity and capacity.  Having your averages, you can calculate the average cost of each iteration and thus the project itself; also, potentially determining when you will run out of budget.  Unfortunately, when using story points and a burndown chart, you still cannot predict with enough certainty when the project will be done. To most stakeholders, stating an approximation or a moving target date is just not acceptable even if you stress the benefits of going agile - early delivery, handling evolving requirements, software that meets specifications, early risk mitigation, etc. 

So what do you tell them?  How do you accurately estimate the time and cost it will take and then convince the stakeholders that the project will be done on time and within budget?   Do you give odds when answering when the project will be done?  What do you do if you run out of budget?  Are you gambling with your budget?

Here's the process we've developed:

a) We bootstrap the stories, budget, timeline, sprint definitions from the sizing tool into our project management tool - the OutSystems Agile Network Project (ANP).  The bootstrap process calculates the budget for each iteration based on the overall budget.

 
b) Each iteration is allocated a budget for development, technical debt, change requests, testing, and sprint review.

 
c) Once the information is in ANP, we review the stories with the business, prioritize them and even negotiate what goes in an iteration, just like any other agile project.

 
d) We also break down the user stories into work-items and estimate them (in hours) in conjunction with the developer(s) assigned to complete them.  It is the work-items that we commit to deliver for the sprint.  This helps us ensure that we do not exceed the iteration's budget. 

In summary, estimating based on patterns enables us to define the project budget.  Resource estimates and other influencing variables coupled with the budget enables us to provide a project timeline and target go-live date.  This information allows us to manage the project at the sprint level ensuring that we are on target while not exceeding our budget.

What method(s) do you use for estimating project budget and timeline?  What method(s) do you use to ensure you deliver within budget and on time?

Today we released a new version of the OutSystems Agile Platform - 5.0! In this version we're taking agile a step further; not only are we supporting the entire application lifecycle management for web applications, we also added support for IT teams to rapidly develop business processes using agile methodologies.

Traditionally, business process development was done at a different pace using different tools than IT used for application development. However, one of the biggest challenges facing the business process world is the integration of business processes with applications; which meant one of them was always waiting for the other.  And, in the case of our customers who already use the Agile Platform, web application development was happening faster than business processes development.

With version 5.0 of the Agile Platform, we have closed that gap! Using the new Business Process Technology capabilities of the platform, IT teams can develop business processes totally integrated with web applications in an agile manner. All artifacts that the Agile Platform provides for Web Application development - like TrueChange technology, 1 Click-Publishing, Real Time Monitoring, and so on - are also available for business process development.

To develop this new capability, the OutSystems R&D team partnered with one of our customers, Van Ameyde, to design and implement this capability. Van Ameyde uses business processes intensively for insurance claims processing and has very heavy change demands for those processes. Customer participation has been key to the development of the new 5.0 functionality, and we believe that it led to a pragmatic implementation of Business Process Technology that will allow IT teams to fulfill the needs of the business from a process perspective, as fast as they have been doing for web applications with the Agile Platform.

Along with Business Process Technology, version 5.0 includes many other improvements that will make developers a lot more productive. If you're already using the Agile Platform, check the videos of some of the improvements we made to the platform. If you want to give it a try for yourself, the best thing to do is download the (free) Community Edition and try out the new capabilities of the platform.

Of course, we could only release such a great version of the Platform with a great team! We had a lot of help from a lot of people, in particular from our Beta customers who provided such excellent early feedback, and a special thanks goes to the 5.0 team that delivered such a great product! Version 5.0 of the Agile Platform will definitely raise the bar for agile.  Give it a try, the 5.0 development team promises you that it ROCKS!

We were all very excited to hear one of our customers, Max Rayner, speak at the recent ALM Expo on how he and his team built and delivered the www.fly.com system using an agile approach. If you're not familiar with it, this is an internet application that's openly available and was built using Agile methodologies, SCRUM techniques and an Agile ALM toolset.
 
Max is the CTO of Travelzoo - a travel publisher with 18 million subscribers and fly.com is an online app that helps you find the exact match to your air travel needs. During the webcast Max discussed the problem space, their agile approach, the innovative metasearch engine, how they managed a distributed team, challenges, key learnings and reasons for their success.With all that said, I can't really do the webcast justice in this blog as there was SO much great information shared - but you can:

1) download the webcast presentation slides.pdf and listen to the podcast

2) view the full recording of the webcast here (unfortunately, you will need to register but its free and painless.)

3) or, you can download the podcast of the presentation here and listen in your own time.

BTW, the recordings include a great Q&A session that I would highly recommend listening to!  Here are some stats Max shared on the fly.com agile project:

The challenge we see in many Enterprise IT shops is that it is hard to get everyone across the business who touches application development 'on board' with an agile approach.  In this blog post I want to share how the CIO of one of our customers has set guidelines which help drive agile project delivery across the business.
 
The customer is a large Portuguese food distribution and consumer goods manufacturing company, with an international presence.  With nearly 25,000 employees they are used to having huge IT projects, involving multiple departments with complex requirements, large budgets and long timelines. 

Before agile, many of these projects were delivered late, over budget and in some cases actually failed.  In 2005 they became an OutSystems customer; embracing the Agile Platform and an agile methodology. As they matured in their agile practices, the CIO saw the value and today they operate under the CIO's mandate that only projects scoped with a timeline of less than three months and a budget under 300K will be approved.

What happened before the CIO mandate was put in place?  Here's what they told us:

1. They would spend two to three months in meetings, aligning all opinions in order to create a huge requirements document;

2. When they finally started developing the systems the business team had disappeared and in most cases forgotten about the project;

3. By the time the project was ready to be tested the key users had changed, the business had changed and the project delivery team immediately entered into a lengthy negotiation phase to reconcile what was delivered versus what the business really needed.

This customer states that with the agile methodology and their new CIO mandate they improved the success rate of their projects in several ways:

1. The business team is more motivated and involved as they are able to see how the projects are progressing on a regular basis;

2. The business gets to be more responsible for the decisions that shape the project direction because they see and constantly test the application;

3. The business and IT avoid the costly, wasteful exercise of building complex requirements documents because they now fully realize that they can never document every detail in a specification;

4. From very early on in the project, IT can see if the project is really what the company needs and identify any mis-matches quickly to reduce the amount of time, dollars and resources that might be wasted.

5. Even for big projects, Agile methodology is used - and forces the team to split the project into phases. This exercise divides the scope into smaller, manageable projects with incremental releases and decreased risk.

So, both for this CIO and many others, we are finding that an agile approach to application development really helps increase project success rate and reduce risks.  Even in a company like this one that has complex application needs, lots of departments and bureaucracy - agile really works! 

What are the technical leaders in your company doing to help drive agile project delivery?  Or, what would you like to see them doing? 

Share their rules, guidelines and mandates along with your ideas!