Nicke Andersson

How Software Factories changed my life

I’ve had three Microsoft-related epiphanies at work this year. The first one was when I finally got my head around how we could use Agile development methodologies to reinvent how we make software with the help of MSF Agile. The second one was when I realized just how good SharePoint 2007 is. The third one was Software Factories.

I don’t know if it is fair to claim that Software Factories are in themselves a Microsoft-specific concept, but MS sure are driving the field forward at the moment. I first saw Jack Greenfield do a presentation on the at TechEd 2005, and while I did find the topic interesting there simply wasn’t enough of an incentive available to actually start using them. Now there is. Guidance automation in Visual Studio 2005 has become mature enough for it to be possible to produce truly useful factories, and there are a couple of really interesting ones out there. Since this spring we’ve been using the Smart Client Software Factory, and it looks like it is changing the way we make Windows Forms applications.

To me factories are about decreasing the amount of “plumbing” code you have to write in an application, and a way to make sure that similar applications are structured and built in the same way. I once saw Rockford Lhotka do presentation in which he mentioned that about 80% of the code that gets written as part of an application is not about producing actual value to the customer (i.e. those bits that the customer feels they are paying for), instead it goes into frameworks and similar things that are required before you actually can start writing code that implements any business-related functionality. You may argue that 80% seems high, I feel it is a pretty good average. I’ve been working on projects where it has at least felt that 95% went into plumbing and only 5 percent actually gave users any value, and on the other hand I’ve seen project where good choice of tools have allowed the developers to spend half their time on the actual functionality. Be as it may, one can argue that anything about 0% is a waste of time and the customer’s money. While I don’t believe we can reach that 0% anytime soon I feel that factories might be a good way to reach 50% or even go below that.

From our point of view the whole thing started with the Composite UI Application Block (CAB), which we started doing work with little over a year ago. The CAB allows you to create Windows application that are composed of discrete parts (Smart Parts) you can use in an MVC pattern to build applications. While it is really a framework and not a factory it is what got us really interested in the factory space. We used to have a great deal of variance in how we architected and developed Windows applications, and the CAB helped give us a reusable structure for that. It was generally felt that the slight overhead introduced by the CAB was well worth it, since it forced the architecture and structure of our solutions towards a similar and more easily understandable model. However, there were still differences in the architecture of the applications and how the CAB was being used. We found that a framework like CAB simply left too many things open and allowed the developers too much freedom in how to use it efficiently. This freedom didn’t lead to the ability to create solutions of higher quality or solutions that where more adapted to specific customer needs; it only led to different ways of achieving the same thing. In essence we were still reinventing the wheel, albeit on a smaller scale.

Last spring the Smart Client Software Factory changed this. The way GAT, GAX and the factory way of thinking gives the developers access to context-sensitive processes (recipes) for achieving pre-defined results does away with the unnecessary differences we are seeing in the architecture and structure of our applications when using only the CAB. I am expecting great things from the Smart Client factory. It seems that it is reducing the need for architecting the client side of the solution, and results in a lot more similar solutions than what we have so far been achieving. With limited architect-resources this is great because it means they can focus on actually designing new stuff, not just doing plumbing. For the developers this is great because it results in applications with similar structures, which makes it easier to get into a project developed by someone else.

Over the summer we also started using another software factory, the Service Factory. This promises to do for the server side what the Smart Client Factory did for the client side. Our experiences are so far limited, but since we’ve been seeing the exact same problems on the server side as we did on the client side, with great variations in how we build Web Services, I’m hope we’ll be seeing some great benefits from this factory as well. I’ll post more on it later.

To summarize, in many cases our architects have been spending most of their time on the plumbing, not on the business functionality, which means that there is not enough focus on the design of the bits that the customer actually see and feels is giving him value. Software Factories will help us reduce the architectural and development work that goes into the background bits that shouldn’t have to be rebuilt every time, and that the customer is not too keen on paying for. While I do not have any actual numbers on the increase in efficiency or on the decrease in complexity I have a strong gut feeling that they are substantial. If I come up with something I will post it here.

TechEd 2006: On the importance of working within set requirements

Requirements and architecture

At the TechEd architecture preconf Ron Jacobs talked about what requirements are, and how they help define the goal for what you are trying to achieve. He drew a parallel to playing soccer on a field with no goal posts, no lines, no constraints whatsoever. Just starting to play, and figuring out the goal positions and so forth as you go along. Obviously this would simply not make any sense. However, people seem to believe it does so when making software. Especially from an architect’s point of view it is important to understand what we’re trying to achieve at least on a hight level before we can decide how to design it. If the architecture is broken in a sufficiently bad way it might cause the whole system to go back to the drawing board (suck on that, XP!). And how might this happen? By not understanding the business problems well enough and screwing up the requirements gathering and therefore designing an architecture to solve the wrong problems. Therefore a certain amount of up-front design is necessary.

But often enough it also happens that architects and developers get too worried about the requirements, and they let this block them from designing the solution. They start worrying about every little bit of unknown area, and refuse to touch it until it’s been specified. So this can really hit two different extremes. Either you do no design because you don’t know the requirements and just start coding, or you get anxious about not knowing everything so you produce nothing.

Learning to balance this line is key to becoming a good developer or architect, IMHO. You need to get enough confidence in yourself to trust that you will find a solution to problems that come up and just get started on designing those parts that can be designed. Focus first on those areas that you do understand, and use that as a way to get going and to create something that you can show the customer and get feedback on. This way the rest of those requirement will start to appear, and you can incrementally improve your design. Experience will teach you which types of things you need to know up front and which kinds of things it is ok to defer until you have more information.

Getting people to commit to the project

One of the really key things we’re trying to improve in our organization right now is to get everyone in a project to have a shared understanding of what the project is supposed to achieve. This sound like something really obvious, but still it is one of the most simple things that companies fail to get right. I’ve worked on many project where management has thought that everyone involved knows the goal of the project, but in reality none of the developers really do. Note that this also includes times when I have been part of the management for these projects.

In a situation like this it’s impossible for them to make good decisions about how to develop something or on how much time to spend on it, simply because they lack context. And that results in a lack of commitment to the project (sometimes I think that developers believe that “commit” is what you do when you check in code to version control). Without commitment and context you will be stuck with a bunch of people that can make no decisions on their own and can only perform exactly the tasks you give them. But by getting them more involved and by giving them context you allow them to contribute to the project beyond simply performing precisely given tasks. And you are sure to benefit from it.

All it takes to sow that first seed of project commitment in all the project members is to create a good vision statement and sit down with the whole team an talk through it. Discuss it and make sure everyone understands it. After this they will feel a lot more connected to the project because they have context. And the commitment will follow.

TechEd 2006: Defining the role of a Software Architect

At work we’ve been looking at how to improve our processes for quite some time. Essentially, our goal is to increase both customer and worker happiness by introducing a new methodology for how we work. We’re basing it on MSF Agile, Scrum and to some extent eXtreme Programming.

One of the things I’ve been struggling with getting right in our methodology is defining the different roles in it. MSF Agile provides a base for this, but there are some things in there that I’m not quite happy with. One of the most difficult areas is the difference between a business analyst and an architect, and how much both are required to understand of and take part in defining the business requirements.

In MSF Agile the Architect’s role is described as “The architect’s main goal is to ensure success of the project by designing the foundations of the application. This includes defining both the organizational structure of the application and the physical structure of its deployment.”

The Business Analyst on the other hand is defined as “The business analyst’s main goal is to define the business opportunity in the form of a solution that will provide value to the customer. The business analyst works with the customers and other stakeholders to understand and translate their needs into persona definitions, scenarios, and quality of service requirements that the development team will use to build the application.”

The problem I run into in real life is that for the architect to be able to design a good solution he really needs to understand the business requirements, but he might not be the person who is best suited for eliciting those requirements. The business analyst, on the other hand, rarely knows enough about software design to sketch out a correct solution. Some of them do try, but rarely with good results. And who should actually be responsible for UI design? MSF Agile says it is the business analyst, but we’ve also observed situations where that might tend to guide the solution in a direction where the UI mimics any existing solution as far as possible. Some times that is a good thing, sometimes not. On the other hand, some architects are seriously unsuitable for UI advocacy, so where should it be? I strongly believe an architect needs to have an understanding of the business, and act as some kind of bridge between the technical and the business side, but I have found it hard to put this in words that clearly define where the responsibility of the architect ends and that of the business analyst begins.

TechEd 2006 Pre-conference session on architecture

I didn’t actually know what to expect from this session. There actually weren’t that many preconf sessions that seemed interesting, because they all appeared to be pretty entry-level. I picked this one based partially on the people running it but also because the field of software architecture is so vast that you can always learn something new. It turned out that this session was valuable to me mostly from the point of view of how to organize the architectural activities in our company and how to train people to become architects.

In the preconf Ron Jacobs talked about what an architect is and what he should be doing. He structured that around three different areas, analyzing different key aspect of an architect.

The architect as an explorer

This one I identified with a great deal. He made comparisons to Vasco da Gama and Christopher Columbus; men with a vision of something that could be done to solve a problem, and with the guts to believe in that vision despite it flying in the face of what was considered accepted knowledge at the time. The architect has to have the vision to see which of the emerging technologies, tools or practices will become commonplace and which can be used to solve a specific business need. He needs to be able to convince people of their suitability, and in the end be accountable for what happens if they are used.

The architect as an advocate

This is about the architect understanding the business domain. About listening to the customer and trying to understand what it is that really is required in order to provide a solution to their needs. Learning the business domain is necessary in order to design a system that is correct for the specific situation. This involves observing the domain in order for sufficient knowledge to be obtained.

Here is one area where I might slightly disagree with Jacobs. His view on this included quite a lot of things that might belong in the domain of the business analyst as well. As I mentioned earlier the architect needs to have the ability to understand the business, but he might not be able to elicit them from the customer efficiently.

The advocate-segment also contained a piece on strategy. Strategic understanding and thinking is required on both the technical (what tools/technologies can solve a particular problem) and business (how can the business develop over time, and which sections of the software are relevant from that point of view) levels, and again the architect and the business roles clash somewhat.

The architect as a designer

Designing the correct solution involves understanding the engineering constraints involved, understand the user needs for the solution, and doing so through a solution that is both beautiful and elegant. In a way this ties together the two previous areas, where you combine a vision for how to solve something with a thorough understanding of what you are trying to solve.

The architect vs. the business analyst

A lot of the stuff in the preconf was interesting and was stuff I agreed with, but didn’t give a clear answer to my question of what the separation of the roles of the architect vs the business analyst is. But when looking at what architecture is an answer started to emerge. Jacobs said that “Architecture is an idea, a plan of the solution that will be built”. And to reach the level of an architect was described as “Becoming an architect is a journey towards becoming one who dreams of the solution rather than the one who builds it”. Pretty inspiring, but how should it be interpreted?

In my view this is where the answer to my question lies. The role of the architect is to envision the correct solution to a given problem. The role of the business analyst is to make sure that we are trying to solve the correct problem, and to define what that problem actually is. That means that eliciting requirements and defining the vision for the project should be the responsibility of the business analyst. The architect needs to have the ability to understand and care about the information given to him by the business analyst, but he does not have to know how to extract these from the customer.

So in the end, should we perhaps think of the Business Analyst as a proxy for the customer? A way to achieve what XP calls an on-site customer, i.e. a person who is available all the time to answer questions about what the customer needs and how it should work? And perhaps as a more coherently packaged version of the customer, one that is actually able to express what he needs, instead of what he thinks the correct solution to these needs are. Because in the end, the reason for introducing a business analyst role is because the customer can not himself express the business problem to the architect or the developers on such a level that makes it possible for them to come up with and develop a solution that solves the customer’s problem. Much too often the customer expresses a need that might not be the actual need, but a symptom of it. This actual need is what the business analyst needs to determine, and to communicate to the architect and the developers.

Seems good. Let’s see how long this lasts until I change my mind about it.

One last thing: I feel it is very important that the architect also writes some of the code for the system. If he isn’t involved in the development there is a significant risk that he becomes so detached from the actual day-to-day process of developing the solution within the given constraints, and then dreaming of the solution becomes just that, a dream that can not be translated into practice. For this reason I think it becomes difficult to combine the roles of the business analyst and the architect, especially if the person in this role comes from the business analyst side. He will most likely not be able to contribute to the code on a level that corresponds to what I wrote here.