A Brief Explanation of Why Processes Don’t Scale

I received a comment on an earlier post from Guido Schneider taking some exception to a Scrum example I used. His thoughtful response got me thinking, and the thinking turned my response to his comment into a whole blog post.

Guido makes the point that in the real world, you often have Scrum teams that are part of larger projects. The result is that we try to scale our processes by creating things like scrum-of-scrums (essentially doubling-down on the practice). We need to create high level iteration plans that look at hundreds of work items across multiple Scrum teams. Sometimes we need to roll up utilization numbers and progress through WI hierarchies, for example. The current Scrum template in RTC doesn’t solve all of these scrum-of-scrum, project, or portfolio level issues.

One reason for this is because Scrum isn’t a project level practice, it’s a team practice (as Guido indicates). But organizations often try to scale Agile techniques like Scrum to larger teams and projects, placing a demand on an RTC Scrum template to do more than just Scrum.

Is it Possible to Scale a Process to a Larger Team Size?

So my concern is, how much can Scrum (or other Agile process) be scaled to larger organizations? Are we just doing what we’ve always done with rolling up the project management information, giving it a new name (Agile), and expecting different results? What evidence is there that you can scale any or all Agile techniques?

Interestingly, Agile was not initially defined as a set of techniques that only small teams can use. But almost all the work that’s been done inventing new Agile techniques focuses on the work of small teams. For example, Scrum uses developers and product owners, not project managers, to prioritize change requests. But larger projects make the project manager responsible for prioritization.

I believe this is because it’s easier to fulfill the Agile Manifesto with a small team. For example, face-to-face interaction is much easier, and gives you much more productivity, on a small co-located team then it does on a large distributed team. More documentation/artifacts are required for that larger team because 20 people can’t have face-to-face communication every day.

When you look at scaling a practice to create scrum-of-scrums, are we really scaling Agile? My experience in recent years is that most Daily Scrums are not actual Scrum meetings. They are status meetings with free-flowing discussion. They don’t keep to strict questions about what was or will be completed. Not only is this not Agile (as defined by Scrum), it’s not scalable. When you get to a scrum-of-scrums, you probably have a worse situation since those Daily Scrums will be filled with project managers who are used to discussing status and roadblocks instead of keeping to a Scrum script.

It’s frustrating, because one of my favorite parts of Scrum is the efficient and effective Daily Scrum format.

Agile Isn’t Everything. Neither is Anything Else

I’m not arguing here that we don’t need to roll up estimates and amount of work completed. I’m making the argument that these aren’t necessarily Agile techniques. They’re certainly not traditional Scrum techniques. And we should consider the cost/benefit of using a practice like Scrum (or any other type of process) when we need to make decisions in a way that is not Scrum-like.

For example, the Agile Manifesto says we prefer responding to change over following a plan. That’s an excellent perspective, but it’s a harder perspective to execute on when you have a large project with lots of moving parts. Customers need delivery dates, dependent teams need to accept new code changes, payment is only made after certain functionality is delivered and the business is counting on that money, etc. Four people working in adjacent cubes can coordinate with quick conversations. Larger projects gotta have a plan to follow or they sink under their own weight.

The Agile Scaling Trap

I assert that there are techniques that make you more effective on a small team that don’t scale to a larger team. The techniques lose their effectiveness in larger teams. The dynamics of larger teams are different, not just bigger. There are things you have to worry about with a large team that you don’t need to worry about with a small team.

What’s a small team? Surprisingly there’s no good research on that in our industry, last I checked. But based on some research with entrepreneurs, and using the n(n-1) formula that illustrates the Law of Diminishing Returns, I believe that a small team is no more than 5 people. More than that, and the communication overhead for each new person on the team increases geometrically. It’s also easier to “hide out” and do less work, which is why I join the largest teams I can find.

And the team needs to be relatively isolated. That is, it has few dependencies on other teams and other teams have few dependencies on it. As you depend more on other small teams, the meaning of “team” changes and you find yourself part of a larger team.

So moving from a small to medium sized team involves more than just adding new people. Everyone needs to communicate with each other differently, and the overhead increases dramatically. You can no longer be successful doing the same thing you did when the team had 3 people on it.

By the way, this doesn’t mean you can’t use Agile techniques in larger organizations. You absolutely can. For example, Story Points is something that an organization of almost any size can use. But we need to be wise about what we use and what can be scaled to larger teams.

Adapt and Overcome

So, what’s this all mean? Guido was making the point that at the project level, you still need to deal with lots of planning work items because you have multiple teams on your project. He’s correct. But looking at a WBS rollup is not Scrum, and it’s not even an Agile practice. It was hard to manage this kind of information 30 years ago, it’s hard to manage it now, and it’s going to be hard tomorrow. Tools make it a lot easier, and tools continue to get better. But it’s not a surprise that total automation eludes us.

We still need to adapt processes to our circumstances, history, and goals. Often, adaptations force us to change the way we view or interpret information, which makes us go back and re-think the complex ways we need to evaluate the progress of our teams and projects. It’s no wonder we haven’t come up with a one-size-fits-all process or project management schema.

We still need to understand our own organizational needs and goals, which means we still need to understand how practices (like Scrum, Story Points, etc) give us value so we can choose wisely, adapt appropriately, and get the most value for the energy we put into defining our processes.

RTC Adoption Problems: Tool, Education, or Perspective?

I work with CLM customers a lot and sometimes I encounter people struggling to adopt RTC. Once in a while the problem is with RTC – a bug, the rare difficult-to-use feature, etc. Once in a while it’s just a technical knowledge thing. For example, if you’re not aware that your Websphere JVM heap space should be no more than half your physical memory, then you’re going to have issues.

And sometimes the problem is human nature. It’s our own difficulty with breaking out of an existing perspective, breaking old habits and old ways of looking at things. These are the most difficult problems to solve, but solving them provides the biggest payback.

Example: Scrum Iteration Planning

To illustrate this issue, let’s say you’re using the RTC Scrum process template. You think that it takes too long to load an iteration plan. You perceive a performance problem. Fair enough.

So we look at your plan and see that you’re loading many hundreds of work items when you display your plan. In other words, you have planned to do hundreds of things in the course of a single Scrum iteration. And these are higher-level items, Epics and Stories, not just tasks and bugs. The problem is, a Scrum iteration shouldn’t have that many planed items. You should generally have dozens of plan items, not hundreds. What gives?

It turns out that you have nested iterations (which is fine), and your iteration plan is focused on the highest level iteration (which is not fine). We find that the timeframe of this high-level iteration lasts for more than a year.

Scrum iterations are designed to be short and provide a deliverable at the end. Four to eight weeks is common. If you have longer iterations, it’s harder to do the planning, estimating, and managing that makes Scrum so useful. Instead of looking at all the planned items in your project, you want to look at just those that are planned for the current (or possibly the next) short iteration. Scrum focuses the team on the “now”, partly because Scrum allows for a lot of change to a project in the future. In fact, Scrum embraces and encourages change, so we can’t know, in a detailed way, what’s going to happen past the next iteration.

This means that you are losing a lot of the value of Scrum if you are trying to see the entire project in one iteration plan. And, if you only look at the plan items for a fine-grained iteration, you’re not going to have performance issues because you’re not loading hundreds of plan items.

It also means that in this example, RTC looks like it has a performance problem, but what’s really happening is that the Scrum template is being used in a way it’s not designed for. You’re trying to pound a square peg into a round hole.

Right Process, Wrong Perspective. Or, Right Perspective, Wrong Process.

So now you’re saying “OK smart guy, but I really want to see the entire scope of my project. I’m used to looking at Gantt charts and critical paths in Microsoft Project. It’s how I’m successful at delivering my software!”

This is the crux of the problem. Scrum (or any process) is designed to deliver value in a certain context. For Scrum that context is small teams that can deliver in short cycles and who embrace plenty of changes as you go. But maybe you’re working on a project that can’t do that. Perhaps you have a large project that must try to define all requirements up-front because of contractual constraints. Or you’re required to report progress against a specific total estimate so you can bill the customer. Or your organization is used to working in a way that’s not the least bit Scrummy, and old habits die hard.

We can argue all day about what’s “good” and “bad” about the process you’re using, but that’s not the point here. It may be that the nature of your project prevents you from adopting all of Scrum (or whatever process you’re trying to adopt). A better way to say that is that the nature of your project should define the type of process you adopt. Maybe you actually do need to look at all plan items across the entire project at all times.

That’s OK, I’m not judging. But let’s not fool ourselves into thinking that Scrum will solve our problems because Scrum is all good, hail mighty Scrum. Let’s use RTC to model the process that we can do and must do, the process that we’re contractually obligated to do or that has proven to be effective in our organization for our particular set of circumstances. Or, use RTC to model the process that we need to do, that will solve some problem in the way we develop our software.

The real problem in the scenario described above is that people want to do Scrum “because it’s Agile”. People want to get that Agile mojo and make their projects less painful and more predictable. But first their project needs to be amenable to Scrum practices, and if so they need to change their perspective and the old, ingrained, unconscious habits they’ve developed.

It’s very, VERY hard to change habits and perspectives. You have to have a clear target and keep that in front of you while your body and mind try to yank you back to your habituated way of doing things. The good news is that a tool like RTC can help you by enforcing new process elements that you want to adopt.

But if you try to use the Scrum template (for example) because you want to adopt Scrum, but you want to view reports and plans like you did in your old Waterfall process, you’re going to be frustrated. You’ll be banging a square peg into a round hole. There’s nothing wrong with square pegs or round holes, but we need to pick the right peg for the right hole.

This means we may choose the Traditional process template in RTC instead of Scrum. Or we choose the Simple process template and add Scrum elements to it, along with some custom stuff we like. Or we choose the Scrum template and add some elements from the Traditional template to it.

We do what we need to do to make sure we:

• Do things that make us successful in delivering software IN OUR PARTICULAR ORGANIZATION.
• Consciously change what doesn’t work, and enforce what does work.
• Continuously review our process and incorporate new best practices. In other words, make a habit of creating new good habits.

And when we bump up against a problem in RTC, we want to ask ourselves if:

• Something is wrong with RTC
• We lack some knowledge we need to acquire
• We’re using the chosen process in the way it was meant to be used

I guess the moral of the story is that there’s no such thing as a free lunch. If you want the advantages of Process X, you need to understand how Process X delivers the value that it does. Then embrace that behavior by forming new organizational habits.

In my experience, this is the most difficult part of adopting a process. Teams and organizations who get good at this are much more successful than those that don’t

Jazz Jumpstart Speakers at Innovate

My Jumpstart colleague Rosa Naranjo has posted a list of Innovate sessions presented by Jumpstart team members. Given the high level of expertise across CLM products the Jumpstart group represents, I recommend you attend these sessions if they cover any areas you’re remotely interested in.

JazzPractices at Innovate 2013

I’ll be attending the IBM Rational Innovate conference this year and leading the Process Enactment Workshop. If you’re going to Innovate, consider attending WKSHP-1116, the Process Enactment Workshop. I’ll be delivering it with my co-creators Ralph Schoon and Jorge Diaz Garcia on Tuesday afternoon.

Ralph is from Germany and Jorge is from Spain, so if your native language is English, German or Spanish we can promise to answer your questions in your mother tongue! Go to the Sessions page and search for “ruehlin” to see the abstract.

Many of us from the Jumpstart Team are going to innovate. Check out the team bios and see if you want to visit any of their presentations as well.

If you’d like to meet with me, please email me and we’ll try to get something set up during the conference. I’m imagining a group process enactment discussion over Mai Tais at the pool…

Share this:

RATL Perf Land Blog

The Jazz Jumpstart team is lucky enough to have Grant Covell join us. He’s a performance guru, and has done a lot of work calculating and validating performance on CLM. 

I’ve added his blog, RATL Perf Land, to the blog roll. If you’re interested in performance issues, you should check out is work.

What State Are You In? And Where Can You Go?

One of the hidden frustrations people encounter when using work items in Rational Team Concert is knowing the current state of the work item AND how to get to other states. People sometimes get frustrated because they can’t switch to the state they want, or they can’t visualize the flow of the work item.

My colleague Ralph Schoon, Germany’s best export, described a way to customize a work item attribute that shows you the current state and available states in your work item. It’s based on original work done by Kristina Florea.¹

Here’s what you need to know about this: IT’S TOTALLY COOL! This should be a built-in feature of RTC. But if you do a little work item customization (and create a simple graphic of your WI states), you can have this extremely cool feature as well.

Here’s a new, unsaved work item. You can see where you are in the flow, the states that are available, and how to get to the other states. The Workflow Information field shows a graphic of all the states, with the current state in green. The Workflow Information field is of type “wiki”.

You can see that the work item is “New”, and it will be “Proposed” after we save it.

After we save the work item, the graphic changes to the following:

We are now in the Proposed state (which is also shown in the field at the top of the window). We can see in green that there is just one possible thing for us to do in this state: research the work item. We can also see that if we want to Reject the work item, we must first Research it.

This makes it much easier to get the work item into the state you want it to be in. It also makes it clear what the entire workflow is, which makes understanding the purpose and direction of work items much easier.

Currently, you can only see the workflow for a work item type (not a specific work item instance), and that can only be viewed from the Process Configuration tab of the project description. Usually the only people who look at that (or know to look there) are the process engineer and the project lead.

Using Ralph’s technique, you can see the workflow for a specific work item (not just the general WI type), and you can see what the current state is in context. From a process perspective, this is a big shortcoming in RTC that Ralph has addressed.

You can do this yourself by using JavaScript to customize work items. Ralph provides easy-to-follow instructions in Lab 5 of the CLM Process Enactment Workshop. I recommend doing the whole workshop, or at least the work item customization labs. But if you want to jump right to the section that describes how to display the work item flow, it’s in section 5.6, Calculated Value to Visualize the State of the Technology Review.

You also might want to request that this feature be added to RTC so it’s easier to implement.

1. Ralph was incorrectly identified as the creator of this feature in the original article. The passage was corrected to show Kristina Florea as the creator.

Why We Need Best Practices

Part 1 in a series.

Every discipline has best practices. These are the behaviors and skills that contribute the most to success. They aren’t objectives in themselves. You don’t “complete” a best practice and then you’re done. Best practices are perspectives and disciplines that you always apply almost every day to make you better at what you do.

To illustrate, I heard a story once about salespeople taking training classes at a conference. All the younger sales folks were going to the cool classes about new sales techniques, the latest psychology of sales, etc. But the more experienced salespeople were going to classes on how to negotiate better, how to close the sale, how to do a better job of networking, etc.

You might wonder why the more experienced people were going to those basic sales classes instead of classes about special circumstances or new trends in sales. The reason is because those basics – negotiating, advancing the sale to the next level, connecting with customers – are fundamental practices that contribute the most to being successful with sales. Negotiation and advancing the sale may be 50% or 80% of actually making a sale. Getting better in those areas, even if they’re areas you’re already good at, will give you more “bang for the buck” for your efforts.

Another example: When I was doing karate, the particular school I was in insisted that you memorize a bunch of tenants and purposes of that particular style. The very first one was “Practice basic techniques all the time”. And every class we did so.

Why practice the most basic blocks and kicks all the time after already practicing them for years? Because they focus on balance, on protecting your must vulnerable areas, on improving your core and speed. These are the things that not only give you the foundation to do dramatic (but not very effective) jump-spinning kicks. They also continue to make you better dealing with the kinds of attacks and defenses that you’re most likely to encounter. Block, block, kick, punch. Don’t worry about flying through the air, just avoid getting hit and disable your opponent. Or better yet, run away!

It’s the same for software development. There are fundamental practices that transcend technology and fads. If you get good at those areas you’ll significantly reduce your risk of failure, you’ll be much more likely to create a product that will solve someone’s problems, and you’ll create more robust and high-quality software. If you don’t focus on those best practices, you’re much more likely to fail to deliver a product on-time that solves someone’s problems.

I started to learn best practices by failing a lot. Like many people, I finally started to migrate to doing what seemed to work and avoiding what didn’t. In fact, after you work on some projects over the years, you can start to “smell failure”. That’s the visceral sense of dread you get when you see too many things happening that just aren’t right. When this happens, you’ve started to get a sense that best practices (even if dimly perceived) are not being performed. Run away!

Rational Software (later purchased by IBM) defined 6 specific best practices of software development and organized their products and solutions around them. That was GREAT! We could actually talk to customers about how to solve their problems and discuss where their shortcomings were. For almost 15 years I’ve used the thinking around those practices to help lots of people get better at delivering software.

These days, it feels to me like we have less understanding of what these best practices are in our industry. Teams do “agile” (whatever that means nowadays), or they repeat what their organization has done for many years, or they just make it up as they go along. While we’ve come a long way over the last 30 years, we’re still addressing the same kinds of problems. And we still need to address the same best practices to be successful.

One more thing about best practices in our industry: they give us a way to help measure the value we deliver. Measuring value can be hard. Is a programmer good because he writes three times as much code as someone else, or is the other person better because it’s hard to find bugs in her code? Is a tester good because she finds a lot of bugs, or is the other tester better because he identifies ways for developers to avoid writing bugs? Is an analyst good because he writes requirements that people can understand? Or is the other person better because she writes requirements that developers can design from?

The fact is, bad software can succeed with good marketing. Good software can fail in the marketplace even if the developers are delivering on their minimum SLOCs. Rewards and punishments for these kinds of results may have little to do with the skills of the practitioners and their contributions. It’s often hard to draw that line between what a practitioner does and the success of a specific product.

So instead of looking at how well a product does in the market as a way to reward practitioners, we want to encourage the behavior that’s most likely to make the product successful, and discourage behavior that won’t. If we identify best practices, we can focus on encouraging behavior that realizes those practices. It doesn’t guarantee success or assure failure will be avoided. But it does give use a way to perform that is more likely to make us successful. That’s what we should be measured by.

I thought I’d do a series of blogs on the best practices for software development as I see them. There aren’t very many practices, but they cover large areas. I would love to see the industry re-dedicate itself to best practices instead of embracing are dearly loved process philosophies. Software development processes should make us better at performing best practices, not be ends unto themselves.

Here are the best practices I’ll be covering in future posts. Stay tuned!

• Define the problem and a general solution. If you only do one thing right, do this.
• Manage requirements. At the least, manage the 20% of critical requirements.
• Test continuously. Like voting in Chicago, do it early and often.
• Deliver iteratively. Always get back to solid ground, aka stable software.
• Model the software. The people currently in the room with you are not the only people who need to understand what you’re doing.
• Define the architecture. No, really. Define the architecture!

Use JavaScript to Customize Attributes in RTC

We just published the fifth lab of the Process Enactment Workshop. Ralph Schoon created this lab to illustrate how to use JavaScript to customize attributes in RTC.

Ralph once again makes it easy to understand things that are hard in Rational Team Concert. Another awesome addition to his technical oeuvre.

Share this:

Importing Work Items to RTC: Uploading Attachments

My colleague Ralph Schoon has posted an article on how to programmatically upload attachments to the appropriate work items when importing those work items from another system. 

This is very useful when migrating from another type of change management system to RTC, since attachments are not automatically copied over.

JazzPractices Blog: 2012 in review

The WordPress.com stats helper monkeys prepared a 2012 annual report for this blog.

Here’s an excerpt:

600 people reached the top of Mt. Everest in 2012. This blog got about 8,700 views in 2012. If every person who reached the top of Mt. Everest viewed this blog, it would have taken 15 years to get that many views.

Click here to see the complete report.