How to quadruple your productivity with an army of student interns
Posted in Ksplice on March 10th, 2010 by Greg Price – 54 CommentsStartup companies are always hunting for ways to accomplish as much as possible with what they have available. Last December we realized that we had a growing queue of important engineering projects outside of our core technology that our team didn’t have the time to finish anytime soon. To make matters worse, we wanted the projects completed right away, in time for our planned product launch in early February.
So what did we do? The logical solution, of course. We quadrupled the size of our company’s engineering team for one month using paid student interns.
The Ksplice interns, ca. January 2010. Photo credit: http://archive.cternus.net
Brooks’s observation usually holds. New people take time to get up to speed on any system—both their own time, and the time of your experienced people mentoring them. They also add communication costs that grow quadratically with the size of the team.
Fortunately, Ksplice benefits from a bit of an engineering reality distortion field (our very product is supposed to be technologically impossible) and, with the right techniques, quadrupling our engineering team’s size for one month worked out great. Every intern was responsible for one of the company’s top unaddressed priorities for the month, and every intern successfully completed their project.
So, how do you quadruple the size of your engineering team in one month and still keep everyone productive?
- Tolerate a little crowding. It took a little creativity to suddenly find a dozen new workspaces in our two-room office. Fortunately, we’ve found that a room can always fit one more person—and by induction, you can fit as many as you need. (All those years we spent proving math theorems came in handy after all.) Seating everyone close to each other has an important advantage, too: when lots of people on your team have just started, it’s handy for them to work right next to the mentors who are answering their questions and helping them ramp up on the learning curve of the organization. With the right team, the crowding can also create an energetic office environment that makes people love to come in to work. (Sometimes it gets in the way of concentration, though—that’s when I put on a good pair of headphones.)
At work in the Ksplice engineering office. Photo credit: http://archive.cternus.net
- Locate next to a deep pool of hackers. OK, so we’re a bit spoiled by being headquartered a few blocks away from the Massachusetts Institute of Technology. At MIT, January is set aside for students to pursue projects outside of the curriculum—perfect for hiring an intern army. Many other institutions have either a similar “January term”, or a program for students to spend time working in industry during the term.
- Know who the best people are and only hire them. Ksplice was born four years ago at SIPB, MIT’s student computing group. When a group of students run computing services thousands of people rely on, and spend hours each week discussing, dreaming, collaborating, and learning from each other on computer systems—some of them get really good at it. Even better, everyone sees everyone else in action and knows exactly what it’s like to work with them. Investing some time into getting involved with technical communities makes it possible to hire people based on personal experience with them and their work, which is so much better than hiring based on resumes and interviews. Companies like Google and Red Hat have known for years that being involved in the open source community can provide an excellent source of vetted job candidates.
- Pay well. In some industries, “intern” means “unpaid”—but computer science students have plenty of options, and you want to be able to hire the best people. We looked at pay rates for jobs on campus, and pegged our rate to the high end of those.
- Divide tasks to be as loosely-coupled as possible. Our internship program would never have worked if we had assigned a dozen new people to hack on our kernel code—the training time and communication costs that drive Brooks’ Law would have swallowed their efforts whole. Fortunately, like any growing business, we had a constellation of tasks that lie around the edges of our core technology: infrastructure upgrades, additional layers of QA, business analytics, and new features in the management side of our product. These had manageable technical interfaces our existing software, so our interns were able to become productive with minimal ramp-up and rely on relatively little communication to get their projects done.
- Design your intern projects in advance. A key challenge when scaling up your engineering team quickly is making sure that the interfaces are all well designed and the new projects will meet the company’s needs. So we spent a good deal of time getting these designs together before the interns started. We also allocated plenty of our core engineers’ time for code reviews and other feedback with our interns in order to make sure their work would be maintainable after they left.
Have you achieved more in one month than anyone thought should be possible? How did you do it?
[Edited to make clear our interns were paid and to say more about how we designed their projects for high-quality output.]
Interesting. I may have to do that this summer on one of my pet projects.
Interesting Post! What methods of communications did you use? Any besides: “Hey You” to the person sitting next to you? How did you document what you wanted each intern to do? Did it all really run super smoothly or did you into any hiccups? If so, what were they?
Thanks!!
@John Azzolina:
We make extensive use of instant messaging in the office in order to allow lengthy conversations without distracting other people and without going to the conference room.
For telling interns what to do, we did most of the communication in person or via IM. Rather than writing up an extensive design document, we focused on a tight feedback loop, checking in on progress at least once a day, and often more, and being as available as we could be to answer questions from the interns about how to proceed.
Something that helped a lot with tight feedback was rearranging the seating so that each intern was sitting next to their permanent employee mentor (see the point about crowding in the post).
It ran pretty smoothly overall. The main hiccups were that there were a few days when certain key employees needed to be doing complex operational work, and thus didn’t have time to fully brief one of the interns on their next project. We generally tried to find something productive for them to do while waiting.
I had similar results with MIT interns when I worked in Boston. The caliber of the MIT students is very high – they really surprise you when you give them a general outline for a project and leave the solution method up to them. We got some very creative solutions.
Not all schools excel in creative problem solvers like MIT. Other students might require more direction – and I am not an MIT alum.
It is also important to pay well. It is a lower pay scale – interns. A little more cash really has an outsized positive result.
Thanks for putting down the thoughts so clearly – JHL
Great post. We see both sides of this coin at CollegeJobConnect:
First of all, we are doing the same thing at our startup to expand our reach and productivity. We hired a group of “on campus marketing directors” to help with our marketing push, which had paid off handsomely for us AND for our interns (they are getting real world experience and getting paid). Win, win.
Secondly, our service itself seeks to help companies (like yours) better connect with juniors and seniors across college campuses. (We just recently launched and are only at Dartmouth and Harvard right now, but looking to expand further). On campus recruiting needs to be improved for both the student’s and employer’s sake, and we are aiming to do just that.
One very important point is that it’s usually against local labor laws to employ interns without paying them. The “work for experience” approach will often get companies into trouble. Interns must be paid at least the prevailing local minimum wage, and as you pointed out, you’ll get better interns if you’re willing to pay more. (They’re doing a lot more than they would if they were working for McDonalds or Wal-Mart, and they should be paid accordingly.)
@Len Feldman: Yes, I wouldn’t recommend hiring software interns for minimum wage (let alone less). As you saw in the post, we took care to pay our interns well.
That photo looks like it was taken in the 80′s – was it really taken in 2010?
@Len: where do you live that volunteer work is illegal? I’ve never heard of this.
It doesn’t seem like the Mythical Man month was “busted” given that you gave interns loosely-coupled projects. Brooks was about the (much more common realistic) single big project that has lots of component and sequencing couplings. The communication points required are a function of that coupling. Remove the coupling and you remove the need for communication and hence the bottleneck. Not so much busted as sidestep. Which is always a smart move.
I think Brooks would be proud of you. You managed to avoid his result by avoiding his preconditions. What this suggests is perhaps that software architects should concentrate on creating embarrassing parallelism wherever they can.
So, every intern completed every project successfully and on time (even without a desk).
But then why have employees?
As the two posts above mention, this is not in any way disproving Brooke’s Law. Rather, you’ve simply side-stepped the issue and applied reasonably intelligent management techniques to solve your problem. That is what this article should be about.
Also, congratulations on making it to the front page of Slashdot. The entire interweb is ripping this article apart.
@Len – “(They’re doing a lot more than they would if they were working for McDonalds or Wal-Mart, and they should be paid accordingly.)”
You mean YOU would pay them more. Should is rather definitive, and not a statement that can be made across the board. Sometimes, experience itself is of such benefit, it is worth it to the worker to do such for free, or for low pay.
You of course logically follow that the amount that something is worth, is what all parties agree that it is worth?
Saying “Mythical Man Month Busted” must surely attract much attention! I agree with some of the other comments. This does not bust it, as you managed to not use the interns in the “core” project, rather they worked on ancillary projects that complemented your core mission. That said, what you did is very impressive, and speaks highly of your preparation, loose yet defined environment and communication process, and your leadership abilities. Your fairness in compensation is a major point that I find refreshing. It must have been very exciting to work there during this time!
Congratulations and thank ls for sharing. Now how can I get some MIT students out here…….
Aha. Avoided Brook’s law, by applying Amdahl’s Law…
Excellent!
@Martin Ross. That was exactly my first thought — myth not busted. Clearly some people need to actually READ The Mythical Man Month. Doing bad science is the first sin. Reporting it is the second.
Another anecdote without facts in an anecdote driven industry, well done, hurray!
Stephan
http://codemonkeyism.com
Actually, not Brooks’ Law – more like Gustafson’s Law. Excellent idea all the same.
Sorry – I meant not Amdahl’s Law — more like Gustafson’s Law. Sorry – it’s been a long day….
Though perhaps the myth is not fully busted, I do think that the myth’s truth depends on a lot of assumptions that are not necessary. For example, high quality people can add to a project with little added work, work independently, and produce meaningful results. Here was one example, but even putting someone good on your core, to read parts of it, understand them, look for bugs, can work. Good programmers can cut through code without help, even though it takes a lot more time. If your development environment is easy to set up (or you have good IT), your code is decently structured and somewhat documented, then the new people could even get to do some actual work rather quickly. Want another idea? Create a Wiki, and have the new people put information there as they discover it.
Now, I’m not claiming that it’d be extremely cost effective, but I do think that it’d move your project forward. The myth assumes a (very common) scenario of mediocrity at all levels, and that’s where is can be overcome.
Seems that your project was not late, you just thought it was. Also you made preparations before adding people, which makes the job easier to do. If project skips preparations like designs, then it is going to go worse. You made preparations, then added workforce like workforce is added after preparations.
In bigger projects there is less room for creativity when you design and implement complex business systems. If you have lots of complex things on your hands half built and then add many people there, the need for extra communication is far greater. The myth is not so much busted as it is proven that you can get things done faster than you would think in the first place.
You did not disprove Brooks’ Law, you cheated your way around it by avoiding it’s preconditions.
You use Linux, which made your ‘intern army’ already knowledgeable in the matter previous to hiring. If you had to train the interns from scratch, on proprietary system (like in Brook’s conditions) you would have VERY different results.
hmm,
so you hired a bunch of hackers (who can find their way around), paid them a lot, made them work on their own projects and had a successful one month. Why dont you try this in a real world scenario, hiring real average programmers that you can get in the market, pay them market rate and make them work on your current project which is late?
Hmm, in recent classes, we put enphasis on good segmentation. If you divide your functions and their methods properly, you can assign a whole function to one person.
There again, if it was everyone workign on the kernel, it’s a different story. Same if the function has 2000 methods and it needs the whole team to be focusing on it. I find it interesting that you managed to get all the things around the main core to be done fast and easy! It’s sounds like a good way to get things done.
I live in canada, and my university is the closest thing to MIT, we even kick their asses in a couple of event every now and then… So it’s really the same kind of thrive to evolve environment. ETS (École de technologie supérieur). Getting student that are interested and involve definately helps with the quality of programming you can get out of them.
As for us, we obviously get paid to “intern”, not because we are greedy, but like every other student, there is only so much peanut butter we can eat! And to be honnest, only people with either a technical background or a 3 years degree in programming can attend to this place (well was the rule when I got in) so technically, everyone is already good at programming, we are specializing in managing a big team effort.
As for Brooks, it’s not exactly the same situation, but definately a way to go around it, and get things done. You just proved that you went against the “more people is less work” problem, and I think it’s what people should be focusing on here, not that the brooks preconditions were not met. I mean thats why he failed in his project, why would you try to meet the preconditions.
Anyhow, still fun to see a project that has done it with lots of people, and thanks for sharing that bit of info with us!
Obviously your setup and setting are not the same as Brook’s. His was one large & late project, but your are (as I understood) many small, novel and unrelated projectlets that all started on a blank slate. So adding manpower obviously works.
What I wonder though … is about maintenance of all those projectlets
who is supposed to maintain and understand all of that 20 projectlets? (that are all written by different people in different styles and up to a hacker’s standard of documentation 
Any assumption that you have “broken” the man-month rule is totally out. You have obviously hired better people, which equates to normal people being able to take on the same task with a little less parallelism in a similar time frame.
Have you looked up Amdahl’s law lately?? Think a little about that before you grow a set of balls too heavy to carry. Better processing power is required to accomplish the same tasks in parallel because not all tasks can be done effectively in parallel systems. It applies to people as well as it does to computer systems.
I think there are lessons to be learned here about how to design your application and how to split the development work.
SOO glad to see others recognize and are promoting this! The Internship Institute did a study several years back that individual supervisors can gain 225+ days of work productivity/year (see link below to view summary). Sounds like you think it’s even more. We won’t argue with that!
http://www.internshipinstitute.org/paradigm.asp
As a non-profit charitable organization [in addition to our many programs] I would also like to acknowledge the timely launch of InternshipSuccess.com, a training portal to encourage, prepare and support students who intern. Anyone who reads this post may have $50 off [to make $29 the final cost] and offer to any college parents/students and those who employ interns to benefit. Use the instant credit code: “BeReady”. Keep evangelizing and happy hunting everyone!
Matthew Zinman
Founder, The Internship Institute
I would have to agree with many, many, of the other posters in that the work you did really, truly, has no bearing on Mr. Brooks’ law. It has nothing to do with cheating, working around or such. The problem you solved by hiring the interns (qualified or not, paid well or not) had very little to do the problem Brooks was trying to solve.
Now, if you were able to hire all those interns and had all of them hacking the main kernel of your system (changing interfaces and function calls, rewriting your scripting system, etc.) and they originally came from a Windows programming environment, AND made the same level of improvements I would truly be suitably impressed. You even said in a response that there were times when you had a communications bottleneck. When one of your key kernel developers was engaged in a heads-down effort and couldn’t manage the intern(s) assigned to them. That! is what Brooks law is all about.
Instead, what you have done is nothing more than what any decent (not good, just decent) project manager would have done in the first place. Parallelize whenever possible. Reduce or eliminate a single critical path. This sort of effort is done all the time, by many companies, large and small. At many schools – not just MIT. In many industries, on many platforms (from Mainframes (which I have direct experience with Mr. Brooks’ ultimate work) through embedded systems.
In short, and to burst your ego-headed bubble, you had them work on the crap you didn’t want to. That makes you practical and efficient, not newsworthy.
Stand in line to become a Project Manager – not a systems architect.
BTW – Where were your scope, change, testing and updated system documents? Without them I don’t want to see your system in my environment as I don’t want to have to wade through the crappy UI’s and interfaces.
Fred in IT.
Have to agree with the “slow down there” comments. It sounds very much like you didn’t add people to a late project – you had a couple of dozen small projects that you didn’t have time to accomplish. There’s a vast difference in those two things.
I’m also curious whether this is an applicable model for general practice. Most companies don’t have the luxury of being right next to MIT. Even if they could, while the idea is novel what do you do next year, when three other companies try the same thing? And the one after that, when Google and Microsoft have decided it’s a good idea? Brooks also said that everyone would want a team like yours, but the reality is you just don’t get it. You definitely managed to do something innovative, but until you can even repeat it yourself – much less apply it broadly – it’s a little rash to be claiming broken laws.
I wonder how much technical debt was accrued up in the process.
hey! this sounds like a great idea.
but prove it with this challenge – get 40 women, no wait, you can have an unlimited number, and produce a child – from conception to full birth – in one week
-
can you drive your car excessively fast? can you ignore the oil change interval? skip maintenance? ignore common sense? – sure, once in a while – but one day it will catch up with you
One last little bit of reality that was missed – and that was – reality.
Your labor was, for all intents and purposes, the equivalent of robotic or slave style labor. They were paid, yes. They were able to leave, yes. But, they did not have the weight of the real world on their shoulders. They were there, pretty much, for the thrill of seeing if it could be done. They had no other scheduling conflicts. The equivalent of seeing how many jocks/geeks, etc. will fit into a phone booth.
There were no, soccer games, swimming practice, runny nose, mortgages, or other such real-world issues that encroach upon real-world work efforts.
Now, don’t think that I don’t know that some of the more modern programming and systems development practices have a definite place in the industry. Continuously evolving systems are great when the system needs to evolve to retain a customer base. But, there is, also, still a place for the old top-down approach. When stability and reliability are required. The strange thing is, you didn’t know which approach should have been used for your system.
Lastly, I wonder what OSHA, health or fire departments would have to say about you cramming that many students into such tight working conditions.
As such, it was a nice experiment – but proves nothing about reality. Other than to illustrate the immaturity and carelessness of your company.
Fred in IT
If you have a small group of people split up the “project” in a well-defined way into much smaller projects, then suddenly brooks law doesn’t apply anymore.
For this to work, the project needs to lend itself for this. So, the fact that ONE project managed to achieve scalar multiprocessing in the man-power department, is not interesting news.
Say you have to build a website where each of hundreds of pages needs to interact in a unique way with the user using javascript. All those interactions are documented, and none of them refer to a central database or anything. As a whole, there is a large lines-of-code number. But in fact, there are just many small projects. Add more programmers to such a decoupled project makes it faster.
In general though, not many (large) projects lend themselves for such complete decoupling. So Brook’s law usually applies.
What you identify as your “core technology” is probably what Brooks would call the System portion of the larger System Product. By your own admission your one month sprint didn’t complete, make progress, or enhance the core product, what he calls a System. What you did is address some of the other issues in the rest of the System Product. This is level of effort you simply failed to accomplish previously, and brings you a step closer to producing a System Product and not just a System.
So, you haven’t broken any new ground here, and you certainly haven’t made The Mythical Man Month obsolete. You have only proven you haven’t read his book, or at least learned anything from it.
Single most important multiprocessing issue: loosely-coupled tasks. (Neglecting all the real-world “people” issues others have mentioned, which are worthy of consideration.) This is exactly the same issue as efficient multi-core or distributed-system design. And, as others have also noted, is not what Brooks was talking about.
At a previous job, I had good success with co-ops and interns working on regression tests. We had a framework and a testbed, and all regression tests had consistent interfaces, so we had nice Lego-block-style items to work on. Students who had done co-ops elsewhere were so happy to actually *accomplish* something. It can only work with planning and design.
I did a similar thing with Berkeley grads for a summer. The results were spectacular. It was amazing having all of these smart, tech savy people around. I would give them a task with general directions and they always exceeded my expectations. It does help when you know the people you are getting. When you get random people, your chances of getting a slug are better than getting someone stellar.
If you are fortunate enough to be well liked by your interns, they will recommend you to their friends which will keep your cycle going.
I studied under Fred Brooks while at the Computer Science department at UNC-CH. I asked Dr Brooks to read this post. He replied: “I read it and the first 50 or so rebuttals. I don’t need to study it out; the commenters did a quite adequate job of disposing of it. The ultra-parallel nature of the tasks fully explains their result, in my view.”
Thanks to Ksplice for the interesting article, and congratulations on a productive month.
alamedated: Glad you had a similar good experience! I hope and believe we will benefit from the same cycle you did.
Mark: Wow, thanks! I did not imagine hearing back from Fred Brooks himself on this post. Of course I completely agree that our results do not contradict or disprove the lessons he taught us all. I think people who have read his book or heard of Brooks’s Law sometimes think it inevitably applies to their own situation, and what I hoped to get across in this post is that that’s not true—in the right circumstances, adding people to a software project can get a lot done, even in a short time. I’m not surprised, but I am pleased, to hear that Prof. Brooks agrees.
So for everyone who says we “cheated” or avoided Brooks’s Law rather than broke it: exactly! With the right techniques and circumstances, we were able to make a lot of progress quickly on our software project by adding a large number of people. Some of those circumstances were under our control, some weren’t—this approach would never have worked with the tightly-integrated hard technology at the core of our product. A few of the comments illustrate that other projects have achieved the same circumstances; maybe your next project can too.
Finally, while MIT is great and a fine place to hire engineers from, I want to emphasize that the real key to our hiring wasn’t MIT itself—it was the technical community, SIPB, that happened to be located there. Every one of our engineering interns came from that community. The good news, as I suggested in the post, is that you don’t have to be at MIT to find a technical community where engineers push each other to higher levels of achievement and their contributions are in the open for everyone to assess. Look around at open-source communities related to your company’s work, and you may find opportunities to get involved, like Google, Red Hat, and many other firms, in technical communities that can provide you a hiring advantage like the one we enjoy.
Greg Price wrote:
“what I hoped to get across in this post is that that’s not true—in the right circumstances, adding people to a software project can get a lot done, even in a short time”
As many people have pointed out, you did NOT add people to a software project. You created a dozen small, one-person projects. Your self-serving reply to all that is just one more mis-representation. Have you no shame?
Barbara Hudson: I don’t think you and I disagree on the substance of anything here. We planned one product launch. From our business perspective, this launch was a single project with a single deadline; we wanted all the pieces to be ready and to work together, and were prepared to push the whole launch back if necessary in order to complete any piece of the launch project.
It’s true that at a technical level we were able to carve out subprojects that were reasonably loosely coupled to each other. That’s one of the main points of the post! From a technical rather than a business perspective it’d be reasonable to reserve the word “project” for these subprojects, as I think you’re asking for me to do. But I think the comments indicate that readers generally understood the story as I explained it, and nobody was misled into thinking that our project was tightly integrated as one piece of software. Not every software project is amenable to being carved into subprojects in this way, but we’re hardly the only company to have a product which requires diverse pieces to be in place in order to ship: the management layer, the GUI, the various separate features.
In any case, even when a project is decomposable as ours was, that’s not the whole story. If we hadn’t been connected to a technical community, we couldn’t have identified and hired so many capable engineers so quickly. If we’d insisted on a big desk for each of us, we’d have blocked on trying to find the space–and if we found it, we’d have lost the communication benefits of working close together. So whether your current project is tightly knit and inseparable at the technical level, or amenable to subprojects like ours, I hope there’s something interesting for you in this post.
If not, well, no blog post is for everyone. Maybe this one wasn’t for you.
Any monolithic project can be broken up into several pieces, this one was clearly no exception. One of the really amazing things about hiring talent is that the better the talent, the more we get from them.
Average programmers produce average results, your talent pool consisted of the top 1 – 2% of the country even among a student population like this, you picked the “geeks geeks”, total server farm rats. Getting in touch with this community seems to have been instrumental in your success. What you did was the equivalent of going to the Iowa State wrestling team to get coaches for a quick wrestling camp.