Rock Hymas

So my wife started homeschooling our older two boys in September after we’d recovered from the summer vacations a little bit. She likes to write about their escapades on her blog, and they’re doing a great job. In deciding to start homeschooling we had planned to have me help by teaching them math, but then decided before our “first day of school” that I was too busy with work, a long commute and Cub Scouts to do that. So for the last two months Kami has been teaching them using a combination of the Saxon Math program and the Khan Academy, and I haven’t been all that involved, other than as a spectator on the sidelines and someone that Kami can bounce her ideas and concerns off of.

She’s not learning math … yet.

Since then, she’s been constantly tweaking things as she goes. She honed how she taught them math over the last couple of months. First, she went back and forth between teaching Reagan and Cade together or separately. Though they can both handle the third grade material fine, she found that trying to teach them together just led to goofing off. They devolved into performing for each other, rather than thinking about and learning the material. So she settled on teaching them separately, but that did take more time. Each day she would take about 45 minutes with each boy and go through the Saxon math lesson for the day, the daily “meeting”, and help them with worksheets, if they needed it. While one boy was doing this with Mom, the other one did some personal worksheets and then got to work on Khan Academy exercises, usually doing whichever ones he wanted to.

While this generally worked ok, there were two challenges that regularly came up.

First, the amount of repetition in the Saxon math curriculum was sometimes boring, both for her and for the kids. Though repetition is good, once the kid is comfortable with the material it can be too easy to be fun. Also, the boys like to brag to Dad at dinner about what they learned each day, and there would be weeks where I’d only hear about math once. And my wife wants you to know that it wasn’t because they weren’t doing it!

Second, the hour and a half of math time was a big chunk of the homeschooling day, and it was regularly interrupted by the younger kids, our three and a half year old Levi and newly-walking Liberty. This wasn’t as bad with other subjects, where it was  usually easier to recover from interruptions, or they didn’t take as long, or they could be done with both boys and sometimes the younger ones too.

Both of these challenges point to deeper issues, which we discussed occasionally over the last two months. Many of you might have responded to the first concern by saying “Throw out the Saxon Math!” or “Just skip the parts that were repetitive!” Both responses discount the value of repetition in learning. Even if they didn’t, you’d have to consider that Saxon math makes it fairly easy for someone who never felt comfortable in the world of math (my wife) to teach her kids. But skipping parts of the curriculum brought back Kami’s discomfort of trying to figure out what was important to repeat and what wasn’t, without feeling like she had a good idea of what they would need later on.

The second challenge doesn’t provoke such simplistic responses. You wouldn’t say “Well, you just got to get rid of those two younger kids,” or “Math isn’t that important, just spend less time on it.” Actually, we could spend less time on math, but only if we felt there were another way to teach them that would be as effective. But the fact is that we’ve got four kids. Life has been getting easier with four over time, as the older kids become more capable of helping and the younger ones get better at playing on their own, but that’s also not an issue that is going away in the short term.

So rather than try to eliminate those challenges, we’re going to sidestep them using an ingenious plan: I’ll teach them math. Ok, so it’s not ingenious. It’s not novel, or newfangled, or noteworthy. But it directly addresses the issues. I’m much more comfortable experimenting with the curriculum than Kami was. I’ll start simply, just doing what Kami was doing, but plan to change things up based on the needs and abilities of the boys. Besides that, I’m curious about how best to teach math, not just in a generic sense (what works for everyone), but also in a specific sense. What works for Reagan? What works for Cade? Why? We will work towards a method that gets them learning, gets them involved, and doesn’t take any longer than our current one.

This also helps with the challenge of homeschooling with toddlers as well. I’ll be teaching in the evenings, after the younger two go to bed. That means the house is quieter, it helps the boys wind down after their active afternoons, and Kami can deal with the interruptions most days. Another awesome benefit is that she and the kids just got an extra hour and half each day. Some of that extra time can still be spent by the boys doing Khan Academy exercises and math worksheets. But the rest can be used to learn about other stuff, play, relax, run errands, do housework, etc. Basically, it gives Kami more flexibility during the day, which reduces her stress level, and it gives us some more structure to our evenings, which is important when trying to get kids to actually go to bed.

Compared to this kid, math is easy.

Of course, it’s not all unicorns and bacon. This new schedule does stretch me a little bit. I’ve sacrificed some of my commute on the train so I can think about the work of teaching math. I tend to be pretty wiped out once the kids go to bed, so I haven’t been able to get as much done on weeknights. And I do still have my other responsibilities: Cub scouts, my job, and finding time to play with my younger two kids. Much to Kami’s chagrin, I haven’t done the dishes much this week. And when I have other responsibilities in the evenings, it can mean we do some schedule juggling. Though Reagan seems to do fine learning math in the evenings, Cade can be a bit more goofy at that time of day. Those challenges won’t just go away, and I’m sure we’ll continue making adjustments to how we balance everyone’s different needs.

All that said, our first week on the new schedule has gone surprisingly well. Since we started on Monday, I come home to a much happier wife and kids, because they’ve all had more time to do other things. The “perfect homeschooling day” is something rarely seen even among experienced homeschoolers, but Kami’s been really happy with every day this last week. She’s also more relaxed about school, which is good for her and for the kids. The older boys get more one-on-one time with Dad, which they really like, and makes me a feel a little bit better about the long commute.

There are certainly disadvantages to homeschooling (it can get expensive), but one of the advantages we have it that we can be way more flexible. I like to think of our little homeschool as a small startup amidst the mega-corporations known as public education and private schools. Our market is significantly smaller – just four kids, two of which wouldn’t be served by the megacorps anyway – so we can understand their needs and meet them much more directly. And we can respond more nimbly to those needs as they change. This change in how we teach math is really just one of the larger changes amidst a constant stream of tweaks to the way we teach our kids. Some of them work, some of them don’t, but it’s easy to see the successes and failures and respond quickly.

“Hey, I have to tell you about the most awesome weekend release ever. It happened last Saturday, and –”.

“Weekend Release?”, you ask, incredulous. “Aren’t you supposed to be doing weekday deploys?”

“Yeah, yeah, I know. We agree, and we’ve taken a break recently as we make changes that involve enough risk that we don’t feel comfortable doing them during the week. But that’s not the point of this story. What I really wanted to tell you is about my crazy night trying to –”.

“But everyone agrees that deploying at night doesn’t make sense, right?”

“Um, yes. Well, not everyone. But we do, and that’s also on our radar. So, we had to update some test accounts to test some new performance improvements this coming week –”.

“Wait, so your test accounts are on your production servers? Don’t you have a staging environment?”

“AARGH! Will you just let me tell my story?! And, yes, we know we need to build out a staging environment. It is one of our top priorities at the moment.”

“Ok, ok, sorry. Go ahead with your story.”

“Thank you.”

So last week we got a FogBugz release ready with some bug fixes and performance improvements. The plan was straightforward: deploy it to our production servers on Saturday night and upgrade our test accounts so that QA could bang on it for a week. Barring release blockers, we could upgrade everyone the following Saturday.

On Saturday, while I was out running errands, it started to snow. I love the snow, and thought, “That’s cool, my boys will enjoy playing in the snow and it should all be melted by Halloween when we go trick-or-treating.” Those tweets and that thought apparently jinxed me. Once I was done with my errands in the early afternoon, I drove home through near-blizzard conditions only to get a text from my wife that the power had gone out. She also mentioned that a bunch of trees were down on our street. As I got off the freeway and drove down the hill towards home, I was following a bus that was swerving all over the road to avoid low-hanging branches and a tree that had fallen across half the road.

Our releases don’t usually look like this

After a few hours it became apparent that we wouldn’t be getting power back in time for me to login to our VPN and take care of the release. So I started contacting some friends from church to find someone who had power and wifi and the willingness to put me up for an hour or so at 10pm. Brandon, Ian, and John all said they could, but Brandon was first, so I let our sysadmins know the release was still on (they like to be alerted to these things, you know). It got dark at about 7, and we put the kids down early, wrapping our one-year-old Liberty up in a snowsuit so she’d be warm enough all night. At 9:30, I packed up my laptop and went out to make the 15 minute drive to Brandon’s home.

As I left I had two roads I could take: the freeway, which involved some pretty steep hills; or driving through town (three towns, actually) along a road with no significant hills. Because the snow was still coming down hard and sticking pretty well I decided to avoid the hills. That was a mistake. The first few minutes were pretty uneventful, but it was fun to drive through downtown Berkeley Heights and into New Providence in complete darkness. No streetlights, no traffic lights, no homes lit up, no businesses, no grocery stores. Ninety percent of the other vehicles on the road were snow plows (doing a great job, by the way).

The drive through town was also along a heavily tree-lined road, which made the darkness creepy in a “Halloween is cool” kind of way. But as I travelled this road it became clear that I had taken the wrong road. Trees had fallen across the road at pretty regular intervals. The snow plows (or kind citizens) had cleared a single lane path where these had happened, though it was often just barely wide enough for my compact Honda Civic. I went ahead and drove past a few road closed signs and one unmanned roadblock.

Not in New Jersey, but you get the picture

Soon after that I realized why they were closed: The downed power lines. Most didn’t block the road at all, but one was just hanging directly in my path and scraped over my car before I had time to notice it and react. That got my adrenaline running. At another point I got to play chicken with a snow plow that was driving the wrong way in my lane.

I finally made it into Summit, where there was power in some parts of town, and pulled into Brandon’s driveway, but of course, the car got stuck as soon as I left the road, so I was blocking the exit for both him and his neighbors. He and I puzzled over that for a couple seconds, then realized his neighbors wouldn’t be crazy enough to go driving in the storm. He set me up in his basement, his wife Becki offered me something to drink, and they went back to watching TV.

After a little over an hour the release was done, and it was time to head back home. This time, I’d learned my lesson. Throughout my drive to Brandon’s the roads had been clear, thanks to the plows, so I knew that it would be an easy ride home on the freeway.

And it was.

Until I got off the freeway.

I took an exit that would avoid the steepest hill, just in case. Turns out that was a good choice, as I found out Sunday that it was completely blocked by fallen trees. It did mean I had a bit further to drive through town, maybe a mile and half instead of just a mile.

And that’s when the fun really started.

The off ramp had a tree across it that had obviously been chain-sawed through in the last couple hours. Just around a bend, I came across a cop car with lights flashing, blocking the road I would have taken. So I decided to risk a side road to get around the blocked off section. Now I really was in rural New Jersey, and now I got to drive over some branches that weren’t quite big enough to block my Civic. I finally made it back to the road leading home (less than a mile to go!), but just a couple blocks further ran into a road block that wasn’t one I could drive around. Power lines and police tape were draped across the road right at car level.

So it was off into side streets again, on the very steep hills I had wanted to avoid. I still wasn’t slipping around on the snow much, but had to take a couple detours anyway (trees across the road). Finally I started heading back down the hill.

The end was in sight!

So, of course, I got stuck.

One more downed tree. At the worst spot. The snow was deep enough that turning around wasn’t an option. Backing up didn’t work. I’d finally gotten my Civic stuck in the snow only about half a mile from home.

Fortunately, I’d left home with a flashlight (so I could see in the garage). I broke that out, bundled up, and started the hike home. My flashlight was pretty weak compared to the car headlights, though, so I just barely avoided walking into a downed power line on the hike down the hill. Once down the hill, I got to pass a powwow of power crews and cop cars at the last intersection before my home where they guided me past more downed lines. From there on out I walked down the middle of the street in complete darkness illuminated only by my flashlight.

Once home, after taking over an hour to make the 15 minute drive, I jumped in the shower to warm my toes back up before getting under the heaps of blankets my wife had piled on our bed.

Follow-up (Monday): The power is still out, and the city is telling us they can’t even give us an estimate for when we’ll get it back. On Sunday we drove for a few hours into southern New Jersey to get the last remaining generator at the only Home Depot who hadn’t sold out the first hour they were open. We’re now using it to power our fridge, freezer, and DSL modem, so I can work from home, since the trains into New York aren’t running. Kami and the kids are doing homeschool in the dark, which the kids think is just awesome. I’m writing this post in a room that is about 60°F, while listening to the dulcet roar of our generator right outside my window.

Follow-up (Friday): The power finally came back on Thursday evening at about 5pm. We spent the evening cleaning up our home, doing laundry, and generally trying to get our lives back in order. And guess what? I’ve got another release to do on Saturday!

Well, I collected them into a series and added a report card over on the Fog Creek blog. I plan to continue adding to the series going forward.

What is build and release management?

Can you build your product with one command?

Where do developers keep their checked-out source code?

How do I eliminate absolute paths from my code and build scripts?

Do your builds have other hard dependencies?

You keep talking about hard dependencies. Are there any soft dependencies?

How are build and deploy scripts different?

What should you include in machine setup scripts for dev and build machines?

Can you deploy your product with one command?

Are your build scripts crusty and old?

Why should my build and deployment scripts be separate?

Can you build all possible configurations using command line parameters?

Do you have an official build machine or set of build machines?

Do you save your old builds?

Do you store built binaries in your VCS?

Do you have different build scripts for different configurations of your build?

Do you have different build scripts for official builds vs. dev builds?

More to come!

Related Posts

Deploying FogBugz 8 and Kiln 2

FogBugz and Kiln Build and Release Report Card

“Have you been flossing regularly?” - Your Dentist

Don’t you hate it when you go to the dentist and, while the dentist is digging around in your mouth, he asks if you’ve been flossing?

“Huh-mhqhg,” you respond, hoping that all the fingers in your mouth made your negative answer sound like a “Yes, sir”  on the way out. But no; dentists (and their assistants) have an uncanny ability to understand the most garbled language correctly. Either that, or they’re just really good at reading the guilty look in your eyes.

And then the kicker comes:

“Yeah, I could tell by the profuse bleeding that happens when I start jabbing your gums with this sharp metal instrument of torture.” Ok, so the dentist doesn’t actually say that, but hey, if he can hear what he wants in my garbled mumblings, I should be able to hear his words the way I want to, right?

Anyway, let’s assume he’s probably right, and that regular flossing would actually eliminate the need for blood transfusions after each visit to replace all that was lost. From the scattered times in my life when I’ve been able to keep up flossing for a week or two, I know that after the first few days the bleeding does go down significantly. So maybe the dentist has a point, and not just at the end of those instruments of torture.

But how do you go from a lifetime of flossing for a week or two after each dentist visit and then forgetting about it completely until the next one, to actually making it something that you do each day without thinking much about it?

The trick is building a habit. No, it’s not easy. Yes, it will take time. If you need help, I recommend following these basic steps. There is no easy way to build a good habit, but you can make it easier. Choose a trigger, make it public, keep it simple, build anticipation, take your time, report to others.

I went through these steps over the last month or so, and now I’m flossing daily, even under the dental wire that is the last, permanent reminder that I had braces as a kid. Yeah, I was even more nerdy back then.

Anyway, I’m writing up this post because my next dentist visit isn’t coming for a few months. But if I tell my small corner of the internet how that visit will go it will help me stick to my habit until then.

So now, the next time my dentist asks, “Have you been flossing regularly?”, I’ll have a better answer for him:

“Huh-mhqhg”

Ok, you might not be able to tell the difference, but somehow the dentist will see the gleam in my eye, and know it is not the glint of guilt he saw at my last visit. 

In Drowning in a C of Interviews, Benjamin points out that he doesn’t see much value in doing interviews exclusively in C. Additionally, he believes that “we’re making candidates nervous about how well they know C, and focusing too much on how well they understand C’s semantics, but not really getting any meaningful indicators back in return.”

I have a different take on our interviews. Though I haven’t done as many interviews here at The Creek as Benjamin, throughout the last intern hiring seasons I saw a sum total of one candidate that actually suffered in an interview from not feeling comfortable with C. While I agree with Ben that allowing the candidate to code in the language they are most comfortable with won’t really change how many candidates get the “hire” decision, I do think there are some benefits to interviewing in C that we might lose.

It will help to step back and consider what we are looking for at Fog Creek. Our listing for a software developer position has this to say about what we’re looking for:

Right now we use C#, JavaScript, XHTML, CSS and even Wasabi (an in-house .NET language) to develop FogBugz. Kiln uses C#, JavaScript and Python, Copilot uses C, C++ and Objective C, and we have some legacy code written in VBScript. Tomorrow we may be using something completely new.

Whatever technologies, languages, or development environments you’ve been using, we expect you have mastered them in depth, and we expect that you will be able to master any technology, language, or development environment that we need in the future.

So we’re really looking for two things. First, do they code enough to have mastery in the languages they are most comfortable with. And second, can they use an understanding of broader principles to quickly learn new languages and technologies.

If we always let candidates code in their language of choice, we’ll be testing more for first quality. As such, we’ll want to dive deep into the way things are implemented, different ways of using the language to tackle the problem presented, and we’ll expect that the code is both syntactically and semantically correct. We would only get indirect hints as to the second quality - if the candidate can handle the problem itself then it means they can learn quickly on a small scale, and it most likely means they have a good grasp of the principles required to quickly get up to speed on a new language. But that’s not a guarantee.

If we did all our interviews in C, then, for most candidates we’ll be testing for the second quality - their ability to get up to speed in a language they don’t use on a regular basis quickly. They may be familiar with C, or may not, but they know we interview in C, so they can prepare for it. I specifically request that candidates code in C or C++, and while it is generally easy to see who is comfortable with it, it’s also fairly easy to tell who picked it up recently and didn’t have any problems with the question. I get excited about these candidates, because I know they can pick up new technologies quickly. I get (slightly less) excited about the ones who are obviously comfortable with C and do well on the actual question.

I can think of two interesting third alternatives to the “all in C” vs “pick your language” debate. One would be to work for a good mix of interviews in a language the candidate chooses with interviews in C, the “desert island language”. That would allow candidates to show off mastery of their tools of choice as well as the ability to work well outside of those tools. And since Ben stopped doing his interviews in C, I guess I’ll have to keep doing mine in C. Especially since his blog post will probably convince a couple other guys here to give up the C requirement. And besides, it makes life easier for me.

Another option would be to use a relatively obscure language, or one we make up (Wasabi, anyone?), in place of C for part of our interviews. We could test for mastery of the tools they use by having them code up the solution to a problem their own language of choice. Then we give them a different language, possibly with a fundamentally different paradigm (OO vs functional vs procedural, etc.), and have them code it up again. At this point, they understand the problem and solution fairly well, so they’ll be more fully demonstrating their ability to learn a new language and write code in it.

I’m not totally happy with either of these options, but they can certainly add to the discussion we’re having at Fog Creek about how to find and interview developers who want to help all developers make better software.

I hope you don’t have one script or set of scripts for dev builds and a separate script for official builds. For teams that have made the good step of setting up a dedicated build machine, this form of build script decay is probably more common than having different scripts for different configurations.

For example, you start getting close to shipping an actual product to actual customers, and so you setup a dedicated build machine. In getting build scripts to work you copy the build scripts the developers have been using, but you copy them, instead of just using them directly. This is because there are all kind of changes you need to make now that your code is being compiled and deployed to production, whether that is a web server, or wrapped up in a client installer, or copied to an embedded device. Once you get everything working, you then forget to refactor and move things back to a single script. In the best case, your official build script just calls the development build script. In the worst case, they both do a ton of similar, or exactly the same, things. But now you have to manage both scripts.

Even in the best case, where the official build script calls the development build script, there are probably improvements you can make. There may be steps in the official script that would be useful and/or necessary for developers to run also. Maybe not for every build, but probably for some builds. Taking some of what was done for the official build script might also improve developers life’s by making it easier for them to have multiple installations on their development box with less overhead work on their part.

Of course, in the worst case, with tons of duplicated script code, you’re going to have issues when developers fix the development build script but forget to fix the official one.  And vice-versa. But if you fix the scripts by combining them, and using appropriate command line parameters for any differences that absolutely need to exist, you get developers to help you maintain the official build scripts for free. By combining them you also encourage separation of deployment and build scripts, which will further improve your process.

One of the most natural ways that build scripts decay is when you create create separate scripts for different configurations of your build. This typically does not happen when configuration just means “debug” or “release”. That’s usually handled quite well with compiler and/or linker flags, or just using command line parameters or other ways of modifying run-time behavior. I’m referring instead to different configurations that you actually ship to customers. Here at Fog Creek we have hosted FogBugz and Kiln as well as licensed FogBugz and Kiln. Other products might have both “lite” and “pro” editions. Or maybe you have a separate configuration of your product for each customer. Or you’ve got an internal app that needs to be a little different at each of the company offices.

The obvious problem with having separate build scripts for each configuration is that you then have to remember to change multiple scripts when something common to all of them changes. And the common stuff will change more than the differences will. When your scripts are used to build every configuration, then you know going into a change that you need to consider each configuration. For example, it may very well be that you’re making a change that will only affect “pro” users, but then you’ll remember that you need to properly disable the feature in the “lite” edition.

Like most cases of repeating yourself, it’s fine to do it once, to understand the differences, for example when you first add a new configuration. But then you should refactor, eliminating the repetition by using common build scripting technology. Whether you’re using msbuild, make, ant, maven, or whatever, it’s quite straightforward to have a single script deal with multiple different configurations of the build, even when fairly major differences arise between them. Often, the largest differences between configurations are not build-related but deployment-related. So if you’ve separated your build and deployment scripts you’re already halfway there.

Of the many positive comments about the new FogBugz and Kiln releases, that was my favorite. Not so much because we did anything better than Twitter, as I haven’t heard of any problems with the rollout of their new version, but mostly because it meant…we did it! As the new build and release manager for Fog Creek, it was satisfying to get to this point—to have shipped two new major releases simultaneously with some interesting changes to our deployment process to handle new releases mixed in along the way.

I’ve mentioned before that your deployment scripts may have many different deployment targets: developer machines, the QA team, internal users, beta testers, and finally a full release. I’d like to document some of our processes here at Fog Creek for deploying FogBugz and Kiln. There are some things we do well…really well. And there are some things we’re still working to improve. Though my focus will be on deploying (rather than building) our products, I will mention our build processes when they affect deployment, and discuss some of the improvements we could make there too.

It’s worth mentioning now that all of the credit for what works in our build and deployment process goes to the great Fog Creek developers that set up this process over the last few years. And one of the reasons our releases went so smoothly this time around is because the current FogBugz and Kiln teams did a great job squashing all of the bugs that our QA team did a great job finding. Besides storing away knowledge in hopes of making some improvements going forward, I didn’t do much more than run a few script files. If that sounds exciting to you, read on.

Dev Deployment

Before anyone else can deploy a new version of FogBugz or Kiln, the FogBugz and Kiln developers do. Work began in earnest for FogBugz 8 and Kiln 2 when our summer interns arrived in late May and early June. Each of them needed to get FogBugz built and deployed on their machine and the Kiln interns needed to get Kiln deployed.

The set of steps a new intern goes through to get their FogBugz and/or Kiln deployments up and running is, sadly, not short. And it’s all manual. It involves getting the source code in the correct directories, installing a large number of tools (all at the correct version for development, of course), setting up virtual directories, preparing the database, building everything, etc. This whole process typically took one or two days, and not all setups were the same—something that occasionally bit our interns. Typical problems included getting the wrong version of some build tool, setting up the databases using the wrong named db instance, naming directories incorrectly, etc. As you might expect, once everything is set up, you don’t want to change anything, because it can be painful.

On the other hand, once everything is set up, doing daily development is pretty darn easy. Building FogBugz or Kiln in place will update your locally hosted instance of that application. There really aren’t any ongoing deployment needs, except when new features necessitate them. If you want two different development environments to make it easier to switch between bug fixing and feature development (we use one repository for the currently shipping version—bugfixes, and another for the “next” version—new features), you have to go through the same steps to create a new deployment as you did for your initial set up. It is no fun repaving your machine, but that’s not something the interns needed to worry about too much, and only occasionally causes problems for us full-time developers.

It’s worth noting here that development deployments most closely resemble a “licensed” deployment (i.e., the type of installation a customer uses when they install FogBugz or Kiln on their own servers). Setting up a development machine to behave like our “hosted” environment (i.e., the type of installation used for our FogBugz On Demand service where Fog Creek hosts everything for you) is much more complicated both to set up in the first place, as well as to update with new code. And honestly, doing the same for Kiln is almost unheard of.

In summary, initial deployment to development machines can be somewhat painful, but ongoing deployment of changes for testing purposes is easy and relatively painless. We can work to make initial deployment more automated, as long as we preserve the ease of ongoing deployment. There are additional development deployment features we could also add, such as deploying to multiple virtual directories from a single copy of the source, scripted deployment to virtual machines, and scripted deployment to a test hosted (as opposed to licensed) environment.

Ourdot

After two to three weeks of intense development, code reviews, and hallway usability discussions, it was time for the interns to show off what they had done to the rest of the company. First, they did demos of their features after lunch one day. We got to see the first iterations of cool new features like Kiln search, the new FogBugz wiki editor, all the wiki navigation changes, repository aliases, “big files” support in Kiln, as well as the much-improved FogBugz permissions model. The more important aspect of their demo was deploying their code to the FogBugz and Kiln instance we use internally, so that everyone else in the company could start using it on a daily basis…and start reporting bugs on a daily basis. This install of FogBugz and Kiln is affectionately known as “ourdot”.

Deployments to ourdot are unique. Unlike deployments to customers, our QA team, or dev machines, a separate set of scripts handles the work of building and deploying both products to ourdot, though they have much in common with our other build/deploy scripts. The ourdot install is a kind of cross between our licensed and hosted environments, and it’s often used as a testing ground for build and deployment changes as well as new features.

We continued deployments to ourdot every two weeks during major feature development and then more frequently as the interns transitioned into bug fixing mode in preparation for the beta release. At the start of the summer, deploying both FogBugz and Kiln to ourdot was a very manual process. Scripts did much of the work of building and copying files, but modifying the configuration on the servers in order to handle a new build had to be done manually, and involved 5-10 manual steps for both products. One of my first projects after taking over build and release responsibilities was to get both of these scripts running completely autonomously. All that I have to do now is just kick off the build and deploy process, and then deal with failures if they occur, which is quite rare.

Generations

In preparation for our first beta release, we needed to make sure that everything was working correctly in our hosted environment. The hosted servers use an extension of blue-green deployment to manage the switchover of accounts from one version to another and also to support having different accounts running different versions at the same time. Each version deployed to our hosted servers becomes a “generation” and each account is running on a specific generation. Individual accounts can be moved from one generation to another (later) generation, and scripts make it easy to move some subset of accounts or to change the default generation. Changing the default generation upgrades accounts on the old default generation and ensures that new accounts are created on that generation.

The reasons this deployment process works is because 1) deploying a new generation to the servers does not affect anyone on existing generations and 2) moving accounts from one generation to another does not affect other accounts. So we don’t have to deploy new generations at horrible hours of the night, or warn all accounts about downtime when only some are actually being upgraded.

While this process works quite well from our users’ standpoint, there are important changes we can make to improve our efficiency. The scripts that build both FogBugz and Kiln for deployment to our hosted servers are not the same scripts that do builds and deployments for ourdot, licensed, or dev machine installs. However, they do share a lot of common functionality that could be consolidated (and will be, in time). Another improvement we’re looking at is combining our generation numbers with our version numbers, so we don’t have to track two different numbers when filing bugs and discussing changes to the software. Moving accounts to a new generation can be surprisingly quick, but deploying a new generation still takes too long, and is something we want to improve. Also, FogBugz generations and Kiln generations are unrelated, despite the tight integration between the two products.

And now, back to the story of deploying FogBugz and Kiln! We went through a few deployments of new generations to get all the bugs worked out of the hosted deployment scripts. These “bugs” were typically things that had changed due to new features being added, but that had not yet been incorporated into our deployment scripts. Once we had a generation that built and deployed cleanly, we moved some testing accounts over so that our QA team could hammer it, and then proceeded to move alpha accounts over. Once the alpha accounts had been using the new code for a while we were ready to give the new version to beta customers.

Beta

As Joel described in Top Twelve Tips for Running a Beta Test, one key to a great product launch is a successful beta program. And the first thing you need for a good beta program are willing guinea pigs…um…I mean, users. For this rollout, we used an old web app that had been coded up for collecting beta applicant information for previous versions of FogBugz and Kiln. While not pretty, it got the job done, and when it didn’t we could always go straight to the beta applicant database and enter things directly if needed.

Another important aspect of running a successful beta is getting the word out, so we gradually let our customers know with announcements on the FogBugz and Kiln support sites, email, announcements on the blog, and via Twitter. We spaced these out with the goal of having applicants come in at about the same rate that we were approving them, in hopes that no one would have to wait too long before being accepted into the beta. The beta application allowed anyone to sign up for the hosted beta, and all current licensed customers to sign up for the licensed beta.

Once we had a stable generation for our hosted beta users to try out, we moved just 3 customer accounts to that generation and started watching for bug reports, support questions, etc. This may seem like an awfully small number, but we targeted customers who were already active on the support sites in hopes of getting good feedback. As that feedback started to come in and we were able to iterate on the issues raised, we slowly at first and then more quickly added more and more beta customers, roughly doubling the number each week until we had around 150 hosted beta accounts active, including some of our biggest customers.

Licensed Beta

Our licensed beta deployment process is probably the weakest link of our entire launch process. It did not break or cause major problems at any point, but it is almost completely manual. Basically, customers who signed up for the licensed beta are added to a list of potential beta users. We can then email any of these users with a unique download link that they can use to access the installer. The process of deploying a new installer involves manually copying each installer to the right location on our web servers, modifying the beta applicant database with the filename and size of each installer, manually verifying that the download links work for each installer, and then emailing all accepted beta customers a notification that the beta installer has been updated, and adding any new beta applicants to the beta.

Overall, the first step in improving this process was just to automate it. Since we were only accepting existing licensed users, we could also improve it by integrating it with our existing shop website (the website that tracks orders, maintenance contracts, etc.), where these customers were already able to download copies of the installer they had purchased.

Despite being so manual, and offering such obvious areas for improvement, we really had no major problems releasing licensed beta installers to our customers. Yes, it was kind of a pain, but it didn’t happen too often, and could have been a lot worse. Just as with the hosted beta, we initially informed just a few licensed beta applicants of the new download. Then as we received bug reports and fixed issues, we updated the download with new builds and informed more and more beta applicants until we had roughly 40 accounts using the new release, putting our total beta accounts (i.e., both hosted and licensed) at almost 200. At this point, the developers and QA staff on both the FogBugz and Kiln teams were feeling ready to start shipping to customers who had not applied for the beta. The final release was about to begin!

Leaking

Just as with the beta release, it’s helpful to roll out the final release to our hosted customers in stages. We call this “leaking the release,” which sounds kind of gross, but I like to think of it like sugar water leaking out of a bird feeder rather than, …well, you can come up with an appropriately negative metaphor yourself. The birds love it. As before, this gradual process helps us to catch issues before they have a chance of affecting everyone. Now that we’re dealing with 2-3 orders of magnitude more customers, it’s much more likely that we’ll run into performance problems with the new code. One serendipitous benefit of this type of release was that buzz about the new versions progressively increased over the course of the month leading up to the full rollout.

Initially we bumped 5% of our hosted customers up to the new release generation. In doing so, we found some performance issues in the background processes that do maintenance work on the FogBugz database. Taking that back to the FogBugz developers, they were able to quickly come up with a fix, and we verified it a week later when we bumped to 15% of customers using the release generation. The rest of the generation bumps went flawlessly up to 65% of customers on the new version just a few days before our final 100% bump was scheduled. Once two thirds of our customers had the new versions of FogBugz and Kiln we saw a noticeable increase in comments on Twitter and other outlets about the new version.

Even though there was still deployment work to do, this felt like a turning point to me. More of our hosted customers were using FogBugz 8 and Kiln 2 than were using versions 7 and 1 (respectively). Deployments had gone surprisingly well, with very few BugzScout reports popping up. The performance issues had been squashed, and no major hiccups occurred. If anything, it almost felt too easy. We did the bump to 65% on a Wednesday night. Thursday’s project was a big one: deploy the new website, release our licensed installer, and update our shop website to handle purchases of and upgrades to the new licensed version.

The Big Switch

Deploying changes to our website is a fairly straightforward process, though it does (currently) take too long. A simple script is run that re-indexes the site, copies the files to our webservers, updates the server configuration, and then checks a few key files to make sure nothing broke. It takes over an hour to run; about two thirds of which is spent copying the files over to the web servers.

But before we could pull the trigger, we needed to make sure that our shop website could handle purchases of the new versions of FogBugz and Kiln. Because major versions of our software are released relatively infrequently, this process has not yet been automated. So I got to get down and dirty with the shop database in order to add the necessary information about the new FogBugz and Kiln licensed products (and the various platforms they operate on). As part of this process we got the actual licensed installers for both products copied over to the web servers.

These installers had gone through the normal beta process described above, and additionally had been hit pretty well by our QA team. With the changes to our shop website tested by doing some fake purchases, we were ready to actually deploy the website…which went off without a hitch. It was fun to announce to the company that the fancy new website (designed by Jason and the rest of our design team) was up and to hear the excitement from everyone in the office. Although we weren’t totally done, this was the day that it felt like we had really shipped the product. New licensed customers were getting the new version, the website was up, and life was good.

100%

…but we weren’t really done…at least not yet. We still had one more generation bump to do for our hosted customers in order to get the remaining 35% of accounts using the new versions. This bump went even more smoothly than our previous ones. It felt so good to look at a simple DB query that showed all of our active accounts on the latest generations and know that we had shipped. It also felt pretty cool to know that we had done so by using the bug tracking abilities of FogBugz and the great source control provided by Kiln.

This final bump to 100% of hosted customers to the new version also bumped our new signups to start getting the new version as well. One thing we really want to fix the next time around is to decouple that switch so that new customers can start getting the new version earlier on in the release process. As things stood, there were three days when new hosted customers were getting the old versions of FogBugz and Kiln despite the fact that we’d already updated our website and were actively promoting the new versions.

Final Notification

Once more, it was time to iteratively “leak” the new version of FogBugz and Kiln–this time by notifying our licensed customers that there was a new version available for them to download and install. True, some of them would notice it on their shop page after having read or heard about the new version online. But most will just wait for the notification in FogBugz, then go to our website to download the new installers and upgrade. Those who don’t have support contracts will need to pay for the new version or get their support contracts up to date.

So, we notified some percentage of our licensed customers every couple of days over the next two weeks until we had 100% of them notified by early October. This helped to spread out the bandwidth load from customers downloading the rather large installer files, and allowed us the same wiggle room to fix any bad bugs in the installers or any licensed-specific code issues before it got out to all of our customers.

Final Thoughts

With all that done, we’ve already shipped our first set of bug fixes to customers—within a week of finishing the process of notifying our customers about the new versions. This is the low-hanging fruit that was important to fix, but not so important as to delay shipping FogBugz 8 or Kiln 2 to our customers. It’s easy at the end of a cycle to get caught up in perfectionism and think that every bug absolutely needs to be fixed, but after a certain point it’s more important to get it to your customers and find out which bugs they are actually seeing (you might be surprised). And besides, your customers might find much worse bugs to focus on that you never would have found in-house. Or they might just want some small feature that didn’t manage to make its way into the initial launch release.

As I said earlier, everything good about this process is credited to the entire FogBugz and Kiln team that got it in place over the past few years and many releases. Ben Kamens owned all of this process before handing it off to me, and did a great job getting me up to speed as I tried to digest all of the new information.

And now that we’ve shipped, we’re already working to improve the way we work internally so that we can get bug fixes and new features to our customers more quickly. You know you work with a great team when an impromptu post-ship meeting where the only agenda item is “Congratulations for the team!” turns into a great technical discussion about next steps to improve the product because everyone is so geared up about making it even better.

Do not store the output of a build in your version control system along with your code. Period. If you work on an interesting product, it is bound to be comprised of multiple components that all fit together in interesting and complex ways. For reasons you may have had no say in, some of those components are built and checked into the source code for use in building a larger product. Do not do this. It will bite you in the end.

First, you have the problem of interfaces. These components interact with the large product through an interface of some sort. Since you’re building the component and the larger product you control that interface. But if a developer has to build the component and check it in for it to be included in the larger product you can get into situations where the interface changes, but the component hasn’t been updated. Oh sure, you can setup rules that remind developers to rebuild and checkin a component if it’s interface changes, but now your developers just have one more thing to remember - and they will forget. Do not do this. It will bite you in the end.

Second, you have the problem of repository size. True, this isn’t as much of an issue for a CVCS such as Subversion. But for a DVCS, which you should be using to store your code anyway, this can cause problems over time. And if you’re patting yourself on the back for using a CVCS, just wait, you’ll come around eventually. And when you do, you’ll care about the size of your whole repository, not just a single revision. Cloning large repositories, made large by many large binary checkins, can take large amounts of time. Do not do this. It will bite you in the end.

Finally, you have the problem of build processes. Because of the need for built binaries to create your entire product, it only makes sense to checkin the binaries for a component once they’ve been built. But now your build scripts are doing more than just building. And you need to make sure the scripts aren’t checking in the binaries each time a developer runs the build on his or her box. And if you have a single script that actually builds the entire product, it is either lying (i.e. you still need to build and checkin the component first), or it’s doing more work than it need to (i.e. you are building the component, checking it into source, then checking it out again). Do not do this. It will bite you in the end.

So what do you need to do to solve this problem? First, make sure your build scripts are doing more work than they need to - build all components, checkin, checkout. Then, remove the checkin/checkout steps.

Of course, this does raise some issues. Now developers who could blithely ignore the component in question will need to make sure they have the code for it, so the build scripts will be able to operate correctly. If you’re doing daily builds, and storing the built files, you can optimize your build scripts in development environments to grab the built components from storage if they don’t have the code. This will work fine for developers who don’t change the interface between the two components. Developers who do change the interface will need the code for the component anyway. And of course, if it builds quickly, all developers can just clone the code and build it.