Rock Hymas

“Hey, I have to tell you about the most awesome weekend release ever. It happened last Saturday, and –”.

“Weekend Release?”, you ask, incredulous. “Aren’t you supposed to be doing weekday deploys?”

“Yeah, yeah, I know. We agree, and we’ve taken a break recently as we make changes that involve enough risk that we don’t feel comfortable doing them during the week. But that’s not the point of this story. What I really wanted to tell you is about my crazy night trying to –”.

“But everyone agrees that deploying at night doesn’t make sense, right?”

“Um, yes. Well, not everyone. But we do, and that’s also on our radar. So, we had to update some test accounts to test some new performance improvements this coming week –”.

“Wait, so your test accounts are on your production servers? Don’t you have a staging environment?”

“AARGH! Will you just let me tell my story?! And, yes, we know we need to build out a staging environment. It is one of our top priorities at the moment.”

“Ok, ok, sorry. Go ahead with your story.”

“Thank you.”

So last week we got a FogBugz release ready with some bug fixes and performance improvements. The plan was straightforward: deploy it to our production servers on Saturday night and upgrade our test accounts so that QA could bang on it for a week. Barring release blockers, we could upgrade everyone the following Saturday.

On Saturday, while I was out running errands, it started to snow. I love the snow, and thought, “That’s cool, my boys will enjoy playing in the snow and it should all be melted by Halloween when we go trick-or-treating.” Those tweets and that thought apparently jinxed me. Once I was done with my errands in the early afternoon, I drove home through near-blizzard conditions only to get a text from my wife that the power had gone out. She also mentioned that a bunch of trees were down on our street. As I got off the freeway and drove down the hill towards home, I was following a bus that was swerving all over the road to avoid low-hanging branches and a tree that had fallen across half the road.

Our releases don’t usually look like this

After a few hours it became apparent that we wouldn’t be getting power back in time for me to login to our VPN and take care of the release. So I started contacting some friends from church to find someone who had power and wifi and the willingness to put me up for an hour or so at 10pm. Brandon, Ian, and John all said they could, but Brandon was first, so I let our sysadmins know the release was still on (they like to be alerted to these things, you know). It got dark at about 7, and we put the kids down early, wrapping our one-year-old Liberty up in a snowsuit so she’d be warm enough all night. At 9:30, I packed up my laptop and went out to make the 15 minute drive to Brandon’s home.

As I left I had two roads I could take: the freeway, which involved some pretty steep hills; or driving through town (three towns, actually) along a road with no significant hills. Because the snow was still coming down hard and sticking pretty well I decided to avoid the hills. That was a mistake. The first few minutes were pretty uneventful, but it was fun to drive through downtown Berkeley Heights and into New Providence in complete darkness. No streetlights, no traffic lights, no homes lit up, no businesses, no grocery stores. Ninety percent of the other vehicles on the road were snow plows (doing a great job, by the way).

The drive through town was also along a heavily tree-lined road, which made the darkness creepy in a “Halloween is cool” kind of way. But as I travelled this road it became clear that I had taken the wrong road. Trees had fallen across the road at pretty regular intervals. The snow plows (or kind citizens) had cleared a single lane path where these had happened, though it was often just barely wide enough for my compact Honda Civic. I went ahead and drove past a few road closed signs and one unmanned roadblock.

Not in New Jersey, but you get the picture

Soon after that I realized why they were closed: The downed power lines. Most didn’t block the road at all, but one was just hanging directly in my path and scraped over my car before I had time to notice it and react. That got my adrenaline running. At another point I got to play chicken with a snow plow that was driving the wrong way in my lane.

I finally made it into Summit, where there was power in some parts of town, and pulled into Brandon’s driveway, but of course, the car got stuck as soon as I left the road, so I was blocking the exit for both him and his neighbors. He and I puzzled over that for a couple seconds, then realized his neighbors wouldn’t be crazy enough to go driving in the storm. He set me up in his basement, his wife Becki offered me something to drink, and they went back to watching TV.

After a little over an hour the release was done, and it was time to head back home. This time, I’d learned my lesson. Throughout my drive to Brandon’s the roads had been clear, thanks to the plows, so I knew that it would be an easy ride home on the freeway.

And it was.

Until I got off the freeway.

I took an exit that would avoid the steepest hill, just in case. Turns out that was a good choice, as I found out Sunday that it was completely blocked by fallen trees. It did mean I had a bit further to drive through town, maybe a mile and half instead of just a mile.

And that’s when the fun really started.

The off ramp had a tree across it that had obviously been chain-sawed through in the last couple hours. Just around a bend, I came across a cop car with lights flashing, blocking the road I would have taken. So I decided to risk a side road to get around the blocked off section. Now I really was in rural New Jersey, and now I got to drive over some branches that weren’t quite big enough to block my Civic. I finally made it back to the road leading home (less than a mile to go!), but just a couple blocks further ran into a road block that wasn’t one I could drive around. Power lines and police tape were draped across the road right at car level.

So it was off into side streets again, on the very steep hills I had wanted to avoid. I still wasn’t slipping around on the snow much, but had to take a couple detours anyway (trees across the road). Finally I started heading back down the hill.

The end was in sight!

So, of course, I got stuck.

One more downed tree. At the worst spot. The snow was deep enough that turning around wasn’t an option. Backing up didn’t work. I’d finally gotten my Civic stuck in the snow only about half a mile from home.

Fortunately, I’d left home with a flashlight (so I could see in the garage). I broke that out, bundled up, and started the hike home. My flashlight was pretty weak compared to the car headlights, though, so I just barely avoided walking into a downed power line on the hike down the hill. Once down the hill, I got to pass a powwow of power crews and cop cars at the last intersection before my home where they guided me past more downed lines. From there on out I walked down the middle of the street in complete darkness illuminated only by my flashlight.

Once home, after taking over an hour to make the 15 minute drive, I jumped in the shower to warm my toes back up before getting under the heaps of blankets my wife had piled on our bed.

Follow-up (Monday): The power is still out, and the city is telling us they can’t even give us an estimate for when we’ll get it back. On Sunday we drove for a few hours into southern New Jersey to get the last remaining generator at the only Home Depot who hadn’t sold out the first hour they were open. We’re now using it to power our fridge, freezer, and DSL modem, so I can work from home, since the trains into New York aren’t running. Kami and the kids are doing homeschool in the dark, which the kids think is just awesome. I’m writing this post in a room that is about 60°F, while listening to the dulcet roar of our generator right outside my window.

Follow-up (Friday): The power finally came back on Thursday evening at about 5pm. We spent the evening cleaning up our home, doing laundry, and generally trying to get our lives back in order. And guess what? I’ve got another release to do on Saturday!