For years, we’ve run an annual Hack Week. We blocked off a week in which engineers broke up into small teams and built some amazing proof-of-concept demos, which they showed off at an all company event at the end of the week.
Last year, we tried a more shipping-oriented style, with a focus on building features that could be live, in the product by the end of the week. Creatively, we named this "Ship Week."
It went so well that now we do it twice a year. You can even use it to expose prioritization gaps and execution issues in your regular engineering processes! Read on for the details.
Limitations of a Hack Week focused on “concepts”
The idea behind our traditional Hack Week was to give engineers the space to pursue high upside ideas – new concepts that could radically change our product. These are sometimes hard to convey in the abstract and can benefit from a prototype instead.
The deliverable at the end of the week was typically a demo, based on throwaway code. For ideas we wanted to move forward with, in theory the next step was to design and build a “real” version of the feature.
This created an annual disappointment: since building a “real” version of something takes much longer than building a demo-ready prototype, we couldn’t move forward with most of what people had created. Every year we’d have a demo day with five or six compelling ideas, of which we had the bandwidth to productize maybe one. The funnel from a conceptual demo into our product was necessarily narrow.
Previous Hack Weeks had led to some pretty exciting features, but, in the spirit of experimentation, we wanted to try something new.
Ship Week instead of Hack Week
The prompt for Ship Week is simple: work on whatever you think is valuable, but it needs to be “live” by the end of the week.[*1] Instead of concepts, we’re building real software that is ready to deliver value within a week.
That includes both customer-facing work and non-customer-visible work, so anything from new features to performance improvements to refactorings to dev experience projects are all in scope.
Engineers work independently or in small teams, sometimes also working with PMs, designers, or people from our solutions org. At the end of the week we still demo to the whole company.
We’ve loved the results. Each time there’s a long list of fantastic projects. Our first Ship Week included:
Group Suggestions, a feature in which Heap proactively suggests interesting ways to drill down into a funnel.
A new backend service that used a textbook data structure to reduce a batch export process from 4 hours to 10 minutes.
Removing some sharp edges in our install flow.
A staging system that lets you deploy any revision to a prod-like environment and QA it.
An internal Retool application for feature flagging, replacing a chatbot-based system.
A 4x speed improvement in one of our test suites.
One engineer singlehandedly removed ~1700 uses of a deprecated library.
A half dozen other projects!
We’ve noticed two phenomena that would have surprised us at the outset.
Projects have been no less ambitious. We were worried that a framing of “it needs to be done within a week” would push people towards more incremental ideas. In practice, a lot of the ideas have still been pretty bold and novel – just scoped more tightly.
Still plenty of “see it to believe it.” One of the motivating ideas behind our original Hack Week was that there are some ideas that are hard to convey in the abstract but easy to appreciate once you see a demo. One thing we’ve noticed is that, in some cases, this is true because the details are what make an idea special. The best way to communicate that kind of vision is with an implementation that’s close to real. So, surprisingly, Ship Week has been just as effective as a mechanism for conveying ideas you have to see to understand.
Also, in retrospect, we realized that some of our more successful Hack Week projects were actually shaped more like Ship Week projects. I.e., they were ready for beta use within a week, but they sat on a shelf until we decided to “officially” productize them. It would have been better if we had them in beta the whole time, gathering feedback, rather than stuck in a post-hack-week purgatory.
The worst part of Hack Week was that the first question after each demo was always “when can our customers use this?” The answer was always very disappointing: “whenever we decide to build a real version.” Now, we can say “it’s already available!”
Ship Week as a diagnostic
Ship Week is fun. It’s always exciting to see all the projects, and it makes for a nice celebration of the spirit that got a lot of us into software in the first place: it’s fun to build stuff!
Ship Week also serves a secondary purpose: it’s a diagnostic.
Every win is a prompt
Every “win” we get out of Ship Week is also a prompt: if this awesome, impactful project took only one week, why didn’t we do it before? Has every single thing we’ve done in the past few months been higher leverage than this project? Was not doing this project the optimal choice for the company? The honest answer is: probably not.
When a group of engineers puts together something really valuable in the span of a few days, it should prompt two reactions: 1) celebration, because we’re excited about it, and 2) introspection, on why that project had to come from a Ship Week.
Put another way: a successful Ship Week project suggests a gap in how we prioritize or execute our usual work.
Introspecting on the factors that have made some successful Ship Week projects possible is often illuminating in this way. For example:
A project that requires collaboration between engineers who don’t typically work together might indicate a lack of connective tissue across a team boundary. Or, the absence of a forum where people would have discussed the idea. Or, an instance of Conway’s Law, in which an organizational boundary manifests as a suboptimal technical boundary.
If a project delivers a win that, upon reflection, wouldn’t have been anyone’s responsibility, it might indicate an important area of work that doesn’t have a clear owner.
If engineers ship something we had all assumed would take much longer than a week to build, there’s something to be learned about how we assess the level of effort of new features.
If a glut of great projects all land on a common theme, e.g. improving dev workflows, reducing AWS costs, or improving the new user experience, that might indicate that, org-wide, we’re underinvesting in that area and leaving a lot of opportunity untapped. If there’s lots of low hanging fruit on a single tree, we should visit that tree more often!
The cost of process
Ship Week also shows us the cost of our processes. Most software processes exist for good reasons: to make sure we don’t waste effort building something that won’t be valuable, to make sure what we build is maintainable, to make sure all of our effort adds up to a coherent overall product, and so forth. These processes give a team long-term velocity at the cost of short-term speed.
How much speed are we sacrificing? A great way to find out is to spend a week sprinting, without our usual processes in place.
Ship Week projects at Heap should have the same bar for technical quality as our usual work. Code still needs to be reviewed, code still needs tests, and sometimes a design document is still called for. We don’t want to expose our customers to something broken or unsafe, nor do we want to spend a week creating a bunch of technical headaches for ourselves over the subsequent months. But beyond that, the process of building a project is as organic and unfettered as it can be: decisions are local and autonomous, and stakeholders are involved as-needed.
This doesn’t tell you the best possible speed, because process is only one factor that slows down work. Another is technical debt, and Ship Week doesn’t control for that – code that’s hard to work with doesn’t magically become easy to work with just because we declare a holiday. But Ship Week should give you a visceral sense for the maximum speed your team can achieve without the overhead of process, and what factors in particular are the most severe.
We had Garry Tan as a guest judge back in 2017, and he put it nicely.
It’s important to remind yourself every once in a while what “fast” feels like. (And, I think, how fun “fast” is!)
Lessons learned, and problems we haven’t solved
There are still some aspects we haven’t gotten right, and we iterate on this event in some way every time we run it.
One thing we’ve observed is that these projects typically don’t end cleanly at the end of the week. There is usually something to be cleaned up, documentation that needs to be written, or some other additional detail that bleeds into the subsequent sprint. We don’t have a great solution for this at the moment, and instead we generally plan around the expectation that this will be the case.
Another issue is that our platform teams often spend a lot of their Ship Week time supporting everybody else’s projects. For the team that owns our build / test / deploy infrastructure and our systems for managing AWS resources, Ship Week means dozens of engineers are doing something new at the same time and are in need of help and consultative support from them. That means Ship Week serves an interesting alternate purpose for a platform team: it exposes sharp edges, under-documented systems, and areas where we lack a good solution, because everyone runs into new problems at once. But it also means one team doesn’t exactly have the same Ship Week experience as everyone else. This year, we’re experimenting with a second Ship Week for the platform team, after everyone else’s, to make sure they have a chance to participate as well.
Best practices for Ship Week projects
We’ve noticed a few aspects that are common to successful projects:
Maintain continuity. It’s helpful to have at least some overlap between a Ship Week team and the team that will ultimately own the work. Otherwise, it’s hard to keep the momentum of the Ship Week project as we return to our regular work. We’ve seen promising projects stall after Ship Week when this wasn’t the case.
Involve design early. Some Ship Week projects benefit from a bit of lead time for preparation. Design, in particular, has been the most common reason. Even an hour jamming on an idea with a designer is a great way to make a feature a lot better.
Get input from product managers.The spirit of Ship Week is to get out of people’s way and let engineers self-organize. That means, generally, nobody can tell an engineer not to pursue something they want to pursue (unless it would be unsafe or unethical). That said, it’s still usually a good idea for engineers to ask relevant PMs if they have thoughts on their projects. Often they’ll have useful input. Or, there might be something imminent on the roadmap that is worth keeping in mind.
I’d love to hear what’s worked best at your company and otherwise nerd out about engineering processes! Find me on Twitter @danlovesproofs. And if this sounds like the kind of place where you might want to build software, we’re hiring!
[*1] Note that “live” often means “behind a feature flag, but ready to enable.” Especially for more ambitious projects, this mitigates risk.