How we ship code at Townsquared
This post was first published on the Townsquared engineering blog at https://townsquared.com/blog/engineering/ship-code-townsquared/.
Townsquared is the local network for small and medium businesses. As a hub for many daily conversations among local business communities, it’s important for us to have a stable product for our users that still allows rapid iteration by our development teams. As a result, our development process aims to balance development speed with maintaining robustness and keeping technical debt manageable.
The following are some of the key pillars of our product development process that allow us to meet those goals.
Parallelizing work. Our development teams use Pivotal Tracker to organize work. Before implementation begins on a project, engineers and product managers split up work into a number of interdependent, sequenced Pivotal stories. The goal of thinking through the work in advance is to parallelize it; by thinking about what components need to be built, and in what order, engineers can work on multiple decoupled pieces of work at the same time.
Decoupling tasks often requires writing API endpoint stubs so that the front-end and back-end development can proceed in parallel. A nice side-effect is that this forces engineers to clearly define the input and output requirements of every new API endpoint before implementation begins, making explicit the contract between the client and server. Before a project begins implementation, a small group of engineers is tasked with performing research on the technical implications of the design; defining the client-server contract is a core piece of this research.
Test servers and design approval. User-facing features require approval by a visual designer before they can be merged and deployed. We have a fleet of testing servers that are heavily used by the front-end engineering team; work in progress can be pushed from a local branch to a server, and is available to anyone in the company (but specifically designers) to review. Front-end engineers work closely with designers throughout feature implementation: it is not uncommon for a single feature to “ping pong” several times between designers and engineers before it is design approved.
Requiring design approval helps ensure that features are polished and easy to use before real users see them.
Code reviews. Engineers can open pull requests (PRs) on Github after their work has been design approved, or in parallel with the design approval process. PRs can be assigned to a number of different engineers depending on the area of change. The process of getting PRs approved is fairly informal: engineers can notify their reviewers in an internal Slack channel, and most engineers review Github daily for outstanding PRs assigned to them. Most PRs are reviewed within a day.
A PR must be approved by at least one reviewer before it can be merged. Approval is usually signaled by the standard mechanism of adding a comment with the acronym “LGTM” (Looks Good To Me) to the PR.
Automated tests. We run an extensive suite of automated front-end and back-end tests using CircleCI on every PR. Our back-end code is written using a combination of Ruby on Rails, which is tested using RSpec, and NodeJS, which is tested using mocha tests. On the client side, our website is a single-page AngularJS application, which is tested using jasmine. Most of these tests are unit tests, while some are integration tests that test interactions between components of the application, or multiple services.
Staging servers. Once a PR is merged into master, the full suite of automated tests runs on the head of master. If the automated test suite passes, the master branch is deployed to a staging server. This means that anyone in the company can try out the latest stable version of the product at any time by going to our staging server. We do this across our main user-facing application, as well as the many microservices that power various facets of our product.
Feature flagging. A core part of our development philosophy is to release small units of code to production. Smaller changes are easier to review, easier to test, and reduce instability by not changing multiple parts of the system at the same time.
In cases where the full implementation for a feature spans multiple PRs, we use feature toggles to hide the feature from real users before it is ready to be seen. A feature toggle can be enabled by anyone on the development team using a special cookie name while the feature is in development. When the feature is ready to release, the check for the feature toggle is simply removed, exposing the feature to all users.
Daily deploys. We deploy the main user-facing application to production daily. Deployment involves looking at the results of the automated test suite, and also testing various parts of the product manually. We use a deployment checklist that covers all major areas of the product. Our Cucumber test suite is expanding to cover the entire deployment checklist, which will allow us to switch to fully automated deployments once coverage is complete.
On-call support. All engineers at Townsquared participate in a daily rotation on-call schedule. The on-call engineer triages emergent issues, and assigns the resulting bug tickets to engineers or product managers. On-call engineers also investigate issues reported from the field, narrowing down reproduction steps and adding useful information to tickets.
On-call engineers are expected to decide, in consultation with product managers, when an issue needs to be fixed immediately. If so, they either fix the issue themselves, if they are knowledgeable about the relevant area of the code, or they escalate the issue through an escalation chain till they reach someone who can fix the issue immediately. We have separate, defined escalation chains for various areas of our system.
In general, the on-call program has been successful at reducing the time to resolution for critical bugs, as well as at allowing engineers who are not on call to stay undistracted by emergent issues.
The development process at Townsquared is designed to encourage rapid development while minimizing product instability. We are fairly happy with our current process, but are continually evaluating and improving the way we develop software. If you enjoy working in a fast-moving team that is constantly evolving the way it works, check out our job openings and drop us a note!