Why Podia doesn’t use review apps anymore

With the recent discussions about moving away from PaaS solutions like Heroku, one of the common objections is “but what about review apps?”. So, what about review apps?

As a quick review (🥁🤣), Heroku has a feature that allows you to spin up a new “review app” for each git branch. This is useful when you’ve been working on a feature for a while and you need to get feedback from a designer, product owner, support team etc. They are kept up to date with changes to the branch and can have their own databases etc. We have them automatically routable by DNS, through automation with Cloudflare to add the DNS entries, and using self-signed SSL certs. They perform exactly like our staging app but are tied to a specific branch.

We used them pretty heavily a few years ago and it was definitely one of the features keeping us on Heroku. Recently though, Andy Croll mentioned the same objection and Kyle came back to Podia after a few years building Rewardful, which all made me pause to consider why we’ve stopped using review apps. We haven’t used them in so long that I’m not even sure all our setup functionality still works!

So after a little reflection, here are the changes in the past few years that have made reviews app irrelevant to us:

Feature flags

We’ve used Flipper to manage feature flags for a few years but I think our usage was initially quite tentative as we figured out how to best use them.

Today, I think feature flags have radically changed our approach to development. Gone are the long-running feature branches of yesteryear which we’d work on for a few weeks at a time.

Today, a typical project at Podia starts with a few database migrations but once any application code changes, it’s wrapped in a feature flag. Our projects are typically shipping PRs to production 3-5 times a week, and the main app is deployed about 5-6 times a day. We’ve had a long-running project for about 9 months now refactoring some core functionality and that project has been shipping a few PR each week to production. With so many changes, it’s just not possible to keep long-running branches around. All functionality needs to be worked in small chunks, finished in a few days, and shipped to production. Anything else will invite merge conflicts and re-work across projects.

Feature flags allow us to safely ship those small changes to production without worrying about their impact on our users, or whether it’s a coherent and final piece of functionality.

With such short-lived branches, there just isn’t any opportunity to spin up a review app.

Instead, we can enable the flag in production for our developers, for our product team, our designer, and eventually for the support team to get early feedback.

Occasionally we’ll release a feature to a percentage of users but this is pretty rare. Feature flags are mostly about enabling small pieces of work and short-lived branches rather than controlling the rollout of a feature. They can do both but controlling the rollout isn’t their main purpose or advantage.

Once we’re happy that the feature is complete and ready to be released, we enable the feature flag for all users.

This process of using feature flags have entirely replaced review apps for us.

I will always favour a process that allows us to iterate quickly and deploy frequently. That’s the number one priority for any small team; feature flags and short-lived branches do that. Review apps cause branches to live for longer, cause more conflicts, and discourage the team from deploying frequently—just say, no 🙅‍♂️

Designer runs local install

Another change has been with our designer who actually runs a local install of the app just like our developers. He’s able to pull in branches that are being actively worked on, and spin them up locally to review them—and even make changes to the views to correct any small discrepancies.

A designer that runs the app locally and ships code changes? Yes, get yourself one of those!

He can also review the changes behind the feature flag in production but for early feedback or specific visual tweaks, there’s nothing like having him work locally and just making the necessary changes.

Or we can also pair on specific problems over Zoom—no review apps required.

Cloudflare tunnels

We also use Cloudflare tunnels to route webhooks from external dev/test services (e.g. video hosting) to our local machines. This also means that we can use the same tunnels to make our local machines available anyone else for testing.

We rarely do this, and the tunnels aren’t always running, but it’s an alternative to review apps.

Difficulty replicating production

It’s relatively easy to build a staging or review app that runs in a production Rails environment. You can also pretty easily replicate the production database too. But from there the problems will start to get harder to solve.

For us, much of Podia’s functionality centres around Stripe and so we have Stripe tokens and references stored in our database for things like subscriptions and payments. When we bring that data over to a staging or review app environment, these tokens are useless unless we’re running also that app against a production Stripe API—which is a very very bad idea. In a test environment, those tokens won’t work so we’re unable to use much of the functionality in that review environment.

It’s even worse with things like files and video hosting as deleting the file in a review environment could also delete it in production too! 😨

So, over time, the value of testing in a review app has been reduced as we couldn’t completely replicate the production experience. It’s been much better, for us, to ship the changes to production and actually test them behind a feature flag. We’d need to do that any way.

In another app, with less production-specific data, you might not encounter the same issues or you could decide that your review apps will operate on the same seed data are your local development install.

So, what about staging?

Honestly, everything that has killed review apps for us can be said for our staging server too. We probably don’t need it.

Its main use these days is simply making sure the app deploys, boots correctly, and the database migrations run to completion on a clone of the production database. We never actually use it to verify functionality and it only exists because you can’t have a Heroku pipeline without one¹

Other alternatives

As we were discussing potentially moving off Heroku, we did discuss replacing review apps and the fairly reasonable response was to just give each developer their own machine. They would then be free to deploy any branch they wanted, and ideally could have a script to pull in production or seed data.

But, really, I would highly recommend you invest in feature flags, ship short-lived branches to production, and conduct your reviews there by enabling the flags.