Rolling back a deploy on Prostack Cito

Know what you’re undoing

A bad deploy usually breaks one of three things, and each has a different way back:

• The code. A bug, a bad dependency, a build that went wrong. Git has the previous version, so this is the easy one.
• The database. A migration that dropped a column or rewrote data. Reverting the code doesn’t undo it.
• Files the app wrote. Uploads, generated files, anything outside git. Only a backup has the old copies.

If you’re not sure which, the logs usually say. In the site’s settings, Logs shows the Nginx error log live, and Audit Logs (from the cog next to your username) lists every Git pull with its time, so you can see exactly when the bad version went out.

Revert and deploy again

The safest way back is forwards. Revert the commit that broke things, push, and press Deploy latest:

git revert HEAD
git push

You can ask your agent to do it too: “revert the last commit and push” is a safe request. Don’t try to roll back by force-pushing an older commit: Deploy latest only moves the site forward along its branch, so a rewound branch won’t deploy. Revert instead.

For a Node.js app, restart it after the deploy so the old code is what’s running:

passenger-config restart-app ~/public_html

Check out a known-good commit

When it can’t wait for a revert to go through review and CI, sign in over SSH as the site’s user and put the last good commit live directly:

cd ~/public_html
git log --oneline -10            # find the last good commit
git checkout <last-good-commit>
npm run build                    # if the app is built on the server
passenger-config restart-app ~/public_html

The site is now running that commit, detached from its branch. Fix the problem in git as normal; the next Deploy latest switches the site back to the branch it’s connected to.

Restore from a backup

Every Prostack Cito server is backed up off-site each night, and the backups are kept for seven days. They cover site files, MySQL databases and the server’s configuration.

Files you can restore yourself. In the site’s settings, open Backups, pick a snapshot and click Browse. Tick the files or folders you want, then Restore selected, and choose whether to skip files that already exist or overwrite them. If you only want to compare, download a file or folder instead (folders come down as a zip).

Databases are restored by our support team: ask, and say which database and which night. Don’t try to restore MySQL from the snapshot files yourself. The backups are taken from MySQL’s data files on disk, and copying those back by hand will break MySQL on the server.

A nightly backup can be up to a day old, so anything written since is lost on restore. That’s why the next section matters.

Before the next risky deploy

Most painful rollbacks are about the database. A few habits make them rare:

• Export the database before a migration. phpMyAdmin’s Export, or mysqldump over SSH, gives you a copy from just before the change, newer than last night’s backup.
• Write migrations that can run backwards. Laravel and most Node migration tools support a down step. Test it on staging.
• Note the commit you’re replacing. Run git log -1 before you deploy, so the known-good commit is to hand if you need it.

Try it on staging first

Any site can be cloned to a staging copy, files and database included. Cito creates a new database and user for the copy, and you give it its own domain, such as staging.example.com. Deploy the risky change there, check it on the staging URL, then deploy to the live site.

A clone doesn’t copy cron jobs, on purpose, so a staging copy won’t send your scheduled emails twice. It does copy the live site’s .env, so check that before you use it. Environment variables on Prostack Cito has the list of what to change.