I work on a test automation framework at my day job. It's Django-powered, and there's a lot of neat stuff going on with it. I love building it!
Anyway, yesterday during a meeting, I got an email from a co-worker who seemed to be in a bit of a panic. He wrote that he accidentally deleted the wrong thing, and, being Django on the backend, a nice cascading delete went with it (why he ignored the confirmation page is beyond me). He asked if we had any database backups that we could restore, also curious as to how long it would take.
Well, lucky for him (and me!), I decided very early on while working on the project that I would implement a custom database driver that never actually deletes stuff (mostly for auditing purposes). Instead, it simply marks any record the user asks to delete as inactive, thus hiding it from the UI. Along with this, nightly database backups were put in place.
I'll be quite honest--I had a moment of fear as I considered how long it had been since I really checked that either of these two things were still working as designed. I implemented the database driver before I learned to appreciate unit testing, and I haven't made it to that piece as I've been backfilling my unit test suite (yet). As for the nightly database backups, I had never actually needed to restore one, so for probably the last year I didn't really bother checking a) that they were still being produced or b) that they were valid backups.
Thankfully, both pieces were still working perfectly. All I had to do was undelete a few things from the database, as I haven't made a UI for this. After doing that, I realized that one set of relationships was not handled by the custom driver. To fix this, I just restored the most recent nightly backup to a separate database and extracted just those relationships I was interested in. And it worked!
This is the first time I've really been bitten by a situation like this personally. I'm very pleased that I had the foresight to implement the precautionary measures early on in my project. I've also learned that I should probably keep up with those measures a bit better. I definitely plan to make some changes to help mitigate the potential for bad stuff to happen in the future. But it looks like I have a good foundation to build upon now.
TL;DR: unseen magic and valid database backups FTW.