What We Learned From 15 Years of IT Disasters (And How to Avoid Them)
When you’ve been in the managed IT business since 2006, you accumulate stories. Not the kind you share at dinner parties — the kind that make IT professionals wake up in a cold sweat. Over nearly two decades, we’ve seen just about every IT disaster lesson imaginable, and we’ve distilled them into hard-won wisdom.
These are the IT disaster lessons that shaped how we work — and the ones that could save your business from making the same mistakes.
Lesson 1: The Backup That Wasn’t
Early in our journey, we inherited a client whose previous provider had set up “automated backups.” The business owner slept soundly, confident their data was safe. Then their server failed.
When we investigated, the backup job had been failing silently for eleven months. No alerts were configured. No one had tested a restore. The backup was a ghost — it existed in name only.
What we learned: A backup isn’t a backup until you’ve successfully restored from it. We now mandate regular restore testing for every client. No exceptions.
Lesson 2: The Password That Opened Everything
A Sydney accounting firm called us in a panic. An attacker had accessed their systems and was exfiltrating client financial records. The entry point? The managing partner’s email account, which used the password “Autumn2019!” — the same password they used for everything.
There was no multi-factor authentication. No conditional access policies. Nothing between the attacker and the keys to the kingdom.
What we learned: Multi-factor authentication (MFA) isn’t optional. It’s the single most effective security measure a business can implement. Every account. Every user. Every time.
Lesson 3: The “Quick” Server Room Tidy-Up
Why Physical Infrastructure Still Matters
A well-intentioned office manager decided to “tidy up” the server room one weekend. They unplugged “unnecessary” cables, reorganised the rack, and even turned off a “noisy” unit that turned out to be the primary firewall.
Monday morning: 47 employees arrived to no internet, no email, and no access to their files. It took the better part of a day to untangle what had been rearranged.
What we learned: Physical access controls matter as much as digital ones. Server rooms should be locked, labelled, and documented. If you wouldn’t let someone walk into your bank vault, don’t let them walk into your server room.
Lesson 4: The Ransomware That Came Through a PDF
One of the most devastating IT disaster lessons we’ve witnessed involved a conveyancing firm. A staff member opened what appeared to be a settlement statement from a known contact. The PDF contained a macro that deployed ransomware across the network within minutes.
The attackers demanded $80,000 in Bitcoin. The firm’s backups were stored on a network drive that was also encrypted. Recovery took three weeks and cost far more than the ransom in lost productivity and emergency IT services.
What we learned: Air-gapped or immutable backups are essential. Your backup must be isolated from your production network. If ransomware can reach your backup, you don’t have a backup — you have a liability.
Lesson 5: The Cloud Migration Shortcut
Why Rushing to the Cloud Backfires
A growing professional services firm wanted to move to the cloud quickly. They chose the cheapest provider, skipped the planning phase, and migrated over a weekend. By Tuesday, they’d discovered that half their legacy applications didn’t work in the new environment, their internet bandwidth couldn’t handle the load, and their phone system had stopped working entirely.
What we learned: Cloud migration requires proper assessment, planning, and staged implementation. There are no shortcuts. We now run a structured migration process that includes application compatibility testing, bandwidth analysis, and user acceptance testing before any cutover.
Lesson 6: The Insider Threat Nobody Saw Coming
A disgruntled employee at a small manufacturer spent their final two weeks systematically downloading client lists, pricing models, and proprietary designs. When they resigned, they took it all to a competitor. The business had no data loss prevention tools, no access auditing, and no offboarding process.
What we learned: IT disaster lessons aren’t always about technology failing — sometimes they’re about people. Proper access controls, activity monitoring, and structured offboarding procedures are critical. When someone leaves, their access should be revoked immediately, not “when IT gets around to it.”
Lesson 7: The Update That Broke Everything
A software vendor pushed an update that was incompatible with a client’s line-of-business application. The application crashed, taking the database with it. Because the update was automatic and happened overnight, no one knew until staff arrived the next morning to a completely non-functional system.
What we learned: Patch management must be controlled, tested, and staged. Critical business applications need update policies that include testing in a non-production environment before deployment. Automatic updates on servers are almost never appropriate.
Lesson 8: The Insurance Claim That Was Denied
When Compliance Gaps Cost More Than the Incident
After a data breach, a client submitted a cyber insurance claim worth $120,000. The insurer’s investigation revealed the business hadn’t maintained the security standards required by their policy — specifically, MFA wasn’t enforced on all accounts and their endpoint protection was out of date. Claim denied.
What we learned: Cyber insurance isn’t a substitute for proper security. Insurers are becoming increasingly stringent about requirements, and they will investigate before paying. Your IT environment must meet the standards you’ve committed to in your policy.
The Common Thread: Prevention Is Always Cheaper
Every one of these IT disaster lessons has something in common — they were preventable. Not with expensive technology or enterprise-grade solutions, but with proper planning, regular maintenance, and expert guidance.
The businesses that weather IT storms aren’t the ones with the biggest budgets. They’re the ones with:
- Tested, verified backup and recovery plans
- Proactive monitoring and maintenance
- Security fundamentals in place (MFA, endpoint protection, access controls)
- A trusted managed IT services partner watching their back
Don’t Learn These Lessons the Hard Way
We’ve spent nearly two decades learning these IT disaster lessons so our clients don’t have to. Whether you need a comprehensive IT health check, a backup strategy review, or ongoing managed IT support, we’re here to help.
Infraworx has been protecting Sydney businesses since 2006. Call us on 1300 277 211 to find out how we can help you avoid the disasters we’ve seen — and be ready for the ones nobody can predict.
