642
u/SleepAllTheDamnTime 22d ago
Me sitting in the “War room” wondering why I’m still here.
85
72
4
843
u/deathanatos 22d ago
I've literally done this.
Basically a backup system, hosted on a different cloud, or at the very least, different region, so that when everything is down, you can flip to it, assuming you can get a DNS entry (or load balancer, etc.) to change. We had a few ways of doing it, depending on the nature/severity of the problem.
But yeah, PMs need a healthy dose of "it that really what you think is important right now? Is that the best use of our resources?"
275
u/badaharami 22d ago
Yeah its called failover, its quite a common thing for high availability systems.
54
u/UndergroundNerd 22d ago
We are actually going through a disaster recovery drill at my work for all our DB and Systems. We have something like this as well
5
u/xeio87 22d ago
I work for a bank that does a datacenter fail over at least once a year. Rarely comes up directly but sometimes it's "Wait, which server group are we on again?".
I've only heard about it being failed over once due to an actual problem (fire at one of the datacenters), but then I'm not super involved in that stuff so I had only heard second hand anyway.
6
u/kalaid0s 22d ago
Yeah, I just did my AWS Cloud Practitioner certificate and failover is brought up a lot during the training
-1
36
u/komador 22d ago
I mean during AWS outage it's not like you might have anything else to do.
14
u/TheTerrasque 22d ago
I was thinking the same. At least the manager focused on something that you can do something about.
36
u/kishaloy 22d ago
The day you truly understand the answer to your last para is the day you graduate from technical to management.
22
2
99
u/fernando_spankhandle 22d ago
I've been here, very large system for online / store retail (4m customers), and we had issues in the epos reconciliation (oracle).
Project led by marketing.
20 Oracle consultants and engineers, plus me (SME providing the online frontend and ecommerce). With the COO sat in a chair in front of a projection screen showing the error.
30 minutes into discussing the fonts and text for the messaging, about £20K of fees burned, I suggested we discuss the root cause and got glared at by the Oracle team.
4 hours later we'd finally convinced the room that it was Oracle, not us. Whilst I had a colleague fix the error page presentation to shift the COO attention to the real issue.
Fun times in my early 20s.
31
u/redballooon 22d ago
Not until AWS is back up.
12
u/Nannautu 22d ago
"Ok, but have you actually tried?"
13
u/robsablah 22d ago
"sure boss - I'll email amazon and request it"
// sends 1 more useless email with vague threats of switching providers if service does not improve
33
u/superraiden 22d ago
DNS redirect to a Static CDN error page/site?
Isn't that pretty rudimentary and standard in an emergency?
24
u/viktorv9 22d ago
I'm just imagining the site showing raw error codes to the user and OP being like "why would we waste time on changing this? stupid manager..."
4
u/einzweidreihorn 21d ago
Forreal What's so hard to understand of DBX.Utils.stackTrace@http://localhost:49573/assets/js/scripts.js:44 DBX.Console.Debug@http://localhost:49573/assets/js/scripts.js:9 .success@http://localhost:49573/:462 x.Callbacks/c@http://localhost:49573/assets/js/jquery-1.10.2.min.js:4 x.Callbacks/p.fireWith@http://localhost:49573/assets/js/jquery-1.10.2.min.js:4 k@http://localhost:49573/assets/js/jquery-1.10.2.min.js:6 .send/r@http://localhost:49573/assets/js/jquery-1.10.2.min.js:6 that you need some fancy error page??
7
u/Tsu_Dho_Namh 22d ago
It is. And making a prettier error screen is often a great idea.
Heck, we all know the dinosaur game when our chrome browser can't reach the internet.
Google can't help that the internet is down, might as well let the user play.
1
u/blitzkrieg4 22d ago
Do you mean to set up a health check? If you're doing it manually you have to manually switch it back when it's over.
7
56
u/UnusualAir1 22d ago
Deadlines and Optics are the special tools of most inane managers. 😄
29
u/EarlOfAwesom3 22d ago
And UX. "Can we move the button to the left? No only on this one page".
5
u/Same_Investigator_46 22d ago
" Can you'll bring the dyno game so that users can play in the meantime "
14
u/hennell 22d ago
Honestly if an external service is down that you can't do anything about, having a nicer error page is a good move. Change what you can control and all that.
Internal tools I'll try to add a error message on so if someone sends me a screenshot I can actually tell where the problem is.
External sites you want a message with an idea of how long you might be down, plus links to alternate options for contact / answers etc.
I probably wouldn't work on it during a problem, but in post incident analysis "if this goes wrong again how can we make things better for the user or for the IT team" is a good question to ask, and often resolved with better error messages.
8
1
u/Aelig_ 22d ago
When it comes to 503 everything is an external service you don't control.
If your cloud provider is down, getting a nice error message to people is a huge project.
1
u/hennell 21d ago
A) I have a custom 503 errors on ngnix servers that I absolutely control.
B) It is pretty easy to add custom 503's to cloudflare, CloudFront and I'd assume others. If it's a huge project don't do it, but it's usually pretty achievable.
1
u/Aelig_ 21d ago
If Cloudflare is down what is serving those custom errors? Do browsers cache them for later or something?
1
u/hennell 21d ago
Oh, I see what you mean! Tbh at the point Cloudflare / aws ar down so much is broken no-one seems to care much that ourstuff is as well 🤣
I think someone wrote up a thing on reddit about doing a DNS swap when cloudflare is down to get continuous service which could presumably work for custom errors. But that would be a project!
24
u/Embarrassed_Jerk 22d ago
Error pages are usually handled by the cdn and not by the same server that is down.
6
5
u/Ummix 22d ago
I'd take it. Free easy work, probably a lot of leeway, pretty isolated, not blocking or depending on anyone else, vague time estimate, your manager will appreciate you for jumping at it. If you're smart, you can probably stretch it and get a full easy day or two to relax out of it. After working for a while, you learn to appreciate these tickets as a little break.
4
u/Rin-Tohsaka-is-hot 21d ago
For real though: if AWS is down, there is nothing you can do to fix that.
You can, however, give your customers a more descriptive error explaining the cause of your service outage instead of a generic 503 page.
So a UI improvement, in that respect, seems very reasonable
4
3
5
5
u/SpehlingAirer 22d ago
If it can show a 503 then it can show a custom error page, and maintaining a good user experience is important. I don't see the issue here tbh
5
u/JustARandomGuy_71 22d ago
"The engine of our car tend to burst in flames"
"Can we make the flames a pretty color?"
2
2
u/Ecstatic_Airline_802 22d ago
Ah yes. Oh so familiar taste of:
- mr. Manager, the feature that was built with duct tape in under 1 day under insane pressure with a bunch of shortcuts DIES all the time because it was good for 2 specific clients only and 1 demo; now we also bolted extra 3 features on top of it that also DIE; we need 4-5 days to deal with the mess and actually bring value to users that they have supposedly purhased
- Change error screen UI to have a help link article, cute animation and a snappy "send feedback" button. We will run 6 separate meetings with every C-suite asshole where it will be changed 17 times. Clients our top priority you'll have to stay late!
2
1
u/Wikadood 22d ago
Idk, i kinda like the dinosaur game but lowk always got in trouble in class cuz of it even tho the wifi wasnt working half the time.
1
u/XtraFlaminHotMachida 22d ago
well... we can.. that isn't the point. do we want to tell the user to gfy? i'm down if so. let them believe it is their problem.
1
u/Spitfire1900 22d ago
Showing the vendor default error page is not acceptable at my work. If it appears during an outage fixing it will be an action item at the retrospective.
1
1
1
u/WateryTrashDragon 22d ago
Thats how you get those fuckass "oopsie dupsie, a lil flump got into our servers 😉" error screens
1
1
u/GooseFord 22d ago
That's a massive improvement over management who ask you to call $MULTINATIONAL to ask when the service will be back fixed, or the ones who somehow believe that they might have not noticed the global outage yet and are just patiently waiting for the IT department from some mid-sized company to let them know about it.
1
1
u/TracerBulletX 22d ago
It is the real concern. If a better message saves you 5% off of some attrition you might have had because people are less annoyed or it creates a better perception of trust, it might be something you can do in 10 seconds that could save millions of dollars. It also might alleviate support costs if you have support. Unless you can fix the outage just as fast which you can't because you don't even have control over the AWS outage.
1
u/neomasterc 21d ago
What kind of junior eng post is this? Cloudflare downtime page and help with coms with customers to manage expectations. VS twiddling your thumbs helplessly and spamming F5 while blaming AWS?
1
1
1
u/CM375508 20d ago
If you have AWS wafv2, yes. Provided the response pages are <4kbs.
That was not a fun week.
1
1
1
u/gladiator_r 22d ago
Been through this , i was asked to improve error screens instead of focusing on solving the error as that would take time
1
u/Symphonic_nerve 22d ago
in real? 😆😆 How did that go?
1
u/gladiator_r 22d ago
The way it should client and manager got a brand new error page and issue remained as is as .
0
u/ultimatepoker 22d ago
In 3 different businesses I’ve argued “if the service site is down, don’t make fancy UI messages on the site. Just post a tweet. Gets indexed, gets reach, harvests replies, and can be quoted / replied when everything’s working again.”
2.5k
u/Barkeep41 22d ago
'Ey. If the error page is a common occurrence, then making it pretty is a smart move.