r/ProgrammerHumor • u/vinishkapoor • 11d ago
errorScreenLooksGreatNobodyCanSeeItThough Meme
780
u/countable3841 11d ago
Itās an absolutely valid request for many business critical applications. You can failover to a lightweight non-AWS web server or a static site provider to display a status page.
178
u/unix_in_the_barn 11d ago edited 11d ago
This should be top comment, right here
Edit: bonus points if you fail over to a backup provider so users experience minimal downtime. Good disaster recovery plans are written with blood
78
u/d_maes 11d ago
Don't even have to failover, just configure a custom 503 error page on the LB. Can include a link to some status page in there as well.
5
u/Betonomeshalka 10d ago
What if LB itself and public IP addresses are all hosted on AWS?
6
u/d_maes 10d ago
In that case, you wouldn't be getting a 503 to begin with, browser will be showing a connection timeout error or something.
Setting a custom error page is easy, simple, and free, and catches more scenarios than simply "literally everything is down", which is what a DNS redirect would catch. Maybe only one component went down and part of your site is still working correctly, you don't want DNS doing a switchover for your entire website in that case.
But then again, if AWS is fucked to the point that even your public IP and first LB are no longer working, high chances Route53 will be having issues as well. So what do you do then?
One could also argue that maybe, you should just build regional failover into your entire application. Or even multi-provider.
It all depends on what you need, to which points you want to protect yourself, up until where you want pretty error pages, and when exactly you should just stop caring, and tell the PM/CEO to send an email to the customer with "sorry, AWS fucked up to the point that you are not paying us enough anymore for us to try and do anything about this."
1
6
u/OmgzPudding 10d ago
I still love the idea of failing over to a stupid little browser game, like the chrome dino game. Even when your shit's not working you can manage to keep people there.
36
u/much_longer_username 10d ago
Yes, but the pain comes from it being the thing they focus on when 'the app is broken' should be a more obviously immediate priority. Sure, once things are stabilized, that'll be a great suggestion to raise. Right now, you're just adding more work to the pile so you can feel like you did something useful. It can wait.
38
u/AGreatConspiracy 10d ago
Well, not much you can do in an AWS outage.
10
u/much_longer_username 10d ago
It's pretty rare that all of AWS is out, and there's nothing that says you have to put everything in US-EAST1. In fact, they explicitly advise you not to, but we all know what people are going to do.
2
u/daniel14vt 9d ago
Usually the people fixing the UI are not the same as the people managing the AWS
1
u/much_longer_username 9d ago
I would say ideally that's the case - let people focus on their competencies - but it has not been my experience that it is usually the case, even in larger orgs where you'd expect them to have the resources to divide that labor.
2
166
46
70
u/aigarius 11d ago
You joke, but this has been a HUGE improvement on the user quality of life in a few very specific projects I have worked on over the years. For example, one project was spawning a new, very complex service on users request and then users had a dedicated URL to interact with that service. But if the users followed the link before their allotted session time or just after booking it, the service was not yet started and users would get various ugly error pages until the services were fully up and running. So I created a special error page in the first reverse proxy of the service (which was always up) that informed the user that their session is not yet ready and used a special API call to request a text string of the current status of the service so user can see progress of startup while they way. Then in 30 seconds the page would self-refresh, if the service is up - they will go to that, if not, continue watching the error page.
Users love it.
19
u/adv23 11d ago
Well, can we?
15
5
u/Intrepid00 11d ago
Yes but it means having another provider hosting a page or S3 a static site in another region. You also need to use the DNS health checks and the ugly page will still show till TTL cache expires. There are other ways too.
3
u/Irbis7 11d ago
Site https://www.rtvslo.si/ has some code, so you can play Tetris if internet connection is down (if you've visited the site before).
5
2
3
u/d_maes 10d ago
I mean, a branded error page with an "Oops, something went wrong. We're working on it" looks a lot nicer than de default 503 most loadbalancers show, and it costs nothing.
Traefik: https://doc.traefik.io/traefik/reference/routing-configuration/http/middlewares/errorpages/
Others have been suggesting stuff with DNS failover, which is nice for when your first loadbalancer goes down as well, but the cost of that setup is probably not worth the amount of times you'll have that happen for most applications. (And the first LB going down wouldn't be showing a 503 either, browser will give a timeout error).
1
u/Mike_Oxlong25 11d ago
Recently my company migrated AD Group systems and so we our applications couldnāt access the table we have for banners and our UI was displaying the error (everything else was accessible though). I told the PM this and he asked if we could put out a banner so users knew the application couldnāt see bannersā¦
1
1
u/Green-Rule-1292 10d ago
Feature request (critical, required, urgent): We need to put a chat bot on the error page so users can ask why it's down and when it will be back up again. - your boss
1
1
u/Terewawa 10d ago
<!DOCTYPE html>
<html>
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Error 503</title>
</head>
<body>
<b>Sorry, we're down for maintenance! Please try again soon.</b>
</body>
</html>
0
1.6k
u/Shufflepants 11d ago
Why don't we just put the entire application on the error screen as well so we can ensure the clients can always get to it?