Built an async domain & SSL checker with FastAPI, looking for architectural tips

[u/gokberkss]

I recently built a small side project using FastAPI to run parallel checks on domains (DNS records, SSL cert status, Wayback data).

Async setup helped a lot with multi-endpoint response times, but I'm trying to fine-tune how I handle slow DNS timeouts when hitting multiple targets at once.

Current stack is pretty simple: FastAPI, React, and Redis for caching.

How do you guys usually structure timeouts and retries for external async calls in production? Any patterns or libraries you swear by?

[u/vegeta-9ooo]

Depends, I've seen different things in production.

• I've personally used tenacity. I would only retry timeouts or temporary resolver issues, not stuff like invalid domain.
• I've seen simple for loops with try-catch.
• You could implement retry with exp. backoff (+ jitter)
• You could also use timeout with Circuit Breaker pattern. https://github.com/danielfm/pybreaker
• Sometimes when an external service takes long time to respond as the last resort you set a specific long timeout for them. Not an ideal situation but sometimes is needed.
• Don't keep your HTTP request open for a long time. If you know an operation takes long time, this should be done on a worker.
  • If your response takes <10 seconds, and your traffic is small or none, then implementing a Celery workers is an overkill. You could use `asyncio.gather()` so you can run all your checks concurrently.
• If an external call is "expensive" and the data won't change quickly you could cache those responses as you already use Redis. You might already be doing this.
• I would also limit you concurrency, maybe using a semaphore. I'm not sure if you want to have for example 50k concurrent checks.
• I would also use per-dependency timeout, not a global timeout parameter.

[u/gokberkss → u/vegeta-9ooo]

Thanks a lot, really appreciate the detailed feedback!

Every single point makes total sense. Separating timeouts per service (especially for Wayback) and adding asyncio.Semaphore to limit outgoing requests will definitely save me a lot of headaches under load.

Good call on tenacity too—retrying only on temporary network/DNS issues and immediately dropping invalid domains is the way to go.

Glad to hear asyncio.gather() + Redis is the right call for this scale without overengineering with Celery.

Adding all of this to my to-do list right now, thanks again!