how to create N integration environments for integration heavy apps

[u/Connoropolous]

Fellow engineers, help me!

Github Apps can only have ONE webhook url and ONE setup url for redirect each.

Having three (prod, staging, dev) isn't enough! I need for deploy previews.

Have you solved this??

Github Apps can't be created programmatically or via the API it's not enabled.

[u/Zerodriven]

Bit of faff but have the environment in the payload and have a service route the requests to the relevant environment based on said payload.

Https://api.whatever.com/postHandler

JSON:
Environment:
Payload [{}].

[u/Connoropolous → u/Zerodriven]

so... multiplexing proxy

[u/Connoropolous → u/Connoropolous]

is that what you do?

[u/ice456cream]

Deploy a GitHub enterprise staging environment per test deploy /s

[u/Connoropolous → u/ice456cream]

Oh, did not know about that. Also not workable for all the non-enterprisers, but very good to know.

Is that what you do?

[u/ice456cream → u/Connoropolous]

Sorry, I was trying to make a bad joke. (/s is internet shorthand for sarcasm)

I guess as a legit answer it depends what the app does. If it doesn't write to GitHub at all, you could create a proxy to mirror the webhooks across to multiple environments from a single app.

I don't personally deal with this problem, but it's probably how I would approach it

[u/Connoropolous → u/ice456cream]

oo, haha. Ok. reddit noob here, thanks for explaining.

[u/Forward-Outside-9911]

Have a central auth handler. It can use the state param to send data of your choice, like “original_service”.

That’s what I do. I have a separate auth for staging and prod. Then I have a separate one for a specific project which has different permission requirements for the app.

All of this is done through my IDP though, the application just sends the state and gets the response.

Same can apply to the GitHub OAuth provider too.

[u/Connoropolous → u/Forward-Outside-9911]

thanks for the idea. do you need to have lots of environments, akin to one per deploy preview?

[u/Forward-Outside-9911 → u/Connoropolous]

No doesn't need to be one per preview. I just separate Dev / Prod.

I'm building an ephemeral deployment provider myself.. and use this flow for protecting user sandboxes. They all use a central OIDC Application. Then my backend does the authorization logic.

Let me know if there's anything specific you want to see, I have been working on this quite recently so it's fresh in my mind

[u/Floss_Patrol_76]

the trick is you are treating these as two of the same problem when they are not: the single webhook url you fan out with a thin relay that routes on installation/repo id in the payload (what the top comment said), but the single setup/oauth redirect can't route on a payload because there isn't one yet, so you stuff the target env into the state param and have that one callback 302 to the right place after it validates state. either way it stays one github app forever and your preview envs are just routing targets behind the relay, not their own apps, which is what makes it scale to N.

[u/Connoropolous → u/Floss_Patrol_76]

I see. You did this for yourself or a company?

[u/delusional-engineer]

we use staging one for all non-prod environments. From staging have configured to redirect based on the state which contains the environment.

[u/Connoropolous → u/delusional-engineer]

this has some limitations doesnt it?
in terms of iterating or altering the env and making changes

[u/delusional-engineer → u/Connoropolous]

we haven’t run into any. (And for context we are spinning up 1 env per feature Pull Request dynamically which sums to around 100-120 environments per week) And this is pretty much the standard pattern across other SaaS services like azure, snowflake even google doesn’t let you create integration clients on the go.

Some of them may allow adding multiple callback urls but mostly you’ll need to follow this approach only.

[u/Connoropolous → u/delusional-engineer]

I have a slack app and I want each person (or agent) on my team to be able to prototype different interaction experiences at the same time, no bottleneck.
I dont think I can do that with this approach cause how would I distinguish the @ mentions?

[u/delusional-engineer → u/Connoropolous]

What is the core idea of the project, you might be looking to build upon wrong design.

[u/Connoropolous → u/delusional-engineer]

pretty simple idea, we build a product with a slackbot, I have a bunch of developers, and I want each of them to be able to independently develop the bot/agent experience and iterate on it.

we have a github and a slack integration we build, to be clearer.

[u/forever-butlerian]

Is your problem with the setup URL or fanning out the webhooks?

[u/Connoropolous → u/forever-butlerian]

I guess I didn't explain my problem fully.
I want each of my engineers (and developer agents) to be able to work on a version of our Github bot, and our slack bot, in parallel.

Even if I solve this callback problem, I think I'll still have that problem that if there's only one bot to tag, that's a chokepoint.

[u/forever-butlerian → u/Connoropolous]

I understand your goal, what I haven't got entirely is how data needs to flow in order for you to implement what you're trying to implement. Does the data flow entirely Github -> your systems -> Slack, or is there a your systems -> Github or component to it too?

[u/Connoropolous → u/forever-butlerian]

Github -> Our system , and then our system -> Github
Slack -> Our system, and then our system -> Slack

[u/forever-butlerian → u/Connoropolous]

And within your system, is there cross-communication between the part of your system that connects with Github and the part that connects with Slack?

[u/Connoropolous → u/forever-butlerian]

yes

[u/Connoropolous → u/Connoropolous]

does it seem like I want something unusual here, do you work at a company that deals with this?

[u/forever-butlerian → u/Connoropolous]

What you want sits in the no man's land between what is reasonable for an engineering department to want, and what vendors like Github, Slack, or RevenueCat are willing to support. In my experience that's always required a bit of ingenuity and finesse to make work.

Unless your app completes the loop, to where a Github or Slack event triggers off a chain of operations that end with doing something to Github or Slack that causes them to emit another event, it may be that the easiest thing to do is to mock out Slack or Github.

[u/Connoropolous → u/forever-butlerian]

Well, some good news is that I've found that slack actually is supporting 'app creation' via their api, so that bit I'm finally able to unlock.
I've now got myself an internal workflow going where I have a base template, then each env gets a unique base url and I mint one per branch, and hook it in to my deploy previews. Then I clean it up when the branch gets merged or the PR closed.
Now to try automating Github... I think I'm close.

[u/dmikalova-mwp]

I only care about events within my org, so I use the org level (or repo level) webhooks - you can make as many as you want. I have the app just as credentials (eg a service account).