Plans for Webhook Endpoints

[jleclanche]

Moving this from an email chain to put some public eyes on it. We're revamping webhook endpoints for next release.

0. Preliminary work:

1. Make sure the WebhookEndpoint model is up to date and working well, that it can be synced to upstream, etc.
2. Check if there is a way to automatically update WebhookEndpoints via webhooks themselves (I don't think there is but Stripe may have added that)
3. Ensure webhook endpoints can be synced via djstripe_sync_models or some such.

I. Dynamic URLs for webhooks

There are two issues that can be solved by the same rework. First one is that webhook URLs are publicly guessable. Usually included with a /stripe/ or /djstripe/ prefix, you can figure out that there is a stripe webhook URL at https://example.com/djstripe/webhook/ for example. This isn't a huge security issue but it's not great either as you can generate noise for the developers among other things.

I'd like to match instead on /<inclusion prefix>/webhook/<suffix>/. So for example generate a uuid for each new webhook and end up with /djstripe/webhook/e1526013-6e23-4e3d-abd6-6d4879947654/.

For this the first step is to change URL matching to include webhook/ as a prefix, match on all of it, then look up the webhooks in the database. We ought to support multiple sites per installation as well so we probably want to check the host header value as well.

II. Multiple webhooks per installation

With the above, we can ensure multiple webhooks are supported per installation. Each account will have a stripe_account key and when we receive a webhook payload, we can instantly know which account this belongs to by looking at the unique key.

III. Automatic setup of webhooks

Since we control all this in the database, it should be possible to automatically set up the webhooks in Stripe from the SDK. What that means is we can generate a new URL, find the hostname via site config, figure out the prefix somehow (no idea how, yet, though -- maybe iterate on all the URLs configured and figuring out where the djstripe inclusion is); create the endpoints via the API and feed them back to the db as such.

At that point we need to seriously consider and document what the actual setup flow for webhooks is. We may need some amount of user interaction, especially if a webhook endpoint is already configured? We also need to make sure the webhook endpoint admin panel is polished.

IV. Deprecation path for old setups

I'd like to ensure a smooth transition between old setups and new ones. We need a way to detect when things are set up "the old way". We probably still need to watch over /djstripe/webhook/ without a suffix and process according to DJSTRIPE_WEBHOOK_SECRET then. Also need to ensure our new method handles suffixless webhook endpoints without issues. (The webhook url itself shouldn't be unique, but we may need a django check to warn if there are multiple webhook endpoints with the same URL set up? Consequence can be incorrect handling of the payload, it going to the wrong account etc).

V. Check if we can still support DJSTRIPE_WEBHOOK_URL

That setting should be deprecated as it will no longer be necessary. But I do not think we should remove it any time soon as it's a pain to update for existing users. Instead we should add a django Check with a warning explaining the deprecation (with a "removed in dj-stripe 3.0" or some such)

VI. Moving other settings to the database.

On top of webhook secrets, we also have DJSTRIPE_WEBHOOK_VALIDATION and DJSTRIPE_WEBHOOK_TOLERANCE settings which could be moved to the database, as part of the webhook administration. This would allow changing those settings on a per-webhook basis. I don't think we should block the release on this if it's not done by the time the rest is ready, but we should at least add the database fields for these (djstripe_tolerance_ms and djstripe_validation_type).

If we have those functional, the settings should be deprecated, but they will override the value in the db if set.
Consider making the db values nullable so we can retain a default in dj-stripe. No idea if that's the right way forward or not.

VII. Documentation updates

There are several documentation pages on webhooks, we need to ensure they are up to date and reflect our changes. This all needs to be detailed in the changelog for the next release as well, including the rationale.

[kavdev]

@jleclanche fantastic writeup. This makes a lot of sense, and will be an awesome add to the library.

[arnav13081994]

@jleclanche Great writeup! I have started work on this and have a few questions and observations.

1. Dj-stripe does not have a WebhookEndpoint model.
2. It is not possible to get the list of all Webhooks from Stripe using a webhook itself so that needs to be maintained manually for now.
   1. It is, however, possible to get all WebhookEndpoints from Stripe and they can be synced using djstripe_sync_models and not any webhook. This is not a biggie but users would need to be told to run the sync command after installing or updating dj-stripe.
3. For dynamic URLs to work, they would also need to be updated upstream, which does not seem like something most would be comfortable with especially in production.
4. Now that the djstripe_owner_account will get populated for every model instance, we can easily know which stripe_account an object belongs to, not only for incoming webhook requests but also whenever djstripe_sync_models command is run, which would also need to sync data for all djstripe_owner_accounts
5. If DJSTRIPE_WEBHOOK_VALIDATION and DJSTRIPE_WEBHOOK_TOLERANCE are set and they should be overrideable then the user would need to be allowed to specify them on a per webhook basis too which would make it more complicated. I think the easiest thing would be to set a default and let the user update these values from the django admin, if they ever do. And we deprecate these settings immediately.
6. Also WEBHOOK_EVENT_CALLBACK will also need to be set on a per webhook basis.

It seems if we can come up with some sort of naming convention for each webhook, a simple function can be used to detect and override these settings in the DB directly. Something like

DJSTRIPE_WEBHOOK_<ID>_<SETTING_NAME> should suffice. Please note Stripe allows one to set multiple webhooks for the same url.

Please let me know what you think.

[jleclanche]

I don't like the idea that we'd be doing dynamic settings with IDs in the name and all. -1 on that. Either we have db-driven settings for db objects, or we have python settings in python objects, not a weird mix.

[arnav13081994]

I agree. And for the first version let's standardise adding secrets from the DB and not touch anything upstream. Future versions could create URLs dynamically etc.

[dbartenstein]

Hi @arnav13081994 and @jleclanche:
With the current version, dj-stripe 2.5.1, is it possible to work with multiple Stripe accounts by turning off DJSTRIPE_WEBHOOK_VALIDATION? [Of course this is a security issue!]

[jleclanche]

You can set it to "retrieve_event" instead which I believe should work. If it doesn't then there's still work to be done I suppose.

[dbartenstein]

You can set it to "retrieve_event" instead which I believe should work. If it doesn't then there's still work to be done I suppose.

@jleclanche: by using the same webhook URL for different Stripe accounts, it’s impossible to correctly assign the event to the proper Stripe account.

The api_key param is skipped when calling obj.validate() in

dj-stripe/djstripe/models/webhooks.py

Line 120 in 5ed7123

obj.valid = obj.validate()

The Stripe account defined in settings is always used and for validation, which of course won’t work.

Step I of your list is required: #1437 (comment)

@arnav13081994: great news that you have begun tackling webhook endpoints 👏! If you need someone to test this with two different Stripe accounts, I’ll gladly help you out.

[arnav13081994]

@dbartenstein Thanks. I will let you know.