🐛 Bug Report: backend search API failures after 1.27.0 upgrade

[brianphillips]

📜 Description

When attempting to search after upgrading to Backstage 1.27.0, a generic error is displayed:

Error: Error encountered while fetching search results

The details of the error show that the types query parameter is being rejected as invalid by the zod schema.

This is a result of the changes made in #23874 where the initialization of the searchIndexService was pushed into a startup hook (here). Because collators are added during the startup process (here), the http router that is created (here) doesn't have the necessary context (i.e. the list of valid document types) to validate incoming requests properly.

👍 Expected behavior

It should perform the search and display the results.

👎 Actual Behavior with Screenshots

It displays an error:

👟 Reproduction steps

1. Upgrade to 1.27.0
2. Attempt to perform a search
3. See error

📃 Provide the context for the Bug.

No response

🖥️ Your Environment

(this is the output from the devtools plugin)

OS: backstage-sstk-backstage-6768fb786d-nnm8j: Linux 5.10.215-203.850.amzn2.x86_64 - linux/x64
Resources: Memory: 9938/31627MB - Load: 0.22/0.47/0.64
node: v18.20.2
backstage: 1.27.0
Dependencies:
@backstage-community/plugin-adr-backend: 0.4.15
@backstage-community/plugin-adr-common: 0.2.23
@backstage-community/plugin-adr: 0.6.18
@backstage-community/plugin-apollo-explorer: 0.2.1
@backstage-community/plugin-badges-backend: 0.4.1
@backstage-community/plugin-badges: 0.2.59
@backstage-community/plugin-github-actions: 0.6.16
@backstage-community/plugin-github-pull-requests-board: 0.2.1
@backstage-community/plugin-jenkins-backend: 0.4.5
@backstage-community/plugin-jenkins-common: 0.1.26
@backstage-community/plugin-jenkins: 0.9.10
@backstage-community/plugin-kafka-backend: 0.3.16
@backstage-community/plugin-kafka: 0.3.35
@backstage-community/plugin-newrelic-dashboard: 0.3.10
@backstage-community/plugin-sonarqube-backend: 0.2.20
@backstage-community/plugin-sonarqube-react: 0.1.16
@backstage-community/plugin-sonarqube: 0.7.17
@backstage-community/plugin-tech-radar: 0.7.4
@backstage/app-defaults: 1.5.5
@backstage/backend-app-api: 0.7.3
@backstage/backend-common: 0.21.7, 0.22.0
@backstage/backend-defaults: 0.2.18
@backstage/backend-dev-utils: 0.1.4
@backstage/backend-openapi-utils: 0.1.11
@backstage/backend-plugin-api: 0.6.18
@backstage/backend-tasks: 0.5.23
@backstage/catalog-client: 1.6.5
@backstage/catalog-model: 1.5.0
@backstage/cli-common: 0.1.13
@backstage/cli-node: 0.2.5
@backstage/cli: 0.26.5
@backstage/config-loader: 1.8.0
@backstage/config: 1.2.0
@backstage/core-app-api: 1.12.5
@backstage/core-compat-api: 0.2.5
@backstage/core-components: 0.13.10, 0.14.7
@backstage/core-plugin-api: 1.9.2
@backstage/dev-utils: 1.0.32
@backstage/e2e-test-utils: 0.1.1
@backstage/errors: 1.2.4
@backstage/eslint-plugin: 0.1.8
@backstage/frontend-plugin-api: 0.6.5
@backstage/integration-aws-node: 0.1.12
@backstage/integration-react: 1.1.27
@backstage/integration: 1.11.0
@backstage/plugin-api-docs: 0.11.5
@backstage/plugin-app-backend: 0.3.66
@backstage/plugin-app-node: 0.1.18
@backstage/plugin-auth-backend-module-atlassian-provider: 0.1.10
@backstage/plugin-auth-backend-module-aws-alb-provider: 0.1.10
@backstage/plugin-auth-backend-module-azure-easyauth-provider: 0.1.1
@backstage/plugin-auth-backend-module-bitbucket-provider: 0.1.1
@backstage/plugin-auth-backend-module-cloudflare-access-provider: 0.1.1
@backstage/plugin-auth-backend-module-gcp-iap-provider: 0.2.13
@backstage/plugin-auth-backend-module-github-provider: 0.1.15
@backstage/plugin-auth-backend-module-gitlab-provider: 0.1.15
@backstage/plugin-auth-backend-module-google-provider: 0.1.15
@backstage/plugin-auth-backend-module-guest-provider: 0.1.4
@backstage/plugin-auth-backend-module-microsoft-provider: 0.1.13
@backstage/plugin-auth-backend-module-oauth2-provider: 0.1.15
@backstage/plugin-auth-backend-module-oauth2-proxy-provider: 0.1.11
@backstage/plugin-auth-backend-module-oidc-provider: 0.1.9
@backstage/plugin-auth-backend-module-okta-provider: 0.0.11
@backstage/plugin-auth-backend: 0.22.5
@backstage/plugin-auth-node: 0.4.13
@backstage/plugin-auth-react: 0.1.2
@backstage/plugin-catalog-backend-module-backstage-openapi: 0.2.1
@backstage/plugin-catalog-backend-module-github: 0.6.1
@backstage/plugin-catalog-backend-module-scaffolder-entity-model: 0.1.16
@backstage/plugin-catalog-backend-module-unprocessed: 0.4.5
@backstage/plugin-catalog-backend: 1.22.0
@backstage/plugin-catalog-common: 1.0.23
@backstage/plugin-catalog-graph: 0.4.5
@backstage/plugin-catalog-import: 0.11.0
@backstage/plugin-catalog-node: 1.12.0
@backstage/plugin-catalog-react: 1.12.0
@backstage/plugin-catalog-unprocessed-entities-common: 0.0.1
@backstage/plugin-catalog-unprocessed-entities: 0.2.4
@backstage/plugin-catalog: 1.20.0
@backstage/plugin-devtools-backend: 0.3.4
@backstage/plugin-devtools-common: 0.1.9
@backstage/plugin-devtools: 0.1.14
@backstage/plugin-events-node: 0.3.4
@backstage/plugin-home-react: 0.1.13
@backstage/plugin-home: 0.7.4
@backstage/plugin-kubernetes-backend: 0.17.1
@backstage/plugin-kubernetes-common: 0.7.6
@backstage/plugin-kubernetes-node: 0.1.12
@backstage/plugin-kubernetes-react: 0.3.5
@backstage/plugin-kubernetes: 0.11.10
@backstage/plugin-org: 0.6.25
@backstage/plugin-permission-backend: 0.5.42
@backstage/plugin-permission-common: 0.7.13
@backstage/plugin-permission-node: 0.7.29
@backstage/plugin-permission-react: 0.4.22
@backstage/plugin-proxy-backend: 0.4.16
@backstage/plugin-scaffolder-backend-module-azure: 0.1.10
@backstage/plugin-scaffolder-backend-module-bitbucket-cloud: 0.1.8
@backstage/plugin-scaffolder-backend-module-bitbucket-server: 0.1.8
@backstage/plugin-scaffolder-backend-module-bitbucket: 0.2.8
@backstage/plugin-scaffolder-backend-module-gerrit: 0.1.10
@backstage/plugin-scaffolder-backend-module-gitea: 0.1.8
@backstage/plugin-scaffolder-backend-module-github: 0.2.8
@backstage/plugin-scaffolder-backend-module-gitlab: 0.4.0
@backstage/plugin-scaffolder-backend: 1.22.6
@backstage/plugin-scaffolder-common: 1.5.2
@backstage/plugin-scaffolder-node: 0.4.4
@backstage/plugin-scaffolder-react: 1.8.5
@backstage/plugin-scaffolder: 1.20.0
@backstage/plugin-search-backend-module-catalog: 0.1.24
@backstage/plugin-search-backend-module-pg: 0.5.27
@backstage/plugin-search-backend-module-techdocs: 0.1.23
@backstage/plugin-search-backend-node: 1.2.22
@backstage/plugin-search-backend: 1.5.8
@backstage/plugin-search-common: 1.2.11
@backstage/plugin-search-react: 1.7.11
@backstage/plugin-search: 1.4.11
@backstage/plugin-techdocs-backend: 1.10.5
@backstage/plugin-techdocs-module-addons-contrib: 1.1.10
@backstage/plugin-techdocs-node: 1.12.4
@backstage/plugin-techdocs-react: 1.2.4
@backstage/plugin-techdocs: 1.10.5
@backstage/plugin-user-settings: 0.8.6
@backstage/release-manifests: 0.0.11
@backstage/repo-tools: 0.9.0
@backstage/test-utils: 1.5.5
@backstage/theme: 0.4.4, 0.5.4
@backstage/types: 1.1.1
@backstage/version-bridge: 1.0.8
@drodil/backstage-plugin-toolbox-react: 1.20.3
@drodil/backstage-plugin-toolbox: 1.20.3
@frontside/backstage-plugin-graphql-backend-module-catalog: 0.3.3
@frontside/backstage-plugin-graphql-backend-node: 0.1.6
@frontside/backstage-plugin-graphql-backend: 0.1.8
@frontside/backstage-plugin-scaffolder-workflow: 0.10.1
@internal/plugin-shutterhound: 0.0.0-use.local
@pagerduty/backstage-plugin-backend: 0.6.1
@pagerduty/backstage-plugin-common: 0.1.3
@pagerduty/backstage-plugin-scaffolder-actions: 0.1.2
@pagerduty/backstage-plugin: 0.12.0
@roadiehq/backstage-plugin-argo-cd: 2.6.5
@roadiehq/backstage-plugin-github-insights: 2.3.29
@roadiehq/backstage-plugin-security-insights: 2.3.17
@spotify/backstage-plugin-analytics-module-insights: 0.7.11
@spotify/backstage-plugin-core-common: 0.5.12
@spotify/backstage-plugin-core-node: 0.9.0
@spotify/backstage-plugin-core: 0.7.1
@spotify/backstage-plugin-insights-backend: 0.2.13
@spotify/backstage-plugin-insights-common: 0.2.9
@spotify/backstage-plugin-insights: 0.3.0
@spotify/backstage-plugin-permission-backend-module-rbac: 0.1.2
@spotify/backstage-plugin-rbac-backend: 0.7.4
@spotify/backstage-plugin-rbac-common: 0.5.13
@spotify/backstage-plugin-rbac-node: 0.1.2
@spotify/backstage-plugin-rbac: 0.6.2
@spotify/backstage-plugin-search-backend-module-skill-exchange: 0.1.2
@spotify/backstage-plugin-skill-exchange-backend: 0.12.6
@spotify/backstage-plugin-skill-exchange-common: 0.1.1
@spotify/backstage-plugin-skill-exchange-node: 0.1.1
@spotify/backstage-plugin-skill-exchange: 0.11.3
@spotify/backstage-plugin-slack-node: 0.1.5
@spotify/backstage-plugin-soundcheck-backend-module-github: 0.5.5
@spotify/backstage-plugin-soundcheck-backend-module-scm: 0.7.1
@spotify/backstage-plugin-soundcheck-backend: 0.16.2
@spotify/backstage-plugin-soundcheck-common: 0.13.1
@spotify/backstage-plugin-soundcheck-node: 0.7.0
@spotify/backstage-plugin-soundcheck: 0.12.7

👀 Have you spent some time to check if this bug has been raised before?

• I checked and didn't find similar issue

🏢 Have you read the Code of Conduct?

• I have read the Code of Conduct

Are you willing to submit PR?

Yes I am willing to submit a PR!

[drodil]

Good catch. The problem when not having this protection is that for example catalog collator cannot reach catalog API as it has not initialized yet (if both plugins are running in same instance). So maybe reverting the original is in order but in the meantime each collator should take care of calling services only after they are available.

Please note that the change also affected the catalog processing and it should not be reverted.

[Pike]

One way to fix this would be to split start into build (naming things) and start?

FWIW, this is reproducible on demo, https://demo.backstage.io/search?query=radar&types%5B%5D=software-catalog

[freben]

Any of you up for making that split? Sounds patch worthy

[brianphillips]

Yes, I've got a patch started. My only question is whether all the collators will have been added by the time this code runs.

[freben]

You mean the modules that add collators through the extension point? Yes, modules run first.

[Vity01]

Hopefully it will be part of 1.27.2 soon!

[Pike]

1.27.2 is out including this fix, and it works for me locally (my 1.26+ migration isn't fully done yet)

[Vity01]

@Pike Ahh, I didn't notice. I just finished with an upgrade to 1.27.1... Thank you. So let's start over.