[Fleet] Support `space_id` in preconfiguration #183920

nchaulet · 2024-05-21T14:00:57Z

Summary

As discussed change the SO type for package to be agnostic #172963

That PR allow to configure space aware preconfigured agent policies. By default preconfigured agent policy are created in the default namespace, by using the space_id property you could create a policy in a different space.

Example

You need to enable the space experimental feature to test this:

xpack.fleet.enableExperimental: ['useSpaceAwareness']

xpack.fleet.packages:
  - name: system
    version: latest
  - name: nginx
    version: latest
xpack.fleet.agentPolicies:
  - name: Test Kibana spaces
    space_id: test
    description: TEST
    id: test-policy
    is_default: false
    is_managed: true
    is_default_fleet_server: false
    package_policies:
      - id: test-nginx-1
        name: nginx-1
        package:
          name: nginx
        description: ''
        namespace: ''
        inputs:
          'nginx-logfile':
            enabled: true
            streams:
              '[nginx.error]':
                enabled: false
              '[nginx.access]':
                enabled: true
                vars:
                  paths:
                    - '/var/log/test/nginx/access.log*'
                  tags:
                    - nginx-access
                  preserve_original_event: false
                  ignore_older: 72h
          nginx-httpjson:
            enabled: false
    keep_monitoring_alive: true

Automated tests

I added unit test to the preconfiguration service

nchaulet · 2024-05-21T14:01:04Z

/ci

apmmachine · 2024-05-21T14:01:11Z

🤖 GitHub comments

Expand to view the GitHub comments

Just comment with:

/oblt-deploy : Deploy a Kibana instance using the Observability test environments.
run docs-build : Re-trigger the docs validation. (use unformatted text in the comment!)

nchaulet · 2024-05-21T14:01:39Z

x-pack/plugins/fleet/common/types/models/preconfiguration.ts

@@ -24,6 +24,7 @@ export type InputsOverride = Partial<NewPackagePolicyInput> & {

 export interface PreconfiguredAgentPolicy extends Omit<NewAgentPolicy, 'namespace' | 'id'> {
  id: string | number;
+  kibana_namespace?: string;


namespace is already used, curious if someone has better naming idea for that one

namespace is generally used more "internally" in Kibana code whereas space is used more in the public-facing docs, UI, etc. Maybe kibana_space_id would make sense here?

namespace is generally used more "internally" in Kibana code whereas space is used more in the public-facing docs, UI, etc. Maybe kibana_space_id would make sense here?

++ either kibana_space_id or space_id would be preferred. @kpollich is right that namespace is the internal name (for fun historical reasons) that we try not to leak.

I will update to space_id as it seems more used in other plugins

nchaulet · 2024-05-21T15:08:44Z

/ci

elasticmachine · 2024-05-21T15:10:24Z

Pinging @elastic/fleet (Team:Fleet)

nchaulet · 2024-05-21T18:28:18Z

@elasticmachine merge upstream

x-pack/plugins/fleet/server/services/preconfiguration.ts

kibana-ci · 2024-05-22T13:27:06Z

💛 Build succeeded, but was flaky

Buildkite Build
Commit: f530ab5

Failed CI Steps

FTR Configs #9

Test Failures

[job] [logs] FTR Configs #9 / endpoint "before all" hook in "endpoint"

Metrics [docs]

✅ unchanged

History

💚 Build #211346 succeeded 50c5699
💔 Build #211309 failed 7fbd009
💔 Build #211278 failed d9aca86

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @nchaulet

jen-huang

Left a few small suggestions but otherwise code LGTM. I tested locally with kibana.yml

I get this error in the UI but I assume this is out of the scope of this PR and will be tackled in future tasks:

jen-huang · 2024-05-22T16:21:22Z

x-pack/plugins/fleet/server/services/preconfiguration.test.ts

@@ -1124,6 +1125,69 @@ describe('policy preconfiguration', () => {
      expect(policies[0].id).toBe('test-id');
      expect(nonFatalErrorsB.length).toBe(0);
    });
+
+    it('should used a namespaced saved objet client if the agent policy space_id is set', async () => {


Suggested change

it('should used a namespaced saved objet client if the agent policy space_id is set', async () => {

it('should used a namespaced saved object client if the agent policy space_id is set', async () => {

jen-huang · 2024-05-22T16:24:10Z

x-pack/plugins/fleet/server/services/preconfiguration.test.ts

+        }),
+        expect.anything() // options
+      );
+    });


it would be nice to add another test for changing the space_id for the same policy id

when I tested manually, it just created another policy in the second space, rather than "moving" the original policy, because space_id becomes part of the saved object ID. I think this is ok but want to make sure

kc13greiner

LGTM!

nchaulet · 2024-05-22T17:57:01Z

I get this error in the UI but I assume this is out of the scope of this PR and will be tackled in future tasks:

@jen-huang Could I ask you how you get the error? just to check if I did not introduced a new bug

jen-huang · 2024-05-22T18:11:02Z

@jen-huang Could I ask you how you get the error? just to check if I did not introduced a new bug

This is when I have a policy using test space_id, but I did not set up that space yet, so I log into Kibana with default space and get that error when I load Fleet.

kc13greiner · 2024-05-22T18:11:44Z

I get this error in the UI but I assume this is out of the scope of this PR and will be tackled in future tasks:

@jen-huang Could I ask you how you get the error? just to check if I did not introduced a new bug

I also see the error if you start up KB locally with the config listed in the description, if the space 'test' is not created and the user navigates to fleet

kc13greiner · 2024-05-22T18:31:10Z

Initial load:

First reload:

Next reload:

nchaulet · 2024-05-22T19:51:08Z

Thanks I am able to reproduce it, it seems to be related to the deploy policy code, I will create a follow up PR for that specific issue

nchaulet commented May 21, 2024

View reviewed changes

nchaulet added the release_note:skip Skip the PR/issue when compiling release notes label May 21, 2024

nchaulet marked this pull request as ready for review May 21, 2024 15:08

nchaulet requested review from a team as code owners May 21, 2024 15:08

[Fleet] Support kibana_namespace in preconfiguration

d9aca86

nchaulet force-pushed the feature-preconfigured-namespaces branch from 099e742 to d9aca86 Compare May 21, 2024 15:10

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label May 21, 2024

Rename kibana_namespace => space_id

7fbd009

kibanamachine and others added 2 commits May 21, 2024 14:28

Merge branch 'main' into feature-preconfigured-namespaces

6735f1d

Refacto better variables names

69009c9

nchaulet requested review from kpollich and legrego May 21, 2024 19:01

remove unsued parameter

50c5699

jen-huang reviewed May 21, 2024

View reviewed changes

x-pack/plugins/fleet/server/services/preconfiguration.ts Outdated Show resolved Hide resolved

x-pack/plugins/fleet/server/services/preconfiguration.ts Show resolved Hide resolved

nchaulet self-assigned this May 22, 2024

nchaulet requested a review from jen-huang May 22, 2024 12:12

fix after review

f530ab5

jen-huang approved these changes May 22, 2024

View reviewed changes

kc13greiner approved these changes May 22, 2024

View reviewed changes

nchaulet merged commit f6c2fec into elastic:main May 23, 2024
21 checks passed

kibanamachine added v8.15.0 backport:skip This commit does not require backporting labels May 23, 2024

kaanyalti pushed a commit to kaanyalti/kibana that referenced this pull request May 23, 2024

[Fleet] Support kibana_namespace in preconfiguration (elastic#183920)

1cde34f

ycombinator changed the title ~~[Fleet] Support kibana_namespace in preconfiguration~~ [Fleet] Support space_id in preconfiguration May 23, 2024

nchaulet mentioned this pull request May 23, 2024

[Fleet] Fleet should allow an integration to be installed/reinstalled into another space #172963

Open

10 tasks

rshen91 pushed a commit to rshen91/kibana that referenced this pull request May 30, 2024

[Fleet] Support kibana_namespace in preconfiguration (elastic#183920)

74b8bb5

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Fleet] Support `space_id` in preconfiguration #183920

[Fleet] Support `space_id` in preconfiguration #183920

nchaulet commented May 21, 2024 •

edited

nchaulet commented May 21, 2024

apmmachine commented May 21, 2024

nchaulet May 21, 2024 •

edited

kpollich May 21, 2024

legrego May 21, 2024

nchaulet May 21, 2024

nchaulet commented May 21, 2024

elasticmachine commented May 21, 2024

nchaulet commented May 21, 2024

kibana-ci commented May 22, 2024

jen-huang left a comment •

edited

jen-huang May 22, 2024

jen-huang May 22, 2024

kc13greiner left a comment

nchaulet commented May 22, 2024

jen-huang commented May 22, 2024

kc13greiner commented May 22, 2024

kc13greiner commented May 22, 2024

nchaulet commented May 22, 2024

	it('should used a namespaced saved objet client if the agent policy space_id is set', async () => {
	it('should used a namespaced saved object client if the agent policy space_id is set', async () => {

[Fleet] Support space_id in preconfiguration #183920

[Fleet] Support space_id in preconfiguration #183920

Conversation

nchaulet commented May 21, 2024 • edited

Summary

Example

Automated tests

nchaulet commented May 21, 2024

apmmachine commented May 21, 2024

🤖 GitHub comments

nchaulet May 21, 2024 • edited

Choose a reason for hiding this comment

kpollich May 21, 2024

Choose a reason for hiding this comment

legrego May 21, 2024

Choose a reason for hiding this comment

nchaulet May 21, 2024

Choose a reason for hiding this comment

nchaulet commented May 21, 2024

elasticmachine commented May 21, 2024

nchaulet commented May 21, 2024

kibana-ci commented May 22, 2024

💛 Build succeeded, but was flaky

Failed CI Steps

Test Failures

Metrics [docs]

History

jen-huang left a comment • edited

Choose a reason for hiding this comment

jen-huang May 22, 2024

Choose a reason for hiding this comment

jen-huang May 22, 2024

Choose a reason for hiding this comment

kc13greiner left a comment

Choose a reason for hiding this comment

nchaulet commented May 22, 2024

jen-huang commented May 22, 2024

kc13greiner commented May 22, 2024

kc13greiner commented May 22, 2024

nchaulet commented May 22, 2024

[Fleet] Support `space_id` in preconfiguration #183920

[Fleet] Support `space_id` in preconfiguration #183920

nchaulet commented May 21, 2024 •

edited

nchaulet May 21, 2024 •

edited

jen-huang left a comment •

edited