Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(addon): Extract all URL Categories to category as a multi-value field #204

Open
wants to merge 3 commits into
base: develop
Choose a base branch
from

Conversation

as3923
Copy link
Contributor

@as3923 as3923 commented Aug 19, 2021

Description

Extract multi-value URL categories from logs that come from Cortex Data Lake. Should also resolve issues similar that mentioned in #147 for logs from Cortex.

Motivation and Context

How Has This Been Tested?

Screenshots (if appropriate)

Types of changes

  • New feature (non-breaking change which adds functionality)

Checklist

  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have added tests to cover my changes if appropriate.
  • All new and existing tests passed.

@welcome-to-palo-alto-networks

🎉 Thanks for opening this pull request! We really appreciate contributors like you! 🙌

Added `isnotnull(URLCategoryList)` check because some events only have URLCategory.
Copy link
Member

@btorresgil btorresgil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for opening this. How has it been tested?

I'm concerned that isnotnull will always return true because Splunk seems to enjoy populating null json values with a string containing the word "null" which would always be non-null.

I'm also not sure if a split is the right way because I thought CDL would produce a list which should be parsed by Splunk into multi-value field. Curious if CDL is offering a comma delineated string instead of a list.

Thanks again!

For some Cortex eventtypes `http_category` is not in the `URLCategoryList`, but for other eventtypes it is in the list.  Updated the eval to check whether `http_category` is in the list, and to append it if it is not already in the list.
@as3923
Copy link
Contributor Author

as3923 commented Aug 23, 2021

@btorresgil This was tested with Cortex data in an on-premise Splunk Enterprise environment.

I'm not concerned about isnotnull for the URLCategoryList field, because the field only exists in the JSON if there is data, otherwise the field does not exist--therefore not populated with "null" like some of the other JSON fields.

I used split to create a multivalue field because that's what was done in #154 for non-Cortex data. If a single comma delineated string is preferred, then use URLCategoryList without the split function.

@btorresgil btorresgil self-assigned this Oct 7, 2022
@paulmnguyen paulmnguyen force-pushed the develop branch 2 times, most recently from de4dfdc to d7bd687 Compare May 17, 2023 20:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants