Syscollector unable to gather python packages in macOS Sonoma #23507

santipadilla · 2024-05-17T11:50:42Z

Wazuh version	Component	Install type	Install method	Platform
4.8.0-rc2	Syscollector	Agent	Packages	macOS Sonoma

Description

It has been detected in Release 4.8.0 - RC 2 - E2E UX tests - Vulnerability Detection that syscollector is currently unable to collect python packages on macOS Sonoma.

Steps to reproduce

Install a manager
Install and register a macOS Sonoma agent
Install a python package, for example:

python3 -m pip install Django==3.2.13

Check that the package was correctly installed

sh-3.2# python3 -m pip freeze 
altgraph @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/altgraph-0.17.2-py2.py3-none-any.whl
asgiref==3.8.1
certifi==2024.2.2
chardet==3.0.4
click==8.1.7
Django==3.2.13
Flask==0.12
future @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/future-0.18.2-py3-none-any.whl
idna==2.6
itsdangerous==2.2.0
Jinja2==3.1.4
macholib @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/macholib-1.15.2-py2.py3-none-any.whl
MarkupSafe==2.1.5
pytz==2024.1
requests==2.18.4
six @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/six-1.15.0-py2.py3-none-any.whl
sqlparse==0.5.0
typing_extensions==4.11.0
urllib3==1.22
Werkzeug==3.0.3

Wait until syscollector scan is finished

2024/05/17 07:52:14 wazuh-modulesd:syscollector: INFO: Module started.
2024/05/17 07:52:14 wazuh-modulesd:syscollector: INFO: Starting evaluation.

Check that after a Syscollector scan, python packages are not being collected. It does not detect vulnerabilities and they do not appear in the inventory either. It has been tested with and without virtual environment and with different packages. With the same reproduction steps. In no case are they detected. On the other hand, npm packages are detected.

Evidences

All the information about the macOS agent in these two comments:
- System information
- macOS Sonoma Agent
OS: OS.json
Packages: Inventory.json

The text was updated successfully, but these errors were encountered:

sebasfalcone · 2024-05-19T15:53:09Z

We requested @santipadilla for the paths of the installed Python packages

To the date, only packages on the following list will be recognized

wazuh/src/data_provider/src/sharedDefs.h

Lines 113 to 124 in 1cd9357

    
           static const std::set<std::string> UNIX_PYPI_DEFAULT_BASE_DIRS 
        
           { 
        
               "/usr/lib/python*/*-packages", 
        
               "/usr/lib64/python*/*-packages", 
        
               "/usr/local/lib/python*/*-packages", 
        
               "/home/*/.local/lib/python*/*-packages", 
        
               "/root/.local/lib/python*/*-packages", 
        
               "/opt/homebrew/lib", 
        
               "/Library/Python", 
        
               "/Library/Frameworks/Python.framework/Versions/*/lib/python*/*-packages", 
        
           };

santipadilla · 2024-05-20T07:09:47Z

Hi @sebasfalcone, the machines were destroyed, I am going to set up the environment again to reproduce the case and bring you the answer. Thank you!

santipadilla · 2024-05-20T08:09:30Z

Update

Python packages installation

sh-3.2# python3 -m pip install Django==3.2.13
Collecting Django==3.2.13
  Downloading Django-3.2.13-py3-none-any.whl (7.9 MB)
     |████████████████████████████████| 7.9 MB 8.6 MB/s 
Collecting asgiref<4,>=3.3.2
  Downloading asgiref-3.8.1-py3-none-any.whl (23 kB)
Collecting sqlparse>=0.2.2
  Downloading sqlparse-0.5.0-py3-none-any.whl (43 kB)
     |████████████████████████████████| 43 kB 10.6 MB/s 
Collecting pytz
  Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB)
     |████████████████████████████████| 505 kB 53.2 MB/s 
Collecting typing-extensions>=4
  Downloading typing_extensions-4.11.0-py3-none-any.whl (34 kB)
Installing collected packages: typing-extensions, sqlparse, pytz, asgiref, Django
Successfully installed Django-3.2.13 asgiref-3.8.1 pytz-2024.1 sqlparse-0.5.0 typing-extensions-4.11.0



sh-3.2# python3 -m pip install Flask==0.3
Collecting Flask==0.3
  Downloading Flask-0.3.tar.gz (1.0 MB)
     |████████████████████████████████| 1.0 MB 8.6 MB/s 
Collecting Werkzeug>=0.6.1
  Downloading werkzeug-3.0.3-py3-none-any.whl (227 kB)
     |████████████████████████████████| 227 kB 58.4 MB/s 
Collecting Jinja2>=2.4
  Downloading jinja2-3.1.4-py3-none-any.whl (133 kB)
     |████████████████████████████████| 133 kB 64.7 MB/s 
Collecting MarkupSafe>=2.0
  Downloading MarkupSafe-2.1.5-cp39-cp39-macosx_10_9_universal2.whl (18 kB)
Building wheels for collected packages: Flask
  Building wheel for Flask (setup.py) ... done
  Created wheel for Flask: filename=Flask-0.3-py3-none-any.whl size=18064 sha256=2ad10fecaa6686b24ec8ca091d16801b04c90b75ad21f7a05e47d33e04e5d8a1
  Stored in directory: /private/var/root/Library/Caches/pip/wheels/66/77/83/5f18781a1a6edb7d17b996be4d148d1846378b0c88b9206b84
Successfully built Flask
Installing collected packages: MarkupSafe, Werkzeug, Jinja2, Flask
Successfully installed Flask-0.3 Jinja2-3.1.4 MarkupSafe-2.1.5 Werkzeug-3.0.3

Packages were correctly installed

sh-3.2# python3 -m pip freeze 
altgraph @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/altgraph-0.17.2-py2.py3-none-any.whl
asgiref==3.8.1
Django==3.2.13
Flask==0.3
future @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/future-0.18.2-py3-none-any.whl
Jinja2==3.1.4
macholib @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/macholib-1.15.2-py2.py3-none-any.whl
MarkupSafe==2.1.5
pytz==2024.1
six @ file:///System/Volumes/Data/SWE/Apps/DT/BuildRoots/BuildRoot11/ActiveBuildRoot/Library/Caches/com.apple.xbs/Sources/python3/python3-141/six-1.15.0-py2.py3-none-any.whl
sqlparse==0.5.0
typing_extensions==4.11.0
Werkzeug==3.0.3


sh-3.2# pip3 list
Package           Version
----------------- -------
altgraph          0.17.2
asgiref           3.8.1
Django            3.2.13
Flask             0.3
future            0.18.2
Jinja2            3.1.4
macholib          1.15.2
MarkupSafe        2.1.5
pip               21.2.4
pytz              2024.1
setuptools        58.0.4
six               1.15.0
sqlparse          0.5.0
typing_extensions 4.11.0
Werkzeug          3.0.3
wheel             0.37.0

Python version

sh-3.2# python3 --version
Python 3.9.6

Note: It was already pre-installed on the macOS machine.

Package details

sh-3.2# pip3 list --format=freeze | cut -d '=' -f 1 | xargs -n1 pip3 show | grep -E '^(Name|Location):'
WARNING: You are using pip version 21.2.4; however, version 24.0 is available.
You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command.
Name: altgraph
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
Name: asgiref
Location: /Library/Python/3.9/site-packages
Name: Django
Location: /Library/Python/3.9/site-packages
Name: Flask
Location: /Library/Python/3.9/site-packages
Name: future
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
Name: Jinja2
Location: /Library/Python/3.9/site-packages
Name: macholib
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
Name: MarkupSafe
Location: /Library/Python/3.9/site-packages
Name: pip
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
Name: pytz
Location: /Library/Python/3.9/site-packages
Name: setuptools
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
Name: six
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages
Name: sqlparse
Location: /Library/Python/3.9/site-packages
Name: typing_extensions
Location: /Library/Python/3.9/site-packages
Name: Werkzeug
Location: /Library/Python/3.9/site-packages
Name: wheel
Location: /Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-packages


sh-3.2# pip3 show Django
Name: Django
Version: 3.2.13
Summary: A high-level Python Web framework that encourages rapid development and clean, pragmatic design.
Home-page: https://www.djangoproject.com/
Author: Django Software Foundation
Author-email: foundation@djangoproject.com
License: BSD-3-Clause
Location: /Library/Python/3.9/site-packages
Requires: pytz, asgiref, sqlparse
Required-by:

sh-3.2# pip3 show Flask
Name: Flask
Version: 0.3
Summary: A microframework based on Werkzeug, Jinja2 and good intentions
Home-page: http://github.com/mitsuhiko/flask/
Author: Armin Ronacher
Author-email: armin.ronacher@active-4.com
License: BSD
Location: /Library/Python/3.9/site-packages
Requires: Jinja2, Werkzeug
Required-by:

Note: Locations in the list of recognized

sebasfalcone · 2024-05-20T18:39:44Z

Python official documentation

On macOS 10.8-12.3, the Apple-provided build of Python is installed in:
- /System/Library/Frameworks/Python.framework and /usr/bin/python, respectively.

We should add this path to the default dirs

Not identified paths

/Library/Python/3.9/site-packages

It was attempted to be identified by this path here:

wazuh/src/data_provider/src/sharedDefs.h

Line 121 in 1cd9357

"/Library/Python",

We should add the following path to the list:

/Library/Python/*/*-packages

/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/site-package

This path is not described in the documentation of Python but seems to be related to macOS itself

We could preemptively add this path

santipadilla mentioned this issue May 17, 2024

Release 4.8.0 - RC 2 - E2E UX tests - Vulnerability Detection #23416

Closed

2 tasks

juliamagan mentioned this issue May 17, 2024

Release 4.8.0 - RC 2 #23405

Closed

Dwordcito added type/bug Something isn't working level/task labels May 20, 2024

sebasfalcone self-assigned this May 20, 2024

sebasfalcone mentioned this issue May 20, 2024

Update python paths locations for macOS #23532

Merged

sebasfalcone linked a pull request May 20, 2024 that will close this issue

Update python paths locations for macOS #23532

Merged

Dwordcito closed this as completed May 22, 2024

SeyiSoneye mentioned this issue May 24, 2024

Release 4.7.5 - RC 1 - E2E UX tests - macOs #23571

Closed

2 tasks

NomanAbdullah mentioned this issue May 29, 2024

Release 4.8.0 - RC 3 - E2E UX tests - Vulnerability Detection #23702

Open

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Syscollector unable to gather python packages in macOS Sonoma #23507

Syscollector unable to gather python packages in macOS Sonoma #23507

santipadilla commented May 17, 2024 •

edited

sebasfalcone commented May 19, 2024

santipadilla commented May 20, 2024

santipadilla commented May 20, 2024

sebasfalcone commented May 20, 2024 •

edited

Syscollector unable to gather python packages in macOS Sonoma #23507

Syscollector unable to gather python packages in macOS Sonoma #23507

Comments

santipadilla commented May 17, 2024 • edited

Description

Steps to reproduce

Evidences

sebasfalcone commented May 19, 2024

santipadilla commented May 20, 2024

santipadilla commented May 20, 2024

Update

sebasfalcone commented May 20, 2024 • edited

Python official documentation

Not identified paths

santipadilla commented May 17, 2024 •

edited

sebasfalcone commented May 20, 2024 •

edited