-
Notifications
You must be signed in to change notification settings - Fork 410
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add BSD support to CPU count check #1535
Comments
Or maybe use os.sysconf(NPROCESSORS_CONF), as suggested by another pkgsrc developer. |
Thank you for bringing this up. It would indeed be very useful to have better support of BSD systems. I am afraid that none of the core devs of joblib run BSD systems. The os.sysconf feels like a better route to me, as a call to subprocess.run will have a lot of overhead. I do wonder how it will behave with regards to physical versus logical cores for processors with hyperthreading. I also wonder if it will handle well the case of programs running in jails, to avoid oversubscription in these settings. Under Linux, we have found that these details mattered to have efficient computing. What would be really great is if you could make the appropriate changes on a branch, check that they work fine, and do a pull request. As we do not run BSD systems, we cannot test the changes, and I would not feel comfortable doing them blindly. Thanks!! |
I can test on FreeBSD, macOS, and NetBSD. I'll leave the Linux case alone. Running "getconf NPROCESSORS_CONF", which I believe is the shell equivalent so os.sysconf() (both are just interfaces to sysconf(3)), does indeed return 4 on my 2-core i5 running FreeBSD and my 2-core i5 MacBook. Maybe the python sysctl package would be a better route for BSDs (including macOS). We should be able to get the same information as |
Maybe the python sysctl package would be a better route for BSDs (including macOS). We should be able to get the same information as subprocess.run("sysctl -n hw.physicalcpu".split(), capture_output=True, text=True) without spawning a subprocess. I'll explore this and get back to you.
Thanks a lot for looking this up. You are right that macOS is a place to experiment. I don't have one around but when I'm in the office I can ask colleagues.
Not spawning a subprocess will probably be a big gain.
And thanks for the pull request!
|
I think we'll have to stick with subprocesses. Apparently there is no portable sysctl interface for python. The sysctl on PYPI is FreeBSD-specific, developed by the TrueNAS project. I found other projects for Linux and NetBSD with completely different APIs. I'll do a pull request with code for FreeBSD and NetBSD, adapted from the existing Darwin case. |
Patch in platform-specific cases for NetBSD and FreeBSD Without this, the core count routine errors out No other changes Maintainer feedback timeout (12 days) Issue was discussed on tech-pkg and upstream joblib/joblib#1535
The text was updated successfully, but these errors were encountered: