Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Python threading to avoid creating temp files #39

Open
wants to merge 8 commits into
base: main
Choose a base branch
from

Conversation

6-6-6
Copy link

@6-6-6 6-6-6 commented Aug 18, 2022

This is an effort to avoid creating temp files under /tmp folder.

It uses Python's threading and queueing library to receive and preprocess the output of powermetrics.

The main change is presented in commit 1daf3ad
The other 2 commits are for code cleanups.

The whole process is described as follows:

  1. powermetrics is configured to output to stdout
  2. a LIFO queue is created for communicating between threads
  3. the output will be passed to another thread and processed by enqueue_powermetrics(), which will
    • convert the lines of stdout into separate plists (by the magic b'\x00' character)
    • each plist will be added to the LIFO queue
  4. instead of reading the whole temp file, parse_powermetrics() will get the latest plist from the LIFO queue, and parse the plist as usual

This PR will hopefully resolve issues #18, #17 and #35.
Also #6 may also be resolved (without restarting powermetrics multiple times).

making asitop not need to write file on the filesystem

Signed-off-by: Zhang Zongyu <zongyu@novazy.net>
Signed-off-by: zongyu <zongyu@novazy.net>
since temp files do not exists anymore,
the memory consumption problem no longer exists.

Signed-off-by: zongyu <zongyu@novazy.net>
@buliaoyin
Copy link

Seems nice, I'll try to check if the memory issue had been resolved.

@buliaoyin
Copy link

Seems nice, I'll try to check if the memory issue had been resolved.

After 6 hours, the python process memory was still only around 30MB, memory issue has gone.

Signed-off-by: Zhang Zongyu <zongyu@novazy.net>
Waiting for the queue to be filled by utils.enqueue_powermetrics()
  implies that we need to sleep for args.interval seconds,
  so we do not need extra lines to sleep between `powermetrcis` outputs.

Signed-off-by: Zhang Zongyu <zongyu@novazy.net>
@6-6-6
Copy link
Author

6-6-6 commented Sep 1, 2022

Hi, I have just added two more commits.

1acead9 is to meet the requirement of "Codacy Static Code Analysis".

5b6b58c is to remove the redundant time.sleep(args.interval) in asitop.asitop.py,
and the reason is explained in the commit message.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants