OMP: Error #15: Initializing libiomp5md.dll, but found libomp140.x86_64.dll already initialized

[elmoBG8]

Describe the bug
OMP: Error #15: Initializing libiomp5md.dll, but found libomp140.x86_64.dll already initialized.
OMP: Hint This means that multiple copies of the OpenMP runtime have been linked into the program. That is dangerous, since it can degrade performance or cause incorrect results. The best thing to do is to ensure that only a single OpenMP runtime is linked into the process, e.g. by avoiding static linking of the OpenMP runtime in any library. As an unsafe, unsupported, undocumented workaround you can set the environment variable KMP_DUPLICATE_LIB_OK=TRUE to allow the program to continue to execute, but that may cause crashes or silently produce incorrect results. For more information, please see http://www.intel.com/software/products/support/.

To Reproduce
Steps to reproduce the behavior:
Targetting in the following script various websites, for example this one: https://github.com/VinciGit00/Scrapegraph-ai/releases

from scrapegraphai.graphs import SmartScraperGraph
from scrapegraphai.utils import prettify_exec_info

graph_config = {
"llm": {
"model": "ollama/mistral",
"temperature": 1,
"format": "json",
"model_tokens": 2000,
"base_url": "http://localhost:11434",
},
"embeddings": {
"model": "ollama/nomic-embed-text",
"temperature": 0,
"base_url": "http://localhost:11434",
}
}

smart_scraper_graph = SmartScraperGraph(
prompt="List me all the news with their description.",
source="https://github.com/VinciGit00/Scrapegraph-ai/releases",
config=graph_config
)

result = smart_scraper_graph.run()
print(result)

Expected behavior
A list of news with the json format.

Desktop:

• OS: Windows 10

[VinciGit00]

Please try again with the new version

[elmoBG8]

Done pip install --upgrade scrapegraphai
installing successfully scrapegraphai-0.10.1

However, the problem persists

[VinciGit00]

try the basic example and let me know if it works

[elmoBG8]

Yes, using as source https://perinim.github.io/projects it works (and it worked even with scrapegraphai-0.10.0).

The problem is using different sources, like https://github.com/VinciGit00/Scrapegraph-ai/releases

You can't replicate my result using that link as source?

[f-aguzzi]

@baggiponte might this be related to the dependency clashes caused by Poetry you mentioned in issue #198?

[baggiponte]

@baggiponte might this be related to the dependency clashes caused by Poetry you mentioned in issue #198?

Not sure. I am afraid it's more to do with Intel/Windows. @elmoBG8: how are you installing scrapegraph? Do you use conda (maybe you don't even build scrapegraph on conda though)? What's happening is that there are two libomp binaries.

[elmoBG8]

@baggiponte might this be related to the dependency clashes caused by Poetry you mentioned in issue #198?

Not sure. I am afraid it's more to do with Intel/Windows. @elmoBG8: how are you installing scrapegraph? Do you use conda (maybe you don't even build scrapegraph on conda though)? What's happening is that there are two libomp binaries.

Hello,
inside a virtual environment in anaconda I go with pip install scrapegraphai and then playwright install.
However, I remember that I have no problem for example using as source https://perinim.github.io/projects

I have problems for example with https://github.com/VinciGit00/Scrapegraph-ai/releases as source. Do you get something with it?

[baggiponte]

@baggiponte might this be related to the dependency clashes caused by Poetry you mentioned in issue #198?

Not sure. I am afraid it's more to do with Intel/Windows. @elmoBG8: how are you installing scrapegraph? Do you use conda (maybe you don't even build scrapegraph on conda though)? What's happening is that there are two libomp binaries.

Hello, inside a virtual environment in anaconda I go with pip install scrapegraphai and then playwright install.

I am not proficient enough when using windows + conda. Is the venv you create a new one or a "recycled" one where there are already previous libraries? What might happen is that you have different versions of libomp on your machine because you have 2/3 conda envs that each have one. I am not an expert in windows at all, but this new env might pick up the two different omp versions even if they are in separate envs. maybe you can try deleting the conda cache (just googled this) but I haven't used it in years so I might be wrong.