Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Asset conversion stalls on media conversion #1624

Open
brett-hamilton opened this issue Feb 15, 2024 · 11 comments · May be fixed by #1649
Open

Asset conversion stalls on media conversion #1624

brett-hamilton opened this issue Feb 15, 2024 · 11 comments · May be fixed by #1649
Labels
bug Behaving differently as it should behave lang: python Done in Python code os: windows Windows-specific issue

Comments

@brett-hamilton
Copy link

System Info

  • OS: Windows 11
  • Python: 3.12.2

Following the build instructions from doc/build_instructions/windows_msvc.md, when running the first Python command the process stalled out for 30+ minutes before I manually stopped it.

Command

py -m openage --add-dll-search-path <path> --add-dll-search-path <path> main

Output Before Stopping

INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] Finished metadata read (10.50 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (0.61 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
-- Files done:   0/275

Error Output After Stopping

INFO [py] -- Exporting graphics files...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<openage directory>\build\openage\__main__.py", line 178, in <module>
    sys.exit(main())
             ^^^^^^
  File "<openage directory>\build\openage\__main__.py", line 171, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<openage directory>\build\openage\main\main.py", line 82, in main
    convert_assets(asset_path, args)
  File "<openage directory>\build\openage\convert\main.py", line 119, in convert_assets
    convert(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<openage directory>\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
rt
    MediaExporter._export_multithreaded(
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 315, in _export_multithreaded
    while outqueue.qsize() < expected_size:
  File "<string>", line 2, in qsize
  File "<user>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\managers.py", line 821, in _callmethod
    kind, result = conn.recv()
                   ^^^^^^^^^^^
  File "<user>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 250, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "<user>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 321, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
^C

Troubleshooting with the dev, I ran the following command and received a different error.

Command

py openage convert --jobs 1

Output

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<openage directory>\openage\build\openage\__main__.py", line 17, in <module>
    from .log import set_loglevel, verbosity_to_level, ENV_VERBOSITY
ImportError: attempted relative import with no known parent package
@heinezen heinezen added os: windows Windows-specific issue lang: python Done in Python code labels Feb 15, 2024
@brett-hamilton
Copy link
Author

Adding -m to the py command made the conversion process execute properly. However, it still required the --jobs 1 argument, so the multiprocessing problem persists. Thanks!

Command

py -m openage --add-dll-search-path <path> --add-dll-search-path <path> convert --jobs 1

Output

INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] using cached wrapper: C:\Users\brett\AppData\Local\Temp\AOC_empires2_x1.dat.pickle
INFO [py] Finished metadata read (1.06 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (1.49 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
INFO [py] -- Exporting terrain files...
INFO [py] -- Exporting sound files...
INFO [py] Dumping metadata files...
INFO [py] Finished modpack export (23.45 seconds)
INFO [py] asset conversion complete; asset version: 7

@heinezen heinezen added the bug Behaving differently as it should behave label Feb 20, 2024
@heinezen heinezen changed the title Windows Build: Asset Conversion Failing Asset conversion stalls on media conversion Feb 20, 2024
@heinezen heinezen mentioned this issue Feb 20, 2024
Merged
@heinezen
Copy link
Member

#1625 added an error callback, so if you run the conversion now, you should see an error message. Could you try converting again to see what's the issue?

@brett-hamilton
Copy link
Author

After updating with the error callback, here is the error message. Looks like a path issue:

INFO [py] -- Exporting graphics files...
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<openage directory>\build\openage\__main__.py", line 178, in <module>
    sys.exit(main())
             ^^^^^^
  File "<openage directory>\build\openage\__main__.py", line 171, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<openage directory>\build\openage\main\main.py", line 82, in main
    convert_assets(asset_path, args)
  File "<openage directory>\build\openage\convert\main.py", line 119, in convert_assets
    convert(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<openage directory>\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<openage directory>\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 120, in export
    MediaExporter._export_multithreaded(
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 310, in _export_multithreaded
    MediaExporter.log_fileinfo(
  File "<openage directory>\build\openage\convert\processor\export\media_exporter.py", line 620, in log_fileinfo
    target_size = os.path.getsize(target_path)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen genericpath>", line 62, in getsize
TypeError: stat: path should be string, bytes, os.PathLike or integer, not NoneType

@heinezen
Copy link
Member

Interesting. I think the reaon is that the path could not be resolved, which is why it's None.

@heinezen
Copy link
Member

Ok I think the issue is that when the the file info is logged, it's still being converted due to multiprocessing...

@heinezen heinezen mentioned this issue Feb 22, 2024
Merged
@heinezen
Copy link
Member

I've fixed the logging but I think this has nothing to do with your original issue :D Could you retry with #1627

@brett-hamilton
Copy link
Author

brett-hamilton commented Feb 23, 2024 •
edited

Yes, after taking the changes from #1627, here is the output:
The Keyboard Interrupt at line 251 is from me hitting ^C after it got stuck at the same point it did in my original issue (-- Files done: 0/275)

converting from 'C:\Program Files (x86)\Microsoft Games\Age of Empires II The Conquerors Expansion Trial'
INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] using cached wrapper: <user directory>\AppData\Local\Temp\AOC_empires2_x1.dat.pickle
INFO [py] Finished metadata read (1.02 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (1.48 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
Exception in thread Thread-3 (_handle_results):
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 789, in _export_texture
    from ...value_object.read.media.slp import SLP
ImportError: DLL load failed while importing slp: The specified module could not be found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()   0/275
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 595, in _handle_results
    cache[job]._set(i, obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 781, in _set
    self._error_callback(self._value)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 263, in error_callback
    raise exception275
ImportError: DLL load failed while importing slp: The specified module could not be found.
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-6:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-7:
Process SpawnPoolWorker-11:
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-12:
Process SpawnPoolWorker-13:
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-10:
Traceback (most recent call last):
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 316, in _export_multithreaded
    MediaExporter._show_progress(outqueue.qsize(), expected_size)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 461, in _show_progress
    print(f"-- Files done: {format_progress(current_size, total_size)}",
KeyboardInterrupt

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 178, in <module>
    sys.exit(main())
             ^^^^^^
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 171, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\main\main.py", line 82, in main
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
    convert_assets(asset_path, args)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\source\repos\openage\build\openage\convert\main.py", line 119, in convert_assets
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
    convert(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 120, in export
    MediaExporter._export_multithreaded(
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 277, in _export_multithreaded
    with multiprocessing.Pool(worker_count) as pool:
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 739, in __exit__
    self.terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 657, in terminate
    self._terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 698, in _terminate_pool
    raise AssertionError(
AssertionError: Cannot have cache with result_handler not alive

@heinezen heinezen mentioned this issue Feb 23, 2024
Merged
@heinezen
Copy link
Member

Ah looks like it's the import error again.

#1628 should produce less import errors, at least for fresh builds that run from the build directory.

@brett-hamilton
Copy link
Author

I used the change in #1628 and re-built the project, here is that output:

converting from 'C:\Program Files (x86)\Microsoft Games\Age of Empires II The Conquerors Expansion Trial'
INFO [py] Looking for compatible games to convert...
INFO [py] Compatible game edition detected:
INFO [py]  * Age of Empires 2: The Conqueror's Trial Version
INFO [py] converting metadata
INFO [py] using cached wrapper: <user directory>\AppData\Local\Temp\AOC_empires2_x1.dat.pickle
INFO [py] Finished metadata read (1.27 seconds)
INFO [py] Starting conversion...
INFO [py] Extracting Genie data...
INFO [py] Creating API-like objects...
INFO [py] Linking API-like objects...
INFO [py] Generating auxiliary objects...
INFO [py] Creating nyan objects...
INFO [py] Creating requests for media export...
INFO [py] Finished data conversion (1.48 seconds)
INFO [py] Starting export...
INFO [py] Dumping info file...
INFO [py] Dumping data files...
INFO [py] Exporting media files...
INFO [py] -- Exporting graphics files...
Exception in thread Thread-3 (_handle_results):
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 789, in _export_texture
    from ...value_object.read.media.slp import SLP
ImportError: DLL load failed while importing slp: The specified module could not be found.
"""-- Files done:   0/275

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1073, in _bootstrap_inner
    self.run()   0/275
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\threading.py", line 1010, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 595, in _handle_results
    cache[job]._set(i, obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 781, in _set
    self._error_callback(self._value)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 263, in error_callback
    raise exception275
ImportError: DLL load failed while importing slp: The specified module could not be found.
Process SpawnPoolWorker-2:
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-5:
Process SpawnPoolWorker-4:
Process SpawnPoolWorker-6:
Process SpawnPoolWorker-9:
Process SpawnPoolWorker-7:
Process SpawnPoolWorker-10:
Process SpawnPoolWorker-8:
Process SpawnPoolWorker-11:
Process SpawnPoolWorker-12:
Process SpawnPoolWorker-13:
Traceback (most recent call last):
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 315, in _export_multithreaded
    while outqueue.qsize() < expected_size:
          ^^^^^^^^^^^^^^^^
  File "<string>", line 2, in qsize
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\managers.py", line 820, in _callmethod
    conn.send((self._id, methodname, args, kwds))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 301, in _send_bytes
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
    nwritten, err = ov.GetOverlappedResult(True)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
Traceback (most recent call last):
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)

  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 131, in worker
    put((job, i, result))
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
During handling of the above exception, another exception occurred:
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\queues.py", line 396, in put
    self._writer.send_bytes(obj)
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 183, in <module>
KeyboardInterrupt
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\connection.py", line 293, in _send_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
    sys.exit(main())
             ^^^^^^
  File "<user directory>\source\repos\openage\build\openage\__main__.py", line 176, in main
    return args.entrypoint(args, cli.error)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\source\repos\openage\build\openage\main\main.py", line 82, in main
    convert_assets(asset_path, args)
  File "<user directory>\source\repos\openage\build\openage\convert\main.py", line 119, in convert_assets
    convert(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 36, in convert
    convert_metadata(args)
  File "<user directory>\source\repos\openage\build\openage\convert\tool\driver.py", line 105, in convert_metadata
    ModpackExporter.export(modpack, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\modpack_exporter.py", line 60, in export
    MediaExporter.export(modpack.get_media_files(), sourcedir, modpack_dir, args)
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 120, in export
    MediaExporter._export_multithreaded(
  File "<user directory>\source\repos\openage\build\openage\convert\processor\export\media_exporter.py", line 277, in _export_multithreaded
    with multiprocessing.Pool(worker_count) as pool:
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 739, in __exit__
    self.terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 657, in terminate
    self._terminate()
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\util.py", line 227, in __call__
    res = self._callback(*self._args, **self._kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<user directory>\AppData\Local\Programs\Python\Python312\Lib\multiprocessing\pool.py", line 698, in _terminate_pool
    raise AssertionError(
AssertionError: Cannot have cache with result_handler not alive

@heinezen
Copy link
Member

I think I have debug this myself on Windows.

@heinezen heinezen linked a pull request May 17, 2024 that will close this issue
Open
@heinezen heinezen linked a pull request May 17, 2024 that will close this issue
Enhance DLL search on Windows #1649
Open
@heinezen
Copy link
Member

Hey, I have finally found the cause of the issue. The reason it didn't work was that the DLL paths weren't propagated to the multiprocessing subprocesses, which then made th import fail. #1649 will fix the issue and add even more sane defaults.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Behaving differently as it should behave lang: python Done in Python code os: windows Windows-specific issue
Projects
openage converter
Status: 📋 Backlog
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Enhance DLL search on Windows heinezen/openage
2 participants
@heinezen @brett-hamilton