Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scylla-gdb/test_misc.py::test_active_sstables is flaky: gdb.MemoryError: Cannot access memory at address 0x300000000000000 #18618

Closed
tgrabiec opened this issue May 10, 2024 · 0 comments
Labels
area/diagnostics debug tools Backport candidate symptom/ci stability Issues that failed in ScyllaDB CI - tests and framework

Comments

@tgrabiec
Copy link
Contributor

=================================== FAILURES ===================================
_____________________________ test_active_sstables _____________________________

gdb = <module 'gdb' from '/usr/share/gdb/python/gdb/__init__.py'>

    def test_active_sstables(gdb):
>       scylla(gdb, 'active-sstables')

test_misc.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

gdb = <module 'gdb' from '/usr/share/gdb/python/gdb/__init__.py'>
cmd = 'active-sstables'

    def scylla(gdb, cmd):
>       return gdb.execute('scylla ' + cmd, from_tty=False, to_string=True)
E       gdb.error: Error occurred in Python: Cannot access memory at address 0x300000000000000

test_misc.py:7: error
----------------------------- Captured stderr call -----------------------------
Traceback (most recent call last):
  File "/jenkins/workspace/scylla-master/scylla-ci/scylla/test/scylla-gdb/../../scylla-gdb.py", line 1884, in invoke
    for sst in find_active_sstables():
  File "/jenkins/workspace/scylla-master/scylla-ci/scylla/test/scylla-gdb/../../scylla-gdb.py", line 1835, in find_active_sstables
    yield sstable_ptr.reinterpret_cast(sstable_ptr_type)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
gdb.MemoryError: Cannot access memory at address 0x300000000000000

Possible cause is that find_single_sstable_readers() yields an object which is mistakenly identified as sstable reader, and hence its _sst._p field doesn't actually point to an sstable object but is some other data.

@tgrabiec tgrabiec added area/diagnostics debug tools symptom/ci stability Issues that failed in ScyllaDB CI - tests and framework labels May 10, 2024
tgrabiec added a commit to tgrabiec/scylla that referenced this issue May 14, 2024
For the purpose of scylla-gdb.py command "scylla
active-sstables". Before the patch, readers were located by scanning
the heap for live objects with vtable pointers corresponding to
readers. It was observed that the test scylla_gdb/test_misc.py::test_active_sstables started failing like this:

  gdb.error: Error occurred in Python: Cannot access memory at address 0x300000000000000

This could be explained by there being a live object on the heap which
used to be a reader but now is a different object, and the _sst field
contains some other data which is not a pointer.

To fix, track readers explicitly in a linked list so that the gdb
script can reliably walk readers.

Fixes scylladb#18618.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/diagnostics debug tools Backport candidate symptom/ci stability Issues that failed in ScyllaDB CI - tests and framework
Projects
None yet
Development

No branches or pull requests

2 participants