master/dtest-debug: test_ignore_invalid_roles - Error from server: code=2200 [Invalid query] message="unconfigured table roles"

[mykaul]

Relevant log:

06:28:27,199 750     dtest_setup                    DEBUG    dtest_setup.py      :61   | test_ignore_invalid_roles: Allocated cluster ID 76: /jenkins/workspace/scylla-master/dtest-debug/scylla/.dtest/dtest-ijihcjrp
06:28:27,253 750     ccm                            DEBUG    cluster.py          :754  | test_ignore_invalid_roles: start_nodes: no_wait=False wait_for_binary_proto=True wait_other_notice=True wait_normal_token_owner=True force_wait_for_cluster_start=True
06:28:27,257 750     ccm                            DEBUG    cluster.py          :754  | test_ignore_invalid_roles: node1: Starting scylla: args=['/jenkins/workspace/scylla-master/dtest-debug/scylla/.dtest/dtest-ijihcjrp/test/node1/bin/scylla', '--options-file', '/jenkins/workspace/scylla-master/dtest-debug/scylla/.dtest/dtest-ijihcjrp/test/node1/conf/scylla.yaml', '--log-to-stdout', '1', '--abort-on-seastar-bad-alloc', '--abort-on-lsa-bad-alloc', '1', '--abort-on-internal-error', '1', '--api-address', '127.0.76.1', '--smp', '2', '--memory', '1024M', '--developer-mode', 'true', '--default-log-level', 'info', '--blocked-reactor-notify-ms', '5000', '--overprovisioned', '--prometheus-address', '127.0.76.1', '--unsafe-bypass-fsync', '1', '--kernel-page-cache', '1', '--commitlog-use-o-dsync', '0', '--max-networking-io-control-blocks', '1000'] wait_other_notice=True wait_for_binary_proto=True
06:28:30,068 750     ccm                            DEBUG    cluster.py          :754  | test_ignore_invalid_roles: node1: Starting scylla-jmx: args=['/jenkins/workspace/scylla-master/dtest-debug/scylla/.dtest/dtest-ijihcjrp/test/node1/bin/symlinks/scylla-jmx', '-Dapiaddress=127.0.76.1', '-Djavax.management.builder.initial=com.scylladb.jmx.utils.APIBuilder', '-Djava.rmi.server.hostname=127.0.76.1', '-Dcom.sun.management.jmxremote', '-Dcom.sun.management.jmxremote.host=127.0.76.1', '-Dcom.sun.management.jmxremote.port=7199', '-Dcom.sun.management.jmxremote.rmi.port=7199', '-Dcom.sun.management.jmxremote.local.only=false', '-Xmx256m', '-XX:+UseSerialGC', '-Dcom.sun.management.jmxremote.authenticate=false', '-Dcom.sun.management.jmxremote.ssl=false', '-jar', '/jenkins/workspace/scylla-master/dtest-debug/scylla/.dtest/dtest-ijihcjrp/test/node1/bin/scylla-jmx-1.0.jar']
06:28:31,423 750     auth_roles_test                DEBUG    auth_roles_test.py  :1116 | test_ignore_invalid_roles: Default role created by ['node1']
06:28:31,435 750     cassandra.policies             INFO     policies.py         :290  | test_ignore_invalid_roles: Using datacenter 'datacenter1' for DCAwareRoundRobinPolicy (via host '127.0.76.1:9042'); if incorrect, please specify a local_dc to the constructor, or limit contact points to local cluster nodes
06:28:31,570 750     errors                         ERROR    conftest.py         :202  | test_ignore_invalid_roles: test failed: 
self = <auth_roles_test.TestAuthRoles object at 0x7fe6e13e6010>

    def test_ignore_invalid_roles(self):
        """
        The system_auth.roles table includes a set of roles of which each role
        is a member. If that list were to get out of sync, so that it indicated
        that roleA is a member of roleB, but roleB does not exist in the roles
        table, then the result of LIST ROLES OF roleA should not include roleB
        @jira_ticket CASSANDRA-9551
        """
    
        self.prepare()
        cassandra = self.get_session(user="cassandra", password="cassandra")
        cassandra.execute("CREATE ROLE mike WITH LOGIN = true")
        # hack an invalid entry into the roles table for roleA
>       cassandra.execute("UPDATE system_auth.roles SET member_of = {'role1'} where role = 'mike'")

auth_roles_test.py:1077: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cassandra/cluster.py:2729: in cassandra.cluster.Session.execute
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="unconfigured table roles"

cassandra/cluster.py:5120: InvalidRequest
06:28:31,570 750     errors                         ERROR    conftest.py         :202  | test_ignore_invalid_roles: test failed: 
self = <auth_roles_test.TestAuthRoles object at 0x7fe6e13e6010>

    def test_ignore_invalid_roles(self):
        """
        The system_auth.roles table includes a set of roles of which each role
        is a member. If that list were to get out of sync, so that it indicated
        that roleA is a member of roleB, but roleB does not exist in the roles
        table, then the result of LIST ROLES OF roleA should not include roleB
        @jira_ticket CASSANDRA-9551
        """
    
        self.prepare()
        cassandra = self.get_session(user="cassandra", password="cassandra")
        cassandra.execute("CREATE ROLE mike WITH LOGIN = true")
        # hack an invalid entry into the roles table for roleA
>       cassandra.execute("UPDATE system_auth.roles SET member_of = {'role1'} where role = 'mike'")

auth_roles_test.py:1077: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
cassandra/cluster.py:2729: in cassandra.cluster.Session.execute
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   cassandra.InvalidRequest: Error from server: code=2200 [Invalid query] message="unconfigured table roles"

Unsure how it failed only on debug?

[mykaul]

I think the test needs adjustment to new auth tables?

[nuivall]

or even delete the whole test if it purpose is only to manually break the data and see what happens, in v2 the list should never get out of sync.

[bhalevy]

Seen also in https://jenkins.scylladb.com/view/master/job/scylla-master/job/dtest-release-with-raft/361/testReport/auth_roles_test/TestAuthRoles/Run_Dtest_Parallel_Cloud_Machines___FullDtest___full_split000___test_ignore_invalid_roles/

[nuivall]

@bhalevy I think it will always fail, it's not flaky.