Bug: Deadlock encountered

[triracle97]

Describe the bug

When I ran surrealdb intensively with live query, the database hang at this point in file src/net/rpc.rs

if WEBSOCKETS.read().await.contains_key(&id) {
   return Err(Error::Request);
}

I used js sdk + reactjs and I tried both surreal docker and self built surreal in an ubuntu server. When I refreshed the page, the connection wasn't cleared.

Steps to reproduce

I can consistently reproduce the error with this scenario

start surreal db with this command

./surreal start --log trace -u test -p test file://./testdb

Then create a simple react project with npx create-react-app my-app

Copy the following snippet to App.js

import logo from './logo.svg';
import './App.css';
import { Surreal } from "surrealdb.js";
import { useEffect } from "react";

const db = new Surreal({
  onError: (err) => {
    console.log('on error', err);
  },
  onClose: (err) => {
    console.log('on close', err);
  },
  onConnect: () => {
    console.log('connected')
  }
});

async function runSingleDB(tableName) {
  // await db.query(`DEFINE TABLE ${tableName}`); <-- uncomment this on first run to create the table
  const res = await db.live(tableName, () => {})
  console.log('Live query id', tableName, res);
  for (let i = 0; i < 1000; i++) {
    await db.update(`${tableName}:${i}`, {
      test: 1,
      haha: Math.random(),
    })
  }
}

async function initDb() {
  try {
    await db.connect('http://<Your server>:8000/rpc', {
      auth: {
        username: "test",
        password: "test"
      },
      namespace: 'test',
      database: 'test'
    })
  } catch (err) {
    console.log('Hi, err', err);
  }
  await runSingleDB('db1')
  await runSingleDB('db2')
  await runSingleDB('db3')
  await runSingleDB('db4')
  await runSingleDB('db5')
}

function App() {
  console.log('App')
  useEffect(() => {
    initDb()
  }, []);

  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <p>
          Edit <code>src/App.js</code> and save to reload.
        </p>
        <a
          className="App-link"
          href="https://reactjs.org"
          target="_blank"
          rel="noopener noreferrer"
        >
          Learn React
        </a>
      </header>
    </div>
  );
}

export default App;

Open 4 tabs with http://localhost:3000, refresh the all tabs several times and then wait around 5 mins, then try to refresh any tab multiple times again, the websocket will be in pending state (the server will hang at above code)

Expected behaviour

I guess this is a deadlock, and it shouldn't happen

SurrealDB version

v1.4.2, built in x86_64-unknown-linux-gnu

Contact Details

No response

Is there an existing issue for this?

• I have searched the existing issues

Code of Conduct

• I agree to follow this project's Code of Conduct

[phughk]

Heya @triracle97 thanks for raising this. The live queries are going through a significant design change that will reduce the chance of this deadlock happening significantly. We are currently seeing another perf issue with 1.4.x in #3906 - it may be loosely related or at least correlated. Will have a look once 3906 is resolved.

[triracle97]

Thanks for replying @phughk!
Is there any side effect if I comment that line to prevent deadlock temporarily, I don't need to reuse live query and I can kill and recreate live query everytime I create new connection

[phughk]

Unfortunately, that isn't possible - the lock is required to check which connection to send live query notifications to.

[triracle97]

Update for this, since updated to 1.5.0, I haven't encountered this and can't reproduce it with above method

[triracle97]

I encountered it again, the issue still persists, just harder to reproduce

[phughk]

Thanks for the update and info - we will try to tackle this in. The solution would be to have channels with messages and a single handler. It's a bit of work and we are very focused on cloud and 2.0 at the moment. But this is on our radar. Hopefully retrying can be a way around this