Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Test 1-byte reads #86

Open
vinipsmaker opened this issue Jul 8, 2021 · 4 comments · May be fixed by #92
Open

Test 1-byte reads #86

vinipsmaker opened this issue Jul 8, 2021 · 4 comments · May be fixed by #92

Comments

@vinipsmaker
Copy link

Could there be an option where read() only reads 1 byte at a time? It'd be useful to test buffer management routines on parsers.

@ligurio
Copy link
Owner

ligurio commented Jul 8, 2021

Hi! It's a good idea! Could you tell more about used scenario? What parsers do you test?

@vinipsmaker
Copy link
Author

Could you tell more about used scenario?

Sure.

I had the same issue in the past, but then I'd just mock the read routines and force 1-byte reads through the mocks. I don't like this approach as effectively I won't be testing the same code going into production (the read routines change).

A few hours ago I was facing this issue again and I thought of searching for a new solution and FUSE seemed like a good idea and I've stumbled upon your project. So, “why not ask?” is what I've thought. For my current project, I have a routine such as follows:

for (;;) {
    std::string_view buffer_view(buffer.data(), buffer_used);
    auto lf = buffer_view.find('\n');
    if (lf != std::string_view::npos) {
        line = buffer_view.substr(0, lf + 1);
        break;
    }

    if (buffer.size() == buffer_used)
        buffer.resize(buffer.size() * 2);

    ssize_t nread = read(iobuf->fd,
                         const_cast<char*>(buffer.data()) + buffer_used,
                         buffer.size() - buffer_used);
    if (nread == -1) {
        *errcode = errno;
        return EOF;
    }
    if (nread == 0) {
        if (buffer_used == 0)
            return EOF;

        line = std::string_view{buffer.data(), buffer_used};
        break;
    }
    buffer_used += nread;
}

In short, I read LF-delimited records. The code is very simple and I'm not afraid of it. However I'll have to change this read-loop to use a chunked JSON parser to find the record delimiter. The chunked JSON parser maintains pointers to the stream. If I fuck it up, the parser will keep references to invalid memory. That's a much more complex beast.

I'd like to run the tests ensuring that every read() call on the target fd/file only performs 1-byte reads. That'll make sure the parser stops at every token to ask for more bytes and make sure I handle incomplete streams at every possible layer of my code (triggering new correct reads as needed).

@ligurio
Copy link
Owner

ligurio commented Jul 9, 2021

I'll try to include it in the next release.

@vinipsmaker
Copy link
Author

Thanks.

ligurio added a commit that referenced this issue Aug 9, 2021
@ligurio ligurio linked a pull request Aug 9, 2021 that will close this issue
ligurio added a commit that referenced this issue Aug 11, 2021
ligurio added a commit that referenced this issue Aug 11, 2021
ligurio added a commit that referenced this issue Jan 24, 2022
ligurio added a commit that referenced this issue Jan 24, 2022
ligurio added a commit that referenced this issue Jan 24, 2022
ligurio added a commit that referenced this issue Jan 24, 2022
ligurio added a commit that referenced this issue Jan 24, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants