Behavior of operator>> should more closely resemble that of built-in overloads.

Basically:

eat up leading blanks or newlines
leave the stream where the parsing ended, allowing for more data downstream, which may or may not be json.

#include "json.hpp"
    
template<typename T>
static void test(const std::string& s)
{
    T val{999};
    std::stringstream sstr(s);
    //sstr.exceptions(std::ifstream::failbit | std::ifstream::badbit);

    try {
        size_t i = 0;
        while(sstr >> val) {
            std::cerr << "    result[" << i++ << "]=" << val 
                      << "; streampos=" << sstr.tellg() << std::endl;
        }   

        std::cerr << "    final values: result=" << val 
                  << " i=" << i
                  << " streampos=" << sstr.tellg()
                  << " streamflags=" << sstr.flags() 
                  << std::endl;
    } catch(std::exception& e) {
        std::cerr << "    std::exception:" << e.what() << std::endl;
    }   

    std::cerr << std::endl;
}

int main()
{
    using json = nlohmann::json;
    size_t i = 0;

    for(std::string s : { 
        "", 
        "   ",
        "111",
        "222 \t\n",
        "\n\t 333", 
        " 111 \n222\n \n  333" })
    {   
        std::cerr << "\nTest-case[" << i++ << "]: '" << s << "'" << std::endl;
    
        std::cerr << "With int:" << std::endl;
        test<int>(s);

        std::cerr << "With json:" << std::endl;
        test<json>(s);

        std::cerr << "----" << std::endl;
    }   

    return 0;
}

Test-case[0]: ''
With int:
    final values: result=999 i=0 streampos=-1 streamflags=4098

With json:
    std::exception:parse error - unexpected end of input

----

Test-case[1]: '   '
With int:
    final values: result=999 i=0 streampos=-1 streamflags=4098

With json:
    std::exception:parse error - unexpected end of input

----

Test-case[2]: '111'
With int:
    result[0]=111; streampos=-1
    final values: result=111 i=1 streampos=-1 streamflags=4098

With json:
    result[0]=111; streampos=-1
    std::exception:basic_string::append

----

Test-case[3]: '222 	
'
With int:
    result[0]=222; streampos=3
    final values: result=222 i=1 streampos=-1 streamflags=4098

With json:
    final values: result=222 i=0 streampos=-1 streamflags=4098

----

Test-case[4]: '
	 333'
With int:
    result[0]=333; streampos=-1
    final values: result=333 i=1 streampos=-1 streamflags=4098

With json:
    result[0]=333; streampos=-1
    std::exception:basic_string::append

----

Test-case[5]: ' 111 
222
 
  333'
With int:
    result[0]=111; streampos=4
    result[1]=222; streampos=9
    result[2]=333; streampos=-1
    final values: result=333 i=3 streampos=-1 streamflags=4098

With json:
    std::exception:parse error - unexpected number literal; expected end of input

----

The lines std::exception:basic_string::append revealed a bug in the fill_line_buffer function where

m_line_buffer.append(n - 1, '\x01');

was executed with n = 0. This must be fixed.

About the inputs:

"" (empty input): An empty input is not a valid JSON document. Hence, I think the message parse error - unexpected end of input is adequate.
" " (white space): After ignoring whitespace, this is again empty input, cf. input 1.
"111" (number): Parsing this input yields a number. Parsing the (then empty) input again, is the same as input 1.
"222 \t\n" (number with trailing whitespace): The input is parsed as a number while reading the trailing whitespace. Parsing the (then empty) input again, is the same as input 1.
"\n\t 333" (number with leading whitespace): Same as input 4.
" 111 \n222\n \n 333" (number with leading whitespace, followed by additional numbers): A number must not be followed by a number, so the message parse error - unexpected number literal; expected end of input makes sense to me (note that after merging Exception line #301 this message will contain the byte offset of the error).

I can understand your thoughts about the behavior of the streams. However, I think it is not a good idea to stop parsing once we found a JSON value, because this would mean that code like

json j;
j << "false foo bar";

would run without exception.

Am I too strict about this?

🐛 fixed a bug if parser was called with a stream at EOF (#367)

✅ added a test for EOF error (#367)

An empty input is not a valid JSON document.

An empty input is not a valid integer either, yet cin>>i will not throw - it will consume the blanks and leave i unchanged.

There are two classes of use-cases:

Parse a singular json document from a string (e.g. using parse(...) method or a free function). Here the library should verify that the input is not empty and that there's nothing past the end.
Parse next json document from a stream (operator>>). Here the code cannot assume that the stream must contain a singleton json document and nothing else. Maybe there's some binary data that follows - it's not our business to know.

Think of a what kind of code a new user of the library would write, assuming they have not read any documentation other some examples, when they try to read a bunch of json values from stdin in a loop. Now make the API work that way, following the "design principle of least astonishment" : )

I understand your point and you answered my "Am I too strict about this?" question with "yes" 😄

It would be nice to hear other opinions about this. I still feel uncomfortable accepting non-valid JSON files by just looking at a prefix...

Seems reasonable for streaming to stop after a JSON document has been read. If this is a behavior change, it might require a major version bump.

Note that both j << cin; and cin >> j; call the same code.

I agree that this seems weird, but I'm not sure it's valid right now, since there is no operator<< that takes a string:

json j;
j << "false foo bar";

Note for the future:
Suppose input stream contains arbitrarily large, or infinite number of json records (e.g. sensor data being piped from some upstream process), which happen to not be newline-delimited. calling getline() on such stream will fail.

+1, this is basically the only thing I wish for here. Is there any way to simulate this behavior with the library as it stands now?

Unfortunately not.

I saw someone do it in a way that required a two-stage parse. They parsed the data once, got the position of the parse error, assumed that was the start of the next whole entry, and so parsed again limiting to just before the parse error, and then moving the start position to the error position for the next entry.

In may add special overloads for ifstream and istringstream that use caches and remove caching for general istream cases. Would this work?

In may add special overloads for ifstream and istringstream that use caches
and remove caching for general istream cases. Would this work?

For me it's also a workable solution. You may also consider having the caching only on ifstream since istringstream it's already on memory.

🔨 trying something #367

✅ added test for #367

🔨 trying to fix #367

@ceztko

@ceztko I openend a question at StackOverflow and found out the code I used to "rewind" the stream was not correct. I fixed it in a feature branch and the example from #367 (comment) is now working with MSVC 2015 and MSVC 2017 (see AppVeyor build).

@nlohmann

@nlohmann great news, thanks! Sorry for not having looked at this: I wanted to debug it but I couldn't find the time.

Merged fd4a0ec which fixes this issue. Thanks everybody for the patience.

Test to confirm parsing of multiple JSON records in a istream nlohman…

Merge pull request #764 from pjkundert/develop-simplify-istream

Navigation Menu

Repository navigation

Behavior of operator>> should more closely resemble that of built-in overloads. #367

48 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Behavior of operator>> should more closely resemble that of built-in overloads. #367

Description

Activity

nlohmann commented on Nov 24, 2016

TurpentineDistillery commented on Nov 24, 2016

nlohmann commented on Nov 24, 2016

gregmarr commented on Nov 25, 2016

TurpentineDistillery commented on Nov 25, 2016

mrkgnao commented on Jan 13, 2017

nlohmann commented on Jan 13, 2017

gregmarr commented on Jan 13, 2017

48 remaining items

nlohmann commented on May 21, 2017

ceztko commented on May 21, 2017

nlohmann commented on Jun 12, 2017

ceztko commented on Jun 12, 2017

nlohmann commented on Jun 13, 2017

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions