TRWTF is encoding/json



  • package main
    
    import (
    	"encoding/json"
    	"fmt"
    )
    
    // error checking omitted for brevity
    
    func main() {
    	// 2⁴⁸ - 1
    	a := int64(1<<48 - 1)
    
    	// encode it in json
    	b, _ := json.Marshal(a)
    
    	// decode it
    	var c interface{}
    	_ = json.Unmarshal(b, &c)
    
    	// back to json
    	d, _ := json.Marshal(c)
    
    	// okay, let's put it back now
    	err := json.Unmarshal(d, &a)
    
    	// what?
    	fmt.Println(err)
    }
    

    json: cannot unmarshal number 2.81474976710655e+14 into Go value of type int64



  • What? A Go standard library failing to work correctly? I am shocked, I tell you. Shocked.



  • @Ben L. said:


    b, _ := json.Marshal(a)

    Returns two values in a tuple. Sounds reasonable. @Ben L. said:

            var c interface{}
    	_ = json.Unmarshal(b, &c)
    

    Returns one value and modifies another value by reference... Yuck. Consistency, anyone?



  • @flabdablet said:

    What? A Go standard library failing to work correctly? I am shocked, I tell you. Shocked.

    It's okay, you can just pull in the patch, and then recompile every Go program on your system*. Sounds much simpler than downloading a patched shared library, right?


    (*Statistically speaking, you probably don't have any Go programs on your computer, since you aren't one of the Go developers. So just pretend for a minute that Go is used to make actual software and that you--perhaps in a moment of profound drunkenness, or after suffering a nasty concussion--installed some on your computer.)



  • @Bulb said:

    Returns one value and modifies another value by reference... Yuck. Consistency, anyone?

    Maybe it was too big to return by value? What's more perplexing is that Go is a modern language that apparently defaults to passing objects by value and makes you explicitly pass them by reference. That's a feather in your cap.



  • The two marshallings give me:

    • [50 56 49 52 55 52 57 55 54 55 49 48 54 53 53]
    • [50 46 56 49 52 55 52 57 55 54 55 49 48 54 53 53 101 43 49 52]

    Since when is that valid JSON?
    Either it's not using a comma to separate entries in the array, or its reading them all as a single number.
    In either case: WTF, Go?



  • We get it. Go sucks.



  • Not to defend Go, but I smell a PEBKAC. Namely, RTFM would fix this non-issue issue.  http://golang.org/doc/articles/json_and_go.html : Specfically the section Generic JSON with interface{}

    I'll now go back to my Sisyphus language, InstallScript.

  • Considered Harmful

    [quote user="TFM"]Cyclic data structures are not supported; they will cause Marshal to go into an infinite loop.[/quote]

    That's so awesome. What a mature language. So if I pass in an invalid input it throws an exception returns an error hangs my program and presumably chews up more and more memory forever.



  • @joe.edwards said:

    So if I pass in an invalid input it throws an exception returns an error hangs my program and presumably chews up more and more memory forever.

    Don't worry, with all the memory you've lost to static linking, it won't take long for the system to run out of memory and crash. They solved the Halting Problem!



  • How do you fuck up JSON parsing? It's a simplified form of YAML. I could write a damn Bash script that can parse JSON.


    Go

    Oh. I see.




  • all json numbers are treated as floating point numbers
    I don't even...


  • @Ben L. said:

    cough

    @minux...@gmail.com said:

    json doesn't preserve value types, and in fact, all json numbers are treated as floating point numbers,
    so if you marshal a floating point value, don't expect to correctly unmarshal it back as an integer.

    That could not be more wrong.

    Unless by "JSON" he means "JSON as interpreted by the retards who wrote this Go library."

    More specifically, it is true that JSON doesn't preserve value types, but it's untrue that all JSON numbers are treated as floating point numbers. Only numbers that either 1) don't fit in a int/Int32 or long/Int64 datatype, or 2) have a fractional component that can't be preserved by a int or long. (And even that isn't really true; if you have an "infinite precision" type like BigInt in Java you could put a JSON number in it. Why not? Spec doesn't limit number of digits of ints.)

    (If so, Twitter would be very interested to learn that, as TweetIDs are well above the 32-bit range of integers, but can't be treated as floating point numbers, as floating point numbers don't have enough digits of integer precision. Twitter is using JSON wrong! And I guess Golang doesn't care if people can't use their language to talk to Twitter APIs! Hah.)



  • @blakeyrat said:

    @Ben L. said:
    cough

    @minux...@gmail.com said:

    json doesn't preserve value types, and in fact, all json numbers are treated as floating point numbers,
    so if you marshal a floating point value, don't expect to correctly unmarshal it back as an integer.

    That could not be more wrong.

    Unless by "JSON" he means "JSON as interpreted by the retards who wrote this Go library."

    More specifically, it is true that JSON doesn't preserve value types, but it's untrue that all JSON numbers are treated as floating point numbers. Only numbers that either 1) don't fit in a int/Int32 or long/Int64 datatype, or 2) have a fractional component that can't be preserved by a int or long.

    (If so, Twitter would be very interested to learn that, as TweetIDs are well above the 32-bit range of integers, but can't be treated as floating point numbers, as floating point numbers don't have enough digits of integer precision. Twitter is using JSON wrong! And I guess Golang doesn't care if people can't use their language to talk to Twitter APIs! Hah.)

    Oh come on. Everybody knows that 2.81474976710655e+14 is the best way to store an integer! What could possibly go wrong?



  • @Ben L. said:

    cough

    WTF? I'm trying to understand the code that reproduces the bug.. so "f" is a float, even though it implements an empty interface? But when Go tries to marshal the float into JSON, it sends it as "1.111111e+06" instead of as an integer? And then it bitches because you can't unmarshal an exponent into a float?

    JSON doesn't specify a size limit on numbers, so how would Go deal with a 300 digit integer? tries it Oh, it just panics immediately. That's good.



  • Ben L I demand that you persue this bug until minux...@gmail.com pulls his head out of his ass.

    Also taunt him into emailing you so I can tell what the rest of his email address is. I'm guessing "minux_is_my_love_child@gmail.com".



  • @morbiuswilters said:

    how would Go deal with a 300 digit integer?

    WONTFIX. 300 digit integers are treated as.,.uh...cabbages. Yeah, that's it. Brassica.



  • @blakeyrat said:

    Ben L I demand that you persue this bug until minux...@gmail.com pulls his head out of his ass.

    Also taunt him into emailing you so I can tell what the rest of his email address is. I'm guessing "minux_is_my_love_child@gmail.com".

    Click on the dots. You can also get my full name that way.



  • @Ben L. said:

    @blakeyrat said:
    Ben L I demand that you persue this bug until minux...@gmail.com pulls his head out of his ass.

    Also taunt him into emailing you so I can tell what the rest of his email address is. I'm guessing "minux_is_my_love_child@gmail.com".

    Click on the dots. You can also get my full name that way.

    minux.ma@gmail.com. I have never been more disappointed in my life.

    :(



  • @blakeyrat said:

    Ben L I demand that you persue this bug until minux...@gmail.com pulls his head out of his ass.

    Also taunt him into emailing you so I can tell what the rest of his email address is. I'm guessing "minux_is_my_love_child@gmail.com".

    I don't even know what "minux" is.. some combination of Minix and Linux?



  • @morbiuswilters said:

    I don't even know what "minux" is.. some combination of Minix and Linux?

    "Minotaur's UNIX": half beast, half POSIX compliant operating system.



  • @Vanders said:

    @morbiuswilters said:
    I don't even know what "minux" is.. some combination of Minix and Linux?
    "Minotaur's UNIX": half beast, half POSIX compliant operating system.
     

    That's called Linux.



  • @morbiuswilters said:

    JSON doesn't specify a size limit on numbers, so how would Go deal with a 300 digit integer? tries it Oh, it just panics immediately. That's good.

    Wasn't panic meant to be Go's way of saying "shit's fucked up hard"?
    I was under the impression that an unexpected object being deserialised was generally a recoverable exception in every programming language ever.



  • @drurowin said:

    @Vanders said:

    @morbiuswilters said:
    I don't even know what "minux" is.. some combination of Minix and Linux?

    "Minotaur's UNIX": half beast, half POSIX compliant operating system.
     

    That's called Linux.

    Zing!



  • @Salamander said:

    I was under the impression that an unexpected object being deserialised was generally a recoverable exception in every programming language ever.

    Go: If Google can't recover from their error, why should you?



  • Oh wait never mind; I fail reading comprehension.



  • I've submitted my rebuttal. I urge everyone who agrees that this is a bug to consider doing so as well.

    Here's the program I posted with that comment:

    package main
    
    import (
    	"encoding/json"
    	"fmt"
    )
    
    func main() {
    	// both of these are legal
    	var a float64 = 1111111
    	var b int64 = 1.111111e+6
    
    	c, err := json.Marshal(a)
    	check(err)
    	d, err := json.Marshal(b)
    	check(err)
    
    	fmt.Printf("%v -> %v -> %q\n", 1111111, a, c)
    	fmt.Printf("%v -> %v -> %q\n", 1.111111e+6, b, d)
    	fmt.Printf("%v == %v -> %v, %v\n", a, b, int64(a) == b, a == float64(b))
    
    	// at this point, you should be convinced that the two literals are interchangable.
    
    	// but encoding/json disagrees:
    	check(json.Unmarshal(c, &b))
    	check(json.Unmarshal(d, &a))
    }
    
    func check(err error) {
    	if err != nil {
    		panic(err)
    	}
    }
    


  • @Ben L. said:

    I've submitted my rebuttal. I urge everyone who agrees that this is a bug to consider doing so as well.

    Godspeed, but I think a better approach would be to point out that while it's true the JSON spec doesn't preserve value types, it makes sense to store numbers in the most accurate format possible... in this case, it should have been stored (and subsequently read-in) as a long/int64 and not a float. Mention that the current implementation makes it impossible to use Go's JSON parser with JSON sources like the Twitter API, if they need a practical example as to why its wrong.

    I'd post myself but since I never have used Go and never will, I don't feel I have a right to.



  • @morbiuswilters said:

    @Salamander said:
    I was under the impression that an unexpected object being deserialised was generally a recoverable exception in every programming language ever.

    Go: If Google can't recover from their error, why should you?

    Morbs, can you post your code? I'm curious.



  • @Ben L. said:

    @morbiuswilters said:
    @Salamander said:
    I was under the impression that an unexpected object being deserialised was generally a recoverable exception in every programming language ever.

    Go: If Google can't recover from their error, why should you?

    Morbs, can you post your code? I'm curious.

    I just modified the value of your code to start with a 300-digit string. Now I get that it can't unmarshal a 300-digit int into an i64.

    I'm assuming Go has some arbitrary-length integer type somewhere in one of its core libraries.. I'm curious if you could unmarshal a 300-digit int into one of those.



  • @morbiuswilters said:

    @Ben L. said:
    @morbiuswilters said:
    @Salamander said:
    I was under the impression that an unexpected object being deserialised was generally a recoverable exception in every programming language ever.

    Go: If Google can't recover from their error, why should you?

    Morbs, can you post your code? I'm curious.

    I just modified the value of your code to start with a 300-digit string. Now I get that it can't unmarshal a 300-digit int into an i64.

    I'm assuming Go has some arbitrary-length integer type somewhere in one of its core libraries.. I'm curious if you could unmarshal a 300-digit int into one of those.

    Turns out: yes



  • @blakeyrat said:

    @minux...@gmail.com said:
    json doesn't preserve value types, and in fact, all json numbers are treated as floating point numbers,
    so if you marshal a floating point value, don't expect to correctly unmarshal it back as an integer.

    That could not be more wrong.

    Unless by "JSON" he means "JSON as interpreted by the retards who wrote this Go library."

    More specifically, it is true that JSON doesn't preserve value types, but it's untrue that all JSON numbers are treated as floating point numbers. Only numbers that either 1) don't fit in a int/Int32 or long/Int64 datatype, or 2) have a fractional component that can't be preserved by a int or long.

    Don't you remember what the JS in JSON stands for? JavaScript doesn't have an integer type, so JSON was designed to treat all numbers as floating point.

    Twitter's numerical ids are a legacy value, you're supposed to use id_str.



  • @MiffTheFox said:

    @blakeyrat said:
    @minux...@gmail.com said:
    json doesn't preserve value types, and in fact, all json numbers are treated as floating point numbers, so if you marshal a floating point value, don't expect to correctly unmarshal it back as an integer.

    That could not be more wrong.

    Unless by "JSON" he means "JSON as interpreted by the retards who wrote this Go library."

    More specifically, it is true that JSON doesn't preserve value types, but it's untrue that all JSON numbers are treated as floating point numbers. Only numbers that either 1) don't fit in a int/Int32 or long/Int64 datatype, or 2) have a fractional component that can't be preserved by a int or long.

    Don't you remember what the JS in JSON stands for? JavaScript doesn't have an integer type, so JSON was designed to treat all numbers as floating point.

    Twitter's numerical ids are a legacy value, you're supposed to use id_str.

     

    I thought that it was the counterpart to FRDY.

     



  • @Ben L. said:

    @morbiuswilters said:
    @Ben L. said:
    @morbiuswilters said:
    @Salamander said:
    I was under the impression that an unexpected object being deserialised was generally a recoverable exception in every programming language ever.

    Go: If Google can't recover from their error, why should you?

    Morbs, can you post your code? I'm curious.

    I just modified the value of your code to start with a 300-digit string. Now I get that it can't unmarshal a 300-digit int into an i64.

    I'm assuming Go has some arbitrary-length integer type somewhere in one of its core libraries.. I'm curious if you could unmarshal a 300-digit int into one of those.

    Turns out: yes

    That's good; makes the JSON implementation less WTFy.


  • Discourse touched me in a no-no place

    @MiffTheFox said:

    Don't you remember what the JS in JSON stands for? JavaScript doesn't have an integer type, so JSON was designed to treat all numbers as floating point.
    The specification doesn't specify that all numbers are floating point; it allows the expression of both floating point and integer numbers. It isn't tightly bound to JS either; that's just where it started from. Transforming everything that matches number in the spec into a float… yeah, that'd be a WTF.



  • @MiffTheFox said:

    Don't you remember what the JS in JSON stands for? JavaScript doesn't have an integer type, so JSON was designed to treat all numbers as floating point.

    FunFact: JSON isn't strictly a subset of JavaScript; there are some things which JSON allows in strings which JavaScript does not.
    Also as has been said, the spec says nothing about the types of numerical values other than them being, well, numerical.



  • @MiffTheFox said:

    Don't you remember what the JS in JSON stands for? JavaScript doesn't have an integer type, so JSON was designed to treat all numbers as floating point.

    Miff. When is the last time you posted something correct.

    It doesn't matter that JSON "came from" JavaScript. Read the fucking spec. Tell me where the spec requires all numerical values to be floating point.

    Oh and BTW: you're wrong, JavaScript does have an integer type, it's just not exposed to the programmer. Easily proven: take the number on this example, 1111111111, store that in JS and write it back out and you get 111111111 not 1.11111111e10.

    @MiffTheFox said:

    Twitter's numerical ids are a legacy value, you're supposed to use id_str.

    The numerical ids are the correct value. id_str is only there for the benefit of broken-ass JSON parsers written by morons like the Golang creators and apparently you.


  • Considered Harmful

    @blakeyrat said:

    Oh and BTW: you're wrong, JavaScript does have an integer type, it's just not exposed to the programmer. Easily proven: take the number on this example, 1111111111, store that in JS and write it back out and you get 111111111 not 1.11111111e10.

    Quick test in Firebug console...

    console.log( 1111111111111111111 );
    Yields:
    1111111111111111200


    Your example of 1111111111 does print 1111111111, but then so does 0.1111111111 * 10000000000.0.


  • @blakeyrat said:

    @MiffTheFox said:
    Don't you remember what the JS in JSON stands for? JavaScript doesn't have an integer type, so JSON was designed to treat all numbers as floating point.

    Miff. When is the last time you posted something correct.

    It doesn't matter that JSON "came from" JavaScript. Read the fucking spec. Tell me where the spec requires all numerical values to be floating point.

    Oh and BTW: you're wrong, JavaScript does have an integer type, it's just not exposed to the programmer. Easily proven: take the number on this example, 1111111111, store that in JS and write it back out and you get 111111111 not 1.11111111e10.

    @MiffTheFox said:

    Twitter's numerical ids are a legacy value, you're supposed to use id_str.

    The numerical ids are the correct value. id_str is only there for the benefit of broken-ass JSON parsers written by morons like the Golang creators and apparently you.

    @The fucking spec said:

    4.3.19

    Number value


    primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value

    I assume this is what you mean. The link you gave doesn't say anything about what a number actually is, and linked to ECMA 262. That's what the quote is from.


  • Considered Harmful

    @JSON Spec said:

    It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999.

    Reading comprehension fail emphasis added.



  • @MiffTheFox said:

    I assume this is what you mean. The link you gave doesn't say anything about what a number actually is, and linked to ECMA 262. That's what the quote is from.

    That's not the JSON spec. That's the ECMA spec. You fucking idiot.

    Protip: when being called "wrong", the best thing to do is not to double-down and be even wronger.



  • @joe.edwards said:

    @JSON Spec said:
    It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999.

    Reading comprehension fail emphasis added.

    Ok, so show me where in the spec it says a number can be an integer. The closest I could find was it saying numbers can be expressed as an integer in that syntax thing on the right.



  • @blakeyrat said:

    @MiffTheFox said:
    I assume this is what you mean. The link you gave doesn't say anything about what a number actually is, and linked to ECMA 262. That's what the quote is from.

    That's not the JSON spec. That's the ECMA spec. You fucking idiot.

    Protip: when being called "wrong", the best thing to do is not to double-down and be even wronger.

    The link you clicked says that JSON is based on that ECMA spec (it was only a click away) and offered no evidence to the contrary that the definition of a number was any different.



  • @MiffTheFox said:

    Ok, so show me where in the spec it says a number can be an integer.

    Show me where in the spec it says that a number must be of floating point precision.



  • @MiffTheFox said:

    The link you clicked says that JSON is based on that ECMA spec (it was only a click away) and offered no evidence to the contrary that the definition of a number was any different.

    Based on.
    Would you expect to be able to write arbitrary chunks of JavaScript inside of JSON because it was based on JavaScript?
    If you didn't expect it, why not?



  • @Salamander said:

    @MiffTheFox said:

    The link you clicked says that JSON is based on that ECMA spec (it was only a click away) and offered no evidence to the contrary that the definition of a number was any different.

    Based on.
    Would you expect to be able to write arbitrary chunks of JavaScript inside of JSON because it was based on JavaScript?
    If you didn't expect it, why not?

    You used to be able to until people realized that's a big security risk! The first JSON parser was result = eval(json). JSONP, a modified version of JSON, is JavaScript that calls a function with the JSON object as it's argument.

    @Salamander said:

    @MiffTheFox said:

    Ok, so show me where in the spec it says a number can be an integer.

    Show me where in the spec it says that a number must be of floating point precision.

    @json.com said:

    It is based on a subset of the JavaScript Programming Language, Standard ECMA-262 3rd Edition - December 1999.

    @ECMA 262 like I quoted above said:
    4.3.19

    Number value

    primitive value corresponding to a double-precision 64-bit binary format IEEE 754 value

    I do realize I might be in error since I trusted that what Blakeyrat said was the actual JSON spec. RFC 4627, the actual spec, says nothing about how a number is to be stored in memory, only that it must support fractional numbers and exponents. So yeah, it doesn't say float specifically, but sure as hell doesn't say int. (Although it does mention the possibility of the decode supporting constants such as Infinity and NaN, strongly suggesting floating-point.)

    @RFC 4627 said:

    A number contains an integer component that may be prefixed with an optional minus sign, which may be followed by a fraction part and/or an exponent part.

    @RFC 4627 said:
    Numeric values that cannot be represented as sequences of digits (such as Infinity and NaN) are not permitted.



  • @MiffTheFox said:

    You used to be able to until people realized that's a big security risk! The first JSON parser was result = eval(json)

    ...

    So yeah, it doesn't say float specifically, but sure as hell doesn't say int.

    Used to does not mean it was correct or intended behaviour.
    The earliest info I can find on JSON also includes a Java-based parser.
    Wanna know what's interesting? It doesn't use eval of any kind. Even more interesting, it tries parsing numbers as integers before falling back to floating-point precision.

    What you are continually failing to grasp is that the spec does not specify what the numeric types are stored as because that is left to the discretion of the implementer.
    There is absolutely nothing stopping you from, say, storing all numbers as infinite precision decimals if you wanted to.


Log in to reply