How not to generate a token

[bardofspoons421]

So, I've been working with my esteemed compatriot developers in the company mine merged with (They were our biggest competitor) on a series of web service calls that take XML in post data, and return XML. So far rather simple and straightforward.

One of the features added was a token to store on the device (The consumer of this service are Motorola/Symbol PDTs) instead of a password. When the first call to the service is made, a user name and password are provided, in return it supplies a token to identify that session, with an expiry set, that we then use for all future calls. Once again, so far, so good

The first attempt, the developer working on the service side of things (I work on the device end) provides a token that was the Unix time, in milliseconds. Not very good from a security point of view. As an added bonus, they built the query that checked the token against the database by string concatenation, without any sort of sanitation. Leading to this:

' or 1=1 and Rownum=1 --

Being a valid token.

So, I write up a nice document, detailing these flaws and make the following suggestion

The token should be a random sequence of bytes, generated by a secure random number generator, of at least 1024 bits (128 bytes). These bytes should then be encoded in a format for easy storage in the database and transmission inside XML, such as Base64 encoding.

Much better from a security standpoint. Since they're in java, it should be a simple matter along the lines of (With classes and functions excluded for brevity):

import java.security;
import org.apache.commons.codec.binary;
SecureRandom sr = new SecureRandom();

byte tokenBytes[] =  new byte[128];

String tokenString;
sr.nextBytes(tokenBytes);

tokenString = Base64.encodeBase64String(tokenBytes);

Seems straightforward enough, even with my incredibly rusty java skills. Sent off these suggestions, and even have the CIO backing me for this. This was about a week or so ago.

Fast-forward to today. I was writing up some tests for the service, XML data sent through curl to the web service, then piped through xmllint to validate. I notice something funny about the new tokens (UUID-formatted for easier reading):

    96FC3ABF-2A32-3E80-E044-00144F80B9CE
    96FC39A7-E802-3BA6-E044-00144F80B9CE

These a from two subsequent calls a few seconds apart. That doesn't look very random to me. I go and look at their code base. To generate this value, they make a call into Oracle to the SYS_GUID function. Which, from the Oracle docs:

SYS_GUID generates and returns a globally unique identifier (RAW value) made up of 16 bytes. On most platforms, the generated identifier consists of a host identifier, a process or thread identifier of the process or thread invoking the function, and a nonrepeating value (sequence of bytes) for that process or thread.

That's not really random at all. Actually, it might have less entropy than the time-stamp value.

And as I write this, I get back the following email (I pointed out these issues and others [Such as date formats not conforming to ISO 8601]):

Thanks Matt, we will discuss how we want to deal with this later.

This isn't something to decide on later. This is something to fix before production.

Add to this things like them building XML by string concatenation (I had an issue with an ampersand being in the data and not escaped). This project is going swimmingly.

[Medezark]

Oracle -- 'nuff said.

[henke37]

It is guaranteed to be unique as long the counter doesn't roll over. And at least the Microsoft GUID generator uses the time in addition to the mac address.

[HighlyPaidContractor]

@bardofspoons42 said:

Add to this things like them building XML by string concatenation (I had an issue with an ampersand being in the data and not escaped). This project is going swimmingly.

A company I worked for had a rule where all XML had to be generated by concatenation "because it's faster than object creation."  This also applied to any dynamic html or SQL.

[HighlyPaidContractor]

@henke37 said:

It is guaranteed to be unique as long the counter doesn't roll over. And at least the Microsoft GUID generator uses the time in addition to the mac address.

Unique, but not secure.

[blakeyrat]

@Medezark said:

Oracle -- 'nuff said.

Yeah, as soon as you hear "Oracle" (combined with "Java" making it even worse), I just think of Hudson from Aliens: "That's it man, game over man, game over!"

[Someone_You_Know]

@blakeyrat said:

@Medezark said:

Oracle -- 'nuff said.

Yeah, as soon as you hear "Oracle" (combined with "Java" making it even worse), I just think of Hudson from Aliens: "That's it man, game over man, game over!"

 

Nuke the website from orbit.

It's the only way to be sure.

[dhromed]

@Someone You Know said:

Nuke the website from orbit.

It's the only way to be sure.

 

FYI, That's thirteen words.

[Someone_You_Know]

@dhromed said:

@Someone You Know said:

Nuke the website from orbit.

It's the only way to be sure.

 

FYI, That's thirteen words.

 

Fourteen if you're a newspaper editor. They seem to be obssessed with spelling it "Web site", with a capital W and a space, just like no one else in the world does.

[bardofspoons421]

I just got off a meeting involving this and other issues (one method always failing the token check, even if it works for others at the same time; another method returning the wrong XML element). Oh my goodness.

First, the token "not being random enough" (not random at all) is put off "for a few weeks". Considering most of that dev team spends their time putting out fires, this is roughly synonymous with "The heat death of the universe".

Next, one call returning non-conformant data (The string 'null' instead of an empty element) was because "I'm lazy"

Another method (The one that was returning the wrong element, actually), is returning xml that doesn't match the schema. Now, I've provided him with both the schema, and a set of samples that match the schema (Validated before I sent them off, actually). And then he tries to say it was in the samples I sent him.

Finally, the one that takes the cake. In each call that deals with the token, there is a <Status> element that contains both a status code, and a message. The code is constrained, per the schema, to be one of three values: Success, TokenFailure, and Failed. I added the TokenFailure response so the device can redisplay the login dialog, instead of a non-helpful message. Right now, any failure is returning Failed, even though the message describes a token failure, and I'm not parsing free-form text. This was met with the following arguments:

1. The schema is only for the structure (Or some argument like that...)
2. He would have to make another database query (Why? You already know the token failed based on the message)
3. He's holding the state as a boolean (Well, that doesn't match what was spec'd out)

Because of all this, I spent the afternoon hacking together a Perl script. At 255 lines, this about increases the amount of Perl I've written by an order of magnitude. And also, from this script, the service we're working on is probably the most tested thing in their entire system.

I suppose I'm done ranting for now. But I really don't want to call this guy and argue with him more...

[Cad_Delworth]

@Someone You Know said:

Fourteen if you're a newspaper editor. They seem to be obssessed with spelling it "Web site", with a capital W and a space, just like no one else in the world does.

That's not actually wrong: it just proves that they haven't bought a new dictionary since 1999. Oh, and obviously they would never think to use a dictionary Web site such as http://oxforddictionaries.com, which—pleasingly—differentiates between US English and World English.

[blakeyrat]

@Cad Delworth said:

@Someone You Know said:

Fourteen if you're a newspaper editor. They seem to be obssessed with spelling it "Web site", with a capital W and a space, just like no one else in the world does.

That's not actually wrong: it just proves that they haven't bought a new dictionary since 1999. Oh, and obviously they would never think to use a dictionary Web site such as http://oxforddictionaries.com, which—pleasingly—differentiates between US English and World English.

We can never be friends.

[bardofspoons421]

Another bit of added fun: Their system runs on Tomcat with JSP and Struts. I know minimal about any of these. But, when developing this software, I noticed that their web server is returning 200 OK for error pages, instead of 404 or 500 or whatever. I managed to convince them that these should really be set correctly, and it would aide my error handling on the device, as well as be proper HTTP.

Shortly after I got this in an email:

We actually have stumbled on a bug in Apache Web server. Even with our custom HTML page, the Apache web server is supposed to return the correct HTTP status code in the response.

I would thing that one thing Apache, Tomcat, JSP and Struts would be able to do, and would be well tested, is set an HTTP status code. Not exactly a demanding thing to expect from any sort of web stack

Eventually, they implemented this, and today I got this:

This broke every single customer application we have. Setting that response code in the JSP is causing havoc.

So, setting error codes on error pages is breaking everything else? WTF were they doing?!

[vyznev]

@bardofspoons42 said:

import java.security;
import org.apache.commons.codec.binary;

SecureRandom sr = new SecureRandom();
byte tokenBytes[]] = new byte[128];
String tokenString;

sr.nextBytes(tokenBytes);
tokenString = Base64.encodeBase64String(tokenBytes);

It sounds like you have the problem mostly solved already. Why not just give up on trying to get the other guy, who obviously doesn't understand either the problem or the obvious solution, to fix it and simply do it yourself?

If the problem is that you lack the authority to do that, or don't want to suffer the consequences of publicly embarrassing him, consider wrapping the code you have here in a simply class, testing that it works and sending it to him.  Tell him that you already had it written anyway, and if he'll just review and commit it you can both stop wasting your time on it.

[blakeyrat]

@bardofspoons42 said:

Another bit of added fun: Their system runs on Tomcat with JSP and Struts. I know minimal about any of these. But, when developing this software, I noticed that their web server is returning 200 OK for error pages, instead of 404 or 500 or whatever. I managed to convince them that these should really be set correctly, and it would aide my error handling on the device, as well as be proper HTTP.

Shortly after I got this in an email:

We actually have stumbled on a bug in Apache Web server. Even with our custom HTML page, the Apache web server is supposed to return the correct HTTP status code in the response.

I would thing that one thing Apache, Tomcat, JSP and Struts would be able to do, and would be well tested, is set an HTTP status code. Not exactly a demanding thing to expect from any sort of web stack

Eventually, they implemented this, and today I got this:

This broke every single customer application we have. Setting that response code in the JSP is causing havoc.

So, setting error codes on error pages is breaking everything else? WTF were they doing?!

... Do they seriously have code that relies on a 404/500 page getting a 200 HTTP status? Or am I misunderstanding?

[bardofspoons421]

@vyznev said:

It sounds like you have the problem mostly solved already. Why not just give up on trying to get the other guy, who obviously doesn't understand either the problem or the obvious solution, to fix it and simply do it yourself?

If the problem is that you lack the authority to do that, or don't want to suffer the consequences of publicly embarrassing him, consider wrapping the code you have here in a simply class, testing that it works and sending it to him.  Tell him that you already had it written anyway, and if he'll just review and commit it you can both stop wasting your time on it.

Because they code by Cargo Cult, it seems. I've already sent that snippet along. As I've mentioned, I'm working with these guys as a result of a merger. We were a .Net/MSSQL shop, them Java/Oracle. Only one of our developers has access to the code base, and I go badger her whenever I want to look at something they did. That's how I discovered the SQL injection possibility, and that I found they were generating a UUID by calling out to Oracle.

Their "Lead Dev" comes from a VB6 background, and then moved to Java. Being a former VB6 programmer (But also picking up C along the way), I know. Source control is a recent addition to them. Large amounts of copy and paste throughout their code base, without any thoughts to reuse or refactoring. They've half-assed-ly re-invented Hibernate, except with more bugs, and more copy and paste code (All of their data objects have the word "Bean" in the class name. Someone read a book on EJB, it seems, and decided they liked the term)

They won't take my code because I'm not one of them, it seems, it would have to be "Tested' and have to conform to whatever specs they have (ha!). It's not like their stuff is tested anyhow. The Perl script I wrote probably increased their tests and code coverage by roughly infinity.

[bardofspoons421]

@blakeyrat said:

... Do they seriously have code that relies on a 404/500 page getting a 200 HTTP status? Or am I misunderstanding?

I seriously hope not. My best guess is they just did it wrong. My own boss emailed me shortly after I received this message (He was cc'd on the huge chain to be kept in the loop) asking "Is he serious?"

I don't know how JSP works, but it seemed to me, from googling, that you just add a code block to the page setting the response status.

[Daniel15]

But, when developing this software, I noticed that their web server is returning 200 OK for error pages, instead of 404 or 500 or whatever.

ASP.NET before .NET Framework 3.5 SP1 is worse - When an error occurs and you have a custom error page set, the default setup does a 302 redirect to the custom error page, which then sends a 200 OK. This means you can't refresh to try again (as it's redirected to a static page). .NET Framework 3.5 SP1 finally added the ability to load the custom error page without doing a redirect

[bjolling]

@HighlyPaidContractor said:

@bardofspoons42 said:

Add to this things like them building XML by string concatenation (I had an issue with an ampersand being in the data and not escaped). This project is going swimmingly.

A company I worked for had a rule where all XML had to be generated by concatenation "because it's faster than object creation."  This also applied to any dynamic html or SQL.

So you can't use ASP.NET? Back to classic ASP?

[Xyro]

@Daniel15 said:

When an error occurs and you have a custom error page set, the default setup does a 302 redirect to the custom error page, which then sends a 200 OK. This means you can't refresh to try again (as it's redirected to a static page).

Hey, that's what Community Server does!  :D

bardofspoons, what would happen if you started getting cocky and superior with them?  Not that I recommend that sort of relation, but it's quite possible that intellectually smacking them around might be a better form of communication for them.  If there's one thing I've learned since working at my current job, it's that I hate Oracle.  If there's another thing I've learned, it's that with some people you just can't be polite.  Sad but true.

[Guest]

 Sounds like your side of the merger needs to stage a revolt.

[b_redeker]

@Xyro said:

If there's another thing I've learned, it's that with some people you just can't be polite.  Sad but true.

Sometimes you have to be firm. But something tells me not much will change until management on the other side changes. This kind of incompetence can only propagate if management has absolutely no clue technically.

[HighlyPaidContractor]

@b-redeker said:

This kind of incompetence can only propagate if management has absolutely no clue technically.

There's an alternative?

[bardofspoons421]

I've been rather demanding through this whole process, and seldom backing down, unless a good point is made. Granted, most of my demands stem from them programming on hulking Sun servers, and me on a 600MHz ARM processor.

First, I wanted to use SOAP, because this seemed like something it was designed to do, communicate well between two dissimilar systems. They seem to have never done anything with SOAP before, and at first wanted to send me pipe-delimited data inside of a SOAP message, I managed to convince them otherwise, stating I want something well defined. That worked to get us going, outside of their stupid class names (Everything ending with Bean). I was using the WCF stuff on Compact Framework to generate the code from their WSDL, since I assumed the WCF would suck less and be more efficient than the older Web Service bindings. My only concern at first was raw data set size, and after running some tests, I discovered they didn't have gzip set up on their web servers, and convinced them to turn that on (Which probably gave them a boost on other things... they enjoy including javascript files several times in one page, and un-minified javascript too)

Then I made wonderful discovery: There was no way to send the "Accept-Encoding: gzip" header through the WCF stuff (Keep in mind, I'm on Compact Framework, and not the full framework). When tossing around a 13mb dataset from that would be roughly the size of the largest site they eventually have. This is rather important, since the data set zipped down to about 110kb. After some googling, I discovered that it was rather straightforward to add the header with the legacy web service stuff. Switched that out (I thankfully had the foresight to have the service interface defined as an interface, and having the UI coder [Another WTF, don't get me started] code to that interface) for web service... and it took double the time, even with the transfer taking less.

Ok, scrap that. I banged out an XML schema that roughly matched what they were giving me (Pipe delimited data was again suggested, I veto'd again, throwing up several cases where it would be much more work, and convincing them t hat with gzip, the angle bracket tax is negligible), sent that off to them, and got to work making straight HTTP calls POST'ing XML back and forth. This is about when I discovered they're returning 200 OK for everything.

I have no clue why I went off on this project history tangent, I guess I just needed to rant.

Anyway, the closest thing our esteemed compatriots have to a project manager is the CIO. He seems to have everything in his head, and is retiring in March. Our side of things predicts mass chaos when that happens, as, I stated before, their lead dev is essentially a fire-fighter, his title even conveys that, being something like "Tactical Project Manager"

As for revolt, our team is already subversive enough. We heckle upper management for sending out 2 sentences in a word doc containing the company approved template (Our "branding guide" demanded exact-to-the-mm spacing in an email signature) to all employees through yet another person, instead of, I don't know, actually sending it in an email body. When we first got access to their code base (Almost 2 years after the merger) the next few weeks was spent with a chain of WTFs. This is when we discovered their bean-anti-pattern, massive amounts of copy and paste (All of the files concerning the web calls for my service have a massive chunk of commented out code at the top from another service, I have no clue why), and general cargo-cult-ness. We may not have the best development methodologies on our side (Cowboy coding, NIH, etc), but we at least have been using source control for years (I think they started using SVN sometime earlier this year), have a build process, and an actual testing environment (Though no unit or integration tests). Thankfully our manager knows what a mess things are, and is trying to convince the CIO (and I'm sure he will the new CIO too) of some of the general mess they have on the other side.

[HighlyPaidContractor]

Just because you reminded me of it...

I was working on a project that recieved a daily feed from the client.  All our other clients were perfectly capable of sending well-formed fixed-width or pipe-delimited files.  One client however helped us develop a new industry standard. 

Their initial attempt at a pipe-delimited file was fairly successful - until it became apparent that their data regularly contained pipes.  Their next few attempts were delimited by absurd never-used ascii characters (like upside-down smileyface).  These attempts failed for similar reasons (yes, upside-down smiley faces regularly occured in their data).  This prompted the move toward multi-character delimiting (\@!vh|) - which also frequently failed.  Any attempt at them moving to fixed-width format, escaping pipes in their data, or any other sane method fell on deaf ears.

At this point, it became apparent that this particular client wasn't worth keeping on, so (with the CTO's blessing) we developed a new standard for delimited files.  The only file type we were willing to accept from them would be delimited with the multi-character string "douchebag" (case insensitive).  We stood firm.

[Guest]

 @bardofspoons42 said:

@blakeyrat said:

... Do they seriously have code that *relies* on a 404/500 page getting a 200 HTTP status? Or am I misunderstanding?

I seriously hope not. My best guess is they just did it wrong. My own boss emailed me shortly after I received this message (He was cc'd on the huge chain to be kept in the loop) asking "Is he serious?"

I don't know how JSP works, but it seemed to me, from googling, that you just add a code block to the page setting the response status.

The proper way is to let Tomcat handle the error page. In your web.xml file you map exceptions to error pages. This particular wheel has already been invented.

[bardofspoons421]

@HighlyPaidContractor said:

The only file type we were willing to accept from them would be delimited with the multi-character string "douchebag" (case insensitive).

That is awesome. One time when working a combined admin/helpdesk/dev position (I was one of two IT people), I got a call from the same user a 3 times in a week because she forgot her password every time. I then reset it to id10t, told her this over the phone. The guy I was working with cracked up the whole time

@Rick said:

The proper way is to let Tomcat handle the error page. In your web.xml file you map exceptions to error pages. This particular wheel has already been invented.

They do something like that in their Struts config:

<global-forwards>
 <forward name="systemerrorpage"
          path="/common/systemerror.jsp" module="" />
 <forward name="genericerrorpage"
          path="/common/genericerror.jsp" module="" />
 <forward name="nopermissions"
          path="/common/nopermissions.jsp" module=""/>
</global-forwards>

That's probably wrong, however. I have no clue how this is invoked. Being completely wrong in doing this would explain their Apache "bug".

[MiffTheFox]

@blakeyrat said:

@bardofspoons42 said:

So, setting error codes on error pages is breaking everything else? WTF were they doing?!

... Do they seriously have code that relies on a 404/500 page getting a 200 HTTP status? Or am I misunderstanding?

I don't know about Java, but in .NET properly handling an HTTP error code would require... handling an exception! (gasp!)

[Guest]

@bardofspoons42 said:

Then I made wonderful discovery: There was no way to send the "Accept-Encoding: gzip" header through the WCF stuff (Keep in mind, I'm on Compact Framework, and not the full framework). When tossing around a 13mb dataset from that would be roughly the size of the largest site they eventually have. This is rather important, since the data set zipped down to about 110kb. After some googling, I discovered that it was rather straightforward to add the header with the legacy web service stuff. Switched that out (I thankfully had the foresight to have the service interface defined as an interface, and having the UI coder [Another WTF, don't get me started] code to that interface) for web service... and it took double the time, even with the transfer taking less.

 

 Have you tried this?

[url]http://blogs.msdn.com/b/maagniho/archive/2008/12/24/gzip-message-encoding-for-wcf-applications-written-using-net-compact-framework.aspx[/url]

[bjolling]

@bardofspoons42 said:

... "Accept-Encoding: gzip" header through the WCF stuff... and it took double the time, even with the transfer taking less.

Ok, scrap that. I banged out an XML schema that roughly matched what they were giving me (Pipe delimited data was again suggested, I veto'd again, throwing up several cases where it would be much more work, and convincing them t hat with gzip, the angle bracket tax is negligible), sent that off to them, and got to work making straight HTTP calls POST'ing XML back and forth. This is about when I discovered they're returning 200 OK for everything.

Via WCF the time probably just got doubled because your device is slow at decompressing the 13mb dataset. So switching to HTTP + gzip would not gain much because your device is still slow at decompressing 13mb of data... Or am I overlooking something?

[Xyro]

If something is slow at decompressing gzip, then it doesn't have a hope of a chance at parsing 13 megs of XML.

[bardofspoons421]

@bjolling said:

Via WCF the time probably just got doubled because your device is slow at decompressing the 13mb dataset. So switching to HTTP + gzip would not gain much because your device is still slow at decompressing 13mb of data... Or am I overlooking something?

WCF wasn't allowing me to send the Accept-Encoding header, so I switched to the older Web Service stuff, for which I could send it. It was the legacy Web Service stuff that was much slower. I did a test with that with and without gzip, and there wasn't much difference, it was all spent decoding the SOAP message, which for some reason the WCF api was much faster than the WS api

And while doing this, I learned what the SOAP overhead was (In addition to their overly verbose element names). When switching from SOAP to a simple XML schema, there was almost an order of magnitude of shrinkage ditching the SOAP envelope. Though, with gzip, this amounted to something like a 10k difference over the wire.

[bjolling]

@bardofspoons42 said:

@bjolling said:

Via WCF the time probably just got doubled because your device is slow at decompressing the 13mb dataset. So switching to HTTP + gzip would not gain much because your device is still slow at decompressing 13mb of data... Or am I overlooking something?

WCF wasn't allowing me to send the Accept-Encoding header, so I switched to the older Web Service stuff, for which I could send it. It was the legacy Web Service stuff that was much slower. I did a test with that with and without gzip, and there wasn't much difference, it was all spent decoding the SOAP message, which for some reason the WCF api was much faster than the WS api

And while doing this, I learned what the SOAP overhead was (In addition to their overly verbose element names). When switching from SOAP to a simple XML schema, there was almost an order of magnitude of shrinkage ditching the SOAP envelope. Though, with gzip, this amounted to something like a 10k difference over the wire.

Thanks for sharing your insight. I was wondering if you also tried the gzip plugin for compact WCF as suggested by ShatteredArm in a previous post? I'm just curious... But I suppose you have a working solution now and you are not considering going back to WCF, right?

[danixdefcon5]

@bardofspoons42 said:

Add to this things like them building XML by string concatenation (I had an issue with an ampersand being in the data and not escaped). This project is going swimmingly.

Someone did this at one place I was working at. He did it because he didn't know how to consume a web service in Java. I'll leave it to you to figure out how he interpreted the XML response from that particular web service.

[immibis]

@bardofspoons42 said:

Add to this things like them building XML by string concatenation (I had an issue with an ampersand being in the data and not escaped). This project is going swimmingly.

 

Swimmingly?

[b_redeker]

@immibis said:

Signature guy just gets old, ya'know?

Yes. Can we please lose the "fake post as signatures" please? Thank you.