The Web, Java, and LDAP, Oh My!



  • The Web, Java, and LDAP, Oh My!

    I had to set up a web application at work. It is open-source, written in Java, and it provides its own web server. This wasn't too bad, really. There were some minor issues but mostly things worked.

    Then we wanted to allow people to log into the web site using their existing username and password. This means using LDAP to talk to Active Directory. I'd never had to deal with either LDAP or AD before but I expected it wasn't going to be easy. There's all kinds of unusual terminology (Forest, Organizational Unit, etc.) and AD doesn't always behave quite exactly like other LDAP providers. So I was expecting a bit of a learning curve. But that wasn't even where I ran into most of the trouble, as it turns out.

    A bit of Googling finds that this particular program has an LDAP plugin that's also open-source. There's a wiki page describing all the settings you need. One to enable the use of the LDAP plugin for authentication. One to specify your "realm" (basically your domain name). There are a couple of oddball settings that you have to specify if you are using AD. These settings have values containing arcane query strings like "(&(objectClass=user)(sAMAccountName={0}))" or "(&(objectClass=group)(member={0}))". Now it would be nice if the application could just let you say something like "usingActiveDirectory = 1" and it would automatically use the right query strings. Or, even better, if it would detect an AD server and just do the right thing. But at least these things are documented somewhere and all I have to do is copy and paste them into my config and I should be good. (Right?)

    I restart the server process again to get the new settings to take effect and try to log in. Failure. Time to check the logs. At startup it shows that it successfully detected the multiple AD servers that are available and that it can connect. But when I actually went to log in the only message in the log is a NullPointerException from the LDAP plugin. This is not very helpful. The application catches and logs all exceptions thrown from plugins so that they don't bring down the entire server. That's good. But it only logs the message, not the stacktrace so I have no idea where to begin looking for the problem. That's bad. I try tweaking various settings at random but they don't seem to have any effect.

    Time to turn on debug logging. Maybe that will provide some extra information. At the bottom of the wiki page there's a note that says something like "To turn on debug logging in versions before X.Y, do this …". But version X.Y is pretty old and I'm using a newer version. There's no mention of how to turn on debug logging for newer versions. In fact there is no supported way to do this in newer versions. There's a ticket in their bug tracking system about this that's a year and a half old. (Later on I found out that the logging configuration is stored in an .xml file inside a .jar file. That's why you can't easily modify it. You have to extract it from the .jar, edit it, and then either store it back in the .jar or make a new .jar and put that new .jar ahead of the old one in the classpath. So it can be done but it isn't exactly convenient. Why this file is stored in a .jar instead of in the "conf" directory that they already have is beyond me.)

    I guess I'll run it under a debugger. That should let me figure out what's going wrong. But I only have the JRE installed on this system, not the JDK, so I don't have jdb. No matter, you can run a Java program on one system and run the debugger on another. All you need to do is specify something on the command line when you start the program in question. I don't remember what it is off the top of my head but some more Googling produces a StackOverflow with the answer:
    "java -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n". Now this is a command line that could only come from the world of Java. Incredibly verbose, full of '-X's and obscure names. What's "jdwp"? What's a "dt_socket"? Is that different from a regular socket? But there's no time for figuring all that out now. We're getting distracted from the task at hand.

    Like most Java programs, this one comes with a shell script to start it. So all I should have to do is find the line near the end that says something like "java -jar foo.jar …" and edit it to contain my -Xdebug switches. Except there is no such line. This shell script doesn't run java directly. Instead it runs a program with the fairly generic name of wrapper. This wrapper program then launches java and monitors it. Thankfully the wrapper program prints out the URL http://wrapper.tanukisoftware.org when you run it with "--help" so I can at least find some documentation on how it works. In that documentation it says that there are properties called "wrapper.java.additional.<n>" that you can set to to pass extra command line parameters to java. So all I have to do is make the command line read "wrapper ... wrapper.java.additional.1=-Xdebug wrapper.java.additional.2=-Xrunjdwp:server=y,transport=dt_socket,address=4000,suspend=n". I guess I was wrong when I thought the command line above was verbose. This one is the new verbosity champion.

    I start the server with this modified script and connect to it from another machine using jdb and set some breakpoints. Then I try to log in again. But my breakpoints don't get hit. Because I'm actually debugging the wrong JVM.

    This application launches two subprocesses when it starts up. One handles the web traffic. I'm not sure what the other one is for but I think it handles the database. So we have a tree that looks like this:
    wrapper
    _ java (main program)
    _ java (database)
    _ java (web server)

    Wrapper launches and monitors the main java program. And that java program launches and monitors two other java programs. I am debugging the main program but what I need to be debugging is the web server. Some digging around and I find that there's a property in the config file for specifying what command line arguments get used when main program launches the web server. Something like foo.web.javaAdditionalOptions. So I revert my changes above to the wrapper command line and put "-Xdebug …" in this property.

    I can put a breakpoint in the initialization method of the plugin and that gets hit so I know that I'm in the correct process now. I put some breakpoints in various methods from the plugin that look like they would be related to authenticating a user. None of them get hit. It is crashing before it even gets to any of those. Now at this point I should have set the debugger to break on throwing a NullPointerException but for some reason that did not occur to me at the time. Instead I stumbled around for a while and eventually got debug logging turned on. That pointed me at the right method in which to place a breakpoint and then I stepped from there to find the place where it was crashing.

    Of course by "place where it was crashing" I mean a filename and a line number. The debugger isn't showing me the source code for this location and I don't know how to convince it to do so. So I'm manually correlating that with the source code that I have open in a separate window. And hoping that I have the version of the source that corresponds to the compiled program because otherwise the line numbers won't match up.

    After all of this I'm sure you're dying to know what the problem was. The plugin keeps a list of all the available servers in a map. If you specify a single server in the config file and don't give it a name then it gets stored in the map with a key of "<default>". If you specify more than one server then you have to give each one a name and this name is used as the key in the map. If you don't specify any servers then the plugin uses SRV Records to automatically discover the servers. For the keys in the map the plugin uses "<default>1", "<default>2", "<default>3", etc. But when it goes to look up a server in this list it notices that you didn't specify any server names in the config file and incorrectly assumes that this means that there's a single server stored under "<default>". It ends up doing map.get("<default>"). This returns null since that key doesn't exist in the map. Then this null gets passed to a constructor and stored in a member that's never supposed to be null. Later on this member gets dereferenced and we crash. I'm not sure how this escaped notice by everyone except me. Perhaps other people have certain settings (like "automatically create new users in the database the first time they try to log in") turned off and this avoids the problem.

    Now that I knew what the problem was, I picked an AD server and specified the hostname manually in the config file, thus disabling the automatic server discovery. And of course everything started working immediately.

    Just kidding! Of course it didn't work. I got errors about how I had to "bind" which is LDAP-speak for "provide a username and password". Which is a bit odd considering that the user is providing a username and password. But you can't use those, apparently. You need a special account which exists just for binding and then you can use that account to search for real users and authenticate them. Once I had such an account created for me everything actually started working.

    And it only took me a whole day.



  • Lots of wtfs, yes... but requiring a username/password to do an LDAP lookup isn't one of them.

    You need security credentials to do AD lookups using .NET's System.DirectoryServices, too.



  • Remind me again why Java is a good idea?



  • Nah! It probably was a Linux hardware.

    How's Java fault that the plugin was badly documented and that the application used some weird startup wrapper? Actually, thanks to Java remote debugging was he able to figure out what should have been something obvious.

    Also, if the plugin was well written, that NPE is very easy to protect against.



  • @jnz said:

    And of course everything started working immediately.

    Just kidding! Of course it didn't work.


    Best part.



  • @lightsoff said:

    Remind me again why Java is a good idea?

    None of the problems in the OP are actually due to Java, per se.

    But every Java application has similar problems. So you have to wonder what it is about the Java ecosystem that makes coders write complete shit.



  • Well this one was:
    @jnz said:

    Later on I found out that the logging configuration is stored in an .xml file inside a .jar file.

    You'd never see a program written in a compiled language do that, unless the entire programming team was totally insane.

    To be fair, while it's something that could happen with any interpreted language with a packaging system, it seems to happen to Java more often than elsewhere.



  • You'd never see a .NET program do that, either, because it'd take a SHITLOAD more work than just using the default Settings class.



  • Absolutely.

    Oh hell. I just agreed with Blakeyrat... Does that mean the world is about to end and/or I should buy a lottery ticket?



  • @lightsoff said:

    To be fair, while it's something that could happen with any interpreted language with a packaging system, it seems to happen to Java more often than elsewhere.

    It's because it's easier to do something like that than to fight the application's classloader.



  • Sounds like trwtf is having the class loader be involved in loading a configuration file to begin with...


  • :belt_onion:

    Problem is that Java has 3 or more "out of the box" configuration methods, none of which are really easy.

    Let's see...

    1. We have system properties (a form of named command-line arguments) which can be queried from anywhere within the JVM.
    2. There is the Properties class which can load either a plain and stupid ini-like format (no sections or placeholders) or some verbose XML format (hurray for property = "name" tags, see here - only introduced in Java 5 back in 2003). Because it has no sections, it was likely impossible to give it a standard path or location and yet make it fit in any random application, so you either need to give it a filepath you just invented or need to do class loader magic to scan for some arbitrarily named file in one of the registered directories. Since JAR files are immediately on the classpath, people sometimes stuff configs in a JAR as it will then always exist on the classpath.
    3. Some Java version introduced a "configuration database". I don't know the particulars about it, but it does have a few drawbacks:
      * I believe there's no easy way to deploy a default configuration from an installer.
      * AFAIK it's global for your OS and managed by the JVM, so the end user might have no control over when or how his settings are used for different versions of your software.


  • @JBert said:

    Because it has no sections, it was likely impossible to give it a standard path or location and yet make it fit in any random application, so you either need to give it a filepath you just invented or need to do class loader magic to scan for some arbitrarily named file in one of the registered directories. Since JAR files are immediately on the classpath, people sometimes stuff configs in a JAR as it will then always exist on the classpath.

    Maybe this is rooted in a classic Java misunderstanding of what being cross platform means. Configuration handling is inherently platform specific, so it would be perfectly appropriate to look for this file in /etc/mymonstrosity, /Library/Application Support/com.wtf.monstrosity, whatever the appropriate place is on Windows, and so on depending on the platform it's deployed on.



  • I think this is why you get all sorts of frameworks with crazy class loaders that add an abstraction layer to the OS. That sort of thing works "well" enough with server stuff.


  • :belt_onion:

    Actually, that makes me remember that there's even a 4th way, originated in the Java Enterprise Edition spec: JNDI, which is a "registry" containing references to all types of objects which you can navigate through. Problem is that the spec only defines the query API. How or when you are meant to set up this registry was never written down by the committee, instead they deployed a SEP field and let the application server vendors come up with mutually incompatible ways of solving the problem.

    That's why you need to have JBoss or IBM-specific deployments, thanks to all the side effects or slightly minor differences in config files and packaging.



  • @calmh said:

    Maybe this is rooted in a classic Java misunderstanding of what being cross platform means. Configuration handling is inherently platform specific, so it would be perfectly appropriate to look for this file in /etc/mymonstrosity, /Library/Application Support/com.wtf.monstrosity, whatever the appropriate place is on Windows, and so on depending on the platform it's deployed on.

    That windows one made me look up if anyone had implemented a Java library for accessing the windows registry.
    I found this

    Here is a class that allows you to read the Windows Registry without having to install any JNI library. It is implemented purely using introspection and will therefore compile and run on any platform.

    :facepalm:



  • I'll bet "any platform" meant 32 or 64 bit Windows to that guy.


  • mod

    He promised it would compile and run, not that it wouldn't throw exceptions and crash when it realizes there is no registry :)



  • @boomzilla said:

    I'll bet "any platform" meant 32 or 64 bit Windows to that guy.

    1. "Run" != "work". So it could be technically correct.

    2. He might be trying to say, in a super-clumsy manner, that it mocks a Registry for systems that don't natively possess one. You have to correct for the fact that no software engineer can communicate worth crap.



  • I considered trying it on Linux to see what would happen but haven't been motivated enough...maybe later if today is slow.



  • @blakeyrat said:

    1) "Run" != "work". So it could be technically correct.

    I thought of that, but it doesn't make sense as an argument as you can catch pretty much everything in Java (including linking errors). It's very hard to write a program which will compile and then not run to some degree when you don't care about whether or not it works.

    @blakeyrat said:

    2) He might be trying to say, in a super-clumsy manner, that it mocks a Registry for systems that don't natively possess one. You have to correct for the fact that no software engineer can communicate worth crap.

    That's what the Preferences class does; what his code does is it takes a specific implementation (WindowsPreferences) from inside of that, then uses methods defined on that implementation. It does nothing but throw a BackingStoreException on Mac and Linux.



  • @boomzilla said:

    I considered trying it on Linux to see what would happen but haven't been motivated enough...maybe later if today is slow.

    It throws this:

    java.util.prefs.BackingStoreException: java.lang.NoSuchMethodException: java.util.prefs.FileSystemPreferences.WindowsRegOpenKey(int, [B, int)
    	at net.infotrek.util.WindowsRegistry.getKeySz(WindowsRegistry.java:82)
    


  • Java logging systems tend to use the Properties system. Or at least the more common ones do.

    java.util.logging, Log4J, and Logback all do anyway.



  • The classloader is used to locate and open files when they're located inside a JAR/WAR/EAR file as those can't be opened directly using File objects.


  • Discourse touched me in a no-no place

    @powerlord said:

    those can't be opened directly using File objects

    They can, but it's a much bigger PITA than letting the classloader do the work for you. The main problem is that you too easily end up with an all-or-nothing piece of customisation; making redistributable code is rather difficult, and logging is one of the things that often needs to be configured precisely at the point when it's awkward to do so. :frowning:



  • @jnz said:

    Then we wanted to allow people to log into the web site using their existing username and password.

    Completely sensible requirement.

    @jnz said:

    This means using LDAP to talk to Active Directory.

    This, however, is WTF. Authentication in Active Directory is handled by Kerberos and that is what one should be using for that purpose.

    Yes, most people end up using LDAP, because, despite how weird it is and how dense it's documentation is, Kerberos manages to do worse. That does not make it right.

    You may also still need access to the LDAP if you need some additional information about the user, but for authentication itself, Kerberos should be used. I believe it should be able to check group membership too, but I am not sure about that and don't really know what to search for to check; the documentation for this is rather dense.

    That said, I've tried to get Apache mod_kerb working a couple of times, but while it works for password authentication, I never managed to feed it correct keytab with it's own key to make it verify tickets for proper SSO. Which, I guess, is the reason everybody ends up using LDAP.



  • To be fair, none of these problems are Java specific. You could easily have the same mess if it was in C# or a dozen other languages.



  • @Bulb said:

    I believe it should be able to check group membership too

    It cannot. Groups are authorization, Kerberos does authentication. You need to follow up with an LDAP lookup to get group membership.



  • @Eldelshell said:

    How's Java fault that the plugin was badly documented and that the application used some weird startup wrapper? Actually, thanks to Java remote debugging was he able to figure out what should have been something obvious.

    Also, if the plugin was well written, that NPE is very easy to protect against.

    The fact that such a wrapper program exists and that people use it (and some apparently pay for it) implies that there's something deficient about the JRE as it stands. Programs written in other languages don't need a wrapper distributed with them. Even if you accept that a wrapper is required or desired here, note that the wrapper is not written in Java but in C. I guess Java isn't up to the task. Having a wrapper written in C has a nice bonus: if you want to be able to run on multiple platforms you need to distribute a wrapper binary for each of them. That's the promise of Java: Write Once, Run ... on a select set of preapproved platforms?

    The NPE is the fault of the plugin but the language really didn't help out the author of the plugin. When calling map.get(key), Java returns null if the key is not in the map. Sometimes this behavior is what you want. But this code implicitly assumed that the key was in the map and did not expect to get null back. Other languages have two different ways to access an element a map: one that throws when the element is not present and one that doesn't. In C++ you have std::map::at and std::map::find (or std::map::operator[]). In Python you have dict[key] and dict.get(key). Rust handles it differently with HashMap::get returning an Option which you cannot accidentally dereference at runtime.

    In Java you get map.get(key) and that seems to be all. (If I missed something here, please tell me.) If there had been a way to easily say "get the value for this key from the map or otherwise throw if the key doesn't exist" then perhaps the author would have used it. That would have changed the problem from a NPE in some far-away part of the code to an exception getting thrown from the line of code causing the problem. The error message might have been informative too, like 'KeyNotFoundException: The key "<default>" is not present in the map' or something. That's certainly more useful than a generic NPE since it gives you some clue as to what went wrong and something to grep for.

    Perhaps the reason that Java doesn't have a method for "get or throw" is that it would be a pain to use. With checked exceptions almost nobody is going to want to use that over the "get or return null" method. So they just left it out entirely.

    There's also the issue that you could call the constructor of an object and pass in a null reference for one of the parameters even though that constructor was not prepared for that parameter to be null. This may be somewhat mitigated if you use type annotations with @Nonnull but not many people do that and retrofitting an old codebase seems painful. I'm going to have to grudgingly give Java a pass here since tons of other languages have this same issue.



  • @blakeyrat said:

    So you have to wonder what it is about the Java ecosystem that makes coders write complete shit.

    It's not a problem with Java, it's a side effect of all the shitty schools teaching it exclusively.

    If the schools had taught C# instead, C#'s community would be the tumor.



  • @ben_lubar said:

    It's not a problem with Java, it's a side effect of all the shitty schools teaching it exclusively.

    I think that is a direct side-effect of the purpose with which Java was created and the attitude with which it was pushed.

    Java was intentionally dumbed down so it would be easy for code monkeys to learn and it was pushed as a simple language that schools should teach as the first language. It is so dumbed down that it makes some abstractions verbose, difficult or not very natural¹. So to help the students learn the techniques that are completely natural and not worth talking about in other languages², the architecture astronauts came and defined The Design Patterns complete with heaps of boilerplate and tons of useless crap. And that's how most Java code out there is either tangled mess (when written by people who didn't bother to learn Design Patterns) or buried beneath layers of useless wrappers (when written by people who did).


    ¹ Objects are a useful abstraction, but there are many problems that just don't fit it.

    @ben_lubar said:

    If the schools had taught C# instead, C#'s community would be the tumor.

    I don't think it would be as bad. C# is constantly adding new abstractions and sane guidelines for using them.

    ² All the factories, observers, visitors, commands etc. were used for millennia before Java, but they were so obvious nobody bothered to name them.

  • Discourse touched me in a no-no place

    @Bulb said:

    All the factories, observers, visitors, commands etc. were used for millennia before Java, but they were so obvious nobody bothered to name them.

    Sensible Java programmers just use a libraryframework like Spring to handle all that stuff. Though there is a bunch of architecture astronautics inside Spring, using it doesn't require doing much fancy stuff at all. The aim is to focus on just writing the code that actually does the work you care about, not the code that supports the application's configuration.

    There are a lot of non-sensible Java programmers about. :unamused:



  • Sensible programmers know when the patterns are worth and when they are not. It is, sadly, difficult to teach.



  • Could be.

    I left my uni when they were still teaching C++ with Borland Builder, but I think they switched to Java either the next year or year after.

    @Bulb said:

    Java was intentionally dumbed down

    I hate the phrased "dumbed down" SO MUCH.

    Could we get like a Rosie to show up in posts that use that phrase or something?

    EDIT: here you go -bz



  • @blakeyrat said:

    I hate the phrased "dumbed down"

    It it adequate to how I hate Java.



  • @Bulb said:

    It it adequate to how I hate Java.

    That is gibberish. You're certainly dumbed down your posts! Hah!!!



  • Ok but you missed Bulb's.



  • Not by accident.



  • This.

    I'm sure someone is going to comment about the high number of frameworks that Java has, though.

    Keeping in mind that Java on the desktop sucks, most of the time you run into Java, it's going to be a web backend.

    For web backends, there are two frameworks that work together that are in the clear majority: Spring MVC for the web part and Hibernate for the database part.

    I would hope by now that if you talked about making a desktop application with Java that someone would look at you as if you were crazy.


  • Discourse touched me in a no-no place

    @powerlord said:

    I would hope by now that if you talked about making a desktop application with Java that someone would look at you as if you were crazy.

    Hi! We're crazy here! :wtf:


  • mod

    @lightsoff said:

    Does that mean the world is about to end and/or I should buy a lottery ticket?

    Go for the lottery ticket. Then you don't really lose if the world ends. ;)



  • If you are not satisfied with open-source software because of (Java/LDAP/your ignorance) just don't use it. Go find another one. Or submit a patch. :smiley: ) Or at least a ticket/bug report.

    OK so you decided to eat the cactus. AD requiring bind auth is OK, the bind is the proof of identity itself. Unusual but I implemented it several times.

    Next is Java logging. Any logging IS hard, stacktraces from Python / Erlang / C# often also arent available.

    Next is wrapper. That's mostly for windows idiot-level users that will get confused looking at .bat files ("So where is my .exe?"). Every wrapper is just a call to java .dll + main class name. Yes, sometimes it's hard to find which main class / params to use (and reverse-engineering IS hard).

    Not sure if JRE/JDK/debugging was next. Was it? That's wrong idea, not being a Java developer you will not succeed in this. I would try this as a last measure.

    Good thing with Java OSS is you can compile it/fix it. I did it lot of times and do constantly. Alternative is "didnt work right now -- throw it away." Buy iPhone and never deal with Java if it hurts you.

    Peace :smile:



  • @vkovalchuk said:

    If you are not satisfied with open-source software because of (Java/LDAP/your ignorance) just don't use it.

    @vkovalchuk said:

    Good thing with Java OSS is you can compile it/fix it.

    So, nothing open source can ever be bad because you can fix it? The fact that he could choose to make it better does nothing to the argument that the software doesn't work when multiple SRV records are returned.



  • @vkovalchuk said:

    If you are not satisfied with open-source software because of (Java/LDAP/your ignorance) just don't use it. Go find another one. Or submit a patch. ) Or at least a ticket/bug report.

    Look, shitty software is shitty software regardless of the license attached to it.

    And if the user can't figure out the solution ("your ignorance"), that's the software's fault-- it wasn't released in a sufficiently usable state.



  • @vkovalchuk said:

    Good thing with Java OSS is you can compile it/fix it

    Unlike any other OSS, I guess?

    And we've been through this time and again - no, we don't want to fix your shit to use it.



  • @vkovalchuk said:

    Next is wrapper. That's mostly for windows idiot-level users that will get confused looking at .bat files ("So where is my .exe?").

    .bat files won't confuse idiot-level windows users. They never turned off 'hide file extensions'.



  • @vkovalchuk said:

    Or at least a ticket/bug report.

    Good idea! I'll just go to their GitHub page and ... oh, they have Issues disabled. Hmm, they use a JIRA instance to track bugs. That's fine. I'll just sign up for an account. Oh, but they have account creation disabled. Mere mortals are only allowed to look at the bug tracker, not actually file bugs. Instead you're supposed to post your bug to a mailing list. And then if somebody decides it is actually a bug and can be bothered they'll open an actual issue in JIRA for you. Assuming they don't just ignore you. Or forget about you.

    There are currently two active bugs in that bug tracker for this plugin. There's no significant activity on this plugin's GitHub repo for a year and half. I'm not feeling optimistic.



  • Hi, stopped reading after you asserted that a shell script was C.


  • sockdevs

    (post withdrawn by author, will be automatically deleted in 24 hours unless flagged)



  • @Gribnit said:

    Hi, stopped reading after you asserted that a shell script was C.

    Where did anyone say a shell script was C?


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.