Intentional subversion of network security
-
Well I need a catchy title to get some answers, right?
Anyway, my employer has been using for years a MitM proxy (
) for all internet traffic, namely ZScaler. Recently (1-2 weeks ago) they upgraded it to a Windows App, and this morning I discover that Opera can't connect anymore to any website.All other browsers installed on my computer (Firefox, Chrome, IE, Edge) work fine, and a couple of quick reboots/updates doesn't fix it, so I assume IT is very subtly telling me to bend over and take it. I haven't found any documentation of the change anywhere on the intranet, but of course why would they bother telling users that they're fucking with them? (also and more realistically, 99.9% of users-who-are-not-IT-admins-and-therefore-can-disable-the-proxy use one of the 4 browsers mentioned above, but please don't interrupt my rant, thank you very much)
I'm a bit surprised though because nowadays Opera is AFAIK basically just a skin over Chromium so I would have thought that if Chrome is allowed, Opera would, but apparently not. Changing Opera's UA string doesn't seem to work either, so at that point I'm stumped. I can disable the ZScaler app, but then of course nothing at all goes through either (which is how it has been with the proxy for quite some time, and fairly obviously normal).
So, any suggestion? Any idea how ZScaler is sniffing my browser (apart from the UA) and how I could hide it? Stuff like PowerShell can get through as well, so it seems that it's really specifically Opera that is blocked, rather than everything except a few listed apps, but that doesn't help me much.
Of course I could switch to IE (that one is to trigger
on readers of this post 
) but I'd rather not have to change my habits just because Mordac.
-
My initial assumption would be that Opera isn’t using your computer’s native “trusted root CA” store and as such you need to install the root cert of the intercepting certificate in the appropriate place for Opera to use it.
-
@izzion The list of certificates displayed in Opera is indeed slightly different from the list that I can see in the Windows settings (lots of overlap, but a few are missing in Opera). Also the simple fact that Opera has a "certificates" dialog that doesn't seem to be the Windows one is a hint that it's probably maintaining a different list (although with W10 I can't be sure that the dialog I'm seeing is not just another flavor of the Windows settings, of course).
I've seen in particular one certificate in the name of my employer that is missing in Opera, however exporting/importing it works (i.e. I get a "success!" message), but it's still not listed in Opera afterwards, which seems weird. The import window is the standard Windows one (i.e. the same used by the export function), so it seems like importing from the Opera window is actually using standard Windows import, which means that maybe it's actually not a different certificate list but just the standard Windows one, somehow reformatted a bit?
Basically I don't know anything about certificates and how they're managed, so I have no idea how to go about it.
A couple of quick searches show that apparently Opera can be more picky than other browsers on the certificates it accepts (see e.g. this), so possibly it's just that my employer-provided certificates are borked and Opera refuses them?
-
More about the certificates: after much fiddling, there seems to be only one certificate that is not in Opera and that is in Windows, and that is an employer-issued one. It's not a recent one so I'm not sure why it would have suddenly caused the issue right now, except maybe if ZScaler didn't start checking it until today, so... maybe?
Now the certificate is of "Certificate Template" "SubCA" and it is also listed in the "Intermediate Certification Authorities" (and in Opera, I can indeed see it in the "intermediate" list). But in Windows it's listed both in the "Intermediate" and in the "Trusted Root Certification Authorities", but not in Opera. Exporting/importing doesn't help, as mentioned above, it's still never listed in the "root" authorities. I tried different formats, one of them showed me a scary warning about accepting an unverified authority which I hoped meant that it was going to work, but no such luck.
So if that is the issue, I would say that it's something in the certificate that prevents it from being properly recognised by Opera as a "root" authority. Whether it's intended or an oversight by IT, no idea (and if the later, it won't help me much as I can't really tell them to re-issue it!)...
Also is there something I can look at e.g. in the developer's tools to see if certificates are indeed the source of the issue?
-
@remi I would tell your employer to disable the MitM proxy or go take a long walk off a short pier, because fuck that shit and fuck any employer using one. That would solve all your problems* with it quite nicely methinks.
*Although depending on the option chosen, other problems may arise instead.
-
If this was indeed a certificate issue, then I would expect you to be seeing a certificate error. Though I'm not an opera expert.
You probably want to look at the network tab in the developer tools and see what the responses look like.
-
@Atazhaia Yeah, that's also what I would like to be able to tell them. But since it's been this way for quite a few years now, and I value the rest of my job more than them snooping on my traffic, there is little I'm willing to do along that avenue.
-
@PleegWat said in Intentional subversion of network security:
If this was indeed a certificate issue, then I would expect you to be seeing a certificate error. Though I'm not an opera expert.
The error is a long wait followed by "www.google.com took too long to respond." Nothing more.
You probably want to look at the network tab in the developer tools and see what the responses look like.
Thanks. I already tried that and I see some requests going out to google.com, they all fail (with "failed to load response data" in the response tab), and... that's all. I'm no web developer so maybe there is some information hidden in there, but I don't see anything.
-
This suggests that Opera 60+ should be using the Windows Certificate store... so my first question is going to be are you Lorneing your Opera version?
Opera Not Accepting Certificate (solved)
What I mean by 101 shit, is trusting the ca... And what a CA even is.. To be honest anyone understands what a CA is and even saw that the cert manager allows...
Other search results are suggesting that Opera will only accept certificates if they have a proper SAN name (DNS-based name of the site that's being intercepted, rather than an IP). Are you in fact getting an Invalid Certificate warning on the pages, or are you getting just a hard fail without a prompt of "hey this certificate is weird do you want to continue?" It looks like this page has some more information about how to verify what certificate Opera is using toward the bottom.
-
@izzion said in Intentional subversion of network security:
This suggests that Opera 60+ should be using the Windows Certificate store... so my first question is going to be are you Lorneing your Opera version?
Nope, I'm on Opera (Beta) 67. Not quite the very last one, but only the last (fourth) minor version number is different from the most recent. I never had to manually install certs in Opera, or at least not a in veeeeery long time, so I suspect it's been using system-ones for quite some time already.
I do get the same dialogs as shown in your linked article, which apparently then are the standard Windows ones... except of course they don't look like what I get when I go to certificates from Windows settings (either the new-look W10 settings or the old-style "control panel" app). What I get then is
certmgrorcertlmwhich look like this:
Other search results are suggesting that Opera will only accept certificates if they have a proper SAN name (DNS-based name of the site that's being intercepted, rather than an IP). Are you in fact getting an Invalid Certificate warning on the pages, or are you getting just a hard fail without a prompt of "hey this certificate is weird do you want to continue?"
Hard fail, no info except "took too long to respond". I've tried looking at the details of the missing certificate and I don't see anything different from, say, another employer-issued certificate that I see listed in both windows (Opera and certmgr). But maybe I'm missing one missing (!) field though. I've also tried looking at the certificate chain shown through the "green padlock" on Chrome and it's going through one of the certs that I can see in Opera, not through the missing one, so maybe the missing cert isn't the issue at all.
It looks like this page has some more information about how to verify what certificate Opera is using toward the bottom.
Meh. I don't get the "green padlock" thingy on Opera since no page loads at all, so of course I can't click it, and the rest doesn't help me much, it just tells me what I know ("to manage certificates click on certificates" well duh).
-
@remi
Hrm. We've exhausted my knowledge/expectations of things that should go wrong with an HTTPS MITM sniffer but only effect one browser. Sorry
-
@izzion That's OK, thanks anyway. At least I got something else than just
...Also that problem was enough to win over
and I've finally done what I wanted to do for a long time, i.e. try Vivaldi. Seems to work nicely with all the little things that I was used to in Opera, so I guess I'll switch to it.
-
Honestly, the problem here likely really is ZScaler. WTPharm uses that as their proxy solution, and it sucks major balls. We've had to uninstall/reinstall the app more times than I can count for people whose O365 won't authenticate (and indeed can't communicate with the activation servers), and other similar issues.
-
@remi Which certificate do you get for TDWTF? Maybe zscaler isn't doing MITM at all, but they're intercepting in a different way (extract master secret from the SSL library?). Normal MITM doesn't require any software on your local machine after all.
-
@PleegWat said in Intentional subversion of network security:
@remi Which certificate do you get for TDWTF? Maybe zscaler isn't doing MITM at all, but they're intercepting in a different way (extract master secret from the SSL library?). Normal MITM doesn't require any software on your local machine after all.
ZScaler has two main methods of operation: Standard proxy setup (where a PAC file denotes where the traffic is routed to be snooped) and this "ZApp" variant, where the local app handles more "intelligent" IP routing to the ZScaler servers based on location/ping/phase of the moon and can handle things like "turn itself off if you're connected to a trusted VPN connection" or other such fanciness.
The app also permits you to use other products, like their "ZScaler Private Access", which is supposed to be a pseudo-VPN-like solution that doesn't require a persistent VPN connection, but instead uses some form of authentication and permissions lists to allow you to access applications that would normally require VPN access via...magic? Mini-tunnels? I didn't understand the whole business about it when WTPharm's people were talking about it, but it allows me to use AD while not on their VPN somehow.
-
@PleegWat said in Intentional subversion of network security:
@remi Which certificate do you get for TDWTF? Maybe zscaler isn't doing MITM at all, but they're intercepting in a different way (extract master secret from the SSL library?). Normal MITM doesn't require any software on your local machine after all.
On Vivaldi (but I assume it's the same everywhere), TDWTF has a "certification path" of
Foo-Root-CA>Foo-Intermediate-Ext-CA1>Foo-SSL>what.thedailywtf.com(whereFoois my employer name, and other similar anonymising). I've left the second one relatively unchanged, because I feel that the "intermediate" bit is relevant, as well as the "ext" (I also have various "int" certificates installed, and guess what, at least one of them is used whenever I'm on the intranet...).On Opera,
Foo-Root-CAis listed, andFoo-Intermediate-Ext-CA1is listed, but only in the "intermediate" certificates (well it's in the name, isn't it?) while incertmgrthat one is listed both in "intermediate" and in "trusted root". Go figure.Foo-SSLisn't listed anywhere so I assume it's generated on the fly as needed?@e4tmyl33t said in Intentional subversion of network security:
ZScaler has two main methods of operation: Standard proxy setup (where a PAC file denotes where the traffic is routed to be snooped) and this "ZApp" variant, where the local app handles more "intelligent" IP routing to the ZScaler servers based on location/ping/phase of the moon and can handle things like "turn itself off if you're connected to a trusted VPN connection" or other such fanciness.
Yeah, as I said in OP, the issue only arose after they switched to the app a couple of days ago (we were using the PAC thing until now). I have had the app for a couple of weeks, but from messages on mailing lists it looks like the deployment was done progressively world-wide, so my guess is that last weekend the deployment was finished and they switched on whatever black magic exists in ZScaler that now causes Opera to no longer go through.
I've hated ZScaler from the day they started using it, for both the basic idea of it (MitM) and the implementation (at one point they were using some Flash-based stuff for browser that caused browsers without Flash to fail, while CLI programs would still go through provided you gave it the right proxy credentials...).
I still have a tiny shred of hope that the issue isn't that they're intentionally blocking Opera as such (if they were, working around the block would be a clear violation of IT policy, which isn't necessarily a wise idea -- but if they were since Opera is niche, I assume they would also have blocked a raft of other niche browser, such as Vivaldi?) and that instead it's just that Opera needs a bit of tweaking to align to whatever ZScaler wants. If not, it looks like I'm fucked.
-
@PleegWat said in Intentional subversion of network security:
If this was indeed a certificate issue, then I would expect you to be seeing a
certificateWebsite doesn't exist error. Though I'm not an opera expert.You probably want to look at the network tab in the developer tools and see what the responses look like.
In recent memory, Windows just pretends the other end just kinda never really worked.
I wonder if non-https traffic works?
-
@Tsaukpaetra said in Intentional subversion of network security:
I wonder if non-https traffic works?
That's a good question. It's a bit hard (for me) to check as most well-known sites use https nowadays. A bit of searching led me to this one which is made to compare http vs. https (no idea how meaningful the comparison here, but I don't care for the purpose of the test) and as far as I can see is itself http, not https.
And the verdict is... it works! I can get the page to load in Opera (although it took a bit of time, longer than in Vivaldi -- something like 20s to load the page vs. <5s in Vivaldi), and the http test works. And then clicking on the https test... fails, as with other pages (i.e. "took too long to respond").
So it seems that the issue is indeed related to https, which would confirm that it's a certificate issue, and also that Opera per se is probably not blocked (it's more likely something that Opera doesn't do properly -- or something that Opera does slightly differently from other browsers and that's not accepted by ZScaler).
It doesn't tell me how to fix it though. Since it seems that Opera is using standard Windows certificates, I'm not sure what else I can do about it...
-
I could imagine it being a matter of supported cipher suites - like Opera insisting on ephemeral while zscaler refuses to deal with anything more expensive than RSA key exchange. I could also see it being related to certificate pinning; I don't get the impression that's that widely deployed yet but one doesn't generally check.
-
@PleegWat ...
...
I'm sorry, were you talking about cricket?
I very vaguely follow what you're saying, but it doesn't give me any hint as to what I could do, if anything... thanks for the comment though, it might help shed some light on other stuff I might read elsewhere!
-
@remi Sorry, my knowledge of SSL/TLS internals might be classified as 'higher than average'. Basically, I'm saying Opera may be insisting on higher-quality encryption than zscaler is prepared to bother with.
-
@PleegWat That's more or less what I guessed. I tried randomly toggling some of the opera://flags that mentioned security or certificates, but that doesn't seem to be enough. Might be hard-coded stuff that I can't control.
It seems I'm out of luck then, except that there is some tiny hope that in the future the security settings of Opera will become the norm and therefore will be properly handled by ZScaler.
Also, a weird thing I noticed is that for some reason Github is working. The certificate chain does not go through my employer's certificate, which probably explains why, although I have no idea why that specific site is allowed to get through ZScaler without requiring a specific certificate???? Maybe this was needed by some dev somewhere in corporate IT and therefore an exception was added for it in ZScaler? (that would tie in with what @e4tmyl33t said about ZScaler requiring fiddling for almost everything)
-
@remi I started having a similar problem a few days ago. Ironically it was Vivaldi that wasn't working. My company uses a different MitM product but the symptoms were exactly the same: Vivaldi failed with ERR_TIMED_OUT while Chrome and Firefox worked fine on the same site.
Turns out Vivaldi was reaching out to pki.mycompany.com, presumably to check the SSL certificate revocation list or something, but a misconfigured firewall was blocking it. Apparently Chrome either doesn't check for revoked certs or doesn't care if the request times out. Once IT fixed the firewall Vivaldi started working normally again.
So I suggest installing a local proxy that can log outgoing requests and see what other URLs Opera might be trying to hit.
-
@mizar-hook Eh, funny that... at least that also would confirm that all these issues are just unintentional side effects (yes, I know, different MitM and obviously different company so it could be a different policy, but still... as always, don't immediately ascribe to malice what can be explained by incompetence).
So I suggest installing a local proxy that can log outgoing requests and see what other URLs Opera might be trying to hit.
This is getting close to being overpowered by
, but in case I get enough motivation to do it, any suggestion for an easy to install and use (and uninstall afterwards!) proxy to try that?
-
@remi I used tinyproxy which was just a 'brew install' on my Mac. Very little configuration required and I ran it in non-daemon mode in a terminal window so I could just hit Ctrl-C when I was done.
-
@remi I tend to use Telerik Fiddler on Windows.
-
@robo2 I've heard of a Fiddler on the Roof, but never a Fiddler on the Windows...
-
@Mason_Wheeler YMMV. I've seen and read Fiddler on the Windows way more often than Fiddler on the Roof. So
-
@robo2 A friend of mine was in Fiddler on the Roof a few years ago. He said he could always tell who was familiar with the show and who wasn't by how impressed they were when he told them he was playing the title character.
-
@Mason_Wheeler Too
to find it, but I have a really strong feeling of deja vu.
-
@robo2 said in Intentional subversion of network security:
@remi I tend to use Telerik Fiddler on Windows.
I second the Fiddler suggestion. Although I've had proxy issues "magically" fix themselves just by virtue of Fiddler being open (even with HTTPS decryption off).
-
I tried Fiddler, but I think ZScaler is fucking with me in more than one way.
To use the proxy I obviously need to tell the browsers to connect to it. But Opera uses the standard Windows proxy setup, so I need to change that one to point to Fiddler. But every time I change anything in that setting window, it's reset to the corporate proxy settings as soon as I apply the changes.
My guess is that the oh-so-wonderful ZScaler app monitors that setting and forces it to whatever it wants it to be. To confirm this, I turned off ZScaler, and magic! now I can change the proxy settings, and Fiddler captures the traffic as expected. But of course now that ZScaler is off, Fiddler itself can't seem to connect to outside, so all requests are blocked. And as soon as I turn ZScaler back on, requests go through again, but the proxy settings are reset and Fiddler doesn't see anything anymore. OK, I'm not that dumb (narrator: yes he is). I can use either the old PAC script, or the ZScaler app, to find the IP of the proxy ZScaler uses, and set that into Fiddler once I've turned ZScaler off. And it works, I can now capture traffic, yay!
Except... on the first page I load I get the ZScaler login interstitial, which isn't really unexpected since I got that one every time I use a new browser (I guess it stores a cookie or whatever other session mechanism it wants to check if I've logged in already, fine). The dialog is all messed up, with no styling and stuff (just the raw HTML), which tells me something is not quite right, but I can type my login/password, so all is well. Except that validating that leads me to a "oops something happened" ZScaler page (properly styled, this time), with no more details ("contact your IT" well that's unlikely to help).
And once I got that page, I can now browse to any https page, and it works. It's logged by Fiddler, but the traffic goes through, the page loads normally.
So maybe I just needed to re-authenticate myself on ZScaler (in Opera) and now that it's done it will work again, so let's turn off Fiddler and turn on ZScaler again and... no luck. Same problem as before. Maybe because that previous authentication was for Fiddler, not for Opera. Or maybe Fiddler filters out some stuff from Opera, or uses itself a certificate that has the right properties that Opera expect, but that my employer's certificate doesn't. Or maybe I am that dumb and just randomly pushing buttons without any understanding of what it does.
Current conclusion: if I pipe all traffic through Fiddler (and turn off ZScaler and manually give the proxy IP to Fiddler), Opera works again. So I guess if I really wanted to, I could use that. Except it will break every other day, whenever ZScaler turns itself back on, or the proxy IP changes, or anything else.
Concerning the https traffic logged in Fiddler:
everything goes through normally, so there is no obvious error to look at. Peeking into a couple of packets doesn't bring me any insight.
-
20 days later (according to NodeBB's timestamp)...
I kept Opera on the side despite it not working, because I had tabs open to some intranet sites. And I just happened to click on another tab (not internal), and... it worked. Tried various sites, all work fine on Opera now.
So it seems that whatever broke Opera is now working again. Which means it might break again for no reason one day. Or Vivaldi might break for the same no reason whenever.
I'm blaming W10 updates. Not that I have any clue they're the cause, but it's always a safe guess when something goes wrong. No, that's not right. I'm blaming W10 updates and corporate IT. Not that I have any clue they're the cause, but it's always a safe guess when something goes wrong. Oh, wait. I'm blaming W10 updates, corporate IT and third-party MitM apps. Not that... well, you get it.
I love computers. sigh
-
@remi Blaming corporate IT is a very good start.
-
@remi said in Intentional subversion of network security:
I'm blaming W10 updates. Not that I have any clue they're the cause, but it's always a safe guess when something goes wrong.
While normally W10 updates are a good place to blame, if you still had Opera open it sounds like you didn't actually reboot to install any updates, so they're probably blameless here.
Corporate IT, on the other hand...
-
@Unperverted-Vixen Remember, blame is not a conserved quantity. Corporate IT can be 100% to blame as well as Windows 10 updates being 100% to blame.
-
@Benjamin-Hall Blame is a quantum vector, with acceptance of responsibility being the observation event?
-
@dkf nah. Blame is gpl, it infects everyone who touched the problem.
-
@Benjamin-Hall That's just quantum entanglement!
