What gives, Nginx?
-
So I made a few changes to my Nginx config, and now it's failing silently on the location I changed. It returns a 500, but it doesn't seem to be logging anything. Can I get more eyes on my config? Maybe one of you knows what I did wrong...
#don't worry about that variable, we run our confs through a preprocessor at build time #It's getting substituted before NGINX sees it resolver $DNS_RESOLVER valid=10s ipv6=off; server { listen 80 default_server; # _ is ALL server_name _; charset utf-8; server_tokens off; client_max_body_size 5m; gzip off; add_header X-WEBSERVER "$WEBSERVER"; add_header X-Robots-Tag "noindex, noarchive, nosnippet"; root /mnt; include conf.d/inc-v6.content; location /robots.txt { add_header Content-Type text/plain; return 200 "User-agent: *\nDisallow: /\n"; } [..snip several other locations that are fine...] location /v6/api/{ rewrite ^/v6/api/(.*)$ /$1 break; set $backend http://apiloadbalancer; proxy_pass $backend:3000; } location /v6/{ rewrite ^/v6/(.*)$ /$1 break; proxy_pass http://apploadbalancer:3000; } }
You can see I've edited the API location, but not the root location; I am attempting to apply a fix for the DNS never being re-resolved after Nginx starts, which is popular across the internet. It's only API calls that throw a 500. The proxy-pass destinations are not anonymized, that's a Service Link in Docker; I have no idea if that's related or not.
-
@yamikuronue First thing I would try would be
rewrite ^/v6/api/(.*)$ /api/$1 break;
-
@wharrgarbl I think what that's doing is, the application running int he API container doesn't know it's at
/api
-
@yamikuronue so, did the changed line I posted fix it?
-
@wharrgarbl No. It changed nothing. I don't understand why you even suggested it...?
I don't have any sign in the application's log that it's even being hit, by the by. If it was a rewrite issue I'd expect to see a 404 :(
-
@yamikuronue Because I think /v6/example and /v6/api/example were being directed to the same place at apploadbalancer:3000/example and I dunno what else could be happening
-
@wharrgarbl They're going to different places: api loadbalancer vs app loadbalancer
-
Normally, the Nginx error logs would have all the details. I'd double check for permissions etc to make sure Nginx is writing logs correctly.
There's nothing wrong with the config. It's probably the load balancers returning something Nginx doesn't like. I'd try and see what
curl --verbose http://apploadbalancer:3000
returns within the Nginx container
-
@cark Could this happen because the backend sent a 401? Like for some reason that's now a 500 instead of returning a 401 to the frontend?
-
@yamikuronue Actually, there seems to be something wrong with the Nginx config. Accessing /v6/api/ with your config gives me this in the log
2017/06/27 23:58:49 [error] 8034#8034: *1 invalid URL prefix in ":3000", client: 127.0.0.1, server: plonk, request: "GET /v6/api/ HTTP/1.1", host: "plonk"
Looks like Nginx isn't setting $backend as you expect. If I take out that variable and directly write
proxy_pass http://apiloadbalancer:3000
then it works
-
@cark said in What gives, Nginx?:
If I take out that variable and directly write
yeah, taht's what I had before. But the variable is the fix I'm meant to be implementing.
-
Oh, that's right, I found the error again: "Invalid URL prefix: :3000"
Meaning the variable isn't resolving for whatever reason.
-
This post is deleted!
-
@powerlord yeah, it's from
ngx_http_rewrite_module
and we're using rewrite rules. :( good thought though
-
@yamikuronue Yeah, I glanced at the alphabetical list of directives to find out which module it was in after I said that. Hence why that post got baleeted before you actually responded to it.
-
@powerlord Googling a bit shows a lot of people complaining about
proxy_pass
not working correctly with variables, with people saying "doing [thing] fixed it for me!" followed immediately with "[thing] didn't fix it for me..."You'd think this would be solved considering at least one of the threads I found talking about it is 6 years old.
-
@yamikuronue said in What gives, Nginx?:
we run our confs through a preprocessor at build time
Have you looked at the build output to see if it's generating something weird?
-
@ben_lubar Yeah, that's fine. We fixed some issues with that before we hit this problem :)
-
@yamikuronue well, if you're getting "Invalid URL prefix: :3000", that seems to indicate that your URL is :3000/foo/bar instead of http://someserver:3000/foo/bar
-
@ben_lubar Yeah. But I don't know why.
-
@yamikuronue my un-researched guess would be that the preprocessor is replacing
set $backend http://apiloadbalancer; proxy_pass $backend:3000;
with
set http://apiloadbalancer; proxy_pass :3000;
-
@ben_lubar the preprocessor only takes uppercase. I was sure that was it for a minute though.
We also tried moving the resolver definition since I pasted into this thread, so our current post-processing block reads:
location /v6/api/{ resolver 169.254.169.250 valid=10s ipv6=off; rewrite ^/v6/api/(.*)$ /$1 break; set $backend http://apiloadbalancer; proxy_pass $backend:3000; }
-
@yamikuronue said in What gives, Nginx?:
@ben_lubar the preprocessor only takes uppercase. I was sure that was it for a minute though.
We also tried moving the resolver definition since I pasted into this thread, so our current post-processing block reads:
location /v6/api/{ resolver 169.254.169.250 valid=10s ipv6=off; rewrite ^/v6/api/(.*)$ /$1 break; set $backend http://apiloadbalancer; proxy_pass $backend:3000; }
why not
location /v6/api/ { proxy_pass http://apiloadbalancer:3000/; }
If the
proxy_pass
destination has a path, the path in theproxy_pass
definition replaces the one in thelocation
definition.
-
@ben_lubar Because, again, that won't update when the DNS changes on the downstream container. It's a common problem, but I can't seem to get the usual workaround working.
-
The docs page lies. You cannot use
set
inside alocation
(or, if you can, not the same location where it's used)
-
@yamikuronue said in What gives, Nginx?:
The docs page lies. You cannot use
set
inside alocation
(or, if you can, not the same location where it's used)Hmmm. One of my teams has an nginx reverse proxy that dynamically proxies to healthy instances of services-- while we're not doing exactly what you're doing, it seems like a similar use case.
Could it be an issue with your version of nginx? We're using nginx 1.7.2 and don't have any issues setting variables inside a location block, passing them to a Lua script, or using the variables the Lua script sets in a subsequent proxy_pass.
-
@heterodox said in What gives, Nginx?:
We're using nginx 1.7.2
I think we're newer than that? not sure, I'm at home now.
Thanks everyone, though.
-
-
@yamikuronue said in What gives, Nginx?:
The docs page lies. You cannot use
set
inside alocation
(or, if you can, not the same location where it's used)Wouldn't the configuration test throw an error then?
-
@homobalkanus I dunno, man, but moving it above the
location
made it work fine.
-
@yamikuronue said in What gives, Nginx?:
wait, doesn't that make you + anyway?
If + means , probably, but we need to query DNS SRV records in order to determine the proper backend, so that's one way to do it.
-
@heterodox nevermind, they just moved lua to OpenResty. You can still get it without paying for the premium product
-
@yamikuronue said in What gives, Nginx?:
@heterodox nevermind, they just moved lua to OpenResty. You can still get it without paying for the premium product
Ah, I wasn't aware of Nginx Plus. No, that employer won't give us budget for shit, so yeah, we're using openresty.
-
@Yamikuronue Place your rewrite after the variable set.
location /v6/api/{ set $backend http://apiloadbalancer; proxy_pass $backend:3000; rewrite ^/v6/api/(.*)$ /$1 break; }
Don't understand exactly why this works but it does.
-
@cAtilre said in What gives, Nginx?:
Don't understand exactly why this works but it does.
Because the lexer parses sequentially line-by-line?
Also, welcome to the forums!
-
@Tsaukpaetra said in What gives, Nginx?:
Because the lexer parses sequentially line-by-line?
Lexers and parsers are two different things.
-
@masonwheeler
So you're saying it's because the parser lexes line-by-line?
-
@masonwheeler said in What gives, Nginx?:
Lexers and parsers are two different things.
One's a luxury car manufacturer and one's a computer program?
-
@loopback0 said in What gives, Nginx?:
@masonwheeler said in What gives, Nginx?:
Lexers and parsers are two different things.
One's a luxury car manufacturer and one's a computer program?
I thought Lexers were the people who have luxury cars...
-
@dcon said in What gives, Nginx?:
@loopback0 said in What gives, Nginx?:
@masonwheeler said in What gives, Nginx?:
Lexers and parsers are two different things.
One's a luxury car manufacturer and one's a computer program?
I thought Lexers were the people who have luxury cars...
Those are "flexers". The "f" stands for "fuck poor people".