AWS issues


  • Impossible Mission Players - A

    @cartman82 said in Testing candidate's mettle S02E04:

    Seems we are on.

    Hope the stupid forum software doesn't crash, it's been laggy whole day.

    I heard we ran out of AWS credits, and everyone uses the forum so consistently it hasn't built up again, so we're throttled down to like 1/3 normal. Which piles on requests. Which keeps CPU high. Which perpetuates the issue. Yay!



  • @Tsaukpaetra said in Testing candidate's mettle S02E04:

    I heard we ran out of AWS credits

    :face_palm:

    Why the fuck does a forum software need so much CPU that it requires a goddamn AWS?!?

    And how the fuck does a forum software manage to EXCEED THE AWS CREDITS?!?!


  • Impossible Mission Players - A

    @Lorne-Kates said in Testing candidate's mettle S02E04:

    @Tsaukpaetra said in Testing candidate's mettle S02E04:

    I heard we ran out of AWS credits

    :face_palm:

    Why the fuck does a forum software need so much CPU that it requires a goddamn AWS?!?

    And how the fuck does a forum software manage to EXCEED THE AWS CREDITS?!?!

    Apparently a bug caused constant crashes, and restarting the forum is CPU intensive or something.


  • sockdevs

    @Lorne-Kates said in Testing candidate's mettle S02E04:

    @Tsaukpaetra said in Testing candidate's mettle S02E04:

    I heard we ran out of AWS credits

    :face_palm:

    Why the fuck does a forum software need so much CPU that it requires a goddamn AWS?!?

    And how the fuck does a forum software manage to EXCEED THE AWS CREDITS?!?!

    why havent we bought more credits to cover the shortfall?

    because i've asked that questiong of the @mods and @admins multiple times and they've never given me an answer.

    if it's money that's the problem i'm sure we could stump up a collection among the regulars to pitch in the 20$ or so that would be needed to refresh our CPU credits and get us back on our feet.


  • mod

    @accalia Why are you asking the mods? We can't do anything about it. Go pester the admins some more.


  • sockdevs

    @abarker said in Testing candidate's mettle S02E04:

    @accalia Why are you asking the mods? We can't do anything about it. Go pester the admins some more.

    because i have pestered the admins and gotten 0 response.

    i figure maybe the mods can apply some pressure on the admins to at least get an answer where apparently i can't.


  • mod

    @accalia said in Testing candidate's mettle S02E04:

    i figure maybe the mods can apply some pressure on the admins to at least get an answer where apparently i can't.

    Why don't you try creating a new thread in meta? It wouldn't be the first time that forum support has been drummmed up for something in a grass-roots sort of way. Really, though, the one you most likely need to talk to is @BenLubar, and he might not be able to do much. You might actually need to talk to @apapadimoulis. I don't know who has access the account for the VM environment, but it certainly ain't the Mods. You pester us when we can't do shit about it, and you'll just piss us off.

    It seems like there was someone else I had a similar conversation with a few weeks ago …



  • @accalia I'm pretty sure we can't buy additional CPU credits. Here, have a graph:

    0_1476981231928_upload-d1571327-b6ca-4921-958d-bd81baa29683


  • Winner of the 2016 Presidential Election

    @ben_lubar what's been using up the credits? Is it still the container restarting when the youtube plugin crashes it or whatever was mentioned the other day?


  • sockdevs

    @abarker said in Testing candidate's mettle S02E04:

    Why don't you try creating a new thread in meta?

    I tried asking on IRC several times when ben_lubar was online and got no answer, i reached out to boomzilla as well, neither request got any answer. i mentioned both on these forums before and have not gotten an answer.

    That's not i got an answer that didn't satisfy, literally my questions were not acknowledged nor responded to. you think my mentioning the mod group is annoying? well i'm sorry about that but my DIRECT attempts to reach out to the relevant admins got me ZERO response, so apparently i'm not important enough to reach out to them and get an answer. Thus, i sought to find someone more important to get an answer through.

    which brings us to today when i finally got an answer, kinda, to my question. An answer that came only after engaging the moderators of this forum.

    @ben_lubar said in Testing candidate's mettle S02E04:

    @accalia I'm pretty sure we can't buy additional CPU credits. Here, have a graph:

    Okay. Fine.

    Why can we not buy more CPU credits and HOW does that graph explain why we're still experiencing the longest and most pronounced cooties event since servercooties.io started tracking the forum.



  • @accalia said in Testing candidate's mettle S02E04:

    Why can we not buy more CPU credits and HOW does that graph explain why we're still experiencing the longest and most pronounced cooties event since servercooties.io started tracking the forum.

    Apparently credits can only be accrued at a set rate during idle time on the instance, if how I'm reading Amazon's description of how that works accurately, and the rate (and maximum banked amount of credits) goes up the larger the instance you buy.

    As for why it might be causing cooties, it's because if you run out of credits, your instance won't scale. It'll be capped at a fixed baseline % of usage of a CPU core until you idle it enough to accrue some credits to allow it to burst again.


  • sockdevs

    @e4tmyl33t said in Testing candidate's mettle S02E04:

    @accalia said in Testing candidate's mettle S02E04:

    Why can we not buy more CPU credits and HOW does that graph explain why we're still experiencing the longest and most pronounced cooties event since servercooties.io started tracking the forum.

    Apparently credits can only be accrued at a set rate during idle time on the instance, if how I'm reading Amazon's description of how that works accurately, and the rate (and maximum banked amount of credits) goes up the larger the instance you buy.

    so what you are saying is that we need to buy a larger instance? cool. let's do it. bump the instance size up then if a single bug can tank our performance to an unrecoverable cooties storm.

    is it a case of money? fine. where should we paypal the money to? a couple of us chip in a coffee or two's worth of cash a month and we have it covered.


  • mod

    @accalia said in Testing candidate's mettle S02E04:

    An answer that came only after engaging the moderators of this forum.

    to be fair, I've asked him point blank repeatedly on my own before today's effort gathered more success. It's not just because you mentioned us, I'd been poking him about it for days.


  • sockdevs

    @Yamikuronue said in Testing candidate's mettle S02E04:

    to be fair, I've asked him point blank repeatedly on my own before today's effort gathered more success.

    so.... our communications from admins to/from forum users is shit all the way around.

    great.

    how do we fix this?

    no, seriously. how do we fix this?

    We have a communication issue that needs to be fixed here when we can have multiple people ask direct questions of the admin staff and the response is either an empty carrier wave or

    I'm pretty sure we can't fix X. Here, have a unrelated image

    and that response only after multiple days of multiple people trying to get information.

    we have a communication problem here, and i'm sick of being part of it. so, What do we need to do to fix it and how can i help be part of that solution?



  • @accalia said in Testing candidate's mettle S02E04:

    a couple of us chip in a coffee or two's worth of cash a month and we have it covered.

    I think that depends on both what size of instance we're looking at, plus what kind of lockin, plus what OS it runs on (if this pricing thing is to be trusted). On-demand servers are more expensive than reserved instances, too, if you're not planning to lock in for a 1 or 3 year period.


  • sockdevs

    @e4tmyl33t said in Testing candidate's mettle S02E04:

    @accalia said in Testing candidate's mettle S02E04:

    a couple of us chip in a coffee or two's worth of cash a month and we have it covered.

    I think that depends on both what size of instance we're looking at, plus what kind of lockin, plus what OS it runs on (if this pricing thing is to be trusted). On-demand servers are more expensive than reserved instances, too, if you're not planning to lock in for a 1 or 3 year period.

    so it takes more than one or two people sacrificing their Friday Coffees.

    whatever, my point is we CAN and SHOULD solve this.


  • mod

    @Tsaukpaetra @Lorne-Kates @accalia @abarker @ben_lubar @Jaloopa @e4tmyl33t

    Does anyone mind if I move this out of the Lounge?


  • sockdevs

    @Yamikuronue said in AWS issues:

    @Tsaukpaetra @Lorne-Kates @accalia @abarker @ben_lubar @Jaloopa @e4tmyl33t

    Does anyone mind if I move this out of the Lounge?

    i'm fine with that. the issue is public so the discussion of the issue and how we're going to solve it should be public.


  • area_can

    @Lorne-Kates said in AWS issues:

    And how the fuck does a forum software manage to EXCEED THE AWS CREDITS?!?!

    :shrug:



  • @Yamikuronue said in AWS issues:

    @Tsaukpaetra @Lorne-Kates @accalia @abarker @ben_lubar @Jaloopa @e4tmyl33t

    Does anyone mind if I move this out of the Lounge?

    Given I'm no dev, just a consumer of the toxic hellstew that is our forum (needs some pepper, by the way), I'm fine with it being brought up wherever, if it's for the betterment of everyones' experience. I just figured I'd look up the stats and pricing on AWS stuff because I was bored at work :D



  • @ben_lubar We'd need to move off the burstable t2 class and on to the m3 or m4 class. Or at least move to a t2.large which gets 36 credits/hour instead of 24.

    @accalia said in AWS issues:

    if it's money that's the problem i'm sure we could stump up a collection among the regulars to pitch in the 20$ or so that would be needed to refresh our CPU credits and get us back on our feet.

    I almost started a GoFundMe yesterday, but I have no way of getting the money to @apapadimoulis


  • Winner of the 2016 Presidential Election

    (MFW I click to the next page of a thread as it's being jeffed...)

    AWS in general seems like an expensive solution to the problem of hosting this forum...


  • Impossible Mission - B

    We still have not answered @Lorne-Kates's original question: why is this forum on AWS in the first place?

    Filed under: Head in the clouds


  • sockdevs

    @pydsigner said in AWS issues:

    AWS in general seems like an expensive solution to the problem of hosting this forum...

    This is quite possible, but we're on AWS now and we have resource problems today. We should fix those resource problems today and once we're out of the cooties we can look to see if a migration to a different VM host would be beneficial.



  • @masonwheeler said in AWS issues:

    We still have not answered @Lorne-Kates's original question: why is this forum on AWS in the first place?

    Filed under: Head in the clouds

    I assumed it was because it was a potential easy solution to the scaling issue. Don't need to run a permanently beefy box if you only need it at certain times...



  • @accalia said in Testing candidate's mettle S02E04:

    Why

    Amazon's documentation shows that you can buy additional initial credits, but that only applies if you're starting hundreds of EC2 instances per week. They only say that we get 24 CPU credits per hour with our setup.

    @accalia said in Testing candidate's mettle S02E04:

    HOW

    We had a number of things happen that used up a lot of processing power. Obviously, the YouTube plugin restarting the forum was an issue, but also the watchdog.bash script restarting the forum repeatedly while it was trying to start was a problem, and "SemrushBot" didn't help either, with its 15000 pageviews so far in the log. (I blocked it with the other banned user-agents, and the forum got faster nearly instantly).

    So hopefully with all those things fixed, the site will be able to regenerate enough CPU credits during low activity times to get back into the normal routine.

    The 4 hours early this morning when the site was stuck in a busy loop didn't help much, either. I think I'll tweak the timeouts watchdog.bash has and re-enable it.


  • Impossible Mission - B

    @e4tmyl33t said in AWS issues:

    I assumed it was because it was a potential easy solution to the scaling issue. Don't need to run a permanently beefy box if you only need it at certain times...

    Yeah, and how's that working out for us so far?


  • sockdevs

    @ben_lubar Now that, that was good communication. i have something to work with there

    so as to your first bit, this is true to my understanding, but i also understand from my interactions with AWS that moving from a t2.medium to a t2.large is totally possible with minimal downtime (basically a host reboot) this will get us earning credits faster than we are using them again, eliminating the cooties. Once we're out of the storm we can look at options for downsizing again.

    as for the second, that only makes it more clear to me that we should be on the t.2 large, or even off of the CPU credit tier entirely and onto a more dedicated resource tier such as the m tiers, we have a wildly unpredictable CPU usage environment here and this incident has made an incident that would have been cleared up days ago on a dedicated compute tier while we struggle to earn compute credits fast enough to get back to being stable even now.

    Is it money that prevents implementing this solution? because if it is i know we can crowdsource enough to cover it. 6$/month across 20 of the forum regulars would do it easily, that's like two cups of coffee, or one if you go to starbucks.


  • Winner of the 2016 Presidential Election

    @e4tmyl33t said in AWS issues:

    @masonwheeler said in AWS issues:

    We still have not answered @Lorne-Kates's original question: why is this forum on AWS in the first place?

    Filed under: Head in the clouds

    I assumed it was because it was a potential easy solution to the scaling issue. Don't need to run a permanently beefy box if you only need it at certain times...

    AWS on demandt2.large [on demand] + 200GB storage + 500GB out$1641.36/year
    AWS reserved t2.large [1 year 100% down reserved] + 200GB storage + 500GB out$664.65/year
    DO on demand 8GB + 200GB extra storage + 5TB bandwidth $1200/year
    SpeedyKVM unresizeable monthly8GB + 400GB storage + 4.5TB bandwidth $768/year
    SpeedyKVM unresizeable reserved8GB + 400GB storage + 4.5TB bandwidth $384/year

    Just a comparison of various hosting services to get at least 2 cores, 8GB RAM, 200GB storage, and 500GB bandwidth. I don't know what we actually need currently.


  • mod

    Ok. I'm going to move it. Feel free to delete any post you feel is sensitive.


  • sockdevs

    @e4tmyl33t said in AWS issues:

    @masonwheeler said in AWS issues:

    We still have not answered @Lorne-Kates's original question: why is this forum on AWS in the first place?

    Filed under: Head in the clouds

    I assumed it was because it was a potential easy solution to the scaling issue. Don't need to run a permanently beefy box if you only need it at certain times...

    this is actually true, IF the forum was set up to scale with load.

    but it isn't, apparently.

    @pydsigner said in AWS issues:

    Just a comparison of various hosting services to get at least 2 cores, 8GB RAM, 200GB storage, and 500GB bandwidth. I don't know what we actually need currently.

    yep.

    we have options for migrating to a different host for additinal resources without additional cost (after teh initial cost of the migration) but that doesn't change that we have a problem today.

    or rather we had a problem earlier this week that we're still not recovered from because we're a compute heavy site on a metered compute VM



  • @accalia said in AWS issues:

    this is actually true, IF the forum was set up to scale with load.
    but it isn't, apparently.

    Based on what I've read, you're right. It IS built to do clustering, but not load-based scaling, so you'd still need an external load balancer to spread the load out over multiple instances/machines/whatever.


  • sockdevs

    @e4tmyl33t said in AWS issues:

    @accalia said in AWS issues:

    this is actually true, IF the forum was set up to scale with load.
    but it isn't, apparently.

    Based on what I've read, you're right. It IS built to do clustering, but not load-based scaling, so you'd still need an external load balancer to spread the load out over multiple instances/machines/whatever.

    hmm, you can do the load based scaling in AWS, it's a bit tricky to set up, and honestly i'd love for AWS to automate the setup much like Azure has, but it is possible to set up (basically you just have an automated process monitor your cluster and spin nodes up/down as necessary. a PITA to set up but once set up is mostly fire and forget)



  • @accalia said in AWS issues:

    @ben_lubar Now that, that was good communication. i have something to work with there

    This.


  • Winner of the 2016 Presidential Election

    @ben_lubar said in AWS issues:

    They only say that we get 24 CPU credits per hour with our setup.

    So a t2.medium.


  • mod

    @pydsigner said in AWS issues:

    t2.large [on demand] $1641.36/year
    t2.large [reserved] $664.65/year

    @reverendryan said in AWS issues:

    We'd need to move off the burstable t2 class and on to the m3 or m4 class.

    An m3.medium reserved would be $420/yr with no up front. An m3.large reserved would be about $832/yr.

    m3.medium: 3.75 GiB of memory, 1 vCPU, 4 GB of SSD-based local instance storage, 64-bit platform
    m3.large: 7.5 GiB of memory, 2 vCPUs, 32 GB of SSD-based local instance storage, 64-bit platform

    So I think the real question is, what is the current budget for this forum, and what are we on now exactly? We're willing to do the cost/benefit analysis and find the best size for the budget, and potentially even willing to cover shortfalls, but we need data to do so.

    @ben_lubar ?


  • Winner of the 2016 Presidential Election

    @Yamikuronue said in AWS issues:

    An m3.medium reserved would be $420/yr with no up front. An m3.large reserved would be about $832/yr.

    m3.medium: 3.75 GiB of memory, 1 vCPU, 4 GB of SSD-based local instance storage, 64-bit platform
    m3.large: 7.5 GiB of memory, 2 vCPUs, 32 GB of SSD-based local instance storage, 64-bit platform

    Just remember that you have to factor in bandwidth and storage as well. Also, IIRC, m4.larges might be cheaper than m3.larges.


  • Winner of the 2016 Presidential Election

    @accalia said in AWS issues:

    fine. where should we paypal the money to?

    @ben_lubar said in AWS issues:

    We had a number of things happen that used up a lot of processing power.

    Call me stupid, but can't we just have some software constantly throttle our server to 90% CPU? According to 0.1 second of googeling CPU-limit exists for such a purpose. Yes, we would probably lose a few miliseconds here and there but at least we wouldn't be put to 33%...

    Filed Under: I mean, you guys can also just throw money at the problem. Glad you all think like programmers and not like managers :trolleybus:


  • sockdevs

    @Yamikuronue said in AWS issues:

    So I think the real question is, what is the current budget for this forum, and what are we on now exactly? We're willing to do the cost/benefit analysis and find the best size for the budget, and potentially even willing to cover shortfalls, but we need data to do so.

    basically yeah.

    I'm totally willing to throw down some hard cash for this thing, at least so long as i have help, i'm not exactly thrilled to carry the whole load but....

    /me calculates on back of envelope

    Assuming wtdwtf covers half the costs at...... ~500/yr$

    That means the forum has to cover ~500/yr

    that's 41. numbers per month.

    call it 45$ per month to cover extraneous fees and shit

    assume each donor chips in a fiver.....

    that's nine. Nine of the regular denizens a month that have to decide to skip one (maybe two) coffees and donate the cash instead to the forum to keep it running and we cna have a yearly bidget of 1k$

    yeah, a M3.large with 100/100 GB transfer is about 105/mo. that storage is a little low, but EBS storage (or even S3 storage) is DIRT cheap.... call it 115/month tops.

    the m4.large could be cheaper depends on exact usage.

    that goes a little over my back of the envelope budget, but not unattainably so


  • Winner of the 2016 Presidential Election

    @accalia said in AWS issues:

    @Yamikuronue said in AWS issues:

    So I think the real question is, what is the current budget for this forum, and what are we on now exactly? We're willing to do the cost/benefit analysis and find the best size for the budget, and potentially even willing to cover shortfalls, but we need data to do so.

    basically yeah.

    I'm totally willing to throw down some hard cash for this thing, at least so long as i have help, i'm not exactly thrilled to carry the whole load but....

    /me calculates on back of envelope

    Assuming wtdwtf covers half the costs at...... ~500/yr$

    That means the forum has to cover ~500/yr

    that's 41. numbers per month.

    call it 45$ per month to cover extraneous fees and shit

    assume each donor chips in a fiver.....

    that's nine. Nine of the regular denizens a month that have to decide to skip one (maybe two) coffees and donate the cash instead to the forum to keep it running and we cna have a yearly bidget of 1k$

    yeah, a M3.large with 100/100 GB transfer is about 105/mo. that storage is a little low, but EBS storage (or even S3 storage) is DIRT cheap.... call it 115/month tops.

    the m4.large could be cheaper depends on exact usage.

    that goes a little over my back of the envelope budget, but not unattainably so

    There is no way we should be paying 1400/year to host this forum. I mean, may as well go dedicated at that point.


  • sockdevs

    @pydsigner said in AWS issues:

    @accalia said in AWS issues:

    @Yamikuronue said in AWS issues:

    So I think the real question is, what is the current budget for this forum, and what are we on now exactly? We're willing to do the cost/benefit analysis and find the best size for the budget, and potentially even willing to cover shortfalls, but we need data to do so.

    basically yeah.

    I'm totally willing to throw down some hard cash for this thing, at least so long as i have help, i'm not exactly thrilled to carry the whole load but....

    /me calculates on back of envelope

    Assuming wtdwtf covers half the costs at...... ~500/yr$

    That means the forum has to cover ~500/yr

    that's 41. numbers per month.

    call it 45$ per month to cover extraneous fees and shit

    assume each donor chips in a fiver.....

    that's nine. Nine of the regular denizens a month that have to decide to skip one (maybe two) coffees and donate the cash instead to the forum to keep it running and we cna have a yearly bidget of 1k$

    yeah, a M3.large with 100/100 GB transfer is about 105/mo. that storage is a little low, but EBS storage (or even S3 storage) is DIRT cheap.... call it 115/month tops.

    the m4.large could be cheaper depends on exact usage.

    that goes a little over my back of the envelope budget, but not unattainably so

    There is no way we should be paying 1400/year to host this forum. I mean, may as well go dedicated at that point.

    there's a point.

    if we go over to our "friends" at OVH we can get a dedicated server for.......

    yeah their HOST-32L configuration is 69$/month and has an Intel Xeon D-1520 with 4 compute cores and hyperthreading, 32GB of RAM, 2TB RAID1, and gigabit bandwidth. that's 828$/year with nop setup costs and < 24 hour from ordering to availability.

    other Dedicating hosting solutions will be competitive with that (and honestly we can probably shave a fair bit off that for lower specs.

    Downside of course is, harder to scale the server than with VM

    we have options here.

    what we don't have is much, if any engagement from the admins of the site

    where did @ben_lubar wander off to anyway?


  • mod

    @pydsigner said in AWS issues:

    that you have to factor in bandwidth and storage as well.

    Oh yes, that's the other big question: What are we using?

    How much disk space have we eaten up in the past couple years of image uploads and rambling on? How much bandwidth do we average per month with all our likeses and lounging? What is our actual usage, and how fast is it growing?


  • Winner of the 2016 Presidential Election

    For the record I'm happy to chip in $10/month

    all the time that's still less than £10, but the way that's going it won't be long before I switch to £10 instead of $10 as $10 will be more than that... but anyway


  • Impossible Mission - B

    @pydsigner said in AWS issues:

    There is no way we should be paying 1400/year to host this forum. I mean, may as well go dedicated at that point.

    See above, re: Head In The Clouds



  • If we can get this to scale out using multiple instances, we can use spot instances, which are hell of a lot cheaper. So like one main instance and one or two spots?

    The spot instances can get killed if there's sudden instance demand in a datacenter for some reason, but should work 99% of the time.



  • I do find it both amusing and terrible that in the middle of trying to load this discussion about how our server situation's fucked up that I got a 502 Bad Gateway response...



  • At current spot prices, with m1.medium being just under a penny an hour, it would cost about $88 a year.

    If we had 2 of those plus a small dedicated instance for cases when these instances go down, I think we can do it for under $500 a year.


  • sockdevs

    From IRC:

    <accalia`work> Ben... not to call you r10pez10 or anything, but could you use words more often? please? the lack of communication and engagement on this issue has been absolutely infuriating and simply posting links or images without context does not help
    <accalia`work> yes, i can read the git message, and understand it, but having your words explain it would make a huge difference to the perceived engagement of the admin staff for the forum

    because people who aren't on IRC should probably see that comment and be able to comment on it too.



  • @Yamikuronue said in AWS issues:

    How much disk space have we eaten up in the past couple years of image uploads and rambling on?

    Uploads:

    2014-02: 0.0MiB
    2014-04: 0.9MiB
    2014-05: 93.1MiB
    2014-06: 205.5MiB
    2014-07: 330.6MiB
    2014-08: 280.2MiB
    2014-09: 229.1MiB
    2014-10: 565.5MiB
    2014-11: 440.8MiB
    2014-12: 501.2MiB
    2015-01: 372.4MiB
    2015-02: 534.4MiB
    2015-03: 1525.3MiB
    2015-04: 606.8MiB
    2015-05: 556.7MiB
    2015-06: 205.8MiB
    2015-07: 163.5MiB
    2015-08: 286.6MiB
    2015-09: 382.6MiB
    2015-10: 473.5MiB
    2015-11: 408.0MiB
    2015-12: 455.6MiB
    2016-01: 445.1MiB
    2016-02: 304.9MiB
    2016-03: 332.3MiB
    2016-04: 263.8MiB
    2016-05: 307.6MiB
    2016-06: 335.8MiB
    2016-07: 438.9MiB
    2016-08: 493.6MiB
    2016-09: 529.6MiB
    2016-10: 269.4MiB

    MongoDB:

    4.0K /data/configdb
    98M /data/db/diagnostic.data
    3.1G /data/db/journal
    24G /data/db
    24G /data

    (if anyone has any ideas on how to split that up between months, let me know)


  • Discourse touched me in a no-no place

    What the fuck is a CPU credit and why has nobody ever mentioned this in my umpteen discussions about using AWS at work?

    I was under the impression that if my AWS VM was on, it's resources were fully allocated for my use.

    Unless there's some janky cutrate tier I scrolled right past.


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.