Async Favorites



  • Last night I was fixing the functionality of a add/remove movies to a user's watch list. You know, you want to see a movie later and you add it to a list you can easily access.

    For any one with common sense, this would mean three remote API endpoints:

    • List
    • Add
    • Remove

    And the service we're working with does this. Except, when you call the endpoints for add/remove this actions don't happen in real time. For some unknown reason, we receive a 200 but the action happens some time in the future. This means that when we try again to list the favorites, we don't receive the previous changes.

    And to add more stupidity, when we try to add again, we receive an error that the movie is already in the watch list. :wtf:

    Is this documented? Of course not! :headdesk:

    I can't imagine the level of WTF that must be this backend where something so simple works in such a stupid way.



  • After how long this sync happens? Because I've been sitting here for 15 minutes and still no changes.


  • BINNED

    @Eldelshell said:

    I can't imagine the level of WTF that must be this backend where something so simple works in such a stupid way.

    Materialized views, maybe? If add/remove operates on the original tables and is instant, but the list is retrieved from a materialized view that doesn't get refreshed immediately you'd see that kind of behaviour.

    Stupid? Yeah, but could explain the stupid you see on the endpoint.



  • That makes sense, but I've never seen this sort of setup, so, in your opinion, how long does it takes to refresh this sort of view? Because right now, an hour has passed and still nothing.



  • They probably have some setup where your request goes into a queue somewhere and a worker later takes it out and performs the update. Inconvenient, but if the whole world overnight decides to start using their service, they can scale.

    In 90% of cases, this is over-engineering it.

    You're not gonna be an overnight success. You can fucking afford to do an SQL UPDATE query and return the fresh data. Not every problem is big fucking data.


  • BINNED

    @Eldelshell said:

    That makes sense, but I've never seen this sort of setup, so, in your opinion, how long does it takes to refresh this sort of view?

    Depends on their scripts / triggers if that is the case. AFAIK there is no defaults for this, though I only looked at Postgres materialized views.

    @cartman82 said:

    They probably have some setup where your request goes into a queue somewhere and a worker later takes it out and performs the update. Inconvenient, but if the whole world overnight decides to start using their service, they can scale.

    So they query the worker's queue instead of a table? Since the failure notification on double adding is returned immediately, as far as I understood the OP. That's even more brillant than my guess! Not saying it's wrong, you never know what kind of stupid you'll find next, but it's more brillant for sure.



  • @cartman82 said:

    They probably have some setup where your request goes into a queue somewhere and a worker later takes it out and performs the update.

    Nah, I think Onyx answer makes more sense since as I stated above, when you try to add the movie again, you get an error. Meaning this state is persisted.

    @cartman82 said:

    You're not gonna be an overnight success.

    Well, this client is kind of big, like world-wide big.

    Status: 90 minutes and still nothing.



  • @cartman82 said:

    They probably have some setup where your request goes into a queue somewhere and a worker later takes it out and performs the update. Inconvenient, but if the whole world overnight decides to start using their service, they can scale.

    I bet the request is sent as an e-mail to the administrator. He then prints it, puts it on a wooden table, makes a polaroid photo, which then gets sent over regular mail to some other location where a guy signs it, faxes it to a third guy who finally reads it and manually updates the record in phpMyAdmin.
    Just a hunch...



  • Well, after more than two hours waiting for the movie to show up in the list, this starts to be a feasible scenario.

    I'm starting to wonder if this is a 24h cron.


  • ♿ (Parody)

    @Eldelshell said:

    I'm starting to wonder if this is a 24h cron.

    In Oracle, I have some materialized views that refresh once per day (and at least one that refreshes once per quarter). It depends on a lot of stuff. I know that Oracle has the ability to only refresh the data that's been updated, and to do it on each and every commit, but there are a lot of restrictions about the data and what goes into the view. Most of my stuff doesn't fit that, but it's not really something like what you're doing, either.


  • BINNED

    @boomzilla said:

    I know that Oracle has the ability to only refresh the data that's been updated

    Automatically or do you have to write a trigger? Postgres only has the manual refresh (for now, I think they plan on adding it), which you can stick into a trigger, if you really want to. But I'd assume that it would be more optimized if done in-engine.


  • ♿ (Parody)

    @Onyx said:

    Automatically or do you have to write a trigger?

    When you create the MV, you optionally create a Job (Oracle's scheduled task sort of mechanism) and tell Oracle how to update. It's not really a trigger, but something built into the MV. You tell it to refresh ON DEMAND or ON COMMIT.

    Trivial example:

    create materialized view mv
      REFRESH FAST ON COMMIT
      as select * from t
    ;
    

    source: http://www.sqlsnippets.com/en/topic-12894.html

    The topic is complicated with lot of nuance (hey, it's Oracle!).



  • @cartman82 said:

    They probably have some setup where your request goes into a queue somewhere and a worker later takes it out and performs the update. Inconvenient, but if the whole world overnight decides to start using their service, they can scale.

    I like how you said that instead of a worker process. 😆


  • Java Dev

    @boomzilla said:

    but there are a lot of restrictions about the data and what goes into the view.

    Mostly obvious ones though. A materialized view is essentially a table. Full refresh means rerun the query to generate a new copy of the table.

    Fast refresh means it does an UPDATE on the materialized view table. This means it must be able to do the UPDATE based on only the records you added to the base table, rather than the entire base table.


  • ♿ (Parody)

    @PleegWat said:

    Mostly obvious ones though.

    I'm too stupid to rise to the level of Oracle obviousness, then. There are weird requirements like putting count(*) in the query. Really, it's not obvious, and instead of telling you that it won't work, stuff just silently breaks.


  • Java Dev

    Not sure on the details. I know it's in the docs - we did extensive research before we found out that 1. it doesn't perform for our (olap) needs and 2. we can't control retention separately for the MV and the base table (so we can't use it to keep aggregated data longer than details).

    On that side, there are a lot of things like 'if AVG(x) is in then SUM(x) and COUNT(x) must be as well'. which makes sense since it needs to recompute the value. Not sure where the count(*) comes in, or what the requirements are for joins.


  • ♿ (Parody)

    @PleegWat said:

    Not sure where the count(*) comes in, or what the requirements are for joins.

    Yeah, it starts getting tricky.



  • Anything yet?



  • IDK, I just forgot about it after four hours have passed and still nothing.



  • As a continuation to this specific WTF, I just got this ticket opened:

    Changes applied to "Favorites" are lost on application exit.

    :facepalm:


Log in to reply