Scraper master's code



  • An example of our old scraper master's code.

    0_1490171109257_upload-65103db6-219b-4a11-a68f-40f19c1eee7e

    It's eloquent ORM in Laravel. get() loads all found records into an array-like thing.

    Just for reference, when I execute the equivalent SELECT statement in database for a particular username, I get 17,000 records.

    When I group by code, I get 30 unique codes.


  • Winner of the 2016 Presidential Election

    @cartman82
    Wait, so he loads 17,000 records into memory (assuming get() executes the query) just to calculate some numbers using simple aggregate functions, which he should have queried in the first place?


  • Dupa

    STORED PROCEDURES TO THE RESCUE!!!!!11111oneoneoneoneone



  • @asdf said in Scraper master's code:

    Wait, so he loads 17,000 records into memory (assuming get() executes the query) just to calculate some numbers using simple aggregate functions, which he should have queried in the first place?

    Yup. Each request was taking like a minute to complete.

    I think he already "solved" this same problem once. Probably by deleting a bunch of data from db.


  • Winner of the 2016 Presidential Election

    @cartman82 said in Scraper master's code:

    I think he already "solved" this same problem once. Probably by deleting a bunch of data from db.

    :headdesk:

    Tell me that he at least asked for permission before he did that? Because even my worst third-world cow-orkers are smart enough to ask for permission before they do something that stupid.



  • @asdf said in Scraper master's code:

    Tell me that he at least asked for permission before he did that? Because even my worst third-world cow-orkers are smart enough to ask for permission before they do something that stupid.

    Permission from whom?

    No one has even seen his code until he got kicked off and now we are looking for a replacement :)



  • @asdf said in Scraper master's code:

    Tell me that he at least asked for permission before he did that.

    He puts the 'rape' in 'scrape'.



  • @cartman82 said in Scraper master's code:

    No one has even seen his code until he got kicked off and now we are looking for a replacement

    I'd do the deleting a bunch of data part for free



  • @cartman82 all of this is improved by the fact that Laravel is the slowest PHP framework.


  • Impossible Mission Players - A

    @fbmac said in Scraper master's code:

    @cartman82 said in Scraper master's code:

    No one has even seen his code until he got kicked off and now we are looking for a replacement

    I'd do the deleting a bunch of data part for free

    Huh. So that's where all the milk went...


  • Dupa

    @kt_ said in Scraper master's code:

    STORED PROCEDURES TO THE RESCUE!!!!!11111oneoneoneoneone

    Seriously though, I really hated that we used stored procedures at my previous company, but I think I learned to appreciate them. (During the first try I typed loved, but since when is justified rape love? Shit, justified? Am I turning into @Lorne-Kates?!).

    Using SPs you can encapsulate all of the necessary data manipulation in one place to make sure you're doing it consistently and efficiently in the code.

    Sure, SQL corrupts, but hey!, since so does power, maybe that's the good kind of corrupt?



  • @TimeBandit You've obviously never had to use Zend, Cake or Symfony. Then again the Eloquent ORM in Laravel does have issues. Zend is...ugh.

    Had an issue with a legacy system in Zend (2.0.something) being slow as fuck. Stripped it out and dropped it into a barebones page to test the query responsiveness. ~12 seconds (Searching a database with around 4million rows and pulling back the right data and formatting it). Laravel was about 15 seconds, Cake and CodeIgniter something simiar. Zend? Over two minutes. Fuck Zend.

    Kohana is my poison of choice at the moment, though I'm still supporting a raft of Laravel 4.2 and CodeIgniter apps. I'd take CoeIgniter as a second if I can't use Kohana and Laravel 5 reluctantly if I have no other choice.



  • @thegoryone said in Scraper master's code:

    @TimeBandit You've obviously never had to use Zend, Cake or Symfony. Then again the Eloquent ORM in Laravel does have issues. Zend is...ugh.

    Had an issue with a legacy system in Zend (2.0.something) being slow as fuck. Stripped it out and dropped it into a barebones page to test the query responsiveness. ~12 seconds (Searching a database with around 4million rows and pulling back the right data and formatting it). Laravel was about 15 seconds, Cake and CodeIgniter something simiar. Zend? Over two minutes. Fuck Zend.

    I've worked with Zend 1.x and it was not too bad.

    If I'm not mistaking, Laravel is based on Symfony.

    BTW, if you are doing your data selection in code instead of the database, you are the :wtf: (I can't imagine another reason for your code taking so much more time in Zend)

    Kohana is my poison of choice at the moment, though I'm still supporting a raft of Laravel 4.2 and CodeIgniter apps. I'd take CoeIgniter as a second if I can't use Kohana and Laravel 5 reluctantly if I have no other choice.

    I work with Code Igniter 3.x now. Good balance between performance and not framing you in only-one-way of doing things.

    See framework benchmark



  • @TimeBandit Interesting

    framework - requests per second
    siler-0.6 - 2,069.69
    laravel-5.3 - 101.94

    Fuck using frameworks



  • @groo

    Code Igniter 3.0: 753.09

    You gain a lot of functionality without too much overhead.

    Laravel is at the very bottom. I guess it's OK if you have a really beefy server and only a couple users :wink:



  • @TimeBandit laravel has symfony components, it isn't based on it.
    Also, it can be tuned to be faster if needed



  • @Jarry said in Scraper master's code:

    it can be tuned to be faster if needed

    When you want speed, you tune a Mustang, not a Smart :stuck_out_tongue_winking_eye:



  • @TimeBandit
    Yeah, but running speed it's not the only speed that matters, most of the time development speed is the one you want



  • @Jarry said in Scraper master's code:

    most of the time development speed is the one you want

    I prefer to spend a little more time on development now, then spend eternity paying 3x more for hosting.

    Laravel is like :disco: :horse: , the hardware to support it should be here in ~10 years.


  • sockdevs

    @TimeBandit said in Scraper master's code:

    When you want speed, you tune a Mustang, not a Smart :stuck_out_tongue_winking_eye:

    But a Smart would actually go around corners :P



  • @RaceProUK said in Scraper master's code:

    But a Smart would actually go around corners

    A Mustang too



  • @RaceProUK said in Scraper master's code:

    But a Smart would actually go around corners

    Meh...most cart paths don't have a lot of those.



  • @TimeBandit Zend2 is slow. Very slow. 3 is a massive improvement without a doubt, slower than 1 too for what we were working with, I suspect.

    BTW, if you are doing your data selection in code instead of the database, you are the :wtf: (I can't imagine another reason for your code taking so much more time in Zend)

    Hell no, give me some credit. But it was an enterprise-ish sized system and couldn't handle the workload like other frameworks could. We ported it to CodeIgniter and had no more problems.

    Re: Benchmarks - Not surprised Laravel5.x is so low down tbh. Found 4.x faster. I suspect most of the issues are with the templating system and Eloquent, but 4.x was at least easier to go back to and run with by comparison.



  • @thegoryone said in Scraper master's code:

    Kohana is my poison of choice at the moment, though I'm still supporting a raft of Laravel 4.2 and CodeIgniter apps. I'd take CoeIgniter as a second if I can't use Kohana and Laravel 5 reluctantly if I have no other choice.

    I found Symfony pretty pleasant. Very comparable to doing an ASP.MVC app or whatever the equivalent is in Java land. Very OOP, very good tooling, lots of modules / plugins. Some clunky crap too, and events are hard to debug.



  • @cartman82 You wouldn't have any issues with Laravel if you find Symfony useful enough.

    I'm planning to try out Yii this month at some point to see what the fuss is about (Assuming I can get my newest novel edited in time to meet my amazon pre-order date)


Log in to reply
 

Looks like your connection to What the Daily WTF? was lost, please wait while we try to reconnect.