Scraper master's code
-
An example of our old scraper master's code.
It's eloquent ORM in Laravel.
get()loads all found records into an array-like thing.Just for reference, when I execute the equivalent
SELECTstatement in database for a particular username, I get 17,000 records.When I group by
code, I get 30 unique codes.
-
@cartman82
Wait, so he loads 17,000 records into memory (assumingget()executes the query) just to calculate some numbers using simple aggregate functions, which he should have queried in the first place?
-
STORED PROCEDURES TO THE RESCUE!!!!!11111oneoneoneoneone
-
@asdf said in Scraper master's code:
Wait, so he loads 17,000 records into memory (assuming get() executes the query) just to calculate some numbers using simple aggregate functions, which he should have queried in the first place?
Yup. Each request was taking like a minute to complete.
I think he already "solved" this same problem once. Probably by deleting a bunch of data from db.
-
@cartman82 said in Scraper master's code:
I think he already "solved" this same problem once. Probably by deleting a bunch of data from db.
Tell me that he at least asked for permission before he did that? Because even my worst third-world cow-orkers are smart enough to ask for permission before they do something that stupid.
-
@asdf said in Scraper master's code:
Tell me that he at least asked for permission before he did that? Because even my worst third-world cow-orkers are smart enough to ask for permission before they do something that stupid.
Permission from whom?
No one has even seen his code until he got kicked off and now we are looking for a replacement :)
-
@asdf said in Scraper master's code:
Tell me that he at least asked for permission before he did that.
He puts the 'rape' in 'scrape'.
-
@cartman82 said in Scraper master's code:
No one has even seen his code until he got kicked off and now we are looking for a replacement
I'd do the deleting a bunch of data part for free
-
@cartman82 all of this is improved by the fact that Laravel is the slowest PHP framework.
-
@fbmac said in Scraper master's code:
@cartman82 said in Scraper master's code:
No one has even seen his code until he got kicked off and now we are looking for a replacement
I'd do the deleting a bunch of data part for free
Huh. So that's where all the milk went...
-
@kt_ said in Scraper master's code:
STORED PROCEDURES TO THE RESCUE!!!!!11111oneoneoneoneone
Seriously though, I really hated that we used stored procedures at my previous company, but I think I learned to appreciate them. (During the first try I typed loved, but since when is justified rape love? Shit, justified? Am I turning into @Lorne-Kates?!).
Using SPs you can encapsulate all of the necessary data manipulation in one place to make sure you're doing it consistently and efficiently in the code.
Sure, SQL corrupts, but hey!, since so does power, maybe that's the good kind of corrupt?
-
@thegoryone said in Scraper master's code:
@TimeBandit You've obviously never had to use Zend, Cake or Symfony. Then again the Eloquent ORM in Laravel does have issues. Zend is...ugh.
Had an issue with a legacy system in Zend (2.0.something) being slow as fuck. Stripped it out and dropped it into a barebones page to test the query responsiveness. ~12 seconds (Searching a database with around 4million rows and pulling back the right data and formatting it). Laravel was about 15 seconds, Cake and CodeIgniter something simiar. Zend? Over two minutes. Fuck Zend.
I've worked with Zend 1.x and it was not too bad.
If I'm not mistaking, Laravel is based on Symfony.
BTW, if you are doing your data selection in code instead of the database, you are the
(I can't imagine another reason for your code taking so much more time in Zend)Kohana is my poison of choice at the moment, though I'm still supporting a raft of Laravel 4.2 and CodeIgniter apps. I'd take CoeIgniter as a second if I can't use Kohana and Laravel 5 reluctantly if I have no other choice.
I work with Code Igniter 3.x now. Good balance between performance and not framing you in only-one-way of doing things.
See framework benchmark
php-framework-benchmark/README.md at master · kenjis/php-framework-benchmark
PHP Framework Benchmark. Contribute to kenjis/php-framework-benchmark development by creating an account on GitHub.
-
@TimeBandit Interesting
framework - requests per second
siler-0.6 - 2,069.69
laravel-5.3 - 101.94Fuck using frameworks
-
Code Igniter 3.0: 753.09
You gain a lot of functionality without too much overhead.
Laravel is at the very bottom. I guess it's OK if you have a really beefy server and only a couple users
-
@TimeBandit laravel has symfony components, it isn't based on it.
Also, it can be tuned to be faster if needed
-
@Jarry said in Scraper master's code:
it can be tuned to be faster if needed
When you want speed, you tune a Mustang, not a Smart
-
@TimeBandit
Yeah, but running speed it's not the only speed that matters, most of the time development speed is the one you want
-
@Jarry said in Scraper master's code:
most of the time development speed is the one you want
I prefer to spend a little more time on development now, then spend eternity paying 3x more for hosting.
Laravel is like
, the hardware to support it should be here in ~10 years.
-
@TimeBandit said in Scraper master's code:
When you want speed, you tune a Mustang, not a Smart
But a Smart would actually go around corners :P
-
@RaceProUK said in Scraper master's code:
But a Smart would actually go around corners
A Mustang too
-
@RaceProUK said in Scraper master's code:
But a Smart would actually go around corners
Meh...most cart paths don't have a lot of those.
-
@thegoryone said in Scraper master's code:
Kohana is my poison of choice at the moment, though I'm still supporting a raft of Laravel 4.2 and CodeIgniter apps. I'd take CoeIgniter as a second if I can't use Kohana and Laravel 5 reluctantly if I have no other choice.
I found Symfony pretty pleasant. Very comparable to doing an ASP.MVC app or whatever the equivalent is in Java land. Very OOP, very good tooling, lots of modules / plugins. Some clunky crap too, and events are hard to debug.
