Page 1 of 3
4scrape Down For Good
Posted: Tue Jul 28, 2009 3:59 am
by Sacchin
This site was awesome =/ Most of you prolly already know. News found by Jackal.
I no longer have the time or energy to continue maintaining the 4scrape codebase. There’s currently a crapton of issues with it –
* Cache for searches (potentially just post searches) is broken.
* Threads need to be cached as a whole unit — assembling them from a 500,000-row table is too slow.
* General consistency errors — there’s a bunch of images missing (???)
* The scraper likes to shit itself to keep things lively.
* The backend would occasionally crash/spinlock (???)
* The JavaScript shit is a horrible mess.
I took the entire server offline for now, but I’m going to re-post the shit code here along with a complete dump of the entire database (over a year’s worth of posts, lawds). I don’t really know how to offload the image data (~150GB) — I’ll talk to Shuugo about setting up a torrent (or just turn the web server back on and let people hammer the extra 150mbps).
It was a fun run; I had a good time. Sorry for crapping out on you :(
EDIT: 4scrape source code [178KB]. I should have the full database dump uploaded some time tomorrow (it’s xbox hueg, a little over 500MB compressed comprising almost a million posts and metadata for over 1/3 million images). I’m not quite sure how I’m going to offload the image data yet but it’ll either be over bittorrent, or providing unthrottled HTTP access to the image data (but not the API/front-end, so you’ll need the database dump (or just 1.jpg 2.jpg 3.jpg etc — though you’ll miss *.png etc) to accidentially the whole thing).
EDIT2: And the SQL dump [505MB]. I was running PostgreSQL 8.3; you can probably massage the dump to fit into something else but it uses some Postgres-only features (ie, for the fulltext indexes) so it isn’t going to fit into anything else (ie, MySQL/MSSQL/whatever) as-is. Additionally, I turned back on the image access (for what’s still there) and turned off the access restrictions. Feel free to hammer the balls out of the server while I figure out a better way to get them off.
Source
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:02 am
by DarkJackal
Sux.
Maybe somoene will make another or somethin.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:04 am
by Sacchin
Or offer to take over.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:05 am
by Amelie
I got a bunch of nice wallpapers from that site.
Too bad is going down
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:06 am
by Doppleganger
Ryoko volunteers i heard him from the back of the room.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:07 am
by cpinney
>_> *sad face*
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:38 am
by /Pi
And I was even planning to change my wallpaper.
Q.Q
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 4:40 am
by vietxboy911
lol, guess im the only one whos never heard of it
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 5:15 am
by Deadsolid
Sucks, where to now?
customize.org is a pain in the ass to navigate and search
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 5:23 am
by CrimsonNuker
/QQ
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 5:45 am
by GrindingEternaly
should we have a moment of silence for this God of images?
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 7:23 am
by Boiler
wow tht sucks they had so many nice pictures ._.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 7:53 am
by XemnasXD
nuuuuuuuuuuuuuuuuuu
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:35 am
by Azilius
got every single desk from there..
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:39 am
by William-CL
Lame =[ i just got into that Site not long ago
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 9:48 am
by BloodyBlade
vietxboy911 wrote:lol, guess im the only one whos never heard of it
nah you're not
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 9:55 am
by Doron
aw damnit.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 10:00 am
by William-CL
I doubt it stays Down. It was getting Really popular and has High Demand lately. I wonder if that's why it crashed lol....(The HDD failure on the 23rd) To much traffic for that puny Box they were running it on.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 2:48 pm
by inky
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 7:03 pm
by *BlackFox
Boiler wrote:wow tht sucks they had so many nice pictures ._.
Yeah, this Is bad.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 7:37 pm
by [SD]happynoobing
nooooooooooooooooo... where am i gonna get all the cool awesome loli wallpapers now? it even had a cool url
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 7:43 pm
by Nantosh
QQ
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:09 pm
by DumboDii
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:14 pm
by Sacchin
I can't tell if all of you are being sarcastic or serious.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:22 pm
by DumboDii
Sacchin wrote:I can't tell if all of you are being sarcastic or serious.
Me?
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:23 pm
by Sacchin
DumboDii wrote:Sacchin wrote:I can't tell if all of you are being sarcastic or serious.
Me?
;d
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 8:50 pm
by DumboDii
Sacchin wrote:DumboDii wrote:Sacchin wrote:I can't tell if all of you are being sarcastic or serious.
Me?
;d
Oh lol. But I am serious.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 9:58 pm
by /Pi
Sacchin wrote:I can't tell if all of you are being sarcastic or serious.
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 10:28 pm
by chesticles
someone will take the lead
Re: 4scrape Down For Good
Posted: Tue Jul 28, 2009 10:39 pm
by Sacchin
noobs -.-;;..
