How long Will it take to scrape around 10.000 games?

Tomsen · 2019-10-16T10:46:18+00:00

Did anybody scrape this amound of games? Its around 300gb.

Tomsen

BZZ thanks for your reply. internal? What do you mean by that? Iternal hard drive you mean? I got An internal drive, and i Will use batocera tot scrape. I'm going to do it over night. Buy when you ad new games and scrape again, Will it Just scrape the new games? Or start all over?

Andearoid

I did it and it can take a long time. Depends the DB server you're requesting to process your scrape.

I think the best way is using Universal XML Scraper or Skraper softwares to get games snapshots. However their DB server (ScreenScraper) has many problems with HTTP caching and server overload. This means ScreenScraper can be unreachable and since few weeks I found that many games are no more available from their DB.

In my opinion, Skraper is quicker than UXS. And it offers more options. Also, I heard ARRM is a good software for roms scraping too. But I didn't try it until now.

Tomsen

Andearoid but how i'm supose to use another scraper outside of batocera when batocera is my only operation system?

BZZ

Tomsen just download one of the ones Andearoid suggested. You just need a PC on the network you put the IP of that PC in one of the ones you will pick and it will scrape for you.... I personally use ARRM (http://jujuvincebros.fr/telechargements2/file/10-arrm-another-recalbox-roms-manager). Check this topic nexusone13 I am sure Nexusone will help you if you have trouble ;-)

Tomsen

BZZ thanks.

Andearoid

I tried ARRM this weekend. I think it's the better choice to manage and scrape games snapshots. Because you can edit and scrap only romfiles you choosed before. It's really useful when you don't need to scan all your roms. But the UI is not easy to understand at the first look. Personaly I need round an hour to understand that "Systems folders" means "Roms path". I was looking everywhere how to set my roms directories. So don't be hurry. Take your time before starting a process because it can reset your previous scraps.

About time for scraping, ARRM is like UXS. It takes longer than Skraper. I think ARRM and UXS use the ScreenScraper API 1. And Skraper uses the ScreenScraper API 2.

ARRM and UXS gived me less "unknown" roms than Skraper.
Ex: Genesis/Megadrive fullset (No-Intro validated by ClrMamepro) = 2373 roms
ARRM = 53 not found
UXS = 69 not found
Skraper = 162 not found

I did it with other romsets and ARRM is the most efficient. Skraper result is curently very wrong. But the problem is not from Skraper software. It comes from the ScreenScraper API 2. Because few weeks ago Skraper gived me good results. As good as UXS and sometimes better.

Tomsen

Andearoid well i tried the internal scraper and only some maps worked. It kept on saying: game GameCube, after a couple of hours i bashed some buttons on my controller en suddenly the proces was done. Some maps where not even done, so i will try ARRM for sure. At the moment i'm still hauling in alot of games, so ill try it when i think im kinda done with getting games.

Andearoid

To get games snapshots, manuals, video, etc from ScreenScraper your roms must have the same MD5/CRC or the same filename than records stored as knowledge in the ScrenScraper's database. My experience thinks most newgen roms (like Gamecube) aren't matching very well. That's probly the reason why you get few snapshots for your roms.

By the past, many downloadable roms over the Internet were wrongly dumped. So, it's important to check your romfiles (MD5 + filename) with ClrMamePro to be sure you've the most known/better version file. With the good roms versions you get better scrap's results.

You can find and download XML dictionnaries at Dat-O-Matic and check your romfiles for many hardware with ClrMamePro.
https://datomatic.no-intro.org/?page=download

This roms manager tool is also a must have to keep your MAME's roms uptodate. Read it's tutorial carrefully : http://clrmamepro.free.fr/clrmamepro.php

If you're looking to start with great fullsets search "No-Intro 2019" at Archive.org, then browse thoroughly the results page.

Tomsen

Andearoid roms manager? Keep my roms up to date? You mean if i update batocera, roms might not work anymore?

Andearoid

If your roms are 100% playable with current Batocera's release, they will continue to work with the next Batocera update.

A RomManager can help you to :
. rename your romfiles properly
. check data integrity of your roms
. get a better matching with scraper's databases to grab games screenshots

BZZ

Andearoid "If your roms are 100% playable with current Batocera's release, they will continue to work with the next Batocera"... This might not be accurate. Remember the forum topics when MAME core was removed as it was no longer supported by devs ? : )

BZZ

Andearoid I am not negating this just pointing out that BZZ ... You however have previously Andearoid ;-) No matter as this has gone of topic twice over ;-)

Tomsen

Well we Will Just have see the;-)

BZZ

Tomsen What is there to see ? It already happened ; - ) Hopefully they will add these cores in future as even if they are no longer supported by devs many people have roms/romsets that require them ;-) Some people went back a version or two just to get them back to work rather than re-downloading the whole library of games ;-)

The cores that no longer are supported should have just been kept in Batocera. People that dont use them will not see the difference and people that do will continue to enjoy them ;-)

Tomsen

Tomsen stupid autocorrect is f*cking up my words

Tomsen

BZZ well i will see when a upgrade comes and i can't run the games anymore ;-)

Andearoid

Batocera's devs have removed MAME 2016 (core 0.174) from the latest release. That's true. But it doesn't really mater. Because they added MAME Core 0.212 (august 2019) which is really newer and better. To compare, Recalbox's owners don't have this new core version available for now.

To upgrade an old MAME romset to a newer version, it's easier to use a RomManager. Because some old roms will not be able to work with new MAME core. All arcade emulators like MAME or FinalBurn Neo work like this.

Andearoid

BZZ Previously I was using my Batocera with a MAME 2016 (0.174) core because it was the most recent version available to play arcade games. I didn't know about the new libretro-mame is supporting the 0.212 core in Retroarch. I wouldn't have wanted to downgrade my 2016 romset compatibilty to a 2010 or 2003 romset version.

I keep my MAME's romset uptodate every month for many many years. Current version is 0.214. And it's easier to upgrade from an old romset than downgrade from the latest version.

Tomsen I converted some CUE/BIN to CHD with chdman.exe (provided with MAME). It works with Sega CD, PC Engine CD or Dreamcast. CHD is an ISO compressed format.

Here is the DOS command => chdman.exe createcd -i "%SOURCE_FILE%.cue" -o "%TARGET_FILE%.chd"

ISOBuster can help you to convert .ISO to CUE/ISO.
Then rename .ISO to .BIN and use the chdman.exe command to make your CHD.

Tomsen

I just download a massive amount of dreamcast games, but offcourse when unpack the them I get a lot files. For example I got 1 map with 19 different files (track,bin,gdi). Now I place them in a map and place them in the dreamcast folder otherwise it will look messy. But I like to scrape them, or get some kind of image. Is they only way to do this, is to place all the games in the main folder and hide like 3000 files?

BZZ

Tomsen You can do that yes. That is what I did ;-) ARRM for example can arrange the list by file extension and then you just click hide on the ones you do not want to see..... I am not sure if Dreamcast can read ISO but if so you can just download sets that are single file.

joinski

Tomsen
you could also use different converter-tools to convert those dreamcast games into single cdi or chd files.
just google it 😉

Tomsen

joinski yes i'm going to try that.