1 (edited by Egen 2016-08-05 11:17:33)

Topic: Why are verifications checked manually?

I'm just curious. I apologize if the staff is getting tired of hearing about how many verifications have been lying around since <insert year here> and blah blah and this isn't meant to be one of those topics, I just wonder why the staff essentially force a gigantic burden upon themselves, ultimately generating the large backlog there is now. Wouldn't it make more sense to have a form a user can submit to that has all the fields you need to fill out for your dump? And then the system just checks all the entered data against the existing data. That way you guys don't have to go through 3847398335 of them manually one-by-one. If there is a data discrepancy, then the submission can be flagged.

It just seems like this would really help all parties involved, but maybe I'm dumb and don't understand a reason, which is only why I ask in my curiosity smile

Re: Why are verifications checked manually?

The important part for verifications isn't the data submitted by the dumper, but the provided logs, so the mod could see, if the game was dumped properly to avoid the case when the first dump was taken incorrectly and the "confirmation" was done the similar way.

Re: Why are verifications checked manually?

What logs are these? For specific systems only? I don't remember ever being told to supply log information anywhere for verifications on this site.

And what goes into evaluating these logs? Is there a consistent way to spot errors across the logs? Certain tracker sites have a system in place to automatically evaluate EAC logs, because the output is the same all the time, and so certain things should look a certain way. I can't imagine any logs you guys read are any more arbitrary and require specific human attention; couldn't you write something similar to evaluate any of these logs for you?

Re: Why are verifications checked manually?

DIC logs are not really needed for certain types of discs like console DVD5 or unprotected single-track CDs.

The way things work may not make complete sense but that's because the submission system wasnt designed for the current dumping/ver. rules and it's not gonna change.

Forum posts are better anyway cause you get more scrutiny and can attach more data/notes (and logs) than a form would allow. Theoretically it produces better quality ringcodes and other metadata too. The downside is you may have to wait a few months/years for a mod to get around to processing them. Or they won't touch it for some reason and it stays there forever and you don't know why... oh well.

5 (edited by MigaMan 2016-08-11 22:54:21)

Re: Why are verifications checked manually?

[double post]

Re: Why are verifications checked manually?

Egen wrote:

What logs are these? For specific systems only? I don't remember ever being told to supply log information anywhere for verifications on this site.

For anything with CDDA and/or protections, otherwise, such "verifications" shouldn't have the green status.

MigaMan wrote:

Forum posts are better anyway cause you get more scrutiny and can attach more data/notes (and logs) than a form would allow. Theoretically it produces better quality ringcodes and other metadata too. The downside is you may have to wait a few months/years for a mod to get around to processing them. Or they won't touch it for some reason and it stays there forever and you don't know why... oh well.

Well, adding matching verifications is a rather boring task, if no new info is given (like, missing ringcodes for PSX discs or cuesheet corrections).

7 (edited by Egen 2016-08-12 07:45:03)

Re: Why are verifications checked manually?

Okay, so anything that requires a log, check manually. Everything else... just create a submission form for it man, dang, seriously. Everyone around here is frustrated with verifications that literally go back years—the users and the mods/admins. We're frustrated that we spent time contributing to the website and the work never got processed, meaning we essentially wasted our time, and mods/admins are frustrated that they have to hear about it. Well then change something, seriously, and this can't be that hard. It honestly can't be. It would look a heck of a lot like the new disc submission form for any given platform, and you enter the data in there and the system does all the comparing for you. If there's a discrepancy in any data, the submission is flagged and a mod/admin can then check it himself. In fact, this could also work for anything requiring a log: you just add a section for the log to be submitted/uploaded!

Redump is only as stagnant as it is for poor reasons, honestly. This can be remedied, and you wouldn't have to put nearly as much time into implementing the remedy as you would doing the verifications one-by-one throughout all the rest of time, which, obviously, is the point—to save time. I have something like 300+ verifications to add and I haven't done jack because I honestly think it will be a waste of time and that's sad, because then why are we here? Just to contribute new discs and not have any of the data verified because "it's boring"?

Come on guys, this problem has a solution.

Re: Why are verifications checked manually?

^^^  You think a few years is bad? (Well it kinda is but) SPS literally has unprocessed dumps from over a decade. now THAT'S a stagnant project!

So please don't let the frustration get to you, redump deserves some constructive criticism but we still have to accept that the methods are not likely to change and they will never add new mods (nitpicker's corner: outside of an established pool of people that come and go) so the bottlenecks and backlogs are just something we have to live with.

redump ain't perfect but they're still 10,000x more productive than SPS and infinitely more transparent than wrongrip.

I think it's o.k. to nudge them a little bit from time to time like we're doing right now as long as you can keep from getting frustrated if nothing changes.

*

RE: Your 300+ verifications, try making them EXCITING so they'll get processed fast! wink For dvds I think the best you can do to make them less boring is post pics/scans especially closeups of the ringcodez. For ps2 cd games submitting subdump logs will get the cuesheet marked green... excitement factor 1000! big_smile big_smile

*

F1ReB4LL wrote:
MigaMan wrote:

Forum posts are better anyway cause you get more scrutiny and can attach more data/notes (and logs) than a form would allow. Theoretically it produces better quality ringcodes and other metadata too. The downside is you may have to wait a few months/years for a mod to get around to processing them. Or they won't touch it for some reason and it stays there forever and you don't know why... oh well.

Well, adding matching verifications is a rather boring task, if no new info is given (like, missing ringcodes for PSX discs or cuesheet corrections).

I agree on this, I dumped many games but I never bothered posting verifications unless I had something important to add like ringcodes or different versions.

But when I wrote that last bit, I wasn't actually thinking about verifications but rather those 2 year old "new" dumps with 0 replies on the last few pages like

http://forum.redump.org/topic/14694/ibm … australia/
http://forum.redump.org/topic/14633/wii … rd-france/
http://forum.redump.org/topic/14064/ibm … on-v11425/
http://forum.redump.org/topic/14062/ibm … -new-disc/
http://forum.redump.org/topic/13965/ibm … bmissions/

Those are just a sample of the ones with 0 replies. They may be crap dumps but without an explanation who is to know? My biggest personal gripe isn't with the database software or slowness but the lack of feedback / education / training. I think that's what this old friend was getting at. I guess it doesn't matter any more now with new acct creation ended.

nitpicker's corner: Some of those dumps may not be "new" any more now that 2 years have passed. It's been so long a few have turned into verifications sad

*

ps egen wanna know how many active "mods" SPS has? 1.25 yikes

Re: Why are verifications checked manually?

If my data isn't going to get processed, I'm not submitting it, it's as simple as that. And if I'm not going to submit any data other than new discs that will never get verified by anyone else, then I wonder why I'm even here to a certain point. So I don't know what you mean by I have to "live with" it. Does that mean just kind of "exist" on the site and check it from time to time and nothing gets done? I don't know why I'd do that over suggest changes that can actually be implemented with not even that much difficulty and make this entire site move into the future. Why would I not do that and instead just go "oh waaah never gonna change oh well." That's what everyone else is doing and so we stay here. At least I'm trying to get something improved. If they don't improve it, whatever, but I also don't have to "live with" it or accept it, I can just not contribute, because right now not contributing is about the same thing as contributing.

*shrug* I don't know why the mods/admins reject ideas that can literally change their website for an entire positive. It's like they don't want to make any progress. I understand that verifications are boring and that people have lives, that why I'm trying to get something in place that will help remedy that a million times over. When there's little to no response about it, it looks like the mods/admins plain don't care about their own project, so then why should I?

Re: Why are verifications checked manually?

I agree that there should be a verification form just like there is a new disc form.
I think the problem is that there's no one to code that among the mods/admins or, if there is, they just don't have the time.

Re: Why are verifications checked manually?

I kind of don't believe that. Someone coded what we have now on this website. Is that person gone? Doesn't this project belong to all the same people that have been around forever? And if they have time to add new discs and a verification every now and then, they have time to code the forms, even if it's over the course of a long time and they do almost nothing else with the site. Sure the site slows down even more for a time, but the result is that it speeds up tremendously FOREVER, PERMANENTLY.

Re: Why are verifications checked manually?

^^^^

I used to be this way too but I've mellowed out a bit. It might seem like a big deal at this time but a few months / years from now and you too will be comfortably numb. And enough time will have passed for your verifications to get approved too, so win/win? (Granted a forms update would also be win/win but that could be years down the road too, if ever.)

Egen wrote:

If my data isn't going to get processed, I'm not submitting it, it's as simple as that.

Withholding dumps to try and force them to change their process / software is not a good strategy imho. iRobot is a nice guy, no need to indirectly rage at him and make veiled threats! yikes

I don't see any pending verification threads by you so this is all hypothetical talk any way. Try my subdump suggestion. They might be like "holy shit! green cue files! let's verify all the dumps!" wink

Oh, wait, I think I get it, this isn't really about your verifications getting approved in a timely manner so much as you wanting to improve the process for everyone, and you're using your discs as leverage right? In that case it's really up to iRobot and further arguing about hypotheticals with dumpers in a forum thread is just going to waste your time. That being the case I'll just be over here by the Wall not aggravating you further.



Which one's Pink?

Re: Why are verifications checked manually?

Egen wrote:

I kind of don't believe that. Someone coded what we have now on this website. Is that person gone? Doesn't this project belong to all the same people that have been around forever? And if they have time to add new discs and a verification every now and then, they have time to code the forms, even if it's over the course of a long time and they do almost nothing else with the site. Sure the site slows down even more for a time, but the result is that it speeds up tremendously FOREVER, PERMANENTLY.

Yes, someone coded this site before, but that was almost 10 years ago. The people who coded it may not be the same that add dumps to the db and could easily be gone now. This is speculation on my part, but, for sure, a lot can change in a person's life in a decade.

14 (edited by Egen 2016-08-16 04:45:20)

Re: Why are verifications checked manually?

MigaMan wrote:

Withholding dumps to try and force them to change their process
you're using your discs as leverage

What are you talking about? I'm not submitting dumps because I simply see no reason to go through all the effort of formatting 300+ of them into posts with bold and color markups when the posts will sit there and never get processed. Why do literally days' worth of work that will never see the light of day? It's ridiculous, there's absolutely zero incentive for me to go through all that work. And you're saying that if I triple my personal workload and do a bunch of scans and crap that will give the mods even MORE busywork to process per dump, THEN they will actually want to process the dumps? Not only does that make no sense at all, I didn't know it was my job to make the mods/admins interested in their own project, but I think I'll pass.

I appreciate that you want to look good by defending the mods/admins, but you're defending complacency with a broken system. It doesn't make you look good. There's no onus on you to say "I'm okay with the broken-ass way the project is right now", it's fine if you take a step back, look at it, and go "yeah, wow, this is ridiculous, virtually nothing ever gets done around here." Some people may say you're "complaining", others realize you're just pointing out the truth. But if you think you have some high ground to take and you want to tell me to put more work in so that not only will 300 verification posts will sit for years, but hours worth of scans along with them, then go ahead and make yourself look "good", but I have to tell you, it's not working; you don't look good doing it.

I don't know why you guys are rejecting the simple fact that this can be improved. It's like no one wants to step on anyone's toes and point out that the way things are being done isn't working, but it's not working. HUNDREDS of verifications, if not thousands, sit around endlessly. Why do we think it's chic to say "yeah you'll get used to it" rather than say "hey here's a legitimate idea to fix it"? Lame.


EDIT: By the way, this isn't about anyone being a "nice guy" or otherwise. iR0b0t is a perfectly fine fellow. F1ReB4LL is a perfectly fine fellow. Shout out to all the mods I missed too. I have nothing against you as individuals who have conducted themselves in whatever fashion around others on a message board; you've been courteous enough, and that's all well and good. This just isn't about person A or B being "nice", it's about getting something gosh-darn done around here. We ALL know that this system sucks. Come on, who around here doesn't know that? Show of hands right now. I don't mean this as any kind of attack on anybody and if it's taken that way, then that's a shame and you're missing the point, which is to get this site moving into the future.

I'm not making "threats" or whatever other crap you can call it, I'm giving you a simple truth: given the way the system works right now, there is no reason for me to go through literal DAYS of work on my personal time, which is not endless, to submit a bunch of stuff that could possibly NEVER be processed. Not "processed in three months" or "processed in two years", NEVER PROCESSED. It's completely possible and we all know it. How is it then worth anyone's time to submit anything other than a new disc here? This is all I'm saying. I want submissions to be processed, and not to make myself feel good, or to make blahblah person feel good, to contribute to Redump and get this great site rolling. Hasn't it been long enough that we all complain and nothing gets done? Let's change something, man.

Re: Why are verifications checked manually?

I didn't reject your idea at all. Actually, I got pretty mad once when I saw tons of new discs being added when there were a couple of my verifications just seating there completely ignored.

What I said was: it's possible that the people who coded the site at the very beginning are not around anymore and/or may not have the time to code a new functionality.

I actually don't know who were the people in charge of coding back then, but I don't think they were the same people that updated the database.

16 (edited by Egen 2016-08-16 13:59:27)

Re: Why are verifications checked manually?

Heh, sorry, I actually missed your post entirely, so none of that was directed towards you. "You guys" was poor general phrasing for anyone that is rejecting the ideas.

I see what you're saying, definitely. Even if the original people are gone though, it doesn't preclude the possibility of somebody on the current staff being able to code some new stuff for the website. Neither of us know anything regarding that, but it would surprise me a little that a website would be maintained without anyone on the staff with some coding knowledge.

Re: Why are verifications checked manually?

Egen wrote:

I kind of don't believe that. Someone coded what we have now on this website. Is that person gone?

Yes, gone. http://forum.redump.org/user/2/ -- no visits since 2010 and no db engine modifications from him since, dunno, 2007 or 2008. iR0b0t is only capable of doing little corrections, AFAIK. I only remember small form/field fixes from him for certain systems, adding some additional CUE fields handling and adding the total image CRC (with the V's help).

MigaMan wrote:

But when I wrote that last bit, I wasn't actually thinking about verifications but rather those 2 year old "new" dumps with 0 replies on the last few pages like

http://forum.redump.org/topic/14694/ibm … australia/
http://forum.redump.org/topic/14633/wii … rd-france/
http://forum.redump.org/topic/14064/ibm … on-v11425/
http://forum.redump.org/topic/14062/ibm … -new-disc/
http://forum.redump.org/topic/13965/ibm … bmissions/

Those are just a sample of the ones with 0 replies. They may be crap dumps but without an explanation who is to know? My biggest personal gripe isn't with the database software or slowness but the lack of feedback / education / training. I think that's what this old friend was getting at. I guess it doesn't matter any more now with new acct creation ended.

Wii dump was overlooked or something, dunno, it's added now. I don't know how to verify DC, GC/Wii and XBOX/360 dumps, anyway, they are in their own tricky format requiring specific drives and specific tools.

PC is a different story. Since all my ideas were totally ignored even by Dremora (while he was still an active coder), the PC section is a total and barely maintainable mess now. What should be done from the very beginning is to split the PC section to Win16, Win32, Win64, MS-DOS, OS/2, *nix, BeOS sections, etc. Only then it would be more or less possible to maintain the dumped/undumped lists and all these sections separatedly as well as properly adding all the stuff instead of putting everyting into a single bottomless bin. And then we have another problem - inability to assign more than 1 system per entry, that's a fundamental database issue, Dremora was going to code so-called "Redump 2" engine able to handle that (as well as subchannels, cover scans, bonus materials and all the other possible stuff), but abandoned the project. So, at the moment, only Jackal and Nexy are adding PC dumps from time to time picking the ones to add by their own logic. And yes, many PC titles are heavily protected with gazillions of different protections (compared to the consoles, where all the discs have the same format), so you can't just add these blindly.

Re: Why are verifications checked manually?

Thanks for the responses, Fireball. I didn't realize the situation was like this.

*
*

I'm not going to bother reading the other posts cause I'm done with this thread- too much scarey quotes yikes but here's a different idea I thought up.

*

Why don't one of you geeks make a little dotnet program that has forms needed for verification and saves the data you entered into a xml database.

Then you could zip up the database with any logs / scans and post it. The mods could then use the same program to review the data you entered.

It's not as good as full redump integration but let's be realistic here. And advanced versions could eventually be better integrated by scraping redump or open a link to a mod edit page.

Not a programmer? If everyone with a zillion verifications pitched in the price of a few games you could pay someone around here to do it at a cut rate.

...Or you can keep trying to threat^H^H^H^H^H^H cajole iRobot / the mods into doing it for free and on your schedule.

Re: Why are verifications checked manually?

Back the other days I was working on an (in my opinion) improved database structure, while the current database structure is based on DISC information, I thought it would be more clear to make it PACKET / REVISION based. The disc information would still remain same, just assigned in an other way to a packet. Which to be honest would make it a little bit difficult in case of discs with unknown release source.

Anyways, the whole project has become dead after a powerfailure on my machine and the entire hdd, which was encrypted, has gone unreadable, and it didn't just contain the redump web project, there was my entire project source collection i ever was working on! I think you only can imagine how bad it was for me.

I still have this hdd and made a clone of it in case i can restore the decryption key one day, but there is no real much hope.

The verification form is on my todo list, i know, it should make the entire verification process a lot faster and easier, there is however a big BUT...
a) the disc matrix still has to be reviewed manually, mainly because in most cases dumpers do not separate it into cells.
b) there can be typos basically in every submission.
c) the disc source, region, languages, and several other submission categories can be either wrong or unknown.
d) logs etc. have to be provided and be in a specifical default form.

If you can work out a flawless template how it should look like and be reliable i will gladly bring it into a code format.

Cheers smile

PX-760A (+30), PX-W4824TA (+98), GSA-H42L (+667), GDR-8164B (+102), SH-D162D (+6), SOHD-167T (+12)

20 (edited by Egen 2016-08-16 22:36:52)

Re: Why are verifications checked manually?

F1ReB4LL wrote:

Yes, gone. http://forum.redump.org/user/2/ -- no visits since 2010 and no db engine modifications from him since, dunno, 2007 or 2008.

And he was the only coder? Redump has not a single coder now? Well, then, it's no wonder we're in so much trouble here. Alright, well, I guess nevermind this then, at least until you get a new coder.


EDIT: Oh, and then iR0b0t ninja'd me big_smile

That SUCKS man. I'm sorry to hear that. But I'm so glad to know that somebody has considered the verification form! I wish I knew anything about website/net coding. I know some C++, some Java, and a hell of a lot of AHK (lol), but I wouldn't know what I'm doing around these systems and it's moot because I'm new here and rightly wouldn't be considered for the position. But thanks for your responses you guys, I just love this website and want to see it make progress.

I'll seriously mull a form template over. Probably harder to come up with one than I think, but if we got one up even for one or two consoles it would be a huge improvement. I think nobody should bother with PC, since it's a gigantic mess, but at least some of the older PlayStation consoles should be possible.

Re: Why are verifications checked manually?

iR0b0t wrote:

Back the other days I was working on an (in my opinion) improved database structure, while the current database structure is based on DISC information, I thought it would be more clear to make it PACKET / REVISION based. The disc information would still remain same, just assigned in an other way to a packet. Which to be honest would make it a little bit difficult in case of discs with unknown release source.

Well, it was not yours, but Dremora's base idea for Redump 2 to show the games as "packages". So you would add CD1, CD2 and bonus Audio CD dumps into 1 "package" entry as well as all the covers and whatever else. Though, it was unclear what to do with all the possible ring variations (like those Saturn's 1M/2M/3Ms).

Re: Why are verifications checked manually?

I didn't really know what Dremora was going to do. I probably just had a similar thought, because i didn't like the disc based structure, it does not preserve the real release contents, everything is more or less mixed into a single entry right now.

ALL binary variations would simply be linked to the same package (as binary variations), as long as the package has the same visual contents, there is not much else to do.

If a package would contain a disc with different label print, misprint or whatever it would qualify for a new package, you know what i mean?

Though, this preservation methode is way to complex. One cannot really confirm a package contents if it was an unsealed second hand purchase, resellers do not bother what they sell as long as they get rid of stuff, theirfore they mix everything together to get a "complete" set.

PX-760A (+30), PX-W4824TA (+98), GSA-H42L (+667), GDR-8164B (+102), SH-D162D (+6), SOHD-167T (+12)

Re: Why are verifications checked manually?

Well, my own idea was to list all the "main" titles first, each title would have "releases" (distinguishable by serial or barcode, 1 serial = 1 release, even if different packages exist) and each release would have subreleases with possible alt rings/revisions/scan variations.

ALL binary variations would simply be linked to the same package (as binary variations), as long as the package has the same visual contents, there is not much else to do.

CD scans would differ due to different ringcodes. One of the main ideas was to make the ringcode scans/photos mandatory.

And after adding all the scans we could make CAPS/SPS-style xml dats (gameinfo.xml with all the scans and binaries and their checksums). That would be the real preservation. We even had D4ve guy who was going to make us a perfect scanning convention and guide, but disappeared.