Segmented downloading techniques

[Excelsium 0]

I think the most accurate version of my arguments come across in post #157 onwards.

For ease of writing this I have defined bug to mean several things.

- "problem, issue, lack of technique, gap in technique"

- Solutions to prevent waste caused by gaps/lack of techniques in the segmented download technique.

- To inform the user about the problem, not fix the problem.

- To automate the detection of failing downloads so the user dosn't have to.

New Rewording

For Big Muscle: Helpful rewording.. of the entire thread perhaps... although If you need full detail on the waste that occurs to convince you the bugs need fixing, check the rest of this post.

Issues with the segmenting download technique

1. Redownloading the same chunk(s) ad infinitum of a download with a corrupted temporary file that is beyond repair by itself. - It should stop the process and inform the user of which temporary files are corrupted in this case or it should at least inform the user with a dialog box.

2. Its corruption handling technique to repair corrupted temporary files does not have a high enough success rate, other P2P applications have a much higher, near perfect or perfect success rate.

Issue 2. is a/the cause of Issue 1.

I once described these bugs as "download same chunk ad infinitum bug caused by inability to correct corrupted temporary file bug".

Description of the issues

1. A Corrupt temporary file for a download queue item exists on the users machine.

2. In some cases StrongDC++s segmenting download technique fails to repair the corrupt temporary file indefinitely. (Other P2P clients never fail at this or fail at a much lower %)

2.1. The segmenting download technique downloads the same chunk(s) of the file ad infinitum.*1* *2*

*1*. Ad infinitum meaning: Its been running for days: It redownloads the same chunk(s) 100s-1000s of times at high speed (10Mbit connection) from 1-10 peers at the same time (Max is 10).

*2*. The user is not informed which temporary file is corrupted.

2.1. Should inform the user if the corrupted temporary file cannot be repaired by itself.

2.1.2. Should pause the download if the corrupted temporary file cannot be repaired by itself.

2.1.3. Should detect if the download cannot be repaired by itself (e.g. redownloaded the same chunk(s) too many times) or some other technique.

2.2. "Informing feature" Only needs to be implemented if the repair corrupt temporary file technique is not fixed 100%. If it cannot be perfected then the informing feature is required for the few remaning % where the temporary file cannot be repaired by itself.

I have not tested leaving it (Download with corrupted temporary file) running for more than a week because of the bandwidth waste.

Addition:

One option instead of just pausing the download after a set number of chunk redownloads:

- If the corruption handling technique fails to repair a corrupt temporary file it should delete the download from the queue and re add it to the queue automatically. (Starts over with a new temporary file)

- The user could be given this option when the corruption is detected or set it to do this automatically.

- Other users have experienced these bugs. (see posts ITT)

- Other users continue to experience these bugs. (see posts ITT)

*ITT = in this thread

About this thread

New thread was moved into the old thread without my permission by forum admin, I'm in the process of cleaning it up.

Cleaning done: you can skip everything except this opening post up until page 3 after the post marked "New Thread begins after this post."

Title of thread censored: Real title:

Bugs found in the download techniques packaged in ApexDC++/StrongDC++

Previous title: [Request] Corruption detection technique that informs the user of corrupt temporary files of download queue items. - this would be an interim technique if a fully automated technique is not implemented (If the bugs in the download techniques are not fixed).

Long version of this opening post

The Request:

1. Fix the bugs so that the user dosn't have to be informed as is the behaviour of other p2p clients.

Last resort: 2. Corruption detection technique that informs the user of corrupt temporary files of download queue items.

There are bug(s) in the way StrongDC++s/ApexDC++s segmenting download techniques corruption handling subtechnique handles corruption in temporary files of download queue items.

There are also bugs/lack of corruption handling techniques in the way the normal download technique handles corruption in temporary files of download queue items.

Hence there are several unfixed bugs and lack of corruption handling techniques in the packages that are StrongDC++/ApexDC++

Facts

Please post corrections or relevant additions to these facts if required - I will post them here.

Segmenting download technique

- Does not complete a download if the local temporary file for that download is corrupt (Good)

- It detects the corruption but does not inform the user of which file the corruption was detected and does not pause the download if the corruption in the local temporary file is beyond repair.(Bad)

- It downloads the same chunk(s) of a download with a corrupted temporary file that is beyond repair (by itself)* ad infinitum (Bad)

- It cannot repair the corrupted temporary file because its technique to do so is bugged. (Bad)

- *1*Time and resource waste occurs because of lengthy manual user detection *2*.(Bad)

*1*. Hence the reason for this feature request - I am in essence asking for a technique to automate what is currently a manual process which involves guessing, trial and error.

*2*. Continually observing the transfers window and making a *Guess* which download(s) is/are not going to complete successfully due to a corrupted temporary file for that download.

2.1. The obvious bandwidth waste caused by downloading the same chunk(s) of a file ad infinitum until the user has engaged in and completed the manual corruption detection process.

2.2. StrongDC++/ApexDC++ crashes after re downloading the same chunk(s) of a download with a corrupted temporary file a random number of times. * This is the only bug so far a developer (Big Muscle in this case) has said they will fix - This fix has not yet been released.

2.2. Note: The cause(s) of the crash bug is/are unproven - the only bug I am unsure of.

Standard legacy download technique

- Completes the download regardless of a corrupt temporary file(Bad)

- Perhaps it dosn't detect the corruption in the local temporary file at all, if it does - it ignores it and completes the download.

Cause of the corrupted temporary files

Unidentified defect in the users machine:

Defect in the users machine possibilities: Hard drive, Filesystem, Windows XP, other software, other hardware.

I will add defect possibilities as they are sent in.

Corruption on the users machine is an inherently unfixable problem for all users for the forseeable future

The main variable being the percentage of corrupted temporary files.

- Hardware and software in/on the users machine can and will fail, I believe these are called

"Error rates" some RAM has ECC for example.

"1 error per 1billion operations" along those lines.

We require the tools of information to prevent said consequences quickly and effectively. (See my reply below to Big Muscles statement: "User should fix his problem on his own".)

Personal experience, opinion

I use the legacy download technique - even If I have a corrupt file - that is going to be obvious to me when I open or verify it - In my opinion It wastes much less time and resources to redownload the file and is also much less annoying compared to the lengthy manual user detection process during the download. Also even without the speed of segmenting I still prefer this option because of the cost in time and resources of manually detecting even a few corrupted temporary files - in my case I get about 2 corrupted temporary files for every 60 files that I download, although I may have been "unlucky" recently and the overall number is perhaps 2 in 150 or more files since 2004.

My relevant replies and questions left unanswered from the last thread:

that's a good solution. the number of detected corruptions would have to be defined, though; 50 is a bit too much...

codewise, i think this would need a counter in the FileChunksInfo class if i'm not mistaken.

however, this is a workaround and i still prefer the client doing nothing automatically and letting the user cancel the download if he thinks he has to.

"I still prefer the client doing nothing automatically and letting the user cancel the download if he thinks he has to."

Doing nothing automatically - my translation: Not having corruption handling techniques for corrupt data on the local machine.

One of these automated techniques is to detect which files the corruption is located and inform the user of these corrupted files.

You are saying the user should not be informed of what files are corrupted, by saying it should do nothing automatically.

Rediculous amount of waste can occur in the mean time during the lengthy manual process of user detection... gone into full detail already.

So you are saying that you think this large amount of waste is acceptable. If these really are your opinions, please reply and say it more explicitly in no uncertain terms.

I'd also be interested to know if Big Muscle or anyone else reading this thinks that this waste is acceptable, clearly I think its unacceptable and I'm fairly sure the vast majority (users/devs) involved in P2P filesharing as a whole think its unacceptable but if you think its acceptable please come out and say so and state the reasons why you think it is acceptable.

Other apps will at least detect local files that are corrupted and inform the user of these corrupted files, so that the user can do something about it and quickly.

now do you understand?

better, corruption in the file "was created" when it had already been saved on your HDD and connection to the remote user had been closed. It has nothing to do with network communicating!

how many users has corrupted HDD? maybe one of 1000 (maybe of more, just a guess).

I already understood exactly what you just said and agree with you.

I am asking for corruption handling techniques for corrupt data on the receiving users machine to prevent wastage (full detail-previous posts) and questions at the end of this post.

So My theory of interacting with certain peers causes the local temp file to be corrupted is extremly unlikely however possible (hackers or ???) - I was just trying to list all possibilities however likely or unlikely so we can get to the bottom of this, which we have.

User should fix the problem on his own.

I agree but with an addition in this case -

[Big Muscle 1]

Such workarounds will never be implemented - just its not the workaround, because the same problem can appear also for non-segmented downloading:

a) but you won't see any message about corruption and file will leave corrupted on your HDD

it will appear in another time than for segmented downloading. Or better, it will appear when your PC/OS wants to access to the corrupted area.

User should fix his problem on its own.

[Zlobomir 3]

Yeah, but he pretends that when file is downloaded via http or torrents, all is OK in terms of accessing... erm? Or he is using AVIPreview?

[Excelsium 0]

Yeah, but he pretends that when file is downloaded via http or torrents, all is OK in terms of accessing... erm? Or he is using AVIPreview?

I check the TTH for the copy downloaded (downloaded completly without any errors) from dc++ peers (with segmenting disabled).

and the copy downloaded with Bittorrent. - they match

I've viewed both copies in full of a few such cases, and scanned quickly through the rest - I've tested the TTH in all cases and all have the same TTH, but why would there be a problem? they all have the same TTH.

I use PowerDVD 7 with the default install of codecs from divx.com to view the files, no other codec packages etc are installed.

I am happy with ApexDC++/strongdc running without segmenting mode - I have no problems whatsoever, but it would be a nice extra to have workable segmenting.

(reply to post in bug+crash forum)

I just can't understand why you can't understand that you have bug in your PC. There can't be any switching, because it's useless. Maybe file ABC gets corrupted in segmented downloading but it's ok for normal downloading. But file RKX will be corrupted only for normal downloading, but you won't see any message informing about corruption. Same thing will happen when you will copy file ČŘ

[Big Muscle 1]

Why you can't understand me?

So again:

Today you download file using segmented downloading and it gets corrupted.

Two days after you will download another file using normal downloading and it gets corrupted too.

Corruption doesn't depend whether you use normal or segmented downloading. It depends on actual memory/HDD position (according to where the bug in your PC is)

[Excelsium 0]

Why you can't understand me?

So again:

Today you download file using segmented downloading and it gets corrupted.

Two days after you will download another file using normal downloading and it gets corrupted too.

Corruption doesn't depend whether you use normal or segmented downloading. It depends on actual memory/HDD position (according to where the bug in your PC is)

I've never had a problem with normal downloading .

And its always the same files with the same TTH that fail in segmenting mode even starting from scratch. deleting temp file etc. the other 95+% of files dont have this problem in segmenting mode.

It just takes a few other people to try downloading the same TTH in segmenting mode to prove it either way.

[Big Muscle 1]

I've never had a problem with normal downloading .

i doubt that you had and you will always check TTH of downloaded files. Normal way just doesn't show the message and normally saves the corrupted file to the disk.

[Excelsium 0]

i doubt that you had and you will always check TTH of downloaded files. Normal way just doesn't show the message and normally saves the corrupted file to the disk.

Ok thats a misswording on my part. the facts below and above still stand.

I'd like some detail on how it would be a HDD/memory bug that causes the exact files with same TTH to fail over and over in segmenting mode when starting from scratch even starting with a default clean install of apex/strong.

Again - downloading these files in normal mode complete fine and are verified (checked and matched TTH) against a copy another source (bittorrent) and are fine in extensive subjective testing. So I think that destroys the theory that these files are corrupt and the segmenting technique is detecting corruption in the peers (all peers) copies and preventing completion.

[Big Muscle 1]

OS selects where the file be saved and i don't know how does it decided it.

I just will try to fix the crash and you will see whether it will correct your problems. That's all I can do.

[Excelsium 0]

OS selects where the file be saved and i don't know how does it decided it.

I just will try to fix the crash and you will see whether it will correct your problems. That's all I can do.

Yea it just seems extremly bizzare, unbelievable (For it to be a bug in hardware/os/etc) that it would happen repeatedly on the same files with the same TTH, but succeed on all other files (in Segmenting mode)

I understand your saying its a bug somewhere in my machine (and my previous machines) but for it to be that specific seems unbelievable.

My thinking is that bugs like that happen randomly on random files

Its clear for reasons stated that this is an incredibly specific precise "bug" with no apparent randomness to it.

Theories: (maybe posted some of these earlier).

1. StrongDC segmenting download technique*1 is detecting that peers copies or chunks are corrupt*2 when they clearly are not*3 (False positives) and is preventing the completion of that chunk and therefore the completion of the download.

*1. Has and is using a corruption detect technique of its own

*2. A small percentage of files, perhaps a tiny percentage of the overall files shared using DC++

*3. As proven by testing

1.2. Something else in StrongDCs technique is causing false positives in an extremly specific manner.

1.3. StrongDC is corrupting specific files, detects its own corruption and prevents completion.

Theories of bugs in my machine:

1. An amazingly specific bug ???

2. ??

add more later

Reply to post in crash+bug forum.

QUOTE(Zlobomir @ Mar 30 2007, 05:58 PM) *

Again I have to double post, excuse me BM. But what about the file in question is really ok, runs perfectly, etc., after downloading in some other way. And Excelsium, please send me again some small files for tests, or even better, attach them here.

Yes, you are true. File can have correct TTH and can be 100% if it has been downloaded in other way than segmented.

But again - it's only this situation.

Let's talk there is some HDD error (just guess, it can be also memory error etc.). This error appears on clusters G and U.

Current situation, user downloads file using segmented downloading and Strong/Apex tries to save them into cluster G. So what happens? Yes, file will be corrupted. Normal downloading saves the file to cluster A, so file will be OK.

Another situation, user downloads file using segmented downloading and it will be saved to cluster J. So no file is OK! But later he wants to download another file using normal downloading and OS selects that it will be saved to cluster U. So where are we now??? File will be corrupted using normal downloading.

I hope, that now the problem is understable.

But I also realized another situation. The question is, when the corruption occurs. Before crash or after crash??? But time can't be detected, because Apex/Strong will report the corruption when it's detected, but not when it occurs. what I want to say:

a) corruption occurs before crash -> so when it is detected, Apex/Strong wants to fix it and it crashes during fixing. This means some HW bug in the PC.

corruption occurs after crash -> Apex/Strong crashes and file gets corrupted due to the crash (because OS doesn't flush to disk).

Both situations mean that the crash must be fixed. But in A) it won't fix corruption messages.

I know that now you will say "it's really situation ", but you can't say it, because it can't be decided

I think Its a bit of situation A and a bit of something else.

What I mean is. I think there is a problem with the technique causing false positives or real corruption due to itself (a bug somewhere in the technique of Strongdc) -

And then it crashes after the false or real corruption that it has falsely detected or caused.

See the list of thories above, this is my current one.

On emulation: I don't use it for those hubs or any others - If I come accross a hub that requires it to connect, I don't use that hub because of some issues I heard of.

Users downloading from when it crashed?:

1. The crashes are not very frequent since I usually spot the problem and pause the file before it crashes.

2. I have not been logging the users it was downloading from when it crashes.

I'm probably not going to do any more testing with segmenting mode until a new version of strong/apex is released or someone has suggested configuration tweaks I have not tested.

On my machine:

I'm still thinking about the hard drive clusters and will add to my reply if needed.

But for now heres what I know.

1. My hdd has no problems: defragmented, 0 bad sectors/problems reported by S.M.A.R.T, 0 problems reported by windows tools

2. How can that problem affect a specific set of files with same TTH indefinetly?, files are written in whatever free space (clusters) is available.. which is random.

All the other details have already been posted I'm sure. for example

"Perfect copies of files can be downloaded using normal mode but cannot be completely downloaded using segmenting mode for the same specific files with same TTH"

Again 95+% succeed in segmenting mode, the exact same (No ramdomness*) 1-5% have the download not completing problem in Segmenting mode.

*Randomness characterizes the data layout on a disk, and RAM aswell.

But this is clearly a precise bug, maybe you have some details on how a hw or sw defect or perhaps configuration setting outside of StrongDC/apex could result in this very precise bug affecting a set of files with the same TTH indefinetly?

Another reason why this workaround should be implemented:

(At the very least the client should pause the download and warn the user to prevent slot and bandwidth wastage, and save the user time identifying the corrupt files)

When the crash is fixed the client will be left to continue to download the same chunk(s) over and over again until the user returns and spots it, of course sometimes they wont spot it. - Wasting slots and bandwidth in the network. For me thats 18hrs a day of unsupervised waste.. except weekends.

Of course I'll leave segmenting disabled but what about others who dont know about these issues yet? I know I've been experiencing it for a long time with various machines but only just taken action to deal with it - If I had not taken action I would waste many more gigabytes of bandwidth without the crash fix but with the crash fix I would waste 2x or 3x+ more.

I've a 10mbit connection, I know I've easily wasted 100s of GB's of my own and others bandwidth because of my ignorance of the only known existing workaround (segmenting set to disabled), for example a good percentage of that on popular files with 100+ sources - always maxing my connection.

I started using RevConnect in 2004 - I've been wasting slots and bandwidth for all that time, about 2 months ago I moved to StrongDC>Apex (easier on the eyes) because of these issues but was disappointed to find they are in these clients aswell, so I began my quest for a workaround here, luckily I found one not ideal.. but it works (Disabling segmenting).

The potential that this kind of waste can happen has to be stopped and of course the ongoing waste by people who have the issue, are ignorant of it and have not disabled segmenting - via an automated method.

Options:

1. Detect corruption and pause the file, put error in the error column.

2. Detect corruption and switch technique for the file.

[Big Muscle 1]

I tried to download all the files you have problem with. And I was in hubs you wrote. I've dowloaded them manytimes and always no crash, no corruption, TTH of downloaded files matched with the original.

[Excelsium 0]

I tried to download all the files you have problem with. And I was in hubs you wrote. I've dowloaded them manytimes and always no crash, no corruption, TTH of downloaded files matched with the original.

Possibilities:

- It only affects users with this "bug" either on the sending or the recieving side - which side is undertermined.

- If its not on the recieving side - The problem with recieving the files is not indefinite - related to which users are online sharing the files - a few of those users may have this problem sending the file.

either way bandwidth and slot wastage is happening. - peers sending and recieving the same chunk(s) of files ad infinitum. the wastage potential will increase if the crash fix you release does not address this (no crash = chunk(s) continue their infinite transmission.)

It would be irresponsible to let this wastage continue and soon perhaps increase by an order of magnitude.

IIRC RevConnect did not crash at all or nowhere near as often, the waste occurs in that client which led me to testing and switching to StrongDC++ and then ApexDC++ as said.

I will be testing segmenting again in the next release - If I encounter this issue again I will post a FRAPS/whatever video of the bandwidth+slots being wasted.

[Big Muscle 1]

it would be better if you had build Apex/Strong in VS as a Debug build and completely watch whole process - if it hits some assertion, what it writes in output window etc...

[Excelsium 0]

it would be better if you had build Apex/Strong in VS as a Debug build and completely watch whole process - if it hits some assertion, what it writes in output window etc...

I will do that if I can otherwise I hope video evidence is enough for you or another developer to take action and implement this or another simple preventative technique - for example: Pausing the download of the affected file when corruption is detected - how hard can it be .

Some more ideas..

- If same chunk fails from same user e.g. 10 times, remove that user for that file

- If same chunk fails 10 times from e.g. 5 different users pause the file, warn the user and offer to change download mode to normal for that file.

I have moved my final summary to my opening post in this thread.

Addition to final summary.

I tried to download all the files you have problem with. And I was in hubs you wrote. I've dowloaded them manytimes and always no crash, no corruption, TTH of downloaded files matched with the original.

I have tested those files again in segmenting mode Big Muscle.. This time they worked without any problems - the list of users online sharing those files has changed a lot.

My theory which I didn't share earlier because I thought it was not probable but has proven to be quite probable:

- The problem is on the transmission side, therefore the problem is not indefinite but lasts only as long as the problem transmitting user(s) is/are online.

- A transmitting user is sharing a corrupt copy.

- The transmitting users copy is not corrupt but somehow the transmission of the chunk is being corrupted ad infinitum, a problem with thier pc or client.

- The version of the transmitting users client has problems interacting properly with StrongDC++, causing said problem.

Also an observation I forgot to mention:

When I tried to redownload the problem files - the percentage of completion that the file would encounter the repeated chunk download failure was somewhat random, sometimes at the same percentage twice or more in a row however, which left me with another "hunch" that it was transmission side.

Hence my current thinking now is that this a problem fundamentally outside of the control of the developers of StrongDC++/ApexDC++ due to it being on the transmitting side therefore an implementation of problem handling is required to deal with the situation - problem handling is of course within the control of StrongDC++/ApexDC++ developers.

The problem of corruption occuring is probably never going to disappear one reason being because of the uncontrolled and diverse set of clients in the network. An individual or group needs to implement the proper "problem handling" in DC++ clients as has been done with other P2P applications such as Bittorrent.

Why? To prevent resource and time wastage, thereby greatly enhancing the users experience, productivity and satisfaction with the product, that is surely one of the ongoing goals for you devs.

Some obvious examples of problem handling.. spam filters, whitelists, prisons for criminals, refrigerators, food containers. Are you catching my drift yet Big Muscle?

If you encounter a problem that fundamentally cannot be fixed

"If you leave milk out long enough it sours" - you must handle it.

Funny analogy (needs some work ) :

If someone spams you with junk mail (the same chunk of the file ad infinitum) through the doorflap in your back door one way of "problem handling" would be to weld the doorflap shut in your back door. Or in this case - pause the file (one of your door flaps).

Of course you cannot fix the problem by assassinating the post men (fixing all the relevant bugs in the transmitters client or machine) - You can only handle the problem.

Another theory:

- Normal downloading versus Segmenting

- Segmenting will carry on downloading from as many users as it can so it dosn't have a chance to bypass the online user(s) whos sending you the corrupt bits - clearly it does not bypass that/those user(s) because it downloads the same chunk over and over again from them and everyone else at the same time - making it impossible to tell which one is causing the problem just by looking at the transfers window. If the user(s) with the corrupt bits is online your download is doomed (not going to complete) until they are offline.

Segmenting technique allows corrupted bits to enter your copy (Is this a possibility?)

Perhaps if the user(s) with corruption were online at any stage during the download, your download is doomed until you delete and start over and the user(s) with corruption is/are offline.

More detailed version of doomed files (with segmenting) theory - I think this "bits of permanent doom" theory is not likely unless of course theres a problem with the way the segmenting technique handles corrupt bits, maybe it allows corrupt bits into your copy or - allows corrupt bits in yes but detects it and prevents completion forever - the normal mode blocks corrupt bits from entering and all is good.

Currently the segmenting mode wastes bandwidth (fails to complete download) until the problem users go offline but perhaps they have already damaged parts of your copy and it doomes your file forever (wont complete) until the following conditions have been met:

- You start download from scratch, erase temp file

- Users with corruption cannot be online at any time during the download proccess (you cannot recieve any bits from them) or it will fail.

- Normal downloading is from one user at a time - if it detects a problem and fails just once it will try another user - well it will try all users, maybe it will come to the same problem user again but not everytime - therefore the repeated chunk download problem is near non existent - it will only happen a few times when it gets a slot from that/those problem user(s).

Clearly the segmented downloading technique requires more built in intelligence to work 100% of the time.

To bypass the problem users it has to do something like this:

Detect that a chunk has been redownloaded 10 times.

Switch to single source mode until the chunk has been completed successfully from a user with a clean copy.

Switch back to segmenting mode.

But Segmenting mode fails again e.g. 5 times in total.. well perhaps a large portion or the entire file the problem user(s) is/are sharing is corrupt.

Switch to normal mode until the download is finished.

Or switch back to segmenting mode at arbitary intervals to check if the problem user(s) are offline yet.

The normal download process is much simpler - it has problem handling that works - it does pretty much "Detection corruption and switch (to a different user) - it bypasses the problem user(s) and the download completes successfully".

I think this is my most likely theory to date, also my best example of a problem handling technique.

Why you can't understand me?

So again:

Today you download file using segmented downloading and it gets corrupted.

Two days after you will download another file using normal downloading and it gets corrupted too.

Corruption doesn't depend whether you use normal or segmented downloading. It depends on actual memory/HDD position (according to where the bug in your PC is)

We have now established that the corruption is located in a portion of the users copies (probably just 1 or 2 bad apples) - and that corruption is being transmitted to the reciever -

Preventing the download completing temporarily until they are offline.

(In a percentage of cases, part of the rest of the percentage could be caused by corruption that happened on the recieving side.)

The certainty of the transmission side problem comes from testing the same files many times and failing then suddenly succeeding many times whilst noticing the online users sharing those files has changed a lot.

Such workarounds will never be implemented - just its not the workaround, because the same problem can appear also for non-segmented downloading:

a) but you won't see any message about corruption and file will leave corrupted on your HDD

it will appear in another time than for segmented downloading. Or better, it will appear when your PC/OS wants to access to the corrupted area.

User should fix his problem on its own.

User should fix his problem on its own. - Just finished answering this.

Such workarounds will never be implemented.- This too (same thing really)... but again in very short form.

These are problem handling techniques to deal with a problem that is fundamentally unfixable.

This problem (corruption) has had good complete (or much closer to complete) packages of problem handling techniques implemented elsewhere such as Bittorrent.

DC++ Segmenting clients don't have the complete (further away from completion) package of good problem handling techniques for corruption built for them yet, which of course results in lots of bandwidth and time wastage.

Out of all the segmenting tools I've used (emule,kademlia,bittorrent,others less popular,http) Only in the DC++ landscape have I ever seen such blatent continuous waste resulting from a lack of problem handling techniques - afaik DC++ is the only place I've seen any waste at all, of course there is waste in other tools but its small enough that its not visible to the user, it dosn't affect the user in terms of bandwidth - and in terms of time - at least not in such an offensive way by making them themselves track down the corrupted files through manual labour observing transfers, it is made very obvious in other tools if there is a problem with a particular file.. which is hardly ever.. since I cant even remember an instance off hand... thats the way it should be.

I view DC++ as a single segment mode only P2P application - Its a Great (really great) filesharing tool in this mode because of the features, developments such as StrongDC++,ApexDC++ and the communities that have formed around it. I believe for single segment mode it has a complete set of techniques with little waste (detects corrupt chunks, redownloads from other user(s) bypassing the problem users) - But it turns into an unintelligent wasteful Beast the second you turn on segmenting mode (no ability to bypass problem users)...

- To me at least the existing segmenting techniques are meaningless until the last mile of corruption handling techniques have been built... Close, but no cigar.

[Big Muscle 1]

Too long text to read it. I don't have time for it.

I still don't understand what you are trying to solve. It has nothing to do whether it is normal or segmented downloading. I said it many times.

Segmented downloading doesn't allow corruption, that's the normal downloading which allows corruption. Because segmented D/Ling warns and redownload corrupted blocks, but normal downloading doesn't.

I think I haven't told you what "Corruption detected" message means, so I will describe it:

a) you are downloading some block of data

it will be stored to Windows internal buffer

c) Strong/Apex will send a command "write it to disk" to your OS.

d) Then it verifies whether this data is OK

d1) if it isn't, it will write "TTH inconsistency", remove user from queue and tries to download this block again. (jump to a)

d2) if it is OK, it will continue downloading next block (jump to a)

e) when some block is being resumed (not from its start position), it will reread the already downloaded part of chunk (which has been downloaded earlier using steps a,b,c,d,d1,d2)

f) this read part of block is completely verified

g) if it is OK, it will continue downloading of this chunk (jump to a)

h) if it is corrupted, it will display "Corruption detected" and completely redownload whole chunk.

So if you don't understand it, it simply means that "Corruption detected" messages informs you about some corruption which occured after data was saved to disk. It has been already verified before saving to disk and it was OK else you couldn't see this message.

[Big Muscle 1]

Appendix: Windows itself don't have any mechanism against the corruption during file copying, so I don't understand why any other program should have it.

[Excelsium 0]

(Important post)

Reply to last post:

An additional corruption handling technique could:

- Detect that an attempt at downloading a chunk of a file has failed 10 times -

Warn the user that this has happened and what file it happened in.

- This is the most basic first step of problem handling

- The user is left uninformed, therefore said resource and time waste occurs.

- Additions to this first step of corruption handling are also obvious and should be added.

Segmenting mode cannot bypass users sharing corrupt bits until they are all offline - Won't complete download until all users sharing corrupts bits are offline. (full detail in other post)

Normal mode can bypass users sharing corrupt bits simply by switching to a user that is not transmitting corrupt bits. - Download completes sucessfully when the users sharing corrupt bits are online by bypassing them.

(full detail in other posts)

Appendix: Windows itself don't have any mechanism against the corruption during file copying, so I don't understand why any other program should have it.

-To prevent resource and time waste.

- Other P2P apps have more complete corruption handling techniques that don't result in a large amount of waste.

- Full detail in my previous posts.

[poy 0]

h) if it is corrupted, it will display "Corruption detected" and completely redownload whole chunk.

this seems to be what causes the segment to be downloaded ad infinitum on Excelsium's computer. one's suggestion would be to act the same way the client acts in the case of a "TTH inconsistency" (remove the user from the queue), but i'd be against this solution: i find the current corruption handling pretty nifty.

that's why i agree with Big Muscle: the user should fix this on his own; the problem is on his computer. and if he can't fix it, he can cancel the download himself...

[Excelsium 0]

this seems to be what causes the segment to be downloaded ad infinitum on Excelsium's computer. one's suggestion would be to act the same way the client acts in the case of a "TTH inconsistency" (remove the user from the queue), but i'd be against this solution: i find the current corruption handling pretty nifty.

that's why i agree with Big Muscle: the user should fix this on his own; the problem is on his computer. and if he can't fix it, he can cancel the download himself...

- The problem is not on my computer for a big percentage of the time or whole percentage or near whole percentage (corrupt bits being transmitted to reciever).

- In the mean time of the user being unaware of the fundamentally Unfixable problem

(corrupt bits being transmitted to reciever) Said resource and time waste occurs.

- see my replies to Big Muscles posts.

I've already fully explained in previous posts why the problem (Corruption) Is not the responsibility of any single user, And that corruption handling techniques are more complete in other p2p apps.

Poy's suggestion: one's suggestion would be to act the same way the client acts in the case of a "TTH inconsistency" (remove the user from the queue), but i'd be against this solution

I'd say; remove the user from the queue for that file only.

This suggestion will work in the case of users transmitting corrupted bits if:

- The segementing download technique can detect which user is sending the corrupt bits.

- For the times when the problem is on the recievers local machine the download should be automatically paused and the user warned (The most basic essential problem handling technique as described in full already)

(Full detail in earlier posts)

[Big Muscle 1]

- The segementing download technique can detect which user is sending the corrupt bits.

please, read my post above again. Or maybe I will try to explain it better.

If you "corruption detected" message, it means that file was corrupted after saving to disk. So it was correctly downloaded and correctly sent to your operating system.

If some remote user sends you corrupted file, it will be caught by "TTH inconsistency" and user will be removed from the queue. "Corruption detected" message has nothing to do with file transmitting from user to you.

Or you could try to tell the solution which user should be removed in this situation.

You download file from user A and everything is downloaded correctly (TTH leaf matches), so it is saved to disk. Then you want to download next chunk of same file from user B. Strong/Apex reread the already downloaded part of file from disk and detects that it's corrupted. Now tell me which user should be removed:

user A ? but why? data from him has been verified and they were OK.

user B ? but why? we didn't download anything from him yet.

[poy 0]

- For the times when the problem is on the recievers local machine the download should be automatically paused and the user warned (The most basic essential problem handling technique as described in full already)

this is what i meant with "one's suggestion". i'm against this solution since i think the current technique (go back to the beginning of the block and re-download it) is very useful for corruption handling in most cases.

now, in your (very rare) case, i believe you pausing/removing said download by yourself would be better than an automatic thing that would cancel every downloads which could have been easily corrected by the current technique.

[Excelsium 0]

reply to Big Muscle

So for this particular corruption problem "corruption detected at position":

The corruption happens on the recieving users machine, because of the following reasons:

- A problem in the users machine causes corruption.

- The corruption on the local machine is caused by an unidentified problem with the interaction with certain peers on the network.

Again all scenarios require at least basic problem handling to prevent said wastage - I've gone into full detail of this waste and how to prevent it in earlier posts.

this is what i meant with "one's suggestion". i'm against this solution since i think the current technique (go back to the beginning of the block and re-download it) is very useful for corruption handling in most cases.

now, in your (very rare) case, i believe you pausing/removing said download by yourself would be better than an automatic thing that would cancel every downloads which could have been easily corrected by the current technique.

The automated pausing would only occur for this particular dead end corruption problem (the corruption is in the local machines unfinished copy, and cannot be fixed for some reason) "corruption detected at position".

Even more detail on automated pausing:

- Detect corruption detection messages in system log.

- Detect files that have been downloading the same chunk.. say 50 times.

- Correlate the 2 and pause the file.

Zlobomir just posted something but deleted it:

"Imho this could be caused by a passive peer or an error in hubsoft"

[Zlobomir 3]

Zlobomir just posted something but deleted it:

"Imho this could be caused by a passive peer or an error in hubsoft"

Yes, you are a good observer, and I happen to write bs smtms. The above can not be true, since the message would be "TTH inconsistency".

[Excelsium 0]

Hehe.. I think it helps with the theory a little "unidentified problem with interaction with certain peers". leading to corruption in the local copy.

Again to Big Muscle: I understand now the whole or parts of the local copy is corrupt when "corruption detection at position" is in the system log.

[Zlobomir 3]

Hehe.. I think it helps with the theory a little "unidentified problem with interaction with certain peers". leading to corruption in the local copy.

Yes, but again, every error "corruption detected at position" is caused by something in your computer, after your NIC card, wireless, modem or whatever.