Excelsium

Looking for something more elegant, refined, automatic Lol Not sure if my full point came across.. I see some riddles about cars..: If the known fastest sources slots are taken then in the special case the user needs the last chunk of a download then grant the user an extra slot - if there is a fast source and the fast source has that chunk... of course. I see... but perhaps when 'it tries to replace this slow chunk(user) with the faster one' it could ask for permission to use the 'reserved for sending last chunk slot'... only in that circumstance of course and of course if the fast source has the required chunk, there is a known fast source etc. But So its difficult to build a fully functioning speed tracking feature.. I see. E

Hi.. Problem: - File 99% downloaded. - Downloads last chunk(s) from slow source(s). - Last 1% takes longer than first 99% (exaggeration). -- Sometimes there is no 'fast' source available. Solution?: - Track speed of sources. - One of the fastest sources grants extra temporary slot to user who needs the last 1%/chunk(s) of his download. - Maybe bandwidth prioritization for that slot. Is this practical? Maybe this is already implemented somewhere? E

why would you want to do that :)

yea.. It must not become more difficult/time/resource consuming to share or the long tail will be cut short :> DC is great in those areas.

Theres files on DC that cannot be found with BT. Also has communities BT does not have. Have to use both :>. Therefore the idea is interesting. But still need to have some kind of hubs 'Virtual' or 'Real' centralized ones. because of the communities and rare files that brings.

I think the most accurate version of my arguments come across in post #157 onwards. For ease of writing this I have defined bug to mean several things. - "problem, issue, lack of technique, gap in technique" - Solutions to prevent waste caused by gaps/lack of techniques in the segmented download technique. - To inform the user about the problem, not fix the problem. - To automate the detection of failing downloads so the user dosn't have to. New Rewording For Big Muscle: Helpful rewording.. of the entire thread perhaps... although If you need full detail on the waste that occurs to convince you the bugs need fixing, check the rest of this post. Issues with the segmenting download technique 1. Redownloading the same chunk(s) ad infinitum of a download with a corrupted temporary file that is beyond repair by itself. - It should stop the process and inform the user of which temporary files are corrupted in this case or it should at least inform the user with a dialog box. 2. Its corruption handling technique to repair corrupted temporary files does not have a high enough success rate, other P2P applications have a much higher, near perfect or perfect success rate. Issue 2. is a/the cause of Issue 1. I once described these bugs as "download same chunk ad infinitum bug caused by inability to correct corrupted temporary file bug". Description of the issues 1. A Corrupt temporary file for a download queue item exists on the users machine. 2. In some cases StrongDC++s segmenting download technique fails to repair the corrupt temporary file indefinitely. (Other P2P clients never fail at this or fail at a much lower %) 2.1. The segmenting download technique downloads the same chunk(s) of the file ad infinitum.*1* *2* *1*. Ad infinitum meaning: Its been running for days: It redownloads the same chunk(s) 100s-1000s of times at high speed (10Mbit connection) from 1-10 peers at the same time (Max is 10). *2*. The user is not informed which temporary file is corrupted. 2.1. Should inform the user if the corrupted temporary file cannot be repaired by itself. 2.1.2. Should pause the download if the corrupted temporary file cannot be repaired by itself. 2.1.3. Should detect if the download cannot be repaired by itself (e.g. redownloaded the same chunk(s) too many times) or some other technique. 2.2. "Informing feature" Only needs to be implemented if the repair corrupt temporary file technique is not fixed 100%. If it cannot be perfected then the informing feature is required for the few remaning % where the temporary file cannot be repaired by itself. I have not tested leaving it (Download with corrupted temporary file) running for more than a week because of the bandwidth waste. Addition: One option instead of just pausing the download after a set number of chunk redownloads: - If the corruption handling technique fails to repair a corrupt temporary file it should delete the download from the queue and re add it to the queue automatically. (Starts over with a new temporary file) - The user could be given this option when the corruption is detected or set it to do this automatically. - Other users have experienced these bugs. (see posts ITT) - Other users continue to experience these bugs. (see posts ITT) *ITT = in this thread About this thread New thread was moved into the old thread without my permission by forum admin, I'm in the process of cleaning it up. Cleaning done: you can skip everything except this opening post up until page 3 after the post marked "New Thread begins after this post." Title of thread censored: Real title: Bugs found in the download techniques packaged in ApexDC++/StrongDC++ Previous title: [Request] Corruption detection technique that informs the user of corrupt temporary files of download queue items. - this would be an interim technique if a fully automated technique is not implemented (If the bugs in the download techniques are not fixed). Long version of this opening post The Request: 1. Fix the bugs so that the user dosn't have to be informed as is the behaviour of other p2p clients. Last resort: 2. Corruption detection technique that informs the user of corrupt temporary files of download queue items. There are bug(s) in the way StrongDC++s/ApexDC++s segmenting download techniques corruption handling subtechnique handles corruption in temporary files of download queue items. There are also bugs/lack of corruption handling techniques in the way the normal download technique handles corruption in temporary files of download queue items. Hence there are several unfixed bugs and lack of corruption handling techniques in the packages that are StrongDC++/ApexDC++ Facts Please post corrections or relevant additions to these facts if required - I will post them here. Segmenting download technique - Does not complete a download if the local temporary file for that download is corrupt (Good) - It detects the corruption but does not inform the user of which file the corruption was detected and does not pause the download if the corruption in the local temporary file is beyond repair.(Bad) - It downloads the same chunk(s) of a download with a corrupted temporary file that is beyond repair (by itself)* ad infinitum (Bad) - It cannot repair the corrupted temporary file because its technique to do so is bugged. (Bad) - *1*Time and resource waste occurs because of lengthy manual user detection *2*.(Bad) *1*. Hence the reason for this feature request - I am in essence asking for a technique to automate what is currently a manual process which involves guessing, trial and error. *2*. Continually observing the transfers window and making a *Guess* which download(s) is/are not going to complete successfully due to a corrupted temporary file for that download. 2.1. The obvious bandwidth waste caused by downloading the same chunk(s) of a file ad infinitum until the user has engaged in and completed the manual corruption detection process. 2.2. StrongDC++/ApexDC++ crashes after re downloading the same chunk(s) of a download with a corrupted temporary file a random number of times. * This is the only bug so far a developer (Big Muscle in this case) has said they will fix - This fix has not yet been released. 2.2. Note: The cause(s) of the crash bug is/are unproven - the only bug I am unsure of. Standard legacy download technique - Completes the download regardless of a corrupt temporary file(Bad) - Perhaps it dosn't detect the corruption in the local temporary file at all, if it does - it ignores it and completes the download. Cause of the corrupted temporary files Unidentified defect in the users machine: Defect in the users machine possibilities: Hard drive, Filesystem, Windows XP, other software, other hardware. I will add defect possibilities as they are sent in. Corruption on the users machine is an inherently unfixable problem for all users for the forseeable future The main variable being the percentage of corrupted temporary files. - Hardware and software in/on the users machine can and will fail, I believe these are called "Error rates" some RAM has ECC for example. "1 error per 1billion operations" along those lines. We require the tools of information to prevent said consequences quickly and effectively. (See my reply below to Big Muscles statement: "User should fix his problem on his own".) Personal experience, opinion I use the legacy download technique - even If I have a corrupt file - that is going to be obvious to me when I open or verify it - In my opinion It wastes much less time and resources to redownload the file and is also much less annoying compared to the lengthy manual user detection process during the download. Also even without the speed of segmenting I still prefer this option because of the cost in time and resources of manually detecting even a few corrupted temporary files - in my case I get about 2 corrupted temporary files for every 60 files that I download, although I may have been "unlucky" recently and the overall number is perhaps 2 in 150 or more files since 2004. My relevant replies and questions left unanswered from the last thread: "I still prefer the client doing nothing automatically and letting the user cancel the download if he thinks he has to." Doing nothing automatically - my translation: Not having corruption handling techniques for corrupt data on the local machine. One of these automated techniques is to detect which files the corruption is located and inform the user of these corrupted files. You are saying the user should not be informed of what files are corrupted, by saying it should do nothing automatically. Rediculous amount of waste can occur in the mean time during the lengthy manual process of user detection... gone into full detail already. So you are saying that you think this large amount of waste is acceptable. If these really are your opinions, please reply and say it more explicitly in no uncertain terms. I'd also be interested to know if Big Muscle or anyone else reading this thinks that this waste is acceptable, clearly I think its unacceptable and I'm fairly sure the vast majority (users/devs) involved in P2P filesharing as a whole think its unacceptable but if you think its acceptable please come out and say so and state the reasons why you think it is acceptable. Other apps will at least detect local files that are corrupted and inform the user of these corrupted files, so that the user can do something about it and quickly. I already understood exactly what you just said and agree with you. I am asking for corruption handling techniques for corrupt data on the receiving users machine to prevent wastage (full detail-previous posts) and questions at the end of this post. So My theory of interacting with certain peers causes the local temp file to be corrupted is extremly unlikely however possible (hackers or ???) - I was just trying to list all possibilities however likely or unlikely so we can get to the bottom of this, which we have. I agree but with an addition in this case -

Results of testing. 0.2.2. Corrupt temporary files are repaired with 1 chunk redownload - havn

Hi, Just a small question

If the file is popular then there will be many sharing and many slots available to get that file. I think in the future upload slots per user will increase if segmented downloading is used by more users to counteract the problem of not enough slots which causes delay in obtaining the file, nothing more. If the file is not popular then theres probably nothing any one can do to speed up the download. -Amount of users that want the file is low. -Amount of users that share the file is low.

#177: I agree partially: It is either bug(s) in hardware or software or a combination that stops 1-5% of downloads from completing* using SDC++ segmented download feature. *1-5% in my case and in several other users cases. *Percentage of failing/failed downloads may vary in other bugged machines. I think I've said this already, I'll refer to this post if I see I need to make the same reply again. Or just not reply, my reply is here . #178: Agreed. except on the title for the mod/patch/etc.

#175 see #157: 3.2. and 3.1.

#173: I believe the bugs would be very similar or the same across the affected machines Making it easy to discover the exact location in a specific machine and fix a particular machines hw /sw or combination. - If bugs cannot be discovered or fixed then my technique in its automated or semi automated form is valid for this small group of users with said circumstances. - The group may grow in size as said in #157.

#171: Agreed, my technique is only valid for me and a small group of other users with said circumstances. - That group may grow if DC++ changes its technique as said in #157 - If the exact actionable specifics are discovered to 'fix the bug(s)', my technique is not valid and made obsolete. Until the exact actionable specifics to 'fix the bug(s)' are known I'll use my technique in its automated form when/if I get around to automating it.

#169: By causes I mean: The root causes of the corrupted temporary file that is beyond repair by the DC++ client. - The exact specifics of the 'bug' in which the user can take action to 'fix the bug' so they can use the segmented download technique in its current state with a 99.99999999%+ success rate. I dont believe the root causes have been proven yet - they could be hardware or software such as Windows XP. - the exact specifics have not been found. - or a combination of hw and sw bugs.

#167: I've always understood its only valid in said circumstances. correction to implemented 'automated' - its already been used by myself and others with said circumstances. - Agreed it wont be automated by you and probably not by anyone else reading this except myself - And if the root causes are discovered soon enough, not by myself either. - See #157,#164,#166 *I guess its ok if your refering to the implementation of the automation I'm done. more posts would be reffering to #157,#164,#166 etc and this one I guess. - Probably only valid for DC++ normal mode in its current state. That would be nice to deal with the small percentage of downloads that have been 'completed' but for whatever reason turn out to be a corrupt copy of the file the user tried to copy 'download' from a peer. Match TTH of just completed download to the one put in the queue. In fact that would probably go a long way to solve the problem of many TTHs for same filename - Also just corrected #157 some more.

#165: - About bittorrent: my technique is only valid in said circumstances. when the user has reliable high success rate using other methods. - The thread was never locked. - Your right, we've made our arguments.. my most relevant I feel are in #157 and #164 - Yup I may release a 'patch' if the root causes are not found... at the moment only a few would use it. - Would probably be never required if DC++ changed its technique because I feel the root causes would be discovered.

#163: StrongDC++ technique does not allow this, DC++ does - You've said DC++ may change its technique so that it too does not allow downloads with corrupted temporary files to complete. - I've corrected #157. - as said in #157 the root cause(s) will probably be discovered and posted online because the amount of users experiencing the redownloading chunk problem will increase if DC++ changes its technique in the way described. - I also acknowledge that my techniques are completly obsolete if the root cause(s) are discovered. - The bug(s) is unidentified and is located somewhere in my machine (hw/sw) - And/Or in the latest DC++ client. - My technique is a workaround for that bug to download a file successfully without corruption in 1-5% of cases where I cannot with StrongDC++s segmented download technique.

Reply to #161: DC++s method allows a download with a corrupted temporary file to complete. StrongDC++s method detects that the download has a corrupted temporary file and does not allow the download to complete. (Redownloads chunk ad infinitum if itself cannot repair the corrupted temporary file.) I will edit #157.

Big muscle replies for #158 and #159 are in my post #157.. I need to shorten it a little perhaps. Finished cleaning post #157. Nm.. cleaned some more :)

1. StrongDC++s segmented download technique does not allow a download with a corrupted temporary file to complete. 2. DC++s download technique in its current state does allow a download with a corrupted temporary file to complete. 3. If DC++s download technique is changed so that it does not allow a download with a corrupted temporary file to complete: 3.1. The amount of users who experience the redownloading chunk problem will be increased. 3.2. I feel several people will put effort into finding the root cause(s) of this issue (they could be software or hardware) - and post online. - Because of 3.1. The users could fix their problem if the root cause(s) are discovered. That bug iceman50 mentioned could be one of the causes.:

See the reply I just made to Zlobomir.

To Zlobomir: - A Knife can be used to slice bread or to kill people. Final repost of this note: - Note: The majority of peers in the network use the 'unsafe' download technique 'dc++ normal mode' - it allows corruption. - When I run out of explanations and arguments I will indeed stop posting and refer people to my earlier replies. Refer to my earlier replies.

1. It is accetable (to users of said circumstances) because it wastes less than the redownloading chunk ad infinitum problem. It also brings them to their destination (complete uncorrupted download). - Redownloading file with (95-99%+) success rate wastes less than said problem. 2. See earlier replies. - It is only used by users that have said circumstances - It is already used by users that have said circumstances. - It is and will only be used (enabled) by users of said circumstances. - The only difference will be that it will be automated. - Users of said cirumstances who use this technique check their downloads before sharing, they dont share 'crap'. - Note: The majority of peers in the network use the 'unsafe' download technique 'dc++ normal mode' - it allows corruption. - People who think its not acceptable for users of said circumstances to use this technique can refer to this and earlier replies. rather than me reposting for infinity.

Incorrect on one point: Could also be software problem as you admitted. (Outside of DC++ client). - Due to the exponential possibilities it would cause much waste to find the exact cause. - Hence my technique is a solution* hence I will automate my technique. * Only for users with high (95+%) success rates with my technique. - Unless of course its something like iceman50 pointed out - a real bug in the latest version of DC++.

- Factually incorrect: could be software. I see value in it because of the success rates I personally experience so I will add it (Automate it) for myself if no-one else. - I've seen at least a few other users that would benefit.