DHT bootstrap error

[horus22 0]

Hi,

When I try to enable DHT from the Advanced Settings tab, I get an error:

[19:27:48] DHT bootstrap error: HTTP/1.1 403 Forbidden

(http://strongdc.sourceforge.net/bootstrap/?cid=HJ5P5VVQSETM2P4JIFFFZYWPIZFV3MB7IVNMY3I&encryption=1&u4=6250)

ApexDC++ has all the activity allowed in my firewall and I don't have a router.

What's the problem?

[Big Muscle 1]

ApexDC++ shouldn't connect to strongdc website. SF complained to me about high overload to my website, so I blocked all version below StrongDC++ 2.40.

[Lee 84]

ApexDC++ shouldn't connect to strongdc website. SF complained to me about high overload to my website, so I blocked all version below StrongDC++ 2.40.

So you've created a feature that you want others to adopt, but you aren't prepared to allow them to use your resources. Why don't you pay for some additional resources so your users can connect to your bootstrap? We've disabled it by default, it's not ApexDC++'s fault you can't level with demand. Because of this reason it remains disabled by default - e.g. you create a feature, yet you can't support it.

Why don't you send us your bootstrap script so we can host it ourselves (and do a much better job at it)?

It's still not ready...

[Crise 26]

ApexDC++ shouldn't connect to strongdc website. SF complained to me about high overload to my website, so I blocked all version below StrongDC++ 2.40.

Then why did you create your DHT in such a way that it so heavily relies on one nexus point (ie. in this case strongdc website).

You could have also allowed it to request routing table from DHT node it finds online in a hub (how you can implement this is a good and secure way is another thing, but it is definitely possible).

[Lee 84]

Then why did you create your DHT in such a way that it so heavily relies on one nexus point (ie. in this case strongdc website).

You could have also allowed it to request routing table from DHT node it finds online in a hub (how you can implement this is a good and secure way is another thing, but it is definitely possible).

Just imagine if RIAA/MPAA get hold of it, or a DoS attack reaches the bootstrap... SF have logs of all accesses to bootstrap presumably.

Disaster is waiting to happen.

It's not even secure.

[Ophite 0]

The lesson: donâ

[Lee 84]

All I can say is I'm grateful we have this disabled by default.

[Big Muscle 1]

Lee, Crise: strongdc++ website is not the only way to bootstrap from. It's only a bonus for StrongDC++ clients. I don't see the reason why other clients should connect to StrongDC++ website. The bootstrap script is very simple, so everyone can make such script (and I think everyone can make it much better then I did). Btw, Crise should have my bootstrap script because I share it and he downloaded it from me some time ago.

The other possibility to bootstrap is how all other DHT clients/implementations do it. Users must download node list from server (e.g. nodes.dat in eMule, dht.xml in StrongDC++).

Bootstrapping from known nodes in ADC is a plan. But NMDC won't be supported in such way, I don't want any more hacks for this old stupid protocol.

[Crise 26]

Lee, Crise: strongdc++ website is not the only way to bootstrap from. It's only a bonus for StrongDC++ clients. I don't see the reason why other clients should connect to StrongDC++ website. The bootstrap script is very simple, so everyone can make such script (and I think everyone can make it much better then I did). Btw, Crise should have my bootstrap script because I share it and he downloaded it from me some time ago.

Well although, it is not correct of you to say that everyone can create such a script, because that is certainly not always the case... you do have a good point.

However, while I do understand your point and situation... I have to say that I disagree with the "bonus for users using client x" mentality. Because I believe every node in DHT should be equal but right now that is hardly the case. Because the whole point of DHT is decentralization unless I have misunderstood something, but if each client will end up having to rely on their own "nexus point", the bootstrap, that point is defeated.

And yes I did have your bootstrap script but I lost it some time ago in a PC switch but that's irrelevant. I could ask though, why have you not made your bootstrap script GPL like all the rest of the code. Because if you had I at least would not be participating in this discussion at all (the reason behind this question should be very clear, because right now web based bootstrap is vital for any client wishing to implement your DHT).

Edit: I guess I should say this though the user-agent string is in no way a reliable detection mechanism... after all there are several firewalls that alert user-agent and other http request headers. Also it is extremely easy to compromise such a check anyways.

Edit: Also one thing to do regarding the bootstrap to improve performance is to use some form of caching if possible to lessen the load. Because the thing that "costs" in the bootstrap script is db access. (After all it has to create a new connection on every bootstrap access).

[Big Muscle 1]

You still forget one thing. You can publish special dht.xml (for example with 1500 known nodes) with the client and it still will be able to co-operate inside this DHT network.

Bootstrapping from server isn't required at all (only improvement in StrongDC++ which is not vital) and it will be removed soon, because I don't like when application connects to external website. Bootstrapping from known nodes in the DC hub would also solve that "private hubs" problems. But still, there's no solution for NMDC (without hacks).

[Crise 26]

You still forget one thing. You can publish special dht.xml (for example with 1500 known nodes) with the client and it still will be able to co-operate inside this DHT network.

I haven't forgotten anything... but you seem to have, where does that special xml with 1500 nodes come from. DHT is not like ed2k where servers stay up for long time (if that was the situation your suggestion would work), the nodes in DHT come and go way too fast for this "special" file to work efficiently.

[Big Muscle 1]

Then why published nodes.dat for eMule's KAD network works very well?

[Crise 26]

Then why published nodes.dat for eMule's KAD network works very well?

Probably because it is more popular... ie. the probability of all nodes in the file being dead is lower, as for this DHT though I would consider the probability to have the file die much higher.... and while I do not know better I would think that emules kad is not only using the nodes.dat to get the nodes, like your DHT is at present, or is it?

[Big Muscle 1]

eMule has 3 options:

a ) bootstrap from external nodes.dat (equivalent in StrongDC++ is to bootstrap from external dht.xml)

b ) bootstrap from saved nodes.dat (as StrongDC++ always does when dht.xml is present in Settings folder - and it's always there when you don't run it for the 1st time)

c ) bootstrap from known nodes (i.e. eMule must be connected to some server, which is not much usable, because I didn't find any working one when I was testing it)

d ) bootstrap from certain node = from IP (not much usable, because you must get this IP somewhere)

So the only possibility to bootstrap when you are running eMule for the 1st time is A. And it's working because this file contain many nodes (as I remember, v1 stores 200 nodes, v2 stores 5000 nodes) and only ONE node is needed to bootstrap from, so there's high probability that 1 nodes is still online. And it is much more reliable and secure than bootstrapping from server DB as StrongDC++ does.

But I think that we should rather develop some good way to bootstrap from known nodes in DC hubs.

[PPK 0]

Here is bootstrap.php

<?php

  // check authorization

  if(strncmp($_SERVER['HTTP_USER_AGENT'], "StrongDC++ ", 11) != 0)

  {

    header("HTTP/1.1 403 Forbidden");

    die("Sorry, but you are not authorized to view this webpage.");

  }


  // check arguments

  if(strlen($cid = $_GET['cid']) != 39)

  {

    $error = "HTTP/1.1 400 Invalid CID.";

    header($error);

    die($error);

  }


  // temporary to kill old clients

  if($_GET['u4'] < 6241)

  {

    header("HTTP/1.1 405 Invalid UDP port (old client?). Bootstrap not allowed.");

    die();

  }


  // connect to database

  require('config.php');

  $db_link = mysql_connect($db_host, $db_user, $db_pass);

  if(!$db_link)

  {

    $error = "HTTP/1.1 500 Could not connect to database: " . mysql_error();

    header($error);

    die($error);

  }


  if(!mysql_select_db($db_name, $db_link))

  {

    $error = "HTTP/1.1 500 Could not select node database: " . mysql_error();

  	header($error);

    die($error);

  }


  // delete all entries with node's IP

  $sql_delete = "DELETE FROM nodes WHERE i4=\"" . $_SERVER['HTTP_X_REMOTE_ADDR'] . "\"";

  mysql_query($sql_delete, $db_link);


  $sql_query = "SELECT cid, i4, u4, timestamp FROM nodes WHERE cid NOT LIKE '$cid' ORDER BY timestamp DESC LIMIT 50";

  $select_result = mysql_query($sql_query, $db_link);

  if(!$select_result)

  {

    $error = "HTTP/1.1 500 Could not query to select data from the nodes table: " . mysql_error();

    header($error);

    die($error);

  }


  // return data in database in XML format

  $output = "<?xml version=\"1.0\" encoding=\"utf-8\" standalone=\"yes\"?>\n";

  $output .= "<Nodes>\n";

  while($query_result_array = mysql_fetch_row($select_result))

  {

    $output .= "  <Node CID=\"" . $query_result_array[0] . "\" I4=\"" . $query_result_array[1] . "\" U4=\"" . $query_result_array[2] . "\"/>\n";

  }

  $output .= "</Nodes>\n";


  ob_start();

  echo gzcompress($output, 9);

  header('Content-Length: ' . ob_get_length()); 

  ob_flush();


  // only active nodes will be saved to database

  if(is_numeric($udp_port = $_GET['u4']))

  {

    // save connected client to database

    $sql_insert = "INSERT INTO nodes(cid, i4, u4) VALUES(\"$cid\", \"" . $_SERVER['HTTP_X_REMOTE_ADDR'] . "\", $udp_port) ";

    $sql_insert .= "ON DUPLICATE KEY UPDATE i4 = \"" . $_SERVER['HTTP_X_REMOTE_ADDR'] . "\", u4 = $udp_port";

    $insert_result = mysql_query($sql_insert, $db_link);

    if(!$insert_result)

    {

      $error = "HTTP/1.1 500 Could not query to insert node info: " . mysql_error();

      header($error);

      die($error);

    }


    // keep only 50 latest entries

    if(mysql_num_rows($select_result) + 1 > 50)

    {

      $sql_delete = "DELETE FROM nodes ORDER BY timestamp LIMIT 1";

      $delete_result = mysql_query($sql_delete, $db_link);

      if(!$delete_result)

      {

        $error = "HTTP/1.1 500 Could not query to delete oldest node: " . mysql_error();

        header($error);

        die($error);

      }      

    }

  }

?>

[Big Muscle 1]

yes, this one I share on DC++, but it's old as hell

Anyway, thank you that you are interested in DHT. But it's logical - your client is dead, impracticable and archaic today, so you must find something new :)

[Lee 84]

yes, this one I share on DC++, but it's old as hell

Still trying to get one up on each other.

Would it be possible for you to send me the up-to-date version so we can do some internal tests?

[Big Muscle 1]

I'll send it to you when I get home next weekend. But I will remove this server-bootstrapping from StrongDC++ and will replace it with better way.