The Overnet Protocol
--------------------
The Overnet protocol is divided in two parts: localisation and download.
Localisation using UDP messages, while download is done using TCP/IP. The
download protocol is mainly the same as in edonkey, except that the
Connect/ConnectReply messages miss the server:port fields (ie they are 6
bytes shorter).
In this draft, I focus on the localisation part.
I Messages Format
=================
1) General Format:
------------------
UDP messages start with two bytes:
byte: 227 eDonkey general magic number for messages
byte: the opcode of the message
char[]: message args
2) Opcodes:
-----------
10 OvernetConnect
11 OvernetConnectReply
12 OvernetPublicize
13 OvernetPublicized
14 OvernetSearch
15 OvernetSearchReply
16 OvernetGetSearchResults
17 OvernetSearchResult
18 OvernetNoResult
19 OvernetPublish
20 OvernetPublished
24 OvernetFirewallConnection
25 OvernetFirewallConnectionACK
26 OvernetFirewallConnectionNACK
27 OvernetGetMyIP
28 OvernetGetMyIPResult
29 OvernetGetMyIPDone
33 OvernetPeerNotFound
3) Conventions:
---------------
int8: one byte
int16: two bytes in little endian format
int32: four bytes in little endian format
ip: an int32, for eg 127.0.0.1 = 127 + 0 << 8 + 0 << 16 + 1 << 24
md4: the 16 bytes md4 of the file/client
peer: an overnet peer in the following format:
md4: the peer md4
ip: the peer ip
int16: the peer port
int8: the peer kind/uptime/last seen ?
tags:
int32: <n> number of tags
tag[n]: the array of tag (as in edonkey proto)
4) Messages args:
-----------------
OvernetConnect:
---------------
md4: client md4
ip: my client ip
int16: my client port
int8: my client kind/uptime ? 0 is ok
OvernetConnectReply:
--------------------
int16: <n> number of peers propagated in this message
peer[n]: the peers propagated
OvernetPublicize:
-----------------
md4:
ip: ip
int16: port
int8: kind
OvernetPublicized: no args
------------------
OvernetSearch:
--------------
int8: kind of search ?
md4: the hash you are looking for
OvernetSearchReply:
-------------------
md4: the hash you have asked for
int8: <n> number of peers that could be interesting for you
peer[n]: the better peers you should ask
OvernetGetSearchResults:
------------------------
md4: the hash you are looking for
kind: ?
int16: min (the range of results you want)
int16: max
OvernetSearchResult:
--------------------
md4: the hash you have asked for
md4: the hash of the client/file of the result (depends if you look for
a file description or for a localisation)
tags: the tags describing the result
OvernetNoResult:
----------------
md4: the hash you have asked for
OvernetPublish:
---------------
md4:
md4:
tags:
OvernetPublished:
-----------------
md4: the hash of the published file
OvernetGetMyIP:
---------------
int8: 54
int8: 18
OvernetGetMyIPResult:
---------------------
ip: your ip
OvernetGetMyIPDone: no args
-------------------
OvernetFirewallConnection:
--------------------------
md4: the md4 of the client you want to access
int16: ?
OvernetFirewallConnectionACK:
-----------------------------
md4:
OvernetFirewallConnectionNACK:
md4:
II Client behavior:
===================
1) Basics:
----------
In Overnet, the description and localisation of files are stored on peers
such that the peer hash is as close as possible to the file hash.
When a peer wants a file, it tries to ask the peers which have the
closest hashs to the file it wants. These peers reply by sending the
closest peers they know to the requested hash.
The distance between two hashs is computed using what Kademlia calls a
Xor metric: a xor is computed between the two hashs, and the result
hash represents the distance. Two distances are then compared by using
a lexicographical order on their bytes (ie first bytes are compared, and
if equal, the second bytes, and so on...)
For this to work, the client needs to maintain a small list of peers it
knows, which are as spreaded as possible in the space of possible hashs.
2) Sequences:
-------------
a) Establishing a "connection" with other peers (booting)
---------------------------------------------------------
Client Peer
------------------------------------------------
1: ------- OvernetConnect ------>
2: <----- OvernetConnectReply ----
OK, the peer is connected, it can be added to the list of available peers
for searches. The peers propagated in the reply can be used for new
connections.
b) Searches
-----------
From the keywords search, remove all keywords of 1 or 2 chars.
Then, hash each keyword separately, and start one search per hash. When you
will get results, you will have to filter them so that all keywords are
matched, the protocol won't do that for you.
The search works as follows. You need a set of peers to be searched for the
hash. Typically, you take the peers with the hashes with the same first char
as the one you search.
Then, periodically (every 5 seconds ?), you take some peers (2 ? with the
closest hashes), and ask them for the hash:
Client Peer
------------------------------------------------
1: --------- OvernetSearch ------->
1': ------- OvernetGetSearchResults -->
2: <----- OvernetSearchReply -------
2': <----- OvernetSearchResult ------
2'': <----- OvernetNoResult -------
1: query for better peers to ask for the hash
2: return new peers which should be queried for the hash
1': query the peer directly for the hash: do you know it ?
2': yes, I know it, here are the files containing this keyword
(or the clients sharing this file if you are looking for a file-hash)
2'': no, I have no info, try someone else...
Now, when you have the results, you need to remember what you are
searching for:
1) keywords: results are file descriptions, closed to edonkey ones
(here, maybe some tags are different :)))
2) files: you should look in the tags for one called "loc". It contains
a string with the format "bcp://ip:port" or "bcp://????:ip:port"
containing the source ip and port in ascii.
Now, you just need to connect and download ...
