Here are some design Ideas for debmirror V2.0
=============================================
Files to Mirror:
- debs: /dists/sid/main/binary-alpha/Packages
- source: /dists/sid/main/binary-alpha/Packages
- D-I: /dists/sid/main/source/Sources
- disks: /dists/woody/main/disks-alpha/* (Any Release files there?)
- experimental: /project/experimental/main/binary-alpha/Packages
/project/experimental/main/sources/Sources
- extras: doc, indices, project, tools, ls-lR
- trace: /project/trace
Source:
1) Central management core
- parse config (apt sources.list format?)
- read options
- start modules (as needed or let them autostart when queues fill up?)
- fetch Release.gpg files
- check Release files and reget as needed
- check Packages/Sources files and reget as needed
- check other files
- finalize cleanup module
- get other files as needed
- collect summaries
- use trace files
+ checking a file registers it with the cleanup module too
+ files leaving a download object are fed into a Signature object
and retried a few times
+ when waiting periodically probe modules for stats and display
2) Modules
- Modules run as threads
- One queue for files to be processed (in)
- One queue for files finished processing (out)
- Some status vars [files/bytes queued, files/byte getting,
speed(1,5,15m ?)...] (out)
- args function to get additional args (run() argument?)
- help function to display help
- finalize function (close in queue)
- pause/continue to suspend the thread
- wait_finalize (wait for finalize to finish and return summary, die)
3) Clean Module
- gather a list of files present
- gather a list of all files that should be there (in queue, from
checking file)
- cleanup files on finalize
+ limit IO/s
+ limit number of files / bytes to be deleted (prevent a mirror wipe)
4) Signature Modules
- MD5sum
- SHA1
- GPG
+ limit IO/s
+ limit throughput
5) Download Modules
- ftp
- hftp (ftp via http://user:pass@proxy:port/)
- http
- rsync
- wget (wget-ftp)
- print (output what should be done)
+ limit IO/s
+ limit bandwith
+ limit traffic
