Speed-up apt-get downloads with apt-fast!

[Neal]

I installed Martin's RPM and issued this command

Code:

apt-fast update && apt-fast dist-upgrade

In return, I got

apt-fast: command not found

[MBantz]

Sorry, my bad,

use apt-fast.sh I've forgot to remove the .sh - fixing it tomorrow (my time:-)

Also, I agree that the check line is confusing:

Code:

#If the first user entered variable string contains apt-get, and the second string entered is either install or dist-upgrade
if echo "$1" | grep -q "[upgrade]" || echo "$2" | grep -q "[install]" || echo "$2" | grep -q "[dist-upgrade]"; then
  echo "Working...";

can't find any reference anywhere on what double longbar means - I guess it is supposed to mean logical OR as in other languages - but here I think it's only double pipes? The remark above the line does not make much sense either.

Reading the logic of the program, I guess it is ment to handle upgrade, install and dist-upgrade - and leave other sequences to apt-get, which makes sense.

[MBantz]

hmm.. it was removed, something else must have slipped,

anyway, here is an updated version

http://dl.dropbox.com/u/2546717/apt-fast-0.1-2pclos2010.i586.rpm

where apt-fast can be used

EDIT: rpm for testing in thread below

[Neal]

Thanks, Martin.

[travisn000]

I just installed the apt-fast package..  my first test result are in:

timed apt-fast update: 27-47 second range over 4 trials
timed apt-get update: 17-18 seconds over 4 trials

Firefox d/l via apt-get (pclosusers.com):
Fetched 1142kB in 3s (318kB/s)
Fetched 1142kB in 2s (506kB/s)
Fetched 1142kB in 1s (729kB/s)
Fetched 1142kB in 2s (409kB/s)
...........avg total time to complete command: 20 seconds

Firefox d/l via apt-flash:
1114.9 kilobytes in 1 second. (643.73 KB/s)
1114.9 kilobytes in 1 second. (595.51 KB/s)
1114.9 kilobytes in 1 second. (617.74 KB/s)
1114.9 kilobytes in 2 seconds. (476.50 KB/s)
...........avg total time to complete command: 24 seconds


..so far it appears that despite the faster download of the individual file, it is still faster overall to use apt-get for a single package;  maybe this will change with a larger number of files

[maik3531]

Make sure the speed advantage only at larger downloads such remarks OpenOffice...

[travisn000]

Here are the changes I was recommending:

Code:

#!/bin/sh
#apt-fast by Matt Parnell http://www.mattparnell.com , this thing is FOSS
#please feel free to suggest improvements to admin@mattparnell.com
# Use this just like apt-get for faster package downloading. Make sure to have axel installed


#If the user entered arguments contain upgrade, install, or dist-upgrade
if echo "$@" | grep -q "upgrade\|install\|dist-upgrade"; then
  echo "Working...";

  #Go into the directory apt-get normally puts downloaded packages
  cd /var/cache/apt/archives/;

  #Have apt-get print the information, including the URI's to the packages
  #Strip out the URI's, and download the packages with Axel for speediness
  apt-get -y --print-uris $@ | egrep -o -e "(ht|f)tp://[^\']+" > apt-fast.list && cat apt-fast.list | xargs -l1 axel -a

  #Perform the user's requested action via apt-get
  apt-get $@;

  echo -e "\nDone! Verify that all packages were installed successfully. If errors are found, run apt-get clean as root and try again using apt-get directly.\n";

else
   apt-get $@;
fi

Not only should this allow for full apt-get syntax replacement,  it might even be a few milliseconds faster as it has a few less calls to grep.  It also won't call xargs / axel when nothing needs to be downloaded like the original script did (causing axel errors to print)..   ..it is smart enough to know when the d/l list is empty!
 

[gseaman]

Brilliant!

[MBantz]

Looks great!

Packaged for testing, get it at:

http://dl.dropbox.com/u/2546717/apt-fast-0.1-3pclos2010.i586.rpm

[MBantz]

I just installed the apt-fast package..  my first test result are in:

timed apt-fast update: 27-47 second range over 4 trials
timed apt-get update: 17-18 seconds over 4 trials

Firefox d/l via apt-get (pclosusers.com):
Fetched 1142kB in 3s (318kB/s)
Fetched 1142kB in 2s (506kB/s)
Fetched 1142kB in 1s (729kB/s)
Fetched 1142kB in 2s (409kB/s)
...........avg total time to complete command: 20 seconds

Firefox d/l via apt-flash:
1114.9 kilobytes in 1 second. (643.73 KB/s)
1114.9 kilobytes in 1 second. (595.51 KB/s)
1114.9 kilobytes in 1 second. (617.74 KB/s)
1114.9 kilobytes in 2 seconds. (476.50 KB/s)
...........avg total time to complete command: 24 seconds


..so far it appears that despite the faster download of the individual file, it is still faster overall to use apt-get for a single package;  maybe this will change with a larger number of files

Sounds right, since apt-get is called twice in apt-fast - I guess the overhead always will be around 4 secs no matter how many packages retrieved. This overhead is peanuts in installation of more packages (I guess the sweet spot is 2-3 packages, anyone want to calculate big O (formulae for speed) for this? :-)

[Matt Parnell/ilikenwf]

Here are the changes I was recommending:

Code:

#!/bin/sh
#apt-fast by Matt Parnell http://www.mattparnell.com , this thing is FOSS
#please feel free to suggest improvements to admin@mattparnell.com
# Use this just like apt-get for faster package downloading. Make sure to have axel installed


#If the user entered arguments contain upgrade, install, or dist-upgrade
if echo "$@" | grep -q "upgrade\|install\|dist-upgrade"; then
  echo "Working...";

  #Go into the directory apt-get normally puts downloaded packages
  cd /var/cache/apt/archives/;

  #Have apt-get print the information, including the URI's to the packages
  #Strip out the URI's, and download the packages with Axel for speediness
  apt-get -y --print-uris $@ | egrep -o -e "(ht|f)tp://[^\']+" > apt-fast.list && cat apt-fast.list | xargs -l1 axel -a

  #Perform the user's requested action via apt-get
  apt-get $@;

  echo -e "\nDone! Verify that all packages were installed successfully. If errors are found, run apt-get clean as root and try again using apt-get directly.\n";

else
   apt-get $@;
fi

Not only should this allow for full apt-get syntax replacement,  it might even be a few milliseconds faster as it has a few less calls to grep.  It also won't call xargs / axel when nothing needs to be downloaded like the original script did (causing axel errors to print)..   ..it is smart enough to know when the d/l list is empty!
 

Very nice! I've updated and posted it with credit to you for the modifications at http://www.mattparnell.com/projects/apt-fast-and-axel-roughly-26x-faster-apt-get-installations-and-upgrades.html ...the source is now located at http://www.mattparnell.com/linux/apt-fast/ for future reference.

Anymore, I don't need it because I use Archlinux now, and it's package system supports axel or any other manager through the config. That said, nice work. I've even found out recently that Linux Format mentioned apt-fast in an article...so I guess that's why I have gotten so much traffic recently on this post.

For those of you packaging this and modding it, if you'd like some space to host all that other than using free file hosts, let me know. I have unlimited space and bandwidth, and would be happy to give you a place to put your packages and modifications of apt-fast. I'll give you access to a single folder (probably mattparnell.com/linux/pclinux/) where you can just arrange folders and files as you wish.

...oh and as an afterthought, you could get more speed (and rape the servers heavily) if you append -n X where X is anywhere between 1 and 10. This tells axel how many connections to use for each download. -n 10 should be fine on faster connections, as long as the servers on the other end can handle the load...just put it directly after axel -a in the script.

[travisn000]

...oh and as an afterthought, you could get more speed (and rape the servers heavily) if you append -n X where X is anywhere between 1 and 10. This tells axel how many connections to use for each download. -n 10 should be fine on faster connections, as long as the servers on the other end can handle the load...just put it directly after axel -a in the script.

..I had considered this, but was not too sure if starting more download threads than I have processor threads would be useful; it seems like they might just be fighting for resources on my pc   (..how this type of thing is handled by the hardware is beyond me though)

[Matt Parnell/ilikenwf]

...oh and as an afterthought, you could get more speed (and rape the servers heavily) if you append -n X where X is anywhere between 1 and 10. This tells axel how many connections to use for each download. -n 10 should be fine on faster connections, as long as the servers on the other end can handle the load...just put it directly after axel -a in the script.

..I had considered this, but was not too sure if starting more download threads than I have processor threads would be useful; it seems like they might just be fighting for resources on my pc   (..how this type of thing is handled by the hardware is beyond me though)

No, it doesn't matter how many cores you have, you're running more than 2, 4, or 6 threads at any given time. The kernel alone has more than that. It's just that YMMV depending on your connection speed.

[MBantz]

I have built a new rpm for testing, this version hooks the real apt-get. After installing apt-fast you just use apt-get as normal, and the three apt-get functions will use the axel downloader.

When removing the package apt-get is renamed back again,

http://dl.dropbox.com/u/2546717/apt-fast-0.1-4pclos2010.i586.rpm

cheers,
MBantz

[Matt Parnell/ilikenwf]

I have built a new rpm for testing, this version hooks the real apt-get. After installing apt-fast you just use apt-get as normal, and the three apt-get functions will use the axel downloader.

When removing the package apt-get is renamed back again,

http://dl.dropbox.com/u/2546717/apt-fast-0.1-4pclos2010.i586.rpm

cheers,
MBantz

Contact me if you'd like a place to start putting packages.