PCLinuxOS users who would like to download all available "updates" packages and store them in a local repository can do so with the following command-line method, as an alternative to using "rsync". The method described below is most useful if the computer retrieving the packages is non-Linux -- Microsoft Windows, for example, and the packages will be transfered to a Linux computer in another location. In my case, I have dial-up access only, and I must rely on another person, who has broadband access, to download "updates" packages. "Wget" is available in a Microsoft Windows version, making this method possible. I provide the "input-file", listing the packages to be downloaded. The other person downloads the packages and copies them to a USB flash drive. I then copy the packages from the USB flash drive to my local repository. To keep downloading sessions relatively short, I try to increment "updates" additions in one-month intervals. The following method, for the sake of brevity, assumes that the complete procedure will be accomplished on one Linux machine, but "ftp" and "wget" commands will be essentially the same on a Microsoft Windows machine.
For Linux, two command-line applications are needed: "lftp" and "wget". To see if they are already installed, enter in terminal-mode:
lftp --help
wget --help
To see a full listing of "lftp" commands, enter:
lftp
?
The first decision to make is to choose an online repository from which to download the packages. Run "Repository Speed Test" to help make the choice. The server maintained by the Federal Institute of Technology, Zurich, Switzerland will be used in this example.
The full path to the "updates" packages on the server is:
ftp://ftp.ch.debian.org/mirror/pclinuxos/apt/pclinuxos/2010/RPMS.updates/
Use Alexander V. Lukyanov's "lftp" to connect to the server and display the directory's contents. Connect to the Internet, open a terminal, enter:
lftp
open ftp://ftp.ch.debian.org
cd /mirror/pclinuxos/apt/pclinuxos/2010/RPMS.updates/
To list the directory's contents by modification date and in reverse order, i.e. earliest to latest, enter:
ls -lrt > updates.txt
This command will also pipe the output to "updates.txt" in the local directory where the command was executed. The "updates.txt" file will be useful in the future as a basis for incremental downloads of "updates" packages, so do not delete it.
To see the amount of disk storage occupied by the "updates" packages:
du -hs
Look at the figure displayed, and if the size is not too daunting, the downloading can soon begin.
First, shutdown the "lftp" session.
exit
Next, "wget" is used to access the remote "updates" directory, and to create an input file that will enable downloading all available "updates" packages. In terminal-mode, enter:
wget ftp://ftp.ch.debian.org/mirror/pclinuxos/apt/pclinuxos/2010/RPMS.updates/ links index.html
The file "index.html" will be created in the local directory where the command was executed. This file contains links to all packages available in the remote "updates" directory.
To begin downloading "updates" packages, enter:
wget -F --wait=2 --input-file=index.html
Downloading will begin to the local directory where the command was executed. If downloading gigabytes of data is too massive an undertaking for one session, the "index.html" file can be manually edited into multiple files, each containing a fewer number of links to process. When the downloading is completed, the packages can be moved to a directory created for the local repository ("/home/userID/Repository/pclinuxos/2010/RPMS.extras" follows a directory-tree scheme similar to that used by the online repositories, but the final name choice is up to the user). Those who are sufficiently Linux-savvy can create the local repository directory first, then download the packages directly to that directory. Modify the necessary command-lines accordingly for the proper paths.
The local repository on my hard disk has the following directory-tree structure (my userID is "rg"):
/home/rg/Repository/pclinuxos/2010/base
/home/rg/Repository/pclinuxos/2010/RPMS.extras
A "base" directory is absolutely necessary. Package lists will be generated and stored there, enabling Synaptic to access the packages stored in "RPMS.extras". Naming the directory "RPMS.extras" is optional, by the way; the final choice is determined by the user. I chose "RPMS.extras" to differentiate packages stored there from "updates" packages that might be stored elsewhere, as in apt's cache. (For the purpose of this tutorial, I am "pretending" that my "updates" packages are stored on my hard disk. In reality, they are stored on a USB flash drive, one of three holding the totality of my local repository; they are formatted FAT32, labeled PRIMARY, SECONDARY, TERTIARY, respectively.)
Assuming that the downloaded "updates" packages now reside in:
/home/userID/Repository/pclinuxos/2010/RPMS.extras
do the following:
cd /home/userID/Repository/pclinuxos/2010/base
genbasedir --progress /home/userID/Repository/pclinuxos/2010 extras
"Genbasedir" will announce when it has finished processing the packages, and the "base" directory will now contain the package lists. The local repository now needs to be added as a "source" for Synaptic.
cd /etc/apt/
su
nano sources.list
Add the following line to "sources.list", using the proper userID:
rpm file:/home/userID/Repository/ pclinuxos/2010 extras
Note the "space" in the line of code above and that each line in "sources.list" ends with a "space". Add a "#" to all online repository lines, disabling them. Press F3, then "Enter" to write the file. Press F2 to exit "nano". Synaptic will now use the local repository, exclusively, for installing "updates" packages. Enter "exit" twice to leave terminal-mode.
Now, open Synaptic. Look at "Settings/Repositories". The local repository should be check-marked. Click "Reload" to make Synaptic aware of the package lists for the local repository. Click "Status", then "Installed (upgradable)", and all upgradable packages will be displayed. Click "Mark All Upgrades", then "Apply".
It is strongly recommended that only one repository source be used at one time; one online repository or the local repository. When finished using the local repository, remove its check-mark, then enable the online repository of your choice -- if it is your habit to use an online repository, as most do. I rely entirely on "pre-downloaded" packages, obtained from purchased repository disc-sets or from USB flash drive storage, so my local repository is enabled all the time. As I mentioned earlier, my local repository is almost entirely stored on USB flash drives, but I also have some packages stored on hard disk, thus making "sources.list" read as follows:
[rg@localhost ~]$ cd /etc/apt/
[rg@localhost apt]$ cat sources.list
# Package repository URL's
# Signed repositories have a [<key>] where <key> is the name of the key
# as it appears in vendors.list. If you remove it, no digital signature check
# will be made.
rpm file:/media/PRIMARY/Repository/ pclinuxos/2010 main special
rpm file:/media/SECONDARY/Repository/ pclinuxos/2010 kde4 nonfree
rpm file:/media/TERTIARY/Repository/ pclinuxos/2010 updates
rpm file:/home/rg/Repository/ pclinuxos/2010 extras
# rpm http://ftp.ch.debian.org/mirror/pclinuxos/apt/ pclinuxos/2010 main updates nonfree games kde4
...
The terminal-mode output that was piped to "updates.txt" might be useful the next time, when more "updates" packages are retrieved. Keeping track of Texstar's prodigious package production can be a full-time "hobby" in itself. That file, "updates.txt", sorted by date, will provide a record of what packages have already been downloaded. For the next downloading session, pipe terminal output to another file, named "new.txt", for example, then "updates.txt" and "new.txt" can be compared to see what files need to be added. Do:
wget ftp://ftp.ch.debian.org/mirror/pclinuxos/apt/pclinuxos/2010/RPMS.updates/ links index.html
again, then edit "index.html" to include only the newest files that need to be downloaded. "Wget" can also use a text file as the "input-file", such as:
[rg@localhost Archive]$ cat list.txt
http://cserver.king-online.com/content/Joseph_thumb?date=20101007&size=hires
http://cserver.king-online.com/content/Joseph_thumb?date=20101008&size=hires
http://cserver.king-online.com/content/Joseph_thumb?date=20101011&size=hires
http://cserver.king-online.com/content/Joseph_thumb?date=20101012&size=hires
http://cserver.king-online.com/content/Joseph_thumb?date=20101013&size=hires
http://cserver.king-online.com/content/Joseph_thumb?date=20101014&size=hires
http://cserver.king-online.com/content/Joseph_thumb?date=20101015&size=hires
"Wget" is then used as follows:
wget --progress=bar --wait=2 --referer=http://joseph.king-online.com/client_temps/kol.php --input-file=list.txt
In the example above, additional code is added for the "referer" option. In its shortest form, the code would be:
wget -i list.txt
where no "referer" page is needed.
=============================
If anyone knows how filenames can be sorted, earliest to latest, filenames only (one column only), accessing a directory on a remote server, please let me know. I have tried:
ls
ls -1rt
and both fail to display filenames only, one-column only. Googling has not yet provided an answer. (2010-10-12)
[SOLVED] Use the "cls" command to display filenames only. Solution found in the "lftp" mailing list archive.
=============================
