Author Topic: Local PCLinuxOS Repository.  (Read 13442 times)

Offline Hairyplotter

  • Full Member
  • ***
  • Posts: 52
Local PCLinuxOS Repository.
« on: June 16, 2009, 03:02:27 PM »
This post does make the assumption that the end user is familiar with making a symlink, starting a service, opening ports on a router (if necessary) ect.
If you have any questions, feel free to either PM me or drop a reply, I will be more than happy to assist in any way possible.

I keep an extra computer tucked away in a closet, and one of the many functions of this spare machine is to maintain and serve up
(via anonymous ftp) a PCLOS repository. This step by step guide will explain how I configured my setup, your mileage may differ.

I am not going to get in to hardware configuration. Basically you will need an Internet capable system that can ideally run 24/7
(this isn't a requirement, but it helps).

1) Plan out the location of your repository and make sure you have enough free disk space.
Since I have most of my disk allocated to /home I put my repository in /home/ftp/pub/linux
If all you want are the standard repository sections, (main extra nonfree kde gnome) then you will need around 15g of free disk space.
If you want to add SRPM (source rpms) then you should have between 30-40g free.

2) Choose a repository close to your geographical location.
This is the longest part of the setup. Downloading the initial repository. This line will differ depending on the repository you download from.
Code: [Select]

rsync -aP --no-motd --stats --exclude=*testing --exclude=*kde4 --exclude=SRPM* spout.ussg.indiana.edu::pclinuxos/pclinuxos /apt/pclinuxos/2007/ /home/ftp/pub/linux
I personally like running the initial rsync from an interactive session so I can monitor its progress and interrupt it if necessary.
Once this part is finished, I wrote the following script and put it in /usr/local/bin then created a link to it in /etc/cron.daily to keep the repository updated daily and maintain a detailed log of file transfers.

Code: [Select]

#!/bin/bash
#
#--------------[ To Do List ]--------------
#--
#--
#--
#--
#--
#----------------------------------------------------------
# Function to perform some housekeeping before exiting.
#----------------------------------------------------------
function cleanup {
#
# Did we create a temporary file?
if [ -f $TMPFILE ]; then
# YES, then delete it before we exit.
rm $TMPFILE
fi
# Terminate the script with a return code of 0 (normal termination) or any other number (abnormal termination).
exit $RETVAL
}
# name the file we are going to log our output to.
LOGFILE="/home/ftp/pub/linux/pclinuxos/rsync.log"
# Create a temporary file.
TMPFILE=`mktemp -t rsync.XXXXXXXXXX`
# Was the temporary file created without errors?
if [ $? -ne 0 ]; then
# NO, print a message to our log file then terminate.
   printf "\n********** `date +'[%A %b %d %Y] - [%r]'` **********\n" >> $LOGFILE
   printf "Repository update FAILED: Unable to create temporary file\n" >> $LOGFILE
   RETVAL=1
   cleanup
fi
# Set our file counter to 0
COUNT=0

# Call rsync to sync our repository with spout.ussg.indiana.edu.
# -a: This is equivalent to -rlptgoD. It is a quick way of saying you want recursion  and  want  to  preserve  almost everything.
# -P: Show progress. This isn't very useful in a non-interactive session, but its nice when we want to sync interactively.
# --no-motd: Most servers have an motd, we don't need to see it so have rsync ignore it.
# --stats: Give us some extra file transfer stats. Good during an interactive session.
# --exclude=<pattern>: Ignore directories matching <pattern>. In this case, testing, kde4 and SRPM (source rpms).
# --log-file=: Log output to the temporary file we created above.
rsync -aP --no-motd --stats --exclude=*testing --exclude=*kde4 --exclude=SRPM* --log-file=$TMPFILE spout.ussg.indiana.edu::pclinuxos/pclinuxos/apt/pclinuxos/2007/ /home/ftp/pub/linux 2>/dev/null
# get the return value from rsync and assign it to RETVAL for later use.
RETVAL=$?
# Did rsync terminate without errors?
if [[ $RETVAL -ne 0 ]]; then
# NO, there was a problem with rsync. Write a FAILED notice to our log file, then exit.
printf "\n********** `date +'[%A %b %d %Y] - [%r]'` **********\n" >> $LOGFILE
case $RETVAL in
1)      REASON="Syntax or usage error";;
2)      REASON="Protocol incompatibility";;
3)      REASON="Errors selecting input/output files, dirs";;
4)      REASON="Requested  action  not supported";;
5)      REASON="Error starting client-server protocol";;
6)      REASON="Daemon unable to append to log-file";;
10)     REASON="Error in socket I/O";;
11)     REASON="Error in file I/O";;
12)     REASON="Error in rsync protocol data stream";;
13)     REASON="Errors with program diagnostics";;
14)     REASON="Error in IPC code";;
20)     REASON="Received SIGUSR1 or SIGINT";;
21)     REASON="Some error returned by waitpid()";;
22)     REASON="Error allocating core memory buffers";;
23)     REASON="Partial transfer due to error";;
24)     REASON="Partial transfer due to vanished source files";;
25)     REASON="The --max-delete limit stopped deletions";;
30)     REASON="Timeout in data send/receive";;
35)     REASON="Timeout waiting for daemon connection";;
*)      REASON="Undefined error";;
esac
printf "Repository update FAILED with error code $RETVAL: $REASON\n" >> $LOGFILE
RETVAL=2
cleanup
else
# YES, rsync finished without errors.
# get the number of package files transferred.
TRANSFERRED=`cat $TMPFILE | grep -c .rpm`
# Were any files transferred durning this update?
if [ $TRANSFERRED -gt "0" ]; then
# YES, we had some files transferred.
# Add a note to the logfile detailing the current time and the number of files transferred.
    printf "\n********** `date +'[%A %b %d %Y] - [%r]'` **********\n" >> $LOGFILE
    printf "Repository update SUCEEDED with $TRANSFERRED files transferred.\n" >> $LOGFILE
printf "Number\tSection\t\tFilename\n" >> $LOGFILE
echo "-------------------------------------------" >> $LOGFILE
else
# NO, there were no files transferred durning this update.
# No need to add a note to the log file.
# Call the cleanup function which will terminate this script after some housekeeping.
RETVAL=0
cleanup
fi
# Since we had some files transferred, we will parse our temp file line by line to get the filenames.
cat $TMPFILE | while read line
do
# Does this line have .rpm in it (denoting a package file)
FILE=`echo $line | grep .rpm`
if [ ! -z "$FILE" ]; then
# YES, lets increment our counter by 1 then write that number along with the section and filename transferred to our log file.
COUNT=$(($COUNT+1))
FILE=`echo $FILE | awk '{ print $NF }'`
SECTION=`echo $FILE | cut -d'/' -f3`
FILENAME=`basename $FILE`
printf "  $COUNT:\t$SECTION\t$FILENAME\n" >> $LOGFILE
fi
done
fi
RETVAL=0
cleanup

3) Now that I had the repository downloaded and ready to go, I needed to set up an ftp server to make the files available to all of my machines.

4) I started by installing vsftpd from the repositories.

5) After doing some reading on the vsftpd configuration file, I opened /etc/vsftpd/vsftpd.conf and added:

Code: [Select]

# Allow anonymous FTP? (Beware - allowed by default if you comment this out).
# I want a read only anonymous ftp.
anonymous_enable=YES
anon_root=/home/ftp
anon_upload_enable=NO
anon_mkdir_write_enable=NO
anon_other_write_enable=NO
anon_world_readable_only=YES

cmds_allowed=PASV,PWD,LIST,TYPE,MKD,CWD,EXIT,QUIT,HELP,RETR

pasv_min_port=10450
pasv_max_port=10500

pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES

use_localtime=YES

6) I set vsftpd to start when the computer is turned on.

7) I opened synaptic on my other machines and went to "settings" -> "repositories", unchecked the default repository then clicked "New".

URI: 192.168.1.105/pub/linux
Distribution: pclinuxos/2007
sections: main extra nonfree kde gnome

8) Click "Ok", then "Reload".  You should now have your very own PCLinuxOS repository that keeps itself up to date.

*NOTE. 192.168.1.105 is the IP address I manually assigned to my spare machine.
It is rumored that Bill Gates spent the summer of '77 killing hookers in Arizona, and some claim that their screams can still be heard every time Windows boots up.

Offline holljac

  • Jr. Member
  • **
  • Posts: 24
Re: Local PCLinuxOS Repository.
« Reply #1 on: June 20, 2009, 03:04:14 PM »
I'm trying to make this work, but just can't seem to get it to update the repo. The rsync update script does not seem to actually update the local repository. It worked fine for downloading the original set of files, but now even after I know there have been some updates, it runs the script but doesn't actually do anything. Running the script from a konsole, I get the following output.

receiving incremental file list

Number of files: 9679
Number of files transferred: 0
Total file size: 14343490084 bytes
Total transferred file size: 0 bytes
Literal data: 0 bytes
Matched data: 0 bytes
File list size: 437725
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 63
Total bytes received: 438800

sent 63 bytes  received 438800 bytes  79793.27 bytes/sec
total size is 14343490084  speedup is 32683.30

Any ideas what could be wrong?

Offline Hairyplotter

  • Full Member
  • ***
  • Posts: 52
Re: Local PCLinuxOS Repository.
« Reply #2 on: June 20, 2009, 10:35:52 PM »
I had the same problem too, then realized that I left off a parameter for the rsync command.

add --delete between --stats --exclude=*testing so that your rsync line looks like this:

Code: [Select]

rsync -aP --no-motd --stats --delete --exclude=*testing --exclude=*kde4 --exclude=SRPM* spout.ussg.indiana.edu::pclinuxos/pclinuxos /apt/pclinuxos/2007/ /home/ftp/pub/linux

Please be very careful with this new rsync line, it will remove any files that are no longer in the repository, make 1000% sure that your repository is away from any non repository files. It will not prompt before deleting.
It is rumored that Bill Gates spent the summer of '77 killing hookers in Arizona, and some claim that their screams can still be heard every time Windows boots up.

Offline travisN000

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1758
Re: Local PCLinuxOS Repository.
« Reply #3 on: June 21, 2009, 12:08:59 AM »
Looks like a very nice write up..  maybe the magazine would be interested??

I think Archie & Parnote are heading up the new effort.


As a side note, the FTP server can be skipped if the repo is on a shared network directory; the URI in Synaptic would then be the path to the local mount point / base repo directory.

Offline parnote

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 4444
  • The truth is out there ... PCLinuxOS!
Re: Local PCLinuxOS Repository.
« Reply #4 on: June 21, 2009, 07:24:05 AM »
 ;D ;D ;D ;)
PCLinuxOS Magazine Chief Editor

Linux Registered User #485009

In a world without walls, who needs Windows?

PCLinuxOS Wiki: Contribute tips/tricks/how-to's!

Offline holljac

  • Jr. Member
  • **
  • Posts: 24
Re: Local PCLinuxOS Repository.
« Reply #5 on: June 21, 2009, 08:40:17 PM »
That seems to have done it. Thank you very much.

Offline Hairyplotter

  • Full Member
  • ***
  • Posts: 52
Re: Local PCLinuxOS Repository.
« Reply #6 on: June 21, 2009, 09:33:11 PM »
You will also need to change the location of the log file.

Code: [Select]

LOGFILE="/home/ftp/pub/linux/pclinuxos/rsync.log"

Should be:

Code: [Select]

LOGFILE="/home/ftp/pub/rsync.log"

With the --delete option, the old location has the log file deleted as part of the rsync command.
It is rumored that Bill Gates spent the summer of '77 killing hookers in Arizona, and some claim that their screams can still be heard every time Windows boots up.

Offline mellon

  • Full Member
  • ***
  • Posts: 215
Re: Local PCLinuxOS Repository.
« Reply #7 on: July 18, 2010, 02:56:47 AM »
For some strange reason my Dad looses the connection to the internet every minute of downloading a larger file. 
I thought it a good idea to put a repository on an external disk and then  update his machine from that disk.

I was reading the article in the June 2010 magazine on this topic and all is fine except when I run the script.

It exits with the following line.
line 34: /home/user/tmp/rsync.ccsrRCAd5U: Permission denied


But if I look at the folder the file is actully created

[user@localhost tmp]$ ls -al
total 60
drwx------  5 user user  4096 2010-07-18 10:20 ./
drwx------ 47 user user  4096 2010-07-18 09:16 ../
drwxrwxrwx  3 user user 28672 2010-07-18 09:46 kde-user/
drwxrwxrwx  2 user user 12288 2010-07-18 10:18 ksocket-user/
drwx------  2 user user  4096 2010-07-18 10:17 orbit-user/
-rw-------  1 user user     0 2010-07-18 10:20 rsync.ccsrRCAd5U
-rw-r--r--  1 user user   116 2010-07-18 10:20 rsync.log

So I'm really puzzled about this. I have no knowledge about scripts. Perhaps the explanation is simple?


Offline rubentje1991

  • PCLinuxOS Tester
  • Hero Member
  • *******
  • Posts: 2110
  • Rubenus Parvus MCMXCI
Re: Local PCLinuxOS Repository.
« Reply #8 on: July 18, 2010, 05:22:10 AM »
what if you execute the following command?
Code: [Select]
chmod 666 /home/user/tmp/rsync.ccsrRCAd5U

Offline mellon

  • Full Member
  • ***
  • Posts: 215
Re: Local PCLinuxOS Repository.
« Reply #9 on: July 18, 2010, 06:43:55 AM »
That would not work well in the script, I think

The rsync.<randomnumber> file is a temporary file generated with a mktemp command in the script. I can run that command from the command line and it creates a file with a (I think) 0600 permissions setting.  No error is comming back.
But when executed from the script it is also created the file but generates also an error at which point $? in the script ends most likely with a none 0 value and the script goes into a branch that ends it. See below part of the script

Code: [Select]
# Create a temporary file.
TMPFILE=`mktemp -t rsync.XXXXXXXXXX`
# Was the temporary file created without errors?
if [ $? -ne 0 ]; then
# NO, print a message to our log file then terminate.
   printf "\n********** `date +'[%A %b %d %Y] - [%r]'` **********\n" >> $LOGFILE
   printf "Repository update FAILED: Unable to create temporary file\n" >> $LOGFILE
   RETVAL=1
   cleanup
fi


mellon