My, My, My! ... MiniMe

[Old-Polack]

Today I found that one of my own MiniMe installations never got the aptupgrade script run on it, to upgrade to the new apt, rpm and Synaptic. When I tried to do a normal upgrade, Synaptic wanted to remove the entire KDE 4.6.5 desktop. Where have we heard that before?  

Anything I tried to upgrade these items separately through apt-get or Synaptic failed for dependency reasons. While it started as a MiniMe installation, in 2010, it now contains some 1300+ added packages and two years of personal tweaks. I really didn't want to lose all this, or have to start from scratch to rebuild it.

I don't have a separate /home partition; /home/polack is part of the / partition. Now we all know I could do the usual backup of /home/polack, reformat, reinstall MiniMe, then reinstall the various apps in the regular way, but I decided to play with it in a slightly different fashion. I had nothing to lose anyway, and I would be working with a copy, in case it went totally belly up.

The first thing I did was generate a list, in /home/polack, of all the installed packages with the genlist script I've posed to the forum at various times. I then created a new partition on my fairly empty 2 TB Green drive, rebooted to my regular KDE installation, mounted the MiniMe partition on /mnt/here and the new partition on /mnt/there, then ran;

[root@fatman ~]# rsync -av /mnt/here/ /mnt/there                <Enter>

...to copy my MiniMe installation to the new partition. When done, I edited the /mnt/there/etc/fstab to reflect the new / partition, and also added a new boot stanza for it in my master menu.lst.

Next I ran mkinitrd from a chroot environment on the copied installation, to rebuild the initrd image of the default kernel, so it would use the proper new / partition and filesystem format when booting from its new home.

I rebooted into the copied MiniMe to verify everything was working the same as the original, then rebooted once again to my liveHDD MiniMe 2012.2 image. Choosing the proper partition of the installed MiniMe copy, I then installed the 2012.2 MiniMe over the top of the copy, unchecking the box that would normally have caused the partition to be reformatted. After the installation I rebooted to the copied MiniMe.

I set the root password, then created the normal user using the same name, UID and GID as the original. I looked for any entry to not create a new /home/polack directory that would overwrite the existing one, but found nothing. Fearing the worst, I then logged in as polack.

My desktop was just as it had been, tweaks and all. All links to applications worked, bookmarks and such were unchanged, and Synaptic was able to run a normal system upgrade without removing the KDE desktop. Oddly, Synaptic only saw the newly installed MiniMe base applications, but not the other added applications, even though they all seemed to work properly when opened from the menu or desktop icons. The system knew they were there, but Synaptic did not.

I tried to rebuild the rpm database, to include those other installed and working packages, so they could be properly upgraded, but that failed. Synaptic still didn't show them as being installed at all. I sort of expected that.

Finally, I ran my newlist script to generate a new installation package list, then ran my difflist script to generate a list of only the missing applications from the original list made before the copy process. I removed the installed kernels from that list, divided it into eight smaller files, then entered each into Synaptic's File --> Read Markings, one at a time, and clicked Apply after each entry. The end result is all the packages have been upgraded to the newest version, and now show up properly in the rpm database and Synaptic.

Not one of my personal tweaks was lost or damaged in any way during the whole process. My /home/polack directory and all its contents is absolutely intact.

I've since installed the 2.6.38.8-pclos3.pae.bfs and rebooted. Dkms informed me there were two versions of two of the nvidia drivers and was confused as to what to do next, so I couldn't immediately log into my GUI desktop. I logged in as root at the VT login, then used locate to find /var/lib/dkms/nvidia<whatever>, removed the older version of each driver module from each of those directories, then rebooted. Dkms then did its thing and I arrived back at the GUI login, logged in as polack and am now here, to tell my tale.  

My, my, my! That worked really slick.

[gseaman]

Excellent! Now, can you write us up a little script, you know, just in case?

Galen

[nixer]

Thanks OP, I am sure this method will come in handy.  For others reading this, I found this method in Tex's twitter (as a retweet, I guess) and I had a need to give it a try.  I found an older remaster that I wanted to dust off and use.  I had read about this procedure that you could use to do the pclinuxos aptupgrade script so that rpm would be updated.  I have used this procedure twice now and it has worked both times.  Please note that there is an "error" notation in the terminal as it completes, but it may be one of those notations that is not a fatal error because it works and the updates will continue.

[Just17]

I have never used 'Read Markings' ......  it seems to look for a difference file ....  but never did come across any information about its use ----  I did not search, just never stumbled across anything ...

[djohnston]

@nixer,

Thanks for the link to Fox Mulder's blog. That zip file will be just the ticket for a small project I had on hold.

[Old-Polack]

I have never used 'Read Markings' ......  it seems to look for a difference file ....  but never did come across any information about its use ----  I did not search, just never stumbled across anything ...

If you choose a group of packages to install in Synaptic, but then find you need to wait for a while before you can complete the package installation, you can save the markings as a text file.

Example:

Here I select dosbox and some dosbox games.



Then I choose to save the markings for later use.





Opening the dos-games.txt file we see this.



The next time you open Synaptic you can use File --> Read Markings to open dos-games.txt and it will remark all the applications for you.









The installed package scripts generate alphabetically sorted lists in the same format that Synaptic saves its own markings. The difflist script saves having to re-install all the default packages, leaving just the added packages in its list. One can then use the difflist to duplicate the same package installations on a fresh installation on one, or any number of additional machines.



I broke my list into smaller lists because the total was 1300+ lines, and past experience has shown Synaptic will choke a bit on that large a list, trying to sort dependencies and installation order. Synaptic will do lists of approx 150 packages with ease. The kernels were removed from the list because the system doesn't much like installing more than one kernel at a time. It messes up the mkinitrd process and the addition of new menu.lst boot stanzas.

If an older package is discontinued, it will result in a Synaptic broken package type error, so any short list that produces one can be temporarily by passed, and all the rest of the short lists can then be processed. The discontinued package and its dependencies can be manually removed from the remaining list so it can then be run without the broken package problem. Whatever replaced the discontinued package, if anything did, can then be installed in the normal manner.

[Just17]

Thanks for the explanation    Useful function.

[gseaman]

That is a great tip! Thanks.

Galen

[djohnston]

That is a great tip! Thanks.

Galen

+1  Thanks, O-P.

[Tony]

Bookmarked ! I love these type of Tutorials (fixs/abnormalities), with pictures too, and obscure links, excellent 

[Old-Polack]

Rather than have to search for the three scripts used in the procedure above, I've reproduced them all below, for convenience sake.


This is the first script that would be run on the installation who's application list you wish to duplicate. Copy/paste the purple text only into the text editor of your choice. (not the start and end lines) Press the Enter key to start a new line at the end, then save the file to /home/<you>/bin as genlist. If you haven't yet created a ~/bin directory, do so now. This will place any scripts you add there in your default path, so they can be run from anywhere on the installation, using just the script name in a terminal. Right click on the file, then choose Properties --> Permissions and check the box in front of Is executable. Do this for each of the three scripts below.

Copy the output file installed-packages.txt to a location where it can be again copied to the new installation's ~/ directory of your new normal user. A USB stick storage media works well if the new installation is on a different machine.

If you are attempting to install an updated base system over the top of an existing system, as outlined in the original post of this thread, generate the installed-packages.txt list before doing the installation, and leave it where it is in the ~/ directory of your normal user. You might want a backup of the list somewhere safe, just in case.

----------------------------------------------- start ------------------------------------------------------
#!/bin/sh
# Name=genlist
# Author=Polack
# Script to generate an installed package list
# to aid in re-installation, and upgrade.
#

rpm -qa --qf '%{NAME}\t\tinstall\n' | sort > ~/installed-packages.txt

exit

---------------------------------------------- end --------------------------------------------------------

This is the same script as above, but with a different name for the output file. This should be run on the new installation after first doing the full upgrade procedure.

Again, copy/paste the purple text only into the text editor of your choice, press the Enter key to start a new line at the end, and save the file in your ~/bin directory, but name this one newlist. Remember to make it executable.

---------------------------------------------------- start --------------------------------------------------------------------
#!/bin/sh
# Name=newlist
# Author=Polack
# Script to generate an installed package list, from a new installation,
# to aid in re-installation, and upgrade.
#

rpm -qa --qf '%{NAME}\t\tinstall\n' | sort > ~/new-installed.txt

exit

----------------------------------------------------- end --------------------------------------------------------------------

Same instructions as the other two scripts, but name this one difflist.

---------------------------------------------------- start --------------------------------------------------------------------
#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit

----------------------------------------------------- end --------------------------------------------------------------------

The output file of the above script will be to-install.txt. The script should be run from the ~/ directory of the normal user, which is the default location when opening a terminal, and also the location of the output file.

This is the file to be used by Synaptic as outlined in the original post of this thread. Open it first in a text editor, find the kernel<whatever> lines, and remove them, then save the file.

If the file is massive in length, and Synaptic chokes on it, break its contents into smaller chunks, (a number of files with fewer lines) then process each separately.

Recap from above:

On a new installation, you will not have a ~/bin directory; create one and place your scripts in it. By default ~/bin is in your path, so scripts placed in it will be executable from any location, just as system applications are. Each script must have the Is executable box checked, on the Properties --> Permissions tab.

I keep a master copy of my bin directory, containing all my scripts, on a separate data partition from which it can either be copied in full, or simply linked to, from the various installations I have on this particular machine.

[YouCanToo]

When I run the difflist file this is what I see in the to-install.txt
NOTE the word eject and install on the first line

Code: [Select]

eject          install#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit

What the heck am I doing wrong? It is doing the same on all my machines.

this is what I see in my difflist file

Code: [Select]

#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit


[Old-Polack]

When I run the difflist file this is what I see in the to-install.txt
NOTE the word eject and install on the first line

Code: [Select]

eject          install#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit

What the heck am I doing wrong? It is doing the same on all my machines.

this is what I see in my difflist file

Code: [Select]

#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit


I don't see the new line after exit. Have you generated both lists from genlist and newlist, and are both output files in your ~/ directory?

[YouCanToo]

When I run the difflist file this is what I see in the to-install.txt
NOTE the word eject and install on the first line

Code: [Select]

eject          install#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit

What the heck am I doing wrong? It is doing the same on all my machines.

this is what I see in my difflist file

Code: [Select]

#!/bin/sh
# Name=difflist
# Author=Polack
# Script to generate a package list containing
# only packages added to a default installation,
# to aid in re-installation, and upgrade.
#

diff -dHrN -- installed-packages.txt new-installed.txt |grep \< |sed "s/< //" > to-install.txt

exit


I don't see the new line after exit. Have you generated both lists from genlist and newlist, and are both output files in your ~/ directory?

I will check the file for a new line after the exit statement. And yes both output files were created in the home directory.

[wedgetail]

YouCanToo
The suspense is unbearable any longer, what happened here? 

Old-Polack
I have been looking at this topic a number of times, since I have recently cleaned out 26 test installs I have a few spare partitions and want to look closer at trying to repeat your exercise.

Since it is a long time since I tried to make an rsync -av copy bootable.  I know I did make an rsync -av copy bootable and it seemed to work, well it ran.  I did not check closer and then many weeks later I thought I found a system that seemed to boot not from the partition I was expecting but from another one. (All history now, the system gone)

Next I ran mkinitrd from a chroot environment on the copied installation, to rebuild the initrd image of the default kernel, so it would use the proper new / partition and filesystem format when booting from its new home.

 
What happens if the quoted stage is left out?  Would the new stanza you added to the menu list still boot, seemingly ok?

I think I am under the misapprehension that an OS rsync copy can be run just by changing fstab in the copy and then make a stanza in the menu.lst?