How can we chroot to a build environment

[Archie]

Heya fellow packagers.

I don't know how standard our build servers are from each other, and I'm sure each one is different. I've only been packaging for PCLinuxOS for about half-a-year and prior to that, I've been a few something else.

The idea of a building RPMS on a chrooted environment was suggested to me on G+ by a friend whose name shouldn't be mentioned in our forum. I've pondered on the idea for a while but always have to put it back because of tasks I do at present.

Since last September, I must have rpmorphan, dupeclean, bleachbit and reinstalled countless time because my install was accumulating so much development files and libraries installed by a previous build.

Recently, I have remastered a build environment based on Minime 2012.02 (and an updated Test4) and use those as LiveHD so after a build and copying the SRPM and RPM to my local repo on a USB HD and reboot, the build environment would be "clean" again.

Inspite of numerous reads on how CentOS and Slackware and Red Hat are able to chroot to RPM packaging environments, I am still in the dark on how this can be done in PCLinuxOS.

Has anyone setup a chrooted RPM build environment using PCLinuxOS? If so, please share with the community the steps you took.

Where? How do we start? I do hope afterglow77 can chip in his expertise and knowledge to help the PCLinuxOS community.

Peace and much respect,
Archie

[AS]

Hi Archie,

a chrooted environment need a replica of the root filesystem:

/chroot-path/bin
                  |
                  /usr/bin
                  |    |
                  |    /sbin
                  |    |
                  |    /lib
                  |
                  /lib
                  |
                  /and so on ...

Once you chroot() to /chroot-path, you can't access anything outside the /chroot-path tree, that means that you will need to include other things, like /etc/passwd ...

using a rolling/dynamic distro like PCLinuxOS, it means that you will need to update that replica-filesystem quite often ... IMO this will be a major problem, an error prone task.

chroot env born as a security feature, to prevent access to the main filesystem, this make sense for some apps, like an http web server, but IMHO not so good for development/testing. Also, CentOS and RedHat are quite stable ... using lot of "stable" packages that in PCLinuxOS would be considered "outdated" ...

Looking forward for others opinions! 

AS

[daniel]

More space will help, maybe...

[Neal ManBear]

Our way of packaging, i.e. in the user's /home, was developed as a way to avoid packaging in /. It is a safer way to build RPMs.     

Perhaps chroot can be used, too, but wouldn't it require more space? More/extra work? Or am I misunderstanding the situation?     

[Archie]

What we really need is a "clean" environment after every build. IMO this will greatly decrease missing out on BuildRequires and Requires that may already be on a system during the build and testing.

Perhaps chroot make not be the ideal solution. Right now, my way is to boot to a stock Minime that I remastered with only pkgutils added. This way, I am sure that any development files, libraries and Requires missing will be included in the specfile. I think this is a good way to minimize the "neglected" dependencies during an install and we will have less Broken Packages.

More important to me is ALWAYS CLEAN environment.

[kjpetrie]

In a sense, this is what we already have in that we build in a dummy environment under our home. The only reason I can see for using chroot would be to have a 32-bit environment buried in a 64-bit system for checking 32 and 64-bit builds at the same time on the same machine.

But I think that's easier done with two machines, which could be two VMs, as it gives greater confidence the two environments have not interacted in some unintended way.

Edit:
However, I realise your real purpose here is to encourage a packager with a background elsewhere to explain how it could be done and discuss the possible benefits, and I'm not one to shut my eyes to good advice, so I'd be interested to hear more.

However, it might or might not be compatible with pclos standards, but we can't know that until we've heard it and those in authority have had a chance to review it. We have to be clear however, that any such discussion is not intended as criticism (in the negative sense of that word) of current procedures or management and it mustn't become that. Oh, hang on, why am I addressing that to a mod?

[AS]

What we really need is a "clean" environment after every build. IMO this will greatly decrease missing out on BuildRequires and Requires that may already be on a system during the build and testing.

Adding to kjpetrie's comments because I agree with his view. As it seems Archie is focusing on "dependencies", two separate guests for each target arch should help, BuildRequires dependencies belong to a development guest, while Requires dependencies belong to a test guest, in fact I'm using 4 guests 2 for 64bit and 2 for 32bit, the host remain untouched. Furtherly, sharing the build environment tree across all the virtual guests will be of help. The drawbacks are virtual machines performance ... There is always a "tax" to pay 

[Archie]

Thank you all for replying. I am getting in a better light in this matter.

@kjpetrie, I wish I wrote it the way you did. I am not going to rattle what we already have in place but would like to know from others ... from you, how else we could improve on what we have.

@AS, my "tax" is constant reboots to change from one to another.

I've also been reading a bit more on mock. IMO mock should just stay with Fedora unless there's something else I didn't see ... or understand (which is quite a lot)

[Just17]

Recently, I have remastered a build environment based on Minime 2012.02 (and an updated Test4) and use those as LiveHD so after a build and copying the SRPM and RPM to my local repo on a USB HD and reboot, the build environment would be "clean" again.

You haven't mentioned it, so ......  are you using 'persistence' in this live environment so that you can take a break from a session and return to it retaining "changes"?
When that packaging work is completed then delete the "changes" and return to the 'clean' boot for the next session ........  or is it that the changes made during a session do not ALL get saved during 'persistence'?

Only me wondering how you are doing things is all .... 

[Archie]

Recently, I have remastered a build environment based on Minime 2012.02 (and an updated Test4) and use those as LiveHD so after a build and copying the SRPM and RPM to my local repo on a USB HD and reboot, the build environment would be "clean" again.

You haven't mentioned it, so ......  are you using 'persistence' in this live environment so that you can take a break from a session and return to it retaining "changes"?
When that packaging work is completed then delete the "changes" and return to the 'clean' boot for the next session ........  or is it that the changes made during a session do not ALL get saved during 'persistence'?

Only me wondering how you are doing things is all .... 

Actually, no. I did not add persistence. But that would be a good idea that I would try. Although I did notice that booting into LiveHD only allows me 1Gb space. And that's pretty tight for medium size packages; never mind those humongous monster packages. I will be over at the appropriate section soon.

[afterglow77]

Thank you all for replying. I am getting in a better light in this matter.

@kjpetrie, I wish I wrote it the way you did. I am not going to rattle what we already have in place but would like to know from others ... from you, how else we could improve on what we have.

@AS, my "tax" is constant reboots to change from one to another.

I've also been reading a bit more on mock. IMO mock should just stay with Fedora unless there's something else I didn't see ... or understand (which is quite a lot)

Mock is really just the beginning, it could stay with Fedora and that is fine but the philosophy of being able to easily build multiple archs as Fedora does, x86_64 and i686 (i586 in you case) and then to be able to go on to support Arm and various other arch options when they arise from one src rpm is something different and something PCLinuxOS needs (of course in my opinion). To be able to create a chrooted environment that can be packaged (itself) and moved anywhere, or generated on the fly has it's benefits. An example of mock (or something like it) being a stepping stone would be koji:

http://koji.fedoraproject.org/koji/

koji uses mock in the back end and allows multiple developers to create packages fast and effienctly when they may not have the hardware to do so. They have koji build servers for Arm and for PPC (for those of us with G5s still) as well. Fedora also has a revisioning system where they keep source packages, patches and specs, that the build server can pull from. So let's say packager A updates glibc and it ends up breaking various packages. So packager A is smart enough to run in his clone of git (svn checkout or what have you) a script finding a majority of the requires where there's possible breakage, then goes on to find in the change log the various email address of the packages maintainers. He would then go on to use a script like fedora's rpmdev-bump (rpmdev-bump -c "Rebuild for new Glibc 2.15 $VARSPEC) bumps the release and sends an email out to all the maintaners of the packages. Getting a quick response back from all of them (in this perfect world) he commits the bumped spec file to git and tells the build server to  search for specs with "Rebuild for new Glibc 2.15" in their changelogs and rebuild them with the new Glibc 2.15 Requires. A few hours later both archs are done, as opposed to in some cases might be a few days later for just i586. There's also ways with chroots you can build packages asyncronous I really don't have to go over the benefits of that for any packager that's been around a few days. So yes Fedora can keep all that stuff, but I think you guys if you truly understood it, could really benefit from something like it of your own. Maintianing two source rpm braches as I have noticed you guys are doing quite frankly exausts me and scares me from doing any packaging myself for you guys. I don't mean that as an insult more of a testament to how hard you have to be working, for which I give you kudos but in my opinion effienctly sometimes supercedes work.

[Neal ManBear]

I think I will stick with PCLinuxOS standards.     

[Hootiegibbon]

I think I will stick with PCLinuxOS standards.     

Neal,

I think this is good news on two fronts

1. it keeps things simple, adding an extra level of complexity to a process not always bad, but when it gives everyone a learning curve I think it is

2. why would anyone want to change what is proven to work? (archie - I hear ya but is it really that much of a chore to reboot?)

I think that those of us here a while, both those that have been here for a while , and those who have been gone and subsequently returned under a different nick, should remember what happened when others attempted to change what works, and how long it took to fix things after

afterglow77,

It does not appear that the chroot environment is not gonna happen as a standard here, so I would save your energies with a "hard sell" and if you still wish to package, learn the stable PCLinuxOS way - which is what I intend to do once I can work it into the distractions that family life brings.

Jase

[kjpetrie]

I hear what you're saying, afterglow77, but we don't intend to maintain separate source packages so far as I know, for the two current archs. Texstar, Neal and O-P require all new SRPMs to build in both. Adding ARM would be a lot of work, as we'd have to do that from scratch, and maybe there's something to be gained if that could be automated (well, there certainly would be, wouldn't there?), but the Lead Developer(s) can only implement what they know and are happy with, and if they are to adopt something else they will have to do it in their own time when they see the need and are ready.

So you may well be right in what you're warning of, but I think we have to let those in charge decide what works here and how things are to be done.

Pclos has a history of adopting features from other distros where they prove useful, so it might happen one day if it's really needed, but when that day comes they will examine all the options and adopt the one they like best. A public forum is not the place to persuade them, though. If you want to make the case to Neal and O-P you would have to do so in personal messages (if they're willing to discuss it which they might not be) and you also need to remember that they're looking after the job for Texstar who needs this distro to stay as he wants it for when he takes over again, so nothing's likely to happen without his approval, and he's not well and might not be up to major decisions just yet. So be patient and be aware this might not be the best time.

[Archie]

(archie - I hear ya but is it really that much of a chore to reboot?)

@Hootie ... no, it's never that much of a chore to reboot to a clean environment ... great stuff, that change_dev= ... but when you got a package you eagerly want to try out because you're convinced that you got it all right, TWO minutes seems like a very long time! 

The PCLinuxOS way IMO is a very good way since most of the tedious stuff have been automated on a single right-click of the mouse ... but I'd also like to know how Arch build their packages, how Red Hat copes with anyone being able to use mock, how Debian does it. It's just me, I'm interested to know. Knowing these things IMO would make me a better PCLinuxOS user.