RPMS - Build a Local Update Repository
The Benefit of zypper's -k (--keep-packages) Option
First, Read the NOTICE to the Right -->
With the release of openSuSE 11.0, when you perform and online-update from the openSuSE update repository, you are using zypper, regardless whether you use yast, the desktop updater applet or, of course, zypper itself. If you added the update repository with the '-k' (--keep-packages) option by using 'zypper ar -k http://repo-url alias', or modified the repository to include the option (zypper mr -k update-alias) all the rpms installed from that repository will be cached on your system for later use in a local rpm repository. This eliminates the need to download the same rpms again and again when updating other machines on your local area network. If your machines have a common setup, then it makes sense to use the --keep-packages option not just on the update repository, but on all repositories where RPMS common to multiple boxes reside.
The challenge becomes how to handle all of the saved rpms from multiple machines so that you can build a very useful local repository holding a complete, consolidated or merged, set of rpms from all machines. The solution is easier than you may think and well worth the effort if you update any significant number of machines.
Getting the Big Picture
For example, lets say you have several openSuSE boxes on your local network. They can be a mix of x86 and x86_64 boxes, it doesn't matter. If you have a dedicated server for your local network, that is the logical choice for holding all of the downloaded RPMS, if not, just choose a box to hold the RPMS for your network and create the local update repository on it. Looking at the diagram, you get the idea.
Identify the Puzzle Pieces
While zypper's ability to save local copies of all downloaded RPMS is a great addition, it isn't exactly useful without a little bit of effort.
Where are the RPMS saved?
For starters, where does zypper save all of the RPMS? … and … Are they saved in a manner that allows you to make immediate use of them? Well, the RPMS are saved by default under the directory '/var/cache/zypp/packages/', and that is where the easy part ends. Depending on the number of repositories you have the --keep-packages option set on, you will see an additional directory for each repository under /var/cache/zypp/packages. Something like:
[08:35 alchemy:/var/cache/zypp/packages] # l total 208 drwxr-xr-x 52 root root 4096 2009-03-20 11:25 ./ drwxr-xr-x 5 root root 4096 2008-07-26 17:37 ../ drwxr-xr-x 4 root root 4096 2009-01-30 04:23 backports/ drwxr-xr-x 3 root root 4096 2008-10-31 18:54 cc++/ drwxr-xr-x 3 root root 4096 2008-10-07 00:23 community/ drwxr-xr-x 4 root root 4096 2008-10-04 14:13 compiz/ <snip about 20> drwxr-xr-x 3 root root 4096 2009-03-20 11:25 updates/ drwxr-xr-x 3 root root 4096 2009-01-16 01:41 updatesgwdg/ drwxr-xr-x 3 root root 4096 2008-10-01 20:31 videolan/ drwxr-xr-x 3 root root 4096 2009-02-17 12:14 wm/ drwxr-xr-x 3 root root 4096 2009-03-13 04:43 wordpress/ drwxr-xr-x 5 root root 4096 2008-10-07 00:23 x11/
What's in the directories?
RPMS, of course, but under directories of their own with names only a computer could love. For example, some of the stranger openSuSE default repository names are:
<snip about 20> http:__download.opensuse.org_repositories_games_openSUSE_11.0_ http:__download.opensuse.org_repositories_home:_pikerhog:_utils_openSUSE_11.0 http:__download.opensuse.org_repositories_KDE:_Backports_openSUSE_11.0_ http:__download.opensuse.org_repositories_KDE:_Community_openSUSE_11.0_ http:__download.opensuse.org_repositories_OpenOffice.org:_STABLE_openSUSE_11.0_ http:__download.opensuse.org_repositories_openSUSE:_Tools_openSUSE_11.0 http:__download.opensuse.org_repositories_server:_database_openSUSE_11.0_ http:__download.opensuse.org_repositories_X11:_XGL_openSUSE_11.0_ http:__download.videolan.org_pub_videolan_vlc_SuSE_11.0_ <snip about 20>
Otherwise, the layout is fairly simple. In your update directory, you will find a directory layout of .../updates/rpm/{architecture} where architecture can be noarch, i586, or x86_64. In the remaining repository directories, there is no /rpm subdirectory so they look like .../repo-name/{architecture}. The complete list of architecture types you will find are i586, i686, noarch, SRPM, src, and x86_64. In each architecture directory, you will find the rpms themselves, just as if you were browsing one of the openSuSE repositories with your web browser.
You could just copy that mess of repository directories to a central location and use them as is, but that would result in a very ugly looking local repository. The problem is compounded if you haven't named the repositories exactly the same on each machine -- giving rise to the possibility that you will be storing multiple copies of the same RPM under different directories on your local update server -- not good.
Enough Already - How do we get started?
Before combining all the RPMS from the half-dozen or so boxes on the local network, choosing a logical directory layout for the local update repository made sense. After looking through what was there, I decided I wanted my local repository to have a single subdirectory for each possible architecture. That way when consolidating the saved RPMS from all the different machines, duplicates would take care of themselves by virtue of the filesystem without any additional effort on my part. The resulting layout looked like this:
00:01 nirvana:/home/backup/rpms> ls -1 openSUSE_11.0/ delta i586 i686 noarch repodata src x86_64
Consider Disabling Delta.rpm Use
Note: "delta" rpms relate to online "update" repositories only. The delta RPMS above are not a different "architecture" unto themselves, but since managing the delta.rpms will take an additional step or two to insure that the 'delta' rpms are made available in lieu of regular rpms, having them all in one directory simplifies things.
Also note that if your internet connection has the speed to download the non-delta rpm once, then disabling delta.rpm use is the recommended way to go since recreating the rpm from the delta.rpm is resource intensive (CPU, memory & IO). To disable the download of delta.rpms, make the following changes to /etc/zypp/zypp.conf
##
## Whether to consider using a .delta.rpm when downloading a package
##
## Valid values: boolean
## Default value: true
##
## Using a delta rpm will decrease the download size for package updates
## since it does not contain all files of the package but only the binary
## diff of changed ones. Recreating the rpm package on the local machine
## is an expensive operation (memory,CPU). If your network connection is
## not too slow, you benefit from disabling .delta.rpm.
##
# download.use_deltarpm = true
download.use_deltarpm = false
Putting the Pieces Together
Putting everything together requires just 3 tools: a little BASH, rsync and ssh. So, how much is a "little?" Don't sweat-it, I've done it for you. If you can copy and paste, your fine... First, let's get a roadmap together of what we are going to do so we don't get lost along the way. What we will do is:
- On each client machine:
- disable download of delta.rpm packages (if your internet connection is fast enough)
- enable rpm caching of update packages (one time only)
- parse each directory under /var/cache/zypp/packages/
- remove any older versions of the rpms (move them)
- build a list of files separated by architecture
- use rsync to transfer rpms to the local server by architecture
- pass control to the server via ssh so it can continue the process
- On the local server:
- configure http:// access to your repository (one time only)
- create a gpg key to use for signing the repo (one time only)
- remove any older versions of the rpms (move them)
- download the latest deltainfo.xml.gz from openSuSE
- parse and build a local deltainfo.xml.gz from the openSuSE master file
- create or update the repository metadata (.../repodata directory)
- sign the repository with the gpg key
- return control to the client and offer to delete the cache
Setting Up The Client Machines
Disable delta.rpm use
As mentioned above in the note about delta.rpms, the use of delta.rpms puts additional demand on the update process because before a delta.rpm is installed the regular rpm must be re-created from the information in the delta.rpm. This is different from a patch rpm, because once downloaded, a patch rpm is ready for install without any additional processing. Don't get me wrong, there is nothing wrong with delta.rpms, they can same a bunch of download time. However, because what I am interested here in minimizing the update time for client machines, I save time by not using delta.rpms.
It is a balancing act that depends on the speed of your internet connection. My connection is probably right around the decision point on whether to use delta.rpms or not. I have a 1 Meg. downstream connection which gives real life downloads of about 8-9 Meg. per minute so downloading even the full OpenOffice package of 250 Meg. with icons is not too bad to only have to do it once. So I disabled delta.rpm use. If I had dial-up, you bet I would be using the delta.rpms. To disable delta.rpm use, edit /etc/zypp/zypp.conf and make the following change:
# download.use_deltarpm = true
download.use_deltarpm = false
Enable Caching of Downloaded RPMS
Whether rpms are cached and save after they are downloaded is controlled by the "keeppackages" parameter in each individual repository configuration file. The repo config files are found in /etc/zypp/repos.d/ directory. The easiest way to check the keeppackages status of each repository is with grep:
grep keep /etc/zypp/repos.d/* /etc/zypp/repos.d/3111updt.repo:keeppackages=0 /etc/zypp/repos.d/cc++.repo:keeppackages=1 <snip>
There are three ways to change the keeppackages value to enable saving of rpms:
- when the repository is added to your system with the '-k' or '--keeppackages' option;
- by modifying the repo and specifying the '-k' or '--keeppackages' option; or
- by simply editing the repo config files in /etc/zypp/repos.d/ and setting 'keeppackages=1'.
What I have found works is just to set all repos to keeppackages=1 with a script and then change any back to 0 that you need to. Either one of the following will set all repositories to keep packages. Just copy and paste 'one' of them into an x-term (as root):
for i in $(zypper lr | sed '1,2d' | awk -F '|' '{ print $2 }'); do \
zypper mr -k $1 \
done
- or -
OLDIFS=$IFS
IFS=$'\n'
for i in $(ls /etc/zypp/repos.d/*); do \
perl -p -i -e s/keeppackages=0/keeppackages=1/ $i \
done
IFS=$OLDIFS
NOTE: After you create your local repository you DO NOT want to set keeppackages=1 for that repo on your client machines. [talk about your circular logic if you do....] Also note that to date, the only reason the IFS change is needed has been to accommodate the subpixel repository, that for reasons unexplained, includes whitespace in its default cache repository name.
Parse, Build and rsync Cached RPMS to Update Server
Believe it or not, this part is the easy part -- it is all done be a simple little bash script called 'zyppmerge'. The script does exactly what the heading says. It parses the /var/cache/zypp/packages directory using find, then builds the separate lists of rpms by architecture, and finally uses rsync to copy the files to your local update server. After you download the script, you will need to edit the following variables to match your local setup. By default the local update repository will be created on your server under the directory '/home/backup/rpms/openSUSE_11.X' where X is either 0 or 1 depending on the client openSuSE version. Here is the part of the script you will need to edit (manditory edits marked with »):
## variables backup host, directory and temp files
LOGFILE=/var/log/zyppmerge.log
OSVER=$(sed -e 's/^.*=[ ]//' -e '1d' < /etc/SuSE-release)
RPMDIR=openSUSE_${OSVER}
» REPOHOST=nirvana.3111skyline.com
REPOPATH=/home/backup/rpms/${RPMDIR}
» RUSER=david
TEMPDIR=/tmp/zyppmerge
TEMPSAVE=/var/log/zyppmerge
DELTABLD=/home/backup/rpms/data/xmlparse-srv
ZYPPCACHE=/var/cache/zypp/packages
zyppmerge.dev is part of the archive found here:
http://www.3111skyline.com/download/openSUSE/pkgmanage/pkgmanage.tar.bz2
NOTE: the $DELTABLD script 'xmlparse-srv' will be discussed below. Additionally note that my local repository is under /home to make sure it is on the /home partition to prevent filling up /. I have a very complete openSuSE 11.0 install and currently (supporting both i586 and x86_64 machines), my local repository has 4450 rpms and takes up 8.7G of space.
I put zyppmerge in my normal script directory and then create a link to it in /usr/local/bin. You can simply put the script in /usr/local/bin if you don't have anywhere else you normally put scripts. To make the process completely automated, you will need to configure passwordless access via ssh to your local update server from your clients. You will need to do this both as a regular user and as root accessing the local server via $RUSER@$REPOHOST.
SECURE PASSWORDLESS RSYNC: Basically, on the client, just run 'ssh-keygen -t dsa' as your regular user, then again as root. (just hit return for the password) Then just add both public_keys to a file "cat /root/.ssh/id_dsa.pub ~/.ssh/id_dsa.pub > keyfile", then just copy or add the contents of the keyfile to your ~/.ssh/authorized_keys file on your local update server $REPOHOST (rsync keyfile $REPOHOST:~/.ssh/authorized_keys)
You will now have passwordless ssh, scp, and rsync access to your local update server.
That concludes the client setup. Now just remember to run zyppmerge, now and after each update on the client
Setting Up The Server Side
Create the Repository Directories
Though the server side script will create the directories for you, you might as well determine where you want them to go and update the script variable now to make sure you don't have any surprises later. As mentioned earlier, I did not want my repositories on my root partition for size reasons so I put in on my home partition in the /home/backup directory I always have any way. Under /home/backup, or where ever you decide you want your repository, create an "rpms" subdirectory and then your version and data subdirectories under rpm. For repository naming convention consistency, I chose to use "openSUSE_11.X" as the version directory to keep it consistent with the openSuSE repo naming convention. The data subdirectory is where the xmlparse-srv script will go. If you want it somewhere else, just makesure you update the DELTABLD variable in the zyppmerge script so the clients will know where to find it. To create all the directories at once, just use:
mkdir -p /home/backup/rpms/{data,openSUSE_11.0,openSUSE_11.1}
**Note: NO spaces are allowed within the braces above. It will create:
12:50 nirvana:~> ls -1 /home/backup/rpms/
data
openSUSE_11.0
openSUSE_11.1
Configure the "xmlparse-srv" Server Script
The xmlparse-srv script is part of the archive found here:
http://www.3111skyline.com/download/openSUSE/pkgmanage/pkgmanage.tar.bz2
Once you have the script downloaded to /home/backup/rpms/data, edit the script to match your environment. (There are no manditory edits here unless you are moving things from the default directories):
#define and set variables
let CLINE=0 DLINE=0 BLOCKNO=0 HEADERLINES=2 NEWDELTA=0 PATCHINFO=1
declare -a ENTRY LOCDELTA
OSVER=${1:-11.0}
OSDIR=openSUSE_${OSVER}
REPOLOC=/home/backup/rpms
DATADIR=${REPOLOC}/data
XMLFILEGZ=${2:-${DATADIR}/deltainfo.xml.gz}
XMLFILE=${XMLFILEGZ%.gz}
NEWFILE=${DATADIR}/${3:-out-$(date '+%s').xml}
ERRLOG=${4:-${XMLFILE%%.*}.err}
SUSEDELTA=rsync://ftp5.gwdg.de/pub/opensuse/update/${OSVER}/repodata/deltainfo.xml.gz
DELTADIR=${REPOLOC}/${OSDIR}/delta
METADIR=${REPOLOC}/${OSDIR}/repodata
FOOTER="\n"
BLOCKDONE=no
DEBUG=no
The additional command line parameters listed above: $2, $3, $4 can be ignored here, I used this script to parse xml files other than deltainfo.xml.gz. When I do the next script update, I'll remove them and provide an updated script, so check back from time to time.
Configuring Apache2 to Provide http:// Access to the Repository
Next you need to configure apache2 to provide access to your local update repository. If you already have a working apache2 install, just edit your /etc/apache2/httpd.conf.local and create a directory definition for the /home/backup/rpms location. If you haven't, then see my apache2 setup information on the openSuSE Server Setup page, and then come back here. (you can ignore the secure https setup section for now, unless you plan to use it).
The following provides two examples of how to add a bit of protection to the directory as well. The first is just the minimal protection of requiring a valid user or restricting access to your local subnet, the second, adds url re-writing to use secure http (https) for outside IP address, requires authentication against a database, and denies access to a number of TLDs I have had problems with in the past:
BASIC:
#
## skyline local update repository directory
#
Alias /update/ "/home/backup/rpms/"
Alias /update "/home/backup/rpms/"
<Directory "/home/backup/rpms">
Options +Indexes +FollowSymLinks
IndexOptions FancyIndexing IconsAreLinks FoldersFirst
AllowOverride AuthConfig Options FileInfo Limit
Require valid-user
Order allow,deny
Allow from 192.168.16.
Satisfy Any
</Directory>
SECURED:
Alias /update/ "/home/backup/rpms/"
Alias /update "/home/backup/rpms/"
<Directory "/home/backup/rpms">
Options +Indexes +FollowSymLinks
IndexOptions FancyIndexing IconsAreLinks FoldersFirst
AllowOverride AuthConfig Options FileInfo Limit
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REMOTE_ADDR} !^192\.168\.16\.
RewriteCond %{HTTPS} !=on
RewriteRule .* https://%{SERVER_NAME}%{REQUEST_URI} [R,L]
</IfModule>
AuthType Basic
AuthBasicProvider dbm
AuthDBMType DB
AuthName "Restricted-Files"
AuthDBMUserFile /usr/local/lib/apache2/accessdb
Require valid-user
Order allow,deny
Allow from 192.168.16.
Satisfy Any
</Directory>
After making the changes, reload the apache2 configuration with "rcapache2 reload" and test access. If you have access to your new repository, then add it as a repository on your system and modify the repository to enable auto refresh and to set the priority of the repository lower than the normal openSuSE update repository so when electing which repository to pull the rpms from, your local repository has higher priority (lower number) than all other repositories. After that, your done, enjoy watching massive update fly!
Last-Step: Adding Your New Update Repository to Your System
Your local update repository works just like any online repository. To add your local repository to your system and change the refresh and priority, enter the following (example):
zypper ar http://www.3111skyline.com/update/openSUSE_11.0/ localupdt
zypper mr -r -p 15 localupdt (See note on priority below)
Do NOT use the '-k' option on your new repository -- It would be kind of pointless.. Additionally, the priority of your local repository must be numerically lower than all of the other repositories (meaning higher in priority), otherwise you will still be downloading packages from the regular repositories. If you are planning to simply update a client solely from your local repository, then you can disable all other repositories and leave the priority of the local repo at the default. If you want to leave all other repositories enabled, just make sure your local repository has the lowest number for priority. Further, when you are ready to download a new set of updates from openSUSE, disable the local update repository temporarily, do the update, and then re-enable your local update repository. Otherwise, zypper will ignore new online updates due to your local repository having the highest priority.
Also note, the priority preference concerning whether zypper will ignore newer updates from one repository if an older version of the package exists in your local repository has changed over the past several factory releases of zypper and yast. Regardless if you stick to the disable/re-enable directions above, you will be fine.
Now you are ready to go. To perform updates from your local repository, use the '-t package' option. Example: 'zypper up -t package -r localupdt'. For some reason, the rpms in the local repository do not retain their 'patch' status so performing a 'zypper up -t patch' (the default for zypper up) does nothing. I still don't have an answer for this one, but maybe someone can help us out here. Since the priority for the local repository is higher than the others, each time you issue the zypper up -t package or zypper in commands, you will be checking against the local repository first, and then only downloading rpms from the internet as needed. After an update to a client, don't forget to run zyppmerge to update your local repository with any new packages. Even with a large number of packages downloaded, the merge of new rpms from the client and update of the server repository won't take more than 20 seconds.
Good luck, have fun, and if you find any bugs or errors, drop me a line so I can fix it. (that includes typos as well) Thanks.
