Remote OpenBSD/i386 upgrades from 3.3 and before
OpenBSD v3.3 was the last release of OpenBSD/i386 using the a.out executable
file format. Starting with v3.4, as part of a number of security
improvements, there was a conversion to the more modern and more
flexible "ELF" format.
Unfortunately, the two formats are not directly compatible. This causes
large problems when attempting to upgrade by remote. Most can be
overcome with some work, but one is a show stopper: the fact that the
3.3 and before boot loader is unable to load the new kernel, and the 3.3
and before installboot is unable to install the new boot loader that
could load the newer kernel.
For OpenBSD 3.4 and 3.5, a special kernel was made available,
bsd.rd-a.out,
which would help you over that barrier. However, this only helps if you
have access to the console on the system -- if your machine is at another
location, you won't be able to run the bsd.rd-a.out kernel.
Fortunately for us, OpenBSD developer Dale Rahn had a large number of
machines he needed to remotely upgrade from 3.3 to ELF-compatible
systems. Physically getting to each machine was almost out of the
question, so he wrote
bsdaoutcopy.c,
(local copy), a <5k source file that
converts an ELF kernel into an a.out-format file which can be
successfully loaded by an a.out-compatible boot loader. This
solves the "show-stopper", so a remote upgrade is now possible.
BUT...
That doesn't mean it is trivial. This is a really scary process, if
you make an error, you will end up with a broken system, and need to
make an emergency trip out to the system you were trying to upgrade.
However, it can work. This document will not be a step-by-step guide,
but rather a list of tasks you will have to perform. If you don't
understand this process, you have some work to do before attempting
a remote upgrade this way.
Since the ELF transition took place back on version 3.4, and as I write
this, 3.7 was just released, I am going to presume any machine you are
updating this way is waaay out of date. Rather than trying to update
in the more traditional way, I'm going to assume you are doing mostly a
remote wipe/reload. We won't actually be wiping the drive or removing
absolutely all old files, but we will be reconfiguring the machine from
scratch.
You need to practice this process locally before attempting it on your
not immediately accessible machine. What follows here is the process I
used twice, once completely successful, once where I forgot to set the
root password, but after going to the machine and giving root a PW, all
was well, that was the only obvious error. However, this process is
not tested as well as my FAQ articles, I do not wish to spend much more
time on this. Most of all:
This process is not endorsed or supported by the OpenBSD
project. Do not expect support of this process from OpenBSD, Dale,
or me.
If you find a flaw or an obviously correct improvement in this process,
let me know, but I will not be responsible for whatever happens to your
machine, your hair or your job for following this process. You are
completely on your own. You have been warned.
The Issues
- Old kernels will not run new ELF binaries
- New kernels will not (initially) run old a.out binaries
- /etc directory is horribly out of date
- Old boot loader won't load new kernel (normally)
- New boot loader won't load old kernel, and can't be installed from
old system
Tasks to be performed
- Make travel plans. If this doesn't work, you will be going out
on-site to get your machine back up and running. Ask me how I
know.
- Jump to root equiv. You must do this before you (eventually) nuke
the functionality of su and sudo.
- Download and compile the bsdaoutcopy.c file on your old OpenBSD box.
- Run it against a new kernel. Don't copy the result to /bsd yet,
however. Nothing else is ready to go, if the system were to reboot now,
best it comes up on the old kernel.
- Make a backup copy of your /etc and /var directories.
- Remove all packages and third-party binaries.
- Make a copy of the primary system binaries. I copied the /bin and /sbin
directories to my ~/bin. You will need these later, at least to reboot
the machine once you install the new binaries.
- Unpack the new etcXX.tgz file in the root of the file system, using
the "xzpf" options of tar. That 'p' option is critical. This will
overwrite your existing /etc directory and some files in the /var
directory.
- Rebuild at least the following functions:
- Network configuration (hostname.if, hosts, myname, mygate,
resolv.conf). Get this right.
- User information. The default passwd files have a root user with
no password. If you forget to change this password or import your
existing users at this point, you will discover that OpenSSH won't
let root log in without a password. Ask me how I know...
- Config sudo or su or whatever you need to jump to root abilities.
- OpenSSH keys. You probably want to copy over your old ssh keys so
you have the same keys after your upgrade.
- Copy over your a.out-converted kernel to /bsd. We are getting close
to reboot here.
- Go do a good deed for someone. You need the karma.
- Unpack baseXX.tgz, again using the 'p' option to tar. At this
point, your machine is almost unmanageable. You have broke all binaries
in the /usr, /bin and /sbin directories. Hopefully you remember where
you put your copy of the /bin and /sbin directory. I goofed this up
once, by dropping the contents of /bin and /sbin into ~, rather than
~/bin. This was difficult to figure out as the 'ls' command was now
not functioning, but tab completion helped me find them and recover the
process.
- Reboot your machine by using the reboot command from your saved
copy.
- Don't forget to breathe. Oxygen is good. Travel won't kill ya.
- With luck, soon, you will be able to ping your machine. With
even more luck, shortly after that, you can ssh into the machine. If
so, wonderful, but there is still work to do, don't cancel your travel
plans yet. It is still possible to screw it up.
- ssh into the computer to complete the process.
- Delete your copy of the old binaries you saved (and used) earlier.
A moment ago, they were critical. Now they could end up frustrating
you. You don't need them anymore.
- Unpack the rest of the *.tgz files, again using the 'p' option.
- Copy over the unaltered bsd kernel to /.
- Copy /usr/mdec/boot to /boot.
- Run installboot from /usr/mdec to properly install the new boot
record. Note the word "properly". Make sure you did this right, and
make sure there were no error messages. Yes, that a.out-converted
kernel could be used indefinitely, but if you forget to a.out-convert
a future kernel, you will lose the machine in the future. Do it right
and fix it now. Besides, you just copied over the new /boot file,
you have to.
- Reboot again. See notes above about breathing.
- If the machine comes up again, you can now cancel your travel plans.
At this point, you now have a successful upgrade semi-completed. You
still have to reinstall and configure any apps you wish to run on the
system, but at this point, it is the same as any other remote upgrade.
Future upgrades can be done completely normally (unless there are other
flag days like the a.out -> ELF conversion, of course).
Holland Consulting home
page
Contact Holland Consulting
since Jun 11, 2005
Copyright 2005, Nick Holland, Holland Consulting.
Permission granted to use, distribute and modify as per a standard
two-term BSD license. Short version: Do what you want with it, but
acknowledge that I wrote it, not you, not someone else, and don't
come running to me if something goes horribly wrong.
$Id: aout-up.html,v 1.5 2005/07/02 03:09:02 nick Exp $