My first post will be on my attempt to create a minimal Raspbian image to run on my Raspberry Pi. The standard Raspbian image is bloated with all kind of tools and software, to make it easier on new users to use their Raspberry. I think this is useful if you want to quickly enable people to start doing stuff (browsing, programming in whatever language, play around with the GPIO ports etc). However, if you know what you want with your Pi (use it as a home server, install it in your home made arcade machine, whatever), a lot of the tools installed are useless and you might want to consider cleaning it up.
Of course there are images available with a clean, minimal Linux distro for the Raspberry. There are even small images based on Raspbian itself. However, my goal is not just to have a clean image, but also to learn from it; how can I install a system from scratch? What packages are required? And for what purpose? Also; who is to say that the creator of a random small image didn't install a key-logger on it? Anyway, I wanted to install Raspbian from scratch, on a clean, empty image, using just dpkg and apt.
Note that I do not want to compile anything if possible. It is fun and interesting to install Linux by compiling everything, however, I think compiling on a Raspberry Pi will be much too slow. And doing it using cross-compilers will teach you more on compiling, configuring and optimizing stuff, than on installing stuff. Plus, the packages in the Raspbian repository are probably configured and optimized specifically for the Raspberry Pi. If you are interested on installing Linux from source code, I would recommend not doing this on a Raspberry Pi. But have a look at Linux From Scratch. That's a nice project and you can learn a lot by doing it.
In this blog post I will show you what I did to create an empty image, prepare it for installation, install basic libraries/packages and dpkg and apt. Finally I will attempt to make the image bootable, meaning you can start your Raspberry Pi with your own image and not much more installed than what is needed to install applications from the Raspbian repository, through apt.
About package managers
First a short word on what dpkg does. There are a lot of ways to get software installed on a Linux machine. For us this going to be done through packages. A package contains a few things: the files (executables, libraries, configuration etc), installation and removal scripts and metadata with, amongst others, the list of other packages that this package depends on. For Debian derivatives (like Raspbian), dpkg is the package manager. It keeps track of what packages are installed and checks whether all dependencies are met before installing a new packages. Besides easing the checking whether you have all dependencies installed, this also helps a great deal when you want to remove software.
However, also a short word on what dpkg does not do. It does not have any notion of repositories (servers containing packages) and will therefore not automatically fetch all dependencies. There is another tool for that, apt. Basically, you configure apt with a list of repositories (in the form of URLs) you want to use. When installing a package through apt, it will check what the dependencies for that package are and download these dependencies and the requested package itself from the repository. Then it calls dpkg to install all these packages. Note that apt does not replace dpkg, but rather uses dpkg to install packages. Apt merely saves you the trouble of having to find out all the dependencies and the locations of these dependencies.
So why is this important? First of all; we want to install all software through dpkg, to make sure that at the end, it knows exactly what is installed. Secondly; because we don't have apt installed on the (new) image yet, we will have to manually download the first packages up until we have installed apt. Basically, we will start with doing 'apt-get install apt' manually.
Last, but not least; there are some dependency problems in the beginning. Two packages mark each other as a dependency (creating a chicken and egg problem). Furthermore, most packages contain installation scripts, which also require some tools to be installed. These tools are not marked as a dependency because they are not needed by the software in the package, only by the installation script. We will run into problems because of this; we cannot install a basic C library without these tools and we cannot install these tools without this C library. Again with the chicken and the egg. Luckily, I studied biology, so I know the egg was first, thus solving this was peanuts (okay, I'll be honest, it took me quite a while).
It is easiest to do this project on a Raspberry Pi with the regular Raspbian image installed. Also, having an internet connection is quite handy. This way, you have access to the Raspbian repositories and you can use any installed packages immediately. Make sure you have enough storage space available though.
I cannot explain everything in detail. What I will try to do is explain what needs to be done (in 'normal' English) and show you the command I used to do this. These commands might not work exactly the same for you. If you want more information on the commands used, please use their man pages, Google and/or Wikipedia. Some of the tools used are probably not installed by default in Raspbian, but you can install everything I use here through apt. I assume you know how to use apt.
I run all my commands as 'root' user. This has a serious security problem: one typo can screw over my whole system. However, it saves me from typing 'sudo' before almost every command (and running something with sudo and making a typo has the same effect!). You can become root in Raspbian by typing 'sudo su -'. To minimize any risk, I did this on a fresh install of Rasbian, with no valuable data that can be destroyed. If I screw up, I can just write the standard Raspbian image to my SD card and start over.
Step 1: prepare a new image
- Create a 512 MB raw file to contain the image (of course you can make a larger image, but you can also enlarge your partitions once the image is on an SD card):
dd if=/dev/zero of=/root/image.raw bs=33554432 count=16
- Attach the image to a loop device. This way, the system can access it as a block device, just like a real SD card:
losetup /dev/loop0 /root/image.raw
- Now create two partitions. The first partitions should be a vfat (vat32) partition of around 60 MB. The second partition should be a ext4 partition of the remaining space. Note that the first partition has to be a vfat partition (although you can choose to make it bigger). You may choose to create more partitions or use a different filesystem for the second partition, but for now I recommend using this configuration as it is the same as the normal Raspbian image. It is easiest to use fdisk to do this interactively from the command line. Alternatively it might be possible to run gparted and do this in in a GUI. Please use google to figure out how to create the partitions. I use sfdisk from the command line, you could try what I did:
sfdisk --head 4 --sectors 16 --unit B /dev/loop0 <<EOS
1024 64512 c *
- Make the system detect your newly created partitions:
kpartx -a /dev/loop0
- This will make your partitions available under /dev/mapper/loop0p1 and /dev/mapper/loop0p2. Now we must create the file systems on those partitions. You will need to install the package 'dosfstools' (apt-get install dosfstools) before you can create the vfat file system:
mkfs -t vfat /dev/mapper/loop0p1
mkfs -t ext4 /dev/mapper/loop0p2
- Create the mount points and mount the file systems. Also, we will bind /proc and /dev in the new partition and create a tmp directory. These will be needed while installing packages:
mount /dev/mapper/loop0p2 /target
mount /dev/mapper/loop0p1 /target/boot
mount --bind /dev /target/dev
mount --bind /dev/pts /target/dev/pts
mount --bind /proc /target/proc
chmod 1777 /target/tmp
mkdir -p /target/var/log
chmod 700 /target/root
Now, your newly created image is ready to be used. However, if you reboot your Raspberry Pi, it will not be attached to the loop device and no partitions will be mounted. You can copy the commands from step 2 and 4 and all 'mount' commands from step 6 to a bash script (keep them in order!). This way, you can simply run this script to prepare everything for installation. You can 'detach' the image file by doing the reverse; unmount everything, release the partition mapping and release the loop device. After that you can move the image file around like a normal file:
kpartx -d /dev/loop0
losetup -d /dev/loop0
Step 2: download packages and prepare image for installation
There are two ways of downloading the initial packages:
- Download the tar file I have available here (md5sum: 3b4236ffb0fb97c36dda6179db4d7359) and unpack the tar to /root/packages. This is really quick and you have the same package versions I used. The drawback is that over time, these packages will really become outdated (up until the point that I decide to remove this option all together).
- Or download the most recent versions directly from the Raspbian repository. This will require some manual work to find out where to download them from, but you will install up-2-date packages right from the start. To do this, follow these steps:
- Create a location for downloading the first packages:
- Download all packages that we will install without apt. We download this directly from the Raspbian repository, for which we can find the location on the Raspbian website. At the time of writing the location is http://archive.raspbian.org/raspbian/
- Here we can find a directory 'dists', containing the different versions of Raspbian. We are working with Wheezy (but this can change in the future of course), so enter that directory. Now choose for the main repository and there for the binary-armhf packages.
- Here you can find a file Packages, which contains the list of packages in this repository. A shortcut: http://archive.raspbian.org/raspbian/dists/wheezy/main/binary-armhf/Packages Open this file with a text editor. Here you can search for 'Package: apt'. You can see the 'Filename' field, which points you to where the package can be downloaded from (prepend the location with the repository root: http://archive.raspbian.org/raspbian/ ). Download this package:
cd /root/packages wget http://archive.raspbian.org/raspbian/pool/main/a/apt/apt_0.9.7.9+rpi1+deb7u1_armhf.debThese commands are just an example. The location can change in the future if there is a new version of apt, so create the url based on the information in the Packages file!
- Are we there yet? No, by far not. As I've told before, packages can have dependencies and we need to download and install those packages first. In the 'Packages' file, look at the 'Depends' field (and at the 'Pre-Depends' field if it is present). There is a list of other packages which need to be installed before we can install apt. You can lookup those packages and will find that they too depend on packages. Basically we will have to find and download all dependencies (of dependencies of dependencies of dependencies.....) until we reach a level where the packages have no dependencies any more. And then we are still not there yet, because, besides apt, we will need some other tools/packages to be able to run the installation scripts: core-utils (contains basic commands like cp), sed, grep and findutils. You will need to lookup and download those packages and their dependencies too. To save you the trouble, here is the current list of packages needed:
apt coreutils debianutils dpkgIn short; find these packages and their download location in the Packages file and then download them to the directory /root/packages/
findutils gcc-4.7-base gnupg gpgv grep libacl1 libapt-pkg4.12 libattr1 libbz2-1.0 libc6 libc-bin libgcc1 liblzma5 libreadline6 libselinux1 libstdc++6 libtinfo5 libusb-0.1-4 multiarch-support raspbian-archive-keyring readline-common sed sensible-utils tar zlib1g
Prepare image for installation
Because dpkg will 'chroot' into the /target/ directory, there should be some directories and files present which dpkg needs to keep track of what is installed etc. If you don't know what 'chroot' is, please look it up. In short; chroot will 'fool' an application into believing some directory is the root of the file system (so while the application thinks it writes to /etc/somefile, in 'reality' it writes to /target/etc/somefile).
Here are the commands for creating the locations required by dpkg:
mkdir -p /target/var/lib/dpkg/updates
And last but not least; all installation scripts require (/target)/bin/sh, but of course we don't have that available yet. There is a package bash-static, which contains a statically linked bash (so it does not need any libraries to be installed). Let's install that on the host system and copy it to the new image and link sh to it. Later on this will automatically be overwritten when we install the normal bash package.
apt-get install bash-static
cp /bin/bash-static /target/bin/bash
ln -s bash sh
Step 3: install dpkg and apt
We are now ready to start installing packages to the empty image. As said before, we have several chicken and egg problems here:
- Several packages have no dependencies specified, but their installation scripts use commands like 'cp'. They are not yet installed on the new system and they require libc-bin to run. Therefore we will only unpack libc-bin and these other packages and tell dpkg to ignore these dependencies while installing other packages. After we have installed coreutils (which contains 'cp'), we will install libc-bin and friends normally.
- libc6 depends on libgcc1 and libgcc1 depends on libc6. This is easy to solve: we just tell dpkg to ignore this dependency for libgcc1 (which we need to do anyway, because libc6 is only unpacked at this stage).
- raspbian-archive-keyring has an installation script that uses binaries form the apt package and the apt packages requires raspbian-archive-keyring to be installed (both by the package metadata (the 'Depends' tag) and the installation script). So we will do the same trick as before; unpack raspbian-archive-keyring, install apt and ignore the dependency and then install raspbian-archive-keyring normally. And the same trick applies to sensible-utils and debianutils
But let's start at the beginning. We instruct dpkg to install packages. The --root flag indicates that the packages should not be installed to the system root (/), but to chroot to the provided path instead. I already explained what we should do, so let's just give you the commands. Note that the order of installing the packages is imported because they depend on each other. Note that I do NOT supply the full filename of the packages, because I do not know what version you have downloaded. You should complete the filename! Normally you can to this by pressing tab at the end of each line (bash will autocomplete the path you typed):
dpkg --root /target --install /root/packages/gcc-4.7-base
dpkg-deb -x /root/packages/libc-bin /target
dpkg-deb -x /root/packages/libc6 /target
dpkg --ignore-depends libc6 --root /target --install /root/packages/multiarch-support
dpkg --ignore-depends libc6 --root /target --install /root/packages/libgcc1
dpkg --ignore-depends libc6 --root /target --install /root/packages/zlib1g
dpkg --ignore-depends libc6 --root /target --install /root/packages/libbz2-1.0
dpkg --ignore-depends libc6 --root /target --install /root/packages/liblzma5
dpkg-deb -x /root/packages/libselinux1 /target
dpkg-deb -x /root/packages/tar /target
dpkg-deb -x /root/packages/dpkg /target
dpkg --ignore-depends libc6 --root /target --install /root/packages/libattr1
dpkg --ignore-depends libc6 --root /target --install /root/packages/libacl1
dpkg --ignore-depends libc6,libselinux1,dpkg --root /target --install /root/packages/coreutils
dpkg --ignore-depends libc6 --root /target --install /root/packages/libselinux1
dpkg --ignore-depends libc6 --root /target --install /root/packages/tar
dpkg --ignore-depends libc6 --root /target --install /root/packages/dpkg
dpkg --ignore-depends libc6 --root /target --install /root/packages/grep
dpkg --ignore-depends libc6 --root /target --install /root/packages/sed
We have now unpacked libc-bin and libc6 and installed all tools required to normally install these packages. So let's do that:
dpkg --root /target --install /root/packages/libc-bin
dpkg --root /target --install /root/packages/libc6
This means we have now fully installed all packages required to run dpkg. However, we want to install apt (and findutils), so we don't have to download all packages manually anymore. Note that besides listed dependencies, we also install debianutils (and its dependency sensible-utils) because it contains tools needed by one or more installation scripts. Also, I create an empty file which is used by raspbian-archive-keyring. Normally this will probably be created by the installation script, but because at first the packages is not installed, but merely unpacked, the file needs to be created manually.
It should be possible to copy all packages to the image, chroot to /target and install everything with dpkg we just installed. However, except for proving that dpkg is correctly installed, this is quite useless. So let's not and instead continue with how we were doing it before:
dpkg --root /target --install /root/packages/libstdc++6
dpkg --root /target --install /root/packages/libusb-0.1-4
dpkg --root /target --install /root/packages/gpgv
dpkg --root /target --install /root/packages/readline-common
dpkg --root /target --install /root/packages/libtinfo5
dpkg --root /target --install /root/packages/libreadline6
dpkg --root /target --install /root/packages/gnupg
dpkg-deb -x /root/packages/sensible-utils /target
dpkg --ignore-depends sensible-utils --root /target --install /root/packages/debianutils
dpkg --root /target --install /root/packages/sensible-utils
dpkg --root /target --install /root/packages/libapt-pkg4.12
dpkg-deb -x /root/packages/raspbian-archive-keyring /target
dpkg --ignore-depends=raspbian-archive-keyring --root /target --install /root/packages/apt
dpkg --root /target --install /root/packages/raspbian-archive-keyring
dpkg --root /target --install /root/packages/findutils
And now we are almost done. Apt is installed on the image, we just need to configure it to use the right repositories and add the repository keys to the keyring. I prefer to use the default raspbian repository and, for the firmware, the raspberrypi.org repository.
Also, because apt downloads packages from the internet using urls with hostnames, the resolver should be configured so it know where/how to look up hostnames.
If you know how the configuration files work, you can create and edit the files yourself. Alternatively, you could copy the files from the host system:
cp -vi /etc/apt/sources.list /target/etc/apt/sources.list
cp -vi /etc/apt/sources.list.d/raspi.list /target/etc/apt/sources.list.d/raspi.list
cp -vi /etc/resolv.conf /target/etc/resolv.conf
chroot /target apt-key add /root/raspberrypi.gpg.key
If you've set up different repositories (or the default repositories on your host system are different than mine), you might need to copy more or different files to /target/etc/apt/ and/or install more or different keys. If this is the case you should really start using google now to find out how to configure repositories, which repositories to use and where to find their keys and how to install them. Note that you need to install them in de chrooted environment (so, prepend 'chroot /target' in front of 'apt-key'), because else you will install them on your host system instead!
Now you've reached a milestone; apt is installed and configured on your image. From now on, no need to check dependencies and download packages manually any more! I think this was the hardest part.
Step 4: making it bootable
The last thing we need to do, is make the image bootable. For that we need some more stuff:
- The bootloader; on 'normal' computers this would have been grub, but on the Raspberry Pi we need a custom bootloader. This is also called the 'firmware'.
- Kernel; This will be a Linux kernel. It is actually also included in the bootloader/firmware package.
- Sysvinit; The 'init' program used by default by Raspbian.
- Udev; Tool for automatically creating device nodes
- Mount; Tools for mounting the filesystems.
- Rsyslog; Syslog daemon used by a lot of programs to log messages/warnings/errors.
- Login and passwd; Login is the application asking for your password (on the command line). Login also starts your shell after a successful login. The package passwd gives you some tools to manage users and groups.
- Bash; The default shell we will be using. Note that we already have the statically linked version of bash copied to the system. But we want the 'regular' package installed.
- A few other tools and packages needed by installation scripts or applications. For example locales should be installed.
Installing these packages is now quite easy, because apt will do the hard work for us. We should install them in the right order however, because some packages do not depend on other packages according to the meta data, but they are needed by the installation scripts.
- So, let's chroot to our new environment and start installing the base packages needed by other tools.
apt-get install base-passwd apt-utils ncurses-term ncurses-base gzip mawk hostname dialog
apt-get install locales
- So, that's a good start, now configure the locales. At least create the locale specified by the environment variables displayed when you run 'locale'. You can create locales by running this command:
- And finally the last packages really needed for booting the system:
apt-get install login
apt-get install passwd
apt-get install raspberrypi-bootloader sysvinit udev mount rsyslog bash
apt-get install kmod
Almost there! Just a little bit of configuration to be done and then we're finished!
- Create /etc/hostname and make up a host name. The file should only contain this host name.
- Either give your root user a password with 'passwd'
Create a regular user and install 'sudo'. Configure 'sudo' so your regular user can run sudo.
- Create /etc/fstab and put your default mount points in it. The file should look like this:
proc /proc proc defaults 0 0
/dev/mmcblk0p1 /boot vfat defaults 0 2
/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
- Edit /etc/network/interfaces and either:
- Configure a static IP address for your eth0 device
- Configure eth0 to use dhcp to get an IP address. For this, you'll need to install 'isc-dhcp-client'.
- And last, but certainly not least, configure the firmware. It is best/easiest to copy the configuration from your host system. In your host environment (so not in the chrooted environment, you can exit by typing 'exit'), run this:
cp -vi /boot/cmdline.txt /target/boot/cmdline.txt
cp -vi /boot/config.txt /target/boot/config.txt
This should be it. You are done! \o/
What is next?
You can now put your image on an SD card and boot your Raspberry Pi with it. If you are still in the chrooted environment, you can exit this by typing 'exit'. After that you can unmount the partitions and disconnect the image file from the loop device. You can do this by shutting down your Raspberry completely or by running the commands mentioned at the end of 'Step 1'. I will not go into details on how to get the image on an SD card, that has been described enough: just google it. Note that you'll have to have console access (meaning; a screen and keyboard attached directly to the Raspberry) at this stage; install network services like openssh-server to access your Raspberry from over the network. Have a look at this discussion of you do not get any signal from your raspberry pi: http://www.raspberrypi.org/forums/viewtopic.php?f=91&t=34061Also, do not forget to resize the root partition (or create extra partitions) to fit the entire SD card.
After booting and logging in, you can't do much more than look around in bash and install packages from the repository. And there are probably plenty of packages you'll want to install. What exactly depends on what you want to do. Remember that a lot of tutorials on the internet that use the commandline, assume certain basic commands to be installed. There is a good change that you do not have them installed (for example 'raspi-config' is missing), however you can install everything that is installed on the regular Raspbian image!
These are some of the first packages I installed after installing the base packages. You can probably install these packages already in your chrooted environment, before writing the image to an SD card:
- ntp; tool to set the system date and time automatically (based on NTP servers on the internet). The Raspberry does not have a battery, so without ntp, you would have to set the date after each (re)boot.
- vim and less; useful tools to edit and view text files.
- openssh-server; as explained above, I use this to access the raspberry over the network.
- isc-dhcp-client; so I can configure my network interface to use DHCP for automatic IP address discovery. I use the Raspberry as a server, but I have configured the DHCP server to always assign it the same IP address.
- sudo; let's you run commands as root while being logged in as a regular user. After installing this, I created a regular user and configured it so the regular user is allowed to use sudo. And then I removed the password for root, making the system just a little more safe (in my opinion).