Assignment 0: An Introduction to OS/161
(Adapted from Harvard assignment 0)
Due: In class on Friday, September 11, 2009.
Objectives |
After this assignment, you should:
- Be familiar with Subversion (SVN) and GDB (the GNU Debugger).
- Understand the source code structure of OS/161, the software system we will be using this semester.
- Understand how System/161 provides the "hardware" environment on which OS/161 runs.
- Understand the source code structure of System/161.
- Be comfortable reading the OS/161 source code and figuring out where things are done and how they are done.
- Be prepared to undertake Assignment 1.
Preliminaries |
NB: Throughout the rest of the assignment we will use <username> to refer to your Denison/sunshine/macfile username; so please replace it with the appropriate string (no brackets).
NB: To shorten the Subversion URLs this document uses tashi instead of tashi.mathsci.denison.edu. If you are within the Math/CS network, you are fine. Otherwise, from external locations, please use the fully qualified domain name.
I have created indivdual subversion repositories for each of you to use for your OS161 development. These reside on the machine 'tashi'. Using your <username> and the password provided, ensure that you can access your Subversion repository on tashi:
% svn list https://tashi/svn/<username>
branches/
tags/
trunk/
Introduction |
In this assignment, we will introduce:
System/161The first part of this document briefly discusses the code on which you'll be working and the tools you'll be using. You can find more detailed information on SVN and GDB in separate handouts. The following sections provide precise instructions on exactly what you must do for the assignment. Each section with (hand me in) at the beginning indicates a section where there is something that you must do for the assignment.
What are OS/161 and System/161? |
The code for this semester is divided into two main parts:
- OS/161: the operating system that you will augment in subsequent homework assignments.
- System/161: the machine simulator that emulates the physical hardware on which your operating system will run. This course is about writing operating systems, not designing or simulating hardware. Therefore, you may not change the machine simulator. If, however, you think you have found a bug in System/161, please let the professor know as soon as possible.
The "161" in OS/161 and System/161 comes from the number of the operating systems course at Harvard University for which this software was orginally developed.
The OS/161 distribution contains a full operating system source tree, including some utility programs and libraries. After you build the operating system you boot, run, and test it on the simulator.
We use a simulator in CS161 because debugging and testing an operating system on real hardware is extremely difficult. The System/161 machine simulator has been found to be an excellent platform for rapid development of operating system code, while still retaining a high degree of realism. Apart from floating point support and certain issues relating to RAM cache management, it provides an accurate emulation of a MIPS processor.
There will be an OS/161 programming assignment for each of the following topics:
- ASST1 : Synchronization and concurrent programming
- ASST2 : System calls and multiprogramming
- ASST3 : Virtual memory
- ASST4 : File systems
CS161 assignments are cumulative. Ideally you will build each assignment on top of your previous submission. If, however, at any point you wish to build on top of a solution set, contact me and we will explore the possibilities.
Using the solution sets may seem like an attractive alternative, since they are guaranteed to work. Keep in mind, however, that using the solution set requires that you understand a code base that you did not write. We encourage groups to refrain from using the solution sets except in the most dire circumstances.
About Subversion |
Most programming you have probably done at Denison has been in the form of 'one-off' assignments: you get an assignment, you complete it yourself, you turn it in, you get a grade, and then you never look at it again.
The commercial software world uses a very different paradigm: development continues on the same code base producing releases at regular intervals. This kind of development normally requires multiple people working simultaneously within the same code base, and necessitates a system for tracking and merging changes. Beginning with assignment 2 you will work in teams on OS/161 and will be developing a code base that will change over the couse of several assignments. Therefore, it is imperative that you start becoming comfortable with Subversion, an open source version control system.
Subversion is a powerful tool, but for this course you only need to know a subset of its functionality. The Introduction to Subversion handout contains all the information you need to know and should serve as a reference throughout the semester. If you'd like to learn more, there is comprehensive documentation available here .
About GDB |
In some ways debugging a kernel is no different from debugging an ordinary program. On real hardware, however, a kernel crash will crash the whole machine, necessitating a time-consuming reboot. The use of a machine simulator such as System/161 provides several debugging benefits. First, a kernel crash will only crash the simulator, which only takes a few keystrokes to restart. Second, the simulator can sometimes provide useful information about what the kernel did to cause the crash, information that may or may not be easily available when running directly on top of real hardware.
To debug OS/161 you must use a version of GDB configured to understand OS/161 and MIPS. This is called os161-gdb. This copy of GDB has been configured for MIPS and also been patched to be able to communicate with your kernel through System/161.
An important difference between debugging a regular program and debugging an OS/161 kernel is that you need to make sure that you are debugging the operating system, not the machine simulator. Type
% os161-gdb sys161and you are debugging the simulator. Not good. The handout Debugging with GDB provides detailed instructions on how to debug your operating system and a brief introduction to GDB.
Setting up your account |
We have created some scripts to help you set up your environment so that you can easily access the tools that you will need for this course.
- Add the following line to the end of your .bashrc file:
source /usr/local/os161/etc/bashrc
For reasons that will become clear later, you want to be able to identify the machine you are logged into. The lines in this file set your prompt to display your username, machine name, and working directory. If you know how to change the format of your prompt yourself, feel free to change it to something you like better directly in your .bashrc file.
- Add the following line to the end of your .bash_profile file:
source /usr/local/os161/etc/bash_profile
The lines in this file tell the shell where to find tools you will use in the course.
- Log out and back in.
Getting the Distribution (hand me in) |
- Script the following session using the script command.
- First, download the OS161 source to your home directory from here. (The compressed tar file will be called os161-1.11-du.tgz.)
- Next, create a directory (assumed here to be directly under your home directory) in which you will do all of your cs372 work. For the purposes of the remainder of this assignment, we'll assume that it will be called cs372.
% cd; mkdir cs372 % cd cs372
- Next, check out a directory from our grading repository into
which you'll place the files that you will submit, and create (through svn) the
asst0 directory into which you'll put the files from
this assignment.
The first statement creates the (empty, but version controlled) directory named submit, and the second adds a new directory to the version-controlled repository.
% svn co https://tashi/svn/grading/<username> submit % svn mkdir submit/asst0
- Move the OS/161 distribution into place and unpack it by typing
% mv ~/os161-1.11-du.tgz ~/cs372
This will create a directory named os161-1.11.
% tar xvzf os161-1.11-du.tgz - Rename your OS/161 source tree to just os161, and remove the
tarball.
% mv os161-1.11 os161 % rm *.tgz
- End your script session by typing exit or by pressing
Ctrl-D. Rename your typescript file to be setup.script in the grading submission directory.
% mv ~/typescript ~/cs372/submit/asst0/setup.script
Setting up your Subversion repository (hand me in) |
- Script the following session:
% script ~/cs372/submit/asst0/svninit.script
- From the preliminaries you should have access to a personal Subversion repository on tashi. Your Subversion repository URL is https://tashi/svn/<username>.
- Change directories into the OS/161 distribution that you
unpacked in the previous section and import your source tree.
% cd ~/cs372/ % svn import os161/ https://tashi/svn/<username>/trunk/ -m "Initial import of os161" % svn copy https://tashi/svn/<username>/trunk https://tashi/svn/<username>/tags/os161-1.11 -m "Tagging initial import."
You can alter the arguments as you like; here's a quick explanation.-m "Initial import of os161" is the log message that Subversion records. (If you don't specify it on the command line, it will start up a text editor). /trunk is where Subversion will put the files within your repository. http://tashi/repos/<username>/trunk is the Subversion URL you will specify when you check out your system. The second svn copy command creates a simple tag of the initial import of your system that you can later use with svn diff and other commands. More on that later (or see this tagging explanation).
- Now, remove the source tree that you just imported.
% rm -rf os161
Don't worry - now that you have imported the tree in your repository, there is a copy saved away. In the next step, you'll get a copy of the source tree that is yours to work on. You can safely remove the original tree. - Now, checkout a source tree in which you will work.
% svn co https://tashi/svn/<username>/trunk src
The svn co command creates a working copy of your tree that you can safely modify in src (feel free to put it wherever you like). - End your script session.
Code Reading (hand me in) |
One of the challenges of the OS/161 project is that you are going to be working with a large body of code that was written by someone else. When doing so, it is important that you grasp the overall organization of the entire code base, understand where different pieces of functionality are implemented, and learn how to augment it in a natural and correct fashion. As you and your partner develop code, although you needn't understand every detail of your partner's implementation, you still need to understand its overall structure, how it fits into the greater whole, and how it works.
In order to become familiar with a code base, there is no substitute for actually sitting down and reading the code. Admittedly, most code makes poor bedtime reading (except perhaps as a soporific), but it is essential that you read the code. It is all right if you don't understand most of the assembly code in the codebase; it is not important for this class that you understand all of the assembly.
You should use the code reading questions included below to help guide you through reviewing the existing code. While you needn't review every line of code in the system in order to answer all the questions, we strongly recommend that you look over every file in the system.
The key part of this exercise is understanding the base system. Your goal is to understand how it all fits together so that you can make intelligent design decisions when you approach future assignments. This may seem tedious, but if you understand how the system fits together now, you will have much less difficulty completing future assignments. Also, it may not be apparent yet, but you have much more time to do so now than you will at any other point in the semester.
The file system, I/O, and network sections may seem confusing since we have not discussed how these components work. However, it is still useful to review the code now and get a high-level idea of what is happening in each subsystem. If you do not understand the low-level details now, that is OK.
These questions are not meant to be tricky -- most of the answers can be found in comments in the OS/161 source, though you may have to look at your textbook for some background information. Place the answers to the following questions in a file called ~/cs372/submit/asst0/code-reading.txt.
Top Level Directory
The top level directory of many software packages is called src or source. The top of the OS/161 source tree is also called src. In this directory, you will find the following files:
Makefile: top-level makefile; builds the OS/161 distribution, including all the provided utilities, but does not build the operating system kernel.
configure: this is an autoconf-like script. It sets up things like `How to run the compiler.' You needn't understand this file, although we'll ask you to specify certain pathnames and options when you build your own tree.
defs.mk: this file is generated when you run ./configure. You needn't do anything to this file.
defs.mk.sample: this is a sample defs.mk file. Ideally, you won't be needing it either, but if configure fails, use the comments in this file to fix defs.mk.
You will also find the following directories:
bin: this is where the source code lives for all the utilities that are typically found in /bin, e.g., cat, cp, ls, etc. The things in bin are considered "fundamental" utilities that the system needs to run.
include: these are the include files that you would typically find in /usr/include (in our case, a subset of them). These are user level include files; not kernel include files.
kern: here is where the kernel source code lives.
lib: library code lives here. We have only two libraries: libc, the C standard library, and hostcompat, which is for recompiling OS/161 programs for the host UNIX system. There is also a crt0 directory, which contains the startup code for user programs.
man: the OS/161 manual ("man pages") appear here. The man pages document (or specify) every program, every function in the C library, and every system call. You will use the system call man pages for reference in the course of assignment 2. The man pages are HTML and can be read with any browser.
mk: this directory contains pieces of makefile that are used for building the system. You don't need to worry about these, although in the long run we do recommend that anyone working on large software systems learn to use make effectively.
sbin: this is the source code for the utilities typically found in /sbin on a typical UNIX installation. In our case, there are some utilities that let you halt the machine, power it off and reboot it, among other things.
testbin: these are pieces of test code.
You needn't understand the files in bin, sbin, and testbin now, but you certainly will later on. Eventually, you will want to modify these and/or write your own utilities and these are good models. Similarly, you need not read and understand everything in lib and include, but you should know enough about what's there to be able to get around the source tree easily. The rest of this code walk-through is going to concern itself with the kern subtree.
The Kern Subdirectory
Once again, there is a Makefile. This Makefile installs header files but does not build anything.
In addition, we have more subdirectories for each component of the kernel as well as some utility directories.
kern/arch: This is where architecture-specific code goes. By architecture-specific, we mean the code that differs depending on the hardware platform on which you're running.
For our purposes, you need only concern yourself with the mips subdirectory.
kern/arch/mips/conf:
conf.arch: This tells the kernel config script where to find the machine-specific, low-level functions it needs (see kern/arch/mips/mips).
Makefile.mips: Kernel Makefile; this is copied when you "config a kernel".
kern/arch/mips/include: These files are include files for the machine-specific constants and functions.
Question 1. Which register number is used for the stack pointer (sp) in OS/161?
Question 2. What bus/busses does OS/161 support?
Question 3. What is the difference between splhigh and spl0?
Question 4. What are some of the details which would make a function "machine dependent"? Why might it be important to maintain this separation, instead of just putting all of the code in one function?
kern/arch/mips/mips: These are the source files containing the machine-dependent code that the kernel needs to run. Most of this code is quite low-level.
Question 5. What does splx return?
Question 6. What is the highest interrupt level?
kern/asst1: This is the directory that contains the framework code that you will need to complete assignment 1. You can safely ignore it for now.
kern/compile: This is where you build kernels. In the compile directory, you will find one subdirectory for each kernel you want to build. In a real installation, these will often correspond to things like a debug build, a profiling build, etc. In our world, each build directory will correspond to a programming assignment, e.g., ASST1, ASST2, etc. These directories are created when you configure a kernel (described in the next section). This directory and build organization is typical of UNIX installations and is not universal across all operating systems.
kern/conf: config is the script that takes a config file, like ASST1, and creates the corresponding build directory. So, in order to build a kernel, you will type the following commands (but don't do it yet):
% cd kern/conf % ./config ASST0 % cd ../compile/ASST0 % make depend % makeThis will create the ASST0 build directory and then actually build a kernel in it. Note that you should specify the complete pathname ./config when you configure OS/161. If you omit the ./, you may end up running the configuration command for the system on which you are building OS/161, and that is almost guaranteed to produce rather strange results!
kern/dev: This is where all the low level device management code is stored. Unless you are really interested, you can safely ignore most of this directory.
kern/include: These are the include files that the kernel needs. The kern subdirectory contains include files that are visible not only to the operating system itself, but also to user-level programs. (Think about why it's named "kern" and where the files end up when installed.)
Question 7. How frequently are hardclock interrupts generated?
Question 8. What is the standard interface to a file system (i.e., what functions must you implement to implement a new file system)?
Question 9. How large are OS/161 pids? How many processes do you think OS/161 could support as you have it now (ASST0)? A sentence or two of justification is fine.
Question 10. A vnode is a kernel abstraction that represents a file. What operations can you do on a vnode? If two different processes open the same file, do we need to create two vnodes?
Question 11. What is the system call number for a reboot? Is this value available to userspace programs? Why or why not.
kern/lib: These are library routines used throughout the kernel, e.g., managing sleep queues, run queues, kernel malloc, etc.
Question 12. What is the purpose of functions like copyin and copyout in copyinout.c? What do they protect against? Where might you want to use these functions?
Question 13.Why are there two separate implementations of 'malloc' (one in kern/lib/kheap.c, called kmalloc(), and one that will eventually be in lib/libc/malloc.c; the standard function prototype for this is given in the libc stdlib.h)? Give one major difference that you anticipate between the two implementations.
kern/main: This is where the kernel is initialized and where the kernel main function is implemented.
kern/thread: Threads are the fundamental abstraction on which the kernel is built.
Question 14 Is it OK to initialize the thread system before the scheduler? Why (not)?
Question 15. What are the possible states that a thread can be in? When do "zombie" threads finally get cleaned up?
Question 16. What function puts a thread to sleep? When might you want to use this function?
kern/userprog: This is where you will add code to create and manage user level processes. As it stands now, OS/161 runs only kernel threads; there is no support for user level code. In Assignment 2, you'll implement this support.
kern/vm: This directory is also fairly vacant. In Assignment 3, you'll implement virtual memory and most of your code will go in here.
kern/fs: The file system implementation has two subdirectories. We'll talk about each in turn. kern/fs/vfs is the file-system independent layer (vfs stands for "Virtual File System"). It establishes a framework into which you can add new file systems easily. You will want to go look at vfs.h and vnode.h before looking at this directory.
Question 17. What does a device pathname in OS/161 look like?
Question 18. What does a raw device name in OS/161 look like?
Question 19. What lock protects the vnode reference count?
kern/fs/sfs: This is the simple file system that OS/161 contains by default. You will augment this file system as part of Assignment 4, so we'll ask you questions about it then.
Building a kernel (hand me in) |
Now it is time to build a kernel. As described above, you will need to configure a kernel and then build it.
- Script the following steps:
% script ~/cs372/submit/asst0/build.script
- Configure your tree for the machine on which you are working.
If you want to work in a directory that's not $HOME/cs161
(which you will be doing when you test your later submissions) you
might want to use the --ostree option. ./configure
--help explains the other options.
% cd ~/cs372/src % ./configure
- Configure a kernel named ASST0.
% cd ~/cs372/src/kern/conf % ./config ASST0
- Build the ASST0 kernel.
% cd ../compile/ASST0 % make depend % make
- Install the ASST0 kernel.
% make install
- Now also build the user level utilties.
% cd ~/cs372/src % make
- End your script session by typing Ctrl-D
Running your kernel (hand me in) |
- Script the following session:
% script ~/cs372/submit/asst0/run.script
- Copy the file /usr/local/os161/etc/sys161.conf to your OS/161 root directory (~/cs372/root).
- Change into your root directory.
% cd ~/cs372/root
- Run the machine simulator on your operating system.
% sys161 kernel
- At the prompt, type
p /sbin/poweroff <return>
This tells the kernel to run the "poweroff" program that shuts the system down. - End your script session.
Practice modifying your kernel (hand me in) |
- Create a file called main/hello.c.
- In this file, write a function called hello that uses kprintf() to print "Hello World\n".
- Edit main/main.c and add a call (in a suitable place) to hello().
- Make your kernel build again. You will need to edit conf/conf.kern, reconfig, and rebuild.
- Make sure that your new kernel runs and displays the new message.
- Once your kernel builds, script a session demonstrating a config and build of your modified kernel. Call the output of this script session newbuild.script.
Using GDB (hand me in) |
- Script the following gdb session (that is, you needn't script the session in the run window, only the session in the debug window), naming the script gdb.script. Be sure both your run window and your debug window are on the same machine.
- Run the kernel in gdb by first running the kernel and then attaching
to it from gdb.
(In the run window:) % cd ~/cs372/root % sys161 -w kernel (In the debug window:) % script ~/cs372/submit/asst0/gdb.script % cd ~/cs372/root % os161-gdb kernel (gdb) target remote unix:.sockets/gdb (gdb) break menu (gdb) c [gdb will stop at menu() ...] (gdb) where [displays a nice back trace...] (gdb) detach (gdb) quit
- End your script session.
Practice with Subversion (hand me in) |
In order to build your kernel above, you already checked out a source tree. Now we'll demonstrate some of the most common features of Subversion. Create a script of the following session in ~/submit/asst0/svn-use.script. The script should contain everything except the editing sessions. Do those in a different window.
- First, make sure that you have completed updating your working copy to include the necessary files from the "Hello World!" exercise above. svn status is your friend here. If you are concerned by all of the .depend files and build directories that show up when you run svn status in ~/cs161/src, edit ~/.subversion/config, uncomment the global-ignores line and add the *.depend pattern. This will eliminate the .depend files. To avoid seeing all of the build directories you may have to do a make clean from your ~/cs161/src directory. There are better ways to avoid these sorts of irritations. Come see the course staff. (A simple solution is to run svn status in src/kern, where you will be doing most of your work.)
- Begin the script:
% script ~/cs372/submit/asst0/svn-use.script
- Change directory to kern/main/ and add a comment with your name to main.c.
- First, from within your ~/cs372/src/kern directory,
execute:
% svn status
Note that main.c shows status 'M', indicating that the file has been modified. For the full list of svn status codes consult this cheat sheet. - Execute
% svn diff -x -uw main/main.c
to display the differences in your version of this file. - Now commit your changes using svn commit.
- Remove the first 100 lines of main.c.
- Try to build your kernel (this ought to fail).
- Realize the error of your ways and get back a good copy of the
file.
% svn revert main/main.c
- Try to build your tree again.
- Now, examine the DEBUG macro in lib.h. Based on your earlier reading of the operating system, add ten useful debugging messages to your operating system.
- Now, show us where you inserted these DEBUG statements by doing
a diff.
% cd ~/cs372/src/kern % svn diff -x -uw
- End the script.
Create a Release |
Finally, you should create a release. Refer to the document on Preparing your Assignments for Submission . Once you are confident that your tree is in the condition you would like to leave it in, i.e. svn status in ~/cs372/src should be empty or full of things you know you didn't change or don't want to add, then follow the following steps.
- Export a copy of your tree into a temporary directory:
% cd ~/tmp/ # Create this directory if it doesn't exist % svn export https://tashi/svn/<username>/trunk <username>-asst0
- The step above creates a "clean" (i.e. free of subversion control files) copy of the tree currently in the repository in ~/tmp/<username>-asst0. Testing this release helps you pick up files you may have forgotten to add or changes you haven't yet committed.
- Enter your ~/tmp/<username>-asst0 directory and make sure that everything is as it should be, particularly that you can build and run your kernel! (You may want to pass a different --ostree=<path> argument to the top level ./configure in order to have the built binaries placed somewhere other than $HOME/cs372/root) Once you are finished, clean up by destroying the contents of the ~/tmp/ directory, i.e. rm -rf ~/tmp/*.
- Tag your repository for the end of asst0:
% svn copy https://tashi/svn/<username>/trunk https://tashi/svn/<username>/tags/asst0-end
Remember the tags directory we created earlier? This is versioned like any other part of your Subversion repository. The step above simply copies from /trunk/ into tags/asst0-end. It's also repeatable, so if you realized later you have changes yet to commit you can repeat the copy using a new version of /trunk/. - Finally, export and tar up a copy of your tagged tree
in your submission directory:
% cd ~/cs372/submit/asst0 % svn export https://tashi/svn/<username>/tags/asst0-end <username>-asst0 % tar -czf <username>-asst0.tgz <username>-asst0
- After you tar your release, be sure to remove the
~/submit/asst0/<username>-asst0 your svn
export created:
% rm -rf <username>-asst0
What (and how) to hand in |
In cs372 this semester, we are experimenting with using Subversion to allow you to submit your assignments. Like your codebase itself, this means that you can submit multiple times, version your submissions, and this will better allow us to collect and collaborate over grading your assignments.
Your ~/cs372/submit/asst0 directory should contain everything you need to submit, specifically:
- setup.script
- svninit.script
- code-reading.txt
- build.script
- run.script
- newbuild.script
- gdb.script
- svn-use.script
- sys161.conf: Copy this from ~/cs372/root.
- <username>-asst0.tgz
Review the document Preparing your Assignments for Submission to learn how to properly submit your assignment. Specifically, for this assignment, and for a first-time submit:
Modified instructions for new system for submitting:
- Save your submission materials in a temporary directory and then delete the verison-controlled submit directory associated with the grading repository:
cd ~/cs372
mkdir ~/asst-tmp
cp submit/asst0/* ~/asst-tmp/
rm -rf submit/
- Next, create a submit directory as part of your individual repository, then list the top level repository to make sure it appears alongside trunk/, branches/, and tags/, and then check it out, creating a working 'submit' directory:
svn mkdir https://tashi/svn/<username>/submit
svn list https://tashi/svn/<username>
svn co https://tashi/svn/<username>/submit
- We next create a version-controlled asst0 directory, move the files we saved in the first step on in, and bring them under version control. An svn status should show all of these, as well as the asst0 directory, as added, but not yet commited.
cd submit/
svn mkdir asst0
mv ~/asst-tmp/* asst0/
ls asst0/
cd asst0/
svn add *
cd ..
svn status
- Finally, we commit all the submitted items to the repository:
svn commit -m "ASST0 submission 1"
Once everything is committed, further changes will require a subsequent commit.