Harvard University creates PaaS using SELinux Sandbox

Harvard goes PaaS with SELinux Sandbox

open education
Image by : 


Running students' submitted programs is a security challenge for any university Computer Science department. When Harvard University contacted me about some work they are doing with the "sandbox" tool on Fedora 17, we decided it would be a great opportunity to see how they could get more out of it and share our findings with the community. 

In a lot of ways, Harvard is setting up a simple PaaS (Platform as a Service). We discussed tools like OpenShift and Secure Linux Containers, but the immediate issue was that once they begin offering 'Intro to 'C'" courses online, the students will upload programs to a Harvard web server that will be compiled and tested. And, to no one's surprise, they were concerned about what the students "C" applications might attempt to do.

One option was to setup a separate account for each user, using UID separation. They decided not to do this because there was no guarantee they would know who the students were, executing useradd for each student would be difficult, and the students might use the system just once. I believe the courseware will eventually be open to anyone and all students will login as the user jhavard (so named after John Harvard). 

I was surprised when I got there that Harvard was using sandbox and not sandbox -X, since most of the time I talk to people about the Desktop version of sandbox. They were using the server (non-GUI) version.

When a student uploads a program to their website, the scripts generate a random homedir and tmpdir on the fly, and then execute the sandbox command to create an isolated environment in which to run the student's "test" program. The sandbox utility takes advantage of Namespacing (Containers) to mount the newly created homedir and tmpdir over $HOME and /tmp. It then runs the student's "test" program with the sandbox_t type and a randomly generated MCS label.

The SELinux kernel prevents the sandbox_t type from being able to execute privileged applications on the server and thus controls what the student's process is allowed to do. sandbox_t is only allowed to modify content in its homedir and tempdir and to read/execute system files—it is not allowed to execute any setuid applications. SELinux uses the MCS label to guarantee that no student's processes can interfere with any other student's processes or files. 

The student's code is then compiled and executed. The tools capture the stdout and stderr for later review by the professor, and finally, the server is programed to erase the scripts from all of the content submitted by the student. 

One problem Harvard had with the default sandbox_t types was that the professors will assign their students programs which include networking code, and require full TCP/UDP network access which sandbox_t denies by default. Harvard works around this by enabling networking and using iptables to constrain the network ports and hosts to which the students applications can connect. This way, if the assignment features connecting to a site such as people.redhat.com/dwalsh to download a document, the professors can allow the student's program to access people.redhat.com while blocking access to all other websites.

Extending sandbox policy

In order to test the policy, we used runcon instead of sandbox, because it is simpler to test.

$ runcon -t sandbox_t nc localhost 80
p_ssl_init: Failed to seed OpenSSL PRNG (RAND_status returned false)

I explained a little about writing an SELinux policy, then we started extending the standard SELinux policy.

# cat mysandbox.te


type sandbox_t;
# extend sandbox to be allowed to connect to TCP and UDP sockets
We are using a policy module type enforcement (te) file here to extend sandbox_t.
First, we needed to specify that sandbox_t is required for this module. Second, we used the corenet interfaces to allow sandbox_t to connect to any TCP port and send/recv from any UDP port. We also enabled the sysnet_dns_name_resolve interface that actually allows the use of the DNS resolver, and automatically uses other policy to allow us to create TCP and UDP sockets. During testing, we noticed that the shell was using /dev/urandom for connections, so we also allowed this. 


Our policy file finished, we compiled it using the supplied make file:

# make -f /usr/share/selinux/devel/Makefile mysandbox.pp

Finally, we installed the compiled policy package, and retested to see if we could connect to the network:

# semodule -i mysandbox.pp
# runcon -t sandbox_t nc localhost 80

sandbox_t is now allowed to connect to port 80.

Harvard would then need to distribute and install the compiled policy file (mysandbox.pp) on all of their test servers by running these same commands. Puppet and Chef are excellent tools for automating this.

Multiple domains

Harvard had another idea—they wanted to allow professors to upload their own test suites along with their students' code. However, the current setup presented a problem:

Students' code could affect the professors test suites!

If I can modify the professors test to always return success, I can probably improve my grades.

We decided to write a new policy where the processes on the system would run with two different types: sandbox_staff_t and sandbox_t. We also created two file types, sandbox_staff_file_t and sandbox_file_t. The $HOME directory and all its contents were labeled sandbox_staff_file_t. $HOME now contained two directories, staff and student. The student directory's contents were labeled sandbox_file_t. The staff tests were stored in the staff directory, labeled sandbox_staff_file_t. The rules for the process types were that sandbox_t can manage all content in sandbox_file_t, and sandbox_staff_t can manage all content in sandbox_staff_file_t and sandbox_file_t.

The sandbox_staff_t process needed to be able to execute scripts within the student directory, but it could not execute as sandbox_staff_t or this would allow the students' scripts to attack the staff directory. We needed a transition rule saying that when the sandbox_staff_t process executes a file labeled sandbox_file_t it transitions to sandbox_t. We also wanted sandbox_staff_t to be allowed to send signals, i.e. 'kill,' to the sandbox_t processes and allow it to ptrace them.

Here is the policy we wrote for this:



type sandbox_t;
type sandbox_file_t;
attribute sandbox_file_type;
# extend sandbox to be allowed to connect to TCP and
# UDP sockets
# Create new sandbox domain sandbox_staff_t, this
# interface also adds the rules to allow sandbox_staff_t to
# manage all content labeled sandbox_file_t.
# Create a type for all staff content
type sandbox_staff_file_t, sandbox_file_type;
# Rules to allow sandbox_staff_t, to manage content in
# sandbox_staff_file_t.
manage_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)
manage_dirs_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)
manage_sock_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)
manage_fifo_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)
manage_lnk_files_pattern(sandbox_staff_t, sandbox_staff_file_t, sandbox_staff_file_t)
# Now we write rules to control what happens when
# sandbox_staff_t executes a program,
# If the process executes a program labeled
# sandbox_staff_file_t it will run as sandbox_staff_t
can_exec(sandbox_staff_t, sandbox_staff_file_t)
# If the sandbox_staff_t process runs a file labeled
# sandbox_file_t, it will transition to sandbox_t.
domtrans_pattern(sandbox_staff_t, sandbox_file_t,sandbox_t)
# Allow the sandbox_staff_t process to control the
# sandbox_t process.
allow sandbox_staff_t sandbox_t:process { signal_perms ptrace };
ps_process_pattern(sandbox_staff_t, sandbox_t)
# Allow the sandbox_t process to search through the
# sandbox_staff_t directories
allow sandbox_t sandbox_staff_file_t:dir search_dir_perms;
# Since the tests are going to be executing shells, shell
# programs seem to want to read /proc files

Compile and install:

# make -f /usr/share/selinux/devel/Makefile mysandbox.pp
# semodule -i mysandbox.pp

Test to make sure we can run as sandbox_staff_t:

# runcon -t sandbox_staff_t id -Z

Good, it worked. Now, setup a test environment.

Setting up a test environment

First, setup a home directory "class" with sub-directories staff and student, and write test scripts which output the SELinux context that the scripts are running with and "Hello World."

$ mkdir -p class/staff class/student
$ cat > class/staff/test.sh
id -Z
echo "Hello world staff"
$ cat > class/student/test.sh
id -Z
echo "Hello world student"
$ chmod +x class/staff/test.sh class/student/test.sh

Set the context of the directories, default for all to sandbox_staff_file_t, and class/student to sandbox_file_t;

$ chcon -R -t sandbox_staff_file_t class
$ chcon -R -t sandbox_file_t class/student

Check the labels using find:

$ find class -printf "%p\t\t%Z\n"

class  staff_u:object_r:sandbox_staff_file_t:s0
class/student  staff_u:object_r:sandbox_file_t:s0
class/student/test.sh  staff_u:object_r:sandbox_file_t:s0
class/staff  staff_u:object_r:sandbox_staff_file_t:s0
class/staff/test.sh  staff_u:object_r:sandbox_staff_file_t:s0

Begin testing.

$ id -Z
$ runcon -t sandbox_t id -Z
$ runcon -t sandbox_t touch class/bad
touch: cannot touch ‘class/bad’: Permission denied
$ runcon -t sandbox_t touch class/staff/bad
touch: cannot touch ‘class/staff/bad’: Permission denied

Excellent! This shows that a student process cannot write to the class or staff directories. If we allowed student code to write to these directories, it could create a .bashrc or profile file and trick the sandbox_staff_t process into doing something bad.

$ runcon -t sandbox_t touch class/student/good

This shows the student's code can write to their own directory, which is what we intended.

Verify that the sandbox_staff_t process can write to all directories:

$ runcon -t sandbox_staff_t touch class/staff/good
$ runcon -t sandbox_staff_t touch class/student/good

With this test passing, it's time to make sure that the context transition is working.

# Test to make sure the sandbox_staff_t process will transition to the proper context when running the students code, ie programs labeled sandbox_file_t.

$ runcon -t sandbox_staff_t class/staff/test.sh
"Hello world staff"
$ runcon -t sandbox_staff_t class/student/test.sh
"Hello world student"

Oops, this looks wrong! Why didn't the process run as sandbox_t? It looks like the transition failed.

The policy says that when a sandbox_staff_t process runs a sandbox_file_t, it should transition to sandbox_t. Instead we see the process running as sandbox_staff_t, so we know something is broken. Let me explain what really happened here.

When the runcon or sandbox command is executed, these commands tell the kernel to label the next process they execute as the specified type. In this case, runcon tells the kernel to run the class/student/test/sh as sandbox_staff_t.

In order to get the transitions to work the way we expect, we need to execute a program (/bin/sh) as the sandbox_staff_t and then allow that process to execute class/student/test.sh:

$ runcon -t sandbox_staff_t sh -c class/student/test.sh
"Hello world student"

This shows the process is running as sandbox_t, which means our policy is working correctly. Harvard can now test the sandbox with a command such as:

# sandbox -t sandbox_staff_t -h class staff/test.sh

Note: Monitoring the audit.log's to see what the students are doing can be daunting. Each day, someone needs to review the /var/log/audit/audit.log for AVC messages, which is one of the things I do for OpenShift.

It's easy to get overwhelmed by the number of AVCs generated by students exploring the system. I see many users doing stuff like find /, rpm -q, su, sudo, passwd--most of which I consider harmless becasue this is just users probing the system. Truth be told, if I was given an account on a system, I would probably do the same. But we found that when we removed the ability for a normal user to execute these programs the noise level was reduced considerably.

I recommend that you remove the "other" execute permission flag following: rpm, yum, debuginfo-install, traceroute, iptables, iptables-multi, dmesg, tcpdump, mtr, ip, su, sudo, denyhosts, chsh, chfn. Also modify the applications group to the wheel group and add any administors to the wheel group.

For example:

# chmod o-x /usr/bin/rpm

# chgrp wheel /usr/bin/rpm

# ls -l /usr/bin/rpm
-rwxr-xr--. 1 root wheel 27280 Jul 21 22:04 /usr/bin/rpm

# grep wheel /etc/group

This change would prevent student accounts from executing these commands and triggering AVC's that you are probably not concerned about.

About the author

Daniel J Walsh - Daniel Walsh has worked in the computer security field for almost 30 years. Dan joined Red Hat in August 2001.