20 Linux commands every sysadmin should know

If your application isn't working—or you're just looking for more information—these 20 commands will come in handy.
978 readers like this.
Avoiding data disasters with Sanoid

Opensource.com

In a world bursting with new tools and diverse development environments, it's practically a necessity for any developer or engineer to learn some basic sysadmin commands. Specific commands and packages can help developers organize, troubleshoot, and optimize their applications and—when things go wrong—provide valuable triage information to operators and sysadmins.

Whether you are a new developer or want to manage your own application, the following 20 basic sysadmin commands can help you better understand your applications. They can also help you describe problems to sysadmins troubleshooting why an application might work locally but not on a remote host. These commands apply to Linux development environments, containers, virtual machines (VMs), and bare metal.

1. curl

curl transfers a URL. Use this command to test an application's endpoint or connectivity to an upstream service endpoint. curl can be useful for determining if your application can reach another service, such as a database, or checking if your service is healthy.

As an example, imagine your application throws an HTTP 500 error indicating it can't reach a MongoDB database:

$ curl -I -s myapplication:5000
HTTP/1.0 500 INTERNAL SERVER ERROR

The -I option shows the header information and the -s option silences the response body. Checking the endpoint of your database from your local desktop:

$ curl -I -s database:27017
HTTP/1.0 200 OK

So what could be the problem? Check if your application can get to other places besides the database from the application host:

$ curl -I -s https://opensource.com
HTTP/1.1 200 OK

That seems to be okay. Now try to reach the database from the application host. Your application is using the database's hostname, so try that first:

$ curl database:27017
curl: (6) Couldn't resolve host 'database'

This indicates that your application cannot resolve the database because the URL of the database is unavailable or the host (container or VM) does not have a nameserver it can use to resolve the hostname.

2. python -m json.tool / jq

After you issue curl, the output of the API call may be difficult to read. Sometimes, you want to pretty-print the JSON output to find a specific entry. Python has a built-in JSON library that can help with this. You use python -m json.tool to indent and organize the JSON. To use Python's JSON module, pipe the output of a JSON file into the python -m json.tool command.

$ cat test.json
{"title":"Person","type":"object","properties":{"firstName":{"type":"string"},"lastName":{"type":"string"},"age":{"description":"Age in years","type":"integer","minimum":0}},"required":["firstName","lastName"]}

To use the Python library, pipe the output to Python with the -m (module) option.

$ cat test.json | python -m json.tool
{
    "properties": {
        "age": {
            "description": "Age in years",
            "minimum": 0,
            "type": "integer"
        },
        "firstName": {
            "type": "string"
        },
        "lastName": {
            "type": "string"
        }
    },
    "required": [
        "firstName",
        "lastName"
    ],
    "title": "Person",
    "type": "object"
}

For more advanced JSON parsing, you can install jq. jq provides some options that extract specific values from the JSON input. To pretty-print like the Python module above, simply apply jq to the output.

$ cat test.json | jq
{
  "title": "Person",
  "type": "object",
  "properties": {
    "firstName": {
      "type": "string"
    },
    "lastName": {
      "type": "string"
    },
    "age": {
      "description": "Age in years",
      "type": "integer",
      "minimum": 0
    }
  },
  "required": [
    "firstName",
    "lastName"
  ]
}

3. ls

ls lists files in a directory. Sysadmins and developers issue this command quite often. In the container space, this command can help determine your container image's directory and files. Besides looking up your files, ls can help you examine your permissions. In the example below, you can't run myapp because of a permissions issue. When you check the permissions using ls -l, you realize that the permissions do not have an "x" in -rw-r--r--, which are read and write only.

$ ./myapp
bash: ./myapp: Permission denied
$ ls -l myapp
-rw-r--r--. 1 root root 33 Jul 21 18:36 myapp

4. tail

tail displays the last part of a file. You usually don't need every log line to troubleshoot. Instead, you want to check what your logs say about the most recent request to your application. For example, you can use tail to check what happens in the logs when you make a request to your Apache HTTP server.

Use tail -f to follow Apache HTTP server logs and see the requests as they happen.

opensource.com

The -f option indicates the "follow" option, which outputs the log lines as they are written to the file. The example has a background script that accesses the endpoint every few seconds and the log records the request. Instead of following the log in real time, you can also use tail to see the last 100 lines of the file with the -n option.

$ tail -n 100 /var/log/httpd/access_log

5. cat

cat concatenates and prints files. You might issue cat to check the contents of your dependencies file or to confirm the version of the application that you have already built locally.

$ cat requirements.txt
flask
flask_pymongo

The example above checks whether your Python Flask application has Flask listed as a dependency.

6. grep

grep searches file patterns. If you are looking for a specific pattern in the output of another command, grep highlights the relevant lines. Use this command for searching log files, specific processes, and more. If you want to see if Apache Tomcat starts up, you might become overwhelmed by the number of lines. By piping that output to the grep command, you isolate the lines that indicate server startup.

$ cat tomcat.log | grep org.apache.catalina.startup.Catalina.start
01-Jul-2017 18:03:47.542 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 681 ms

7. ps

The ps command, part of the procps-ng package which provides useful commands for investigating process IDs, shows the status of a running process. Use this command to determine a running application or confirm an expected process. For example, if you want to check for a running Tomcat web server, you use ps with its options to obtain the process ID of Tomcat.

$ ps -ef
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  2 18:55 ?        00:00:02 /docker-java-home/jre/bi
root        59     0  0 18:55 pts/0    00:00:00 /bin/sh
root        75    59  0 18:57 pts/0    00:00:00 ps -ef

For even more legibility, use ps and pipe it to grep.

$ ps -ef | grep tomcat
root         1     0  1 18:55 ?        00:00:02 /docker-java-home/jre/bi

8. env

env allows you to set or print the environment variables. During troubleshooting, you may find it useful for checking if the wrong environment variable prevents your application from starting. In the example below, this command is used to check the environment variables set on your application's host.

$ env
PYTHON_PIP_VERSION=9.0.1
HOME=/root
DB_NAME=test
PATH=/usr/local/bin:/usr/local/sbin
LANG=C.UTF-8
PYTHON_VERSION=3.4.6
PWD=/
DB_URI=mongodb://database:27017/test

Notice that the application is using Python and has environment variables to connect to a MongoDB database.

9. top

top displays and updates sorted process information. Use this monitoring tool to determine which processes are running and how much memory and CPU they consume. A common case occurs when you run an application and it dies a minute later. First, you check the application's return error, which is a memory error.

$ tail myapp.log
Traceback (most recent call last):
MemoryError

Is your application really out of memory? To confirm, use top to determine how much CPU and memory your application consumes. When issuing top, you notice a Python application using most of the CPU, with its memory usage climbing, and suspect it is your application. While it runs, you hit the "C" key to see the full command and reverse-engineer if the process is your application. It turns out to be your memory-intensive application (memeater.py). When your application has run out of memory, the system kills it with an out-of-memory (OOM) error.

Issuing top against an application that consumes all of its memory.

opensource.com

Pressing C while running top shows the full command

opensource.com

In addition to checking your own application, you can use top to debug other processes that utilize CPU or memory.

10. netstat

netstat shows the network status. This command shows network ports in use and their incoming connections. However, netstat does not come out-of-the-box on Linux. If you need to install it, you can find it in the net-tools package. As a developer who experiments locally or pushes an application to a host, you may receive an error that a port is already allocated or an address is already in use. Using netstat with protocol, process and port options demonstrates that Apache HTTP server already uses port 80 on the below host.

netstat verifies that Apache is running on port 80

opensource.com

11. ip

If ip address does not work on your host, it must be installed with the iproute2 package. The subcommand address (or just ip a for short) shows the interfaces and IP addresses of your application's host. You use ip address to verify your container or host's IP address. For example, when your container is attached to two networks, ip address can show which interface connects to which network. For a simple check, you can always use the ip address command to get the IP address of the host. The example below shows that the web tier container has an IP address of 172.17.0.2 on interface eth0.

$ ip address show eth0
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether d4:3b:04:9e:b2:c2 brd ff:ff:ff:ff:ff:ff
    inet 10.1.1.3/27 brd 10.1.1.31 scope global dynamic noprefixroute eth0
       valid_lft 52072sec preferred_lft 52072sec

12. lsof

lsof lists the open files associated with your application. On some Linux machine images, you need to install lsof with the lsof package. In Linux, almost any interaction with the system is treated like a file. As a result, if your application writes to a file or opens a network connection, lsof will reflect that interaction as a file. Similar to netstat, you can use lsof to check for listening ports. For example, if you want to check if port 80 is in use, you use lsof to check which process is using it. Below, you can see that httpd (Apache) listens on port 80. You can also use lsof to check the process ID of httpd, examining where the web server's binary resides (/usr/sbin/httpd).

lsof reveals the origin of process information

opensource.com

The name of the open file in the list of open files helps pinpoint the origin of the process, specifically Apache.

13. df

You can use df (display free disk space) to troubleshoot disk space issues. When you run your application on a container orchestrator, you might receive an error message signaling a lack of free space on the container host. While disk space should be managed and optimized by a sysadmin, you can use df to figure out the existing space in a directory and confirm if you are indeed out of space.

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
devtmpfs              7.7G     0  7.7G   0% /dev
/dev/mapper/RHEL-Root  50G   16G   35G  31% /
/dev/nvme0n1p2        3.0G  246M  2.8G   9% /boot
/dev/mapper/RHEL-Home 100G   88G   13G  88% /home
/dev/nvme0n1p1        200M  9.4M  191M   5% /boot/efi
/dev/sdb1             114G   55G   54G  51% /run/media/tux/red

The -h option prints out the information in human-readable format. By default, as in the example, df provides results for everything under the root directory, but you can also limit results by providing a directory as part of your command (such as df -h /home). 

14. du

To retrieve more detailed information about which files use the disk space in a directory, you can use the du command. If you wanted to find out which log takes up the most space in the /var/log directory, for example, you can use du with the -h (human-readable) option and the -s option for the total size.

$ du -sh /var/log/*
1.8M  /var/log/anaconda
384K  /var/log/audit
4.0K  /var/log/boot.log
0 /var/log/chrony
4.0K  /var/log/cron
4.0K  /var/log/maillog
64K /var/log/messages

The example above reveals the largest directory under /var/log to be /var/log/audit. You can use du in conjunction with df to determine what utilizes the disk space on your application's host.

15. id

To check the user running the application, use the id command to return the user identity. The example below uses Vagrant to test the application and isolate its development environment. After you log into the Vagrant box, if you try to install Apache HTTP Server (a dependency) the system states that you cannot perform the command as root. To check your user and group, issue the id command and notice that you are running as the "vagrant" user in the "vagrant" group.

$ dnf -y install httpd
Loaded plugins: fastestmirror
You need to be root to perform this command.
$ id
uid=1000(vagrant) gid=1000(vagrant) groups=1000(vagrant) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

To correct this, you must run the command as a superuser, which provides elevated privileges.

16. chmod

When you run your application binary for the first time on your host, you may receive the error message "permission denied." As seen in the example for ls, you can check the permissions of your application binary.

$ ls -l
total 4
-rw-rw-r--. 1 vagrant vagrant 34 Jul 11 02:17 test.sh

This shows that you don't have execution rights (no "x") to run the binary. chmod can correct the permissions to enable your user to run the binary.

$ chmod +x test.sh
[vagrant@localhost ~]$ ls -l
total 4
-rwxrwxr-x. 1 vagrant vagrant 34 Jul 11 02:17 test.sh

As demonstrated in the example, this updates the permissions with execution rights. Now when you try to execute your binary, the application doesn't throw a permission-denied error. Chmod may be useful when you load a binary into a container as well. It ensures that your container has the correct permissions to execute your binary.

17. dig / nslookup

A domain name server (DNS) helps resolve a URL to a set of application servers. However, you may find that a URL does not resolve, which causes a connectivity issue for your application. For example, say you attempt to access your database at the mydatabase URL from your application's host. Instead, you receive a "cannot resolve" error. To troubleshoot, you try using dig (DNS lookup utility) or nslookup (query Internet name servers) to figure out why the application can't seem to resolve the database.

$ nslookup mydatabase
Server:   10.0.2.3
Address:  10.0.2.3#53

** server can't find mydatabase: NXDOMAIN

Using nslookup shows that mydatabase can't be resolved. Trying to resolve with dig yields the same result.

$ dig mydatabase

; <<>> DiG 9.9.4-RedHat-9.9.4-50.el7_3.1 <<>> mydatabase
;; global options: +cmd
;; connection timed out; no servers could be reached

These errors could be caused by many different issues. If you can't debug the root cause, reach out to your sysadmin for more investigation. For local testing, this issue may indicate that your host's nameservers aren't configured appropriately. To use these commands, you will need to install the BIND Utilities package.

18. firewall-cmd

Traditionally, firewalls were configured on Linux with the iptables command, and while it retains its ubiquity it has actually been largely replaced by nftables. A friendly front-end for nftables, and the one that ships with many distributions by default, is firewall-cmd. This command helps you set up rules governing what network traffic, both outgoing and incoming, your computer allows. These rules can be grouped into zones, so you can quickly and easily move from one set of rules to another, depending on your requirements.

The command syntax is straightforward. You use the command and some number of options, all of which are named in ways that help you almost construct a human-readable sentence. For instance, to see what zone you're currently in:

$ sudo firewall-cmd --get-active-zones``
corp
  interfaces: ens0
dmz
  interfaces: ens1

In this example, your computer has two network devices, and one is assigned to the corp zone, while the other is assigned to the dmz zone.

To see what each zone permits, you can use the --list-all option:

$ sudo firewall-cmd --zone corp --list-all
corp
  target: default
  interfaces: ens0
  services: cockpit dhcpv6-client ssh
  ports: 
  protocols: 
[...]

Adding services is just as easy:

$ sudo firewall-cmd --add-service http --permanent
$ sudo firewall-cmd --reload

Interacting with firewall-cmd is designed to be intuitive, and it has an extensive collection of predifined services, plus the ability to write nft rules directly. Once you start using firewall-cmd, you can download our firewall-cmd cheat sheet to help you remember its most important options.

19. sestatus

You usually find SELinux (a Linux security module) enforced on an application host managed by an enterprise. SELinux provides least-privilege access to processes running on the host, preventing potentially malicious processes from accessing important files on the system. In some situations, an application needs to access a specific file but may throw an error. To check if SELinux blocks the application, use tail and grep to look for a "denied" message in the /var/log/audit logging. Otherwise, you can check to see if the box has SELinux enabled by using sestatus.

$ sestatus
SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Max kernel policy version:      28

The output above indicates that the application's host has SELinux enabled. On your local development environment, you can update SELinux to be more permissive. If you need help with a remote host, your sysadmin can help you determine the best practice for allowing your application to access the file it needs. If you're interacting with SELinux frequently, download our SELinux cheat sheet for quick reference.

20. history

When you issue so many commands for testing and debugging, you may forget the useful ones! Every shell has a variant of the history command. It shows the history of commands you have issued since the start of the session. You can use history to log which commands you used to troubleshoot your application. For example, when you issue history over the course of this article, it shows the various commands you experimented with and learned.

$ history
    1  clear
    2  df -h
    3  du

What if you want to execute a command in your previous history, but you don't want to retype it? Use ! before the command number to re-execute.

Re-execute a command in your history

opensource.com

Basic commands can enhance your troubleshooting expertise when determining why your application works in one development environment but perhaps not in another. Many sysadmins leverage these commands to debug problems with systems. Understanding some of these useful troubleshooting commands can help you communicate with sysadmins and resolve issues with your application.


This article was originally published in July 2017 and has been updated by the editor.

Tags
User profile image.
As a developer, engineer, and enthusiast of cloud computing and infrastructure automation, Rosemary Wang bridges the technical and cultural barrier between infrastructure engineers and application developers.

Contributors

31 Comments

Great list. I learned a few more thanks to your wonderful article. Thank you for sharing.

Excellent article. I learned some new things. Thanks!

I can't figure out the difference between:
$ curl -I -s database:27017
And:
$ curl database:27017

Yeh they should BOTH resolve ok not just the first.

In reply to by Marcos Alano (not verified)

I use chmod u+x. yep, one can never be to careful

Great article. One `tail` trick that comes in really handy for me is to use `-F`. This follows log files as they get rotated and waits for the file to exist if it doesn't. This makes it really useful for watching daemon logs.

Wonderful article! Very helpful and thoughtful.

you still amaze me.. This is really helpful.

ls, cat and tail? Really? Any 'sysadmin' who doesn't know these by finger muscle memory should hang up their keyboard.

Totally right ! These are all basic "entry level" commands.

In reply to by Alastair Montgomery (not verified)

Even for Desktop users, GNU has hundreds of good programs.

youtube-dl can download an entire youtube channel at once
and curl can dowload a range of images/audios at once
curl -O http://example.com/file[1-100].txt

I think rsync deserves a mention.
Great article.

Instead of netstat (deprecated) sysadmin nowadays should rather know ss command.

How do get the output of "netstat -s" with ss? Please don't say "use ss -s".

In reply to by Bartosz Fenski (not verified)

The second example would resolve DNS if the first did though.
Looks like mistake in article

Thanks for pointing that out! It seems that the formatting cut out the difference out. The first command is issued from a local desktop while the second is issued from within the application's VM. The clarification was added to the text.

In reply to by dean linux (not verified)

I want to report cat abuse in example 6. Abusing cats is wrong! :D

Totally right! Demonstrating poor use of command on a tutorial for commands is bad form! Sad! :-P

In reply to by severanka (not verified)

Not sure iptables belongs here, but if it does, then firewall-cmd does as well.

I recently resolved an issue with WordPress not installing themes, plugins, etc and it was due to SELinux.... Good times...good times.

I'm surprised no one gave the perl answer to #2:

cat test.json | perl -MData::Dumper -MJSON=from_json -ne'print Dumper(from_json($_))'

I prefer wget to curl, atop to top.

But why the heck do you need to install iproute2 to find your IP address instead of just using ifconfig?!?

Oh, and why are basic tools like ls, cat, grep, and ps on the list? It makes the list sound like it's for people who think they are sysadmins but could never get hired as such.

I would add awk to the list. I couldn't live without it. It's much more powerful than most of the basic commands such as grep, if you take the time to learn it as a programming language. In this day and age of heavier languages such as Perl, Python and Ruby, most folks have forgotten how small, fast and powerful awk is. And as a bonus, you will find some version of awk on just about every Unix/Linux variant, including minimal and hardened installs. Perhaps awk deserves another dedicated article. And not just one liners. :-)

Furthermore, every "sysadmin" should know the native package managers such as rpm (yum/zypper) and dpkg (apt). Once they enlighten themselves with those tools, installing software directly from Github or using "configure/make/make install" to blather software all over a system with no governance will seem like a joke. And better yet, making packages graduates you to the next level and impresses your colleagues. :-)

Absolutely agree.

I use awk a lot combined with grep, cat, ls, find, etc.

Once I wrote a program in awk to parse a binary file, it was easier an run faster than use a program language. I was impressed.

And like others I think the list has several basic commands, not just sysadmin commands.

I'd add

ssh, scp, sftp
ftp
find (it's basic, but it has advanced iptions)
chattr
setacl, getacl
Install and manage software
Managing services
cron
rsyslog
grub
rsync

In reply to by storm9c1 (not verified)

i usually make HTTP Request by CURL.
CURL is better than socket.

Missing some useful ones like ln, rsync, strace, tac, rev, sed, awk, cut, watch, diff, more/less. Great beginners list indeed. Cheers

Nice list, Rosemary. Commands like 'cat', 'tail', 'ls' are so basic they probably belong under the category of commands that any "Linux user" should know.
Commands that every sysadmin (even beginners) should know: find, awk, traceroute, tar. There are others but these come to mind right away. Very useful article, nonetheless.

I would add 'less' to the list

Creative Commons LicenseThis work is licensed under a Creative Commons Attribution-Share Alike 4.0 International License.