Learn Expect by writing and automating a simple game

While trying to automate my workflow, I hit upon a configuration utility that defied meaningful automation. It was a Java process that didn't support a silent installer, or support stdin, and had an inconsistent set of prompts. Ansible's expect module was inadequate for this task. But I found that the expect command was just the tool for the job.

My journey to learn Expect meant learning a bit of Tcl. Now that I have the background to create simple programs, I can better learn to program in Expect. I thought it would be fun to write an article that demonstrates the cool functionality of this venerable utility.

This article goes beyond the typical simple game format. I plan to use parts of Expect to create the game itself. Then I demonstrate the real power of Expect with a separate script to automate playing the game.

This programming exercise shows several classic programming examples of variables, input, output, conditional evaluation, and loops.

Install Expect

For Linux based systems use:

$ sudo dnf install expect
$ which expect
/bin/expect

I found that my version of Expect was included in the base operating system of macOS:

$ which expect
/usr/bin/expect

On macOS, you can also load a slightly newer version using brew:

$ brew install expect
$ which expect
/usr/local/bin/expect

Guess the number in Expect

The number guessing game using Expect is not that different from the base Tcl I used in my previous article. 

All things in Tcl are strings, including variable values. Code lines are best contained by curly braces (Instead of trying to use line continuation). Square brackets are used for command substitution. Command substitution is useful for deriving values from other functions. It can be used directly as input where needed. You can see all of this in the subsequent script.

Create a new game file numgame.exp, set it to be executable, and then enter the script below:

#!/usr/bin/expect

proc used_time {start} {
	return [expr [clock seconds] - $start]
}

set num [expr round(rand()*100)]
set starttime [clock seconds]
set guess -1
set count 0

send "Guess a number between 1 and 100\n"

while { $guess != $num } {
	incr count
	send "==> "

	expect {
		-re "^(\[0-9]+)\n" {
			send "Read in: $expect_out(1,string)\n"
			set guess $expect_out(1,string)
		}

		-re "^(.*)\n" {
			send "Invalid entry: $expect_out(1,string) "
		}
	}

	if { $guess < $num } {
		send "Too small, try again\n"
	} elseif { $guess > $num } {
		send "Too large, try again\n"
	} else {
		send "That's right!\n"
	}
}

set used [used_time $starttime]

send "You guessed value $num after $count tries and $used elapsed seconds\n"

Using proc sets up a function (or procedure) definition. This consists of the name of the function, followed by a list containing the parameters (1 parameter {start}) and then followed by the function body. The return statement shows a good example of nested Tcl command substitution. The set statements define variables. The first two use command substitution to store a random number and the current system time in seconds.

The while loop and if-elseif-else logic should be familiar. Note again the particular placement of the curly braces to help group multiple command strings together without needing line continuation.

The big difference you see here (from the previous Tcl program) is the use of the functions expect and send rather than using puts and gets. Using expect and send form the core of Expect program automation. In this case, you use these functions to automate a human at a terminal. Later you can automate a real program. Using the send command in this context isn't much more than printing information to screen. The expect command is a bit more complex.

The expect command can take a few different forms depending on the complexity of your processing needs. The typical use consists of one of more pattern-action pairs such as:

expect "pattern1" {action1} "pattern2" {action2}

More complex needs can place multiple pattern action pairs within curly braces optionally prefixed with options that alter the processing logic. The form I used above encapsulates multiple pattern-action pairs. It uses the option -re to apply regex processing (instead of glob processing) to the pattern. It follows this with curly braces encapsulating one or more statements to execute. I've defined two patterns above. The first is Is intended to match a string of 1 or more numbers:

"^(\[0-9]+)\n"

 The second pattern is designed to match anything else that is not a string of numbers:

"^(.*)\n"

Take note that this use of expect is executed repeatedly from within a while statement. This is a perfectly valid approach to reading multiple entries. In the automation, I show a slight variation of Expect that does the iteration for you.

Finally, the $expect_out variable is an array used by expect to hold the results of its processing. In this case, the variable $expect_out(1,string) holds the first captured pattern of the regex.

Run the game

There should be no surprises here:

$ ./numgame.exp 
Guess a number between 1 and 100
==> Too small, try again
==> 100
Read in: 100
Too large, try again
==> 50
Read in: 50
Too small, try again
==> 75
Read in: 75
Too small, try again
==> 85
Read in: 85
Too large, try again
==> 80
Read in: 80
Too small, try again
==> 82
Read in: 82
That's right!
You guessed value 82 after 8 tries and 43 elapsed seconds

One difference you may notice is the impatience this version exhibits. If you hesitate long enough, expect timeouts with an invalid entry. It then prompts you again. This is different from gets which waits indefinitely. The expect timeout is a configurable feature. It helps deal with hung programs or during an unexpected output.

Automate the game in Expect

For this example, the Expect automation script needs to be in the same folder as your numgame.exp script. Create the automate.exp file, make it executable, open your editor, and enter the following:

#!/usr/bin/expect

spawn ./numgame.exp

set guess [expr round(rand()*100)]
set min 0
set max 100

puts "I'm starting to guess using the number $guess"

expect {
     -re "==> " { 
        send "$guess\n"
        expect {
             "Too small" {
                 set min $guess
                 set guess [expr ($max+$min)/2]
             }
             "Too large" {
                 set max $guess
                 set guess [expr ($max+$min)/2]
             }
             -re "value (\[0-9]+) after (\[0-9]+) tries and (\[0-9]+)" {
                 set tries  $expect_out(2,string)
                 set secs   $expect_out(3,string)
            }
        }
        exp_continue
    }

    "elapsed seconds" 
}

puts "I finished your game in about $secs seconds using $tries tries"

The spawn function executes the program you want to automate. It takes the command as separate strings followed by the arguments to pass to it. I set the initial number to guess, and the real fun begins. The expect statement is considerably more complicated and illustrates the power of this utility. Note that there is no looping statement here to iterate over the prompts. Because my game has predictable prompts, I can ask expect to do a little more processing for me. The outer expect attempts to match the game input prompt of `==>` . Seeing that, it uses send to guess and then uses an additional expect to figure out the results of the guess. Depending on the output, variables are adjusted and calculated to set up the next guess. When the prompt `==>` is matched, the exp_continue statement is invoked. That causes the outer expect to be re-evaluated. So a loop here is no longer needed.

This input processing relies on another behavior of Expect's processing. Expect buffers the terminal output until it matches a pattern. This buffering includes any embedded end of line and other unprintable characters. This is different than the typical regex line matching you are used to with Awk and Perl. When a pattern is matched, anything coming after the match remains in the buffer. It's made available for the next match attempt. I've exploited this to cleanly end the outer expect statement:

-re "value (\[0-9]+) after (\[0-9]+) tries and (\[0-9]+)"

You can see that the inner pattern matches the correct guess and does not consume all of the characters printed by the game. The very last part of the string (elapsed seconds) is still buffered after the successful guess. On the next evaluation of the outer expect , this string is matched from the buffer to cleanly end (no action is supplied). Now for the fun part, let's run the full automation:

$ ./automate.exp 
spawn ./numgame.exp
I'm starting to guess with the number 99
Guess a number between 1 and 100
==> 99
Read in: 99
Too large, try again
==> 49
Read in: 49
Too small, try again
==> 74
Read in: 74
Too large, try again
==> 61
Read in: 61
Too small, try again
==> 67
Read in: 67
That's right!
You guessed value 67 after 5 tries and 0 elapsed seconds
I finished your game in about 0 seconds using 5 tries

Wow! My number guessing efficiency dramatically increased thanks to automation! A few trial runs resulted in anywhere from 5-8 guesses on average. It also always completed in under 1 second. Now that this pesky, time-consuming fun can be dispatched so quickly, I have no excuse to delay other more important tasks like working on my home-improvement projects :P

Never stop learning

This article was a bit lengthy but well worth the effort. The number guessing game offered a good base for demonstrating a more interesting example of Expect processing. I learned quite a bit from the exercise and was able to complete my work automation successfully. I hope you found this programming example interesting and that it helps you to further your automation goals.