In part 1 and part 2 of this series on the Mycroft open source voice assistant, I laid the groundwork for learning how to create a skill. In part 3, I walked through creating an outline for a skill and recommended creating the skill in pure Python first to ensure the methods work as intended. That way, when things go wrong, you know it is related to how your Mycroft skill is constructed and not the code itself.

In this article, you will enhance the outline from part 3 by adding:

• Mycroft entities
• Padatious intents
• Adapt intents
• Dialogs
• Conversational contexts

The code for this project can be found in my GitLab repository.

Let's get to it!

Improve your skill

As a reminder, the purpose of this project is to use Mycroft to add items to shopping lists in the OurGroceries app. However, this tutorial can be applied to a variety of home automation applications like turning on your lights, getting the morning weather report, or controlling your entertainment systems.

Here's what the skill's outline looks like so far:

from mycroft import intent_file_handler, MycroftSkill, intent_handler
from mycroft.skills.context import adds_context, removes_context

class OurGroceriesSkill(MycroftSkill):
    def __init__(self):
        MycroftSkill.__init__(self)

    # Mycroft should call this function directly when the user
    # asks to create a new item
    def create_item_on_list(self, message):
        pass

    # Mycroft should also call this function directly
    def create_shopping_list(self, message):
        pass

    # This is not called directly, but instead should be triggered
    # as part of context aware decisions
    def handle_dont_create_anyways_context(self):
        pass

    # This function is also part of the context aware decision tree
    def handle_create_anyways_context(self):
        pass


    def stop(self):
        pass

def create_skill():
    return OurGroceriesSkill()

As it stands, Mycroft will load the skill successfully, but it won't do anything because all the methods have the command pass in them. For now, ignore the __init__(self) method and start working with the create_item_on_list method. From the comments, you can see the intent is for Mycroft to call this method directly. That means that you need to declare an intent. How do you do that?

Working with intents

You may have noticed that the outline mycroft-msk created (in the third article) has what looks like a weird function, @intent_file_handler('ourgroceries.intent'), above the handle_test method. These are special notations in Python called decorators (if you want, head over to Real Python for a primer on Python decorators). For this tutorial, it is sufficient to know that a decorator is a way to pass your function into a prebuilt function developed by Mycroft. This saves a lot of work and boilerplate code.

Recall from part three of this series that this project uses two intent parsers: Padacious and Adapt, which I described in the second article.

Padatious intents

So how do you know which decorator to use and where to use it? Great questions! I'll start with Padatious, which is the easier of the two to understand. If you recall from the second article, Padatious skills are trained based on phrases that the skill's developer decides are relevant for the skill. As Mycroft may install many skills using the Padatious intent engine, each intent is given a score by the neural network module that Mycroft employs. Mycroft then chooses the highest-scoring intent and executes its functions.

The phrases which Mycroft will use to train the intent are placed inside a file with a .intent file extension. You can have multiple .intent files, but you must reference each file explicitly. That means if you have create.item.intent and create.category.intent, there will be no confusion about which file your variables are populated from, as you must call them by file name. As you can see in the output from mycroft-msk, the decorator is intuitively named @intent_file_handler(). Simply use the name of the file as the argument for the decorator, such as @intent_file_handler("create.item.intent").

Think about what phrases someone might use to add an item to a shopping list. Since the motivating factor for this skill was using Mycroft to create a grocery list, the example code uses food-related terms, but you could use generic terms. With that said, here are some phrases you might say to add an item to your grocery list:

• Add tomatoes to my shopping list
• Add tomatoes to the grocery list
• Add tomatoes to Costco list

You may choose to have some grammatically incorrect phrases as well, to account for Mycroft misunderstanding the user's voice. From the list above, what pieces of information are programmatically relevant? tomatoes, shopping list, grocery list, and Costco list. The official documentation refers to this type of object as an entity. You can think of an entity as a variable, if that makes more sense to you. This will become clearer later when you create an intent file. While the mycroft-msk command will put intents in locale/en-us by default, I put mine under vocab/en-us/. Why? Well, that's because the Adapt intent parser stores its files in vocab, and I prefer to keep all my intent files in the same location. My file vocab/en-us/create.item.intent starts with:

add {Food} to my {ShoppingList}

This defines the entities Food and ShoppingList.

IMPORTANT NOTE:  Padatious entities are not case-sensitive, and Padatious interprets everything in lower-case. For example, ShoppingList will be shoppinglist.

Now that you have an intent, have Mycroft say a phrase containing your entities. Don't forget to add the intent decorator! Your new function will look like this:

    @intent_file_handler("create.item.intent")
    def create_item_on_list(self, message):
        """
        This function adds an item to the specified list

        :param message:
        :return: Nothing
        """
        item_to_add = message.data.get('food')
        list_name = message.data.get('shoppinglist')
        self.speak("Adding %s to %s" % (item_to_add, list_name))

The graphic below uses three phrases:

• Add tomatoes to my shopping list
• Add nails to my hardware list
• Add buns to groceries list

Mycroft will not be able to figure out the intent behind one of these phrases. Can you guess which one and why?

Image by:

^{(Steve Ovens, CC BY-SA 4.0)}

In case the video was a little too fast for you, here's the answer: Mycroft can't process the phrase add buns to groceries list because it is missing the keyword my. The intent explicitly says add {Food} to my {ShoppingList}. Without the word my as input from the user, the Padatious intent for the skill scores low and, therefore, Mycroft doesn't choose this skill to handle the request. The simplest solution is to add a new line to your intent file, like this:

add {Food} to {ShoppingList}

Mycroft can often reload skills when it detects a change, but I prefer to restart Mycroft's skills section to make sure. I also clear the logs a lot during testing, so I run the following command to do everything in one line:

 ./stop-mycroft.sh skills;sudo rm -f /var/log/mycroft/skills.log; ./start-mycroft.sh skills; mycroft-cli-client

Testing the skill after Mycroft restarts yields the following results:

add buns to groceries list                                                                 
 >> Adding buns to groceries list 

In case it is not clear, any response Mycroft makes in the mycroft-cli-client is prefixed by >> to indicate its response. Now that you have a basic intent, go back and review the objectives for this skill from part 3 of this series:

1. Login/authenticate
2. Get a list of the current grocery lists
3. Add item to a specific grocery list
4. Add item to a category under a specific list
5. Be able to add a category (since OurGroceries allows items to be placed in categories)

Ignore the first two items for now—those deal with the online portion of the project, and you need to complete the other objectives first. For the third item, you have a basic intent that, in theory, should be able to take the entities Mycroft detects and turn them into variables in the Python code. For the fourth item on the list, add two new lines to your intent:

add {Food} to my {ShoppingList} under {Category}
add {Food} to {ShoppingList} under {Category}

You also need to alter your function slightly. When using the Padatious intent parser, entities are returned via the message.data.get() function. This function will return None if the entity is undefined. In other words, if Mycroft cannot parse {Category} from an utterance the user makes, message.data.get() will return None. With this in mind, here is some quick test code:

    @intent_file_handler("create.item.intent")
    def create_item_on_list(self, message):
        """
        This function adds an item to the specified list

        :param message:
        :return: Nothing
        """
        item_to_add = message.data.get('food')
        list_name = message.data.get('shoppinglist')
        category_name = message.data.get('category')
        if category_name is None:
            self.speak("Adding %s to %s" % (item_to_add, list_name))
        else:
            self.speak("Adding %s to %s under the category %s" % (item_to_add, list_name, category_name))

Here is an example that tests these code changes:

Image by:

^{(Steve Ovens, CC BY-SA 4.0)}

In the example, Mycroft responds with >> Adding nails to my hardware list under, yet the only time you told Mycroft to say the word under is when category_name has a value other than None. This is because the intent parser is interpreting the word under as a part of the entity ShoppingList. Because the utterance had the word my in it, the sentences that matched the utterances could have been either:

1. add {Food} to my {ShoppingList}
   
   or
2. add {Food} to my {ShoppingList} under {Category}

Since the user did not state the {Category}, Mycroft selected the first statement as the most correct. This means that anything after the word my would be cast into the entity {ShoppingList}. Since, therefore, {Category} is None, Mycroft speaks, "Adding nails to my hardware list under" instead of "Adding nails to my hardware list under None."

Padatious may seem a bit simplistic at first. For each phrase you need Mycroft to match, simply add a line to the intent file. However, with a complex intent, you could have several dozen lines attempting to cover all the different utterances you want to handle.

There is another option that may be worth considering. Padatious intents support parentheses expansion. This means you can use a form of OR statements to cut down on the number for lines in an intent. Going back, the example is trying to account for three cases:

add {Food} to my {ShoppingList}
add {Food} to my {ShoppingList} under {Category}
add {Food} to the {ShoppingList}
add {Food} to the {ShoppingList} under {Category}
add {Food} to {ShoppingList}
add {Food} to {ShoppingList} under {Category}

If you want to rewrite this using an OR statement to combine the my and the keywords, you could write:

add {Food} to ( my | the ) {ShoppingList}
add {Food} to ( my | the ) {ShoppingList} under {Category}
add {Food} to {ShoppingList}
add {Food} to {ShoppingList} under {Category}

This removes two lines from the intent. Parentheses expansion also supports making something optional. So, if you want to make the and my optional and thus allow for the phrase add {Food} to {ShoppingList}, it would look like:

add {Food} to ( | my | the ) {ShoppingList}
add {Food} to ( | my | the ) {ShoppingList} under {Category}

This simple change covers all three scenarios (once you restart the Mycroft skills subsystem). You can go a step further and condense this into a single line if you wish:

add {Food} to ( | my | the ) {ShoppingList} ( | under {Category})

Note: For easier readability, use white spaces in your intent parentheses expansion.

To summarize the important points about Padatious intent parsing:

• You have to give several examples of phrases for Mycroft to come up with the correct match.
• Padatious intents use entities such as {Food} to identify object values that can be retrieved from your Python code.
• Entities are always lower-case regardless of how you declare them in the intent file.
• If an entity cannot be parsed from an utterance, its value is None.
• The decorator for Padatious intents is @intent_file_handler('my.intent.file.intent').

Adapt intents

Unlike Padatious intents, where you specify entities in the intent file, the Adapt intent parser works with a series of keywords that work in combination with regular expression (regex) files to attempt to capture an entity. You would use Adapt over Padatious when you:

1. Expect the utterance to be complex and the more robust parsing of regex is required
2. Want or need Mycroft to be context-aware
3. Need intents to be as lightweight as possible

That said, the voc files Adapt uses are quite flexible. They can include a single word (as shown in the official documentation), or they can include the start of a sentence you want to react to.

As one goal of this project is to have Mycroft create a new shopping list in the OurGroceries app, I wanted to add in some rudimentary checking so that the user is informed if a list with a similar name exists and asked if they still want to create a new list. This should cut down on list duplication and misplacement of items.

Mock up some code, and then you can deal with the vocab and regex files. While you could use Pytest or similar unit tests to assert specific values, for the sake of simplicity, you will create a list called "shopping list." The Python mock function will look like this:

    def create_shopping_list(self, message):
        fake_list = ["shopping list"]
        self.new_shopping_list_name = message.data['ListName'].lower()
        for current_shopping_list in fake_list:
            try:
                if self.new_shopping_list_name in current_shopping_list:
                    if self.new_shopping_list_name == current_shopping_list:
                        self.speak("The shopping list %s already exists" % self.new_shopping_list_name )
                        break
                    else:
                        self.speak("I found a similar naming list called %s" % current_shopping_list)
                        # This hands off to either handle_dont_create_anyways_context or handle_create_anyways_context
                        # to make a context aware decision
                        self.speak("Would you like me to add your new list anyways?", expect_response=True)
                        break
                else:
                    self.speak("Ok creating a new list called %s" % self.new_shopping_list_name)
            except Exception as ex:
    		print(ex)
                pass

Notice I am using a forloop to iterate over fake_list. That is because, in theory, multiple lists will be returned from the OurGroceries app. Also note the try/except block; I have given a general pass on the exception because, right now, I don't know what kind of exceptions I may run into. As you use and debug your code, you can tighten this up a bit.

Another line to note is:

self.speak("Would you like me to add your new list anyways?", expect_response=True) 

This bit of code will have Mycroft prompt the user for a response and store the result. I'll talk more about this bit of code in the conversational contexts section.

Regular expressions, entities, and Adapt intents

Now you have some pseudo code, but you need to add the decorator for Mycroft to action your code. You need to create three files for this to work: two vocab files and one regex file. The regex file, which I will name add.shopping.list.rx, looks like this:

start a new list called (?P<ListName>.*)
create a new list called (?P<ListName>.*)
add a new list called (?P<ListName>.*)

You could make this a one-liner, but for simplicity's sake, keep it as three lines. Note this strange-looking notation: (?P<ListName>.*). This is the part of the code that captures and creates the entity. The entity, in this case, is called ListName. For checking your syntax, I recommend Pythex. It is very helpful when I am debugging my regex (I'm pretty terrible at regex).

IMPORTANT NOTE: Adapt intents are case sensitive.

Adapt and vocab files

Now that your regex includes the full sentences you expect, create your two vocab files. The first file is called CreateKeyword.voc. As you can surmise from the file name, all the words you want to associate with the create action should reside here. This file is very simple:

start a new
create a new
add a new

In the documentation, you will often see only a single word per line. However, due to some Mycroft default skills using start and create, I need to add words so that Mycroft will pick my skill appropriately.

The second file is even easier. It's called ListKeyword.voc and has a single word in it:

list

With these files defined, you can now construct your decorator:

@intent_handler(IntentBuilder('CreateShoppingIntent').require('CreateKeyword').require('ListKeyword').require("ListName"))

The first argument in the IntentBuilder is 'CreateShoppingIntent'; this is the name of the intent and is completely optional. If you want to leave this blank, you can. The require section is a bit confusing. When it comes to keywords, the argument for require is the name of the file without the file extension. In this case, one of the files is called ListKeyword.voc, so the argument being passed into require is just 'ListKeyword'.

While you can name your vocab files anything you want, I highly recommend using the word Keyword in the file so that when you are building your intent_handler decorator, it is clear what you are requiring.

If require is actually an entity from a regex file, the argument for require is the name of the entity as you defined it in the regex. If your regex was start a new list called (?P<NewList>.*), then you would write require('NewList').

Restart the Mycroft skills subsection and try it out. You should see this in the Mycroft command-line interface:

 add a new list called hardware
 >> Ok creating a new list called hardware
 
 create a new list called hardware
 >> Ok creating a new list called hardware
 
 start a new list called hardware
 >> Ok creating a new list called hardware

Conversational contexts

Great, it works! Now add the following decorator to your function:

@adds_context("CreateAnywaysContext")

This decorator is tied to the conversational context that Mycroft supports. Conversational contexts are essentially where you can speak normally to Mycroft and it will understand your meaning. For example, you could ask: "Who was John Quincy Adams?" After Mycroft responds, saying something like "John Quincy Adams was the sixth president of the United States," you could ask: "How old was he when he became president?" If you ask the second question first, Mycroft has no way to know who the pronoun he refers to. However, in the context of this conversation, Mycroft understands that he refers to John Quincy Adams.

Getting back to creating a conversational context, the argument for its decorator is the name of the context. This example calls the context CreateAnywaysContext and hence, the full decorator is @adds_context("CreateAnywaysContext"). This mock method is now complete. However, you now need to add two simple methods to handle the user's feedback. You can simplify the grocery list skill by requiring either a yes or a no answer. Create a YesKeyword.voc and a NoKeyword.voc, and place the words yes and no in them, respectively.

Now create two more methods in your Python:

@intent_handler(IntentBuilder('DoNotAddIntent').require("NoKeyword").require('CreateAnywaysContext').build())
@removes_context("CreateAnywayscontext")
def handle_dont_create_anyways_context(self):
    """
    Does nothing but acknowledges the user does not wish to proceed
    Uses dont.add.response.dialog
    :return:
    """
    self.speak_dialog('dont.add.response')

    @intent_handler(IntentBuilder('AddAnywaysIntent').require("YesKeyword").require('CreateAnywaysContext').build())
@removes_context("CreateAnywayscontext")
def handle_create_anyways_context(self):
    """
    If the user wants to create a similarly named list, it is handled here
    Uses do.add.response.dialog
    :return:
    """
    self.speak_dialog('do.add.response')

There are two things here you have not seen so far:

1. @remove_context
2. self.speak_dialog

If a method that requires CreateAnywaysContext is called, the decorator @remove_context gets rid of the context so that Mycroft does not accidentally action a context more than once. While multiple contexts can be applied to a method, this project will not use them.

Dialogs

Dialogs are files that have several prebuilt responses that Mycroft can pick from. These dialogs are stored in dialog/{language tag}/, and the language tag is based on the IETF standard. Examples can be found on Venea.net in the IETF LanguageTag column.

Mycroft picks randomly from the list of sentences in a specified dialog file. Why would you use a dialog file instead of implementing self.speak in Python? The answer is simple: When you create and use a dialog file, you do not have to change the Python code to support other languages.

For example, if the dialog file called dont.add.response.dialog exists under en-us with the following content:

Ok... exiting
Gotcha I won't add it
Ok I'll disregard it
Make up your mind!

You could also create de-de/dont.add.response.dialog with the following content:

Ok... Beenden
Erwischt Ich werde es nicht hinzufügen
Ok, ich werde es ignorieren.
Entscheiden Sie sich!

In your Python code, you would use self.speak_dialog('dont.add.response') to randomly select one of the answers for Mycroft to use. If a user's Mycroft language is set to German, Mycroft will automatically select the correct dialog and play the dialog in German instead of English.

To wrap up this section, create two files under dialog/en-us. For dont.add.response.dialog, use the same content as in the above example. For do.add.response.dialog, use:

Ok adding it now
Sure thing
Yup yup yup

At this point in this project, your tree should look something like this:

├── dialog
│   └── en-us
│       ├── do.add.response.dialog
│       └── dont.add.response.dialog
├── __init__.py
├── regex
│   └── en-us
│       └── add.shopping.list.rx
└── vocab
    └── en-us
        ├── create.item.intent
        ├── CreateKeyword.voc
        └── ListKeyword.voc

Note that I created the files by hand. If you used the mycroft-msk create method, you might have locale directories, settingsmeta.yaml, or other artifacts.

Wrapping up

We've covered a lot so far. You have implemented the Padatious intent parser to, theoretically, add a new item to a list, whether or not you put it under a category. You have also used the Adapt intent parser to add a new category. You used conversational context to prompt the user for confirmation if a similar list already exists. Finally, you learned the concept of dialogs as a way for Mycroft to provide varied confirmation responses to the user.

Currently, the code looks like:

from mycroft import intent_file_handler, MycroftSkill, intent_handler
from mycroft.skills.context import adds_context, removes_context
from adapt.intent import IntentBuilder

class OurGroceriesSkill(MycroftSkill):
    def __init__(self):
        MycroftSkill.__init__(self)

    # Mycroft should call this function directly when the user
    # asks to create a new item
    @intent_file_handler("create.item.intent")
    def create_item_on_list(self, message):
        """
        This function adds an item to the specified list

        :param message:
        :return: Nothing
        """
        item_to_add = message.data.get('food')
        list_name = message.data.get('shoppinglist')
        category_name = message.data.get('category')
        if category_name is None:
            self.speak("Adding %s to %s" % (item_to_add, list_name))
        else:
            self.speak("Adding %s to %s under the category %s" % (item_to_add, list_name, category_name))

    # Mycroft should also call this function directly
    @intent_handler(IntentBuilder('CreateShoppingIntent').require('CreateKeyword').require('ListKeyword').require("ListName"))
    def create_shopping_list(self, message):
        fake_list = ["shopping list"]
        self.new_shopping_list_name = message.data['ListName'].lower()
        for current_shopping_list in fake_list:
            try:
                if self.new_shopping_list_name in current_shopping_list:
                    if self.new_shopping_list_name == current_shopping_list:
                        self.speak("The shopping list %s already exists" % self.new_shopping_list_name )
                        break
                    else:
                        self.speak("I found a similar naming list called %s" % current_shopping_list)
                        # This hands off to either handle_dont_create_anyways_context or handle_create_anyways_context
                        # to make a context aware decision
                        self.speak("Would you like me to add your new list anyways?", expect_response=True)
                        break
                else:
                    self.speak("Ok creating a new list called %s" % self.new_shopping_list_name)
            except AttributeError:
                pass
    # This is not called directly, but instead should be triggered
    # as part of context aware decisions
    @intent_handler(IntentBuilder('DoNotAddIntent').require("NoKeyword").require('CreateAnywaysContext').build())
    @removes_context("CreateAnywayscontext")
    def handle_dont_create_anyways_context(self):
        """
        Does nothing but acknowledges the user does not wish to proceed
        Uses dont.add.response.dialog
        :return:
        """
        self.speak_dialog('dont.add.response')

    # This function is also part of the context aware decision tree
    @intent_handler(IntentBuilder('AddAnywaysIntent').require("YesKeyword").require('CreateAnywaysContext').build())
    @removes_context("CreateAnywayscontext")
    def handle_create_anyways_context(self):
        """
        If the user wants to create a similarly named list, it is handled here
        Uses do.add.response.dialog
        :return:
        """
        self.speak_dialog('do.add.response')

    def stop(self):
        pass

def create_skill():
    return OurGroceriesSkill()

In the next article, I will go into logging, getting settings from the web UI, and continuing to fill out the skill into something more useful.