• Get the human's command
• Verbally joystick the robot
• Use Lists to keep records
• Remove elements from a list
• Calculate the difference between two times
• Use recursion to perform the learned task

See Also:

• Exclusivity of control flows in VPL
• Recursion, Activities and Exclusivity in VPL
• Extension: Learn more about CCR through Codegen

Hardware

Software

This is the final VPL tutorial in this set, so it is good time to discuss some more advanced VPL concepts. You have already learnt quite a lot about concurrency in VPL. You are used to using merges, joins and notifications to construct your programs. Something we have not talked about a great deal yet is how VPL decides which control flows are exclusive (i.e., which control flows will not be executed concurrently). We will start this tutorial by discussing how this process works.

When you write RDS code in C#, you decide, as the programmer, which control flows are exclusive and which are not. Concurrency and asynchrony is handled with the support of the Concurrency and Coordination Runtime (CCR). CCR forms a large part of the architecture underlying RDS. CCR makes it relatively easy for programmers to handle concurrency and asynchrony in their code. In a later tutorial you will learn more about managing concurrency directly using CCR. When you use VPL however much of this management is taken care of for you.

The following figure shows an example of when VPL will/will not run code concurrently.

Concurrency Example - where you set variables affects which code branches are declared exclusive.

You can see from the figure, that the top block of code (inside the dashed rectangle) will not be run concurrently with other code in the diagram. This is because there is a variable set in this part of the control flow, and variables are shared by the whole diagram. Inside this block, the control flow is split on the same event. When this happens, even though a variable is set these two branches can be run concurrently with each other. It is left up to the programmer in this instance to manage any getting and setting of variables.

The following diagram shows another interesting case.

Concurrency Example - different events from the same VPL service/activity.

In the previous example, we saw that if the control flow splits on the same event, from the same activity/service, then the two flows would be run concurrently. This is not the case if two different events split the control flow from an activity/service. Consider the case where we are receiving notifications from the an iRobotCreateRoomba service. One notification might be about the bumper being pressed and another from the cliff sensors. Clearly different notifications from the same activity/service can be quite independent. If one of these control flows sets a variable, it will be exclusive, and not run concurrently with the other control flow.

The following diagram shows one further case of interest.

Concurrency Example - same events from the same VPL service/activity - but a different instance of the code block.

We saw previously if the same event split into two different control flows from the same code-block for an activity/service, then the two control flows would be run concurrently. If you divide your code as in the preceding diagram, however, the behavior changes as described.

Exclusive control flows have priority when it comes to execution. It is important not to have exclusive control flows that run for a long period of time blocking the execution of other flows. While the execution of other flows is blocked, it is not possible to do a get on the state of the service to view the variables. For example, you should not in general call Wait on a Timer in an exclusive flow.

We will now start working on the Task Learning problem. First, we need to outline the problem in more detail.

The robot must be able execute the following drive related commands:

• "Drive Forwards" - set the DrivePower in both wheels to 0.1.
• "End Drive" - set the DrivePower in both wheels to 0.0
• "Turn Left" - turns the robot left by 45 degrees (by calling RotateDegrees).
• "Turn Right" - turns the robot right by 45 degrees.
• "Turn Around" - turns the robot 180 degrees.

There are three additional commands:

• "Begin Learning" - when in the learning state the robot records the commands it executes so that the human can teach it a task.
• "End Learning" - this command instructs the robot to stop recording commands.
• "Perform Actions" - if the robot has learnt a task (and learning is ended) it should respond to this command by executing the actions in the task.

We would like the human to be able to joy-stick the robot verbally, as such the robot should always execute any drive command it is given. The additional commands just control whether or not the robot is recording the commands it is executing.

Some of our command choices may seem unusual to you. For instance you might think that "Stop" is a better command than "End Drive". The issue is that we are using a sample service that utilizes the Windows Speech recognition software. This software is not only listening for the commands we have entered. Certain words have special meaning to this software such as "Stop" and "Start", so we avoid using those words. You will likely find it is not difficult to avoid using key-words in situations that would confuse the software.

In order to receive verbal input from the human, we need to add the speech recognition service like we did in tutorial four. Select SRNet30withUI from the Services pane. As before, whenever the human speaks, we want to know what the speech recognizer thought he/she said. So add a Calculate block, connect it to the notifications port of SRNet30withUI, and select Hypothesis. Inside the Calculate, type Result to get the string the service believes was just spoken.

We are going to pass the string that was spoken to a new activity, which we will call TaskLearner. Add this new activity to your diagram. Open it up and rename its action TakeHumanInput. Add one input, a string, called PersonSaid.

Actions and Notifications - dialog for TaskLearner should look like this.

Back in the main diagram, connect the new activity, TaskLearner, to the Calculate block.

Main Diagram - your main diagram should now look like this.

The first part of the task we are going to work on, is verbally joysticking the robot. Add a new activity to the TakeHumanInput action in TaskLearner. We will name this activity ExecuteWithoutTiming. Later when the robot is executing actions and learning, we will be using timing. For instance, we need to know how long to spend drive forwards, before executing the next command.

We will call the input to our new activity, Command (type string), and the activity requires no outputs.

Once you have setup your activity's input, it is time to code the activity itself. This is a very simple activity to write. All you need is an If statement that evaluates the Command input and decides what command to pass to the GenericDifferentialDrive. The output from the GenericDifferentialDrive blocks should then be merged and connected to the activity's output. Once you have finished, set the manifest for the GenericDifferentialDrive to iRobot.Drive.Manifest.xml.

Diagram for ExecuteWithoutTiming action - your diagram should look similar to this.

In TakeHumanInput connect up the ExecuteWithoutTiming action to the input and output of the activity. Make sure you pass in the value of PersonSaid as the Command.

Connect up the ExecuteWithoutTiming block - your diagram should look similar to this.

You are now ready to run your program, and test out the joystick capabilities! The first time you run it, you will need to enter some commands to the speech recognizer. When the speech recognizer pops up, add in the following phrases leaving the Semantic Value field blank.

• "Turn Left"
• "Turn Right"
• "Turn Around"
• "Drive Forwards"
• "End Drive"

Now is also a good time to add in the other phrases that are required:

• "Begin Learning"
• "End Learning"
• "Perform Actions"

Speech Recognizer - enter these phrases.

Recall from tutorial 4, to get the speech recognition service listening to you, say "Start listening". Also recall, that if you want VPL to receive the speech notifications, then VPL needs to have the focus, not the speech recognizer window!

We now have the ability to verbally joystick the robot, the next step is to enable the robot to learn a task and repeat it. In order to do this we need to track the robot's state.

When the robot is instructed to "Begin Learning" we need to update the robot to record that the robot is now in the Learning state. If the human says "End Learning" when the robot is in the Learning state then we will record that the robot has Learnt. It may be the case that the robot was given no commands in between these two instructions of course, so if the robot is instructed to "Perform Actions" when no actions have actually been learnt our code should be able to handle this.

On the Start page of the TaskLearner activity initialize two variables, Learning and Learnt to false.

We now want to update these two variables in TakeHumanInput according to the value of PersonSaid. Add an If block to TakeHumanInput and connect it to the input of the action. Disconnect ExecuteWithoutTiming from the input of the action and reconnect it to the output of the Else branch of the If statement (make sure the input is still being passed in correctly).

Next, add the conditions, value.PersonSaid == "Begin Learning" and value.PersonSaid == "End Learning" && state.Learning. Set the variables appropriately for each condition.

The final step is to merge the control flow from the two conditions you just created with ExecuteWithoutTiming control flow that is currently connected to the action result. The easiest way to do this is to click on the output of the last part of one of your If condition control flows and drag the cursor over to the line connected to the action output. When your cursor reaches the line, just let go of the mouse button and VPL will ask you if you would like to make a merge or a join. Choose merge, and do not forget to also hook up the remaining control flow.

TakeHumanInput - the action should currently look like this.

We are now going to start working on the part of the code where the robot must record the actions it takes to learn a task. Every time the robot receives a command from the human we are going to execute it. We will then record the command, and the time at which the robot started executing the command.

There are a number of ways you could choose to structure this code. We have chosen to encapsulate the part of the code that instructs the robot to carry out the action and queries the Timer for the time in an activity, called ExecuteAction. This activity will be very similar to the previous activity we made ExecuteWithoutTiming, except we will pass as output everything we want stored, namely the command and the time information.

Add the new activity ExecuteAction to TakeHumanInput. To start with, add one input, a string, Command. Next add the same code as for ExecuteWithoutTiming, but don't connect the merge up with the action result.

ExecuteAction - your code should currently look like this.

Our next task is to get the current time. Connect the merge to a Timer, which you can add from the Services panel and select GetCurrentTime. The current time is now available by connecting to the output of the Timer. There is no List of type Time in VPL, this means we can't store a series of Time objects. As such we will have to return the Hour, Minute, Second, and Millisecond components of the time instead. These components are all ints and can thus be stored in Lists. Note that we are assuming that our robot is not running across a day boundary in time!

Connect four calculate blocks to the Timer to extract each of the required Time components. We need to pass all this information as part of the action result, so combine these values in a join. In addition we need to pass the Command as part of the action result, so add this in as well similar to the following code.

ExecuteAction - your code should currently look like this.

Before connecting up to the action result, add the appropriate outputs in the Actions and Notifications dialog.

Actions and Notifications - add these outputs.

When you connect to the action result make sure you set each of the outputs correctly. Once you have done that, your activity is complete!

Result - make sure you correctly assign the data to the outputs.

When the robot receives commands in the Learning state it needs to record these commands so it can execute them sequentially when later instructed. The robot also needs to store the time each command was given so it can correctly carry out the task. We are going to use the List type to store this information.

Go to the Start page for the TaskLearner activity. To create a List variable drag in a green List box from Basic Activities and connect it to a Variable block. We will call the VariableActionList and its type will be List of string.

Create a List - this part of your diagram should now look like this.

We also need to create lists for the hours, minutes, seconds and milliseconds. These variables should be of type List of int.

Start page of TaskLearner - this part of your diagram should now look like this.

Next we need to write code to add to the lists. In TakeHumanInput, make add a new Activity block and name it AppendToLists. The input of this activity should be a list of each of the types we just created, as well as an element to append to each of the lists. The output should be the new lists.

Actions and Notifications - add the inputs and outputs shown here.

To append to a list we use ListFunctions from Basic Activities. Add one of these blocks to your diagram and select Append from the drop-down menu. Note that the ListFunctions work by returning a new List, equivalent to the old list, except for the requested change. Append requires two inputs, an item to append, and a List (of an appropriate type) to append it to. It is very important to note that a ListFunctions block act like join. The control flow must reach the ListFunctions block at both inputs.

Keeping all this mind, it is time to fill out the code in your AppendToLists activity! As always, remember to carefully set the outputs.

Append to the Lists - your activity should look similar to this.

We can now use the two activities we made, ExecuteAction and AppendToLists to write the code for the case when the robot receives a command when it is in the Learning state.

Start by adding another condition to the If statement in TakeHumanInput - state.Learning. If this condition is satisfied we want to call the action in ExecuteAction, passing in value.PersonSaid. ExecuteAction will return the Command, as well as the Hour, Minute, Second, and Millisecond at which it was begun. We need to pass these values to AppendToLists, so connect the two activities. The Data Connections dialog for this line is shown in the following figure.

Data Connection - your values should look like the ones here.

AppendToLists returns new lists, so we need to assign these new lists to our list variables. Make these assignments and combine the data flow in a join. You can then connect the join to the result merge and you are finished handling the case where the robot receives a command when Learning!

Learning Case - code to handle the case when the robot receives a command in the Learning state.

The final command we need to be able to respond to is the "Perform Actions" command. In other words, we need to write the code to carry out a sequence of actions the robot has learned.

In TakeHumanInput add the appropriate condition to the If statement: value.PersonSaid == "Perform Actions" && state.Learnt. Next, create a new activity, and name it ExecuteLearnedActions. This activity has no outputs, but requires each of the lists as input. Connect your new activity to the new If condition, and connect its output to the result merge.

TakeHumanInput - your code for this action should look similar to this.

We need some helper code to write ExecuteLearnedActions. We are going to write the action in ExecuteLearnedActions as a recursive method. This will be good practice at writing recursion in VPL! The idea behind the recursion, is we will pass the action the list of commands and time information. It will start executing the first command in the list and then call the action again after waiting an appropriate amount of time, passing in the lists with the first elements popped off the top.

Add a new Activity box to your code and called it PopFromLists. The input and output to this activity are the lists we need to pop from.

To pop from a list we can use a ListFunctions block and select RemoveItem from the drop-down menu. This function requires two inputs, a List and an index (int). Since we want to pop from each of the lists, the index we want is zero. When you finish writing this activity, your code should look similar to the code in the figure following. Remember to set the return values.

PopFromLists - your code for this action should look similar to this.

If the command currently at the top of the Command list is not "Drive Forwards", we can simply call ExecuteActionWithoutTiming, PopFromLists and recurse. However, if the Command is "Drive Forwards", we need to compute the amount of time between when this command was started and when the next command in the list was started. This is the amount of time we should wait before recursing.

Create a new activity called CalculateTimeDifference. The input is the start time and the end time (in terms of the hour, minute, second and millisecond components) and the output should be the int valued time difference in milliseconds.

Actions and Notifications - make these inputs and output.

To calculate the difference in terms of milliseconds, you can simply calculate the pair-wise difference, normalizing each difference to milliseconds, and then add the result of each difference.

CalculateTimeDifference - code to calculate the time difference.

Let's start by renaming the action in ExecuteLearnedActions, Run. Now connect an If block to the input of Run. We need to test if we are done or not. So if value.ActionList.Count > 0 we are going to continue, but otherwise we are done, so we want to connect the Else case to the output of Run. By doing this, we make the recursion terminate when there are no more commands in the learned task to execute.

If the lists are non-empty there are two cases. Add another If block, and connect it to the first one. If the command we need to execute is "Drive Forwards" then we have the more complicated case, where we have to handle the timing.

To access an element of a list in VPL you can use the square brackets notation used for arrays in many programming languages. The condition then is value.ActionList[0] == "Drive Forwards".

We will start by working on the Else branch of this If statement. The If block forwards on the input lists, so we can connect both a copy of our PopFromLists activity to the Else branch, and our ExecuteActionWithoutTiming activity. To the PopFromLists we need to pass each of the lists, and to the ExecuteActionWithoutTiming, value.ActionList[0]. Note that since we are passing by value to PopFromLists this is not a problem. Once both of these activities are complete, we want to call Run recursively. To wait for them both to complete, we add a join. To get a block for ExecuteLearnedActions to connect to the join, one method is to just left-click on the join and drag the cursor to where you want the activity to appear. When you release the mouse-button you will be able to select the activity.

Case where command is not Drive Forwards - this part of your code should currently look like this.

Remember that when values pass through a join, the join label gets attached to them, so when you pass your inputs into Run they will have the form Lists.ActionList.

Now, if the condition value.ActionList[0] == "Drive Forwards" evaluates to true, we want to connect to PopFromLists, and ExecuteWithoutTiming as before, but we also need to connect to CalculateTimeDifference.

When the activities ExecuteWithoutTiming and CalculateTimeDifference both return we need to wait the appropriate amount of time. Add a join to wait and until both activities return and then set the Timer to wait for value.TimeDiff.MillisecondDiff. Note, it is OK to use Wait in this instance because we are not setting any variables in this code. As such, none of this code is exclusive and having a Wait will not cause bad blocking behavior.

Once we have finished waiting, and PopFromLists has also returned we can recurse!

The final step is to connect the result of our recursive calls up to the result of Run. Note that is just like in Java or C, when you type return methodName(input); as your recursive call. We do this in the normal way using merges. The final code for Run is shown in the following image.

Run: ExecuteLearnedActions - your code should currently look like this.

You can now try out your program! Make sure the timing is working fairly accurately. You should also be able to have the robot incrementally learn a task, by having it begin and end learning multiple times.

There are plenty of ways you can extend this exercise. This is already a large project for VPL however, so it might make sense to work in C# instead. One obvious extension is to have the robot learn multiple, named tasks. You could also extend the complexity of the command language.

When we wrote our recursive action, you might have noticed we did not ever set a variable. This was actually the key to the whole thing working! If you are using recursion where the result depends on the output of the recursive call i.e., where the recursive call is connected to the activity result, as in our case, you cannot set a variable. If you set a variable the action will become exclusive and your code will get stuck at the recursive call.

This does not mean that setting variables and recursion can't go together, but for this you need to use a different paradigm. Instead of having the action return, you can use notifications instead. When your action detects that on this call processing is complete, it can post to the notification port of the action (the round connection instead of the square connection). To see an example of using notifications in this manner, watch the second VPL video tutorial available from the DSS homepage.

Similar to the recursive example, you need to be careful about setting variables when one action in an activity calls another action in the same activity. If no variables are being set this can work really well as a way of breaking up code. However, if the two actions are declared exclusive and depend on each other, your code will get stuck!

In general, if you are not sure about how something works, the best way to learn is to write a small test program and see what happens! You can even watch your program run in the debugger to see how it is working in more detail.

At the start and end of this tutorial, we discussed how concurrency is handled in VPL. When you write C# code you will use CCR directly to handle concurrency. One way you can start to learn more about how the VPL concepts relate to the use of CCR primitives is to write some small VPL code examples and then generate code.

When you read the generated code, you can refer to the CCR user guide (part of the RDS documentation) to find out more details.

In this tutorial, you learned how to:

• Get the human's command
• Verbally joystick the robot
• Use Lists to keep records
• Remove elements from a list
• Calculate the difference between two times
• Use recursion to perform the learned task