Imagine how easy it would be to learn skating, if only it doesn’t hurt everytime you fall. Unfortunately, we, humans, don’t have that option. Robots, however, can now “learn” their skills on a simulation platform without being afraid of crashing into a wall. Yes, “it learns“! This is possible with the reinforcement learning algorithms provided by OpenAI Gym and the ROS Development Studio.
You can now train your robot to navigate through an environment filled with obstacles just based on the sensor inputs, with the help of OpenAI Gym. In April 2016, OpenAI introduced “Gym”, a platform for developing and comparing reinforcement learning algorithms. Reinforcement learning is an area of machine learning that allows an intelligent agent (for example, robot) to learn the best behaviors in an environment by trial-and-error. The agent takes actions in an environment so as to maximize its rewards. We have deployed the gym_gazebo package from Erle Robotics S.L. in the ROS Development Studio. It enables the users to test their reinforcement learning for their robots in Gazebo.
How to Train your Robot
Robot training to navigate through an environment with obstacles
In this example we will be seeing how a turtlebot is able to learn navigating through an environment without hitting an obstacle. The turtlebot will use a reinforcement learning method known as Q-learning.
There are four environments already available for the user to test their simulations with. These environments can be launched using the respective launch files:
GazeboCircuitTurtlebotLidar_v0.launch
GazeboCircuit2TurtlebotLidar_v0.launch
GazeboRoundTurtlebotLidar_v0.launch
GazeboMazeTurtlebotLidar_v0.launch
The images of the different environments are given below:
The various environments already available:
Circuit
Circuit2
Round
Maze
The user is requested to try out the existing environment before developing their own environments for training the robot. An environment is where the robot’s possible actions and rewards are defined. For example, in the available environments, there are three possible actions for the Turtlebot robot:
– Forward ( with a reward of 5 points) – Left ( with a reward of 1 point) – Right ( with a reward of 1 point)
If it collides into the walls, then the training episode ends (with a penalty of 200 points). The turtlebot has to learn to navigate through the environment, based on the rewards obtained from different episodes. This can be achieved using the Q-learning algorithm. Let us see, how it can be done using the ROS Development Studio.
Using the ROS Development Studio for training the robot
First, we have to set the path in the jupyter-notebook as given below:
The python scripts in the folder gym_construct/src/ help us simulate the reinforcement learning techniques for a Turtlebot. Currently, the number of episodes has been set to 20. Feel free to increase the number of episodes in the python scripts (usually up to 5000) to actually train the robot to navigate the environment completely.Run the only the script corresponding to the environment:
The robot undergoes several training episodes and each of these episodes are rewarded based on the number of steps taken by the robot before hitting the environment. We will be able to see that the robot incrementally increases its rewards over time compared to its previous versions. With a very large number of episodes, the robot will learn to navigate the environment without hitting the obstacle.
Plotting the learning curve of the robot
The machine learning algorithm generates the output files in the output directory specified in the python script file. In order to plot the curve, we run the display_plot.py. But before that, don’t forget to restart the kernel and set the path once again.
As the number of episodes increases, we will see that the robot’s mean rewards also increases. The user can choose his own robot, environments, action and rewards for testing his reinforcement learning algorithms in OpenAI Gym and RDS. Watch the video below for more on this.
Video describing the procedure to run OpenAI Gym on RDS
I hope you were able to follow the tutorial. So, that’s all folks. It’s now up to you to develop and test reinforcement learning algorithms in OpenAI Gym and RDS.
What if you could program your ROS based robots from any type of computer?
What if you could test your ROS programs on a robot simulation prior to send to the real robot?
What if you could perform all those actions from a web browser without having to install or prepare anything?
Welcome to the
ROS Development Environment (RDS)
The RDS is the latest product created by The Construct. It allows robotics programmers to concentrateon:
Programming their robots.
Testing the programs in real time on the integrated simulator.
Connect the environment to the real robot and test with the robot.
And everything integrated on a single (web based) development environment… pretty cool eh?
Work Flow
Any ROS development starts by selecting the robot you want to program to. Several robots are available off-the-shelf for programming and many more are in the process of being integrated. Additionally, you can create your own robot.
Selecting a robot is as simple as pressing play on the picture of the robot you want to program for.
After a while, the development environment will appear, where you can learn about the robot you are programming for, program it, and test in the simulation.
The environment is fully configurable, allowing you to change windows, and even detach to independent tabs and monitors.
The whole development environment is web based. This means that you can code, test, and debugfrom any type of computer.
Useful graphical ROS tools are also included in the web RDS, like RViz, rqt, or any other that you may imagine in the future.
Simulations are Gazebo or Webots compatible. This means that if you have already existing simulations for any of those two simulators, you can upload them to the RDS and start using them in your programs.
We provide a clear distinction between your private simulations and the ones provided off-the-shelf.
PUBLIC SIMULATIONS: the robot simulations provided by The Construct off-the-shelf. Anybody can access them, clone, modify, make them private, etc…
PRIVATE SIMULATIONS: the ones that you create, upload, or generate from the public ones. Only you can access them.
Forget about having to have that so expensive robot in order to learn how to program. You can learn to program Robonaut, Baxter, Pepper, PR2 using the ROS Development Studio without having to waste a penny.
You can create programs for robots that you don’t own, by using the RDS in the same way as mobile apps are created at present.
Connection to real robot
Once your program runs smoothly on the simulation, you can decide to switch the execution of the program to the real robot. For that purpose, you can connect the RDS to the robot by establishing a VPN and switching the roscore. Now you can test in the real robot what you have developed in the simulation.
Conclusions
Can you imagine developing ROS programs on a Windows computer?
Or to forget about ROS versions?
What about do not caring about your computer’s graphic card?
This is the future that the ROS Development Studio proposes you.
If you are a ROBOT PROGRAMMER and want to have a specific robot on the RDS, contact us.
If you are a ROBOTICS COMPANY and want your robot be included in the RDS, contact us.
Our contact details: info@theconstructsi.ai
If you want to work in the robotics industry, we provide you with a complete path: The Robotics Developer Masterclass (roboticsdeveloper.ai), a 6-month program that helps you master robotics development from scratch and gets you 100% job-ready to work at leading robotics companies.
Want to learn the latest robotics programming standards in 5 days?
Want to avoid all that tedious documentation in order to learn what really matters to program a real robot?
Then keep reading…
Learn to program robots is difficult.
I mean real robots.
I am not talking about learning to program Lego and the like. I’m talking about making a Reem robot move around a shopping mall helping people to carry their bags. Or a Fetch robot cooperate on a warehouse with humans helping them to fetch the packages.
The programming of those robots is difficult.
In order to simplify the programming of those robots, many frameworks have been created:
Those are the real thing. They are frameworks that are installed in real robots and used to make them work. I’m not talking about using Matlab for programming your algorithm in a robot. Matlab is not used in real life, only in academia for non-replicable results. There is no robot out there running in Matlab. Here we are talking about the real thing.
But don’t be wrong: even if those frameworks are quite complex, anybody can learn to program them.
The question is how long does it take?
Why does it take so long to learn to program robots?
We have identified a series of problems that prevent quick learning of robotics.
1. You don’t have access to robots to play with
There are many different robots available and they are really expensive. Acquiring, maintaining, and explaining the hardware are real problems that limit class size and the learning experience of the students.
Only a subset of the course material of a robotics curriculum can be assessed conceptually or mathematically. Demonstrating a thorough understanding of an algorithm requires implementation. Unlike purely computational algorithms, which can be assessed automatically using test-input, authentic assessment scenarios are difficult to create for a robotics class.
Hence students can hardly practice what they are learning.
We strongly believe in the CONSTRUCTIONISM approach. Defined by Seymor Papert around 1980, the constructionism approach advocates for learning by making. That is, you are not learning because somebody is teaching you, but because you practise it.
If you don’t have a robot to practise with, you cannot learn properly and at full speed.
2. Concepts are complex to understand
The concepts to learn to make a robot work are complex (how to make a robot grasp, how to build a navigation program, how to make it reason…), what makes the learning curve very stepped.
How many of you out there use ROS?
Ok, how would you explain what is ROS to others that know nothing about it? Difficult, isn’t?
The most meaningful unit in robotics for a newcomer is making the robot do something. Given the student’s program to make the robot do something, the robot either does it or not. This is very clear to understand. You push this button and the robot does that (or not, if you have a failure in your program). The feedback of your performance is very clear and straight.
It is not the same to get the feedback about the robot doing an action than to do it because the compilation said you forgot to include something in the CMakeLists.
The longer it takes to associate the errors with the result in the robot, the more difficult it is for the student to close the loop of understanding, and hence, learn why or why not this works that way. This is the concept of delayed reward. The more delayed reward, the more difficult to understand the relation cause-effect and the longer it will take to learn the concept associated.
The challenge is then, to provide such feedback very quickly, even if there are many steps and concepts involved in a robot action. Hence, a teaching material cannot start by indicating how to install everything, how to compile, or how to create a download new packages.
Nobody needs to know the definition of ROS to start using it. It is not a good idea to start teaching ROS by starting defining what a package is, then what a workspace is, then how catkin works, then what is the ROS_MASTER_URI, then ….
I mean all of those are really important, but not at this stage. Going at that peace, it will take for ages to make the robot do something, and the student to understand it.
3. Documentation made by teachers for teachers
We found that documentation, tutorials and classes are made by people who already know for people who already know. I call this type of tutorials the reference tutorials. Those tutorials basically start by describing concepts one by one, following a hierarchy very well defined by people who already know the subject. If you ask the reason of that concept or for what do you need it, they will tell you that you will understand later (but later is too late!!!!).
Generally speaking, robotics is taught as if it was taught for the teacher. Very well structured, this concept first because it goes first in the concepts hierarchy. Then the next concept because it is that what goes next in the concepts hierarchy. It is a problem that most documentations of robot programming frameworks work as a reference tutorial for the engineers that wrote it. Of course you can learn from those but it is a lot harder and slower (I learnt ROS that way).
I do believe that we do not work this way. One thing is the way we structure information once we know the whole space, and another is the way we structure it when we are learning. Very different things.
Solution we propose
Themethod we propose here is to make the students forget as many concepts as possible and concentrate on doing things with the robots. Complex concepts will be taught later, when the student has the base. This approach is based on how we learn and not in how we order our thoughts once we have them inside.
We propose a solution to speed the learning that provides the following steps:
1. In media res, that is the trick Star Wars used to make the movie more appealing (new Star Trek movies have copied successfully, I think).
This means, start in the middle, not in the edges. The idea is to put the student in the middle of the knowledge to learn and make it move to the sides, discarding a lot of things in the way to the side.
2- Learning by doing, not by reading or by typing, by doing… on a robot, I mean.
This is the successful case of Code Academy. There you can learn to code by doing and quickly observing the result of what you have done.
We must have the same type of environment for robotics learning. Do and then observe on the robot, what is the result of your program.
3- A step by step guide, showing only what is required for the next action of the robot, and removing out of the equation what is not required/useful at that specific learning moment (even if important for the success of the step).
For example, you must have a catkin_ws in order to launch a ROS package, but we do not believe that we must explain to a new student how to create a catkin_ws. Instead we provide it done ready to be used, and once he becomes used to it, then we show how to create your own.
Implementation
At The Construct, we have integrated all those ideas into a platform that allows students to quickly learn (in 5 days) any concept of robotics programming. I mean, for real robots!
Our platform is called Robot Ignite Academy, and we like to call it the Code Academy for Robotics. We integrated step by step explanations with a robot simulation and program development environment, everything inside a web browser (hence you can learn using any type of device).
At Ignite Academy, students will learn complex robotics concepts by:
Using simulations of the real robots. Many. Examples include Turtlebot-2, Husky, Sphero, Wam, quadcopters… even BB-8! Students use a simulation of the robots as if they had the robot.
Use a step by step lesson process that engages the student on doing something with the robot after every step, in order to check that he is understanding the explanation.
Use the description not only to explain but also to do. Include interactions in the explanation. For example generate graphics dynamically on the explanation based on what the robot is doing at that moment.
Forget about certain basic concepts (for example, what is ROS… who cares? You don’t need to know what is ROS in order to use it and make a robot work) We believe that once you have mastered the real thing, getting those concepts will be easy peace. But if you put them on the pile of things you have to learn together with the others, it makes the whole thing difficult to swallow.
All that without having to install anything in their computers, and with simulations at the core of the learning system, not only as something additional.
In our system the simulation is at the center. We provide different simulated robots that the student has to practice with. They use simulation as if they had the real robot, with the difference that they don’t have to care about building. It is just provided working and ready to be used.
Conclusion
We are providing the Ignite Academy platform online, for anyone. Students can use it for their personal learning. Teachers can use it for teaching during the class. Companies can use it to upgrade their workers.
What is important to state is that you don’t have to care about anything. Just connect using a web browser from any type of computer and start learning.
One of the most wanted robot simulations is a robot that can be used for anything. Robonaut is one of this kind. NASA kindly gave this simulation for public use and we though in The Construct that we could use it to make an even better user friendly version. We created a test to demonstrate the possibilities that TheConstruct has to offer in the Space Zero gravity simulations. We also envisioned this as a proof of concept of how TheConstruct could be used as platform for competitions as Robocup , special for Space Exploration themes.
To launch this simulation you simply have to select the Robonaut2 simulation and select the following launch: $r2_robocup_at_space_1.launch
And there you have it! All the technical hastle was solved by TheConstrut and you just have a ready to use Robonaut. This launch will spawn the torso version of Robonaut, with all its controllers in a zero gravity space. It will also spawn a taskboard full of functional buttons and switches without the third panel which will be spawned separately. One its done you should get something like this:
To check that Robonaut is functioning correctly, it should have its arms in an L shape ready to start moving like in the picture above. Once all is running, please check that the taskboard is publishing. The taskboard publishes in a topic the state of the switches and buttons in the panel. In this early version it only tracks the three lower switches and the security switch on the top right hand corner of the first panel. To check all the taskboard systems are working, this is the correct output to the following command in the Webshell:
$rostopic info /taskboard/TaskboardPanelA
Type:
gazebo_taskboard/TaskboardPanelA
Publishers: * /taskboard_state_tracker
Subscribers: * /taskboard_systems_actuator
You should see the publisher of the Topic and a subscriber that controlles when to activate the magnet that maintains the panel to be placed in position.
Once you are ready just launch the grab and place panel 3 launch:
$roslaunch nasa_r2 r2_first_event.launch
This will reset the power switch and the panel to their starting positions.
Once done, Robonaut will start moving to grab the panel.
Once Grabbed, it will place it just in front of the desired spot.
Then it will activate the magnet by firstly opening the security lid and then activating the power switches.
Once activated, the panel will receive a force that will hopefully place it inside the taskboard.
This launch can be executed as many time as you like, so that there is no need of relaunching the simulation every time. There are other features that you might want to explore like the “model_destination_validator.py” which you can use to evaluate correct placing of objects based on their relative position to prior placed markers. Very useful for unsupervised learning and competitions.
The Humanoids Sumo Challenge is a worldwide contest created by The Construct and Cyberbotics. It consists of two simulated Darwin Humanoid robots that must fight against other in a simulated sumo dojo. Participants, must build a controller for their robot which should try to knock out the other robot or take it out of the dojo.
The contest starts on May 2016 and will last for 6 months. Along those 6 months, every week-day all the participants will automatically fight in the dojo, obtaining a result of the contest for that day (showed in the Leaderboard the final status). The winner on each Friday will receive a 20€ Amazon gift card. The winner of the last day of the contest will win a Webots Pro license.
The status of the contest can be checked anytime in the Leaderboard. The fights between participants follow a Bubble Sort algorithm. This means that any new participant that enters in the contest today can reach the top in a single day, provided that her controller is better than the rest (it will have to fight against all the others and win). By using this algorithm we are providing a chance to all new participants along the six months of the contest.
Once you are enrolled, you can participate creating your controllers. However, in order to put it easy for you we provide an already working fighting controller. So the development strategy may be:
Launch the default controller provided by us.
Modify the default at will, improving it.
Save changes.
Relaunch the simulation with your changes and see results.
Repeat from point n.2 until happy with the results.
Store your controller in your simulations area of The Construct.
Set your stored controller as your contest controller.
See your controller fight with the other participants.
If you are not happy with your performance, reload your stored controllers.
Then repeat from point n.2.
Launch the default controller
To launch the default controller, log in into The Construct and start the Webots simulator. The default world that loads when Webots boots up is the default sumo controller (see video below). This is the example fighting controller that we provide to all the participants.
If you watch this simulation, you will see how the two robots start fighting against each other. The robots will keep fighting until one wins the contest. You can follow the status of the competition by looking at the messages displayed on the log console below the main simulation screen.
To restart the controllers again, just press the Reload red icon. The whole simulation will be loaded again and the robots will start fighting again.
Modify the default controller
The default controller makes the robot look for another robot, walk towards it and take it down. It is written in C++ and uses the Webots Darwin C++ API that you can learn here.
In order to modify the default controller, first open the development environment by clicking on the red IDE icon. The following screen will appear showing all the files that describe the sumo fighting simulation.
In the files displayed on the left part of the IDE you can see the sumo_fighter directory. This is the directory that contains the default fighter controller that we provide to you for your use. Use the IDE to modify the controller. Edit, add new files, delete… you can modify this controller at your will. The only thing that you need to keep is the name of the directory (sumo_fighter) and the name of the main.cpp file containing the main function. Remember to press Ctrl+S to save your mods.
On the files of the left you can also see two types of controllers: the sparring_fighter and the sumo_supervisor. You should not modify or touch the sumo_supervisor because it is the one that controls that the sumo simulation works properly. However, you can modify and change the sparring_fighter. The sparring_fighter is the controller of the other robot that is fighting against your robot. You can change that controller in order to train your sumo_fighter for a specific technique or else.
Save changes
For each file that you have modified on the IDE, press Ctrl+S to save those mods.
Relaunch to see changes
Once you have done changes in the controllers and saved them, you can check how your saved mods behave on the simulation by pressing the Reload button in the simulation window (see picture). By pressing that button, the system will reload the changes you made and restart the simulation with those changes, so you can see how the robots behave now.
Keep doing changes, saving them and reloading this way until you are happy with the results. Then you have to store the whole simulation in The Construct.
Store your modified controller for later use
The default example controller we provide is just that, an example. In order to maintain your modifications in The Construct and keep working on it in the following days, you need to store your mods with your own selected name.
To store your simulation, press the Save button (see picture). A pop up will appear asking you for a name for your simulation. Here you have to provide a different name than the one of the default controller, in order to keep your own version of the controller.
Once this controller is saved, you will be able to load it the next time that you come to The Construct, and keep working from that point. You will also be able to define this controller as your controller for the contest.
You need to do this step every time you want that your mods stay for the next time you come to The Construct. Also, until you do not store, the mods won’t be used in the official contest.
Terminate your simulation
Once you are done for today, do not forget to terminate the simulation. VERY IMPORTANT: you need to finish your simulation once you are done for today. Otherwise you will loose simulation time for nothing.
The Construct provides free accounts that allow you to use the simulators for 10 hours per month. In case you need more hours to improve your controller, please consider subscribing for one of our plans.
NOTE: The Construct has a mechanism to detect inactivity that will automatically finish your simulation if it does not detect activity on it for half an hour.
Check your status in the competition
You can check your current position in the world competition by visiting the Leaderboard. Every week-day at 12:00 a.m. CEST the competition for that day will be executed and the results displayed on the Leaderboard. The actual combats will be displayed also on that page, so you can see in real time the different combats if you visit that page at that time.
The Leaderboard shows how fought with who, who won and why she won (see reasons for winning below).
Next time you come to work on the simulation
Next time you come to modify your controller, may be the next day after watching your result in the competition, you will have to start again the Webots simulator as described above. However, once the default sumo simulation is loaded, you will need to load your own controller instead of the default one.
To load your controller, open the scrolling menu and look for the name of your stored controller (see picture). Select your controller and then press the red Play button on the right of the scroll menu. The sumo contest with your controllers will be loaded and executed. Now you can keep working from the point you left last time.
How to send your controller for fighting in the contest
Up to now you have created and modified your own fighting controller. You need to indicate to the contest which one of your controllers will fight in the contest. For that, first go to your List of Simulations.
Once you are in the List of Simulations section, look for the simulation you created for the Humanoids Sumo Contest (in our case it was the PioneerSumoRingLine). As you will see, on the right hand side of the simulation a sumo icon appears. Press that icon to define the controller of your simulation as your controller for the contest.
By confirming this way, your controller will be taken each day automatically and make it fight against the rest of participants in a Bubble Sort style league.
You can modify your controller as many times as you want as described in the section above, but you do not need to set it every time. Only need to set your sumo contest once. You can modify it all the times you want, using the IDE (as described above).The competition will always take the latest version of it (if you stored it in The Construct with the same name).
Additionally, you can leave the contest at anytime. To leave the contest, just go to the List of Simulations page, and press again the sumo icon to de-select your controller. Then you will be out of the contest and the system will not take your controller for competition anymore.
You can re-enter the contest again at anytime by pressing the sumo icon once more.
How to win the contest
In order to win a fight there are three options:
You kick the other robot out of the dojo.
You knock out the other robot. Knocking out means that the robot is not upright for more than 10 seconds. It may happen that the reason why a robot is not upright is because the robot felt down by itself. It doesn’t matter. Any robot that maintains for more than 10 seconds a non standing position, it will be considered a knock out.
If after 1 minute 30 seconds no robot has kicked the other out, or no robot has been knocked out, then the winner will be the robot that displaced more.
Competition rules
The following are the main rules for the contest:
It is not allowed to cheat. You cannot do anything . Any participant caught cheating will be disqualified.
You cannot exploit errors of the system. Any participant caught exploiting errors will be disqualified.
You cannot modify the sumo environment. You can only modify the robots controllers.
The organizers provide an already fighting controller made in C++. You can change it a will or completely replace it by your own. Just remember to maintain the name of the executable and directory that contains it.
Controllers provided do not use ROS, but you can at your will.
We will create videos of the winners in order to show how they win and check that they are not cheating. The organizers may require some of the participants to provide their controllers in order to check that they are not cheating.
Conclusions
Experiment with our Humanoids Sumo Contest, use it for your robotics courses, propose us modifications… enjoy it!
The Robot Race to Hawaii Contest is a robotics contest were participants have to program a humanoid Nao robot to make it run the 10 meters race in the shortest amount of time possible. The whole competition is run in a Webotssimulation which is run inside The Construct, the platform that hosts the competition. Participants do not have to install anything in their computers and only need a computer equipped with web browser to participate. Furthermore, participation is free (like in free beer ;-).
The level of complexity of the competition is not very high, what makes this competition a good entry point for newcomers to robotics. The competition already provides the simulated environment with the robot in it, and it even includes a sample controller that can be used as a base for the participants. Participants only have to build the walking controller, and can use as a base the provided one (not mandatory). This makes the competition a good hands on exercise on robotics (teachers, you can use this contest as an assignment for your students ;-).
Along the competition, participants will have to learn several disciplines related to robotics and STEM:
Learn to code in Python. The controllers of the robots must be written in Python language. Python is one of the best languages for introducing someone to programming and specially to robotics. A good place to start learning about python is CodeAcademy.
Learn how a robotics simulator works. The simulator used for the contest is Webots. Participants, need to learn how this simulator works in order to understand how all the parts interact together (robot simulation, controller, computer, etc). Fortunately, there are nice tutorials to learn about it here.
Learn how to build models of objects for robot simulators. Along the competition, participants will be asked to create simulated models of simple objects, like a chair, a glass or a blackboard, once per week. This will provide them with deeper knowledge of what a simulation means. The same Webots tutorial above, contains information about how to create a model.
Learn about biped walking. Even if the contest provides a sample controller that already makes the robot walk, participants may be interested in applying more complex techniques for robot walking. Learning about how to make a biped walk can help create faster controllers. You can learn about robot walking here or read many of the scientific papers available in the web.
This challenge leaves out of the equation learning other important robotics subjects like writing and compiling C++ controllers, using ROS as a robotics framework, or the programming of more complex robot skills like opening doors or grasping a objects. That is why we consider this challenge specially suited for people with low knowledge in robotics like STEM high school students and first year of robotics students.
Furthermore, the whole challenge is hosted in The Construct, hence participants do not have to engage in installing and make work the simulator, since everything is stored in the cloud, and can be handled by the participant with a web browser and a few clicks.
You must be aware, though, that participation is not restricted to anyone, independently of the level of knowledge. So, you may expect some pros around…
By the way, did I mention that the winner of the contest will get a trip to Hawaii?!?!
Now that I got your attention, you can enrol the contest and know more about it by visiting the page contest.theconstructsim.com
