The PX4 Vision Autonomy DevKit is a robust and inexpensive kit for enabling computer vision development on autonomous vehicles.
The kit contains a near-ready-to-fly carbon-fiber quadcopter equipped with a Pixhawk 4 flight controller, UP Core companion computer, and Occipital Structure Core depth camera sensor. The vehicle comes with software for PX4 Obstacle Avoidance and Collision Prevention enabled “out of the box”. Developers can use the kit to try out other features provided by the PX4 Avoidance project, modify the existing code, or experiment with completely new computer vision-based functionality.
The guide explains the minimal additional setup required to get the vehicle ready to fly (installing an RC system and battery). It also covers the first flight, and how to get started with modifying the computer vision code.
Do not power the UP Core computer using the power socket (this may destroy the computer). The UP Core should only be powered using the battery.
The kit is intended for computer vision projects that use a forward-facing camera (it does not have downward or rear-facing depth cameras). Consequently it can’t be used for testing Safe Landing, or other features that require a downward-facing camera.
Obstacle avoidance in missions can only be tested when GPS is available (missions use GPS co-ordinates). Collision prevention can be tested in position mode provided there is a good position lock from either GPS or optical flow.
The port labeled
USB1may jam the GPS if used with a USB3 peripheral (disable GPS-dependent functionality including missions). This is why the boot image is supplied on a USB2 memory stick.
The DevKit contains following components:
- PX4 Vision drone kit with these internal parts:
- 1x Pixhawk 4 flight controller (with custom PX4 firmware)
- 1x pmw3901 optical flow sensor
- 1x distance sensor
- 1x Structure Core depth camera
- 1x UP Core computer (with Ubuntu and PX4 avoidance)
- Atom CPU
- FTDI UART connected to flight controller
USB1: USB3 A port used for booting PX4 avoidance environment from a USB2 stick (connecting a USB3 peripheral may jam GPS).
USB2: USB2 port on a JST-GH connector. Can be used for second camera, LTE, etc. (or keyboard/mouse during development).
USB3: USB2 JST-GH port connected to depth camera
HDMI: HDMI out
- SD card slot
- WiFi (attached to external antenna #1). Allows computer to access home WiFi network for Internet access/updates.
- ESP8266 connected to flight controller (attached to external antenna #2). Enables wireless connection to the ground station.
- A USB2.0 stick with pre-flashed software that bundles:
- Ubuntu 18.04 LTS
- ROS Melodic
- Occipital Structure Core ROS driver
- PX4 Avoidance
- Assorted cables, propellers, and other accessories. To be used to attach additional peripherals.
The kit contains all the essential drone hardware except a battery and a radio control system, which must be purchased separately:
- 4S LiPo with XT60 female connector
- Less than 115mm long (to fit between power connector and GPS mast)
- Radio control system
- Any PX4-compatible RC System can be used.
- An FrSky Taranis transmitter with R-XSR receiver is one of the more popular setups.
- An H2.0 Hex Key (to unscrew the top plate so that an RC receiver can be connected)
In addition, users will need ground station hardware/software:
- Laptop or tablet running QGroundControl (QGC).
Attach a compatible RC reciever to the vehicle (not supplied with kit):
- Remove/unscrew the top plate (where the battery goes) using an H2.0 hex key tool.
- Connect the receiver to the flight controller.
- Re-attach the top plate.
- Mount the RC receiver on the UP Core carrier board plate at the back of the vehicle (use zipties or double-sided tape).
- Ensure the antennas are clear of any obstructions and electrically isolated from the frame (e.g. secure them under the carrier board or to the vehicle arms or legs).
Bind the RC ground and air units (if not already done). The binding procedure depends on the specific radio system used (read the receiver manual).
Raise the GPS mast to the vertical position and screw the cover onto the holder on the base plate.
Power the vehicle with a fully charged battery.
Ensure propellers are removed before connecting the battery.
- Connect the Pixhawk 4 and ground station using a USB cable.
- Start QGroundControl on the ground station.
Configure/calibrate the vehicle:
The vehicle should arrive pre-calibrated (e.g. with firmware, airframe, battery, and sensors all setup). You will however need to calibrate the radio system (that you just connected) and it is worth doing the following basic checks.
(Optional) Configure a Flight Mode selector switch on the remote controller.
Modes can also be changed using QGroundControl
We recommend RC controller switches are define for:
Attach the propellers with the rotations as shown:
The propellers directions can be determined from the labels: 6045 (clockwise) and 6045R (reversed).
Screw down firmly using the provided propellor nuts:
When the vehicle setup is complete:
Insert the pre-imaged USB2 stick from the kit into the UP Core port
Connect the battery to power the vehicle.
Wait until the boot sequence completes and the avoidance system has started (the vehicle will reject arming commands during boot).
The boot/startup process takes about 2.5 minutes from USB stick (or 30 seconds from internal memory).
Check that the avoidance system has started properly:
The QGroundControl notification log displays the message: Avoidance system connected.
- A red laser is visible on the front of the Structure Core camera.
- Wait for the GPS LED to turn green. This means that the vehicle has a GPS fix and is ready to fly!
Connect the ground station to the vehicle WiFi network using the following default credentials:
- SSID: pixhawk4
- Password: pixhawk4
WiFi network SSID, password, and other credentials may be changed after connecting (if desired), by using a web browser to open the URL:
http://192.168.4.1. The baud rate must not be changed from 921600.
Find a safe outdoor location for flying, ideally with a tree or some other convenient obstacle for testing PX4 Vision.
To test collision prevention, enable Position Mode and fly manually towards an obstacle. The vehicle should slow down and then stop within 6m of the obstacle (the distance can be changed using the CP_DIST parameter).
To test obstacle avoidance, create a mission where the path is blocked by an obstacle. Then switch to Mission Mode to run the mission, and observe the vehicle moving around the obstacle and then returning to the planned course.
The following sections explain how to use the kit as an environment for developing computer vision software.
The PX4 Avoidance system consists of computer vision software running on a companion computer (with attached depth camera) that provides obstacle and/or route information to the PX4 flight stack running on a flight controller.
Documentation about the companion computer vision/planning software can be found on github here: PX4/avoidance. The project provides a number of different planner implementations (packaged as ROS nodes):
- The PX4 Vision Kit runs the localplanner by default and this is the recommended starting point for your own software.
- The globalplanner has not been tested with this kit.
- The landing planner requires a downward facing camera, and cannot used without first modifying the camera mounting.
PX4 and the companion computer exchange data over MAVLink using these interfaces:
- Path Planning Interface - API for implementing avoidance features in automatic modes.
- Collision Prevention Interface - API for vehicle based avoidance in manual position mode based on an obstacle map (currently used for collision prevention).
You can install the image on the UP Core and boot from internal memory (instead of the USB).
This is recommended because booting from internal memory is much faster (~30 seconds vs 2.5 minutes), it frees up a USB port, and may well provide more memory than your USB stick.
To flash the USB image to the UP Core:
- Insert the pre-flashed USB drive into the UP Core port labeled
- Login to the companion computer (as described above).
Open a terminal and run the following command to copy the image onto internal memory (eMMC). The terminal will prompt for a number of responses during the flashing process.
cd ~/catkin_ws/src/px4vision_ros sudo ./flash_emmc.sh
All information saved in the UP Core computer will be removed when executing this script
Pull out the USB stick.
- Restart the vehicle. The UP Core computer will now boot from internal memory (eMMC).
First insert the provided USB2 stick into the UP Core port labeled
USB1, and then power the vehicle using a 4S battery.
The avoidance system should start within about 2.5 minutes.
Fly the Drone (with avoidance) additionally explains how to verify that the avoidance system is active.
If you've already installed the image on the companion computer you can just power the vehicle (i.e. no USB stick is needed). The avoidance system should be up and running within around 30 seconds.
Once started the companion computer can be used both as a computer vision development environment and for running the software.
To login to the companion computer:
Connect a keyboard and mouse to the UP Core via port
Use the USB-JST cable from the kit to get a USB A connector
- A USB hub can be attached to the cable if the keyboard and mouse have separate connectors.
Connect a monitor to the UP Core HDMI port.
The Ubuntu login screen should then appear on the monitor.
- Login to the UP Core using the credentials:
- Username: px4vision
- Password: px4vision
The PX4 Vision’s UP Core computer provides a complete and fully configured environment for extending PX4 Avoidance software (and more generally, for developing new computer vision algorithms using ROS2). You should develop and test your software on the vehicle, sync it to your own git repository, and share any fixes and improvements with the wider PX4 community on the github PX4/Avoidance repo.
The catkin workspace is at
~/catkin_ws, and is preconfigured for running the PX4 avoidance local planner.
The launch-from-boot file (
avoidance.launch) is in the
px4vision_ros package (modify this file to change what planner is launched).
The avoidance package is started on boot. To integrate a different planner, this needs to be disabled.
Disable the avoidance process using the following command:
systemctl stop avoidance.service
You can simply reboot the machine to restart the service.
Other useful commands are:
# restart service systemctl start avoidance.service # disable service (stop service and do not restart after boot) systemctl disable avoidance.service # enable service (start service and enable restart after boot) systemctl enable avoidance.service
The source code of the obstacle avoidance package can be found in https://github.com/PX4/avoidance which is located in
Make changes to the code! To get the latest code of avoidance pull the code from the avoidance repo:
git pull origin git checkout origin/master
- Build the package
catkin build local_planner
The ROS workspace is placed in
For reference on developing in ROS and using the catkin workspace, see the ROS catkin tutorials.
The kit is designed for creating computer vision software that runs on the companion computer, and which integrates with PX4’s flexible path planning and collision prevention interfaces.
Modification of PX4 code is not needed to meet most computer vision use cases. To discuss the interfaces or how to integrate other features join the PX4 slack channel: #computer-vision.
Firmware modification is not recommended in the Early Adopter Edition (we suggest that you use the shipped version).
- UP Core Wiki - companion computer technical information
- Occipital Developer Forum - Structure Core camera information
- Pixhawk 4 Overview
- PX4 Avoidance software/documentation
- Path Planning Interface
For hardware issues, please contact Holybro at: firstname.lastname@example.org.
For software issues, use the following community support channels:
- PX4 discuss: Computer Vision category.
- PX4 slack channel: #avoidance