Language-Guided Drone Simulator for Emergency Search and Rescue

Our project integrated multiple different AI tools to allow humans to easily interact with drones. You can use a text to speech feature to tell the drone what you want it to do, another algorithm will transfer these human instructions into actions for the drone to execute. You can also activate an object detection model to help you find the humans in this search and rescue setting. Can you find the humans?

In this project we have looked at high stakes emergency scenarios such as bushfires, floods and in this scenario floods. Drones can be massively helpful in these scenarios in helping to assess the situation and find missing persons. However many drones are difficult and unintuitive to use. We are filling this gap by creating a drone that can be guided by natural language and uses other AI features such as object detection and position memory. The main feature of the drone is it's ability to take the human's natural language and convert it into machine instructions. This enables a human to maneuver the drone without training on advanced equipment. Instead of needing precise inputs on a controller a human can simply say "go there". Other supporting features include gaze detection and object detection. When the user says "go there" this can allow the LLM to determine whether "there" is an object or simply a direction.