People easily recognize the dramatic moments that unfold in human events. Dramatic turns of events are key to recognizing and communicating effective reports or stories about events. Autonomous systems will work more effectively with humans in obtaining and conveying such narrative when they too can recognize what is dramatic (or tragic, or comical) about human events. The challenge is to effectively convey such concepts to a computer in such a way that humans and autonomous systems can effectively work together in this.

This research studies how to direct a team of robots to obtain video footage to produce clips that trace a dramatic story arc. It is an examination of how such systems might achieve goals that people consider to be abstract or high-level. Within this project, the programs that command teams of robots must predict likely events, direct the robots to be in position for obtaining the desired footage, and re-plan based on observed events. This challenge encompasses a rich and previously unstudied class of problems for robot systems. It will constitute a unique demonstration of robots that are capable of achieving high-level goals as they process data in forms which combine both continuous and discrete views of the world in a new and unusual way. More broadly, the research will advance how computers can fuse and summarize video streams. Both skills are needed for automatically generating synopses and in editing videos.

Obvious places where this is useful include helping secure the nation (for surveillance), taming the deluge of online multimedia content (for summarization), and advancing applications in the creative industries (for editing). The research project will also use the ideas underlying these pieces in a new robotics course with students at three institutions going head-to-head in a series of competition-based class projects. This course (taught, among other places, at a Hispanic-Serving Institution) will contribute to the development of the STEM workforce of the future, helping increase American competitiveness.

The project advances current knowledge by formulating new theory and developing novel algorithms for autonomous and robot systems, with a focus on those systems with minimal or no human operator intervention. The research contributes novel data representations for robots that will inhabit rich environments such as those characterized by uncertain, unanticipated, and dynamically changing circumstances. One of the foundational ideas of the project is a means to specify sophisticated mission objectives via a recursive structure using prior work in compiler theory for computer languages. The project involves a strong connection between this theoretical work and demonstrated systems.

(National Science Foundation Award Number #1849291)

This research tackles the problem of constructing a communication map of a known environment using multiple robots. A communication map encodes information on whether two robots can communicate when they are at two arbitrary locations and plays a fundamental role for a multi-robot system deployment to reliably and effectively achieve a variety of tasks, such as environmental monitoring and exploration. Previous work typically considered only scenarios with a fixed base station and designed offline methods, which did not exploit data collected online by the robots.

This paper proposes Gaussian Process-based online methods to efficiently build a communication map with multiple robots. Such robots form a mesh network, where there is no fixed base station. Specifically, we provide two leader-follower online sensing strategies to coordinate and guide the robots while collecting data. Furthermore, we improve the performance and computational efficiency by exploiting prior communication models that can be built from the physical map of the environment. Extensive experimental results in simulation and with a team of TurtleBot 2 platforms validate the approach.

(Alberto Quattrini Li, Phani Krishna Penumarthi, Jacopo Banfi, Nicola Basilico, Jason M. O'Kane, Ioannis Rekleitis, Srihari Nelakuditi, Francesco Amigoni. Multi-robot Online Sensing Strategies for the Construction of Communication Maps. Autonomous Robots, 2019)

Environmental monitoring and surveying operations on rivers currently are performed primarily with manually-operated boats. In this domain, autonomous coverage of areas is of vital importance, for improving both the quality and the efficiency of coverage. This research leverages human expertise in river exploration and data collection strategies to automate and optimize these processes using autonomous surface vehicles (ASVs). In particular, three deterministic algorithms for both partial and complete coverage of a river segment are proposed, providing varying path length, coverage density, and turning patterns. These strategies resulted in increases in accuracy and efficiency compared to human performance. The proposed methods were extensively tested in simulation using maps of real rivers of different shapes and sizes. In addition, to verify their performance in real world operations, the algorithms were deployed successfully on several parts of the Congaree River in South Carolina, USA, resulting in total of more than 35km of coverage trajectories in the field.

(Nare Karapetyan, Adam Braude, Jason Moulton, Joshua A. Burstein, Scott White, Jason M. O'Kane, Ioannis Rekleitis. Riverine Coverage for Large Scale Surveying Operations. In Proc. IEEE/RSJ International Conference on Intelligent Robots and Systems, 2019)