The Road to Understand Aerial Cinematography

How does AI picks out the best views and shoot an aerial documentary? At Ailabs, we’ve built a system, equipped a drone and a 360 camera, to have an eye of a videographer.

We want to enable filmmaking to achieve camera movement and tracking shot by artificial intelligence itself. We design our system to be able to pick interesting object and features such as lighthouse or coastline from a 360-degree video and create a flat, standard documentary without manual control of camera movement and angle.

 

The drone carries a 360-degree 4K camera and flies alongside Tainan city, Taiwan, to collect videos that are later used for post editing done by our system.

 

This project is inspired by Chi Po-lin‘s documentary film “Beyond Beauty: Taiwan from Above (看見台灣) which captures Taiwan completely in aerial cinematography and broke the Taiwan box office records for the largest opening weekend and the highest total gross of a locally produced documentary. Unfortunately, Chi died in a helicopter crash in a mountainous area in Hualien County’s when the group was shooting footage for the sequel.

Chi’s pointed out it is due to the poor image quality and the monotone of the camera movement that he did not consider to make film by drones. Besides, flying on helicopters for aerial cinematography put photographer in tremendous danger. For this reason, we started the “Chi Po-Lin project” using a drone, a 360 camera and our AI-powered post-editing software algorithm.

Automatic Cinematography for 360

When using a helicopter for videography, the pilot flies the route, and the videographer operates the video camera respectively. In the case of drone,  the videographer is replaced by a 360 camera and an algorithm in post-editing to determines where to focus on in the 360 images.

 

 

As in the scenario above, we installed a 360 camera on the drone to let the AI control the perspective in the 360 image and render a portion of the 360 image to create virtual camera movements such as pan, tilt, and zoom.

 

Algorithmically controlled perspectives from the 360 images 

 

A reason for the monotone of the aerial videos is that it is very difficult for a person to manipulate the controls of the drone and consider the composition details at the same time. Moreover, the conventional methods of automatic cinematography simplify the problem to aligning the camera to the center of the object of interest. So, we give the control of the camera to the AI which, unlike the existing method, takes into account the composition and the semantic flow of the scene.

360 cameras record every point of view. After the videos are collected, the AI recognizes the scenes and objects of interest encountered during the flight, selects the best angles of each moment, and automatically plans multiple sets of suitable trajectories to assist the user in editing the movie.
 
 

Partners:

– 台灣人工智慧實驗室
– 科技部南科管理局
– 台南市政府
– 成大航太
– 經緯航太

Sponsored:

– Microsoft
– Nvidia
– Garmin

 
 

Featured image by YELLOW Mao. 黃毛, Photographer / CC BY

0 replies

Leave a Reply

Want to join the discussion?
Feel free to contribute!

Leave a Reply

Your email address will not be published. Required fields are marked *