Anagrafe della ricerca

Smartphone photography is nowadays the dominant means of shooting pictures in our everyday life. Due to the multiple limitations affecting smartphone cameras, computational approaches to image generation, processing and coding that overcome them are enjoying great success and are an active field of research. Today, we are on the verge of a new paradigm in mobile photography, with the integration of active sensing instruments, namely LiDAR, onboard smartphones. LiDAR allows capturing 3D information of a scene for 3D reconstruction and understanding tasks. However, in this project our aim is to study how LiDAR can assist the traditional RGB camera in improving its operations. While fusion of LiDAR and RGB has been around for a long time, it has always been addressed from the opposite point of view, i.e., using RGB images to improve LiDAR data. This has been historically motivated by the fact that depth sensing occurred on dedicated devices rather than on devices that are used to snap everydays pictures. In particular, the LICAM project will focus on developing novel AI-powered methods that use LiDAR depth maps acquired by smartphones to enhance restoration and coding of images acquired by the same smartphone, thus constructing a “virtual camera” that can benefit from the strengths of both sensors. The first key objective of the project is enhancing the performance of smartphone RGB cameras under low-light conditions, a notoriously challenging scenario, by using the LiDAR depth map as a guidance to denoise and remove motion blur from the low-light photo. Since LiDAR is an active sensing instrument, it does not suffer from low-light conditions and can provide valuable information to regularize the restoration process. The second key objective is the development of an image compression algorithm that can exploit depth information to define regions of interest and optimize their coding in a rate-distortion sense. We will further explore the novel idea of machine-driven compression, where compression of a low-light image is optimized, not for the goal of consumption of the raw image, but with the objective of maximizing the effectiveness of the downstream restoration process. Deep learning will be instrumental in building the sophisticated image models needed for effective information fusion. Indeed, since the problems addressed by this project are largely unexplored by current literature due to the recency of simultaneous RGB and LiDAR acquisition on smartphones, novel deep learning methodologies will need to be developed. The work performed in this project will be publicly released, including any new dataset created, in order to jump start research into this new and currently unexplored field. This will boost its scientific impact and allow for public reproducible research on the topic. It will also stimulate European advances in AI-driven computational photography, as Europe currently lags behind American and Asian corporations.