Amazon Research Award Recipients
Amazon Research Award Recipients: Dr Jianbo Jiao and Prof Ales Leonardis

We are pleased to announce that Dr Jianbo Jiao and Prof Ales Leonardis have been successful in a prestigious Amazon Research Award, which will explore PCo3D: Physically Plausible Controllable 3D Generative Models.

Along with awardees from 51 universities in 15 countries, they will have access to Amazon public datasets, in addition to AWS AI/ML services and tools.

We asked Jianbo and Ales to reflect on the award in a short interview:

What are the main goals of your research?

  • The primary objectives of this proposed research focus on developing 3D generative deep models that are both physically plausible and controllable. Specifically, we aim to achieve the following:
  • Developing 3D Generative Models: Create 3D generative models using simple inputs, such as images captured by smartphones or natural language descriptions (text or speech).
  • Utilise state-of-the-art stable diffusion models and incorporate physical properties learned from physics simulators.
  • Achieving Precise Control and Manipulation: Ensure physically plausible control and manipulation of the 3D models. Consider intuitive interactions with the environment, adhering to basic deterministic physical laws.
  • Integration of control mechanisms during model training or application to pre-trained generative models.
  • Comprehensive Analysis and Evaluation: Conduct thorough analysis and evaluation of the proposed generative models. Potentially showcase proof-of-concept demos.

What will the research award enable to you to do?

This prestigious research award will enable us to achieve the aforementioned goals, assist in setting up the experimental system, and facilitate the recruitment of researchers to work on the problem. Another valuable aspect of this award is the access to extensive Amazon resources, including datasets and AWS computational resources. Given the data-driven, deep learning-based nature of this project, it will greatly benefit from such support.

What are the significant impacts of your research?

The impact of this research could be substantial for both academia and industry. As mentioned earlier, controllability and physical understanding are critical challenges in 3D generative models. However, the exploration of physically plausible 3D generative models remains notably lacking in the literature. Existing generative models struggle to produce reasonable results when given instructions related to physical properties. While recent works have delved into physically-aware 3D generation and human generation, they are limited in scale and scope. This proposed research introduces a potential new direction, encouraging follow-up studies within the research community. Furthermore, due to its strong ties to smart mobile applications, the resulting research holds promising applications in competitive industrial settings (e.g. novel apps and smart wearable devices). For example, the ability to generate a physically plausible 3D scene with controllable instructions and interact with the generated objects, as if in the real physical world, holds significant potential. Ultimately, this research has the potential to further dissolve the barrier between the real and virtual worlds.

What are you particularly excited about?

Being able to get recognised and be supported by this award to do such an exciting research is great! Generating a realistic 3D world that is indistinguishable from the real world has always been a longstanding goal for my research, I’m very excited to get the opportunity to move forward towards this goal with the support from the Amazon Research Award!

Dr Jianbo Jiao

This grant provides an exciting opportunity for research that pushes the boundaries of realistic 3D modelling, bridging the gap between virtual and physical worlds. With access to Amazon’s computational resources, we can make faster progress and potentially create exciting new applications for mobile and wearable devices. Additionally, being awarded the Amazon Research Award for the second time is a nice recognition of the work I’ve done with my PhD students and research fellows.

Prof Ales Leonardis