Shunsuke Saito / 齋藤隼介 - Reality Labs Research

Name	Shunsuke Saito (齋藤隼介)
Interest	Effective Data Representations for Deep Human Digitization
Email	{firstname}.{lastname}16 (at) gmail.com

I am a Research Scientist at Reality Labs Research in Pittsburgh, where I work on next generation digital humans (relighting, faces, hands, full-body, hair, clothing, and accessories). For example, I am one of the core contributers of Codec Avatar 2.0, which was featured in Meta Connect 2022 and Lex Fridman's metaverse interview with MZ. Before that, I completed my Ph.D. at University of Southern California. If you are interested in an internship with us, feel free to email/DM me!

See video for high-quality details and all-frequency reflections under any illuminations!	Relightable Gaussian Codec Avatars CVPR 2024 (Oral) [Project (video, demo)] Shunsuke Saito, Gabriel Schwartz, Tomas Simon, Junxuan Li, Giljoo Nam
Spatially varying linear model for universal relightable hand prior!	URHand: Universal Relightable Hands CVPR 2024 (Oral) [Project (video)] Zhaoxi Chen, Gyeongsik Moon, Kaiwen Guo, Chen Cao, Stanislav Pidhorskyi, Tomas Simon, Rohan Joshi, Yuan Dong, Yichen Xu, Bernardo Pires, He Wen, Lucas Evans, Bo Peng, Julia Buffalini, Autumn Trimble, Kevyn McPhail, Melissa Schoeller, Shoou-I Yu, Javier Romero, Michael Zollhöfer, Yaser Sheikh, Ziwei Liu, Shunsuke Saito
Pose-guided SDS for decomposing a single mesh into animatable layered assets!	GALA: Generating Animatable Layered Assets from a Single Scan CVPR 2024 [Project (video, code)] Taeksoo Kim, Byungjun Kim, Shunsuke Saito, Hanbyul Joo
Unified two-hand interaction prior with a novel cascaded diffusion!	InterHandGen: Two-Hand Interaction Generation via Cascaded Reverse Diffusion CVPR 2024 [Project (video, code)] Jihyun Lee, Shunsuke Saito, Giljoo Nam, Minhyuk Sung, Tae-Kyun Kim
Drivable and articulated 4D Gaussian Splatting of Avatars!	Drivable 3D Gaussian Avatars Preprint [Project (video)] Wojciech Zielonka, Timur Bagautdinov, Shunsuke Saito, Michael Zollhöfer, Justus Thies, Javier Romero
Diffusion prior for wrinkle-accurate cloth registration!	Diffusion Shape Prior for Wrinkle-Accurate Cloth Registration 3DV 2024 [Project (code coming soon)] Jingfan Guo, Fabian Prada, Donglai Xiang, Javier Romero, Chenglei Wu, Hyun Soo Park, Takaaki Shiratori, Shunsuke Saito
RelightableHands can be useful for boosting monocular two-hand estimation!	A Dataset of Relighted 3D Interacting Hands NeruIPS 2023 (Datasets and Benchmark Track) [Project (code, data)] Gyeongsik Moon, Shunsuke Saito, Weipeng Xu, Rohan Joshi, Julia Buffalini, Harley Bellan, Nicholas Rosen, Jesse Richardson, Mallorie Mize, Philippe de Bree, Tomas Simon, Bo Peng, Shubham Garg, Kevyn McPhail, Takaaki Shiratori
Diffusion prior for high-quality illumination estimation via inverse rendering!	Diffusion Posterior Illumination for Ambiguity-aware Inverse Rendering ACM Transaction on Graphics (SIGGRAPH Asia 2023) [Project (video, code)] Linjie Lyu, Ayush Tewari, Marc Habermann, Shunsuke Saito, Michael Zollhoefer, Thomas Leimkuehler, and Christian Theobalt
No 3D superivision for 360˚ texturing of clothed humans!	Single-Image 3D Human Digitization with Shape-guided Diffusion SIGGRAPH Asia 2023 [Project] Badour AlBahar, Shunsuke Saito, Hung-Yu Tseng, Changil Kim, Johannes Kopf, and Jia-Bin Huang
Scalable 3D compositional modeling of humans and many objects!	NCHO : Unsupervised Learning for Neural 3D Composition of Humans and Objects ICCV 2023 [Project (video, code, data)] Taeksoo Kim, Shunsuke Saito, Hanbyul Joo
See through inside hair via CT to build accurate and complete hair strands!	CT2Hair: High-Fidelity 3D Hair Modeling using Computed Tomography ACM Transaction on Graphics (SIGGRAPH 2023) [Project (video, code, data)] Yuefan Shen, Shunsuke Saito, Ziyan Wang, Olivier Maury, Chenglei Wu, Jessica Hodgins, Youyi Zheng, Giljoo Nam
Real-time rendering of relightable hands with global illumination!	RelightableHands: Efficient Neural Relighting of Articulated Hand Models CVPR 2023 [Project (video)] Shun Iwase, Shunsuke Saito, Tomas Simon, Stephen Lombardi, Timur Bagautdinov, Rohan Joshi, Fabian Prada, Takaaki Shiratori, Yaser Sheikh, Jason Saragih
Learning compositional generative models of glasses and human heads (bonus: relightable)!	MEGANE: Morphable Eyeglass and Avatar Network CVPR 2023 [Project (video)] Junxuan Li, Shunsuke Saito, Tomas Simon, Stephen Lombardi, Hongdong Li, Jason Saragih
Physics-based simulation meets Neural Rendering!	Dressing Avatars: Deep Photorealistic Appearance for Physically Simulated Clothing ACM Transaction on Graphics (SIGGRAPH Asia 2022) [Project (video, supmat)] Donglai Xiang, Timur Bagautdinov, Tuur Stuyck, Fabian Prada, Javier Romero, Weipeng Xu, Shunsuke Saito, Jingfan Guo, Breannan Smith, Takaaki Shiratori, Yaser Sheikh, Jessica Hodgins, Chenglei Wu
High-res and generalizable volumetric avatar for face and body! (no SMPL/3DMM needed)	KeypointNeRF: Generalizing Image-based Volumetric Avatars using Relative Spatial Encoding of Keypoints ECCV 2022 [Project (video, code)] Marko Mihajlovic, Aayush Bansal, Michael Zollhöfer, Siyu Tang, Shunsuke Saito
Learning history-dependent dynamics from raw scans by efficient autoregression with neural fields!	AutoAvatar: Autoregressive Neural Fields for Dynamic Avatar Modeling ECCV 2022 [Project (video, code)] Ziqian Bai, Timur Bagautdinov, Javier Romero, Michael Zollhöfer, Ping Tan, Shunsuke Saito
Fast-rendering of photorealistic hair via joint modeling of strands and appearance!	Neural Strands: Learning Hair Geometry and Appearance from Multi-View Images ECCV 2022 [Project (video)] Radu Alexandru Rosu, Shunsuke Saito, Ziyan Wang, Chenglei Wu, Sven Behnke, Giljoo Nam
Hyper-realistic animatable avatars just from a quick phone scan!	Authentic Volumetric Avatars From a Phone Scan ACM Transaction on Graphics (SIGGRAPH 2022) Chen Cao, Tomas Simon, Jin Kyu Kim, Gabe Schwartz, Michael Zollhoefer, Shunsuke Saito, Stephen Lombardi, Shih-en Wei, Danielle Belko, Shoou-i Yu, Yaser Sheikh, Jason Saragih
Real-time drivable full-body volumetric avatars!	Drivable Volumetric Avatars Using Texel-aligned Features SIGGRAPH 2022 (Conference Track) [Code] Edoardo Remelli, Timur Bagautdinov, Shunsuke Saito, Chenglei Wu, Tomas Simon, Shih-En Wei, Kaiwen Guo, Zhe Cao, Fabian Prada, Jason Saragih, Yaser Sheikh
Implicit body model as a drop-in replacement of SMPL for fast and robust collision handling!	COAP: Compositional Articulated Occupancy of People CVPR 2022 [Project (code)] Marko Mihajlovic, Shunsuke Saito, Aayush Bansal, Michael Zollhoefer, Siyu Tang
Wanna know more about neural fields papers? Check out our search engine and survey!	Neural Fields in Visual Computing and Beyond Eurographics STAR 2022 [Search Engine] Yiheng Xie, Towaki Takikawa, Shunsuke Saito, Or Litany, Shiqin Yan, Numair Khan, Federico Tombari, James Tompkin, Vincent Sitzmann, Srinath Sridhar
Achieving SoTA performance on animatable avatar modeling by revisiting its formulation!	ARCH++: Animation-Ready Clothed Human Reconstruction Revisited ICCV 2021 [Project (results)] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, Tony Tung (* - equal contribution)
Enabling relighting of 3D animatable avatars with any illumination in real-time!	Deep Relightable Appearance Models for Animatable Faces SIGGRAPH 2021 [Project (video, paper)] Sai Bi, Stephen Lombardi, Shunsuke Saito*, Tomas Simon, Shih-En Wei, Kevyn McPhail, Ravi Ramamoorthi, Yaser Sheikh, Jason Saragih ( - equal contribution)
Clothed avatar with pose-driven deformation from raw scans without template or surface registration!	SCANimate: Weakly Supervised Learning of Skinned Clothed Avatar Networks CVPR 2021 (Oral Presentation) [Project (video, code)] CVPR Best Paper Nominee!! Shunsuke Saito, Jinlong Yang, Qianli Ma, Michael J. Black
SCALE is a fast and expressive clothed human representation using a collection of patches!	SCALE: Learning to Model Clothed 3D Humans with a Surface Codec of Articulated Local Elements CVPR 2021 [Project (video, code)] Qianli Ma, Shunsuke Saito, Jinlong Yang, Siyu Tang, Michael J. Black (* - equal contribution)
Pixel-aligned NeRF for human head modeling!	Pixel-Aligned Volumetric Avatars CVPR 2021 [Project (video)] Amit Raj, Michael Zollhöfer, Tomas Simon, Jason Saragih, Shunsuke Saito, James Hays, Stephen Lombardi
Real-time extension of PIFu by fast surface localization!	Monocular Real-Time Volumetric Performance Capture ECCV 2020, SIGGRAPH RTL (Best in Show Award) [Project (video, code)] Ruilong Li, Yuliang Xiu, Shunsuke Saito, Zeng Huang, Kyle Olszewski, Hao Li (* - equal contribution)
High-fidelity clothed human reconstruction at 1k resolution!	PIFuHD: Multi-Level Pixel-Aligned Implicit Function for High-Resolution 3D Human Digitization CVPR 2020 (Oral Presentation) [Project (video, code)] Shunsuke Saito, Tomas Simon, Jason Saragih, Hanbyul Joo
Learning neural implict surface without 3D supervision!	Learning to Infer Implicit Surfaces without 3D Supervision NeurIPS 2019 Shichen Liu, Shunsuke Saito, Weikai Chen, Hao Li
Pixel-aligned neural implict function for high-quality 3D reconstruction (with texture)!	PIFu: Pixel-Aligned Implicit Function for High-Resolution Clothed Human Digitization ICCV 2019 [Project] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, Hao Li ( - equal contribution)
Single-view 3D clothed human reconstruction with texture by synthesizing silhouettes and back-side texture inference!	SiCloPe: Silhouette-Based Clothed People CVPR 2019 (Oral Presentation - Best Paper Finalist) [Supplemental] Ryota Natsume, Shunsuke Saito, Zeng Huang, Weikai Chen, Chongyang Ma, Hao Li, Shigeo Morishima (* - equal contribution)
3D morphable hair model leveraging volumetric VAE and single-view hair reconstruction! (even from stylish dogs!)	3D Hair Synthesis Using Volumetric Variational Autoencoders ACM Transaction on Graphics (SIGGRAPH Asia 2018) Shunsuke Saito, Liwen Hu, Chongyang Ma, Hikaru Ibayashi, Linjie Luo, Hao Li
Expressive avatar modeling by novel expression synthesis with GANs!	paGAN: Real-time Avatars Using Dynamic Textures ACM Transaction on Graphics (SIGGRAPH Asia 2018) Koki Nagano, Jaewoo Seo, Jun Xing, Lingyu Wei, Zimo Li, Shunsuke Saito, Aviral Agarwal, Jens Fursund, Hao Li
Single-view reconstruction of high-quality skin reflectance and geometry!	High-Fidelity Facial Reflectance and Geometry Inference From an Unconstrained Image ACM Transaction on Graphics (SIGGRAPH 2018) Shugo Yamaguchi, Shunsuke Saito, Koki Nagano, Yajie Zhao, Weikai Chen, Kyle Olszewski, Shigeo Morishima, Hao Li (* - equal contribution)
Modeling meso-scale facial geometry using image-to-image translation in UV space!	Mesoscopic Facial Geometry inference Using Deep Neural Networks CVPR 2018 (Spotlight Presentation) Loc Huynh, Weikai Chen, Shunsuke Saito, Jun Xing, Koki Nagano, Andrew Jones, Hao Li, Paul Debevec
Complete head avatar reconstruction from a single photograph!	Avatar Digitization from a Single Image for Real-Time Rendering ACM Transaction on Graphics (SIGGRAPH Asia 2017) Liwen Hu, Shunsuke Saito, Lingyu Wei, Koki Nagano, Jaewoo Seo, Iman Sadeghi, Jens Fursund, Yen-Chun Chen, Hao Li ( - equal contribution)
Fabrication-ready garment transfer from human to animals!	Quasi-Developable Garment Transfer for Animals SIGGRAPH ASIA 2017 Technical Brief Fumiya Narita, Shunsuke Saito, Tsukasa Fukusato, Shigeo Morishima
Fast and robust 6 DOF estimation of headsets!	Outside-in Monocular IR Camera based HMD Pose Estimation via Geometric Optimization VRST 2017 Pavel A. Savkin, Shunsuke Saito, Jarich Vansteenberge, Tsukasa Fukusato, Lochlainn Wilson, Shigeo Morishima
Dense correspondences on faces in the wild!	Learning Dense Facial Correspondences in Unconstrained Images ICCV 2017 Ronald Yu, Shunsuke Saito, Haoxiang Li, Duygu Ceylan, Hao Li
Novel expression synthesis of faces in the wild using GANs!	Realistic Dynamic Facial Textures From a Single Image Using GANs ICCV 2017 Kyle Olszewski, Zimo Li, Chao Yang, Yi Zhou, Ronald Yu, Zeng Huang, Sitao Xiang, Shunsuke Saito, Pushmeet Kohli, Hao Li ( - equal contribution)
High-quality facial performance capture with deep learning!	Production-Level Facial Performance Capture Using Deep Convolutional Neural Networks SCA 2017 Samuli Laine, Tero Karras, Timo Aila, Antti Herva, Shunsuke Saito, Ronald Yu, Hao Li, Jaakko Lehtinen
Representing skin details as high-dimentional "style" using style transfer for photoreal avatar from a single image!	Photorealistic Facial Texture Inference Using Deep Neural Networks CVPR 2017 (Spotlight Presentation) Shunsuke Saito, Lingyu Wei, Liwen Hu, Koki Nagano, Hao Li (* - equal contribution) featured in fx guide, Gizmodo, Daily Mail, Seamless, 3ders, and Wired
Simplifying production-level facial performance capture leveraging a template model!	Multi-View Stereo on Consistent Face Topology Eurographics 2017 Graham Fyffe, Koki Nagano, Loc Huynh, Shunsuke Saito, Jay Bush, Andrew Jones, Hao Li, Paul Debevec
Real-time facial performance capture from HMD cameras using neural nets!	High-Fidelity Facial and Speech Animation for VR HMDs ACM Transaction on Graphics (SIGGRAPH Asia 2016) Kyle Olszewski, Joseph J. Lim, Shunsuke Saito, Hao Li
Occlusion-aware facial performance capture from a live-streaming video! (also real-time face swapping)	Real-Time Facial Segmentation and Performance Capture from RGB Input ECCV 2016 [Data] Shunsuke Saito, Tianye Li, Hao Li
Garment transfer from human to animals!	Garment Transfer for Quadruped Characters Eurographics 2016, short paper Fumiya Narita, Shunsuke Saito, Takuya Kato, Tsukasa Fukusato, Shigeo Morishima
Wanna bulk up in metaverse? Interactive body shape modeling inspired by bio-physics!	Computational Bodybuilding: Anatomically-based Modeling of Human Bodies ACM Transaction on Graphics (SIGGRAPH 2015) Shunsuke Saito, Zi-Ye Zhou, Ladislav Kavan
Efficient and robust frame interpolation by extending PatchMatch!	PatchMove: Patch-based Fast Image Interpolation with Greedy Bidirectional Correspondence Pacific Graphics 2014, short paper Shunsuke Saito, Ryuuki Sakamoto, Shigeo Morishima
Extending homogenization FEM analysis to cloth simulation!	Macroscopic and Microscopic Deformation Coupling in Up-sampled Cloth Simulation Computer Animation and Virtual Worlds Journal (CASA 2014) Shunsuke Saito, Nobuyuki Umetani, Shigeo Morishima
Better representing character-specific expression for facial retargeting!	Character Transfer: Example-based individuality retargeting for facial animations International Conference in Central Europe on Computer Graphics, Visualization and Computer Vision 2014 (WSCG 2014) Takuya Kato, Shunsuke Saito, Masahide Kawai, Tomoyori Iwao, Akinobu Maejima, Shigeo Morishima
Estimating material parameters of cloth from 4D scans!	Estimating Cloth Simulation Parameter from 3D Cloth Motion Separating Dynamic from Static Motion ACM SIGGRAPH EuroGraphics Symposium on Computer Animation (SCA 2012), poster Shunsuke Saito, Ryusuke Sagawa, Naoya Iwamoto, Shigeo Morishima

Shunsuke Saito

Web: www-scf.usc.edu/~saitos/
E-mail: shunsuke.saito16 (at) gmail.com

Technical Paper Reviewer:

ECCV, ICCV, IJCV, SIGGRAPH, CVPR, TPAMI, ToG, AAAI, WACV, 3DV, ACCV, TVCG, Signal Processing, SIGGRAPH ASIA, CGF, PG, Eurographics, etc.

Internship:

MPI Tübingen (2020), FAIR, Adobe Research (2019), Pinscreen (2018), FRL (2017), FOVE (2015), Yahoo! Japan (2012-13)

Talk:

CMU VASC Seminar, Brown VC Seminar, BIRS Workshop, Asiagraphics Webinar (2023), Dagstuhl Seminar (2022), MPI Saarbruecken/Tuebingen, Industrial Light & Magic, Dagstuhl Seminar (2019), University of Tokyo, CMU Graphics Group, VC Symposium (2018), VC/GCAD Symposium, Autodesk Research (2015)

Scholarship and Awards:

Outstanding Reviewer (ICCV2023, 3DV2020, ACCV2020),
CVPR Best Paper Finalist (2019, 2021),
Super Creator, IPA Exploratory Software (2015),
ACM Student Research Competition Semi-Finalist (2014),
MIRU Interactive Presentation Award, Okayama, Japan (2014),
UCLA CSST Scholarship Program $9,036 (2013),
Okuma Memorial Special Scholarship $5,000 (2011)