Research Areas
Real-time graphics
This research focuses on scalable graphics algorithms for rendering large and complex scenes at real-time frame rates. We investigate novel approaches to techniques such as level of detail, image-based rendering, potentially visible sets, frame extrapolation and geometry compression. We are particularly interested in rendering methods suitable for streaming rendering and virtual reality displays.| <a href="/pdf/Schmalstieg_{"id"=>"447", "title"=>"Disocclusion buffer for potentially visible sets"}.pdf"><img src="/img/Schmalstieg_{"id"=>"447", "title"=>"Disocclusion buffer for potentially visible sets"}.jpg" width=200 alt="Disocclusion buffer for potentially visible sets"></a> | <a href="/pdf/Schmalstieg_{"id"=>"446", "title"=>"NeuralPVS for potentially visible set prediction"}.pdf"><img src="/img/Schmalstieg_{"id"=>"446", "title"=>"NeuralPVS for potentially visible set prediction"}.jpg" width=200 alt="NeuralPVS for potentially visible set prediction"></a> | <a href="/pdf/Schmalstieg_{"id"=>"427", "title"=>"Laced wires for geometry decompression on GPU"}.pdf"><img src="/img/Schmalstieg_{"id"=>"427", "title"=>"Laced wires for geometry decompression on GPU"}.jpg" width=200 alt="Laced wires for geometry decompression on GPU"></a> |
| <a href="/pdf/Schmalstieg_{"id"=>"414", "title"=>"Trim regions for potentially visible set generation"}.pdf"><img src="/img/Schmalstieg_{"id"=>"414", "title"=>"Trim regions for potentially visible set generation"}.jpg" width=200 alt="Trim regions for potentially visible set generation"></a> | <a href="/pdf/Schmalstieg_{"id"=>"396", "title"=>"Temporally adaptive shading atlas for incremental shading"}.pdf"><img src="/img/Schmalstieg_{"id"=>"396", "title"=>"Temporally adaptive shading atlas for incremental shading"}.jpg" width=200 alt="Temporally adaptive shading atlas for incremental shading"></a> | <a href="/pdf/Schmalstieg_{"id"=>"362", "title"=>"Shading atlas streaming for virtual reality rendering"}.pdf"><img src="/img/Schmalstieg_{"id"=>"362", "title"=>"Shading atlas streaming for virtual reality rendering"}.jpg" width=200 alt="Shading atlas streaming for virtual reality rendering"></a> |
Situated visualizations
Situted visualization uses augmented reality displays to present information to a human user that is dynamically derived from the user's physical environment. It provides guidance, instructions and context in everyday situations. We investigate when, where and how to visualize such information, and what information to visualize.| <a href="/pdf/Schmalstieg_{"id"=>"451", "title"=>"Situated Brushing and Linking in VR and AR"}.pdf"><img src="/img/Schmalstieg_{"id"=>"451", "title"=>"Situated Brushing and Linking in VR and AR"}.jpg" width=200 alt="Situated Brushing and Linking in VR and AR"></a> | <a href="/pdf/Schmalstieg_{"id"=>"440", "title"=>"CECILIA: embedding visualizations in existing games"}.pdf"><img src="/img/Schmalstieg_{"id"=>"440", "title"=>"CECILIA: embedding visualizations in existing games"}.jpg" width=200 alt="CECILIA: embedding visualizations in existing games"></a> | <a href="/pdf/Schmalstieg_{"id"=>"433", "title"=>"AR assembly guidance with error management"}.pdf"><img src="/img/Schmalstieg_{"id"=>"433", "title"=>"AR assembly guidance with error management"}.jpg" width=200 alt="AR assembly guidance with error management"></a> |
| <a href="/pdf/Schmalstieg_{"id"=>"416", "title"=>"guitARhero: augmented reality for guitar learning"}.pdf"><img src="/img/Schmalstieg_{"id"=>"416", "title"=>"guitARhero: augmented reality for guitar learning"}.jpg" width=200 alt="guitARhero: augmented reality for guitar learning"></a> | <a href="/pdf/Schmalstieg_{"id"=>"419", "title"=>"AR visualization patterns for reusable situated visualizations"}.pdf"><img src="/img/Schmalstieg_{"id"=>"419", "title"=>"AR visualization patterns for reusable situated visualizations"}.jpg" width=200 alt="AR visualization patterns for reusable situated visualizations"></a> | <a href="/pdf/Schmalstieg_{"id"=>"404", "title"=>"RagRug: a unified development framework for situated analytics"}.pdf"><img src="/img/Schmalstieg_{"id"=>"404", "title"=>"RagRug: a unified development framework for situated analytics"}.jpg" width=200 alt="RagRug: a unified development framework for situated analytics"></a> |
Photorealistic augmented reality
Ideally, virtual objects displayed in augmented reality would be indistinguishable from real objects. This grand challenge requires capturing the entire image formation process in reality and simulating the virtual counterpart. It encompasses reality capture, light transport, camera simulation and much more.| <a href="/pdf/Schmalstieg_{"id"=>"452", "title"=>"HandLight"}.pdf"><img src="/img/Schmalstieg_{"id"=>"452", "title"=>"HandLight"}.jpg" width=200 alt="HandLight"></a> | <a href="/pdf/Schmalstieg_{"id"=>"428", "title"=>"Neural Bokeh"}.pdf"><img src="/img/Schmalstieg_{"id"=>"428", "title"=>"Neural Bokeh"}.jpg" width=200 alt="Neural Bokeh"></a> | <a href="/pdf/Schmalstieg_{"id"=>"337", "title"=>"Learning lightprobes"}.pdf"><img src="/img/Schmalstieg_{"id"=>"337", "title"=>"Learning lightprobes"}.jpg" width=200 alt="Learning lightprobes"></a> |
Image-based modeling and rendering
Capturing the geometry and appearance of the real world and novel view sythesis from images is a key technique for compelling extended reality techniques, such as telepresence or mediated reality. In this work, we investigate emerging scene representations, such as light fields and radiance fields.| <a href="/pdf/Schmalstieg_{"id"=>"448", "title"=>"Sorted Opacity Fields"}.pdf"><img src="/img/Schmalstieg_{"id"=>"448", "title"=>"Sorted Opacity Fields"}.jpg" width=200 alt="Sorted Opacity Fields"></a> | <a href="/pdf/Schmalstieg_{"id"=>"442", "title"=>"AAA Gaussians"}.pdf"><img src="/img/Schmalstieg_{"id"=>"442", "title"=>"AAA Gaussians"}.jpg" width=200 alt="AAA Gaussians"></a> | <a href="/pdf/Schmalstieg_{"id"=>"418", "title"=>"VR Photo Inpainting"}.pdf"><img src="/img/Schmalstieg_{"id"=>"418", "title"=>"VR Photo Inpainting"}.jpg" width=200 alt="VR Photo Inpainting"></a> |
| <a href="/pdf/Schmalstieg_{"id"=>"406", "title"=>"Good Keyframes to Inpaint"}.pdf"><img src="/img/Schmalstieg_{"id"=>"406", "title"=>"Good Keyframes to Inpaint"}.jpg" width=200 alt="Good Keyframes to Inpaint"></a> | <a href="/pdf/Schmalstieg_{"id"=>"380", "title"=>"InpaintFusion: RGB-D Inpainting"}.pdf"><img src="/img/Schmalstieg_{"id"=>"380", "title"=>"InpaintFusion: RGB-D Inpainting"}.jpg" width=200 alt="InpaintFusion: RGB-D Inpainting"></a> | <a href="/pdf/Schmalstieg_{"id"=>"374", "title"=>"MR Light Fields"}.pdf"><img src="/img/Schmalstieg_{"id"=>"374", "title"=>"MR Light Fields"}.jpg" width=200 alt="MR Light Fields"></a> |
3D reconstruction and authoring
Augmented reality applications require new types of multimedia content to deliver convincing instructions and guidance. This content is generated using techniques in the area of semantic and parameterized reconstruction combined with procedural and computer-assisted 3D authoring.| <a href="/pdf/Schmalstieg_{"id"=>"429", "title"=>"Subsurface infrastructure reconstruction"}.pdf"><img src="/img/Schmalstieg_{"id"=>"429", "title"=>"Subsurface infrastructure reconstruction"}.jpg" width=200 alt="Subsurface infrastructure reconstruction"></a> | <a href="/pdf/Schmalstieg_{"id"=>"434", "title"=>"AuthXR: An immersive authoring system for industrial procedures"}.pdf"><img src="/img/Schmalstieg_{"id"=>"434", "title"=>"AuthXR: An immersive authoring system for industrial procedures"}.jpg" width=200 alt="AuthXR: An immersive authoring system for industrial procedures"></a> | <a href="/pdf/Schmalstieg_{"id"=>"444", "title"=>"IntelliCap: Intelligent Guidance for Consistent View Sampling"}.pdf"><img src="/img/Schmalstieg_{"id"=>"444", "title"=>"IntelliCap: Intelligent Guidance for Consistent View Sampling"}.jpg" width=200 alt="IntelliCap: Intelligent Guidance for Consistent View Sampling"></a> |
| <a href="/pdf/Schmalstieg_{"id"=>"408", "title"=>"Model-Free Authoring by Demonstration of Assembly Instructions"}.pdf"><img src="/img/Schmalstieg_{"id"=>"408", "title"=>"Model-Free Authoring by Demonstration of Assembly Instructions"}.jpg" width=200 alt="Model-Free Authoring by Demonstration of Assembly Instructions"></a> | <a href="/pdf/Schmalstieg_{"id"=>"330", "title"=>"Authoring of AR surface instructions"}.pdf"><img src="/img/Schmalstieg_{"id"=>"330", "title"=>"Authoring of AR surface instructions"}.jpg" width=200 alt="Authoring of AR surface instructions"></a> | <a href="/pdf/Schmalstieg_{"id"=>"288", "title"=>"Technical documentation retargeting to AR"}.pdf"><img src="/img/Schmalstieg_{"id"=>"288", "title"=>"Technical documentation retargeting to AR"}.jpg" width=200 alt="Technical documentation retargeting to AR"></a> |
Localization and tracking
Wide-area localization and tracking is key enabling technology for augmented reality. Our research investigates scalable scene descriptors, sensor fusion, and multimodal localization techniques.| <a href="/pdf/Schmalstieg_{"id"=>"453", "title"=>"Change-Resilient Localization"}.pdf"><img src="/img/Schmalstieg_{"id"=>"453", "title"=>"Change-Resilient Localization"}.jpg" width=200 alt="Change-Resilient Localization"></a> | <a href="/pdf/Schmalstieg_{"id"=>"417", "title"=>"Bag of Wor(l)d Anchors"}.pdf"><img src="/img/Schmalstieg_{"id"=>"417", "title"=>"Bag of Wor(l)d Anchors"}.jpg" width=200 alt="Bag of Wor(l)d Anchors"></a> | <a href="/pdf/Schmalstieg_{"id"=>"407", "title"=>"Compact World Anchors"}.pdf"><img src="/img/Schmalstieg_{"id"=>"407", "title"=>"Compact World Anchors"}.jpg" width=200 alt="Compact World Anchors"></a> |
| <a href="/pdf/Schmalstieg_{"id"=>"402", "title"=>"HoloLens stereo tracking"}.pdf"><img src="/img/Schmalstieg_{"id"=>"402", "title"=>"HoloLens stereo tracking"}.jpg" width=200 alt="HoloLens stereo tracking"></a> | <a href="/pdf/Schmalstieg_{"id"=>"366", "title"=>"TrackCap"}.pdf"><img src="/img/Schmalstieg_{"id"=>"366", "title"=>"TrackCap"}.jpg" width=200 alt="TrackCap"></a> | <a href="/pdf/Schmalstieg_{"id"=>"364", "title"=>"VR Upper Body Pose"}.pdf"><img src="/img/Schmalstieg_{"id"=>"364", "title"=>"VR Upper Body Pose"}.jpg" width=200 alt="VR Upper Body Pose"></a> |
Immersive displays
Conventional XR displays generate only certain types of depth cues and fall short of a hypothetical ultimate display. Our research on immersive display technology improves some of the limitations of existing XR display technology using light field approximation and focal cue synthesis.| <a href="/pdf/Schmalstieg_{"id"=>"431", "title"=>"Gaze-Contingent Layered Optical See Through"}.pdf"><img src="/img/Schmalstieg_{"id"=>"431", "title"=>"Gaze-Contingent Layered Optical See Through"}.jpg" width=200 alt="Gaze-Contingent Layered Optical See Through"></a> | <a href="/pdf/Schmalstieg_{"id"=>"412", "title"=>"Off-Axis Layered See-Through Head Mounted Display"}.pdf"><img src="/img/Schmalstieg_{"id"=>"412", "title"=>"Off-Axis Layered See-Through Head Mounted Display"}.jpg" width=200 alt="Off-Axis Layered See-Through Head Mounted Display"></a> | <a href="/pdf/Schmalstieg_{"id"=>"403", "title"=>"Video See Through Display with Focal Cues"}.pdf"><img src="/img/Schmalstieg_{"id"=>"403", "title"=>"Video See Through Display with Focal Cues"}.jpg" width=200 alt="Video See Through Display with Focal Cues"></a> |