Switch all rendering fully to Vulkan and update filament to latest. - #7550
Draft
ssheorey wants to merge 73 commits into
Draft
Switch all rendering fully to Vulkan and update filament to latest.#7550ssheorey wants to merge 73 commits into
ssheorey wants to merge 73 commits into
Conversation
…ackaging Enable BUNDLE_OPEN3D_ML automatically when PyTorch or TensorFlow ops are built, ship Open3D-ML pip deps as extras_require[ml], and validate imports via require_ml_extra. Bundle Filament LLVM libc++ on Linux GUI wheels with a small ldd helper; refresh CI/Docker for Open3D-ML main, CUDA 12.6+ arch defaults, and local style --changed-only.
Make BUNDLE_OPEN3D_ML an option again, defaulting to ON only when ML ops are built and OPEN3D_ML_ROOT resolves, so local builds can opt out and CI jobs without an Open3D-ML checkout configure instead of erroring out. Windows wheels now bundle Open3D-ML too. Import the Open3D-ML backed submodules of open3d.ml, open3d.ml.torch and open3d.ml.tf lazily and gate them on require_ml_extra(), so the ops and layers stay usable with a core-only install and a missing dependency reports how to install the ml extra. Wheel tests install that extra. Keep Turing usable after dropping its SASS by emitting compute_75 PTX, which also needs a -virtual case when translating architectures for PyTorch. Fall back to the pybind extension when probing for libc++ in static builds, where libOpen3D is not packaged. Co-authored-by: Cursor <cursoragent@cursor.com>
NVIDIA's static CUDA libraries carry device code back to sm_50, ~47MB of which no GPU that Open3D supports can run. nvprune only accepts static libraries, not the linked Open3D library, so the imported CUDA targets are pruned and re-pointed before linking, cached per architecture list. Also fix the Windows path separator assertion in the TensorBoard plugin test, which started running there now that Windows wheels bundle Open3D-ML, and skip the module cleanly when the `ml` extra is not installed. Co-authored-by: Cursor <cursoragent@cursor.com>
…ask written from a parallel loop Fixes #7538
…hing PyTorch/NumPy (#7363)
…7497) Replace AssertTensorDevice(CPU) with .To(CPU) in InverseTransformation and TransformIndexer so callers with CUDA/SYCL tensors do not get device mismatch errors. Both functions read small float64 tensors (3x3 and 4x4) and copy the data into host-side arrays, so a device transfer is cheap and semantically correct. This is a general fix that covers all call sites including VoxelBlockGrid, PointCloud, and Image kernel paths.
Fix compilation error when building with USE_SYSTEM_EIGEN3=ON on macOS with clang++. The AddPoint template functions used Eigen::ArrayBase<Derived3> as an unconstrained parameter type, which causes substitution failures when the compiler instantiates Eigen::ArrayBase for types that don't derive from it. This manifests under stricter C++17 template resolution when Eigen is included from a system path (-isystem) rather than as a project-local header. Fix: - Add std::enable_if constraint requiring Derived3 to be derived from Eigen::ArrayBase<Derived3> - Change parameter from 'const Eigen::ArrayBase<Derived3>&' to 'const Derived3&' (semantically identical since Derived3 IS-A ArrayBase) Applied to both Accumulator::AddPoint and AccumulatorBackprop::AddPoint.
Previously BUNDLE_OPEN3D_ML=ON hardcoded GIT_TAG origin/main in the ExternalProject and ci_utils.sh forced a main branch checkout. This prevented reproducible builds using matching release tags. Add OPEN3D_ML_VERSION CMake cache variable (default: main) that controls the git ref checked out from the Open3D-ML repo. Users can set -DOPEN3D_ML_VERSION=v0.19.0 for reproducible builds or any other tag/branch. Also exposed as a Docker build arg in Dockerfile.ci, forwarded to cmake, and added to ci_build() in docker_build.sh (defaults to main). Removed the checkout -b main workaround in ci_utils.sh and Dockerfile.ci; replaced with fetch --tags so version tags are available when OPEN3D_ML_VERSION is set to one.
… ss/gs-vulkan-backend
Use the shared RGBA16F Vulkan image as Filament's color attachment and composite splats in place, preserving mesh color in offscreen readback. Harden shared-device setup and update the focused mixed mesh/splat tests and design documentation.
… ss/gs-vulkan-backend
Align VMA dispatch settings with Filament to prevent allocator initialization crashes. Preserve required Vulkan device features for Filament compatibility.
Prefer suitable discrete Vulkan GPUs over integrated adapters with a simple score-based selection policy, while retaining the first device when scores tie. Keep the Vulkan context shared with Filament so Gaussian splat compute and Filament rendering use the same device and queue configuration. Route CPU Vulkan implementations such as llvmpipe to Filament's OpenGL backend. Filament's Vulkan path can crash inside llvmpipe, whereas the OpenGL path provides working interactive and offscreen software rendering. Align the Filament build configuration and Vulkan context feature setup with the allocator and shared-device requirements. Update the Gaussian splat design notes and visualization documentation to describe the current backend behavior and software-rendering environment variables.
- Correct mesh depth decoding and GS depth encoding for Filament's infinite-far reversed-Z render projection, preventing distance-dependent false occlusion. - Separate Vulkan in-place color composition from Metal's transparent overlay path, and keep offscreen color/depth targets and readback compatible with each backend. - Extend the per-frame depth flags and renderer plumbing so scene-depth presence and composition mode are explicit to the composite shader. - Update mixed-scene and splat-only rendering goldens to cover the corrected depth values, and document the depth conventions and formulas in GaussianSplatDesign.md.
- Avoid marking redraws pending before the GUI run loop starts, because macOS rejects those startup requests and otherwise leaves windows blank. - Preserve the existing WindowSystem API and backend behavior while retaining redraw coalescing once the application is running. - Update AGENTS.md to require descriptive commit bodies, and apply the current formatting cleanup across the affected rendering and Gaussian splat files.
- Incorporate the latest main branch build, packaging, documentation, and CI updates. - Preserve the new Gaussian splat design without GPU adapter selection or legacy OpenGL/Vulkan interop context files. - Resolve registration, SYCL Sobel indexing, Docker, and CI configuration conflicts while retaining both branches' compatible behavior.
- Windows XPU: disable BUILD_FILAMENT_FROM_SOURCE for SYCL (Intel) builds, as those don't need Vulkan patches and the Intel compiler cannot build Filament's filameshio.lib target. - Apple Filament: supply v1.57.2 source tarball hash (was empty, causing CMake ExternalProject_Add to fail). - Vulkan layout tracking: register externally-imported images with COLOR_ATTACHMENT_OPTIMAL / DEPTH_STENCIL_ATTACHMENT_OPTIMAL initial layouts to match Filament's post-render state, preventing the first compute transition from discarding mesh color. - FilamentRenderToBuffer: don't call callback with stale buffer when Vulkan direct readback fails; return empty frame instead. - Docs: fix RST underline length in headless_rendering.rst and remove orphaned sentence fragment in GaussianSplatDesign.md.
…e inherited RenderToBuffer::Buffer type. Fixed Vulkan Gaussian texture readback in ComputeGPUVulkan.cpp: removed an incorrect row flip after vkCmdCopyImageToBuffer. Made the GPU golden assertions in GaussianSplatRender.cpp tolerate ±1 half-float conversion variance and one mesh-edge rasterization pixel in the mixed scene.
Serialize shared-queue access before GS geometry and preserve mixed GS/Filament output across FAST and BEST LOD transitions. Track packed GS revisions independently from Filament scene changes to avoid redundant buffer uploads, add focused coverage, and record the Windows Vulkan debugging evidence.
Add an idempotent Windows development bootstrap and correct imported Vulkan image handoff bookkeeping for Gaussian splat rendering. End the composite frame before checking submission status so release barriers are recorded before submission. Document that interactive and offscreen rendering pass while depth-layout VUID 09600 remains unresolved.
Reduce duplication and nesting in the shared-VkDevice Gaussian splat code without changing rendered output, then remove the test tolerance that was hiding two incorrect golden values. GaussianSplatVulkanContext: extract ScorePhysicalDevice() and FindGraphicsComputeQueueFamily() so device selection is no longer a nested ternary inside the enumeration loop, and so the queue-family search is not duplicated between scoring and device creation. Replace CheckExtensions() (whose out-param was never read) with HasExtension/HasAllExtensions. Build images through vk::raii handles with release() on success, replacing three try/catch blocks that each repeated manual cleanup. ComputeGPUVulkan: replace TransitionImageLayout()'s four five-way nested ternaries with SrcScopeForLayout()/DstScopeForLayout() switches, and replace two hand-rolled raw-C VkImageMemoryBarrier2 blocks with MemoryBarrierInLayout() so each call site states why the barrier exists. Collapse LoadShader()'s five sequential try/catch blocks into one, since every intermediate was already RAII. Consolidate the duplicated descriptor binders, buffer resizers, texture resizers, and init helpers. FilamentRenderToBuffer: give each platform one contiguous branch instead of interleaved __APPLE__ blocks with endFrame()/flushAndWait() scattered between them, and add DeliverFrame() for the callback-and-mark-done pattern that appeared five times. Rename leftover "interop" locals and comments; the GL interop path they refer to no longer exists. GaussianSplatDesign.md: a prose section had been inserted into the middle of the Filament integration table, orphaning its last three rows from the rendered table. Move the rows back and fold the prose into the paragraph that already described the same mechanism. Tests: compare all 64 pixels exactly instead of allowing one outlier at tolerance 1. That allowance was masking two wrong goldens in the mixed splat/mesh scene: the cube at z=2.8 was recorded as depth 0 rather than 140 (2.2 m from the camera x 64) and its colour as 150 rather than 15. Correct both, plus several off-by-one colour values, and print a paste-ready grid on failure so future golden updates are deliberate. Verified on two Linux Vulkan vendors, each five consecutive runs, all three GaussianSplatRenderTest cases passing with byte-identical output: NVIDIA RTX 3070 Laptop (driver 580.173.02) and Intel UHD Graphics CML GT2 (Mesa 25.2.8). Two independent implementations agreeing means the off-by-one colour values are not driver rounding, so no tolerance is warranted. Intel exposes a single queue in family 0, so that run also covers the single-queue branch of the queue setup touched here (GS and Filament share queue index 0), which NVIDIA never reaches. Record both in GaussianSplatDesign.md and correct a stale claim there that the only test GPU was an RTX 4090. Windows, AMD and macOS/Metal remain unverified.
Upgrade source Filament to v1.76 on Linux and Windows, retain the minimal external-image import patch, and remove the obsolete swapchain backport. Synchronize imported image layouts with Filament, enable depth clamping on the shared device, support software Vulkan 3DGS, and record validation results across NVIDIA, Intel, and Lavapipe.
Upgrade macOS Filament integration to v1.76, update offscreen readback sequencing, and preserve Gaussian splat output across scene updates and Metal resize target recreation.
|
Thanks for submitting this pull request! The maintainers of this repository would appreciate if you could update the CHANGELOG.md based on your changes. |
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Shared-context ABI mismatches, unpatched default binaries, and rendering lifecycle regressions can cause crashes or incorrect output.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Replaces hybrid OpenGL–Vulkan Gaussian-splat rendering with a shared Vulkan device on non-Apple platforms and updates Filament integration, build infrastructure, documentation, and rendering tests.
Changes:
- Shares Vulkan devices and images directly between Filament and Gaussian-splat compute.
- Updates Filament to v1.76 and adjusts platform builds, packaging, and CI.
- Adds offscreen rendering tests and revises CPU-rendering guidance.
File summaries
| File | Description |
|---|---|
util/setup_windows_development.ps1 |
Adds Windows development bootstrap script. |
examples/cpp/DrawWebRTC.cpp |
Increases example window sizes. |
docs/tutorial/visualization/headless_rendering.rst |
Labels legacy OpenGL rendering. |
docs/tutorial/visualization/cpu_rendering.rst |
Documents software Vulkan rendering. |
docker/Dockerfile.openblas |
Adds Vulkan and Filament build dependencies. |
docker/Dockerfile.ci |
Builds and exports patched Filament. |
docker/docker_test.sh |
Propagates CI state into containers. |
docker/docker_build.sh |
Exports Filament build artifacts. |
cpp/tests/visualization/rendering/GaussianSplatRender.cpp |
Adds Gaussian-splat rendering goldens. |
cpp/tests/visualization/CMakeLists.txt |
Registers the new rendering tests. |
cpp/open3d/visualization/rendering/Open3DScene.cpp |
Refactors LOD visibility selection. |
cpp/open3d/visualization/rendering/GpuAdapterSelection.h |
Removes GL adapter-selection API. |
cpp/open3d/visualization/rendering/GpuAdapterSelection.cpp |
Removes adapter-selection implementation. |
cpp/open3d/visualization/rendering/gaussian_splat/shaders/gaussian_composite.comp |
Adds in-place Vulkan compositing. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatVulkanInteropContext.h |
Removes GL–Vulkan interop context. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatVulkanContext.h |
Declares shared Vulkan context. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatVulkanContext.cpp |
Implements Vulkan device and image ownership. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatVulkanBackend.h |
Updates backend documentation. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatVulkanBackend.cpp |
Imports shared Vulkan images into Filament. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatRenderer.h |
Splits geometry and composite invalidation. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatRenderer.cpp |
Revises rendering lifecycle and readback. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatPassRunner.cpp |
Adapts passes to native Vulkan images. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatOpenGLContext.h |
Removes shared OpenGL context API. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatOpenGLContext.cpp |
Removes shared OpenGL implementation. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatMetalBackend.mm |
Updates Metal caching and readback. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatDesign.md |
Documents the shared-device architecture. |
cpp/open3d/visualization/rendering/gaussian_splat/GaussianSplatDataPacking.h |
Adds scene revision tracking. |
cpp/open3d/visualization/rendering/gaussian_splat/ComputeGPUVulkan.h |
Exposes direct Vulkan image registration. |
cpp/open3d/visualization/rendering/gaussian_splat/ComputeGPUMetal.mm |
Adds aligned RGBA16F readback. |
cpp/open3d/visualization/rendering/gaussian_splat/ComputeGPU.h |
Adds RGBA16F download abstraction. |
cpp/open3d/visualization/rendering/filament/FilamentView.cpp |
Exposes composited Vulkan color output. |
cpp/open3d/visualization/rendering/filament/FilamentScene.h |
Adds render scheduling and revisions. |
cpp/open3d/visualization/rendering/filament/FilamentScene.cpp |
Revises invalidation and rendered-view tracking. |
cpp/open3d/visualization/rendering/filament/FilamentResourceManager.h |
Generalizes imported texture handles. |
cpp/open3d/visualization/rendering/filament/FilamentResourceManager.cpp |
Updates imports and texture usage flags. |
cpp/open3d/visualization/rendering/filament/FilamentRenderToBuffer.h |
Adds readback state and delivery helper. |
cpp/open3d/visualization/rendering/filament/FilamentRenderToBuffer.cpp |
Implements backend-specific GS readback. |
cpp/open3d/visualization/rendering/filament/FilamentRenderer.h |
Tracks rendered views and callbacks. |
cpp/open3d/visualization/rendering/filament/FilamentRenderer.cpp |
Reorders Vulkan and Metal compositing. |
cpp/open3d/visualization/rendering/filament/FilamentNativeInterop.mm |
Avoids RTTI for Metal platform access. |
cpp/open3d/visualization/rendering/filament/FilamentEngine.cpp |
Defaults non-Apple rendering to Vulkan. |
cpp/open3d/visualization/gui/Window.cpp |
Coalesces redraw requests. |
cpp/open3d/visualization/gui/SceneWidget.cpp |
Keeps overlays only on Metal. |
cpp/open3d/visualization/gui/GLFWWindowSystem.cpp |
Removes GL adapter steering. |
cpp/open3d/visualization/gui/Application.cpp |
Adds resource-path layouts. |
cpp/open3d/visualization/CMakeLists.txt |
Replaces interop sources with Vulkan context. |
CMakeLists.txt |
Changes Windows runtime defaulting. |
AGENTS.md |
Expands commit-message guidance. |
3rdparty/find_dependencies.cmake |
Adjusts source-built Filament paths. |
3rdparty/filament/patches/0001-importTextureR.patch |
Adds external Vulkan-image imports. |
3rdparty/filament/filament_download.cmake |
Updates prebuilt Filament packages. |
3rdparty/filament/filament_build.cmake |
Updates and patches source-built Filament. |
.github/workflows/windows.yml |
Builds and uploads Windows Filament artifacts. |
.github/workflows/ubuntu.yml |
Preserves artifacts when tests fail. |
.github/workflows/ubuntu-sycl.yml |
Preserves SYCL artifacts on test failure. |
.github/workflows/ubuntu-openblas.yml |
Uploads ARM64 Filament artifacts. |
.github/prompts/code-simplifier.md |
Adds a code-simplification prompt. |
Review details
- Files reviewed: 60/60 changed files
- Comments generated: 9
- Review effort level: Balanced
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| include(FetchContent) | ||
|
|
||
| set(filament_LIBRARIES filameshio filament filaflat filabridge geometry backend bluegl bluevk ibl image ktxreader meshoptimizer smol-v utils vkshaders) | ||
| set(filament_LIBRARIES filameshio filament filaflat filabridge geometry backend bluegl bluevk ibl image ktxreader meshoptimizer smol-v utils shaders) |
| option(WITH_IPP "Use Intel Integrated Performance Primitives" ON ) | ||
| option(BUILD_SYCL_MODULE "Build SYCL module with Intel oneAPI" OFF) | ||
| if(BUILD_SHARED_LIBS OR WIN32 AND BUILD_SYCL_MODULE) | ||
| if(BUILD_SHARED_LIBS) |
Comment on lines
120
to
+121
| if (backend == filament::backend::Backend::DEFAULT) { | ||
| backend = filament::backend::Backend::OPENGL; | ||
| backend = filament::backend::Backend::VULKAN; |
Comment on lines
+64
to
+70
| struct FilamentVulkanSharedContext { | ||
| VkInstance instance = VK_NULL_HANDLE; | ||
| VkPhysicalDevice physical_device = VK_NULL_HANDLE; | ||
| VkDevice logical_device = VK_NULL_HANDLE; | ||
| std::uint32_t graphics_queue_family_index = 0xFFFFFFFFu; | ||
| std::uint32_t graphics_queue_index = 0xFFFFFFFFu; | ||
| }; |
Comment on lines
+261
to
262
| const bool run_gs_pipeline = | ||
| gaussian_splat_renderer_ && scene_->HasGaussianSplatGeometry(); |
Comment on lines
+265
to
+267
| for (FilamentView* view : rendered_views_) { | ||
| gaussian_splat_renderer_->RequestCompositeForView(*view); | ||
| gaussian_splat_renderer_->RenderCompositeStage(*view); |
Comment on lines
+239
to
+243
| void FilamentScene::MarkGaussianSplatChanged() { | ||
| ++gaussian_splat_revision_; | ||
| if (merged_gs_attrs_) { | ||
| merged_gs_attrs_->revision = gaussian_splat_revision_; | ||
| } |
Comment on lines
+258
to
+260
| if (!gpu_ || targets.color_vk_image == 0) return false; | ||
| return gpu_->DownloadTextureRGBA16F(targets.color_vk_image, | ||
| targets.width, targets.height, out); |
Comment on lines
+315
to
+319
| const char* ci = std::getenv("CI"); | ||
| // Very rough way to tell if a CI machine has a GPU | ||
| if (ci && !core::cuda::IsAvailable() && | ||
| core::sy::GetDeviceCount() < 2) { | ||
| GTEST_SKIP() << "Gaussian splat rendering requires GPU in CI"; |
Member
Author
There was a problem hiding this comment.
software rendering should work with the updated filament everywhere. No need to skip the test in CI now.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The previous 3DGS rendering design with hybrid OpenGL - Vulkan rendering for 3DGS was buggy in the case of multi-GPU machines. Windows provides a way to query the GPU in use and create an independent context on the same device (# GPU selection PR ) , but Linux does not have a portable way to do that. This led to crashes if opengl and vulkan were using different GPUs, or if there were attempts / miitigations to force that (such as env vars).
This PR simplifies the whole design by switching completely to Vulkan - so that a single context can be used for both filament and 3DGS shaders.
Type
Motivation and Context
Checklist:
python util/check_style.py --applyto apply Open3D code styleto my code.
updated accordingly.
results (e.g. screenshots or numbers) here.
Description