Photo2Mesh: SAM3D 插件用户手册

Photo2Mesh: SAM3D - User Manual

Dragonfly Prototype Apps · Photo2Mesh: SAM3D...

版本 Version 1.0 · 2026-07-04

第一部分 中文手册

目录

1. 简介

2. 适用场景

3. 安装与启用

4. 运行环境与首次配置

5. 界面说明

6. 使用步骤

7. 参数说明

8. 输出结果

9. 常见问题与故障排除

10. 注意事项与已知限制

11. 参考资料

1. 简介

Photo2Mesh: SAM3D 是一款 Dragonfly 插件,可从单张照片重建物体的三维表面网格。它把整条流程拆成两个先后衔接的阶段(对应面板的两个选项卡):第一阶段用 Meta 的 SAM 3D Objects 人工智能模型,从一张图像重建出三维点云;第二阶段用 Open3D 把点云转换为三角形表面网格,并作为网格对象导入 Dragonfly 场景,可直接用于可视化与测量。

该插件是 Photo2Mesh 系列的一员(其他成员为 Meshroom、COLMAP、gsplat),在菜单中显示为 Prototype Apps ▸ Photo2Mesh: SAM3D...。与需要多视角照片的摄影测量方法不同,SAM3D 只需要一张照片即可推断出三维形状。

底层引擎与算法

• SAM 3D Objects(仓库 facebookresearch/sam-3d-objects):Meta 发布的单图到三维模型的 AI 模型。它接收一张图像和一个覆盖整幅图像的掩膜(即整幅画面就是目标物体),输出一组高斯泼溅(Gaussian Splatting)表示的三维点。插件随后从中提取出干净的 XYZ(可含 RGB)点云 points.ply。
• Open3D(表面重建):把点云估计并定向法线后,用三种方法之一重建三角面片——Poisson(泊松,默认)、Ball-Pivoting / BPA(球旋转)或 Alpha(阿尔法形状),再清理重复顶点与退化三角面,得到 mesh.ply。
• 重型计算栈基于 torch 2.5.1(CUDA cu121),并依赖 PyTorch3D 与 kaolin,全部运行在插件自建的专用虚拟环境中,不占用 Dragonfly 自带的 Python。

许可证要点

• SAM 3D Objects 采用 SAM License:免版税、允许商业使用,不设月活跃用户(MAU)或企业规模上限。使用条件:随任何再分发附上许可证副本、在论文中注明出处、遵守贸易管制与不用于军事用途等条款。
• Open3D 采用 Apache-2.0 许可证。
• 因此本插件可安全用于产品化场景。

2. 适用场景

当你只有一张照片、却希望快速把实物样品、零件或标本数字化为三维模型时,本插件非常适用。典型用途包括材料科学、工业检测和生命科学实验室中的逆向工程、记录存档、教学演示与成果展示。

• 对不透明、纹理丰富、且占满画面的物体效果最佳。
• 生成的网格比例与朝向是任意的——导入 Dragonfly 后通常需要重新缩放和摆正,才能与真实尺寸对应。
• 第二个选项卡(Point Cloud to Mesh)也可单独使用:为任何已有的点云文件(.ply / .pcd / .xyz)生成表面网格,而无需经过第一阶段的 AI 重建。

SAM3D 是单图三维推断,结果是模型对物体形状的“猜测”,不等同于经过标定的高精度测量重建。它擅长快速得到可视化用的形状,不适合作为计量级尺寸依据。

3. 安装与启用

本插件随 Prototype Labs & Apps 完整安装包(Full Package) 一起分发。安装步骤如下:

1. 把安装包解压到任意较短路径的文件夹(例如 C:\PL\,避免过深或 OneDrive 重定向的路径)。

2. 双击运行 `Install_FullPackage.bat`。

3. 在弹出的对话框中选择核心安装模式(Fresh 全新安装 / Compatible 兼容安装,该选择只影响 Prototype Labs 核心的 blocks 和 recipe,不影响任何插件的环境)。

4. 在插件列表中勾选 Photo2Mesh: SAM3D。注意:所有插件默认未勾选(默认关闭),必须手动勾选后才会安装。

5. 点击 Install,等待控制台完成。

6. 完全退出并重启 Dragonfly(菜单只在启动时扫描)。

重启后,插件出现在 Prototype Apps ▸ Photo2Mesh: SAM3D... 菜单下(位于 Photo2Mesh 分组)。点击即可打开一个可停靠(默认浮动)的面板。

以后修改勾选

最方便的方式是在 Dragonfly 内修改:打开 Developer ▸ Prototype Labs... ▸ Menu Item Manager,在底部的 “Prototype Apps (Full Package)” 列表里找到本插件的勾选框——勾选=部署菜单项,取消=移除菜单项,修改后重启 Dragonfly 生效。停用从不删除插件已搭建好的环境,重新启用即刻可用。也可以随时重跑安装器,它会记住上次的勾选作为默认值。

卸载

双击 `Uninstall_FullPackage.bat` 移除所有 Full Package 菜单项与插件。卸载会保留各插件已搭建的环境(venv / 下载内容),其路径会在结束时列出,若需腾出磁盘空间可手动删除。

4. 运行环境与首次配置

SAM3D 的重型 AI 计算栈不会在安装时下载,而是在首次使用时由你在面板的 Environment(环境) 区域点击 Setup Environment 一键搭建。该区域为两个选项卡共享。

Setup Environment 具体做什么

1. 从一个基础 Python 创建专用虚拟环境(venv)。若 “Base Python” 留空,默认使用当前 Dragonfly 自带的 `Python_env\python.exe`(它是完整 CPython,自带 venv 与 pip),因此通常无需另装 Python;也可指定任意 CPython 3.9+ 的路径,或形如 py -3.11 的启动器。

2. 安装 torch 2.5.1(cu121) 与匹配的 torchvision(索引地址来自 “Torch CUDA wheel index” 字段,默认 https://download.pytorch.org/whl/cu121)。

3. 用 git 克隆 `facebookresearch/sam-3d-objects` 仓库,并执行其可编辑安装 .[dev]、.[p3d]、.[inference](含 PyTorch3D 与 kaolin)。

4. 安装 Open3D + plyfile + huggingface_hub 命令行工具。

5. 在填写了 HF token 的前提下,下载受限(gated)的 facebook/sam-3d-objects 模型权重到 checkpoints/hf 目录。

6. 运行一次冒烟自检(检查 torch/CUDA、Open3D、sam-3d-objects 是否可导入)。

下载体积与前置条件

• 联网:必须。整个环境是数 GB 级下载(仅 torch cu121 约 2.5 GB,再加模型权重)。
• NVIDIA GPU:必须。SAM 3D 推断需要 CUDA(cu121),CPU 无法运行。
• HuggingFace 账号与令牌:模型是受限模型。首次下载前须先在 huggingface.co 注册免费账号,在 https://huggingface.co/facebook/sam-3d-objects 页面接受许可,在 Settings ▸ Access Tokens 创建一个令牌,并粘贴到面板的 HF token 字段。若不填令牌,Setup 会跳过权重下载(其余组件照常安装),之后 Photo 选项卡会提示 “checkpoints not found(找不到权重)”。
• Git:Setup 需要系统 PATH 中有 git 才能克隆仓库;若没有,请先安装 Git,或在面板中直接指向一个已有的仓库检出目录。

环境安装到哪些路径

由于环境位于代码目录之外,重新安装插件代码不会触碰已搭好的环境,可放心更新。

Setup 失败时的替代方案(常见于 Windows)

PyTorch3D 与 kaolin 在裸 Windows venv 上以 pip 方式安装是出了名的脆弱。可靠的替代路线是按上游项目文档使用 conda / mamba 手动搭建环境,然后跳过 Setup 按钮,直接在面板的 Environment 区域手工填写三个路径即可:

• venv python → 该 conda 环境的 python.exe
• sam-3d-objects repo → 你的仓库检出目录
• checkpoints dir → 包含 pipeline.yaml 的文件夹(通常是 …/checkpoints/hf)

实际运行只依赖上述三个路径(venv python、仓库、权重目录)。Setup 按钮只是一个便利工具,并非必需——你完全可以手动搭好环境后让插件指向它。

5. 界面说明

面板左侧是 Environment 区域 + 两个选项卡,右侧是进度条、结果提示与日志。左上角有一段蓝色提示文字,概述插件功能与首次搭建环境较重这一事实。

Environment(环境,两个选项卡共享)

• venv python:虚拟环境的 python 路径。由 Setup 自动填写,或手动指向一个已有的 SAM 3D python。
• sam-3d-objects repo:模型仓库检出目录(带 “…” 浏览按钮)。
• checkpoints dir:权重目录,必须包含 pipeline.yaml(带浏览按钮)。
• Base Python (build):搭建环境时使用的基础 Python;留空=当前 Dragonfly 的 python,也可填路径或 py -3.11。
• Torch CUDA wheel index:torch 轮子索引地址,默认 https://download.pytorch.org/whl/cu121。
• HF token:HuggingFace 访问令牌(输入框以密码方式隐藏);下载受限模型前须先在网站接受许可。
• Setup Environment 按钮:一键搭建 venv + 克隆 + 下载权重。

选项卡 1:Photo to Point Cloud(照片 → 点云)

Image input(图像输入) 区域,二选一(SAM 3D 只接收一张图像,整幅画面即为目标物体):

• Active channel slice(活动通道切片) 单选:从当前会话的通道中取一张切片。可选 Red / primary(红/主)、Green(可选)、Blue(可选) 三个通道下拉框以合成彩色;Refresh 按钮刷新通道列表;Slice index(切片索引) 数值框选择第几张切片(范围随所选主通道的层数自动设定)。
• Image file(图像文件) 单选:直接选择磁盘上的一张图像(支持 png/jpg/jpeg/bmp/tif/tiff)。

Options(选项) 区域:Job root(作业根目录)、Seed(随机种子)、Max image size(最大图像边长,像素)。底部为蓝色的 Reconstruct Point Cloud(重建点云) 按钮。

选项卡 2:Point Cloud to Mesh(点云 → 网格)

• Point cloud input(点云输入):一个路径框(带浏览按钮),接收 .ply / .pcd / .xyz;由选项卡 1 自动填入,也可手动浏览。
• Reconstruction (Open3D)(重建参数):Method(方法) 下拉框(poisson / bpa / alpha)、Normal neighbours (k)(法线邻域数)、Poisson depth(泊松深度)、Poisson density trim (quantile)(泊松密度裁剪分位)、BPA radius x avg-spacing(球旋转半径倍数)、Alpha(0 = 自动)。
• 底部为蓝色的 Reconstruct Mesh(重建网格) 按钮。

右侧公共区域

• Cancel(取消) 按钮:请求中止当前运行。
• Open Job Folder(打开作业文件夹) 按钮:在资源管理器中打开最近一次的作业目录。
• 进度条、结果提示行(绿色显示成功摘要,如点数/顶点数/面数),以及 Log(日志) 文本框。

6. 使用步骤

工作流 A:从照片重建表面网格(端到端)

1. 首次使用先在 Environment 区域完成搭建(见第 4 章):填入 HF token 后点击 Setup Environment,等待其结束并自动回填 venv python / repo / checkpoints。

2. 切换到 Photo to Point Cloud 选项卡,选择图像来源:若用通道切片,点 Refresh 后在 Red / primary 里选主通道(可再选 Green/Blue 合成彩色),用 Slice index 选层;若用文件,选 Image file 并浏览一张照片。

3. 按需设置 Job root、Seed、Max image size。

4. 点击 Reconstruct Point Cloud。插件导出图像、生成整幅白色掩膜、运行 SAM 3D 推断(GPU,较耗时),提取干净点云并发布到场景;点云路径会自动填入选项卡 2。

5. 切换到 Point Cloud to Mesh 选项卡(点云路径已预填)。选择 Method,调整对应参数。

6. 点击 Reconstruct Mesh。Open3D 估计法线并重建表面,生成的网格作为 Dragonfly 网格对象导入,结果行显示顶点数与面数。

7. 在 Dragonfly 中查看网格;由于比例/朝向任意,通常需要重新缩放和摆正。

工作流 B:仅把已有点云转为网格(独立使用)

1. 直接打开 Point Cloud to Mesh 选项卡。

2. 在点云输入框浏览选择一个已有的 .ply / .pcd / .xyz 文件。

3. 选择方法与参数,点击 Reconstruct Mesh,即可导入网格。此流程不调用 SAM 3D,但仍需 venv 中的 Open3D 可用。

运行过程中可随时点击 Cancel 请求中止;点击 Open Job Folder 可查看中间文件(输入图像、splat.ply、points.ply、mesh.ply 等)。

7. 参数说明

环境参数

照片 → 点云(选项卡 1)

点云 → 网格(选项卡 2,Open3D)

8. 输出结果

在 Dragonfly 场景中,本插件会生成以下对象:

• 点云对象(选项卡 1 产出):命名形如 SAM3D_cloud_<时间>。由于 Dragonfly 没有一等的点云对象,点云通过原生网格加载器以“仅顶点的网格”方式发布(尽力而为);磁盘上的 points.ply 才是交给选项卡 2 的权威数据。
• 表面网格对象(选项卡 2 产出):命名形如 SAM3D_mesh_<时间>,是一个 Dragonfly 网格(FaceVertexMesh);结果行会报告顶点数与面数。

对应的磁盘文件保存在作业目录(Job root 下带时间戳的子文件夹)中,可通过 Open Job Folder 查看:

结果的比例与朝向是任意的。导入后请在 Dragonfly 中重新缩放/摆正,再进行可视化或测量。

9. 常见问题与故障排除

问:Photo 选项卡报 “SAM 3D checkpoints not found(找不到权重)”。

答:说明权重未下载或 checkpoints 目录未指向包含 pipeline.yaml 的文件夹。请先在 huggingface.co 接受 facebook/sam-3d-objects 的许可并创建令牌,把令牌填入 HF token 后重跑 Setup Environment;或手动下载权重后把 checkpoints dir 指向 …/checkpoints/hf。

问:Setup 过程中 PyTorch3D 或 kaolin 安装失败(WARNING)。

答:这在裸 Windows venv 上很常见。可靠做法是按上游文档用 conda/mamba 手动搭建环境,然后跳过 Setup 按钮,直接在面板里手工填写 venv python、sam-3d-objects repo、checkpoints dir 三个路径即可运行。

问:点击 Reconstruct 提示 “venv not set(未设置 venv)”。

答:venv python 字段为空。请先运行 Setup Environment,或手动指向一个已有的 SAM 3D python。

问:网格重建后 “no faces / reconstruction produced no faces(没有面片)”。

答:换一种 Method 重试,或提供更密的点云,或把 Alpha 调大;泊松法可适当降低 Poisson density trim 的分位以少裁面片。

问:通道下拉框是空的。

答:确认当前 Dragonfly 会话中已加载图像数据,然后点击 Refresh 重新扫描通道。

问:venv 创建失败,提示基础 Python 没有 venv 模块。

答:把 Base Python (build) 指向一个正常的 CPython 3.10–3.12(例如 C:\Python311\python.exe)后重试。

10. 注意事项与已知限制

• 必须有 NVIDIA GPU 与联网;首次搭建环境是数 GB 级下载,请预留时间与磁盘空间。
• 模型受限,必须先在 HuggingFace 接受许可并使用令牌,否则权重不会下载。
• SAM 3D 是单图三维推断,结果为形状估计,比例与朝向任意,不等同于计量级精确重建;导入后需重新缩放。
• 对不透明、纹理丰富、占满画面的物体效果最佳;透明、反光、镂空或纹理稀少的物体效果较差。
• 当前版本使用整幅图像掩膜(整幅画面即目标);尚未提供自动分割抠图。
• 点云在 Dragonfly 中以“仅顶点网格”尽力发布,磁盘上的 .ply 才是权威数据。
• PyTorch3D / kaolin 的 pip 安装在 Windows 上可能失败——已提供 conda 手动搭建的替代路线。

11. 参考资料

• SAM 3D Objects 项目:https://github.com/facebookresearch/sam-3d-objects
• 受限模型权重(需接受许可):https://huggingface.co/facebook/sam-3d-objects
• Open3D 项目主页:https://www.open3d.org
• PyTorch(CUDA 轮子索引):https://download.pytorch.org/whl/cu121
• Full Package 安装 / 启用 / 卸载说明:随包 README.md

Part II English Manual

Contents

1. Overview

2. Use Cases

3. Installation and Enabling

4. Environment and First-Run Setup

5. Interface Guide

6. Usage Steps

7. Parameter Reference

8. Outputs

9. FAQ and Troubleshooting

10. Notes and Known Limitations

11. References

1. Overview

Photo2Mesh: SAM3D is a Dragonfly plugin that reconstructs a 3D surface mesh of an object from a single photograph. The workflow is split into two consecutive stages that map to the panel's two tabs: the first stage uses Meta's SAM 3D Objects AI model to reconstruct a 3D point cloud from one image; the second stage uses Open3D to turn that point cloud into a triangle surface mesh, which is imported into the Dragonfly scene as a mesh object ready for visualization and measurement.

The plugin belongs to the Photo2Mesh family (alongside Meshroom, COLMAP and gsplat) and appears in the menu as Prototype Apps ▸ Photo2Mesh: SAM3D.... Unlike photogrammetry methods that require many views, SAM3D infers 3D shape from just one photo.

Engine and algorithms

• SAM 3D Objects (repo facebookresearch/sam-3d-objects): Meta's single-image-to-3D AI model. It takes one image plus a whole-image mask (the whole frame is the object) and emits a Gaussian-splat 3D point set. The plugin then extracts a clean XYZ (optionally RGB) point cloud points.ply.
• Open3D (surface reconstruction): after estimating and orienting normals, it reconstructs triangles with one of three methods — Poisson (default), Ball-Pivoting / BPA, or Alpha shapes — then cleans duplicate vertices and degenerate triangles to produce mesh.ply.
• The heavy compute stack is based on torch 2.5.1 (CUDA cu121) and depends on PyTorch3D and kaolin. It runs entirely in the plugin's own dedicated virtual environment, never inside Dragonfly's bundled Python.

Licensing

• SAM 3D Objects ships under the SAM License: royalty-free, commercial use allowed, no MAU or enterprise cap. Conditions: include a copy of the license with any redistribution, attribute in publications, and obey the trade-control / no-military terms.
• Open3D is licensed under Apache-2.0.
• The plugin is therefore safe to ship in a product.

2. Use Cases

This plugin is ideal when you have only a single photograph but want to quickly digitize a physical sample, part, or specimen into a 3D model. Typical uses include reverse engineering, documentation, teaching, and presentation in materials science, industrial inspection, and life-science labs.

• Works best on opaque, well-textured objects that fill the frame.
• The mesh has arbitrary scale and orientation — after import into Dragonfly you usually need to rescale and reorient it to match real-world dimensions.
• The second tab (Point Cloud to Mesh) also works standalone: it meshes any existing point-cloud file (.ply / .pcd / .xyz) without the AI reconstruction stage.

SAM3D performs single-image 3D inference, so the result is the model's best guess at the object's shape — not a calibrated, metrology-grade reconstruction. It excels at producing shapes quickly for visualization, not at supplying dimensional ground truth.

3. Installation and Enabling

This plugin is distributed with the Prototype Labs & Apps Full Package. Install it as follows:

1. Unzip the package to a short folder path (e.g. C:\PL\; avoid very deep or OneDrive-redirected paths).

2. Double-click `Install_FullPackage.bat`.

3. In the dialog, choose a core install mode (Fresh = clean install / Compatible = keep your own blocks & recipes; this choice only affects the Prototype Labs core, not any plugin's environment).

4. Tick Photo2Mesh: SAM3D in the app list. Note: all plugins are unchecked (off) by default and must be selected manually.

5. Click Install and wait for the console to finish.

6. Fully quit and restart Dragonfly (menus are only scanned at startup).

After the restart the plugin appears under Prototype Apps ▸ Photo2Mesh: SAM3D... (in the Photo2Mesh group). Clicking it opens a dockable (floating by default) panel.

Changing your choices later

The easiest way is from inside Dragonfly: open Developer ▸ Prototype Labs... ▸ Menu Item Manager and, in the "Prototype Apps (Full Package)" list at the bottom, find this plugin's checkbox — tick = deploy the menu item, untick = remove it. Restart Dragonfly to apply. Disabling never deletes a plugin's built environment; re-enabling is instant. You can also re-run the installer at any time; it remembers your previous choices as the new defaults.

Uninstall

Double-click `Uninstall_FullPackage.bat` to remove all Full-Package menu items and plugins. Uninstall keeps each plugin's built environment (venv / downloads); their paths are listed at the end so you can delete them manually to reclaim disk space.

4. Environment and First-Run Setup

SAM3D's heavy AI stack is not downloaded at install time. Instead, on first use, you build it by clicking Setup Environment in the panel's Environment box. This box is shared by both tabs.

What Setup Environment does

1. Creates a dedicated virtual environment (venv) from a base Python. If "Base Python" is blank, it defaults to the current Dragonfly's `Python_env\python.exe` (a full CPython with venv and pip), so usually no separate Python install is needed; you may also specify any CPython 3.9+ path or a launcher like py -3.11.

2. Installs torch 2.5.1 (cu121) and a matching torchvision (index from the "Torch CUDA wheel index" field, default https://download.pytorch.org/whl/cu121).

3. Clones the `facebookresearch/sam-3d-objects` repo with git and runs its editable installs .[dev], .[p3d], .[inference] (including PyTorch3D and kaolin).

4. Installs Open3D + plyfile + huggingface_hub CLI.

5. If an HF token is provided, downloads the gated facebook/sam-3d-objects checkpoints into the checkpoints/hf folder.

6. Runs a smoke test (checks torch/CUDA, Open3D, and that sam-3d-objects imports).

Download size and prerequisites

• Internet: required. The environment is a multi-GB download (torch cu121 alone is ~2.5 GB, plus the model checkpoints).
• NVIDIA GPU: required. SAM 3D inference needs CUDA (cu121); CPU cannot run it.
• HuggingFace account and token: the model is gated. Before the first download, create a free account at huggingface.co, accept the license at https://huggingface.co/facebook/sam-3d-objects, create a token (Settings ▸ Access Tokens), and paste it into the HF token field. Without a token, Setup skips the checkpoint download (everything else still installs) and the Photo tab later reports "checkpoints not found".
• Git: Setup needs git on PATH to clone the repo; if you don't have it, install Git or point the panel at an existing checkout.

Where the environment is installed

Because the environment lives outside the code directory, reinstalling the plugin code never touches the built environment, so updates are safe.

If Setup fails (common on Windows)

PyTorch3D and kaolin are notoriously hard to pip-install into a bare Windows venv. The reliable fallback is the project's documented conda / mamba setup; then skip the Setup button and simply fill three fields in the Environment box by hand:

• venv python → that conda env's python.exe
• sam-3d-objects repo → your repo checkout
• checkpoints dir → the folder containing pipeline.yaml (usually …/checkpoints/hf)

The run only needs those three paths (venv python, repo, checkpoints dir). The Setup button is a convenience, not a requirement — you can build the environment manually and just point the plugin at it.

5. Interface Guide

The panel's left side holds the Environment box plus two tabs; the right side holds the progress bar, result label, and log. A blue note at the top left summarizes what the plugin does and warns that the first Setup is heavy.

Environment (shared by both tabs)

• venv python: path to the virtual environment's python. Filled by Setup, or point it at an existing SAM 3D python.
• sam-3d-objects repo: the model repo checkout directory (with a "…" browse button).
• checkpoints dir: the weights directory; must contain pipeline.yaml (with browse button).
• Base Python (build): the Python used to build the env; blank = this Dragonfly's python, or a path, or py -3.11.
• Torch CUDA wheel index: the torch wheel index URL, default https://download.pytorch.org/whl/cu121.
• HF token: HuggingFace access token (masked as a password); accept the license on the site before downloading the gated model.
• Setup Environment button: one-click build of venv + clone + checkpoints.

Tab 1: Photo to Point Cloud

Image input area, choose one (SAM 3D takes a single image; the whole frame is the object):

• Active channel slice radio: take one slice from a channel in the current session. Optional Red / primary, Green (optional) and Blue (optional) dropdowns compose colour; Refresh rescans channels; Slice index spinner picks the slice (its range is set automatically from the chosen primary channel's depth).
• Image file radio: pick an image on disk directly (png/jpg/jpeg/bmp/tif/tiff).

Options area: Job root, Seed, and Max image size (px). At the bottom is the blue Reconstruct Point Cloud button.

Tab 2: Point Cloud to Mesh

• Point cloud input: a path field (with browse button) accepting .ply / .pcd / .xyz; auto-filled from Tab 1, or browse.
• Reconstruction (Open3D): Method dropdown (poisson / bpa / alpha), Normal neighbours (k), Poisson depth, Poisson density trim (quantile), BPA radius x avg-spacing, and Alpha (0 = auto).
• At the bottom is the blue Reconstruct Mesh button.

Right-side common area

• Cancel button: requests cancellation of the current run.
• Open Job Folder button: opens the most recent job directory in Explorer.
• Progress bar, a result label (green success summary such as points / vertices / faces), and a Log text box.

6. Usage Steps

Workflow A: photo to surface mesh (end to end)

1. On first use, build the environment in the Environment box (see Chapter 4): enter your HF token and click Setup Environment, then wait for it to finish and auto-fill venv python / repo / checkpoints.

2. Switch to the Photo to Point Cloud tab and choose the image source: for a channel slice, click Refresh, pick the primary channel in Red / primary (optionally Green/Blue for colour), and choose the layer with Slice index; for a file, select Image file and browse to a photo.

3. Set Job root, Seed, and Max image size as needed.

4. Click Reconstruct Point Cloud. The plugin exports the image, generates a full white mask, runs SAM 3D inference (GPU, may take a while), extracts a clean point cloud and publishes it to the scene; the cloud path is auto-filled into Tab 2.

5. Switch to the Point Cloud to Mesh tab (the cloud path is pre-filled). Choose a Method and adjust its parameters.

6. Click Reconstruct Mesh. Open3D estimates normals and reconstructs the surface; the resulting mesh is imported as a Dragonfly mesh object and the result label reports vertex and face counts.

7. Inspect the mesh in Dragonfly; because scale/orientation are arbitrary, you usually need to rescale and reorient it.

Workflow B: mesh an existing point cloud (standalone)

1. Open the Point Cloud to Mesh tab directly.

2. Browse and select an existing .ply / .pcd / .xyz file in the point-cloud input.

3. Choose the method and parameters, then click Reconstruct Mesh to import the mesh. This path does not invoke SAM 3D, but still needs Open3D available in the venv.

You can click Cancel at any time to request a stop; click Open Job Folder to inspect intermediate files (input image, splat.ply, points.ply, mesh.ply, etc.).

7. Parameter Reference

Environment parameters

Photo to Point Cloud (Tab 1)

Point Cloud to Mesh (Tab 2, Open3D)

8. Outputs

In the Dragonfly scene, the plugin creates the following objects:

• Point cloud object (from Tab 1): named like SAM3D_cloud_<time>. Because Dragonfly has no first-class point-cloud object, the cloud is published as a vertices-only mesh through the native mesh loader (best-effort); the on-disk points.ply is the authoritative hand-off to Tab 2.
• Surface mesh object (from Tab 2): named like SAM3D_mesh_<time>, a Dragonfly mesh (FaceVertexMesh); the result label reports vertex and face counts.

The corresponding disk files are saved in the job directory (a timestamped subfolder under Job root), viewable via Open Job Folder:

Results have arbitrary scale and orientation. After import, rescale/reorient in Dragonfly before visualization or measurement.

9. FAQ and Troubleshooting

Q: The Photo tab reports "SAM 3D checkpoints not found".

A: The weights were not downloaded, or the checkpoints dir does not point at a folder containing pipeline.yaml. Accept the license for facebook/sam-3d-objects on huggingface.co, create a token, paste it into HF token, and re-run Setup Environment; or download the weights manually and point checkpoints dir at …/checkpoints/hf.

Q: PyTorch3D or kaolin fails to install during Setup (WARNING).

A: This is common on a bare Windows venv. The reliable route is to build the environment manually with conda/mamba per the upstream docs, then skip the Setup button and fill venv python, sam-3d-objects repo, and checkpoints dir by hand.

Q: Clicking Reconstruct says "venv not set".

A: The venv python field is empty. Run Setup Environment first, or point it at an existing SAM 3D python.

Q: After meshing I get "no faces / reconstruction produced no faces".

A: Try a different Method, provide a denser point cloud, or increase Alpha; for Poisson, lower the Poisson density trim quantile so fewer faces are trimmed.

Q: The channel dropdowns are empty.

A: Make sure image data is loaded in the current Dragonfly session, then click Refresh to rescan channels.

Q: venv creation fails, saying the base Python has no venv module.

A: Point Base Python (build) at a normal CPython 3.10-3.12 (e.g. C:\Python311\python.exe) and retry.

10. Notes and Known Limitations

• An NVIDIA GPU and internet are required; the first environment build is a multi-GB download, so allow time and disk space.
• The model is gated — you must accept the license on HuggingFace and use a token, otherwise the checkpoints are not downloaded.
• SAM 3D performs single-image 3D inference; the result is a shape estimate with arbitrary scale and orientation, not a metrology-grade reconstruction; rescale after import.
• Works best on opaque, well-textured objects that fill the frame; transparent, reflective, hollow, or texture-poor objects give poorer results.
• The current version uses a whole-image mask (the whole frame is the object); automatic segmentation/cut-out is not provided.
• The point cloud is published in Dragonfly as a best-effort vertices-only mesh; the on-disk .ply is the authoritative data.
• pip installation of PyTorch3D / kaolin may fail on Windows — a manual conda setup is provided as a fallback.

11. References

• SAM 3D Objects project: https://github.com/facebookresearch/sam-3d-objects
• Gated model weights (license acceptance required): https://huggingface.co/facebook/sam-3d-objects
• Open3D project home: https://www.open3d.org
• PyTorch (CUDA wheel index): https://download.pytorch.org/whl/cu121
• Full Package install / enable / uninstall guide: the bundled README.md