Potential-Guided Particle Steering for Negation-Constrained Dexterous Grasping
Under review · 2026
Chung-Ang University
* Equal contribution · † Corresponding author
Language-driven dexterous grasp models perform well when an instruction says where to grasp, but fail systematically when it also says where not to. Training corpora such as DexGYSNet contain virtually no avoidance instructions, and because every part named in training marks a contact target, a forbidden part is easily read as one more region to grasp rather than one to avoid. We handle negation entirely at inference time, with no negation-specific training examples. Classifier-free guidance (CFG) steers contacts away from the forbidden region while Sequential Monte Carlo (SMC) prunes particles already headed for it, and a frozen 3D part-grounding model localizes that region from language alone. On NegGrasp, the benchmark we construct for this setting, our method cuts the violation rate of the strongest baseline from 57.9% to 17.2% while improving both constraint-aware and physical success.
A flow-matching DiT denoises grasp pose and contact coordinates together, so every step carries a spatially checkable estimate of the finished grasp that the sampler can still revise. All steering acts on that estimate, and the trained generator is never touched.
An LLM splits the instruction into a positive rewrite (prompt A) and a prompt naming the forbidden part (prompt B). Prompt B is phrased affirmatively on purpose, since that is exactly the direction a negation-blind model drifts in. Find3D then labels the point cloud to yield the forbidden region ℱ.
A one-step look-ahead predicts the finished contacts. Guidance fires only when they land near ℱ, pushing the contact velocity along the component of vA−vB orthogonal to vA. The push goes sideways off the region and leaves the positive target intact.
16 particles run in parallel, resampled by a potential that peaks when the look-ahead is clear of ℱ and already close to the object. A running maximum keeps credit for earlier safe checkpoints.
Deterministic steps keep resampled clones identical forever. A marginal-preserving stochastic update separates them within a single step, leaving the distribution each particle targets unchanged.
Every row shares the same decomposition-and-grounding scaffolding and the same particle budget, so the comparison isolates the generator and the steering. Macro averages over the 27 (category, forbidden part) cells of the NegGrasp test split.
| Method | Instr. | N | Viol. (%) | Cont. (%) | Clear. (mm) | CSR (%) | TSR (%) |
|---|---|---|---|---|---|---|---|
| DexGYS | mixed | 1 | 55.3 | 28.2 | 10.5 | 43.6 | 27.6 |
| DexGYS + BoN | decomp. | 16 | 36.2 | 14.9 | 14.0 | 56.0 | 36.0 |
| DextER | mixed | 1 | 57.9 | 33.4 | 12.5 | 34.1 | 22.4 |
| DextER + BoN | decomp. | 16 | 29.9 | 14.4 | 22.3 | 51.5 | 31.0 |
| Ours (BoN) | decomp. | 16 | 19.6 | 8.5 | 19.8 | 58.6 | 38.3 |
| Ours (BoN + Steering) | decomp. | 16 | 17.2 | 6.6 | 21.1 | 61.5 | 38.7 |
The forbidden part is rendered in red. Baselines repeatedly contact it, and often seize it outright. Our grasps stay clear in every case shown while still reaching the instructed part, and they often differ from the ground-truth reference, so the sampler is finding its own valid solution rather than reproducing GT.
@inproceedings{kim2026potential,
title = {Potential-Guided Particle Steering for
Negation-Constrained Dexterous Grasping},
author = {Kim, Geonho and Kim, SooGon and Lee, Jongmin},
booktitle = {Under review},
year = {2026}
}