What if the safest way to generate a labeled object image is not to redraw the object at all? This paper paints outside the box, then filters the synthetic examples before training. Visual grounding aims to localize the image regions based on a textual query.

Given the difficulty of large-scale data curation, POBF investigates how to effectively learn visual grounding under data-scarce settings. POBF synthesizes images by inpainting outside the box, tackling a label misalignment issue encountered in previous works. POBF also uses a filtering scheme that combines a hardness score and an overfitting score, balanced by a penalty term.

Visual grounding is a fundamental task in vision-language understanding, and its objective is to locate a specific image region based on a free-form textual query. Constructing visual grounding datasets with dense region-text annotations is both time-intensive and costly, which hinders progress in the field.

The problem is learning visual grounding under data-scarce settings, where only few real training data are accessible. Off-the-shelf generative models can synthesize images across diverse domains, but they struggle to directly generate data with region-text labels because they are not trained with dense annotations.

A naive approach is inpainting the object within the bounding box, but that can result in label misalignment issues. The generated object may not align spatially with the box or may fail to appear entirely; the synthetic right cow misaligns with the box, and the batter disappears.

The paint-outside-the-box strategy preserves the content within the box while generating the background outside, resulting in higher-quality images that align well with the original annotations. In the top row, the red box around the cow stays centered on a plausible cow in the POBF column, while the corresponding generated examples show either a distorted cow, a shifted subject, or a heavily altered scene.

In the bottom row, POBF preserves the batter inside the box, whereas Gen2Det leaves the boxed region empty and GeoDiffusion produces a visibly unrelated, artifact-heavy image. Training neural networks with synthetic data presents another challenge: not all samples contribute positively to model performance.

The filtering scheme consists of three components: a hardness score, an overfitting score, and a penalty term. The hardness score prioritizes the easiest samples because it is challenging to model outliers when basic patterns are not adequately captured in the data-scarce regime.

The overfitting score penalizes samples that exacerbate overfitting to unintended features outside the box, while the penalty term manages the trade-off between the two scores. The selected synthetic data are combined with the real data to train the visual grounding model.

Visual grounding identifies the target region within an image that a natural language description refers to. The target region is typically represented as a bounding box, with center coordinates, width, and height. Data-scarce visual grounding means learning when only a very limited amount of annotated real training data is available, such as one percent of the dataset.

POBF is a model-agnostic framework for data-scarce visual grounding, and it consists of four stages. The first stage synthesizes images and captions using image inpainting and captioning techniques, with an image generator and captioner pretrained on large-scale image-caption pairs.

Next, a teacher model is trained using the limited real data, and a filtering scheme selects effective synthetic images. The teacher network assigns a hardness score, an overfitting score, and a penalty term to each synthetic image, after which high-scoring synthetic images are combined with real data to train the student model.

In the top panel, one real bus example feeds both the captioner and image generator, producing a shared caption alongside several related inpainted images. The lower panels then show the teacher trained from real data, its scores filtering the generated-data pool, and the retained data being combined with real data for student training.

Image inpainting regenerates masked regions within an image based on a given prompt while keeping the unmasked areas intact. To address label misalignment, POBF paints outside the bounding box instead. The captioning model generates a caption, and the bounding box, image, and caption are input into the inpainting model; the red bus within the bounding box remains unchanged and strictly aligns with the ground truth bounding box.

The synthetic image can be paired with the original text and bounding box to form a new training sample, and generating multiple synthetic images per real sample increases the training dataset by a factor of that number. A high strength parameter is used to encourage data diversity; strength controls the noise added to the base image and influences similarity between the generated output and the original image.

Not all synthetic samples contribute positively to model performance, so POBF uses a filtering scheme to select the most effective images for training. The limited real training dataset first trains a teacher model, and the teacher evaluates synthetic images using two scores and one penalty term.

The score calculations are illustrated in Figure 3 for a real data sample and the set of images generated from it. The top input retains the red bus, so the teacher’s predicted box is compared with the ground-truth box to produce S one. In the middle example, the bus is obscured and the resulting prediction feeds S two, while the bottom image has no ground-truth label and instead produces the penalty term P.

The first three datasets used in this evaluation are RefCOCO, RefCOCO+, and RefCOCOg, and all three are derived from MSCOCO 2014. The evaluation also includes the ReferItGame dataset, which was collected through an online two-player game, as described in the benchmark setup.

Performance is reported on the testA and testB splits for RefCOCO and RefCOCO+, the umd-test splits for RefCOCOg, and the test set for ReferItGame. For each dataset, randomly sampled subsets comprising one percent of the entire dataset simulate data-scarce scenarios.

Models using synthetic data from various baselines consistently outperform training solely on real data, with at least a one point nine eight percent improvement. POBF achieves an average performance gain of five point eight three percent over the real-data-only baseline.

POBF surpasses GeoDiffusion by an average margin of three point eight five percent, and the text attributes GeoDiffusion's limitation to fine-tuning over a fixed set of objects within a specific domain. POBF exceeds the strongest image-editing baseline, X-Paste, by two point two nine percent; X-Paste produces unnatural images, while Gen2Det suffers from label misalignment issues.

The improvement on ReferIt is relatively modest, which is hypothesized to stem from small objects and the visible artifacts that may arise when very large background regions must be generated. In the lower section, the three-sample averages show POBF at thirty-three point five four, compared with twenty-eight point two six for real-only training.

The accompanying deviations indicate variation across samples, ranging from plus or minus zero point three zero on ReferIt to plus or minus one point four nine on RefCOCO TestA. Each individual filtering component produces a gain over the variant without filtering: one point two three percent, zero point seven one percent, and one point one four percent.

The variant that excludes only the penalty term reaches an accuracy of thirty-three point zero eight percent, surpassing all configurations that use a single filtering component. The full filtering scheme yields the highest accuracy across all datasets, reaching an average accuracy of thirty-three point six one percent.

That result is a one point nine six percent improvement over the baseline without filtering, and the combination of synthetic data and filtering yields the best results. Table 3 separates the effects of synthetic data from the filtering scheme.

Starting from an average of twenty-seven point seven eight, adding synthetic images gives thirty-one point three one, while adding synthetic texts gives twenty-nine point two nine. In the filtering ablation, the individual components S one, S two, and P produce average gains of one point two three, zero point seven one, and one point one four percent over the unfiltered variant, showing that each contributes to selecting useful synthetic training data.

POBF is a framework for effectively learning visual grounding in data-scarce scenarios, using generative models pretrained solely on image-caption pairs to enable high-quality synthetic data generation. By avoiding specialized networks trained on extensive dense annotations, POBF gains flexibility and generalizability.

Its paint-outside-the-box strategy addresses label misalignment issues, and its filtering scheme is tailored for inpainted data. The filtering scheme integrates a hardness score, an overfitting score, and a penalty term. Table five tests whether POBF remains effective when its pretrained image generator or captioner is replaced.

It reports results on RefCOCO+ TestA and TestB, RefCOCOg Test, and an average, including the default POBF configuration, alternative captioners LLAVA and ViT-GPT2, and alternative generators Stable Diffusion 2 and Dreamshaper. This matters because it examines whether the filtering approach can integrate with multiple state-of-the-art generative models rather than depending on one fixed configuration.

Table six compares the real-data baseline with POBF across RefCOCO-plus and RefCOCOg at two training-data sizes. With zero point five percent of the training data, the reported averages are thirteen point ninety-seven for Real and nineteen point eighteen for POBF, a gain of five point two two; with two percent, they are thirty point forty-five and thirty-five point fifty-one, a gain of five point zero six.

This matters because the framework is evaluated under severely limited data conditions. Table 7 examines POBF across three architectures with different parameter counts: TransVG at one hundred forty-nine million, Dynamic-MDTRE at one hundred sixty million, and QRNet at two hundred seventy-three million trainable parameters.

For each architecture, the table reports RefCOCO+ TestA and TestB, RefCOCOg Test, and the average accuracy for real-data training versus POBF. The listed average gains are five point seventy-nine, seven point zero three, and four point forty-one, respectively, showing improvements across all tested architectures.

POBF combines paint-outside-the-box generation with filtering based on hardness, overfitting, and a penalty term. In four benchmarks, it improves over real-data-only training by an average of 5.83%.