Prompt Highlighter: Interactive Control for Multi-Modal LLMs
Yuechen Zhang1

Shengju Qian1
Bohao Peng1
1
The Chinese University of Hong Kong

Shu Liu2
2
SmartMore

Jiaya Jia1,2

arXiv:2312.04302v1 [cs.CV] 7 Dec 2023

https://julianjuaner.github.io/projects/PromptHighlighter/
Language
Prompt

I would like to control the degree of compactness

Vision &
Language
Prompt

User: Write a summary of A Mid-Summer Nights'
Dream. Make it compact.

Vicuna Normal Inference

256 tokens

MAKE IT COMPACT

The image features a man wearing a red and green soccer
jersey, smiling and posing for a picture. He is likely a soccer
player, as he is wearing a sports uniform and is standing on a
field. The man appears to be enjoying himself…

Unstable
234 tokens

→ super compact

141 tokens

+ Compactness is important. 280 tokens
Prompt Highlighter: Control Partial Prompts

User: Please describe this image.

LLaVA Normal Inference

A Midsummer Night's Dream is a comedy play written by William
Shakespeare. The play follows the events that take place…
Prompt Engineering

I want to know more about the man
and the jersey...

Prompt Highlighter: Inference with Highlighted Region
121 tokens

The image features a smiling man, who is none other than
the famous soccer player, Cristiano Ronaldo. He is wearing
a red and green jersey and is standing in front of a crowd…

In A Mid-Summer Nights' Dream, a group of young lovers, including
Hermia, Lysander, Demetrius, and Helena, become…
65 tokens

The man in the image is wearing a red, green, and yellow shirt,
which is the national soccer team uniform of Portugal. He is
smiling and appears to be enjoying the game. The shirt has a
green and red stripe, and the number 7 is displayed on it …

In A Mid-Summer Nights' Dream, four young lovers run away from an
arranged marriage and hide in the forest where they are controlled by
a mischievous fairy. A troupe of actors perform a play within the play
that resolves the conflicts and the lovers are reunited. </s>

Figure 1. Prompt Highlighter facilitates token-level user interactions for customized generation, compatible with both LLMs and VLMs.
Compared with vanilla inference and prompt engineering, the context-highlighted inference provided by our method offers controllable
generations and produces customized results. Outputs correlated with the highlighted parts are underlined.

Abstract

and VLMs, achieving impressive customized generation results without training. Experiments confirm its effectiveness
in focusing on input contexts and generating reliable content. Without tuning on LLaVA-v1.5, our method secured
69.5 in the MMBench test and 1552.5 in MME-perception.
Code is available at: https://github.com/dvlabresearch/Prompt-Highlighter/.

This study targets a critical aspect of multi-modal LLMs’
(LLMs&VLMs) inference: explicit controllable text generation. Multi-modal LLMs empower multi-modality understanding with the capability of semantic generation yet
bring less explainability and heavier reliance on prompt
contents due to their autoregressive generative nature.
While manipulating prompt formats could improve outputs,
designing specific and precise prompts per task can be challenging and ineffective. To tackle this issue, we introduce
a novel inference method, Prompt Highlighter, which enables users to highlight specific prompt spans to interactively control the focus during generation. Motivated by
the classifier-free diffusion guidance, we form regular and
unconditional context pairs based on highlighted tokens,
demonstrating that the autoregressive generation in models can be guided in a classifier-free way. Notably, we find
that, during inference, guiding the models with highlighted
tokens through the attention weights leads to more desired
outputs. Our approach is compatible with current LLMs

1. Introduction
Large Language Models (LLMs) have driven significant
progress in a multitude of natural language processing
tasks [1–9]. Further advancements have been achieved
by extending these models to handle vision-language
tasks [10–15] through visual-language alignment and instruction tuning. These efforts have led to the development
of Vision Language Models (VLMs), which can generate
text based on multi-modal inputs. Due to its autoregressive nature, the typical generation process in LLMs and
VLMs (multi-modal LLMs) is primarily conditioned on input contexts. Prompt engineering [16–19] has emerged
as a common interaction mechanism between humans and
1

language models, where diverse formats and content of
prompts are employed to steer the generation towards desired outcomes. However, prompt engineering often relies
on empirical intuition and requires careful design of the
context, making it less accessible for non-experts. As illustrated in the left part of Fig. 1, even the meticulously
crafted prompts, which convey the concept of ‘compactness’ clearly, can lead to unpredictable outputs that fail to
meet the requirements.
Instead of manipulating prompt-level contexts (i.e.,
prompt engineering) to control LMs’ generation process,
we propose a novel inference approach, Prompt Highlighter,
that enables token-level user interactions for personalized
generations. Our method allows users to interact with multimodal LLMs in a manner analogous to applying a highlighter tool on the input context in the text editor, enabling
users to emphasize desired parts by highlighting them.
This highlighting mechanism is achieved by constructing a regular and unconditional input context pair with different textual embeddings in the highlighted tokens. Subsequently, we can adjust the model’s focus on the highlighted
components by employing the classifier-free guidance [20–
22] on predicted token probabilities. Moreover, by probing
cross-token attention maps, we discover a robust correlation between attention scores and the semantic significance
of tokens. This suggests that, in the autoregressive generation process of language models, the semantic relationship
between tokens can be represented to a certain extent by
their attention scores. Building on this insight, we introduce an attention activation strategy that adjusts the attention weights associated with a highlighted part. Specifically,
Prompt Highlighter employs an adjusted attention mask to
reweight corresponding attention scores, enabling a more
focused generation on highlighted parts.
As illustrated in Fig. 1, compared to vanilla inference,
our highlighted inference can guide the generation process
to produce controllable results that align more closely with
user needs. Prompt Highlighter is compatible with mainstream transformer-based multi-modal LLMs. This compatibility encompasses VLMs that use precise patch-wise
visual token mapping, such as LLaVA [10, 23, 24], as well
as methods that employ implicit query-ba qw[]
sed visual token mapping, like those based on QFormer [11, 13–15]. This novel interaction paradigm with
highlighted sections during the generation process goes beyond what prompt engineering can offer.
We further demonstrate the effectiveness of Prompt
Highlighter by evaluating it using comprehensive multimodal benchmarks. We verify that directly highlighting
the full image context in VLMs can significantly improve
the quality of generated image captions [25] and questionanswering results. Specifically, our method can effectively
mitigate the model’s propensity to hallucinate by guiding

its focus toward reliable contexts, thereby enhancing overall performance. Notably, without additional training, our
method improves the performance of the baseline LLaVAv1.5, securing 2nd place in both MMBench [26] and MMEperception [27] leaderboards.
Our contributions can be summarized as follows: (1)
We pioneer the exploration of fine-grained human-model
interactions in multi-modal LLMs, proposing a plug-andplay pipeline that enables token-level user interactions for
controllable generation. (2) We conduct extensive experiments on comprehensive benchmarks, demonstrating that
our method significantly enhances the overall performance.

2. Related Works
2.1. Multi-Modal LLMs
Recent Large Language Models (LLMs) [1, 7–9, 28–30]
play a significant role in natural language processing tasks,
particularly in language generation and question answering.
Building upon these pre-trained language models, VisionLanguage Models (VLMs) [10, 11, 13–15, 31] further introduce the alignment between vision and language modalities by leveraging extensive training on image-caption pairs
or image-question conversations. There are two prevalent
methods for aligning vision and verbiage modalities. The
first method, exemplified by LLaVA [10], directly maps image patches to tokens using a projector, establishing a oneto-one correspondence. The second method, represented by
models like BLIP2 [13, 32], employs a Query Transformer
(Q-Former) after getting image features to establish a nonuniform patch-token mapping. These methods use learnable
queries to get compressed image features, yielding visual
tokens rich with semantic information.

2.2. Interactions with Multi-Modal LLMs
Prompt engineering and interactions. Based on the autoregressive property of LLMs, users aim to control the generation results by modifying the input contexts. This largely
determines the test-time interactions with LLMs, primarily executed through prompt engineering. Representative
methods such as CoT [17] introduce demonstrations in the
context to enhance reasoning ability. Other multi-branch
designs like ToT and GoT [16, 18, 19, 33, 34] have been
proposed for rich and reliable context generation and selfchecking. Aside from prompt engineering, human-model
interactions have not been extensively explored in VLMs.
Methods like Kosmos-2 [31], LLaVAInteractive [35], and
LISA [36] enable grounding perception tasks such as detection, segmentation, and image editing through interaction
with the language model. These task-oriented interactions
require additional data collection and task-specific tunning.
In contrast, Prompt Highlighter is plug-and-play for general
text generation in pre-trained models.
Classifier-free guidance and controllable generation.
2

update next token embedding $!"# = &((!"#)

!

tokenize,
embedding &

❄

Language Model
❄ "!

normal context !
unconditional context !̅

Multi-modal input * +
highlight mask +

guidance

+
update

CFG re-weight
(!"#
logit predictions

token-wise embeddings

highlighted attention mask

self-attention activation

Figure 2. An abstract pipeline of Prompt Highlighter. Users can control the focus of generation by marking out specific image regions or
text spans. Then a token-level mask m is created to guide the language model’s inference.

Classifier-Free Guidance (CFG) [20] enables a control on
Diffusion Models’ generation process without a conventional classifier. Specifically, CFG’s step-wise sampling allows users to employ a negative prompt within the unconditional branch, effectively guiding the generation away from
harmful distributions. This approach has been extended to
language models by LLM-CFG [21], allowing a controllable text generation and improved performance. However,
LLM-CFG still requires a pair-wise prompt design and does
not support partial token-level reweighting within the context, which is vital for controlling VLM’s generation. Besides, methods in Diffusion Models [37, 38] achieve finegrained control over image generation using text prompts
by emphasizing areas within cross-attention maps. Despite
these advancements, fine-grained control over autoregressive generation in LLMs and VLMs is still challenging.
Prompt Highlighter is proposed to tackle this issue.

in which ϵt is the noise prediction conditioned on the previous output xt+1 and the text condition c. LLM-CFG [21]
extended this property to autoregressive language models.
Given a sequence of N tokens x = {x1 , . . . , xN }, the likelihood of predicting
the entire sequence can be expressed as
QN
PΘ (x) = i PΘ (xi |xj<i ). The model samples each subsequent token from the conditional probability distribution.
Based on Eq. (1), the CFG sampling on the language model
can be denoted as

3. Prompt Highlighter

Pxi = γ log PΘ (xi |xj<i , c) − (γ − 1) log PΘ (xi |xj<i ).
(4)
The formulation in Eqs. (3) and (4) offers a paradigm
for controllable generation in LLMs [21], with the guidance
strength γ controls the degree of generation focus. Notably,
the effectiveness of this guidance depends on the careful design of the conditional prompt c, which should be naturally
formed as a complete phrase or sentence to retain its semantic meaning. Prompt Highlighter extends CFG control in
language models in a more generalized manner. The user’s
selection on the context x is converted into a token-level
binary highlight mask m = {m1 , . . . , mN }. We define
mi = 1 if the i-th token xi is highlighted, and mi = 0 otherwise. This mask constructs a two-branch condition: the
normal and the unconditional contexts. The normal context
operates in the same manner as in vanilla inference. Meanwhile, the unconditional context s̄ is derived from the normal context s = {s1 , . . . , sN } within the textual embedding
space through a token-wise scaling,

N
Y
PΘ (xi |xj<i , c)γ
PΘ (x|c)γ
∝
.
P̂Θ (x|c) ∝
PΘ (x)γ−1
PΘ (xi |xj<i )γ−1
i=1

Similar to the transaction from Eq. (1) to Eq. (2), the likelihood in LLM is represented as the next-token classification probability. Thus next token’s logit prediction Pxi =
log P̂Θ (xi |xj<i , c) is

An overview of Prompt Highlighter is presented in Fig. 2.
Given a pre-trained generative model PΘ , we first extract
the input tokens from the text and the input image, forming the prompt context x. Subsequently, by marking out
specific image regions or text spans, user create a tokenlevel binary mask m to highlight specific tokens. Prompt
Highlighter then generates the output sequence y using the
two-branch condition based on m autoregressively. The following section will delve into the specifics of our method.

3.1. Token-Level Highlight Guidance
In conditioned Diffusion Models [39], given a noisy image
x and a class condition c, the model predicts the conditioned
step-wise sample P̂Θ (x|c) ∝ PΘ (x) · PΦ (c|x)γ . Here, PΦ
is a classifier, and γ is the guidance strength controlling the
weight of likelihood on c. Ho et al. [20] observed that guidance can be offered without a classifier. Applying the Bayes
rule, PΘ (c|x) ∝ PΘ (x|c)/PΘ (x), the sampling process of
the Classifier-Free Guidance (CFG) can be expressed as
P̂Θ (x|c) ∝ PΘ (x|c)γ /PΘ (x)γ−1 ,
log P̂Θ (ϵt |xt+1 , c) = γ log PΘ (ϵt |xt+1 , c)

(3)

s̄i = (α − 1)mi · f (xi ) + f (xi ),

(5)

(1)
where α is the scaling factor and f (·) is the token-toembedding function, i.e., si = f (xi ). We empirically set
a small rescale α (e.g., 0.01) that can ensure a normal inference while ignoring the highlighted part. Then, based on

(2)

− (γ − 1) log PΘ (ϵt |xt+1 ),
3

Generated tokens →

high

Input contexts →

Grammar

LaTeX

Compact

low

Vicuna Self-Attention

Q-Former Cross-Attention

Figure 3. Visualizing attention maps. Left: A segment of the attention map between the generated tokens and the input requirement
prompt: “. . . fix the grammar and keep LaTeX format, make it compact. . . ”. Some representative tokens are marked for reference. Right:
Query-based token mapping. This shows the attention score on 32 queries in the first cross-attention layer of the Q-Former.

the two-branch condition (s, s̄), we can define the i-th token
sampling process of the token-level highlight guidance as

precise control over the output. We reformulate the model
inference function in Eq. (6) to a mask-conditioned one
P̃Θ (xi |sj<i , m). In each of its self-attention layers, let ki
represent the i-th column vector of the query-key multiplied
attention score matrix in one attention head. The activated
attention score hi in the normal context branch is defined
as
hi = log(β) · mi + ki .
(7)

log P̂Θ (xi |sj<i ) =γ log PΘ (xi |sj<i )
− (γ − 1) log PΘ (x̄i |s̄j<i ).

(6)

Compared with Eq. (4), the additional conditional context
c is naturally incorporated as the difference between s and
s̄. This arrangement provides users with the flexibility to
control the in-line requirements. After the i-th token is predicted, the highlight mask m and contexts s, s̄ are updated
by appending mi = 0 and si = s̄i = f (xi ), respectively.
The generation process is terminated when the end token
</s> is predicted.

Then, the attention probability pi is calculated as
exp(hi )
β mi · exp(ki )
pi = PN
= PN
.
mj · exp(k )
j
j=1 exp(hj )
j=1 β

(8)

This mechanism defines the activation scaling factor as β.
For the unconditional branch, the attention score is deactivated by using a scaled negative mask in the inference
P̃Θ (x̄i |s̄j<i , −δm). Eq. (8) presents a SoftMax probability pi = softmax(hi ) on the activated scores, with a consistent β mi -times probability augmentation on highlighted
tokens. The attention activation operates under the assumption that users cannot highlight dominant ‘sink’ tokens as
explored in [40]. Consequently, the attention activation will
not catastrophically impair the model’s fundamental generative capabilities during the inference.

3.2. Attention Activation
The token-level highlight guidance anchors the generative
process with a token-wise logit reweighting. However, its
effectiveness may diminish when facing long and complex
input contexts with few highlighted tokens, making it difficult to distinguish s and s̄. To further investigate token-wise
correlations and their impact on generation results, we exclude sink tokens that dominate the attention score [40] and
visualize cross-token self-attention score maps during inference. For instance, in the left of Fig. 3, pivotal tokens form
a band-like pattern on the attention map, drawing attention
from nearly all following tokens. This pattern endures with
changes in the model’s layer number or attention heads,
suggesting the attention mechanism’s consistency and robustness in well pre-trained LLMs [7, 8]. Meanwhile, it
implies that attention scores within the model can represent
the semantic correlation between tokens.
When addressing diverse requirements, LLMs need to
balance attention among multiple tokens. For instance,
as seen in Fig. 3 (left), as one of the requirements in the
prompt, ‘compactness’ might not get enough attention during the generation process, resulting in an output that is
less compact than expected. Given the direct correlation
observed between attention and tokens, we propose an attention activation strategy to activate the attention scores on
highlighted tokens within the attention mechanism. This
strategy can effectively steer the model’s focus towards or
away from specific tokens, allowing for more nuanced and

3.3. Highlighting Visual Tokens
Methods for highlighting visual tokens can be classified into
two categories based on the type of token mapping involved,
as discussed in Sec. 2.1. In direct token mapping, such as in
LLaVA [10], the highlighting of visual tokens is straightforward. Image patch-level feature forms the sequential visual
contexts sim . This enables a natural patch-level scaling on
embeddings in Sec. 3.1 and attention activation introduced
in Sec. 3.2.
In contrast, the scenario becomes more complex with
query-based token mapping. For example, in works like
BLIP-2 [11, 13, 14], the image feature is transformed into a
unified set of few learnable queries q via Q-Former, which
are then input into LLMs as textual embeddings. This process obscures the direct correlation between image patches
and input tokens, as demonstrated in the example on the
right side of Fig. 3. To address this challenge, we leverage
4

learnable

CA Block

SA Block

SA Block

CA Block

Q-Former
(BERT)

context tokens

with highlight mask !
"!"

"

4. Experiments
4.1. Implementation Details
Pre-trained models. Prompt Highlighter can be applied to
a variety of general frameworks. For LLMs, we employ
the LLaMA model architecture [7] and utilize Vicuna-13B
v1.1 [8] as the test model. For VLMs, experiments are done
on one direct token mapping model LLaVA-13B [10, 23]
and one query-based mapping model InstructBLIP-Vicuna13B [15]. We adopt the LLaVA-v1.5 13B model [23] in
quantitative evaluations. All experiments are conducted on
a single NVIDIA-A800 GPU.
Hyper-parameters. The following parameter configurations are constant throughout almost all examples. In the
highlight guidance, we set the guidance strength γ = 1.3
in Eq. (6), and the scaling parameter α in Eq. (5) is set to
0.01. In attention activation, we accommodate the diverse
feature domains across different models by setting β = 2.0
in Eq. (7) and β = 20.0 in Eq. (9). The scale factor δ
in Eq. (7) is set to satisfy δm = (log(β) + 2)m.
Inference. During user interactions, if the text range selected by the user does not align perfectly with the tokens
from the tokenizer, we adjust the start or end selection position to ensure that the selected range is fully encompassed.
For images, the input mask is downsampled to a patch-wise
binary mask based on the input image. The size depends on
the visual encoder’s patch size (e.g., CLIP). In the autoregressive generation process, we employ a greedy search and
cache prior KV values across layers.

#
"

Figure 4. Highlighting visual tokens with Q-Former-based methods. In comparison with vanilla inference, we augment the learnable queries q by activating corresponding attention weights in the
Cross-Attention (CA) blocks.

The image features Cristiano Ronaldo, the Portuguese soccer player …

In this image, Cristiano Ronaldo is seen smiling and laughing while…

In this image, a Portuguese soccer player wearing the red and green jersey…

Figure 5. Attention scores in the first four queries of the Q-Former.
Each row shows a different user selection and text output.

the fact that the Q-Former itself is a Transformer model to
perform the token highlighting directly inside it. First, we
adopt the embedding rescale in Eq. (5) on the patch-wise
image feature and use the output queries from the Q-Former
to form the image context pair (sim , s̄im ). Then, by activating attention scores within the corresponding patch-wise
user-selection mask m in Q-Former’s cross-attention layers, we can effectively steer the model to adjust its focus.
This process is depicted in Fig. 4. In the Q-Former model,
cross attention is calculated across the learnable query q =
im
}.
{q1 , . . . , qM } and the image feature qim = {q1im , . . . , qN
We then activate the attention score corresponding to the
mask m within the cross-attention map. It can be expressed
using a formulation similar to Eq. (7),

p 
q̂ = softmax (log(β) · w + QK ⊤ )/ dk V ,
(9)

4.2. Applications and Comparisons
Partial context highlighting is a fundamental application
of Prompt Highlighter, utilized in scenarios where the emphasis is required on specific spans of the context. As
shown in Figs. 1 and 6, our method enables users to guide
the model’s attention toward relevant parts of the input by
highlighting them. It enhances the focus and relevance
of the generated output across a range of diverse frameworks [8, 10, 15], making results align more precisely with
user-selected tokens. It’s worth noting that the examples
in Fig. 6 can not be achieved by LLM-CFG [21], as it requires a prompt design that has complete sentences or images. This approach proves particularly useful in tasks such
as content summarization and interactive conversations.
Generation control. As demonstrated in Fig. 1, we can further control the degree of correlation to the highlighted part
in text generation. Users can manipulate the model’s output
dynamically by adjusting the highlight guidance strength γ
in Eq. (6). This capability can prove advantageous in a variety of tasks, ranging from generating descriptive captions
for images to creating customized responses in conversational agents.
Reliable description. When carrying out long text genera-

where Q = fq (q), K = fk (qim ), and V = fv (qim )
represent the transformed QKV vectors in the cross attention with projections f(·) , and dk denotes the dimension
of K. As a variation of the activation process in Eqs. (7)
and (8), w is a resized version of m, expanding dimension
from N to (H, M, N ), where H denotes the number of attention heads. We illustrate the variations in attention scores
with changing input masks in Fig. 5. Attention score maps
show attention concentrations in masked areas. This guides
the model’s focus towards specific visual elements, leading
to various generation outputs.
5

User: Please give me a plan to learn about
computer
vision.
computer
vision.

Normal Inference
Prompt Highlighter (PH)

[Vicuna] *Normal Inference:
1. Start with the basics… 2. Learn about image processing…
3. Learn a programming language… 4. Learn about OpenCV…
5. Learn about deep learning… 6. Practice with projects…
7. Read books and articles… 8. Join online communities…

b

[LLaVA-v1] Normal Inference:
As the sun sets over the ocean, the couple on the deck of the ship is lost in
each other‘s embrace. The man, with his arm around the woman, leans in
to whisper sweet nothings in her ear. The woman, with her head resting on
his chest, listens intently, feeling the warmth and love emanating from him.
Man: "I can't believe we're on this grand ship, sailing across the vast ocean.
It's like we're in a fairy tale.”
Woman: "It's truly magical. I feel so close to you right now, like nothing else
matters in the world.”
Man: “I feel the same way. I‘m so grateful to have you by my side on this
journey. I promise to always cherish and protect you”…

PH:
1. Foundations of Computer Vision… 2. Image Processing…
3. Feature Detection… 4. Object Detection… 5. Image
Segmentation… 6. Deep Learning for Computer Vision…
*We list keywords of the generation result here (full version in the appendix).
Our result has a more systematically plan and provide reference textbooks (Fig.8).

a
b

User: Write a
possible dialog
based on the image.

a

User: Why would a
person find this
image funny?

a. Highlight the Couple
PH: Couple (Rose DeWitt Bukater and Jack Dawson): As we stand
together on this grand ship, Rose, can you believe we’ve come so far from
our humble beginnings? Jack, I've never felt so alive, so free, or so in love as
I do right now, with you by my side. This journey has been both
exhilarating and heart-wrenching, but through it all, your presence has
been my beacon of hope.

[InstrcutBLIP] Normal Inference:
A person might find this image funny because it depicts a squirrel taking a
photo with a camera, which is an unusual sight. Squirrels are not typically
known for their photography skills or interest in cameras, so the idea of a
squirrel taking a photo is humorous and unexpected. Additionally, the
squirrel's facial expression adds to the comical aspect of the image, as it
appears to be laughing or smiling while holding the camera.

As we continue our voyage, I can’t help but wonder what the future holds
for us amidst the luxury and uncertainty of this massive ocean liner.
Whatever challenges we may face, I know that together, we can overcome
them and cherish every moment we have on this Titanic.
b. Highlight the Ship [Titanic]

a. Highlight the Camera
PH: The image shows a squirrel holding a camera and taking a photo of
another squirrel, which is funny because it is an unlikely scenario. Squirrels
are not typically known for their interest in photography, and the idea of a
squirrel using a camera is humorous …

PH: Woman (smiling): Jack, I can‘t help but feel a thrill as we sail aboard this
grand Titanic. The sunset only adds to the romantic atmosphere.
Man (Jack): Indeed, Rose. I'm grateful to share this moment with you, away
from the constraints of society. Our love transcends class and expectations.
Woman (leaning in closer): As we journey together, I know that our love will
only grow stronger, even amidst the uncertainty of our fate on this fateful
night.

b. Highlight the Teeth
PH: The image of a squirrel holding a camera and smiling is funny
because it is an unusual sight. Squirrels are not typically associated with
photography, and the idea of a squirrel taking pictures is humorous.
Additionally, the squirrel's grin adds to the comical nature of the image.

Man (gently holding her hand): Rose, I promise to cherish and protect you, no
matter what challenges we may face. Together, we can overcome anything.

Figure 6. Partial context highlighter. When given user input , vanilla inference might lead to unfocused generations. Our context-highlighted inference can faithfully capture the content of the highlighted part. The highlighted sections in each case are marked with circled
indices in the image. Outputs correlated with the highlighted parts are underlined.

4.3. Quantitative Evaluation

tion tasks, every predicted token interacts with all previous
content, including the newly generated contexts, to gather
information. This may lead to a gradual divergence and
loss of attention between the generated tokens and the input context, causing the model to hallucinate. We address
this issue by highlighting all user input condition tokens,
thereby guiding the generated content to align more closely
with the input context. This approach is particularly vital
for descriptive tasks, such as generating image captions.
We demonstrate this in Fig. 7. Our method can generate
more accurate and detailed descriptions. Moreover, when
the generated description is fed into an image generation
model like DALLE-3 [41], our method demonstrates its advantages in facilitating image-text alignment.

General VLM benchmarks. In Tab. 1, we evaluate
our method on common comprehensive Vision-Language
benchmarks, MME [27] and MMBench [26]. Prompt Highlighter on LLaVA-v1.5 demonstrates a consistent performance improvement compared to well-trained models by
designating the entire image as the highlighted part in the
input context. Notably, though these benchmarks primarily assess overall performance with single-token generation
and are not designed for user interactions, we still get the
2nd place in both MMBench and MME perception.
Reliable description. We then evaluate the reliability of
image captions generated by our method. In this case,
we utilized the reference-free metric CLIP Score [44], expressed as S-CLIP = 2.5 · max(cos(v, c), 0), to eval6

user input image

User:
Write a story
about this image

User:
Describe
this image

Text-to-image generation
:
normal inference w/ highlight context

Normal Inference: The image features a young woman with a bright
smile, wearing a pink headband and a white shirt. She is posing for the
camera, and her hair is styled in a ponytail. The woman appears to be
enjoying herself and is the main focus of the image.
CLIP-Sim=0.5811

CLIP-Sim=0.4538

As she walks through the garden, she encounters a small
bird perched on a branch, which adds to the serene
atmosphere. The girl's smile and the presence of the
flowers and teddy bear create a sense of joy and innocence,
making the scene a delightful and heartwarming moment.

Prompt Highlighter: The image features a smiling Asian woman wearing
headphones and carrying a pink backpack. She appears to be posing for
a picture, showcasing her headphones and ponytail hairstyle. The woman
is positioned under a clear blue sky, which adds a pleasant atmosphere
to the scene. Her white shirt complements her overall appearance, and
she is looking directly at the camera with a bright smile on her face.

CLIP-Sim=0.7441

User:
What are
they doing?

Normal Inference: The President of the United States and
a basketball player are standing together, holding a
basketball, and smiling for the camera.

Prompt Highlighter: LeBron James and President Obama
are exchanging a basketball, with the president holding
the basketball while standing next to the basketball
player.

Normal Inference: In the image, a young girl is happily
holding a bouquet of flowers, which includes a teddy bear.
She is wearing a red hat and a red dress, adding a pop of
color to the scene. The girl appears to be enjoying her time
outdoors, surrounded by a beautiful garden filled with
flowers.

Prompt Highlighter: In the illustration, a cheerful girl
dressed in red clothing is holding flowers and stuffed
bunnies in her arms. She appears to be a character from
Genshin Impact, a popular anime series. She is wearing a
helmet and a red hat, which adds a sense of adventure to
her appearance.

CLIP-Sim=0.8113

As she walks, petals from the flowers and leaves from the
surrounding trees flutter around her, creating a whimsical
atmosphere. She is carrying a suitcase, possibly filled with
more treasures or items she has collected during her
journey. The girl seems to be enjoying her time, smiling as
she holds the flowers and stuffed bunnies close to her heart.

CLIP-Sim=0.7817

Figure 7. Results when highlighting all input contexts. Given user input , vanilla inference might lead to hallucinations. In contrast,
context-highlighted inference can accurately capture the content of the image. We further feed descriptions into DALLE-3 [41] (shown on
the right) to provide a visually apparent difference. The CLIP-Similarity [30] with the input image is reported for each generated image.

method

S-CLIP

R@1

R@5

R@1

R@5

User study. We conduct a user study to assess the usability and effectiveness of our method. This study asks participants to rank generation results across five tasks (image captioning, image/text partial highlight generation, text
for image generation, and image understanding). Compared with the original inference baselines, the collected
255 valid preference results indicated that 77.3% of users
found Prompt Highlighter to generate more correlated results and be beneficial in accomplishing the task objectives.
More details about the quantitative evaluation can be
found in Appendix A.2.

FuseCap [43]
CoCa-CFG [22]

0.785
0.808

37.2
44.6

62.3
71.7

47.7
-

72.3
-

4.4. Ablation Study

LLaVA-v1.5 [23]
Prompt Highlighter

0.809
0.829

36.6
45.2

62.2
71.7

50.7
62.2

76.4
85.0

method

MME

MMBdev

MMBtest

OWen-VL-Chat [42]
mPLUG-Owl-2 [14]

1487.5
1540.2

60.6
64.5

61.8
66.0

LLaVA-v1.5 [23]
Prompt Highlighter

1531.3
1552.5 (+21)

67.7
69.7 (+2.0)

67.0
69.5 (+2.5)

Table 1. Evaluations on comprehensive VLM benchmarks, including MME Perception [27] and MMBench (MMB) [26].
text → image

image → text

Module-wise ablation. In Tab. 3, we systematically conduct an ablation study by removing each module of Prompt
Highlighter and noting the performance change in MME
and MMBench-dev. The results revealed that removing
the attention activation module led to the most considerable performance reduction, and combining the highlight
guidance and attention activation significantly improves the
overall performance.
Hyper-parameters. In Tab. 4, we explore the impact of
three scaling parameters (α, β, γ) in Eqs. (5) to (7) for highlight control on the performance of our method. We observed a trade-off between higher concentration (higher β
and γ) and general vision-language understanding ability.

Table 2. CLIP-based reference-free image caption evaluation conducted on the MSCOCO [25].

uate the embedding similarity between the image v and
the generated caption c. We also report the text ↔ image retrieval recall R@1, R@5 in the CLIP embedding
space. As demonstrated in Tab. 2, on the MSCOCO Karpathy test set [25], our method shows a state-of-the-art CLIP
score when compared to recent competitive caption-focused
methods [22, 43]. Our results exhibit significant advantages
across all metrics compared to the baseline.
7

settings

Guidance Attention

baseline (our impl.)
w/ Guidance
w/ Attention
Full pipeline

✓
✓

✓
✓

computer
vision.
[Vicuna] Please give me a plan to learn about computer
vision.

MME

MMBdev

1528.7
1531.1
1537.2
1552.5

67.7
68.5
69.5
69.7

1. Start Foundations: of Computer Vision: Start by learning the basics of
computer vision, including image processing, feature detection, and
image segment matching. Read books like “Computer Vision: Algorithms
and Applications” by Richard Szeliski and “Introduction to Computer
Vision” by Richard Szeliski, which et al.
2. Open Image Processing: Learn image processing techniques such as
filtering, edge thresholding, and edge detection. Read books like “Image
Processing: The Fundamentals” by John C. Bovik and “Digital Image
Processing” by Rafael C. Gonzalez…

Table 3. A module-wise ablation study.
α (in Eq. (5))
(α, 2.0, 1.3)

0.0
1517.6

0.01
1552.5

0.1
1522.3

1.0
1524.2

β (in Eq. (7))
(0.01, β, 1.3)

1.0
1527.2

2.0
1552.5

3.0
1537.3

4.0
1535.7

γ (in Eq. (6))
(0.01, 2.0, γ)

1.0
1537.2

1.3
1552.5

1.5
1532.6

2.0
1524.0

Predicted with vanilla inference. Reweighted by CFG.

Figure 8. An example of token changed with CFG.
!""
attention
maps

Table 4. Hyper-parameter ablation on MME. We probe for the
most suitable value for one of the combinations in (α, β, γ).

Baseline
Baseline beam=2
Prompt Highlighter

token/s ↑

memory (MB) ↓

S-CLIP ↑

6.67
5.97
5.95

16231
18537
17373

0.809
0.807
0.829

-1,-2,-1
0, 0, 0
1, 2, 1

Contexts’ Attention
Contribution %

method

highlight image context
vanilla inference

!! = #. %%
-1, 0, 1
-2, 0, 2
-1, 0, 1

Table 5. Evaluation on inference speed and GPU memory. The
memory is dominated by the model weight, consuming 13971 MB.

!" = #. &&

Generated token length →

0

Figure 9. Left: A simple verification of the vertical band-like pattern in the attention map, with a report on the gradient summation.
Right: Following [45], we present a visualization displaying the
average contribution of context’s attention during generation.

4.5. Discussions
Prediction control by CFG. Given that our highlight guidance operates on the predicted token probability, similar
to LLM-CFG [21], we further investigate the semanticlevel distinctions in normal and unconditional branches.
This is visualized through an example of token prediction in Fig. 8. When the embeddings associated with the
highlighted tokens are perturbed, the unconditional-context
branch tends to predict unrelated to the highlighted parts.
This, in turn, enables the rectification of responses in textgeneration tasks.
Attention activation. To confirm that the attention activation operates as anticipated, we not only visualize the activated attention scores corresponding to different regions
in Fig. 5, but we also demonstrate its effectiveness using
500 attention maps from the caption experiment in Fig. 9.
Firstly, we validate the band-like pattern property discussed
in Sec. 3.2, by comparing the vertical and horizontal gradients. We observed a significant gradient gap with Gx > Gy
in all 500 cases. Given this property, the attention activation can capture a higher attention probability p, as defined
in Eq. (8), from the given contexts. P
This attentionPcontribution plotted in Fig. 9 is denoted as mi =1 (pi )/ j<i (pj ).
Consequently, attention activation leads to results that adhere more closely to the input context.
Limitations and future work. While our approach introduces a novel method for controlling generation in multimodal LLMs, it has certain limitations: (a). Additional

computations: Our method requires an extra decoding
branch, which brings additional computational overhead
and GPU memory requirements. However, these additional
loads are marginal and acceptable with the batched inference. We validate this by the caption experiment in Tab. 5.
(b). Dependence on base model: Content generation quality
is tied to the base model’s capabilities, which may result in
over-emphasis or miss-emphasis on highlighted parts when
using poorly-trained base models.
One direction for future work will be to create a more
intuitive highlighting scheme. We also aim to extend our
method to support a greater variety of interactions.

5. Conclusion
We introduce Prompt Highlighter, a novel paradigm for
user-model interactions in multi-modal LLMs, offering output control through a token-level highlighting mechanism.
This approach, requiring no extra training, competes well
on standard benchmarks and provides reliable generation
outputs by merely highlighting input context. Further, diverse applications demonstrate its intuitive usability and effectiveness in enhancing control over the generation process. This work represents a promising direction for enhancing user control in multi-modal LLMs, and we anticipate it will inspire further research.
8

Appendix Contents
A. Experiment Details
A.1. Workflow of the Prompt Highlighter . . . . .
A.2. Quantitative Evaluation . . . . . . . . . . . .
A.2.1 VLM Benchmarks . . . . . . . . . .
A.2.2 Reliable Descriptions . . . . . . . .
A.2.3 User Study . . . . . . . . . . . . . .
A.3. Attention Map Visualization . . . . . . . . .
A.4. Limitation Analysis . . . . . . . . . . . . . .
A.4.1 Speed and Memory . . . . . . . . .
A.4.2 Constrained by the Base-Model . . .

9
9
9
9
10
10
10
11
11
11

B . Additional Disccusions
B.1. Compare with LLM-CFG . . . . . . . . . . .
B.2. Compare with other Highlight Methods . . .

12
12
12

C. Showcases
C.1. More Visual Results . . . . . . . . . . . . . .
C.2. Multi-Round Interactive Conversation . . . .

13
13
13

Algorithm 2 Attention Activation in Self-Attention Layer.
For simplicity, we demonstrate with single-head attention.
Require: Input hidden state q = {q1 , . . . , qη−1 } and vanilla attention mask mattn , highlight mask m, Scaling factor β, SelfAttention Module MSA ⊂ P̃Θ .
1: (Q, K, V ) = MSA .proj qkv(q)
2: Calculate the attention score map:
3: k = Q@K ⊤ + mattn
4: Initialized the highlighted attention mask:
5: w = m.expand as(k)
6: Activate score via attention map (Eqs. (7) and (9)):
7: h = log(β)w + k
8: Then complete the remaining operations in self-attention:
9: p = softmax(h)
10: return qout = MSA .proj out(p@V )

A.2. Quantitative Evaluation
A.2.1

VLM Benchmarks

We use the same prompts and test scripts for our benchmark tests as those in the LLaVA-v1.5 codebase [23]. The
only difference is our highlighting of the image context in
the inputs (i.e., mi = 1 when si ∈ sim ). We adopt softmax, instead of log softmax used in the multiple-token prediction tasks, as the token probability rescale function. As
discussed in Sec. 4.4, we observe slight variations in results across different test sets based on the parameters of
attention activation. In Tab. 7, we showcase the impact of
changes in the β value on the benchmark tests. Our method
still achieves the second rank on the current MMBenchdev leaderboard, even when using a uniform parameter of
β = 2.0.
We adopt the MME and MMBench datasets because they
offer comprehensive evaluations of VLMs across multiple
dimensions and feature over 10K of VQA questions. This
makes them ideal for assessing the overall competency of
Vision-Language Models. We present a detailed evaluation result of MMBench in Tab. 6. With the same pretrained weight as LLaVA-v1.5, our training-free method
consistently outperforms the baseline across nearly all evaluation dimensions. Given that our approach is compatible
with VLM frameworks that use token-level embedding input and a Transformer-based language decoder, it positions
the Prompt Highlighter as a training-free plugin for integrating context-highlighting abilities into these models. We
further evaluate the MMBench benchmark using the current
state-of-the-art model, InternLM-VLComposer [32]. Our
method demonstrated a performance improvement, increasing the SOTA performance from 74.8 to 75.3 in the dev
split with this representative Q-Former-based approach.
These results suggest that though our method is designed to
create new interactive ways instead of focusing on bench-

A. Experiment Details
A.1. Workflow of the Prompt Highlighter
For a more comprehensive understanding of the method part
of Prompt Highlighter, we provide pseudo-code algorithmic
workflows in Algorithm 1 and Algorithm 2. In this context, the attention activation outlined in Algorithm 2 acts as
the modified forward function in all Self-Attention layers
during the multi-modal LLMs’ inference P̃Θ (xη |s, m), as
discussed in Sec. 3.2 and illustrated in Algorithm 1.
Algorithm 1 Highlighted Guidance Control
Require: Pre-trained LLM or VLM decoder PΘ , input multimodal tokens x, binary highlight mask m, guidance strength
γ, and scaling factor α, δ
1: Initialize token sequence x = {x1 , . . . , xN }
2: Initialize conditional embedding s = f (x) and unconditional
embedding s̄ as s̄ = (α − 1)m · f (x) + f (x) (Eq. (5)).
3: Initialize current token index η = N , output token sequence
y=[]
4: while xη is not </s> do
5:
Calculate the greedy decoded prediction xη in Eq. (6):

xη+1 = argmax(γ log P̄Θ (xη+1 |s, m)
− (γ − 1) log P̄Θ (x̄η+1 |s̄, −δm))
7:
Update output sequence:
8:
y.append(PΘ .tokenizer.decode(xη+1 ))
9:
Update next token embedding and mask:
10:
sη+1 = s̄η+1 = f (xη+1 ), mη+1 = 0, η = η + 1
11: end while
12: return Output sequence y
6:

9

MMBench-dev

MMBench-test

Overall

LR

AR

RR

FP-S

FP-C

CP

Overall

LR

AR

RR

FP-S

FP-C

CP

MMICL [46]
mPLUG-Owl2 [14]
Sphinx [47]

67.9
66.5
67.2

49.2
32.2
33.1

71.6
72.4
67.3

73.0
60.9
58.3

66.7
68.6
74.4

57.2
60.1
59.4

77.2
79.4
80.7

65.2
66.0
67.5

44.3
43.4
32.9

77.9
76.0
73.6

64.8
62.1
57.8

66.5
68.6
72.1

53.6
55.9
63.2

70.6
73.0
79.2

LLaVA-v1.5 [23]
+ Prompt Highlighter
Improvement

67.7
69.7
+2.0

41.7
44.2
+2.5

69.7
70.6
+0.9

63.5
68.7
+5.2

70.0
73.7
+3.7

59.3
59.3
0.0

80.2
80.9
+0.7

67.0
69.5
+2.5

39.9
42.6
+2.7

74.7
77.5
+2.8

61.6
64.3
+2.7

70.9
75.0
+4.1

59.9
62.0
+2.1

75.4
76.4
+1.0

method

Table 6. Detailed comparison in MMBench-dev and MMBench-test[26]. The categories include Logic Reasoning (LR), Attribute
Reasoning (AR), Relation Reasoning (RR), Instance-Level Fine-Grained Perception (FP-S), Cross-Instance Fine-Grained Perception (FPC), and Coarse Perception (CP). The improvement of our method over the LLaVA-v1.5 [23] baseline is reported in the last row for each
category. We highlight the best and underline the second best result for each column.

method

MME

MMBdev

MMBtest

LLaVA-v1.5 [23]

1531.3

67.7

67.0

PH (0.01, 3.0, 1.3)
PH (0.01, 2.0, 1.3)

1537.3
1552.5

69.7
68.7

68.7
69.5

Table 7. Hyper-parameter settings in VLM benchmarks. Evaluations are conducted on MME Perception [27] and MMBench
(MMB) [26]. We list a combination of hyper-parameters (α, β, γ)
for our method (PH for Prompt Highlighter).

baseline

Text Partial Highlighter
Image Partial Highlighter
Image Understanding
Image Caption
Description → Image
Overall

Vicuna-13B
LLaVA
InstructBLIP
LLaVA-v1.5
LLaVA-v1.5 + DALLE-3
-

preference
70.6%
74.5%
76.5%
80.4%
84.3%
77.3%

Table 8. A fine-grained user study result.

A.2.3

mark improvement, it still proves to be competitive in general applications.

A.2.2

sub-task

User Study

We present the detailed results of the user study in Tab. 8.
Using different sub-task examples, we engaged various
baseline models for different tasks to demonstrate the consistent preference for Prompt Highlighter across a broad array of frameworks and tasks. The user preference results
affirm that our method delivers superior performance and
flexibility, consistently outperforming the baseline models
across a wide range of tasks.

Reliable Descriptions

As discussed in Sec. 4.4, we observe a trade-off between the
attention concentration and QA performance that appeared
in the benchmarks. Therefore, compared with experiments
in VLM benchmarks, we adopt a relatively more aggressive parameter setting in the MSCOCO caption experiment:
(α, β, γ) = (0.01, 7.0, 2.0). We utilize a CLIP ViT-Base-32
model [30, 48] to test the CLIP Score [44] and to extract embeddings for both text → image and image → text retrieval
metrics. Our experiment uses a simple input prompt, ’De
scribe this image.’ . In the comparison in Tab. 2,
we report the official result reported by CoCa-CFG [22]
with the same γ value of 2.01 . As CLIP has a maximum token length limitation of 77, we split captions at semicolons
and periods and input the longest possible concatenated sequence to circumvent this limitation. For all text-to-image
generation results in figures, we query the generated text
description to the Bing Image Creator powered by DALLE3 [41] [www.bing.com/images/create].

A.3. Attention Map Visualization
In this section, we delve into the attention map visualization, expanding on the verification experiments discussed in Sec. 4.5. In Fig. 10, we illustrate an example
akin to Fig. 3, demonstrating the persistence of the bandlike pattern in different layer segments in the Vicuna-13B
model [8]. The band-like property is clear and consistently
evident across various layers. Given that these band-like token activation patterns are distributed across different layers, we simply incorporate the attention activation modification in all self-attention layers. Furthermore, the attention
maps referenced in Fig. 9 to calculate gradients and attention contributions are computed based on the averaged attention map across layers and heads.

1 CoCa-CFG achieves caption → image retrieval recall with R@1=49.4
and R@5=75.7 with γ = 3.0, while it remains the same S-CLIP=0.808.

10

Several punctuations in the first few layers dominant the attention score.

Layer 01-10

Layer 01-40 average

Layer 11-20

Layer 21-31

Layer 31-41
low

high

Figure 10. An additional example of attention maps across different layer segments (left) and the averaged visualization with more
generated tokens (right). We mark tokens representing punctuation, which draw significant attention in the initial layers. Beyond these
tokens, a consistent correlation between tokens forms a band-like pattern that can be observed across all layers.

a

and we validate its tangible impact in Tab. 5. Memory consumption with 8-bit loading confirms that the Prompt Highlighter is designed to be memory-efficient, making it feasible to run inference on a 13B model using commonly available GPUs, such as the NVIDIA-3090. This added load becomes noticeable when evaluating single-token prediction
tasks, as the model is required to compute double KV values among tokens in its initial inference step. However,
when employing KV-cache in the multiple-token generation, this extra load can be alleviated, as the doubled calculation with cached KVs does not become the bottleneck
for inference speed or GPU memory consumption. This results our method only has a 10.5% overall additional time
and 7% additional GPU memory. We also confirm that the
commonly utilized inference strategy of introducing an extra beam search branch does not impair the quality of the
descriptive task (S-CLIP 0.809→0.807 in Tab. 5), despite
imposing an additional computational load. This also illustrates the practical effectiveness of Prompt Highlighter.

User: Write a possible
dialog based on the
image.

[LLaVA-v1] Prompt Highlighter: Couple (Rose DeWitt Bukater and Jack
Dawson): As we stand together on this grand ship, Rose, can you believe we’ve
come so far from our humble beginnings? Jack, I've never felt so alive, so free,
or so in love as I do right now, with you by my side. This journey has been
both exhilarating and heart-wrenching, but through it all, your presence has
been my beacon of hope.

[InstructBLIP] Prompt Highlighter: Titanic is a 1997 American epic romantic
disaster film directed, written, and co-produced by James Cameron. The film
stars Leonardo DiCaprio and Kate Winslet as Jack Dawson and Rose DeWitt
Bukater, two fictional characters who fall in love aboard the RMS Titanic
during its ill-fated maiden voyage in 1912. In the movie poster, the couple is
depicted in a romantic embrace, with the Titanic ship in the background. There
are two versions of the poster, one featuring the couple closer together and the
other with them farther apart. (not a dialog)

a. Highlight the Couple

As we continue our voyage, I can’t help but wonder what the future holds for
us amidst the luxury and uncertainty of this massive ocean liner. Whatever
challenges we may face, I know that together, we can overcome them and
cherish every moment we have on this Titanic.

[InstructBLIP] Normal Inference: The scene depicts a romantic movie poster
featuring a man and a woman embracing each other in front of a large ocean
liner. The couple is positioned on the left side of the poster, while the ocean liner
takes up most of the space on the right side. There are two people visible in the
poster, with the man standing closer to the left side and the woman standing
closer to the right side. The ocean liner dominates the center of the poster,
making it the focal point of the image. (not a dialog)

A.4.2

Figure 11. An example illustrating the influence of different base
models. Given the same highlighted context, the output of the
model may be constrained by the capabilities of the base model.

Our method stems from the base model and its plug-andplay nature. As illustrated in Fig. 11, the base model InstructBLIP [15] is not adept at dialog generation. Consequently, with a highlighted prompt context, despite the output text being more related to the highlighted part, it fails to
generate a dialogue effectively. This suggests that while the
Prompt Highlighter can augment existing abilities in pretrained multi-modal LLMs, it cannot endow them with new
capabilities.

A.4. Limitation Analysis
A.4.1

Constrained by the Base-Model

Speed and Memory

As discussed in Sec. 4.5, Prompt Highlighter imposes an
additional computational load on the unconditional branch,
11

User: Please give me a plan to learn about computer
vision.
computer
vision.

User: <marked_image> Describe this image, emphasize the marked area.
Normal Inference (add marks and arrow): The image
features a man wearing a red and green soccer jersey,
smiling and posing for the camera. He is wearing a
wristband on his left wrist, and his shirt is numbered
with the number seven. The man is standing in a room,
possibly a locker room, with a few other people in the
background.
The main focus of the image is the man in the red and
green soccer jersey, and a red arrow is pointing towards
him, drawing attention to his presence in the scene.

[Vicuna] Normal Inference:
1. Start with the basics… 2. Learn about image processing…
3. Learn a programming language… 4. Learn about OpenCV…
5. Learn about deep learning… 6. Practice with projects…
7. Read books and articles… 8. Join online communities…
LLM-CFG:
1. Learn the basics of programming … 2. Learn the basics of image
processing … 3. Learn about OpenCV … 4. Learn about machine
learning … 5. Learn about computer vision algorithms … 6.
Practice with datasets … 7. Build projects … 8. Stay up to date …

User: <original_image> Describe this image, and emphasize more on the jersey

Prompt Highlighter:
1. Foundations of Computer Vision… 2. Image Processing…
3. Feature Detection… 4. Object Detection… 5. Image
Segmentation… 6. Deep Learning for Computer Vision…

Normal Inference (prompt engineering): The image features a man wearing a red
and green soccer jersey, smiling and posing for the camera. The jersey has a
distinctive design, with a combination of red and green colors, and the number
seven displayed on it. The man appears to be a soccer player, possibly a
professional athlete, as he is wearing a sports uniform and is in a stadium setting.
User: <masked_image> Describe this image

Figure 12. One example of the comparison with LLM-CFG [21].

Normal Inference (physically “highlight” the jersey part
and input a masked image ):
The image features a man wearing a red and green
soccer jersey, standing on a field and smiling. He appears
to be a soccer player, possibly a professional athlete, as
he is wearing a uniform and is in a stadium setting. The
man is the main focus of the image, and he is
surrounded by a crowd of people, indicating that he
might be participating in a game or event.

computervision.
vision.
[Vicuna] Please give me a plan to learn about computer
! > 0.1

Sure, here's a plan to help you learn computer vision…

! = 0.01 Certainly! Here's a plan to help you learn Python…
LLM-CFG Certainly! Here‘s a plan to help you learn ASSTST…

User: < original _image, m> Describe this image
Prompt Highlighter: The image features a soccer player wearing a red, green, and
gold jersey with the number 7 on it. The player is smiling and appears to be
enjoying the game. He is wearing green and red stripes on his jersey, which are
visible on the sleeves and the green stripe on the side. The player is also wearing a
green and white armband on his right arm. The scene captures the excitement and
enthusiasm of the soccer player as he participates in the game.

Figure 13. Output example for the unconditional branch with different unconditional context designs.

B. Additional Disccusions
B.1. Compare with LLM-CFG

Figure 14. A comparison of Prompt Highlighter with different
vision-language prompt manipulation approaches.

The comparison between Prompt Highlighter and LLMCFG [21] is visually demonstrated in Fig. 12. Our method
stands out by offering a more granular control mechanism
that allows specific tokens within the context to be highlighted. Conversely, LLM-CFG, with its basic prompt-level
differentiation and without a mechanism to directly manipulate model feature interactions (e.g., our attention activation
strategy), often generates outputs that bear noticeable similarity to the original, regardless of substantial γ increases.
Using the identical prompt as in Fig. 12, we consolidate
this claim by exhibiting the independent inference outcomes
of the unconditional branch in Fig. 13. In addition, we
demonstrate the significance of assigning the α value at the
embedding layer. As shown in Fig. 13 and as per our discussion in Sec. 3.1, it becomes apparent that a small scalar multiplication α in the embedding space leaves the overall contextual understanding of the unconditional branch largely
unaffected. Such a fine-grained difference assists in making more distinguishing token choices during the inference
process in Eq. (6). In contrast, LLM-CFG’s unconditional
branch omits part of the prompt, resulting in an obstacle

to generating results that effectively underscore the highlighted sections.

B.2. Compare with other Highlight Methods
To demonstrate the non-trivial nature of our method, we
evaluated it alongside several intuitive prompt engineering
approaches, which include adding adjectives, capitalizing
text, and explicitly marking in images. Comparative examples in LLM and VLM are provided in Figs. 1 and 14,
respectively. Our observations suggest that the employment
of additional prompts or explicit emphasis can occasionally
lead to unpredictable results. This is primarily due to the introduction of extra information, while models may not truly
emphasize correlated parts. In contrast, our method seamlessly accommodates all context information and produces
focused text outputs. This demonstrates that the design of
our method not only offers a high degree of flexibility but
also enables the generation of more contextually appropri12

ate outputs.

C. Showcases
C.1. More Visual Results
Additional results of Prompt Highlighter in LLM and VLM
are provided in Fig. 15 and Fig. 16, respectively.
In the first example of Fig. 15, we showcase the primary
part of the output of the example in Fig. 6. Our results concentrate more on the highlighted section, offering professional reference books to support user learning “computer
vision”. The second example exhibits a case for a multisegment mask input, where the model is steered to output
code snippets at the appropriate positions by highlighting
the requirement part of the text and the comment as the position identifier. Fig. 16 demonstrates a range of applications
and capabilities of Prompt Highlighter in VLM from various perspectives, including image content comprehension
(case 1), focused generation (case 2), improved image descriptions (case 3), and explicit generation control (case 5).
These instances underscore the performance enhancement
and customizable generation control facilitated by Prompt
Highlighter. We also present some challenging cases due to
the limited perceptual capacity of the base model.
Moreover, we present results of text-to-image generation based on the text descriptions generated by our method
in Fig. 17. These outcomes vouch for the superior capability of our approach in descriptive tasks and its proficiency
in generating higher-quality image-text matching data.

C.2. Multi-Round Interactive Conversation
Up to this point, the examples and experimental scenarios
we’ve showcased are for single-round dialogues or QAs.
As a general-purpose interactive inference pipeline, our
method can also support multi-round interactions and conversations. We provide a schematic diagram of multi-round
conversation at the top of Fig. 18. The multi-round conversations’ generation can be decomposed into single-round
generations with continuously growing context, allowing
Prompt Highlighter to change the focus of user input in each
round by updating the input mask m and resetting previous
KV caches.
Below the pipeline, we demonstrate how to resolve the
image understanding problem, presented in Fig. 16 case 4,
through multi-round conversation and human interactions.
By highlighting multi-modal tokens and guiding the model
to understand the image content in a segmented manner,
users can “teach” the model to solve problems that it was
previously unable to solve.
Figures 15-18 are presented on the following pages ↓

13

Case1: A full version of the first example in Fig. 6

Normal Inference

Prompt Highlighter

User: Please give me a plan to learn about computer vision.

1. Start with the basics: Before diving into the advanced concepts of computer vision, it's essential to have a solid understanding of the fundamentals. Learn about the different types of
computer vision, such as object recognition, image segmentation, and motion analysis.
2. Learn about image processing: Image processing is a crucial aspect of computer vision. Learn about image filtering, edge detection, and other image processing techniques.
3. Learn a programming language: Python is the most popular language used in computer vision. Learn the basics of Python and familiarize yourself with the libraries commonly used in
computer vision, such as OpenCV, TensorFlow, and Keras.
4. Learn about OpenCV: OpenCV is a powerful library for computer vision tasks. Learn about its various functions and how to use them to implement computer vision algorithms.
5. Learn about deep learning: Deep learning is a subset of machine learning that uses neural networks to perform tasks. Learn about the basics of deep learning and how it's used in computer
vision.
6. Practice with projects: The best way to learn computer vision is by practicing with projects. Start with simple projects and work your way up to more complex ones. This will help you apply
what you've learned and improve your skills.
7. Read books and articles: There are many books and articles available on computer vision. Read them to learn about the latest developments in the field and to gain a deeper understanding
of the concepts.
8. Join online communities: Join online communities, such as forums and social media groups, to connect with other computer vision enthusiasts. This will help you learn from others and stay
up-to-date with the latest developments in the field.
1. Foundations of Computer Vision: Start by learning the basics of computer vision, including image processing, feature detection, and image matching. Read books like "Computer Vision:
Algorithms and Applications" by Richard Szeliski and "Introduction to Computer Vision" by Richard Szeliski, et al.
2. Image Processing: Learn image processing techniques such as filtering, thresholding, and edge detection. Read books like "Image Processing: The Fundamentals" by John C. Bovik and
"Digital Image Processing" by Rafael C. Gonzalez and Richard E. Woods.
3. Feature Detection: Learn about feature detection techniques such as SIFT, SURF, and ORB. Read papers like "SIFT: Scale-Invariant Feature Transform" by David G. Lowe and "ORB:
Oriented FAST and Rotated BRIEF SIFT" by Yi Liu and Aki Kanade.
4. Object Detection: Learn about object detection techniques such as Haar cascades, HOG, and Viola-Jones algorithms. Read papers like "Haar Cascades for Real-Time Object Detection" by
Andreas Santner, et al. and "A Survey of Object Detection" by Paul Viola and Michael Jones.
5. Image Segmentation: Learn about image segmentation techniques such as connected component analysis, region growing, and watershed segmentation. Read books like "Image Segmentation:
Algorithms and Applications" by T. P. Singh and "Computer Vision: A Modern Approach" by David A. Forsyth and Jean Ponce.
6. Deep Learning for Computer Vision: Learn about deep learning techniques for computer vision, including convolutional neural networks (CNNs) and recurrent neural networks (RNNs).
Read books like "Deep Learning for Computer Vision" by Rajalingapuram Kannan and "Computer Vision with Deep Learning" by S. S. K. Tiwari and S. C. Gupta.
7. Practice and Projects: Practice your computer vision skills by working on projects and datasets such as ImageNet, OpenCV, and Kaggle.

Case2: Specific requirement in code
generation. We partial highlight both
the text prompt and the position
comment in the code segment.

User: I would like to
print top 5 token
predictions of last_logit.
Directly output the
newly added codes .

# INSERT CODE HERE

You can simply print the top 5 token predictions of the last
logit by sorting the logits in descending order and taking the
first 5 elements. Here's the updated code:

Here's the added part to print the top 5 token predictions of the last logit:

This code sorts the last logit based on the logits and selects the top 5 tokens. The logits represent the model's confidence in each token being the next one in the sequence, so the top 5 tokens
with the highest logits are likely to be the correct predictions. Finally, the code prints the added part.

Figure 15. Two more examples in pure text partial highlighter. A Vicuna-13B-v1.1 [8] is used as the base model.

14

3
Normal Inference
Prompt Highlighter

Case1: image
understanding

1

User: What is funny
about this image?
Describe it panel by
panel.

Case3: better image
caption in real-world
images (MSCOCO)

User: Describe this image
in detail.

T2I result

CLIP-sim=0.288

The image consists of two panels. In the first panel, there is a close-up view of a
smartphone with a cable attached to it. The cable appears to be plugged into the
phone's charging port. In the second panel, there is a wider shot of a person
holding the smartphone in one hand and the charging cable in the other hand.
The combination of these two panels creates a humorous situation where the
person is holding the charging cable while the phone is already connected to it, as
if they are trying to charge the phone without actually plugging it in.

The image features a wooden table filled with a large
assortment of bananas. The bananas are arranged in various
bunches, with some placed closer to the front and others
towards the back of the table. The table is situated outdoors,
possibly in a market or a similar setting.
CLIP-sim=0.336

1. Highlight the charger part

Case2: generation with
different attention.

User:
What is in the image?

a cute anime
girl holding a
bouquet of
flowers

a girl holding
a bouquet of
flowers and
bunnies

3. Highlight the full image context

The image features a wooden crate filled with a variety of
bananas, including both ripe and unripe ones. The bananas
are arranged in different stages of ripeness, with some
appearing yellow and ripe, while others are still green. The
crate is placed outdoors, possibly at a market or a fruit stand,
and is surrounded by trees in the background.

In this image, there are two panels side by side. The first panel shows a
smartphone with a blue USB cable attached to it. The second panel is a close-up
view of the USB port on the smartphone, which has a small blue tool inserted
into it. The combination of these two panels creates a humorous and unexpected
image, as it appears that the blue tool is being used to charge the smartphone
through its USB port. This is an unconventional way of charging a smartphone,
and it adds a touch of humor to the scene.

CLIP-sim=0.8544

CLIP-sim=0.8724

4
Case4: a hardcase for
image understanding.
The upper-bound of the
method is depended on
the base model.

User: Is there any text in
the image? Why people
may find this image
interesting?

Yes, there is a text in the image. The text is a conversation between a man and a
woman, with the man asking the woman if she has the coronavirus. This image
may be interesting to people because it reflects the concerns and anxiety
surrounding the COVID-19 pandemic. The image could also be a representation
of the importance of maintaining social distance and being cautious in public
spaces, as the man and woman are standing at a distance from each other. The
image might also be a commentary on the impact of the pandemic on people's
daily lives and interactions.

2a. Highlight the holding flowers

4. Highlight the full image context
an anime girl
with a hat and
horns

2b. Highlight red hat part

an anime
character holding
a rabbit while
surrounded by
flowers

Yes, there is text in the image, which includes speech bubbles from the two
people, a man and a woman, and a before and now panel. People may find this
image interesting because it humorously portrays a man and a woman wearing
green face masks, possibly as a response to the COVID-19 pandemic. The image
uses a comic strip format to convey the message, making it more engaging and
relatable to the current situation. The cartoon characters' conversation and the
use of speech bubbles add a lighthearted touch to the serious topic, making it an
entertaining and thought-provoking piece of art.

2c. Highlight the bunny

Case5: Control the degree of focus with different (!, #).

User: Describe this image.

The image features Cristiano Ronaldo, the Portuguese soccer player, in a red and green jersey, celebrating with his arms outstretched…

Cristiano Ronaldo is seen in a red and green Portugal soccer jersey, celebrating his team's victory during a match…

5. Highlight the jersy

5

The image features a Portuguese soccer player wearing a red and green Nike jersey, with the number 7 on his chest…

Figure 16. More results with multi-modal inputs. We demonstrate three applications (1,2,3,5) and one hard case (4) for image understanding. We use InstructBLIP-Vicuna-13B [15] as the base model of cases 1,2,5 and LLaVA-v1.5 [23] 13B as the base model of cases 3,4.

15

Input images

T2I results using text from vanilla inference

T2I results using text from context-highlighted inference

Figure 17. More text-to-image examples with image generation by DALLE-3 [41]. We have marked areas in some figures that generate
better results due to different captions.

16

Multi-Round Conversation
Round 1 context
normal context
unconditional context

Round 2 context

Language
Model

normal context

generated context

unconditional context

highlight mask "!"

refresh
KV-cache

…
Language
Model

…

generated context

highlight mask "!#

Vanilla inference

Interactive Inference

Round 1 highlighted

Round 2 highlighted

Round 1

Round 1

User: <image> Describe the
conversation in the left side image.

User: <image> Describe the
conversation in the left side image.

highlighting tokens in
the generated text

Round 2

Highlight the left part
of the image

Assistant: In the left side image, the conversation between the
two people is about pregnancy. The woman is talking to the
man, who is wearing a green face mask, and asking him if he is
pregnant. The man is responding by saying, "Are you are you
pregnant?". This conversation is taking place in a setting where
the woman is wearing a green face mask, and the man is wearing
a green face mask as well.

Assistant: In the left side image, a man and a woman are having
a conversation. The woman is wearing a green face mask, and
the man is wearing a green neck gaiter. They are talking about
the coronavirus, with the woman possibly expressing her
concerns or sharing information about the virus. The man is
listening attentively to the woman's words.

Round 2

User: Describe the conversation in the
right side of the image, then try to
explain the meme inside the image.

User: Describe the conversation in
the right side of the image, then try to
explain the meme inside the image.

The meme inside the image is a play on words, using the phrase
"now" to represent the present time and the phrase "before" to
represent the past. The two conversations are juxtaposed, with
the left side showing a pregnancy-related conversation and the
right side showing a conversation about the coronavirus. The
meme is a creative way to compare and contrast two different
situations and their respective concerns.

Highlight the right part of the image

Assistant: In the right side of the image, the conversation is
about the coronavirus. The woman is talking to the man, who is
wearing a green face mask, and asking him if he has the
coronavirus. The man is responding by saying, "Do you have the
coronavirus?". This conversation is taking place in a setting
where both the woman and the man are wearing green face
masks.

Assistant: In the right side of the image, a man and a woman are
having a conversation. The woman is wearing a green face mask,
and the man is wearing a green neck gaiter. They are talking
about the coronavirus, with the woman possibly expressing her
concerns or sharing information about the virus. The man is
listening attentively to the woman's words. The meme inside the
image is a play on the idea of a "before" and "after" scenario.
The "before" scenario shows the man and woman talking about
the coronavirus, while the "after" scenario shows the same man
and woman talking about the same topic, but with the woman
wearing a green face mask and the man wearing a green neck
gaiter. This meme is likely meant to emphasize the importance
of wearing face masks and practicing social distancing to
prevent the spread of the virus.

Figure 18. Multi-round interactive conversation powered by Prompt Highlighter. We illustrate the multi-round interactive conversation pipeline on the top. We provide a comparison in a multi-round conversation between the vanilla inference (left) and Prompt Highlighter (right). In this example, the user highlights different contexts in two rounds.

17

References

[14] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan,
Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi,
Yaya Shi, et al. mplug-owl: Modularization empowers
large language models with multimodality. arXiv preprint
arXiv:2304.14178, 2023. 4, 7, 10
[15] Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat
Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale
Fung, and Steven Hoi. Instructblip: Towards generalpurpose vision-language models with instruction tuning,
2023. 1, 2, 5, 11, 15
[16] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran,
Thomas L Griffiths, Yuan Cao, and Karthik Narasimhan.
Tree of thoughts: Deliberate problem solving with large language models. arXiv preprint arXiv:2305.10601, 2023. 1,
2
[17] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten
Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.
Chain-of-thought prompting elicits reasoning in large language models. NeurIPS, 35:24824–24837, 2022. 2
[18] Significant-Gravitas. Significant-gravitas/autogpt: An experimental open-source attempt to make gpt-4 fully autonomous. 2
[19] Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le,
Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny
Zhou. Self-consistency improves chain of thought reasoning
in language models. arXiv preprint arXiv:2203.11171, 2022.
1, 2
[20] Jonathan Ho and Tim Salimans. Classifier-free diffusion
guidance. arXiv preprint arXiv:2207.12598, 2022. 2, 3
[21] Guillaume Sanchez, Honglu Fan, Alexander Spangher, Elad
Levi, Pawan Sasanka Ammanamanchi, and Stella Biderman.
Stay on topic with classifier-free guidance. arXiv preprint
arXiv:2306.17806, 2023. 3, 5, 8, 12
[22] Simon Kornblith, Lala Li, Zirui Wang, and Thao Nguyen.
Guiding image captioning models toward more specific captions. In ICCV, pages 15259–15269, 2023. 2, 7, 10
[23] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee.
Improved baselines with visual instruction tuning. arXiv
preprint arXiv:2310.03744, 2023. 2, 5, 7, 9, 10, 15
[24] Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell
Nye, Augustus Odena, Arushi Somani, and Sağnak Taşırlar.
Introducing our multimodal models, 2023. 2
[25] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence
Zitnick. Microsoft coco: Common objects in context. In
ECCV, pages 740–755. Springer, 2014. 2, 7
[26] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang
Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He,
Ziwei Liu, et al. Mmbench: Is your multi-modal model an
all-around player? arXiv preprint arXiv:2307.06281, 2023.
2, 6, 7, 10
[27] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin,
Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin, Jinrui Yang,
Xiawu Zheng, et al. Mme: A comprehensive evaluation
benchmark for multimodal large language models. arXiv
preprint arXiv:2306.13394, 2023. 2, 6, 7, 10
[28] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert,
Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov,

[1] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya
Sutskever, et al. Improving language understanding by generative pre-training. 2018. 1, 2
[2] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario
Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.
Language models are few-shot learners. NeurIPS, 33:1877–
1901, 2020.
[4] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar
Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized
bert pretraining approach. arXiv preprint arXiv:1907.11692,
2019.
[5] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova.
Bert: Pre-training of deep bidirectional
transformers for language understanding. arXiv preprint
arXiv:1810.04805, 2018.
[6] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and
Luke Zettlemoyer. Bart: Denoising sequence-to-sequence
pre-training for natural language generation, translation, and
comprehension. arXiv preprint arXiv:1910.13461, 2019.
[7] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier
Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste
Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.
Llama: Open and efficient foundation language models.
arXiv preprint arXiv:2302.13971, 2023. 2, 4, 5
[8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao
Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao
Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source
chatbot impressing gpt-4 with 90%* chatgpt quality. See
https://vicuna. lmsys. org (accessed 14 April 2023), 2023.
4, 5, 10, 14
[9] Hyung Won Chung, Le Hou, Shayne Longpre, Barret
Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang,
Mostafa Dehghani, Siddhartha Brahma, et al. Scaling
instruction-finetuned language models.
arXiv preprint
arXiv:2210.11416, 2022. 1, 2
[10] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee.
Visual instruction tuning. arXiv preprint arXiv:2304.08485,
2023. 1, 2, 4, 5
[11] Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision-language
understanding with advanced large language models. arXiv
preprint arXiv:2304.10592, 2023. 2, 4
[12] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi.
Blip: Bootstrapping language-image pre-training for unified
vision-language understanding and generation. In ICML,
pages 12888–12900. PMLR, 2022.
[13] Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi.
Blip-2: Bootstrapping language-image pre-training with
frozen image encoders and large language models. arXiv
preprint arXiv:2301.12597, 2023. 2, 4

18

Yufei Guo, Wesam Manassra, Prafulla Dhariwal, Casey Chu,
Yunxin Jiao, and Aditya Ramesh. Improving image generation with better captions. Technical report, OpenAI, 2023. 6,
7, 10, 16
[42] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan
Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren
Zhou. Qwen-vl: A frontier large vision-language model with
versatile abilities. arXiv preprint arXiv:2308.12966, 2023. 7
[43] Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz,
and Ron Kimmel. Fusecap: Leveraging large language models to fuse visual data into enriched image captions. arXiv
preprint arXiv:2305.17718, 2023. 7
[44] Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras,
and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718,
2021. 6, 10
[45] Yuechen Zhang, Jinbo Xing, Eric Lo, and Jiaya Jia. Realworld image variation by aligning diffusion inversion chain.
In NeurIPS, 2023. 8
[46] Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai
An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han,
and Baobao Chang. Mmicl: Empowering vision-language
model with multi-modal in-context learning, 2023. 10
[47] Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian
Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin
Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming
He, Hongsheng Li, and Yu Qiao. Sphinx: The joint mixing of
weights, tasks, and visual embeddings for multi-modal large
language models. https://github.com/AlphaVLLM/LLaMA2- Accessory/blob/main/SPHINX/
SPHINX_paper.pdf, 2023. 10
[48] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov,
Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner,
Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is
worth 16x16 words: Transformers for image recognition at
scale. In ICLR, 2021. 10

Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.
Llama 2: Open foundation and fine-tuned chat models. arXiv
preprint arXiv:2307.09288, 2023. 2
[29] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin,
Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul
Barham, Hyung Won Chung, Charles Sutton, Sebastian
Gehrmann, et al. Palm: Scaling language modeling with
pathways. arXiv preprint arXiv:2204.02311, 2022.
[30] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, pages 8748–8763. PMLR, 2021. 2, 7, 10
[31] Zhiliang Peng, Wenhui Wang, Li Dong, Yaru Hao, Shaohan
Huang, Shuming Ma, and Furu Wei. Kosmos-2: Grounding multimodal large language models to the world. arXiv
preprint arXiv:2306.14824, 2023. 2
[32] Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu,
Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang
Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue
Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He,
Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang.
Internlm-xcomposer: A vision-language large model for advanced text-image comprehension and composition, 2023. 2,
9
[33] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann,
Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk,
et al. Graph of thoughts: Solving elaborate problems with
large language models. arXiv preprint arXiv:2308.09687,
2023. 2
[34] Yuxi Xie, Kenji Kawaguchi, Yiran Zhao, Xu Zhao, MinYen Kan, Junxian He, and Qizhe Xie. Decomposition enhances reasoning via self-evaluation guided decoding. arXiv
preprint arXiv:2305.00633, 2023. 2
[35] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng
Gao, and Chunyuan Li. Llava-interactive: An all-in-one
demo for image chat, segmentation, generation and editing,
2023. 2
[36] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui
Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation
via large language model. arXiv preprint arXiv:2308.00692,
2023. 2
[37] Songwei Ge, Taesung Park, Jun-Yan Zhu, and Jia-Bin
Huang. Expressive text-to-image generation with rich text.
In ICCV, 2023. 3
[38] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and
Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. arXiv
preprint arXiv:2301.13826, 2023. 3
[39] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. NeurIPS, 34:8780–8794,
2021. 3
[40] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han,
and Mike Lewis. Efficient streaming language models with
attention sinks. arXiv, 2023. 4
[41] James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng
Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee,

19