stream_infer input_embeddings

[JidongZhang-THU]

how could I use the input_embeddings in stream_infer? Was it designed for mix embedding?

[lvhan028]

@irexyc please help clarifying this question

[irexyc]

For some tasks like qwen-vl or internlm-xcomposer, the decode process are same with normal llm. The only difference is the embedding layer. Normal llm use embedding layer to encode token_ids to input_embs. These multimodal model concat the image features and input_embs as final input.

To make the code simpler, we add dummy ids to token_ids and after embedding layer, we replace that dummy embeddings with real image features.

This is a web demo #874

[JidongZhang-THU]

Thanks for reply.
I was using adaptor model to combine text and img input to inputs_embeds. Model loaded by AutoModelForCausalLM.from_pretrained can take the inputs_embeds by generate function which is defined in GenerationMixin.
I want to know how I can use the stream_infer to fulfill the same inference by generate function.

1. I have adapted text and img to tensor, so I don't need input_ids
2. I don't know how to use input_embeddings and input_embedding_ranges

[JidongZhang-THU]

According to #874 , I wrote a test.
stream_infer(input_ids=[0] * input_feature.shape[0], session_id=0,
input_embeddings=[input_feature], input_embedding_ranges=[(0, input_feature.shape[0])])
'#' input_feature.shape (N, 4096)
It seems that works for me.