Is there a way to stop required parameters from being inferred and added to structured output?

Question

I'll use a synthetic example - the real one is more complex. Consider trying to validate a freeform address. You want to be sure the user provided all the information needed and if not prompt them to do so. Here's a system prompt for same:

@SystemMessage(""" Your role is to extract american postal address information ONLY from the user input message. If a piece of the required address information is not provided you MUST NOT infer it. Instead inform the user of the piece[s] of information that is/are missing. """)

Now consider input such as

The address is 123 East Street, Winchester, VA

that you want to be mapped into a data structure similar to this

@Description("Client Address") public static final class Address { @Description("apartment number") String apartmentNumber; @Description("street number") Integer streetNumber; @Description("street name") String street; @Description("city") String city; @Description("state") String state; @Description("zip code") String zip;

Ok that's probably more detail than you need. On to the actual question. I have a tool to validate the mapped object. By time it gets to the tool [one of] the four possible zip codes for the given address - Winchester, VA has been mapped into it, and the apartment number has been set to be the same as the street number. Which means I can't stop and ask that the zip be furnished.

If I provide less information - a partial address such as 100 Eat St for example, I get an address in Seattle, WA with an appropriate zip.

I've tried looking at the manual, the examples and googling, but can't seem to find a way to stop unwanted extraneous information being mapped into my object. I've tried numerous combinations of system message to attempt to guide the llm to not inject spurious information but with no success.

What am I missing?

Dharman · Accepted Answer · 2025-03-30 10:18:26Z

0

As a temporary (ugly, but might work) solution you could add boolean containsZip, boolean containsApartmentNumber, etc to the Address and ignore actual value if the flag is false

edited Mar 30 at 10:18

Dharman♦

33.9k27 gold badges106 silver badges157 bronze badges

answered Mar 11 at 17:29

LangChain4j

964 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

aatk Mar 11 at 18:16

Mistral-Nemo-Instruct-2407 inside vllm - comms through an openai server.

aatk Mar 11 at 19:53

And as an fyi - adding the booleans - even with @description that labor the point - are not deterministic. On some occasions they are correct, in others they return that the user provided the [field] even though it's quite clearly not there in the user message. There seems to be no pattern to it either.

LangChain4j Mar 12 at 9:25

You will never get determinism with LLMs, so try to find the strategy that maximizes the success rate. Regarding the model - try using a better one

Collectives™ on Stack Overflow

Is there a way to stop required parameters from being inferred and added to structured output?

1 Answer 1

3 Comments

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Related