Make approximate assertions as "expectations" on test results.
This module is designed to be used within test cases decorated with the @pytest.mark.decorator decorator
It allows you to log scores about a test case and optionally make assertions that log as "expectation" feedback to LangSmith.
Example:
import pytest from langsmith import expect @pytest.mark.langsmith def test_output_semantically_close(): response = oai_client.chat.completions.create( model="gpt-3.5-turbo", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Say hello!"}, ], ) response_txt = response.choices[0].message.content # Intended usage expect.embedding_distance( prediction=response_txt, reference="Hello!", ).to_be_less_than(0.9) # Score the test case matcher = expect.edit_distance( prediction=response_txt, reference="Hello!", ) # Apply an assertion and log 'expectation' feedback to LangSmith matcher.to_be_less_than(1) # You can also directly make assertions on values directly expect.value(response_txt).to_contain("Hello!") # Or using a custom check expect.value(response_txt).against(lambda x: "Hello" in x) # You can even use this for basic metric logging within tests expect.score(0.8) expect.score(0.7, key="similarity").to_be_greater_than(0.7)Client for interacting with the LangSmith API.
Use the client to customize API keys / workspace connections, SSL certs, etc. for tracing.
Also used to create, read, update, and delete LangSmith resources such as runs (~trace spans), datasets, examples (~records), feedback (~metrics), projects (tracer sessions/groups), etc.
For detailed API documentation, visit the LangSmith docs.
Decorator for creating a run tree from functions.
Schemas for the LangSmith API.
Generic utility functions.