Hi! I maintain EvalPort, an open, model-agnostic spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets that can move between eval tools instead of every framework inventing its own file format.
I've been going through inference-client SDKs looking for the natural point where someone would want to save a batch of runs as a reusable eval fixture, and replicate-python's Prediction object is a good fit — it already carries input, output, status, error, metrics, and timestamps for every model call.
A small conversion helper (thinking an example, not core functionality) could look like:
from evalport import ResultSet, TestResult
def predictions_to_evalport(predictions: list[Prediction]) -> ResultSet:
return ResultSet(results=[
TestResult(
test_case_id=p.id,
input=p.input,
actual_output=p.output,
passed=p.status == "succeeded",
error=p.error,
metadata={"model": p.model, "version": p.version, **(p.metrics or {})},
)
for p in predictions
])That would let anyone already scripting a batch of replicate.predictions.create() calls (e.g. regression-testing a fine-tune, or comparing two model versions on the same prompt set) dump the results into a format other eval tools can read, without writing a one-off serializer each time.
Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
This is genuinely a "does this seem useful to you" check rather than a PR — happy to put one together if there's interest, and totally fine if it's not a fit for this library. Thanks for reading either way!
Hi! I maintain EvalPort, an open, model-agnostic spec (JSON Schema + Python/TS SDKs) for portable LLM eval datasets — test suites, test cases, and result sets that can move between eval tools instead of every framework inventing its own file format.
I've been going through inference-client SDKs looking for the natural point where someone would want to save a batch of runs as a reusable eval fixture, and replicate-python's
Predictionobject is a good fit — it already carriesinput,output,status,error,metrics, and timestamps for every model call.A small conversion helper (thinking an example, not core functionality) could look like:
That would let anyone already scripting a batch of
replicate.predictions.create()calls (e.g. regression-testing a fine-tune, or comparing two model versions on the same prompt set) dump the results into a format other eval tools can read, without writing a one-off serializer each time.Spec, for context: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
This is genuinely a "does this seem useful to you" check rather than a PR — happy to put one together if there's interest, and totally fine if it's not a fit for this library. Thanks for reading either way!