Eval operators

The Eval family contains 4 operators. Grade outputs against assertions, regexes, and judge prompts.

Operators in this family

• eval.assertion
• eval.grade
• eval.regex
• eval.score

eval.assertion

Evaluates a CEL boolean expression against merged inputs and fails the step if it returns false.

Version 1.0.0 · Semantics passthrough · Idempotent

Config

Field	Type	Required	Default	Description
expression	string	yes	—	CEL expression returning a boolean. Inputs are bound under ctx.inputs.* and inputs.*.
target	string	no	—	Optional target value exposed as ctx.inputs.target.

Input ports

Port	Type	Required	Description
input	any	no	Input

Output ports

Port	Type	Required	Description
output	object	no	Output

eval.grade

Scores a response against a rubric using an LLM judge. Returns a score, justification, and per-criterion results.

Version 1.0.0 · Semantics passthrough · Secrets OPENROUTER_API_KEY

Config

Field	Type	Required	Default	Description
max_score	number	no	10
model	string	no	—	Model identifier (resolved against the customer catalog).
response	string	no	—	Response to evaluate. Falls back to inputs.output if omitted.
rubric	string	yes	—	Evaluation rubric or criteria. Embedded into the judge’s system prompt.

Input ports

Port	Type	Required	Description
input	string	no	Input

Output ports

Port	Type	Required	Description
output	object	no	Output

eval.regex

Native regex match/find_all/replace against a target string.

Version 1.0.0 · Semantics passthrough · Idempotent

Config

Field	Type	Required	Default	Description
mode	”match” | “find_all” | “replace”	no	"match"
pattern	string	yes	—	Regular expression pattern (Rust regex syntax).
replacement	string	no	—	Replacement string (required when mode=replace).
target	string	no	—	String to apply the pattern to. Falls back to inputs.output if omitted.

Input ports

Port	Type	Required	Description
input	string	no	Input

Output ports

Port	Type	Required	Description
output	object	no	Output

eval.score

Scores output quality against a reference or rubric.

Version 1.0.0 · Semantics passthrough · Visibility advanced · Secrets OPENROUTER_API_KEY

Config

Field	Type	Required	Default	Description
max_tokens	integer	no	4096
model	string	no	—	Model identifier (e.g. ‘deepseek/deepseek-r1’).
reference	string	no	—	Optional reference answer.
rubric	string	yes	—	Evaluation rubric or criteria.
stop_sequences	string[]	no	—
temperature	number	no	0.7
top_p	number	no	—

Input ports

Port	Type	Required	Description
input	string	yes	Input

Output ports

Port	Type	Required	Description
explanation	string	no	Explanation
score	number	no	Score
usage	object	no	Usage

Source schemas

Every operator above is rendered from its JSON schema in /opt/glyph/schemas/operators/:

• schemas/operators/eval.assertion.json
• schemas/operators/eval.grade.json
• schemas/operators/eval.regex.json
• schemas/operators/eval.score.json