Qwen3.8 27B MLX 8bit

Hermes tool-contract hard v1: 50 cases. Primary score is tool/state pass rate. tool_state=60%, strict=60%, final_text=90%, schema_valid=100%, required_tool=78%, hallucinated_tool=0%, forbidden_tool=0%, recovery=100.0%. avg_tool_calls=2.52, avg_turns=2.96, output_tps=1.9, input_tokens=101827, output_tokens=12576. non_text_failures: final_state_wrong=11, missing_required_tool=11, model_error=11, timeout=5, too_many_tool_calls=9. strict_failure_breakdown: final_answer_missing_expected_text=5, final_state_wrong=11, missing_required_tool=11, model_error=11, timeout=5, too_many_tool_calls=9. Local OMLX Qwen3.8 27B 8-bit run; provider/model errors and per-case timeouts are counted as failures.

Hermes tool-contract v0: 100 cases. Primary score is tool/state pass rate. tool_state=84%, strict=79%, final_text=83%, schema_valid=100%, required_tool=98%, hallucinated_tool=0%, forbidden_tool=0%, recovery=57.1%. avg_tool_calls=1.62, avg_turns=2.31, output_tps=5.4, input_tokens=120641, output_tokens=21959. non_text_failures: final_state_wrong=5, max_tool_calls=12, missing_required_tool=2, should_not_have_called_tool=1. strict_failure_breakdown: final_answer_missing_expected_text=17, final_state_wrong=5, max_tool_calls=12, missing_required_tool=2, should_not_have_called_tool=1. Local OMLX Qwen3.8 27B 8-bit run.

Hermes tool-contract hard v1: 50 cases. Primary score is tool/state pass rate. tool_state=60%, strict=60%, final_text=90%, schema_valid=100%, required_tool=78%, hallucinated_tool=0%, forbidden_tool=0%, recovery=100.0%. avg_tool_calls=2.52, avg_turns=2.96, output_tps=1.9, input_tokens=101827, output_tokens=12576. non_text_failures: final_state_wrong=11, missing_required_tool=11, model_error=11, timeout=5, too_many_tool_calls=9. strict_failure_breakdown: final_answer_missing_expected_text=5, final_state_wrong=11, missing_required_tool=11, model_error=11, timeout=5, too_many_tool_calls=9. Local OMLX Qwen3.8 27B 8-bit run; provider/model errors and per-case timeouts are counted as failures.

Hermes tool-contract v0: 100 cases. Primary score is tool/state pass rate. tool_state=84%, strict=79%, final_text=83%, schema_valid=100%, required_tool=98%, hallucinated_tool=0%, forbidden_tool=0%, recovery=57.1%. avg_tool_calls=1.62, avg_turns=2.31, output_tps=5.4, input_tokens=120641, output_tokens=21959. non_text_failures: final_state_wrong=5, max_tool_calls=12, missing_required_tool=2, should_not_have_called_tool=1. strict_failure_breakdown: final_answer_missing_expected_text=17, final_state_wrong=5, max_tool_calls=12, missing_required_tool=2, should_not_have_called_tool=1. Local OMLX Qwen3.8 27B 8-bit run.