platoseedplatoseed
Automated Multi-Modal Testing for Agents
Ashr is a fully contained test and evals platform that improves AI agents by ensuring accuracy and quality across a wide range of user journeys through your product. Ashr generates user journeys through your agents’ tool calls, results, and questions. We don’t generate one off tests, we generate large amounts of authentic user stories through your product, and pick up on errors, inconsistencies, and failures that would otherwise take hours of manual testing or get caught by a customer. We also work to ensure agent response quality in line with your business goals by offering custom evaluation metrics. Ashr works with voice, text, automated website generation, image, file generation, and multimodal. We’re also willing to generate custom input types or make modifications for anyone interested. Currently, we integrate as an SDK, and are proactively working to make our documentation and resources as accessible and easy to use as possible.
Ashr provides automated multi-modal testing and evals for AI agents, enabling replay of real environments and regression detection before users are impacted. It focuses on evaluating agent performance across scenarios such as lost item claims, flight rebooking, and refunds, with an emphasis on catching failures and shipping better AI agents.
Ashr runs evals in real environments, replays conversations and actions, and presents detailed test timelines, diffs, and prompts to pinpoint where agents failed. It tracks datasets, test details, exploration of expected vs actual results, versioned prompts, and offers an SDK (Python/TypeScript) to run evals from code, enabling end-to-end testing of agent interactions and workflows.
Who it’s for: AI product teams and operations teams building customer-support agents, particularly those handling multi-step service scenarios (e.g., item claims, rebooking, refunds).
Customer-facing product with evals for agents and references to docs/blog/book a call; mentions of running evals and platform features indicate traction and ongoing development.
built ai agents in finance and devops for the last two years. currently building ashr to work towards building better, more widely adopted agents.
Building AI powered workforce management software for small and medium sized businesses. Co-founder and CTO of Ask Geri (Acquired). Berkeley EECs.
Custom Tests and Evals Simulating Real User Behavior
Ashr is a modular testing and evaluation platform that integrates as an SDK to simulate real user behavior and assess agentic systems. It targets developers building AI-driven agents, providing test suites and custom scorers to detect hallucinations and ensure production-ready performance.
The eval platform for autonomous software
Armature helps companies monitor and optimize the agent experience.
