Repository navigation
test(evals): add cross-system Slack customer flow - #749
Merged
Merged
Conversation
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Repository UI Review profile: CHILL Plan: Team Run ID: 📒 Files selected for processing (2)
Included review availability: Your plan provides up to 8 included reviews per hour; 4 remain after this review. 📝 WalkthroughWalkthroughThe change adds Slack-style messaging emulation to evaluation trials, introduces Salesforce and Zendesk customer-support fixtures and tools, adds a grounded Slack reply evaluation case, and runs it in the integration suite. ChangesCustomer-support evaluation
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: ⚪ Minimal · up to This adds a Slack-style customer-support evaluation with synthetic Salesforce and Zendesk data and verifies threaded, grounded replies. No merge-blocking current-head risk remains. Sequence Diagram(s)sequenceDiagram
participant ScenarioRunner
participant MessagingTeamChatEmulator
participant EvalApp
participant EvalServices
ScenarioRunner->>MessagingTeamChatEmulator: Emit Slack inbound message
MessagingTeamChatEmulator->>EvalApp: Deliver inbound event
EvalApp->>EvalServices: Request Salesforce and Zendesk records
EvalApp->>MessagingTeamChatEmulator: Send grounded thread reply
ScenarioRunner->>MessagingTeamChatEmulator: Wait for outbound delivery
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
Greptile SummaryThe PR adds a deterministic cross-system customer-support evaluation that receives a Slack-style message, retrieves matching Salesforce and Zendesk records, and delivers a grounded response to the originating thread.
|
| Filename | Overview |
|---|---|
| packages/adapters/src/messaging-team-chat-emulator.ts | Implements the complete messaging surface and now correctly treats unsupported typing and foreign-provider threads as no-ops. |
| packages/testkit/src/evals/runner.ts | Wires deterministic inbound messaging, bot linking, run discovery, outbound-delivery waiting, and messaging evidence into eval trials. |
| packages/testkit/src/evals/cases.ts | Adds the Slack customer-update scenario and grades cross-system reads, grounded content, thread delivery, distractor avoidance, and read-only behavior. |
| packages/testkit/src/evals/services.ts | Adds coherent synthetic Salesforce and Zendesk providers, tools, records, and similar-name distractors. |
| packages/testkit/src/eval-customer-support.postgres.test.ts | Adds an offline replay through the API, database, agent runtime, tool execution, messaging mirror, and outbound delivery paths. |
Sequence Diagram
sequenceDiagram
participant Colleague as Slack colleague
participant Messaging as Messaging emulator
participant App as Rakazo API / agent
participant SF as Salesforce emulator
participant ZD as Zendesk emulator
Colleague->>Messaging: Ask about Fairhaven Robotics
Messaging->>App: Emit inbound DM
App->>SF: Search account and list opportunities
SF-->>App: Owner, renewal stage, and next step
App->>ZD: Search organization and list tickets
ZD-->>App: Urgent SSO ticket and latest update
App->>Messaging: Send grounded response to origin thread
Messaging-->>Colleague: Record outbound Slack reply
Reviews (2): Last reviewed commit: "fix(evals): preserve best-effort typing ..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
Contributor
Playwright screenshotsNo new feature frames; gallery is suite-vs-main drift. Open screenshot gallery · Dashboard · CI run Updated for commit |
This branch was previously deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The eval suite did not verify a realistic inbound messaging task that requires grounding an answer across multiple business systems and returning it to the original conversation. This adds that coverage without making local or CI verification depend on Slack, Salesforce, Zendesk, Era, or another hosted service.
What changed
Testing
pnpm checkpnpm test(3,472 passed; 156 skipped)pnpm test:integration(all 19 isolated suites passed)pnpm test:evals --listA live model trial was not run locally because it requires an inference-provider credential. The deterministic Pi replay exercises the complete orchestration path. Repository-wide
pnpm lintstill reports existing diagnostics in files unchanged by this PR; every changed source file passes Biome.Summary by CodeRabbit
New Features
Documentation