Repository navigation
Add autonomous agent control and MCP integration - #559
Merged
Merged
Conversation
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical issues allow cancelled requests to delete or upload files, automation edits to merge with gesture undo groups, and concurrent annotation requests to overwrite each other’s output.
8 open issues
Automation edits can join an active gesture undo group · New Concurrent exports can overwrite the same destination · New Approval handler ignores client disconnects · New Automation edits can join an active gesture undo group · New Relative config path can crash pairing and unpairing · New Connection failure prevents trying other endpoint candidates · New Report encoded screenshot dimensions, not capture dimensions · New Persistence interval resets on every command · New
What changed in this PR
Adds opt-in agent control to TinyClips, connecting the macOS and Windows apps to a shared .NET CLI, MCP server, and portable Agent Plugin.
Changes:
- Adds scoped access, pairing approvals, audit logging, revocation, and automation hooks for capture and Studio workflows.
- Packages contract-driven CLI/MCP tools with five Skills and in-app setup guidance.
- Adds policy, integration, and package checks, plus cross-platform CI and separate NuGet publishing.
| File | Description |
|---|---|
| windows/tests/TinyClips.Core.Tests/TinyClips.Core.Tests.csproj | Copies shared automation fixtures. |
| windows/tests/TinyClips.Core.Tests/StudioEditorSessionEditTests.cs | Tests saved, undoable agent edits. |
| windows/tests/TinyClips.Core.Tests/AutomationSettingsCatalogTests.cs | Tests settings access and validation. |
| windows/tests/TinyClips.Core.Tests/AutomationPolicyFixtureTests.cs | Checks shared permission decisions. |
| windows/tests/TinyClips.Core.Tests/AutomationContractTests.cs | Tests contract scopes and parameters. |
| windows/tests/TinyClips.Core.Tests/AutomationClientStoreTests.cs | Tests tokens, revocation, and audit rotation. |
| windows/src/TinyClips.Core/TinyClips.Core.csproj | Embeds the shared command contract. |
| windows/src/TinyClips.Core/Studio/StudioEditorModel.cs | Applies agent updates through undo history. |
| windows/src/TinyClips.Core/Studio/Editing/StudioEditorSession.cs | Adds revision-checked agent edits. |
| windows/src/TinyClips.Core/Automation/IAutomationCommandHandler.cs | Defines handler and approval interfaces. |
| windows/src/TinyClips.Core/Automation/AutomationProtocol.cs | Defines JSON requests and responses. |
| windows/src/TinyClips.Core/Automation/AutomationPolicy.cs | Enforces scopes and approval requirements. |
| windows/src/TinyClips.Core/Automation/AutomationPipeServer.cs | Serves requests over a per-user pipe. |
| windows/src/TinyClips.Core/Automation/AutomationErrorCodes.cs | Defines protocol errors and session commands. |
| windows/src/TinyClips.Core/Automation/AutomationContract.cs | Loads shared command definitions. |
| windows/src/TinyClips.Core/Automation/AutomationClientStore.cs | Persists paired clients and token hashes. |
| windows/src/TinyClips.Core/Automation/AutomationCaptureTargetRegistry.cs | Maps capture handles to opaque IDs. |
| windows/src/TinyClips.Core/Automation/AutomationCaptureRequest.cs | Validates explicit capture targets. |
| windows/src/TinyClips.Core/Automation/AutomationAuditLog.cs | Writes rotating request audit logs. |
| windows/src/TinyClips.App/Views/SettingsWindow.xaml.cs | Creates the Agent Control settings section. |
| windows/src/TinyClips.App/Views/SettingsWindow.xaml | Adds Agent Control navigation. |
| windows/src/TinyClips.App/Views/ScreenshotEditorWindow.xaml.cs | Exposes image encoding for automation. |
| windows/src/TinyClips.App/Views/GuideWindow.xaml | Adds agent setup guidance. |
| windows/src/TinyClips.App/Views/AutomationApprovalWindow.cs | Presents cancellable approval prompts. |
| windows/src/TinyClips.App/ViewModels/Studio/StudioViewModel.cs | Exposes Studio automation operations. |
| windows/src/TinyClips.App/ViewModels/AgentControlSettingsViewModel.cs | Manages access, clients, and setup commands. |
| windows/src/TinyClips.App/Settings/SettingsSectionKind.cs | Adds the Agent Control section identifier. |
| windows/src/TinyClips.App/Services/Studio/StudioWindowService.cs | Provides access to open Studio editors. |
| windows/src/TinyClips.App/Services/MediaDevicePermissionService.cs | Exposes current camera and microphone permissions. |
| windows/src/TinyClips.App/Services/IMediaDevicePermissionService.cs | Adds permission-status properties. |
| windows/src/TinyClips.App/Services/Automation/IAutomationAppActions.cs | Defines automation-facing app actions. |
| windows/src/TinyClips.App/Services/Automation/AutomationApprovalPrompt.cs | Serializes pairing and sensitive-action prompts. |
| windows/src/TinyClips.App/Services/Automation/AppAutomationCommandHandler.Capture.cs | Implements targeting, OCR, and annotations. |
| windows/src/TinyClips.App/Services/Automation/AgentControlService.cs | Owns agent access and pipe lifecycle. |
| windows/src/TinyClips.App/Controls/Settings/AgentControlSettingsSection.xaml.cs | Connects settings UI and lifecycle cleanup. |
| windows/src/TinyClips.App/Controls/ScreenshotEditor/EditorController.cs | Renders annotations without an editor window. |
| windows/CHANGELOG.md | Documents Windows agent capabilities. |
| shared/automation/README.md | Explains shared automation contracts. |
| shared/automation/fixtures/policy.json | Supplies cross-platform permission cases. |
| plugins/tinyclips/skills/tinyclips-studio/SKILL.md | Guides Studio editing and export. |
| plugins/tinyclips/skills/tinyclips-setup/SKILL.md | Guides installation and pairing. |
| plugins/tinyclips/skills/tinyclips-record-demo/SKILL.md | Guides recording workflows. |
| plugins/tinyclips/skills/tinyclips-library/SKILL.md | Guides capture-library operations. |
| plugins/tinyclips/skills/tinyclips-capture/SKILL.md | Guides capture, OCR, and annotations. |
| plugins/tinyclips/README.md | Documents plugin setup and installation. |
| plugins/tinyclips/plugin.json | Declares portable plugin metadata. |
| plugins/tinyclips/mcp.json | Configures the stdio MCP launch. |
| plugins/tinyclips/LICENSE | Supplies the plugin’s MIT license. |
| mac/TinyClipsTests/AutomationPolicyTests.swift | Tests policy and contract consistency. |
| mac/TinyClipsTests/AutomationClientStoreTests.swift | Tests pairing, storage, and audit behavior. |
| mac/TinyClips/Views/Studio/StudioWindow.swift | Exposes open Studio models to automation. |
| mac/TinyClips/Views/Studio/StudioViewModel.swift | Adds agent edits and export preparation. |
| mac/TinyClips/Views/SettingsView.swift | Adds the Agent Control settings tab. |
| mac/TinyClips/Views/Settings/AgentControlSettingsSection.swift | Adds access controls and setup instructions. |
| mac/TinyClips/Views/MenuBarViews.swift | Shows access status and a disable action. |
| mac/TinyClips/Views/GuideWindow.swift | Adds agent setup to the Guide. |
| mac/TinyClips/TinyClipsApp.swift | Connects automation to app startup and shutdown. |
| mac/TinyClips/Studio/StudioEditorModel.swift | Applies editable agent fields through undo history. |
| mac/TinyClips/Capture/WindowSelector.swift | Exposes system-window filtering identifiers. |
| mac/TinyClips/Capture/ScreenshotCapture.swift | Separates window-image capture from saving. |
| mac/TinyClips/Automation/AutomationSettings.swift | Defines allowed settings and value validation. |
| mac/TinyClips/Automation/AutomationPolicy.swift | Implements scoped permission decisions. |
| mac/TinyClips/Automation/AutomationContract.swift | Loads shared commands and error definitions. |
| mac/TinyClips/Automation/AutomationApproval.swift | Queues timed in-app approval prompts. |
| CONTRIBUTING.md | Documents agent CI and build triggers. |
| CHANGELOG.md | Records shared and macOS agent features. |
| agent/TinyClips.Agent.slnx | Defines the CLI development solution. |
| agent/tests/TinyClips.Package.Tests/TinyClips.Package.Tests.csproj | Configures separate package smoke tests. |
| agent/tests/TinyClips.Cli.Tests/TinyClips.Cli.Tests.csproj | Configures CLI tests and fixtures. |
| agent/tests/TinyClips.Cli.Tests/PluginTests.cs | Checks plugin metadata and Skills. |
| agent/tests/TinyClips.Cli.Tests/McpServerTests.cs | Tests official SDK interoperability. |
| agent/tests/TinyClips.Cli.Tests/FakeApp.cs | Supplies a transport-level fake app. |
| agent/tests/TinyClips.Cli.Tests/ContractTests.cs | Tests schemas, validation, and scopes. |
| agent/tests/TinyClips.Cli.Tests/CliTests.cs | Tests CLI pairing and command behavior. |
| agent/src/TinyClips.Cli/TinyClips.Cli.csproj | Packages the CLI and MCP server. |
| agent/src/TinyClips.Cli/Program.cs | Adds the cancellable CLI entry point. |
| agent/src/TinyClips.Cli/McpServer.cs | Exposes contract commands as MCP tools. |
| agent/src/TinyClips.Cli/Contract.cs | Builds and validates command schemas. |
| agent/src/TinyClips.Cli/CliEnvironment.cs | Resolves endpoints and configuration paths. |
| agent/src/TinyClips.Cli/Cli.cs | Implements commands, pairing, and JSON output. |
| agent/src/TinyClips.Cli/AutomationClient.cs | Sends requests through sockets or pipes. |
| agent/src/TinyClips.Cli/AgentConfig.cs | Stores CLI pairing configuration. |
| agent/src/TinyClips.Cli/.mcp/server.json | Declares MCP registry metadata. |
| agent/README.md | Documents usage, testing, and publishing. |
| agent/.gitignore | Excludes generated output and local configuration. |
| .github/workflows/windows-build.yml | Adds shared automation and configuration triggers. |
| .github/workflows/build.yml | Adds automation triggers and manual dispatch. |
| .github/workflows/agent-release.yml | Publishes validated, tag-versioned agent packages. |
| .github/workflows/agent-build.yml | Builds and smoke-tests packages across platforms. |
Review effort: Balanced
Preserve the required Windows checks alongside agent CI triggers. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
editor.trim trims videos and GIFs (optional audio removal) and editor.saveFrame exports a still, on macOS and Windows, without opening a trimmer. Outputs are new library items; sources are never modified. Stale Studio edits now return conflict. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Unresolved critical security flaws can expose bearer tokens, trigger unintended approvals, and allow sensitive operations to execute after access is revoked.
6 open issues
Restrict automation pipe connections to the current user · New Require deliberate input to approve sensitive requests · New Cancel pending approvals and recheck authorization after revocation · New Invalidate pending requests when a client is revoked · New Handle fractional GIF frame boundaries consistently · New Allow configurable timeouts for long-running media operations · New
8 resolved since last review
Automation edits can join an active gesture undo group Approval handler ignores client disconnects Concurrent exports can overwrite the same destination Automation edits can join an active gesture undo group Persistence interval resets on every command Report encoded screenshot dimensions, not capture dimensions Connection failure prevents trying other endpoint candidates Relative config path can crash pairing and unpairing
Review effort: Max
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve agent-control release notes alongside upstream Studio window pointer fixes. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Track NuGet package metadata and workflow state, and publish version-matched agent tags after explicit confirmation. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.


Summary
agent-release.ymlNuGet Trusted Publishing workflow. Ignore generated outputs, local IDE state, and optional agent-local credentials.Validation
dotnet test agent/TinyClips.Agent.slnx -c Release --no-build— 41 tests passed.Publishing
Configure the NuGet.org Trusted Publishing policy for
jamesmontemagno/tiny-clips, workflow fileagent-release.yml, no environment, and packageTinyClips. Allow new packages and new versions. Merging this PR does not publish; anagent-v*tag triggers the dedicated release workflow.Relates to jamesmontemagno/tiny-clips-website#19.