Skip to content

Add autonomous agent control and MCP integration - #559

Merged
jamesmontemagno merged 7 commits into
mainfrom
motz-agent-app-integration
Nov 3, 2026
Merged

jamesmontemagno merged 7 commits into
mainfrom
motz-agent-app-integration

Conversation

@jamesmontemagno

Copy link
Copy Markdown
Owner

Summary

  • Add opt-in, scoped Agent Control to macOS and Windows, with pairing, audit logs, revocation, and approval for sensitive actions.
  • Provide autonomous capture, OCR, annotations, recording, library, and Studio workflows through a shared contract, a .NET CLI, and an MCP server using the official C# SDK.
  • Bundle five Skills and MCP configuration in a portable Agent Plugin, and add setup guidance to Settings, the Guide, and documentation.
  • Add cross-platform CLI/package smoke CI, shared-contract triggers for both app builds, and a separate agent-release.yml NuGet Trusted Publishing workflow. Ignore generated outputs, local IDE state, and optional agent-local credentials.

Validation

  • dotnet test agent/TinyClips.Agent.slnx -c Release --no-build — 41 tests passed.
  • Packaged CLI and MCP smoke tests using the official SDK client — 3 tests passed.
  • macOS unit tests and Debug builds of both TinyClips and TinyClipsMAS passed with signing disabled.
  • Workflow lint, plugin schema validation, trigger checks, and gitignore checks passed.
  • Native Windows WinUI build and tests could not run on this macOS host; the Windows CI jobs need to validate them on a Windows runner.

Publishing

Configure the NuGet.org Trusted Publishing policy for jamesmontemagno/tiny-clips, workflow file agent-release.yml, no environment, and package TinyClips. Allow new packages and new versions. Merging this PR does not publish; an agent-v* tag triggers the dedicated release workflow.

Relates to jamesmontemagno/tiny-clips-website#19.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Copilot AI balanced review requested due to automatic review settings November 1, 2026 15:14

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved critical issues allow cancelled requests to delete or upload files, automation edits to merge with gesture undo groups, and concurrent annotation requests to overwrite each other’s output.

8 open issues
What changed in this PR

Adds opt-in agent control to TinyClips, connecting the macOS and Windows apps to a shared .NET CLI, MCP server, and portable Agent Plugin.

Changes:

  • Adds scoped access, pairing approvals, audit logging, revocation, and automation hooks for capture and Studio workflows.
  • Packages contract-driven CLI/MCP tools with five Skills and in-app setup guidance.
  • Adds policy, integration, and package checks, plus cross-platform CI and separate NuGet publishing.
File Description
windows/​tests/​TinyClips.Core.Tests/​TinyClips.Core.Tests.csproj Copies shared automation fixtures.
windows/​tests/​TinyClips.Core.Tests/​StudioEditorSessionEditTests.cs Tests saved, undoable agent edits.
windows/​tests/​TinyClips.Core.Tests/​AutomationSettingsCatalogTests.cs Tests settings access and validation.
windows/​tests/​TinyClips.Core.Tests/​AutomationPolicyFixtureTests.cs Checks shared permission decisions.
windows/​tests/​TinyClips.Core.Tests/​AutomationContractTests.cs Tests contract scopes and parameters.
windows/​tests/​TinyClips.Core.Tests/​AutomationClientStoreTests.cs Tests tokens, revocation, and audit rotation.
windows/​src/​TinyClips.Core/​TinyClips.Core.csproj Embeds the shared command contract.
windows/​src/​TinyClips.Core/​Studio/​StudioEditorModel.cs Applies agent updates through undo history.
windows/​src/​TinyClips.Core/​Studio/​Editing/​StudioEditorSession.cs Adds revision-checked agent edits.
windows/​src/​TinyClips.Core/​Automation/​IAutomationCommandHandler.cs Defines handler and approval interfaces.
windows/​src/​TinyClips.Core/​Automation/​AutomationProtocol.cs Defines JSON requests and responses.
windows/​src/​TinyClips.Core/​Automation/​AutomationPolicy.cs Enforces scopes and approval requirements.
windows/​src/​TinyClips.Core/​Automation/​AutomationPipeServer.cs Serves requests over a per-user pipe.
windows/​src/​TinyClips.Core/​Automation/​AutomationErrorCodes.cs Defines protocol errors and session commands.
windows/​src/​TinyClips.Core/​Automation/​AutomationContract.cs Loads shared command definitions.
windows/​src/​TinyClips.Core/​Automation/​AutomationClientStore.cs Persists paired clients and token hashes.
windows/​src/​TinyClips.Core/​Automation/​AutomationCaptureTargetRegistry.cs Maps capture handles to opaque IDs.
windows/​src/​TinyClips.Core/​Automation/​AutomationCaptureRequest.cs Validates explicit capture targets.
windows/​src/​TinyClips.Core/​Automation/​AutomationAuditLog.cs Writes rotating request audit logs.
windows/​src/​TinyClips.App/​Views/​SettingsWindow.xaml.cs Creates the Agent Control settings section.
windows/​src/​TinyClips.App/​Views/​SettingsWindow.xaml Adds Agent Control navigation.
windows/​src/​TinyClips.App/​Views/​ScreenshotEditorWindow.xaml.cs Exposes image encoding for automation.
windows/​src/​TinyClips.App/​Views/​GuideWindow.xaml Adds agent setup guidance.
windows/​src/​TinyClips.App/​Views/​AutomationApprovalWindow.cs Presents cancellable approval prompts.
windows/​src/​TinyClips.App/​ViewModels/​Studio/​StudioViewModel.cs Exposes Studio automation operations.
windows/​src/​TinyClips.App/​ViewModels/​AgentControlSettingsViewModel.cs Manages access, clients, and setup commands.
windows/​src/​TinyClips.App/​Settings/​SettingsSectionKind.cs Adds the Agent Control section identifier.
windows/​src/​TinyClips.App/​Services/​Studio/​StudioWindowService.cs Provides access to open Studio editors.
windows/​src/​TinyClips.App/​Services/​MediaDevicePermissionService.cs Exposes current camera and microphone permissions.
windows/​src/​TinyClips.App/​Services/​IMediaDevicePermissionService.cs Adds permission-status properties.
windows/​src/​TinyClips.App/​Services/​Automation/​IAutomationAppActions.cs Defines automation-facing app actions.
windows/​src/​TinyClips.App/​Services/​Automation/​AutomationApprovalPrompt.cs Serializes pairing and sensitive-action prompts.
windows/​src/​TinyClips.App/​Services/​Automation/​AppAutomationCommandHandler.Capture.cs Implements targeting, OCR, and annotations.
windows/​src/​TinyClips.App/​Services/​Automation/​AgentControlService.cs Owns agent access and pipe lifecycle.
windows/​src/​TinyClips.App/​Controls/​Settings/​AgentControlSettingsSection.xaml.cs Connects settings UI and lifecycle cleanup.
windows/​src/​TinyClips.App/​Controls/​ScreenshotEditor/​EditorController.cs Renders annotations without an editor window.
windows/​CHANGELOG.md Documents Windows agent capabilities.
shared/​automation/​README.md Explains shared automation contracts.
shared/​automation/​fixtures/​policy.json Supplies cross-platform permission cases.
plugins/​tinyclips/​skills/​tinyclips-studio/​SKILL.md Guides Studio editing and export.
plugins/​tinyclips/​skills/​tinyclips-setup/​SKILL.md Guides installation and pairing.
plugins/​tinyclips/​skills/​tinyclips-record-demo/​SKILL.md Guides recording workflows.
plugins/​tinyclips/​skills/​tinyclips-library/​SKILL.md Guides capture-library operations.
plugins/​tinyclips/​skills/​tinyclips-capture/​SKILL.md Guides capture, OCR, and annotations.
plugins/​tinyclips/​README.md Documents plugin setup and installation.
plugins/​tinyclips/​plugin.json Declares portable plugin metadata.
plugins/​tinyclips/​mcp.json Configures the stdio MCP launch.
plugins/​tinyclips/​LICENSE Supplies the plugin’s MIT license.
mac/​TinyClipsTests/​AutomationPolicyTests.swift Tests policy and contract consistency.
mac/​TinyClipsTests/​AutomationClientStoreTests.swift Tests pairing, storage, and audit behavior.
mac/​TinyClips/​Views/​Studio/​StudioWindow.swift Exposes open Studio models to automation.
mac/​TinyClips/​Views/​Studio/​StudioViewModel.swift Adds agent edits and export preparation.
mac/​TinyClips/​Views/​SettingsView.swift Adds the Agent Control settings tab.
mac/​TinyClips/​Views/​Settings/​AgentControlSettingsSection.swift Adds access controls and setup instructions.
mac/​TinyClips/​Views/​MenuBarViews.swift Shows access status and a disable action.
mac/​TinyClips/​Views/​GuideWindow.swift Adds agent setup to the Guide.
mac/​TinyClips/​TinyClipsApp.swift Connects automation to app startup and shutdown.
mac/​TinyClips/​Studio/​StudioEditorModel.swift Applies editable agent fields through undo history.
mac/​TinyClips/​Capture/​WindowSelector.swift Exposes system-window filtering identifiers.
mac/​TinyClips/​Capture/​ScreenshotCapture.swift Separates window-image capture from saving.
mac/​TinyClips/​Automation/​AutomationSettings.swift Defines allowed settings and value validation.
mac/​TinyClips/​Automation/​AutomationPolicy.swift Implements scoped permission decisions.
mac/​TinyClips/​Automation/​AutomationContract.swift Loads shared commands and error definitions.
mac/​TinyClips/​Automation/​AutomationApproval.swift Queues timed in-app approval prompts.
CONTRIBUTING.md Documents agent CI and build triggers.
CHANGELOG.md Records shared and macOS agent features.
agent/​TinyClips.Agent.slnx Defines the CLI development solution.
agent/​tests/​TinyClips.Package.Tests/​TinyClips.Package.Tests.csproj Configures separate package smoke tests.
agent/​tests/​TinyClips.Cli.Tests/​TinyClips.Cli.Tests.csproj Configures CLI tests and fixtures.
agent/​tests/​TinyClips.Cli.Tests/​PluginTests.cs Checks plugin metadata and Skills.
agent/​tests/​TinyClips.Cli.Tests/​McpServerTests.cs Tests official SDK interoperability.
agent/​tests/​TinyClips.Cli.Tests/​FakeApp.cs Supplies a transport-level fake app.
agent/​tests/​TinyClips.Cli.Tests/​ContractTests.cs Tests schemas, validation, and scopes.
agent/​tests/​TinyClips.Cli.Tests/​CliTests.cs Tests CLI pairing and command behavior.
agent/​src/​TinyClips.Cli/​TinyClips.Cli.csproj Packages the CLI and MCP server.
agent/​src/​TinyClips.Cli/​Program.cs Adds the cancellable CLI entry point.
agent/​src/​TinyClips.Cli/​McpServer.cs Exposes contract commands as MCP tools.
agent/​src/​TinyClips.Cli/​Contract.cs Builds and validates command schemas.
agent/​src/​TinyClips.Cli/​CliEnvironment.cs Resolves endpoints and configuration paths.
agent/​src/​TinyClips.Cli/​Cli.cs Implements commands, pairing, and JSON output.
agent/​src/​TinyClips.Cli/​AutomationClient.cs Sends requests through sockets or pipes.
agent/​src/​TinyClips.Cli/​AgentConfig.cs Stores CLI pairing configuration.
agent/​src/​TinyClips.Cli/​.mcp/​server.json Declares MCP registry metadata.
agent/​README.md Documents usage, testing, and publishing.
agent/​.gitignore Excludes generated output and local configuration.
.github/​workflows/​windows-build.yml Adds shared automation and configuration triggers.
.github/​workflows/​build.yml Adds automation triggers and manual dispatch.
.github/​workflows/​agent-release.yml Publishes validated, tag-versioned agent packages.
.github/​workflows/​agent-build.yml Builds and smoke-tests packages across platforms.

Review effort: Balanced

Comment thread agent/src/TinyClips.Cli/AgentConfig.cs Resolved
jamesmontemagno and others added 2 commits November 1, 2026 07:28
Preserve the required Windows checks alongside agent CI triggers.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
editor.trim trims videos and GIFs (optional audio removal) and editor.saveFrame
exports a still, on macOS and Windows, without opening a trimmer. Outputs are new
library items; sources are never modified. Stale Studio edits now return conflict.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

jamesmontemagno and others added 3 commits November 1, 2026 23:46
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Preserve agent-control release notes alongside upstream Studio window pointer fixes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Track NuGet package metadata and workflow state, and publish version-matched agent tags after explicit confirmation.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@jamesmontemagno
jamesmontemagno merged commit de78c26 into main Nov 3, 2026
16 checks passed
@jamesmontemagno
jamesmontemagno deleted the motz-agent-app-integration branch November 3, 2026 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants