Quickstart: Optimize a hosted agent (preview)

• azd CLI (Azure Developer CLI).

• Azure CLI for authentication.

• The microsoft.foundry extension for azd (0.1.40-preview or later of the azure.ai.agents dependency):

  azd ext install microsoft.foundry
  

  If already installed, upgrade:

  azd ext upgrade microsoft.foundry
  

• Azure CLI for authentication.

• Python 3.10 or later.

• The Python packages used in this path:

  pip install "azure-ai-projects>=2.4.0" azure-ai-agentserver-optimization azure-identity python-dotenv
  

• An existing Foundry project that already contains the hosted agent, registered dataset, and evaluator you want to use for optimization.

• Visual Studio Code.

• Microsoft Foundry Toolkit for Visual Studio Code version 1.6.4 or later, signed in to Azure. Foundry Toolkit installs and updates the Microsoft Foundry Skill used by this workflow.

• GitHub Copilot in Visual Studio Code with access to agent mode. Foundry Toolkit sends the Agent Optimizer request to GitHub Copilot after you select the agent workspace.

• Azure CLI and Azure Developer CLI (AZD) installed and authenticated:

  az login
  azd auth login
  

• A coding agent host with the Microsoft Foundry Skill installed.

• Azure CLI and Azure Developer CLI (AZD) installed and authenticated:

  az login
  azd auth login
  

• The microsoft.foundry extension for AZD. Install it before you start the workflow:

  azd ext install microsoft.foundry
  

  If it's already installed, upgrade it:

  azd ext upgrade microsoft.foundry
  

Step 1: Create the project

Initialize a new project from the optimization sample template:

mkdir my-agent && cd my-agent
azd ai agent init -m https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/responses/optimization-customer-support/azure.yaml .

This template imports the Optimization Customer Support sample, an optimization-ready Python hosted agent that uses the bring-your-own approach and the Responses protocol. It represents a consumer electronics support agent that handles order inquiries, returns, warranty claims, troubleshooting, complaints, recommendations, and escalation. The deliberately minimal baseline instruction makes the improvements from instruction optimization and skill discovery easy to compare.

The sample calls load_config() to load baseline or candidate configuration and includes .agent_configs/baseline/, eval.yaml, full and quick evaluation datasets, container configuration, and the Foundry deployment manifest. The interactive flow imports these files and prompts for your Azure subscription, region, and model deployment settings.

Step 2: Provision and deploy

Authenticate and provision the Azure resources:

az login
azd auth login
azd provision

Provisioning takes approximately two minutes and creates a Foundry account, project, Azure Container Registry, and model deployments.

Deploy the agent:

azd deploy

Test the deployment:

azd ai agent invoke "What is 2+2?"

Step 3: Generate evaluation suite and optimize

Generate an evaluation dataset and evaluators for your agent:

azd ai agent eval generate

This step creates eval.yaml, a test dataset, and scoring evaluators based on your agent's instructions. The optimizer uses these files to measure improvement.

Run the optimizer:

azd ai agent optimize --max-candidates 2

The CLI prompts you to select an optimization model. To skip the prompt, pass it directly:

azd ai agent optimize --max-candidates 2 --optimize-model gpt-5

The CLI detects your agent from azure.yaml and uses the generated eval.yaml automatically. With two candidates, optimization typically completes in about 8 minutes. Real-time progress is shown:

Optimizing agent "customer-support-py"...
  Config: eval.yaml
  Baseline saved to .agent_configs/baseline/metadata.yaml
  Job ID: opt_162bd0f09....
  Status: pending
  Portal: <OPTIMIZATION-JOB-URL>

Use the portal URL to monitor your job in the Foundry portal.

The eval model scores each response (any chat-completion model works). The optimization model (--optimize-model) generates improved candidates and must be from the supported list (gpt-5 family or DeepSeek). You can also set optimization_model under options: in eval.yaml to avoid passing the flag each time.

Step 4: Deploy the winner

The star (*) in the output indicates the best candidate. Apply the optimized config locally, then deploy:

azd ai agent optimize apply --candidate <candidate-id>
azd deploy

The apply command downloads the optimized configuration into .agent_configs/<candidate_id>/ and updates your azure.yaml to use the new instructions. The deploy command pushes the optimized agent live using code deploy.

Invoke your agent to verify the improvement:

azd ai agent invoke "What is your return policy?"

You can also run evaluation to confirm the score improvement:

azd ai agent eval run

Python SDK path

Use the following steps if you want to run the optimizer from Python instead of the Azure Developer CLI workflow described earlier.

This path assumes you already have the following resources in an existing Foundry project:

• A hosted agent to optimize.
• A registered training dataset.
• A registered evaluator.

Unlike the Azure Developer CLI flow described earlier, the Python SDK path doesn't scaffold a project or generate eval.yaml, a dataset, or evaluators for you. If you want the sample to create those assets automatically, use azd ai agent eval generate first.

1. Create a .env file

Create a working folder, and then add a .env file with these values:

FOUNDRY_PROJECT_ENDPOINT=<your-project-endpoint>
FOUNDRY_AGENT_NAME=<your-hosted-agent-name>
DATASET_NAME=<your-registered-dataset-name>
EVALUATOR_NAME=<your-registered-evaluator-name>
DATASET_VERSION=1
POLL_INTERVAL_SECONDS=10
EVAL_MODEL=<your-eval-model-deployment-name>
OPTIMIZATION_MODEL=<your-optimization-model-deployment-name>

Run the script from this same working folder so load_dotenv() can load the .env file automatically. If you prefer to run it from another directory, set the same values in your shell environment first.

Use the exact project endpoint from your Foundry project's Overview page. The Python script sends its first request immediately. If FOUNDRY_PROJECT_ENDPOINT is only a placeholder or points to the wrong project, the run fails with ResourceNotFound: The project does not exist.

Set EVAL_MODEL and OPTIMIZATION_MODEL to deployment names that already exist in your Foundry project, not just model family names. For example, if your project deployment is named gpt-4.1-mini or DeepSeek-V3.2, use that exact deployment name in .env.

2. Run the optimization job

Create a file named optimize_hosted_agent.py in the same folder as .env:

import os
import time

from azure.ai.agentserver.optimization import load_config
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import (
  OptimizationAgentIdentifier,
  OptimizationEvaluatorRef,
  OptimizationJob,
  OptimizationJobInputs,
  OptimizationOptions,
  OptimizationReferenceDatasetInput,
)
from azure.identity import DefaultAzureCredential
from dotenv import load_dotenv

load_dotenv()

endpoint = os.environ["FOUNDRY_PROJECT_ENDPOINT"]
agent_name = os.environ["FOUNDRY_AGENT_NAME"]
dataset_name = os.environ["DATASET_NAME"]
evaluator_name = os.environ["EVALUATOR_NAME"]
dataset_version = os.environ.get("DATASET_VERSION", "1")
eval_model = os.environ.get("EVAL_MODEL", "gpt-4o")
optimization_model = os.environ.get("OPTIMIZATION_MODEL", "gpt-5")
poll_interval_seconds = int(os.environ.get("POLL_INTERVAL_SECONDS", "10"))

optimization_config = load_config() # Reads agent optimization config from .agent_configs/baseline/metadata.yaml

with (
  DefaultAzureCredential() as credential,
  AIProjectClient(endpoint=endpoint, credential=credential) as project_client,
):
  job = OptimizationJob(
    inputs=OptimizationJobInputs(
      agent=OptimizationAgentIdentifier(agent_name=agent_name),
      train_dataset=OptimizationReferenceDatasetInput(
        name=dataset_name,
        version=dataset_version,
      ),
      evaluators=[OptimizationEvaluatorRef(name=evaluator_name)],
      options=OptimizationOptions(
        max_candidates=2,
        eval_model=eval_model,
        optimization_model=optimization_model,
        optimization_config={
          "system_prompt": optimization_config.instructions,
          **({"tools": optimization_config.tool_definitions} if optimization_config.tool_definitions else {}),
          **({"skills": optimization_config.skills} if optimization_config.has_skills else {}),
        }
      ),
    )
  )
  poller = project_client.beta.agents.begin_create_optimization_job(job=job)

  print(f"Optimization job started, waiting for completion...")
  while not poller.done():
    print(f"\tstatus=`{poller.status()}`")
    time.sleep(poll_interval_seconds)

  result = poller.result()

  if result:
    print(f"Baseline candidate: {result.baseline}")
    print(f"Best candidate: {result.best}")

    for candidate in result.candidates or []:
      print(
        f"{candidate.name}: candidate_id={candidate.candidate_id}, "
        f"avg_score={candidate.avg_score:.4f}, "
        f"avg_tokens={candidate.avg_tokens:.0f}"
      )

Run the script:

python optimize_hosted_agent.py

When the job succeeds, the script prints the winning candidate and its candidate_id.

Unlike azd ai agent optimize, the Python SDK flow doesn't create a local .agent_configs/baseline/metadata.yaml file. The optimization job metadata stays in the returned job object and in the Foundry service response, including the baseline candidate, best candidate, and scored candidate list.

3. Apply the winning candidate

If you're also working from the local azd project used in the CLI flow above, apply the winning candidate by using the candidate_id returned by the Python script:

azd ai agent optimize apply --candidate <candidate-id>
azd deploy

If you only need to inspect the result, use the candidate scores and evaluation identifiers printed by the script to review the winning configuration in Foundry before promoting it.

Run the optimization in VS Code

Foundry Toolkit includes a native Agent Optimization experience for deployed hosted agents. From the agent playground, you can start an optimization run, compare candidates with the baseline, inspect configuration changes, and deploy the best candidate.

Step 1: Select a deployed hosted agent

1. Select Foundry Toolkit in the Activity Bar.
2. Under My Resources, select Agents.
3. If you have a deployed hosted agent, select it to open the hosted agent playground.
4. If you don't have a deployed hosted agent, complete the VS Code path in Quickstart: Deploy your first hosted agent. After deployment finishes, return to Agents and select the new hosted agent.

Step 2: Start an optimization run

1. Select the Optimize tab, which is marked Preview.

1. Select New Optimization.

2. In Select Workspace, choose the workspace that contains the selected hosted agent's code:

   • Select Current workspace if the current workspace contains the agent code and its azure.yaml file.
   • Select Browse... to open the workspace that contains the agent code.

   Foundry Toolkit uses the workspace files to prepare the optimization and to apply a candidate to the matching azure.ai.agent service.

1. Foundry Toolkit opens GitHub Copilot Chat and sends an Agent Optimizer request populated with the selected agent's kind, name, and Foundry project endpoint.

2. Answer the four optimization questions in Copilot Chat:

   Input	What to provide
   Evaluation metrics	Enter the metrics or evaluators to use. If you don't have them, choose whether to run azd ai agent eval generate or use the optimizer's built-in defaults.
   Dataset	Select the optimization dataset. If you don't have one, choose whether to run azd ai agent eval generate or use the optimizer's built-in defaults.
   Maximum candidates	Enter the maximum number of candidates to generate, such as 2.
   Optimization model	Select an existing deployment from the supported optimization models.

GitHub Copilot waits for these inputs before it starts optimization. The generated request directs Copilot to use the Microsoft Foundry Skill's Agent Optimizer workflow and Azure Developer CLI commands exclusively. It doesn't use Foundry MCP tools. Copilot:

• Inspects the agent code in the selected workspace.
• Initializes an AZD environment from existing azure.yaml and .env values if the project doesn't already have one.
• Wires the agent for optimization and deploys the updated hosted agent.
• Creates eval.yaml in the agent service folder.
• Starts the optimization after you review and approve the proposed file changes and commands.

After Copilot submits the job, return to the Optimize tab. The run appears under Optimization runs. The table shows its run ID, status, candidate count, baseline score, best score, and creation time.

Step 3: Compare and deploy the best candidate

1. When the run succeeds, select it under Optimization runs.
2. Compare the Baseline and Best scores. Review Score details for each candidate, and select View changes to inspect its configuration changes.
3. If the best candidate improves on the baseline, select Deploy best candidate to update the current agent. To deploy it as a new agent or change deployment settings, select Custom deploy instead.

Run the optimization with the Microsoft Foundry Skill

Use this path in any coding agent host that supports the Microsoft Foundry Skill, such as GitHub Copilot in Visual Studio Code, Copilot CLI, or Claude Code. The skill resolves the agent context from azure.yaml, loads its Agent Optimizer workflow, and keeps candidate application and deployment behind review gates.

Step 1: Open the agent workspace

Open an empty folder in your coding agent host. Confirm that the microsoft-foundry skill is available. If the skill isn't available, follow Use the Microsoft Foundry Skill in coding agents.

Step 2: Ask the skill to run Agent Optimizer

Submit this prompt to your coding agent:

Use the Microsoft Foundry Skill to run the Agent Optimizer workflow for a
Python hosted agent. If this workspace doesn't contain an agent, initialize the
customer support optimization sample from this template:
https://github.com/microsoft-foundry/foundry-samples/blob/main/samples/python/hosted-agents/bring-your-own/responses/optimization-customer-support/azure.yaml
Resolve the AZD environment and hosted-agent service, verify that the agent is
optimizer-ready, and deploy and invoke the baseline. Generate and show me the
evaluation dataset, evaluators, and eval.yaml before running optimization.
Verify that the project has a supported optimization model deployment, then run
Agent Optimizer with two candidates. Stop after reporting the operation ID,
portal URL, candidate IDs, and scores. Don't apply or deploy a candidate yet.

The coding agent might ask you to select a subscription, region, Foundry project, agent service, evaluation model, or optimization model when it can't resolve those values from the workspace. Review generated files and cost-bearing resources before you approve any changes or commands.

Step 3: Apply and deploy an approved candidate

After you review the optimization results, submit this follow-up prompt:

Recommend the best optimization candidate and explain the score improvement.
Summarize the candidate changes before applying anything. After I approve the
candidate, apply it locally, show the source diff, and stop again before
deployment. After I approve deployment, run azd deploy, invoke the agent with
"What is your return policy?", and rerun the evaluation to confirm the
improvement.

The skill uses azd ai agent optimize apply --candidate <candidate-id> so you can review the optimized configuration locally. It deploys only after your approval, then invokes and evaluates the updated hosted agent.