Skip to content

Packer 1.2.5 fails for AWS with 'ResourceNotReady: exceeded wait attempts' #6569

Closed

Description

@kmacoskey

I am shameless copying this issue but updating slightly: #5792

It may be related to #6526

I have a CentOS Packer build on AWS which works with version 1.1.2 but fails with version 1.2.5

The template used to build the image:

{
  "variables": {
    "packer_script_path": "",
    "force_deregister": "",
    "image_name": "",
    "aws_secret_key_id": "{{env `AWS_SECRET_KEY_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}",
    "source_ami": "ami-476e4c22",
    "ami_regions": "us-west-2",
    "ssh_username": "{{env `DEFAULT_PLATFORM_USER`}}"
  },
  "builders": [
    {
      "name": "aws",
      "type": "amazon-ebs",
      "region": "us-east-2",
      "source_ami": "{{user `source_ami`}}",
      "instance_type": "t2.medium",
      "ssh_username": "{{user `ssh_username`}}",
      "ssh_pty": "true",
      "run_tags": {
        "Source": "Packer",
      },
      "tags": {
        "Source": "Packer",
      },
      "ami_name": "{{user `image_name`}}",
      "ami_description": "{{user `image_name`}}",
      "force_deregister": "{{user `force_deregister`}}",
      "ami_regions": "{{user `ami_regions`}}",
      "shutdown_behavior": "terminate"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "execute_command": "{{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "scripts": [
        "{{user `packer_script_path`}}/install-packages.sh"
      ]
    },
    {
      "type": "shell",
      "execute_command": "{{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "scripts": [
        "{{user `packer_script_path`}}/configure-os.sh"
      ]
    },
    {
      "type": "shell",
      "execute_command": "{{ .Vars }} sudo -E -S sh '{{ .Path }}'",
      "scripts": [
        "{{user `packer_script_path`}}/set-kernel.sh"
      ]
    }
  ],
  "post-processors": [
    {
      "type": "manifest",
      "output": "image-metadata/aws-{{user `file_name`}}.json",
      "strip_path": true
    }
  ]
}

As you can see, I am not doing anything complicated.

When the build fails using version 1.2.5, it times out with the following error. This happens 75% of the time:

==> aws: 1 error(s) occurred:
==> aws: 
==> aws: * Error waiting for AMI (ami-192c0861) in region (us-west-2): ResourceNotReady: exceeded wait attempts
  • Packer version from packer version
    1.2.5

  • Host platform
    Alpine Linux 3.8.0

I am using this image from dockerhub, https://hub.docker.com/r/hashicorp/packer/, with a tag of light.

If I run the above template using v 1.1.2, it works fine.

Regard,
Kris

Activity

  1. kmacoskey commented on Aug 6, 2018

    @kmacoskey
    Author

    Setting the env var AWS_MAX_ATTEMPTS=150 has been a valid workaround for now.

  2. SwampDragons commented on Aug 6, 2018

    @SwampDragons
    Contributor

    I'm going to close this as a duplicate since it's caused by the same underlying issue as #6526. I'd like to track all instances of this bug in the same place. Thanks for sharing your information :)

  3. kmacoskey commented on Aug 6, 2018

    @kmacoskey
    Author

    The linked issue is already closed?

  4. SwampDragons commented on Aug 6, 2018

    @SwampDragons
    Contributor

    Sorry, realized I linked the wrong one and updated 😬

  5. kmacoskey commented on Aug 6, 2018

    @kmacoskey
    Author

    The issue occurs with and without encrypt_boot: true. Should I add some comments to the other linked issue to reflect that the issue extends beyond the scope of the original issue?

  6. SwampDragons commented on Aug 6, 2018

    @SwampDragons
    Contributor

    There's already another issue linked to that one that's closer in symptoms to yours. I promise, it's the same root cause :) . If you want to add a little note though, mentioning your symptoms and that increasing the wait attempts helped, it may help other users.

  7. kmacoskey commented on Aug 6, 2018

    @kmacoskey
    Author

    I had not followed the next issue. It makes more sense now. Thank you!

  8. SwampDragons commented on Aug 6, 2018

    @SwampDragons
    Contributor

    No problem! And thanks for your diligence.

  9. mattolenik commented on Aug 28, 2018

    @mattolenik

    I'm also hitting this but setting AWS_MAX_ATTEMPTS=150 does nothing. Neither does any combination of these variables, according to the logic laid out here: https://github.com/hashicorp/packer/blob/master/builder/amazon/common/state.go#L298

    I'm just exporting them from the shell script where I invoke Packer. It's pretty straightforward, but I can't seem to configure it to not time out. It's as if they're all being ignored entirely.

    I'm confused in general with this behavior...AMI import is a process that can frequently take 1-2 hours, even longer. The updated default timeout of 30 minutes is still nowhere near enough for any moderately sized image.

  10. SwampDragons commented on Aug 28, 2018

    @SwampDragons
    Contributor

    I can update the default further, but those env variables absolutely should work. That code is a little hairy but ultimately the behavior is: use the env variables if the user has set them and if not, use waiter-specific defaults rather than global ones. If you want help debugging I'll need to see your debug logs (set env var PACKER_LOG=1)

  11. SwampDragons commented on Aug 30, 2018

    @SwampDragons
    Contributor

    @mattolenik and @WeekendsBull can one or both of you please share configs and debug logs with me so I can try to replicate?

  12. 8 remaining items

  13. SwampDragons commented on Aug 31, 2018

    @SwampDragons
    Contributor

    I lied. What I thought was a problem is fine.

  14. SwampDragons commented on Aug 31, 2018

    @SwampDragons
    Contributor

    The good news is, I can reproduce now.

  15. SwampDragons commented on Aug 31, 2018

    @SwampDragons
    Contributor

    Found it. New PR 6649 should fix this for you. Sorry for the saga. If you'd like I can build you a copy of the branch.

  16. WeekendsBull commented on Aug 31, 2018

    @WeekendsBull

    i'm able to build mine now. @SwampDragons. Really appreciate your help.

  17. mattolenik commented on Aug 31, 2018

    @mattolenik

    @SwampDragons thank you very much for your help!! A local build will be just fine for me for now.

  18. SwampDragons commented on Sep 4, 2018

    @SwampDragons
    Contributor

    Okay you're on some kind of posix system going by logs so here's a
    linux build:
    packer.zip

    and here's an OSX build:
    packer.zip

  19. cixelsyd commented on Sep 19, 2018

    @cixelsyd

    I'm hitting this also; I tried to look and see if this made it into 1.3.1 or is still as-yet-unreleased however my github foo is not strong enough. I believe that it did not make it into a release yet, would you please confirm for me? Thanks!

  20. SwampDragons commented on Sep 19, 2018

    @SwampDragons
    Contributor

    This should be in 1.3.1. If you're still having problems, update the environment variables.

  21. cixelsyd commented on Sep 20, 2018

    @cixelsyd

    Thank you! I used 1.3.1 and updated my environment variables and it is working as expected.

  22. lmayorga1980 commented on Sep 20, 2018

    @lmayorga1980
    Contributor

    The environment variables fixed the problem when uploading an image that usually takes 1 hour. With Packer 1.2.3 it runs smoothly and after 1.2.5+ I have to provide.

    AWS_POLL_DELAY_SECONDS=10 AWS_MAX_ATTEMPTS=400.

  23. royerhq commented on Sep 26, 2018

    @royerhq

    I am getting the same issue with Packer Version 1.3.1 running the Amazon-Import Post Processor.

    The AMI gets created in AWS but Packer fails with, ResourceNotReady: exceeded wait attempts, which results on an un-tagged and improperly named AMI.

    ==> virtualbox-iso: Exporting virtual machine...
        virtualbox-iso: Executing: export CentOS 7.5.1804 (x86_64) Minimal [09-26-18] --output output-virtualbox-iso\CentOS 7.5.1804 (x86_64) Minimal [09-26-18].ova
    ==> virtualbox-iso: Deregistering and deleting VM...
    ==> virtualbox-iso: Running post-processor: amazon-import
        virtualbox-iso (amazon-import): Uploading output-virtualbox-iso\CentOS 7.5.1804 (x86_64) Minimal [09-26-18].ova to s3://ami.packer.net/packer-import-1538001480.ova
        virtualbox-iso (amazon-import): Completed upload of output-virtualbox-iso\CentOS 7.5.1804 (x86_64) Minimal [09-26-18].ova to s3://ami.packer.net/packer-import-1538001480.ova
        virtualbox-iso (amazon-import): Started import of s3://ami.packer.net/packer-import-1538001480.ova, task id import-ami-015fd54147463778b
        virtualbox-iso (amazon-import): Waiting for task import-ami-015fd54147463778b to complete (may take a while)
    Build 'virtualbox-iso' errored: 1 error(s) occurred:
    
    * Post-processor failed: Import task import-ami-015fd54147463778b failed with error: ResourceNotReady: exceeded wait attempts
    
    ==> Some builds didn't complete successfully and had errors:
    --> virtualbox-iso: 1 error(s) occurred:
    
    * Post-processor failed: Import task import-ami-015fd54147463778b failed with error: ResourceNotReady: exceeded wait attempts
    
    ==> Builds finished but no artifacts were created. 
    

    Note
    My environment Variables where set and it still fail. I will set it to a higher number just like @lmayorga1980 did.. will provide feed back once I find a number that would work for me.

    Update
    Thank you @lmayorga1980 --- I totally messed up my AWS_POLL_DELAY_SECONDS variable.. I had it set to 5 which is the default value so I kept timing out... Value of 10 like you did worked like a charm.. Thanks!

  24. lmayorga1980 commented on Sep 27, 2018

    @lmayorga1980
    Contributor
  25. shyam2j commented on Nov 16, 2018

    @shyam2j

    The environment variables fixed the problem when uploading an image that usually takes 1 hour. With Packer 1.2.3 it runs smoothly and after 1.2.5+ I have to provide.

    AWS_POLL_DELAY_SECONDS=10 AWS_MAX_ATTEMPTS=400.

    THis worked for me as well, had to make to 400 from 150 for AWS_MAX_ATTEMPTS

  26. ghost locked and limited conversation to collaborators on Mar 30, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions