Ansible troubleshooting - AWS Failed to import the required Python library (botocore or boto3)

By Luca Berton · Published 2024-01-01 · Category: installation

Learn how to troubleshoot and fix the \"Failed to import the required Python library (botocore or boto3)\" error in Ansible for AWS with a live Playbook.

Ansible troubleshooting - AWS Failed to import the required Python library (botocore or boto3)

Introduction

Today we’re going to talk about Ansible troubleshooting, specifically about the “Failed to import the required Python library (botocore or boto3)” message and enable Ansible For AWS.

This fatal error message happens when we are trying to execute some code against your AWS EC2 Infrastructure without the necessary Python libraries for the AWS.

These circumstances are usually related to the configuration of your Ansible Controller node and usually are not related to Ansible Playbook.

I’m Luca Berton and welcome to today’s episode of Ansible Pilot.

## Playbook

The best way of talking about Ansible troubleshooting is to jump in a live Playbook to show you practically the “Failed to import the required Python library (botocore or boto3)” and how to solve it!

In this Playbook, I’m going to reproduce the error and fix using the PIP, the Python Package Manager on a demo machine.

error execution

$ ansible-playbook ami_search.yml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit
localhost does not match 'all'
PLAY [AMI search] *********************************************************************************
TASK [search for AMI] ********************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Failed to import the required Python library (botocore or boto3) on demo.example.com's Python /usr/bin/python3.8. Please read the module documentation and install it in the appropriate location. If the required library is installed, but Ansible is using the wrong Python interpreter, please consult the documentation on ansible_python_interpreter"}
PLAY RECAP ****************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

fix code

• python version

The first step is to determine your Python version (3.8 in this example):

$ ansible --version
ansible [core 2.12.2]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/devops/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.8/site-packages/ansible
  ansible collection location = /home/devops/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.8.12 (default, Sep 16 2021, 10:46:05) [GCC 8.5.0 20210514 (Red Hat 8.5.0-3)]
  jinja version = 2.10.3
  libyaml = True
[devops@demo aws]$ python --version
-bash: python: command not found
[devops@demo aws]$ python3 --version
Python 3.6.8
[devops@demo aws]$ python3.8 --version
Python 3.8.12
[devops@demo aws]$ whereis python3.8
python3: /usr/bin/python3.6 /usr/bin/python3.6m /usr/bin/python3 /usr/bin/python3.8 /usr/lib/python3.6 /usr/lib/python3.8 /usr/lib64/python3.6 /usr/lib64/python3.8 /usr/local/lib/python3.8 /usr/include/python3.6m /usr/include/python3.8 /usr/share/man/man1/python3.1.gz

• before Let’s use the pip3.8 command because we are running python 3.8. In other Linux distributions, you might need to specify different Python versions. For example Python 3.9 using pip3.9, pip3 or just pip.

[root@demo aws]# pip3.8 list | grep boto

• fixed with PIP

[root@demo aws]# pip3.8 install boto3
WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3.8 install --user` instead.
Collecting boto3
  Downloading https://files.pythonhosted.org/packages/d4/c0/59513bab408fe6dd366f55c5e6ea4128daffe501fa8986749932219bdd59/boto3-1.24.27-py3-none-any.whl (132kB)
     |████████████████████████████████| 133kB 616kB/s 
Collecting jmespath<2.0.0,>=0.7.1
  Downloading https://files.pythonhosted.org/packages/31/b4/b9b800c45527aadd64d5b442f9b932b00648617eb5d63d2c7a6587b7cafc/jmespath-1.0.1-py3-none-any.whl
Collecting botocore<1.28.0,>=1.27.27
  Downloading https://files.pythonhosted.org/packages/20/7a/06940ad3f89e19ef065dd2600b14148673efe775da4fa61f39b7c63724a8/botocore-1.27.27-py3-none-any.whl (9.0MB)
     |████████████████████████████████| 9.0MB 2.5MB/s 
Collecting s3transfer<0.7.0,>=0.6.0
  Downloading https://files.pythonhosted.org/packages/5e/c6/af903b5fab3f9b5b1e883f49a770066314c6dcceb589cf938d48c89556c1/s3transfer-0.6.0-py3-none-any.whl (79kB)
     |████████████████████████████████| 81kB 3.2MB/s 
Collecting python-dateutil<3.0.0,>=2.1
  Downloading https://files.pythonhosted.org/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247kB)
     |████████████████████████████████| 256kB 4.9MB/s 
Requirement already satisfied: urllib3<1.27,>=1.25.4 in /usr/local/lib/python3.8/site-packages (from botocore<1.28.0,>=1.27.27->boto3) (1.26.9)
Requirement already satisfied: six>=1.5 in /usr/lib/python3.8/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.28.0,>=1.27.27->boto3) (1.12.0)
Installing collected packages: jmespath, python-dateutil, botocore, s3transfer, boto3
Successfully installed boto3-1.24.27 botocore-1.27.27 jmespath-1.0.1 python-dateutil-2.8.2 s3transfer-0.6.0

• after

[root@demo aws]# pip3.8 list | grep boto
boto3              1.24.27  
botocore           1.27.27  
[root@demo aws]#

fix execution

$ ansible-playbook ami_search.yml 
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit
localhost does not match 'all'
PLAY [AMI search] *********************************************************************************
TASK [search for existing AMI] ********************************************************************
ok: [localhost]
TASK [debug] **************************************************************************************
ok: [localhost] => {
    "ami_found": {
        "changed": false,
        "failed": false,
        "images": [
            {
                "architecture": "x86_64",
                "block_device_mappings": [
                    {
                        "device_name": "/dev/sda1",
                        "ebs": {
                            "delete_on_termination": true,
                            "encrypted": false,
                            "snapshot_id": "snap-03f2e24f30f580353",
                            "volume_size": 10,
                            "volume_type": "gp2"
                        }
                    }
                ],
                "creation_date": "2020-11-02T11:01:38.000Z",
                "deprecation_time": "2022-11-02T11:01:38.000Z",
                "description": "Provided by Red Hat, Inc.",
                "ena_support": true,
                "hypervisor": "xen",
                "image_id": "ami-096fda3c22c1c990a",
                "image_location": "309956199498/RHEL-8.3.0_HVM-20201031-x86_64-0-Hourly2-GP2",
                "image_type": "machine",
                "name": "RHEL-8.3.0_HVM-20201031-x86_64-0-Hourly2-GP2",
                "owner_id": "309956199498",
                "platform_details": "Red Hat Enterprise Linux",
                "public": true,
                "root_device_name": "/dev/sda1",
                "root_device_type": "ebs",
                "sriov_net_support": "simple",
                "state": "available",
                "tags": {},
                "usage_operation": "RunInstances:0010",
                "virtualization_type": "hvm"
            }
        ]
    }
}
PLAY RECAP ****************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Wrong Python Interpreter (library already installed)

Sometimes boto3 is installed but you still get the error. Notice that the message ends with a hint: “if the required library is installed, but Ansible is using the wrong Python interpreter…” This happens when you installed boto3 into one Python but Ansible runs a different one — very common when the system has both python3.6 and python3.8, or when you use a virtual environment.

First confirm which interpreter Ansible uses and whether it can see boto3:

ansible localhost -m ansible.builtin.setup -a 'filter=ansible_python' | grep executable

Then point Ansible at the interpreter that actually has the library by setting ansible_python_interpreter — in the inventory:

[local]
localhost ansible_connection=local ansible_python_interpreter=/usr/bin/python3.8

or as a play variable:

- name: AMI search
  hosts: localhost
  vars:
    ansible_python_interpreter: /usr/bin/python3.8

If you run Ansible inside a Python virtual environment, install the library into that same venv (pip install boto3 botocore) and point ansible_python_interpreter at the venv's bin/python. When using an execution environment, add boto3 and botocore to the EE image instead.

Conclusion

Now you know better how to troubleshoot the Ansible “Failed to import the required Python library (botocore or boto3)” message and move forward with your Ansible For AWS project.

• AWS automation with Ansible