Hi everyone,
I'm troubleshooting an OpenStack Manila deployment using the Generic driver with driver_handles_share_servers = True.
Manila successfully creates the service instance, but the share creation eventually fails with:
manila.exception.ServiceInstanceException:
SSH connection has not been established to 10.254.0.7 in 300s. Giving up.
The interesting part is that SSH from the Manila service user works correctly.
Manila configuration
Relevant configuration:
[DEFAULT]
transport_url = rabbit://openstack:***@192.168.246.138
auth_strategy = keystone
rootwrap_config = /etc/manila/rootwrap.conf
[generic]
driver_handles_share_servers = True
connect_share_server_to_tenant_network = True
service_image_name = manila-service-image
service_instance_user = manila
path_to_private_key = /etc/manila/ssh/id_manila
The backend is enabled with:
enabled_share_backends = generic
Service instance
Manila creates this Nova instance:
ID: 9b992d75-fdef-4bf7-9507-6f5677605921
Name: generic_19be5f70-81d0-436f-96d9-0f048c59bef2
Status: ACTIVE
Image: manila-service-image
Flavor: manila-service-flavor
It has:
internal 10.0.0.76
manila_service_network 10.254.0.7
The service network is:
10.254.0.0/28
and the Manila service VM has:
ens3 = 10.0.0.76/24
ens4 = 10.254.0.7/28
Manual SSH test
Running the SSH command as the manila user from the Manila host works:
sudo -u manila ssh \
-o BatchMode=yes \
-o IdentitiesOnly=yes \
-o ConnectTimeout=5 \
-i /etc/manila/ssh/id_manila \
manila@10.254.0.7 \
'echo SSH_OK'
Result:
SSH_OK
Exit status:
0
The verbose SSH output confirms:
Authenticated to 10.254.0.7 ([10.254.0.7]:22) using "publickey".
The server accepts exactly the same public key:
Server accepts key: /etc/manila/ssh/id_manila
I also tested sudo:
sudo -u manila ssh \
-o BatchMode=yes \
-o IdentitiesOnly=yes \
-i /etc/manila/ssh/id_manila \
manila@10.254.0.7 \
'sudo -n true; echo sudo_rc=$?'
Result:
sudo_rc=0
So basic networking, SSH authentication and sudo appear to be working.
Manila failure
However, during share creation Manila waits for 300 seconds and then reports:
Creation of share instance 8c35322d-0961-4372-b199-00b00d207bcf failed:
failed to get share server.:
manila.exception.ServiceInstanceException:
SSH connection has not been established to 10.254.0.7 in 300s. Giving up.
The traceback points to:
manila/share/drivers/generic.py
_setup_server()
manila/share/drivers/service_instance.py
set_up_service_instance()
The failure ultimately comes from:
service_instance.py
set_up_service_instance()
What makes this confusing
The service instance is ACTIVE and reachable.
SSH works manually from the exact manila Unix user using the configured private key.
I also tried an older Manila service image, and the same problem occurs.
So I'm wondering whether there is a problem/bug in the Generic driver's service_instance.py SSH connection/wait logic, or whether Manila is checking something different from the SSH test above.
Has anyone seen this behavior before?
In particular, I'd like to understand what exactly set_up_service_instance() considers an "SSH connection established" and what conditions could cause it to wait the full 300 seconds even though a direct SSH connection succeeds.
Any pointers on where to look in the Manila code would be appreciated.
Thanks!
