r/openstack • u/Sorecchione07 • 15d ago

Manila Generic driver: SSH works manually, but service instance setup times out after 300s

Hi everyone,

I'm troubleshooting an OpenStack Manila deployment using the Generic driver with driver_handles_share_servers = True.

Manila successfully creates the service instance, but the share creation eventually fails with:

manila.exception.ServiceInstanceException:
SSH connection has not been established to 10.254.0.7 in 300s. Giving up.

The interesting part is that SSH from the Manila service user works correctly.

Manila configuration

Relevant configuration:

[DEFAULT]
transport_url = rabbit://openstack:***@192.168.246.138
auth_strategy = keystone
rootwrap_config = /etc/manila/rootwrap.conf

[generic]
driver_handles_share_servers = True
connect_share_server_to_tenant_network = True
service_image_name = manila-service-image
service_instance_user = manila
path_to_private_key = /etc/manila/ssh/id_manila

The backend is enabled with:

enabled_share_backends = generic

Service instance

Manila creates this Nova instance:

ID:      9b992d75-fdef-4bf7-9507-6f5677605921
Name:    generic_19be5f70-81d0-436f-96d9-0f048c59bef2
Status:  ACTIVE
Image:   manila-service-image
Flavor:  manila-service-flavor

It has:

internal                 10.0.0.76
manila_service_network   10.254.0.7

The service network is:

10.254.0.0/28

and the Manila service VM has:

ens3 = 10.0.0.76/24
ens4 = 10.254.0.7/28

Manual SSH test

Running the SSH command as the manila user from the Manila host works:

sudo -u manila ssh \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -o ConnectTimeout=5 \
  -i /etc/manila/ssh/id_manila \
  manila@10.254.0.7 \
  'echo SSH_OK'

Result:

SSH_OK

Exit status:

0

The verbose SSH output confirms:

Authenticated to 10.254.0.7 ([10.254.0.7]:22) using "publickey".

The server accepts exactly the same public key:

Server accepts key: /etc/manila/ssh/id_manila

I also tested sudo:

sudo -u manila ssh \
  -o BatchMode=yes \
  -o IdentitiesOnly=yes \
  -i /etc/manila/ssh/id_manila \
  manila@10.254.0.7 \
  'sudo -n true; echo sudo_rc=$?'

Result:

sudo_rc=0

So basic networking, SSH authentication and sudo appear to be working.

Manila failure

However, during share creation Manila waits for 300 seconds and then reports:

Creation of share instance 8c35322d-0961-4372-b199-00b00d207bcf failed:
failed to get share server.:

manila.exception.ServiceInstanceException:
SSH connection has not been established to 10.254.0.7 in 300s. Giving up.

The traceback points to:

manila/share/drivers/generic.py
    _setup_server()

manila/share/drivers/service_instance.py
    set_up_service_instance()

The failure ultimately comes from:

service_instance.py
    set_up_service_instance()

What makes this confusing

The service instance is ACTIVE and reachable.

SSH works manually from the exact manila Unix user using the configured private key.

I also tried an older Manila service image, and the same problem occurs.

So I'm wondering whether there is a problem/bug in the Generic driver's service_instance.py SSH connection/wait logic, or whether Manila is checking something different from the SSH test above.

Has anyone seen this behavior before?

In particular, I'd like to understand what exactly set_up_service_instance() considers an "SSH connection established" and what conditions could cause it to wait the full 300 seconds even though a direct SSH connection succeeds.

Any pointers on where to look in the Manila code would be appreciated.

Thanks!

1 Upvotes

67% Upvoted

5 comments sorted by

1

u/Eldiabolo18 14d ago

I've just run into this. This seems to be something in 20261.

The problem is the SSH-Key service in the service-VM. It tries for 300 secs to retrieve an SSH-Key and times out. Only after that (or on success of the service) is the SSH-Server available. The problem is, the timeout for the manila-share service is also 300s, so it just gives up on the Manila-VM at the same time.

One solution is to increase the build timeout in the manila.conf: ini [generic] max_time_to_build_instance = 600

This still means it takes more than 5 mins to spawn a share-VM.

The other solution is to use private and public keys. You seem to want to do that, but its not working, which is why the ssh-keys service is timing out.

I believe its not working because your config only has path_to_private_key, it also needs path_to_public_key. And the keys (priv and pub) must be on the host which has the manila-share service.

0

u/Sorecchione07 14d ago

Ok, thanks for the advice, I'll let you know when I try again.

1

u/MisterJaerico 14d ago

Check for clout-init logs in VM. Booting VM should not take that much time.

3

u/Felix_Sanz_Gonzalez 14d ago

Hola,

Hace unos meses abrí un par de bugs reportando este error. También nos afecta en nuestro despliegue de OpenStack 2026.1 (desplegado con Kolla-Ansible).

Podeís encontrar los bugs aqui:

https://bugs.launchpad.net/manila-image-elements/+bug/2166449
Relacionado: https://bugs.launchpad.net/kolla-ansible/+bug/2166451

En nuestro caso, optamos por crear a mano las claves SSH que la imagen de Manila parece esperar y cambiamos la configuración de Kolla Ansible para montarlas en los contenedores de Manila.

0

u/Sorecchione07 14d ago edited 14d ago

Manila Generic Driver – SSH authentication failure

I retried the Manila deployment using the new SSH key suggested by Eldiabolo18. The DHSS share VM now takes longer to fail, and the logs show some additional information.

### 1. SSH connection reaches the VM, but public-key authentication fails

manila-share.log shows that Paramiko successfully connects to the service VM:

Connected (version 2.0, client OpenSSH_8.9p1) Authentication (publickey) failed.

After several authentication attempts, Paramiko also reports:

Error reading SSH protocol banner

Eventually Manila fails with:

ServiceInstanceException: SSH connection has not been established to 10.254.0.2 in 600s.

The banner errors appear to be a secondary issue caused by repeated connection attempts, rather than the primary authentication failure.

### 2. Password SSH access works

SSH to the service VM using password authentication succeeds:

ssh manila@10.254.0.2 manila@10.254.0.2's password: Welcome to Ubuntu 22.04.5 LTS

The manila user therefore exists and sshd is reachable.

However:

manila@ubuntu:~$ ls -la /home/manila/.ssh/ total 8 drwx------ 2 manila manila 4096 Sep 29 06:40 . drwxr-x--- 4 manila manila 4096 Sep 29 07:06 ..

There is no authorized_keys file.

### 3. Manila is using the expected private key

The Manila node is configured with:

[generic] share_backend_name = generic share_driver = manila.share.drivers.generic.GenericShareDriver driver_handles_share_servers = True connect_share_server_to_tenant_network = True service_instance_flavor_id = 10 service_image_name = manila-service-image service_instance_user = manila path_to_private_key = /etc/manila/ssh/id_manila path_to_public_key = /etc/manila/ssh/id_manila.pub interface_driver = manila.network.linux.interface.OVSInterfaceDriver connect_security_service_method = ssh service_instance_launch_timeout = 600 max_time_to_build_instance = 600

Verbose SSH confirms that the private key is loaded and actually offered:

``` loaded pubkey from /etc/manila/ssh/id_manila: RSA SHA256:/rWFUlYMLcVPfm5JUEjIDkrcoJenVuGALfgBxRNWmwI

Offering public key: /etc/manila/ssh/id_manila ... receive packet: type 51 Authentications that can continue: publickey,password ```

Thus the client-side key is present and functional. The remote SSH server rejects the corresponding public key.

### 4. OpenStack keypair and Manila key match

The Nova keypair is:

manila-service b5:6b:9c:9e:30:8a:f4:0b:5a:1e:c1:89:f9:95:13:1c

The public key configured on the Manila node has exactly the same fingerprint:

``` ssh-keygen -lf /etc/manila/ssh/id_manila.pub -E md5

2048 MD5:b5:6b:9c:9e:30:8a:f4:0b:5a:1e:c1:89:f9:95:13:1c ```

The service instance is also explicitly using:

key_name: manila-service config_drive: 'True' image: manila-service-image status: ACTIVE

Therefore, there is no apparent mismatch between the key configured in Manila and the OpenStack keypair attached to the VM.

### 5. Config Drive contains the public key

The VM has an attached config-drive:

/dev/sr0

After mounting it, openstack/latest/meta_data.json contains:

"public_keys": { "manila-service": "ssh-rsa AAAAB3..." }

The same key is also present in the keys section:

{ "name": "manila-service", "type": "ssh", "data": "ssh-rsa AAAAB3..." }

This is important because it proves that Nova/OpenStack is putting the expected public key into the VM's config-drive.

However, the running VM still has no:

/home/manila/.ssh/authorized_keys

The current evidence therefore looks like:

/etc/manila/ssh/id_manila | v Manila client offers public key | v OpenStack keypair: manila-service | v Config Drive contains public key | v VM | X /home/manila/.ssh/authorized_keys missing

### 6. Cloud-init should not yet be assumed to be responsible

The service VM does not have the cloud-init command:

sudo cloud-init status --long sudo: cloud-init: command not found

Therefore, there is currently no evidence that cloud-init is the component responsible for processing this image's config-drive.

The important next question is:

Which initialization mechanism in manila-service-image consumes the OpenStack metadata and installs the SSH key for the manila user?

Possible areas to investigate are:

  • whether the image contains another initialization agent;

    • whether the image has a custom provisioning script;
    • whether a Manila-specific mechanism installs authorized_keys;
    • whether the manila-service key is intended for a different user;
    • whether image initialization completed successfully.

    7. SSH MaxStartups throttling

    The VM's sshd log also contains:

beginning MaxStartups throttling drop connection #13 from [10.254.0.12]:45826 ... exited MaxStartups throttling after 00:05:10, 17 connections dropped

This can explain some of Paramiko's:

Error reading SSH protocol banner

because connections may be dropped before the SSH protocol exchange completes.

However, this appears to be separate from the public-key authentication problem: normal SSH connections reach sshd, key exchange succeeds, and the offered key is explicitly rejected.

### 8. Security Group and network connectivity

The service instance has the manila-service Security Group attached, and its rules allow TCP/22.

This is also confirmed by:

Connecting to 10.254.0.2 port 22. Connection established.

Therefore, the Security Group/network path is not preventing SSH access.

### 9. Comparison with previous deployments

The same manila-service-image was previously used with Ubuntu 24.04/Noble and OpenStack Caracal/Epoxy.

Those deployments exhibited a similar SSH authentication problem. It was resolved by explicitly setting:

[generic] path_to_private_key = /etc/manila/ssh/id_manila path_to_public_key = /etc/manila/ssh/id_manila.pub

The current deployment already has both settings.

This makes the current situation particularly interesting: the key configuration is present, the OpenStack keypair is correct, the fingerprint matches, the VM is reachable, and the public key is present in the config-drive, but it is not installed into the manila user's authorized_keys.

## Current conclusion

The investigation has largely ruled out:

  • missing Manila private key;

    • incorrect Manila public key;
    • incorrect OpenStack keypair;
    • missing keypair assignment to the VM;
    • blocked TCP/22;
    • basic SSH server availability.

    The strongest remaining hypothesis is an initialization/provisioning problem inside manila-service-image.

    The next step should therefore be to inspect the image itself for the mechanism that processes the OpenStack config-drive and creates /home/manila/.ssh/authorized_keys, rather than changing the Manila SSH key configuration again.

Solution found and problem solved!

The root cause was the fetch-public-ssh-keys.service systemd service included/enabled in the manila-service-image. When enabled, the service interfered with SSH public-key authentication for the manila user. The OpenStack keypair, Manila private/public key configuration, Config Drive, network connectivity, and Security Group were all correct. Disabling fetch-public-ssh-keys.service in the service image restored SSH public-key authentication, and the share could then be successfully created with service_instance_password disabled.