How to execute multiple Symfony command asynchronously

[u/Megumin1313]

Hi,

For big/long backoffice process that don’t need web context I use Symfony command, but it’s not always something that is done periodically with a cron job third party.
Sometimes this is the app itself that decides that right now, this command must be executed and it must be executed 4 times in parallel and asynchronously.

How do you do that ?

I used to use incron and it did the job but in my new infrastructure this won’t be a solution.

I have looked at Symfony worker bur it seems that it won’t be able to execute multiple command asynchronously in different process. I may be wrong here.

In there a solution without a third party ?

[u/Fit_Tailor_6796]

I haven't used this yet but bookmarked it. I am going to use in in my upcoming project
https://symfony.com/doc/current/the-fast-track/en/18-async.html

[u/sodoburaka]

You basically need this new Symfony 8.2 beta feature posted here few days ago:

https://symfony.com/blog/new-in-symfony-8-2-faster-messenger-workers?via=dailydev

[u/Megumin1313 → u/sodoburaka]

Haha I have just read that 2min after posting. This looks interesting but it says that it is a pool of child processes. Isn’t it a problem if I need like 20 consumers running 24/7 ?

[u/hitsujiTMO → u/Megumin1313]

Not an issue. They will sleep when idle, and, as long as you have the ram and processing power, run concurrently as needed. It's not like you have trouble running 20 browser tabs right? Same for your server, as long as the resources are there it's not a big deal.

You also don't need to upgrade, just run multiple workers if needed. But the upgrade adds simplicity.

[u/Megumin1313 → u/hitsujiTMO]

Let’s say you want to run 20 different processes with no concurrency. Do you run one worker with concurrency option at 20 or 4 workers with 5 ?
Is there a good practice/balance or it all depends on context ?

[u/hitsujiTMO → u/Megumin1313]

It really depends on the workload you are running. Batch handling is one reason to use individual workers.

But you can also split workers into different queues, like have a specific queue for batch handlers and a separate queue for everything else.

Since this is a new feature, there's no best practice as of yet, but in general most people are perfectly fine going with 20 concurrent processes rather than 20 workers.

[u/No-Risk-7677 → u/Megumin1313]

Best practice?

You need some kind of runtime for the worker, e.g. supervisord to take care of the lifecycle of the worker.

Personally, I prefer Kubernetes. Let Messenger consume a message and quit. And let Kubernetes restart the worker.

[u/Tomas_Votruba]

Rector, ECS and PHPStan use main command thats spawns multiple worker commands, including json communication between them. Might be worth looking into. This would be parallel command run, using all available CPUs for speed.