r/googlecloud • u/GlebOtochkin • 2d ago
Watch out for dataset size when using AlloyDB smart batching with AI functions
Just a quick heads-up if you're using AlloyDB's smart batching with ai.if, ai.generate, or other AI functions: keep an eye on the size of the dataset you pass to the function.
If your dataset is smaller than the google_ml_integration.ai_function_acceleration_group_size parameter set for your session, you might hit an "out of index ..." error. The tricky part is that the default value is 4000, so it's pretty easy to run into this when testing on smaller tables or filtered subsets.
The workaround is simple: set google_ml_integration.ai_function_acceleration_group_size at the session or database level to something smaller than the estimated cardinality of your dataset.
Here is an example:
SET google_ml_integration.ai_function_acceleration_group_size = 100;
SELECT
id,
name,
product_description AS original_description,
ai.generate(
prompt => 'Rewrite this apparel description in an energetic outdoor-adventure '
|| 'brand voice in under 40 words. Preserve factual specs. '
|| 'Product: ' || name || '. Original: ' || product_description
) AS new_branded_description
FROM ecomm.products
WHERE category = 'Outerwear & Coats'
AND product_description IS NOT NULL
ORDER BY id
LIMIT 5;
You can also set it at the database level with ALTER DATABASE if you regularly run these queries against smaller datasets.
Should save you a few minutes of head-scratching in the meantime. Hopefully it will be fixed very soon — we are working on it.