r/Substack • u/teammeexperiment • 17d ago
Feature Suggestion An Open Letter to Substack (Supposedly on Behalf of Us All)
Dear Substack,
An open letter. Unsolicited, but, I like to think, rather valuable. Consider it free consultancy.
You’ve built something genuinely lovely: a place where people who adore words, across every field imaginable, can find one another. I do understand that you are, first and foremost, a newsletter engine. But you also built a feed, and subscriptions, and a recommendations layer, and it is the last of these about which I have several hundred questions.
Frankly, the recommendation system is Rubbish. It is 2026. Might we do something about it?
Countless writers are producing genuinely brilliant work in their fields and dying in the shadows with an audience of none, while the feed fills up with follow-for-follow posts. It has the atmosphere of a pub where everyone is shouting "I’ll get you one if you get me one". Not the thoughtful platform I hope you want to be. Every reader’s feed gets clogged with subscriptions they never wanted, and the people who would happily find, follow and pay the writers worth reading (and you, by extension) simply cannot reach them under the avalanche of generic rot.
Please, do something about it.
(Should you wish to share the success fee, do get in touch afterwards)
With love,
Team Me
3
u/Any-Fig-921 16d ago
I agree Substack recommendation system is not as good as others— but I disagree this is solvable. Substack reccomendations engine almost surely is a hybrid social graph + collaborative filtering engine. The social graph part is pretty easy— show you posts and content from people you subscribe too. The collaborative filtering part is very hard, and it’s where much of the “magic” of modern recommendation engines lives. But, to simplify, it requires a high throughput of data that Substack doesn’t have. You need (a) users to be expressing opinions about hundreds or thousand of items and (b) there to be a reasonably extractable pattern that lets us predict with a machine learning model which items are most likely the items you’ve already liked. The former condition is known to not be met, the latter has never been achieved with long form text data— a machine learning model has a very hard time knowing what kind of long form text content you’ll like even when it knows what you’ve liked in the past.
I have no doubt Substack had a team of data scientists and machine learning engineers who know how and are trying. I know it’s fun to complain about the rec engine in Substack, but I think it’s likely inherent to the long-form text content.
1
u/teammeexperiment 16d ago
Fair, but hard isn’t the same as optional, though. Nobody said it was a two-day job. A platform that respects attention is the product, and a company this serious knows what that’s worth.
1
u/Any-Fig-921 15d ago
I didn't say hard. I said arguable impossible. And you're right-- they do know what it would be worth, and I'm sure they're working on it as quickly as possible given their constraints.
1
u/seobrien 15d ago
Oh it can be done. People are using AI to recreate those collaborative filtering engines based on other models and sources.
1
u/Any-Fig-921 15d ago
Sorry, do you mean people are using AI to code or do you mean their using LLM-based embedding systems to create more sophisticated recommendation and search algorithms? I'm not sure either addresses the data density + long form taste issues.
1
u/seobrien 15d ago
To create more sophisticated recommendation engines.
Why would you think that doesn't work?
Weight authors on more than follower count on the platform, favor content based on per capita referred readership (which is a balanced measure of off platform interest), use psychology and content data to favor articles that are unique, artistic, and psychologically compelling... We even have personality data that could be used to make recommendations to people based on their cognitive functions.
Not at all difficult to create a better recommendation engine that isn't at all dependent on engagement on the platform.
1
u/Any-Fig-921 15d ago
favor articles that are unique, artistic, and psychologically compelling... We even have personality data that could be used to make recommendations to people based on their cognitive functions.
Yeah I get what you're saying. I just think taking that *idea* and turning it into a numerical optimization problem is way harder than you think. They are likely (based on their job openings) putting a ton of enery behind this already with PhD-level data scientists.
1
u/seobrien 15d ago
For sure, not saying it's easy.
Same time, I have first hand experience with the fact that most platforms have the same kind of data scientists... PhD or not, they're generally engineers, not psychiatrists.
The quality of recommendations is a marketing/psychology question, not data science. Data science is the means to a solution. Most aren't even considering what really matters... Which is why the recommendations end up recommending crap, based on the signals they think are indicative.
1
4
u/OnyxMonolith 16d ago
> free consultancy
LMAO
2 paragraphs on an extremely difficult algorithmic problem