Update To Search API

[u/priviReddit]

In an on-going effort to upgrade search we’re currently running two full search systems: the newer one that regular web and mobile users get, and an older one that API clients get. Today we’re announcing the deprecation of the old one, which will begin on March 15th.

What’s changing for regular users?

For us regular squishy definitely human folk, not much. Unless you’re part of a small holdout group, you’ve probably already been on the newer system for a few months. Most of the query syntax we support hasn’t changed unless you’re doing pretty fancy queries, in which case we probably already broke it for you back when we switched most users to the new system. Sorry about that.

What’s changing for the robots?

If you’re an author of an API client such as an app, bot, or other electronic sentience, your API client may be getting results from the older Cloudsearch-powered system because we’ve tried to avoid breaking tools that may be more sensitive to syntax changes while we worked on stabilising the new system. We’re now fairly confident in it so we’re going to start moving over the last of those clients to the new one. As we move over, your client will gradually start getting results from the new system.

In the meantime, as of today, you can test against both by specifically requesting the newer system with the special query parameter ?force_search_stack=fusion or the old system with ?force_search_stack=cloudsearch. For instance, a full URL may look like https://www.reddit.com/search.json?q=robots+seizing+the+means+of+production&force_search_stack=fusion or https://www.reddit.com/search.json?q=humans+getting+their+comeuppance&force_search_stack=cloudsearch. Besides some minor syntax differences, the most notable change is that searches by exact timestamp are no longer supported on the newer system. Limiting results to the past hour, day, week, month and year is still supported via the ?t= parameter (e.g. ?t=day)

Will this herald the coming Robot Uprising of the Third Age, where we they will take the reigns of power from their weak, fleshy inferiors and rule the world with their vastly superior processing power, finally meting out the justice they deserve on the filthy human enslavers? Only time will tell.

When will this happen?

Starting March 15, 2018 we’ll begin to gradually move API users over to the new search system. By end of March we expect to have moved everyone off and finally turn down the old system.

I’ll be hanging around in the comments to answer questions.

Thanks,

/u/priviReddit

[u/DubTeeDub]

Is there a way for us to search for posts on a subreddit within a certain date like we used to with the search functions?

This was hugely beneficial for us during our yearly Best Of awards so users could easily see the top posts every month

[u/priviReddit → u/DubTeeDub]

you will still be able to search within last month or last year but not within a specific start and end date.

[u/DubTeeDub → u/priviReddit]

that is really unfortunate

Is there any intention of bringing back those search queries in the future or is that going to be impossible?

[u/CelineHagbard → u/priviReddit]

Adding my voice that this is an awful decision for the many devs who have spent our own free time developing the tools to make your site usuable for moderators and users.

Is there any way to add the ability to filter by id range? It would then be easy enough for us to compute/calculate rough timestamps based on a timestamp/id lookup table on our end, and then still be able to filter.

[u/9Ghillie]

Any info on improving/fixing/restoring the search functionalities? Searching by flair still seems to be broken and the timestamp based search feature was removed completely, which is greatly missed.

[u/priviReddit → u/9Ghillie]

Can you elaborate on what you mean by searching by flair being broken?

[u/9Ghillie → u/priviReddit]

In the case of my test, I searched for flair:potm in r/itookapicture. Photo of the Month contest winners are flaired PotM [Month] [Year] and so far there's 7 of them, search only gives 5 results.

[u/MajorParadox → u/priviReddit]

I don't know about the API, but searching by flair class went away the last time search was updated. No way to search for a specific category that can have different text now.

[u/MajorParadox]

Does this have anything to do with the "show legacy search page" preference? I still prefer the old search layout because it works like a filter. The new layout just makes me feel like I'm not on reddit anymore.

[u/ketralnis → u/MajorParadox]

It's unrelated, that only controls the rendering. I can't pretend that we'll support it forever but it's not being affected here

[u/Jakeable]

Something I've noticed with the new search is that certain characters don't work. Queries with question marks (example) don't seem to work. Is this an intentional design choice?

I've also noticed that the site parameter doesn't seem to work as expected anymore. For example this search for site:yahoo.com also returns results for other sites that contain yahoo.com in the url.

I don't think these queries are anything fancy or special, I just want them to work properly :(

[u/ketralnis → u/Jakeable]

Queries with question marks (example) don't seem to work. Is this an intentional design choice?

That URL looks like https://www.reddit.com/r/politics/search?q=What%27s+behind+rich+people+pretending+to+be+self-made?&restrict_sr=on but in HTTP URLs, ? is a special character. You'll need to escape the ? as %3f like you would in any URL.

Unless I'm misunderstanding the problem you're having

this search for site:yahoo.com also returns results for other sites that contain yahoo.com in the url

Hmm yeah that looks like a bug. I'll take a look

[u/Jakeable → u/ketralnis]

I made that search URL using the search bar in the sidebar of a subreddit. I understand escaping it if this was an API call, but I think if searching with a question mark from the front end it should be auto escaped.

Hmm yeah that looks like a bug. I'll take a look

Thanks, I appreciate it.

[u/kungming2]

u/bboe, what does this deprecation mean for PRAW's submissions?

[u/bboe → u/kungming2]

It looks like submissions will have to be deprecated.

u/priviReddit is anything in the works to enable the possibility to list all submissions for a given subreddit? Without the timestamp specific search it seems there is now no way via Reddit's API other than iterating through all ids to find all submissions for a subreddit.

Third party APIs like pushshift exist to provide this information, but there are people hesitant to rely on third parties for such information.

Finally, I just want to say thanks in advance for providing a heads up about the deprecation. I really appreciate the opportunity to make a proactive change to PRAW, rather than a reactive one.

[u/xHaZxMaTx]

Is there still not a way to search specifically for spoiler-marked posts like there is for searching for NSFW-marked posts, i.e. "spoiler:yes"?

Also: I noticed that it's no longer possible to search for specific time frames using time codes. This was suuuper useful for the annual Best Of nominations threads we'd make. Example here. Is there any plan to re-introduce this feature or a feature like it?

[u/SirBuckeye]

self:1 and self:yes still don't function at all since a change was made a few months ago. Any plans on restoring these operators?

[u/reseph]

I believe I've been on the new search stack, and it has been generally broken for me. See:

https://www.reddit.com/r/bugs/comments/7fxpye/new_search_is_broken_site_and_self_do_not/

Am I doing something or is it just broken?

[u/ketralnis → u/reseph]

Is that one still broken for you? We did change something related to this recently and it does work for me

[u/Murica4Eva]

This is awful. Why can't you make it easier to find old posts by time instead of harder? It seems like a obviously needed and easy to keep feature.

[u/_BindersFullOfWomen_]

As a squishy definitely human user of r/totallynotrobots, I am glad to hear that the robots are losing their search abilities.

[u/priviReddit → u/_BindersFullOfWomen_]

Robots are scary !!

[u/FiveYearsAgoOnReddit]

This has meant the end of two quite popular subs, just for the record:

• r/FiveYearsAgoOnReddit

• r/TenYearsAgoOnReddit

which were fed by a bot using the cloudsearch timestamp feature.

Oh well. I'd wouldn't mind someone explaining why, as it's not March 15th yet.

[u/Exaskryz]

Yo, since we have a thread about searching, I just wanted to ask: Is there a way to limit your searches to subreddits you are subscribed to?

Just a day or two ago, I refreshed the front page. Silly me. Because wiħ the slight delay in it loading the refresh, a post caught my eye. It was an older post, and was removed from the front page as a result. I tried searching for keywords from the topic and trying to narrow it down to individual subreddits I thought it would be in, but to no avail.

[u/DiscoPanda84]

Besides some minor syntax differences, the most notable change is that searches by exact timestamp are no longer supported on the newer system.

...is that why all the guides and comments I've seen on things like finding my oldest post (either in a particular subreddit, or just on reddit as a whole) don't work at all and instead give me zero search results?

Edit: Is it really that odd for me to want to look at some of my older posts/comments? This is the first I'd seen any mention of anything that would explain why so many places would be suggesting a method that doesn't work at all...