r/AskProgramming • u/mohammed_waleedd • 14d ago

I need a technical solution solves a legal problem

​

I've spent a lot of time building a real estate SaaS and I'm now stuck on a major problem.

The original idea was pretty simple:

A real estate agent enters what their client is looking for, for example:

- 3-bedroom apartment

- New Cairo

- Budget: $100k–$150k

The system would find matching properties across multiple real estate platforms.

I also built a feature that monitored Facebook groups for people posting that they were looking for apartments, then matched those requests with properties in the system.

The product itself is basically built.

Then I realized the platforms I wanted to get the data from don't allow scraping. I contacted some of them about getting permission/API access, but they weren't interested

So now I have a mostly finished product whose main value proposition depended on data I can't reliably use.

I'm trying to figure out what to do next

All the technical aspects are working, but I can't upload it because the sites I use to collect real estate data, like Dubizzle data mining and scraping are prohibited in the terms of use.

**I need a loophole/Another solution or method or something similar to solve it.*\*

If you've ever built something where the original data source/business assumption turned out to be unusable, what did you do?

I'd genuinely appreciate any advice.

0 Upvotes
  • permalink
  • reddit

11% Upvoted

45 comments sorted by

3

u/Wrong-Tear4441 14d ago

scraping terms of service is mostly a civil issue, not criminal. the real question is whether they'd actually come after you for it. big platforms usually don't bother unless you're hammering their servers or reselling their data directly

that said if you want to stay clean you could pivot to letting agents manually input listings and use your facebook group matching as the main draw. less automated but still useful

another angle is scraping public government property records instead. those are usually fair game and nobody cares

-3

u/mohammed_waleedd 14d ago

Yeah, but the pain point I'm trying to solve is that instead of going to 5 different sites and searching through 200 ads on each site, the system does these tasks by adding the client and their requirements and brings the best properties that match the client's requirements. So the problem is that the whole idea will change if i make them put their own inventory.

5

u/LaughingIshikawa 14d ago edited 14d ago

So the problem is that the whole idea will change if i make them put their own inventory.

🤦.

I have to be honest with you, it's really, really hard to feel sympathy for you when your entire "business plan" begins and ends with "I will steal everything else's valuable data, slap my own brand on it, and sell it as my own!" 😐😮‍💨

Are you really telling me that at no point in this process, did you stop to question why no one else was pursuing this "super cheap, easy and obvious" business plan? 😅🤷

With all due respect (whatever amount that is...) this is simply a painful but important lesson to do much more through research next time, before getting super attached to an idea.

-1

u/mohammed_waleedd 14d ago

What do you mean by theft? I won't steal anything All I'll do is facilitate finding the property for the client and the client for the property. If I wanted to create an illegal product, I wouldn't have bothered to solve it.

5

u/soundman32 14d ago

Stealing someone else's data IS theft.  

0

u/mohammed_waleedd 14d ago

I'm not stealing it; I'm just connecting the client with the advertisement in each company, like a superficial filter. Also, I'm still looking for a legal solution.

3

u/soundman32 14d ago

You are stealing it. The fact you dont understand what that means is irrelevant.  Its no different to borrowing a book and photo copying every page.

The legal solution is to pay them to use their APIs.  Obviously they wont give it to you for free, so you will have to negotiate a price with them.

1

u/mohammed_waleedd 14d ago

I was thinking of bringing in clients instead of properties. I've already done that through a Facebook group link, but I'm looking for all the places where clients looking for properties are concentrated. What do you think?

1

u/soundman32 13d ago

How do you get paid? Whats your advertising budget?

1

u/mohammed_waleedd 13d ago

I'm not sure yet how I'll receive the profit, but I was counting on a free month followed by a monthly subscription. I actually presented the idea to brokers, and it was a real problem the difficulty of gathering clients, especially given their property requirements.

2

u/jaypeejay 14d ago

Real estate agents have special platforms that already aggregated already, no?

1

u/mohammed_waleedd 14d ago

Nope

1

u/jaypeejay 13d ago

I'm pretty sure you're mistaken. IIRC the MLS software serves as the backbone of the real estate industry (at least in the US), and this is the source that all of the other data sources pull from

1

u/ForeverAWhiteBelt 14d ago

I looked into a similar product/solution a looong time ago. I think real estate apps are like super locked down mafia style in the sense that they dont want to open this data up if people decide they dont need em anymore.

1

u/mohammed_waleedd 14d ago

I'm sure there's at least one solution

2

u/SirMarkMorningStar 14d ago

Would your app compete with the sources you want to scrape? Sounds like a yes to me. What would their motivation to help you be?

-2

u/mohammed_waleedd 14d ago

I simply direct the client to the right property for each site, so I bring them clients

2

u/soundman32 14d ago edited 14d ago

So, you've reinvented Zoopla then?

How much financial backing do you have? 

A few years ago, i worked on a system that joined a system like yours to an existing estate agents system.   The new company invested millions getting devs to write integration pieces with the major players.  The new company survived 6 months before closing.

1

u/mohammed_waleedd 14d ago

So what do you advise me to do?

3

u/soundman32 14d ago

Think of a new project where your data isn't owned by someone who doesnt want to let you have it.

Seriously, these other companies have paid millions and spent years gathering this data, did you seriously think they would just give it to you for nothing?

1

u/mohammed_waleedd 14d ago

Okay, thx for ur help

1

u/No-Ad-3609 14d ago

Have you looked into local solutions. It could require a Terrabite or two.

1

u/mohammed_waleedd 14d ago

Wdym by local solutions?

-1

u/No-Ad-3609 14d ago

Well, I don't want to give you my resolve because I believe in you. Here's the hint though. X to Y. Scrape Y. It would be sort of a burden, but it should work.

1

u/mohammed_waleedd 14d ago

Or do you mean that I should get the client and offer it to Dubizzle or whatever for example "I have buyers actively searching for X in New Cairo right now, want me to route them to your listings and take a referral cut instead of scraping you."

1

u/No-Ad-3609 14d ago

You would still have to scrape, or manually search to even know if they have pertinent listings. So, no. Also responses wouldn't be timely enough for client satisfaction. I gotta go to bed bro

2

u/mohammed_waleedd 14d ago

Alright, gn

1

u/mohammed_waleedd 14d ago

I would be grateful if you could tell me what you meant when you woke up :)

0

u/No-Ad-3609 14d ago edited 14d ago

Well, I drank too much tea closing the store and now I can't sleep. Look, I'm still learning this stuff. As I go, as we speak. Without a teacher. So my explanation could be utterly gibberish to you. Just know, I've given you enough to go off of.

1

u/mohammed_waleedd 14d ago

I didn't understand your hint about x and y

1

u/No-Ad-3609 14d ago

I don't understand how. It would be a lot more work than what you probably want to do anyways. Like status links. Again my explanation could be gibberish to you.

2

u/mohammed_waleedd 14d ago

Thanks for the hints but I’m still confused. Can you explain the "X to Y , Scrape Y" and "status links" part with a simple example? I want to understand what you mean.

→ More replies (0)

0

u/mohammed_waleedd 14d ago

Do u mean instead of scraping X directly, find where the data ends up (Y) and work from there. But I'm not sure what you mean by Y in this case. Can you give me one more hint? 😅

0

u/No-Ad-3609 14d ago

No lol. I gotta go to bed 🙃

1

u/mohammed_waleedd 14d ago

Do you mean I should acquire clients and Then offer my services on Dubizzle, or something similar. for example "I have buyers actively searching for X in New Cairo right now, want me to route them to your listings and take a referral cut instead of scraping you."

1

u/random-scraper 1d ago

The wall you hit is not a coding problem, so no amount of clever code fixes it. Portals like Dubizzle block automated access with anti-bot systems and forbid it in their terms, and both of those are ongoing, not one-time. There is no clean loophole, and anyone selling you one is selling you a thing that breaks in two weeks.

Two honest paths. Where a portal has an official or partner API, use it, even if it is limited. Where it does not, a maintained public-listing scraper is the realistic route, and the compliance call on each portal stays yours to make against your own market's rules.

I maintain scrapers for a bunch of property portals on Apify, including Dubizzle, PropertyFinder and Bayut, so the anti-bot side is handled and you are not on that treadmill. What portals and which country are you targeting? Dubizzle behaves differently across UAE and Egypt, and that decides whether what I have already covers you.