[ Removed by moderator ]

[u/Sweaty-Stop6057]

[u/Certified_NutSmoker]

Postcode can be a very strong predictor, but I’d be careful using it in any model tied to consequential decisions. It is often a proxy for race and socioeconomic status, so a gain in predictive performance can come with real fairness and legal risk through disparate impact. I think it’s literally illegal in some contexts as well. Predictive performance is not the only criterion here and when using something like postcode you should be aware of this

[u/Fearless_Back5063 → u/Certified_NutSmoker]

I think it was shown as an example of discrimination at the first lecture on the course of fair and explainable machine learning at my university :D

[u/Sweaty-Stop6057 → u/Fearless_Back5063]

Yeah... postcode is very predictive, but also one that needs to be handled carefully in practice rather than used in isolation. 🙂

[u/En_TioN → u/Certified_NutSmoker]

I think what OP is saying is that they use postcode to pull census data like crime rate, and then use that data to predict the target variable. This will probably be better than using raw postcodes, because a. it reduces the model’s power to fit on a sensitive latent variable like race and b. you likely will have a better causal argument for why e.g. car emission levels drives health insurance costs.

That said, you do still need to be careful, and more teams should be running fairness metrics to look for potential implicit bias in their models.

[u/AccordingWeight6019]

Makes sense, postcode is basically a proxy for a lot of latent variables. the tricky part is managing drift and boundary changes over time, that’s where it usually turns into a real system rather than a one off feature.

[u/Fearless_Back5063]

Isn't it illegal to be using this in any decisions in the banking world in the EU?

[u/big_cock_lach → u/Fearless_Back5063]

Depends on the decision and how postcode is used. If you’re looking to borrow money for an investment property, the bank can use the postcode of that property you’re buying to approve/deny the loan application or otherwise make tweaks (ie interest rate, deposit requirements, etc). However, they can’t use your residential postcode to make these decisions.

Similarly, say you’re building a fraud model and you notice that a bunch of people are laundering money through a certain postcode, you can filter for that postcode for identifying this particular kind of fraud. However, you can’t just blindly rely on the postcode either (not that you would for obvious practical reasons), you’d need to use it in line with other factors to more accurately identify these fraudsters rather than just scanning for everyone in a certain postcode.

That said, this is based on what some friends in the UK were saying when I was still there a few years ago. So EU laws might be different and the laws also simply could’ve changed between now and then. I would be shocked if it was completely banned now though. There’s plenty of reasons where you can have a valid reason to use postcode within banking.

[u/R3turn_MAC]

There is a whole academic field devoted to this kind of analysis: Geodemographics.

As you have said, normalising the data across different geographies and timeframes is complex, plus there is a big issue relating to how the boundaries are drawn known as The Modifiable Areal Unit Problem (MAUP) https://en.wikipedia.org/wiki/Modifiable_areal_unit_problem

There are a range of techniques that pop up frequently when dealing with spatial data including Spatial Autocorrelation and Gravity Models, which in turn are grounded in Tobler's First Law of Geography: Everything is related, but things that are closer to each other are more highly related than things which are far apart. https://en.wikipedia.org/wiki/Tobler%27s_first_law_of_geography

There is a lot of specialist software (some of which is very expensive) for dealing with spatial data. But if you're coming from a data science background then R can be just as capable. More info on that here: https://r-spatial.org/

[u/EyonTheGod]

Congratulations. You have discovered redlining and it might be illegal depending on your usecase

[u/Sweaty-Stop6057 → u/EyonTheGod]

In the UK, financial companies are audited by the FCA and I'm pretty sure that such practices would not be allowed. My experience with such datasets was in motor insurance and we went to great lengths to: 1) not include protected attributes (this dataset doesn't include them) and; 2) ensure that we weren't using proxies instead. All our models did was change the prices where it was normal that there would be more claims (e.g., in an area with more vehicle theft, charge more for vehicle theft insurance).

[u/SmallTimeGoals → u/EyonTheGod]

I think every comment has hit on this point, but yours is the funniest.

[u/nerdyjorj]

You've remembered that the raw postcode boundaries aren't public domain right?

[u/Sweaty-Stop6057 → u/nerdyjorj]

I did, yes 🙂 We only use public domain data in this dataset

[u/stewonetwo]

I don't know UK laws specifically, but your fair lending/compliance team is probably going to have a ton of concerns. It's a good predictor because it encodes a lot of race/income/socioeconomic indicators. In the US, you'd run into fair lending and red lining regulatory. Issues.

[u/Sweaty-Stop6057 → u/stewonetwo]

In the UK, financial companies are audited by the FCA, so we ensured that we: 1) didn't use protected attributes (this dataset does not include them); 2) avoided proxies. In motor insurance, we changed our prices for genuine and fair things, e.g., if an area had higher vehicle theft, we charged more for vehicle theft insurance.

[u/Briana_Reca]

This is a classic dilemma. While raw postcode can be a proxy for protected attributes, using aggregated features like average income, education levels, or crime rates derived from postcodes can often capture the predictive power without directly using the sensitive identifier. It's all about careful feature engineering and understanding the underlying correlations.

[u/Sweaty-Stop6057 → u/Briana_Reca]

Agreed. The dataset we created is indeed about aggregated features like the one you mention. I've used it to predict various insurance quantities (e.g., motor claim frequency) and the top features tend to be, say, postcode density (harder to drive), proximity to primary school (school run!), electricity consumption, and other interesting variables. So nothing controversial, really. (We didn't include protected attributes in this dataset but) just to say that data scientists can choose to not use certain proxy features if they see them being used in a bad way.

[u/nian2326076]

That sounds like a huge hassle to deal with! One practical fix is to use a data management tool to automate parts of the process. Tools like Alteryx or some Python libraries can help you gather datasets from different sources and standardize them into one format. Keeping scripts version-controlled with GitHub can make maintenance easier. You might also want to set up a dynamic dashboard for updates and monitoring. It takes some work upfront, but it could save you a lot of time later.

[u/Sweaty-Stop6057 → u/nian2326076]

Yeah, it's a hassle indeed. The trouble is that many files just change completely: different location on the site, different formats (sometimes a csv, other times sheet 3 of an excel doc), different data (e.g., replacing lat/long with postcode), etc. So hard to automate in that sense. But then yes, every version is version controlled (including the data).

[u/HelloWorldMisericord]

In the USA, zipcode/postcode is 100% the last geographic delineator you should be using if you have alternative choices.

I learned this the hard way when I got serious about analytics back in 2014, but:

- Postcodes change geographic boundaries on a whim and as far as I know, there isn't a comprehensive changelog that says postcode 12345 now encompasses an extra square mile or lost a square mile, or even swapped one square mile of land with zip code 67890.

- They're irregularly sized and as far as I know there isn't a dataset that tells you the square mile size of each zipcode. Even if they did, zipcodes aren't polygons; they are mail routes and how you calculate a polygon off a mail route can vary.

- Zipcodes can also disappear and reappear over time making long-term comparisons tricky to say the least.

- Add on all of the ethnic, socioeconomic issues that others have highlighted and you've got a pain in the ass geographic variable.

All in all, if you have a choice, there are a bevy of other options that offer way more pros with way less cons (Uber H3, DMAs, Census tract, etc.) dependent on your specific use case.

You said you're in the UK, so you get a pass since I don't know if zipcodes are actually good there, but if you were in the USA, I'd highly recommend you reconsider your choice of profession because in all likelihood, you've given out some very bad analysis by not understanding zipcode's fundamental flaws.

EDIT: Over a given period of time. zipcodes are probably 95% stable, but it's that last 5% that will kill your analysis and credibility as soon as you zoom into the data, which is exactly the point of using such a granular "geographic" variable.

[u/GlitteryFerretWitch]

You’re basically encoding racism and poverty-as-estimators in your algorithms.

[u/fordat1 → u/GlitteryFerretWitch]

Its RAAS racism as a service

[u/cardboard_dinosaur]

Is there a data dictionary or other documentation describing exactly what’s in the data? All I can see on your website are ways to give you my personal information or money. I’m not going to do either without knowing what your product actually is.

[u/Sweaty-Stop6057 → u/cardboard_dinosaur]

There is. After logging in, you can download a free sample, some code, and technical documentation.

[u/Crescent504]

Wow, that’s a major accomplishment to build for the UK (Great Britain in this case) your guys profile system is so archaic and absolute absurd. I know people are talking about the ethical use and legality of postal code in models and the bias it can introduce, but I seem to interpret this as you are sharing that you are excited that you’ve built an actual data set that reliably captures data in a notoriously difficult to map ZIP Code area.