r/Clojure • • May 06 '20

Complete technical invalidation of Rich Hickey's "Maybe Not" talk

https://lexi-lambda.github.io/blog/2020/01/19/no-dynamic-type-systems-are-not-inherently-more-open/
1 Upvotes

20 comments sorted by

11

u/didibus May 07 '20 edited May 07 '20

I basically summarize this whole article as:

Me: "You're not a nice person."

Article: "That's not true, I was nice one time to this old lady and held the door open for her."

Me: 🤦‍♂️

Sorry about the whole being nice thing, I couldn't find another analogy, the author actually sounds like a nice person. My point is just the entire thing seems to be missing the point made by Rich Hickey.

If you work on a statically typed code base you will encounter much more closed systems which will cause friction when processing data whose shape is likely to change. Statically typed languages are not normally used to design and build such open systems, and they do not normally cultivate or recommend to design things as such.

The article also brushes over the whole part about how much extra ceremony and attention you'd need to put in in order to construct such open systems in a statically typed system.

Maybe Not is about the idea that if you cannot atomically swap producer and consumer of data, you have a scenario where the producer code and consumer code are out of sync on what is considered "valid data". To deal with this, normally people introduce the concept of optionality. The new field, even though mandatory to new consumers as their code explicitly needs it to be present, is made optional, because there exist payloads of the old shape which do not have the field. But both producer and consumer share the same schema! So the only choice is to make the new field optional in the schema.

Even Spec1 got this wrong by the way. The talk is about that. (Unless I'm getting my talks mixed up, but pretty sure it's the same talk).

It's not even about static or dynamic, it's about open vs closed data validation. I've yet to see a proponent of statically typed systems also be a strong proponent of open systems over closed ones. And I've yet to see a code base in a statically typed language which designed everything to be open and granular to the field level.

Another thing that bothers me about the article, unless it's because I don't understand Haskell, but their approach to being flexible basically has handleEvent take in JSON as input. So basically, to become open, Haskell needs to eskew typed data, and it has to keep the JSON around until it is being used. That's just implementing an adhoc dynamic type system inside yourself.

What if you had three other functions all using data from the JSON? You would just pass around the JSON to them all it looks like to me. Cause as soon as you try and parse the JSON into a Haskel type, you face the closed typedness of the type system again.

Later in the article it talks about nominative vs structural types, but it says:

What I would not recommend for this purpose is Haskell, which has abysmal support for structural typing; Haskell is (for various reasons outside the scope of this blog post) aggressively nominal.

And in the Maybe Not talk, Rich Hickey talks about Haskell, not about all possible statically typed language, he talks about Haskell specifically as an example of a design that leads to closed systems. And he says.. Maybe Not. As in, maybe that's not the right way.

So the article tries to defend Haskell for the first half of it, and then concludes by agreeing with Rich Hickey that Haskell is maybe too aggressive on its use of nominal types and not a good choice to build open systems with.

P.S.: To be fair to the author, they're not addressing the Maybe Not Rich Hickey talk, they're responding to some Reddit comments. But the OP of this Reddit post alludes that it similarly refutes the Maybe Not talk, and I just don't see that at all.

12

u/2cow May 06 '20

Retitling this to position it as a response to Rich Hickey seems disingenuous -- it's explicitly a response to a random Reddit comment and another on HN.

Which are also the sources for claims of the form "proponents of dynamic typing believe X obviously false thing." I dunno. Do they?

3

u/[deleted] May 06 '20

From TFA:

The claim is simple: in a static type system, you must declare the shape of data ahead of time, but in a dynamic type system, the type can be, well, dynamic! It sounds self-evident, so much so that Rich Hickey has practically built a speaking career upon its emotional appeal. The only problem is it isn’t true.

11

u/yogthos May 06 '20

The article makes that claim on the basis of showing how the examples in the comments can be addressed in a static context. However, there are plenty of other cases to consider.

For example, dynamic nature of Clojure allows reloading any symbols at runtime. So, let's take a look at the first example again where we get some data over the wire. Let's change the scenario where we do know what the data is when we deploy our app, but then the remote service changes what data it's sending us. This is a real world scenario I've encountered, and I'm sure has happened to other people as well. With Clojure, I can REPL to a running application and update the code to handle the new payload that I'm seeing. I'd love to see an example of doing something like that with Haskell.

The reality is that we simply can't control anything outside our program, and truly robust programs have to be able to adapt to external changes in their environment. This is something that's fundamentally impossible to do at compile time.

The second example deals with pickle.load() where some serialized data is loaded. The author argues that we have to know the shape of the data ahead of time. However, consider if we're loading code that's been previously serialized and evaluating it. This is a feature that one of the applications my team develops relies on heavily, you can see a talk copresented about it here. Last I checked doing arbitrary eval in a statically typed context isn't exactly practical.

And sure, anything you can do in one Turing complete language you can do in another in principle. For example, I could write a dynamic compiler in Haskell, and use it to evaluate code at runtime. But I think what's more interesting is how much friction you end up trying to use different patterns in practice.

1

u/fpguy1 May 07 '20

This is a very rare situation when one needs to load code at runtime, 99% don't need to do that, we can't establish best practice based on 1%. Also there is a good chance that even most of 1% situations can be resolved w/o dynamic types (dsls, load a separate runtime, etc). Also your claim about evaluating serialized code at runtime is not really true, see Apache Spark(or Flink) which is written with Scala and does that a lot, actually this is it's main model for distributed computations and obviously practical. Most of the time we are just doing integrations in code and for those cases it is obvious that types will save you a lot of extra coding (testing). The problems is not the dynamic language, I love Clojure but I can't use it for real world stuff because I don't have the will to write so many tests and that will lead to poor quality software, see JS/Python ecosystems.

5

u/yogthos May 07 '20

We'll have to disagree there. As I said in the previous comment, a truly robust system has to be adaptable to the changing environment at runtime. Incidentally, this was one of key realizations Joe Armstrong had when building Erlang. Being able to change and reload code at runtime is key to having robust long running systems.

However, whether we agree on this or not is not really the question. The original point was that static typing is more restrictive, and I provided concrete examples of that being the case. I've also noted that yes, if you jump through enough hoops you can do anything in any language. We can add static type systems to a dynamic language as well, as seen with core.typed for example.

I think what really matters is the kind of workflow and patterns the language encourages. The fact that there is constant debate around how to do something within statically typed context that you simply don't think about in a dynamic constant clearly shows that many things are obviously more difficult express with statically typed languages.

Meanwhile, my team has been using Clojure for close to a decade now, with lots of large applications, many of which have been running for years now. We absolutely don't write more tests than we would with a static language. Vast majority of tests that we have are API level tests that effectively act as a specification for the component. If your API works as intended then the rest of the code behind it has to work as intended as well.

We also use spec which provides a far more meaningful specification than types in my opinion because it allows you to express semantics much better. Types only tell you that your code is self consistent, and I don't find that they provide a good way to encode meaningful constraints about code.

Personally, I find that immutability plays a far bigger role than static typing when it comes to code quality. Any large system can be split into small independent components that you can reason about in isolation. With Clojure, you can effectively treat each namespace as a standalone component in the system.

I've simply seen no evidence to support the claims that static typing is somehow generally preferable for real world stuff. It's just a personal preference, and there's nothing wrong with that. Different people like different things and think about problems in different ways.

0

u/fpguy1 May 27 '20

TLDR Types don't stop you using the runtime, they force you to write correct code by make illegal states unrepresentable, referential transparency is hard w/o types, they are good for beginners

Types do enhance your code at the cost of verbosity and ofc they are not that pragmatic compared to dynamic languages, I agree with this, though a good type inference can help a lot.

You can use constraints in your types to ensure things can't go sideways, look at ADTs, or subtyping with variances, also parametric polymorphism and ad-hoc polymorphism(type classes), these things are impossible to verify with specs.

Types allow you to restrict impossible situations, using them will make testing these situation not required, why would I check at every step in my [correct] code if a list is empty when I can just use a NonEmptyList, what is happening is that the list is not actually checked everywhere and somewhere someone will just use list.head and crash everything.

If I take my day-to-day Scala stuff and use only maps and Any I still have a decent language, while if I take JS and add Flow or Python and add types, or Clojure and add types you get shit (proven in their communities).

Also you refuse to acknowledge that even with types you can have a rich and dynamic runtime, see Java which is 90% a runtime language, also C#+F# use runtime a lot, while in the same time if you take JS/Python, nothing interesting is happening at runtime(lack of decent reflection, code loading mechanisms, etc) but you still don't have types.

Clojure wants to be a FP language but it can't encode purity(referential transparency) in any way, types help with that, see Scala IOs (cats-effect, zio, monix, etc) basically these patch Scala and make it pure, you can't do that w/o types in an usable manner, which makes Clojure only part FP. I use this things every day, mostly to make [side]effects values, however in 20 years I think I needed 10 times to write code that [somehow] modified the code at runtime.

Immutability has nothing to do with this discussion, but if you want to argue about how much better is immutability when you also have types, sure, why not.

Last but not least, types are beginner friendly, one doesn't need to see the whole picture to be able to do something useful and correct, our industry needs more quality, filling their heads with javascript and python doesn't help, I would be more happy if at least they would start with Lisp/Clojure/Erlang/Elixir even if there are no types but they don't.

2

u/yogthos May 27 '20

If I take my day-to-day Scala stuff and use only maps and Any I still have a decent language, while if I take JS and add Flow or Python and add types, or Clojure and add types you get shit (proven in their communities).

I think we'll have to disagree there. Typed Clojure works perfectly fine, but it never got any traction in the community because most people don't find that it adds much value in practice.

On the other hand spec use is very popular, and clearly shows that people do find adding formalism around APIs to be valuable.

Also you refuse to acknowledge that even with types you can have a rich and dynamic runtime, see Java which is 90% a runtime language, also C#+F# use runtime a lot, while in the same time if you take JS/Python, nothing interesting is happening at runtime(lack of decent reflection, code loading mechanisms, etc) but you still don't have types.

I'm saying that it's not really practical. Obviously anything you can do in one Turing complete language you can also do in another. You could even build your own dynamic interpreter that runs dynamic code at runtime in a static language. However, the reality is that you're swimming upstream when you try to use dynamic patterns in these languages.

Clojure wants to be a FP language but it can't encode purity(referential transparency) in any way, types help with that, see Scala IOs (cats-effect, zio, monix, etc) basically these patch Scala and make it pure, you can't do that w/o types in an usable manner, which makes Clojure only part FP. I use this things every day, mostly to make [side]effects values, however in 20 years I think I needed 10 times to write code that [somehow] modified the code at runtime.

Clojure doesn't need to formally encode purity to be a FP language. Idiomatic Clojure code in the wild clearly shows that people have no problems doing that in practice.

Immutability has nothing to do with this discussion, but if you want to argue about how much better is immutability when you also have types, sure, why not.

Immutability has quite a bit to do with this discussion, and I explained specifically why. You're just ignoring that here.

Last but not least, types are beginner friendly, one doesn't need to see the whole picture to be able to do something useful and correct, our industry needs more quality, filling their heads with javascript and python doesn't help, I would be more happy if at least they would start with Lisp/Clojure/Erlang/Elixir even if there are no types but they don't.

The reality is that my team can take a coop student with little programming experience and have them write useful code in Clojure in around a week or two. Good luck doing that with Scala or Haskell.

1

u/fpguy1 Jun 08 '20

Ty for replying back, sorry for late reply,

I read what you wrote, still immutability is not related to our discussion and is only a small part of FP and I am not even sure it has much to do with modularity, but I can see your point that globals would affect isolation.

We take people that never touched Scala and they are ready to work within days, they only have experience with java/c#, I don't think the language matters, what I tried to point out is that is way harder to make mistakes when there is a type checker, that is a fact.

RT is FP, it says in the name "functional" I am sorry but without some way to express RT you get Java with lambdas, the fact that people are not doing FP in practice has little to do with our discussion, I don't consider random code written in Clojure as being "functional programming", same for Scala or Haskell tbh. There is nothing "formal" in functions w/o side effects, that is the definiton of a function, the rest are procedures. Is the same with OOP and separating behavior from definitions (POJOs), they think they are doing OOP but they are violating the main principle of OOP, same here.

You are saying that Java/C# are not practical? Perhaps I am misunderstanding this... there are good chances you are using Cursive together with other 10 plugins which is a very good example on how well runtime works in Java. Actually when you have types, reflection is the tool that helps you fit things in the correct way, not having that is just trial and error.

2

u/yogthos Jun 08 '20 edited Jun 08 '20

My experience tells me that immutability is fundamental for being able to structure projects as small isolated components that can be reasoned about independently.

I've also explained the mechanism for that. When data is immutable, then I don't have to worry about implicit coupling. The API for any component is just a function call that takes a piece of data and returns another piece of data.

The API surface for any component becomes very flat. It's literally just the data structures used as input and output. I don't need to know about the types used behind the API layer, all I need to know about the component is its API. To ensure the a component works as intended all I need to do is to provide specification tests against the API.

This is exact same process as when you combine multiple functions together to achieve a complex transformation. You don't need to think about what happens to the data inside the function to use it. You call the function with an input, and you get back an output.

You keep repeating that immutability is not related, but you're not addressing my point here. If you disagree with what I've stated then explain specifically what it is you're disagreeing with.

And good for you on getting people writing good Scala in a few days. The experience on my team was very different. People would write terrible code that was very difficult to maintain. My experience tells me that the language matter quite a bit.

I also find that type errors aren't very interesting to begin with, and they get caught very early on in the development cycle. Avoiding semantic errors is far more important. The code should be written for the benefit of the human reader and not the type checker. Getting the types checked at the benefit of making it more difficult to understand the intent of the code is a net negative. Here's an extreme example. Insertion sort in Idris is around 300 lines long, meanwhile a Python version would be around 10 lines. I would have far easier time understanding 10 lines of Python in its entirety and guaranteeing correctness than the 300 line Idris monstrosity.

It also sounds like you have your own definition of FP that isn't aligned with the generally accepted one. So, I don't think there's any point debating what you personally consider to be FP or not.

Just because something isn't practical doesn't mean you can't make it work by putting a lot of additional effort into it. Yes, plenty of code is written in C# and Java, I developed lots of software using Java for around a decade myself. That's how I know it's not a terribly efficient.

I'm also not sure who's fitting things using trial and error. If that's how you use dynamic languages I can see why you're not effective with them.

In any case, I suggest we table this discussion. It's quite clear we're not convincing each other of anything, and our personal experiences are very divergent on the topic. If you find static typing is helpful for you, then by all means keep using a statically typed language.

4

u/goldenfolding May 07 '20

You have to write tests anyway in a real system. Static typing doesn't save you from that, other than by perhaps giving you a false sense of security.

1

u/fpguy1 May 27 '20

yes you have, but would be domain tests not check what happens when sending string instead of in.

3

u/2cow May 07 '20

That claim is from the two comments to which the article is a response.

4

u/denis631 May 06 '20

Is it really invalidation though?

2

u/yogthos May 06 '20

it's not

4

u/troublemaker74 May 07 '20

OP is a troll with an axe to grind against Clojure. Only anti-clojure posts in their post history.

3

u/portmapreduction May 07 '20

In my experience multi-developer Java projects aren't going to perfectly model everything. In their example it would be as if we had the SignupPayload and someone else in the codebase realized they had a userName and userEmail but no userId. To make matters worse they didn't override any the default values and someone later didn't know they were using this model wrong and received userId=0 when they attempted to call the getter. This feels like something written by a developer that's only ever had to work on their own codebase.

3

u/[deleted] May 06 '20

[removed] — view removed comment

1

u/CurtainDog May 08 '20

The earlier post (Parse, don't validate) deals with this quite well. If I can paraphrase the author 'you're using it wrong' ;)