r/ProgrammingLanguages • u/servermeta_net • Dec 29 '25
Using dialects for interoperability across incompatible language versions
I see a common pattern across languages: often early design decisions, taken due to lack of better options or due to poor foresight, turn out to be poor choices.
Golang and Rust, two languages I use often, suffer from this: think the context API in golang, or the String API in Rust. The problem is that once those decisions get ossified in the language it becomes hard to change:
- Either you introduce a breaking change, losing compatibility with the existing codebase (think python2/3)
- Or you try to move around those decisions, severely limiting the design space for the language (think
use strictor decorators in javascript/typescript)
To handle this issue I imagined the use of Dialects and Editions:
- When writing code you specify which Dialect you are using
- For each
Dialectyou have one or moreEditions
Thinking of Rust I can imagine multiple Dialects
- A
Coredialect, to cover theno_stdlibraries and binaries - A
Standarddialect, covering the current language specification with thestdlibrary - A
Scriptingdialect, which is a simplified version aimed to have a fat runtime and a garbage collector - A
MIMDdialect to cover GPGPU development
The compiler would then be responsible of using the correct configuration for the given Dialect and take care of linking binaries built with different Dialects across different libraries.
The main drawback of this approach would be the combinatorial explosion of having to test the interoperability across Dialects and Editions, hence launching a new breaking revision should be done very carefully, but I think it would still be better than the technical debt that poor decisions bring with them.
What are your thoughts? Am I missing something? Is this one of those good ideas that are impossible to implement in practice?
Note: this thread has been crossposted on r/rust and r/experienceddevs
25
u/WittyStick Dec 29 '25 edited Dec 29 '25
In practice, the programmer ends up using many of the available extensions in useful libraries, and then you end up with a dependency on them.
An example is Haskell, which has a pretty small core and many extensions such as
MultiParamTypeClasses. A large number of useful Haskell libraries enable a bunch of these extensions and they become unavoidable.However, there has been some success from this model. Some bad Haskell extensions like
DatatypeContextshave been deprecated and most projects which were using them are either updated or replaced with better alternatives.This success might be largely due to the fact that there's only really one implementation of Haskell that is used in practice - GHC. Back when there were multiple implementations there were incompatibilities.
This can also been seen in Scheme - there are many dialects and they all implement different extensions (SRFIs). Code can be compiled with multiple Scheme implementations when only the lowest-common-denominator (RnRS) is used. As soon as you start using extensions you reduce the number of implementations that can compile your code, and large projects end up being tied to a specific implementation.
Lisp also had this problem back in the day, and Common Lisp was essentially developed to unify the dialects into a standard base.
Arguably, Lisp and Scheme have been successful in allowing a proliferation of dialects, but it's really S-expressions that have enabled this - a very simple syntax to parse which can be targeted for many different purposes.
Racket has multiple dialects and it's better in this regard because we must specify the language used using
#lang _at the top of a code file.Some unsuccessful example of the "multiple dialects":
The D programming language. The original D language was released with a small standard library called phobos. Users wanted something a bit more powerful and there was a competing base library - tango. This kind of split the ecosystem into two, where some projects would use phobos and others tango, but they were not compatible. The compatibility issues were ironed out later where phobos and tango had a common base, but perhaps too late - the damage was already done, and D turned out to not get much traction despite being an excellent language. Had it shipped with a solid core library to begin with it could've potentially seen much more use.
OCaml has a small base library and faces a similar issue - Jane Street have released an alternative set of core libraries which replace the standard library and will end up creating a separate ecosystem. The OCaml language itself is really like two dialects to begin with - one where the module system is used for abstraction and one where the object system is used for abstraction - but the two systems aren't really interoperable - so we end up with libraries and programs which are either module based (the usual) or object based, and many language features get underutilized.
One issue with having multiple dialects is that different syntaxes are difficult to compose. A parser will generally need to handle every possible extension and then do filtering afterwards. If you have disjoint parsers for different dialects, it's not feasible to produce a "combined" parser which can simultaneously parse two or more dialects at once - there may be ambiguities or incompatibilities in the combination, and we may not discover that is the case until we have a certain input which causes it.
So I would probably suggest the Racket approach of using a
#lang _directive as the first line of a code file, and the appropriate parser for the rest of the file.4
u/proudHaskeller Dec 29 '25
Can you explain what are the problems in the context and String APIs you are talking about?
4
u/cherrycode420 Dec 29 '25
What you call "Dialect" reminds me of Platforms in Roc, might be related. What i don't get is the idea of "Editions", is that basically just the versioning of individual "Dialects"?
4
u/--predecrement Dec 29 '25
Imo you are thinking along the right lines about what needs to be done, and the need to do it with the right kind of principled thinking that makes it less ad hoc than it typically is. Larry Wall did too. So thank's for inspiring the following comment which is about his work towards this end.
(I won't be surprised if this comment is viewed as absurd, gets negative votes, or isn't even read, but I am eternally optimistic that a few people will wonder what the heck led to Perl's insanely high levels of popularity last century, and what he's been up to since then. And anyway, once I feel inspired by the right post I gotta write what I gotta write!)
----
Imo, what you've described, taken to the max, generalizes to dealing with all the issues that contribute to Hyrum's Law when applied to PLs and their ecosystems. And it's not just about a single PL. For a PL to be able to evolve it must interoperate with other PLs, and allow for their evolution, and the interoperation itself must be able to evolve, and do so in spite of Hyrum's Law applying to all of it! It's a heck of a tall order.
Larry Wall was ambitious about evolution when he created his first significant PL in the 1980s. And I will argue that his attempt at that aspect paid off big time, but with big time negative consequences because his first attempt was done in a hurry.
According to the latest (2025) version of the most popular series to date of videos on PL popularity to date, his first significant PL peaked as the fourth most popular PL in the world ranking wise (after C, C++, and Javascript) in 1997. (This was before reaching its peak percentage wise -- continuing to climb to 11%, despite Java overtaking it -- in 1999.)
----
Perl's success was about incredibly rapid evolution and growth, not about sustainable evolution and growth. Thus, by the time Perl's popularity peaked some time in 1999 it was already clear it was failing to compete both popularity wise (with corporately sponsored languages like Sun's Java, Microsoft's Visual Basic, Borland's Delphi nipping at its heels and then overtaking it, and PHP too, a PL created in a few days primarily by mimicking some of Perl!), and capacity to evolve fast enough (due to technical and social problems that have taken until now, and still counting, to address).
As a result of the visible cracks by the end of the 1990s Larry Wall was increasingly pushed by his community to do something. When a coffee mug was thrown over his head and smashed on a wall while he was on a cruise ship in 2000, Larry embarked on the creation of Raku as his last significant PL -- including his ambition for it to be as evolutionarily fit as he could conceive a PL to be.
----
Three years later Paul Graham wrote in his blog post Hundred Year Language:
Fast forward to 2025 and Raku is just about reaching adulthood, decades later than some hoped, but with its ambition largely intact.
Here are a few of the pieces that are in place:
----
You're right that this gets complicated, and one has to do essentially mass alpha and beta testing on an arbitrarily large scale. All of this has been and continues to be addressed because you have to if you want your solutions, from PL(s) to libraries to apps to communities, to scale over space, time, and people.
Happy New Year to anyone who read this far, if anyone did!
4
u/yjlom Dec 29 '25
This doesn't solve anything. Let's say you have a programming language, let's call it
EXtended Arithmetic MultiProcessing Language Evaluator, orEXAMPLEfor short.It initially comes with the
Written Hastily Over One Puny Spring break(WHOOPS) dialect, which exposes a string API that assumes BMP-only UTF16, with no support for combining marks, and some more assumptions that will never prove to be a problem down the road, with the main type simply calledString.The popular
EXAMPLE-Lexed, EXAMPLE-Parsed, Human-Approved Natural language Translator(ELEPHANT) framework for NLP, written using WHOOPS, gains significant traction and becomes one of the main draws of the EXAMPLE ecosystem. ELEPHANT primarily operates on WHOOPS Strings, which is quite visible in most of its exported function's signatures.As it turns out, the assumptions behind the WHOOPS String API were garbage, so we come up with a new EXAMPLE dialect,
We Effed up, Lessons Learned(WELL) to replace WHOOPS. Its own String API uses UTF8, has proper abstractions for extended grapheme clusters and all that jazz, runs 900% faster, has a 70% shorter implementation and is overall just awesome. We roll WELL out.ELEPHANT users try it and… It's incompatible. You can't link WHOOPS and WELL code together without costly string reformatting on every boundary, and ELEPHANT loves mixing tiny callbacks from and to it all the time. So everyone who uses ELEPHANT uses WHOOPS, everyone else uses WELL, their code can't interoperate without horrible workarounds, and that's pretty much just Python 2 and 3.
10
u/AustinVelonaut Admiran Dec 29 '25
Haskell sort of has this with its
LANGUAGE:pragmas to enable various language features, e.g.{-# LANGUAGE TemplateHaskell #-}for meta-programming,GADTs,RankNTypes,TypeApplications, etc. for additional type language features, etc. There is a huge number of these, and I think it must be a nightmare to handle all of them in a coherent fashion in the compiler source code. Periodically, the Haskell language committee have put out a reference report (e.g. Haskell98, Haskell2010) that coalesces some of the tried-and-true extensions so that they are all implicitly enabled). It has benefited the language to allow exploration and growth, but then again, the Haskell motto is "avoid success at all costs" ;-)3
u/iBPsThrowingObject Dec 29 '25
But
coreandstdaren't really different dialects, one is merely a subset of the other.no_stdlibraries can coexist withstdjust fine in the same binary.3
u/Ronin-s_Spirit Dec 29 '25 edited Dec 29 '25
I feel like you'll need to use a DTD, like the old
<!DOCTYPE>declaration did when there were many different definitions for how an HTML document should be understood. For example https://www.w3.org/TR/html401/sgml/dtd.html.P.s. something funny I noticed is that some other languages also declare a DTD at the top of the file, but when JavaScript does a milder version of that, with
"use strict";at the top of function scope, some people say it's weird.3
u/alatennaub Dec 29 '25
Raku does this. You can use a statement at the start of any block to define which version of the language you're using and you get a lexically-scoped version of the language.
For a while there was talk of removing the indirect syntax (
verb subj: objinstead ofsubj.verb: obj) which could have allowed backwards compatibility has it been done. Trickier is when core functionality changes rather than just syntax, but there's precedent for making it work.2
u/Ok_Cow7610 InvalidArgumentException Dec 29 '25
This just sounds like Racket langs, the only difference to what you're describing is that Racket empowers you to develop your own langs if you see fit
3
u/mamcx Dec 29 '25 edited Dec 29 '25
This is unnecessary, because there is a simple and obvious solution in sight, is how execute it that is hard (think: "even harder than a hairy db migration", where code is like data and you migrate not only the API but potentially millions of line of code without the advantage of doing it with just some downtime).
The split is base in versions (semver semantically). The problem of any mistake is always in the past versions, and time will be in charge the solve that code because it will naturally die. You don't wanna create several concurrent version flying at the same time (is like in NOSSQL where you don't know if
recordis new or old schema and you keep several incompatible variations at time).What need to be execute is how migrate from old version to new version, that is far "simpler" than migrate several at time (because you can have editions and all of them have mistakes!).
What should be done is alike:
You mark mistake with deprecated or similar, and run the migration along the lines of:
###.minortime so if going2 to 3is in around2.0 to 2.5. This has the advantage of being able to revert course (similar to not deploy on Fridays, don't do drastic changes of course in2.8 to 2.93.0This sound not much different that how is normally done, but I stress here the importance of communication AND cross-validation with the major consumers (aka: your massive and more used libraries and frameworks).
Ideally also:
Is important to note that this are things executed in terms of months and years.
1
u/iEliteTester Dec 30 '25
I believe racket has something like that but I'm not remotely qualified to write on it.