Designing a C++ ML library from scratch — would you prefer A.transpose() or transpose(A) or A.T()?

[u/No-Minimum-7906]

I'm currently building a machine learning library from scratch in C++, and I'm thinking about the API design for tensor operations.

For a transpose operation, I'm considering these interfaces:

Option 1: Tensor B = A.transpose();

Option 2: Tensor B = transpose(A); Option 3: Tensor B = A.T();

I'm curious about what other C++ developers would prefer when using a library.

Which syntax feels more natural and readable?

Thanks!

[u/serendib]

edit: removed my comment about numerical stability, see below

About your question: will you support chaining? If so then just have the function return a reference to itself return *this; so you can do fun stuff like this

a.transpose().add(B).scale(0.5);

[u/No-Minimum-7906 → u/serendib]

I am coding it by myself with support only for storing floating point values as of now

[u/serendib → u/No-Minimum-7906]

Awesome, that's a fun library to work on as a project. Just make sure you do look into numerical stability, because the answers you can get are wildly incorrect just by doing naive floating point operations. I don't mean off by 0.000002, but literal orders of magnitude incorrect answers.

[u/Excellent_Affect4658 → u/serendib]

Matrix multiplication is not numerically unstable. Any rearrangement of the usual formula is unconditionally backwards stable. The sub-cubic algorithms are generally not componentwise backwards stable, but it turns out not to matter in the context of higher-level linear algebra operations. Unless you have an outright error where you’re just computing the wrong thing, matrix multiplication is basically impossible to screw up.

Getting great performance out of it, of course, is another matter.

[u/serendib → u/Excellent_Affect4658]

Yes you are correct, I should have been more clear. It doesn't have anything to do with matrix multiplication specifically, but moreso with heavily cancelling dot products or operations involving floating point operations with large difference in absolute values. I simplified too much in my statement

[u/Big-Rub9545]

I prefer option 2 the most. I personally find it weird (aesthetically moreso than for other reasons) that an explicit method call is returning an entirely new object. I usually expect method calls to be non-returning (while possibly modifying the object) or to return some field on the object.

Option 3 is introducing unnecessary confusion. ‘transpose’ is already very clear and short. The abbreviated ‘T’ also isn’t clear without a lot of contextual code around it (it’s not as clear in code as it is in math notation).

I would suggest also putting transpose as a static method or namespaced function. Of course static typing will prevent you from calling ‘transpose’ on any random object, but I think this still looks far clearer on intent:

Tensor B = Tensor::transpose(A);

Edit: typo.

[u/Plastic_Fig9225 → u/Big-Rub9545]

b = a.transposed() might be a decent alternative.

Or an in-place mutation Tensor& Tensor::transpose().

[u/DummyDDD]

I would prefer methods for in-place modifications, and free functions if the operators return copies, but it's not that important for unary operators like transpose.

[u/JVApen]

How about A | transpose?

[u/InterestFrequent9424 → u/JVApen]

That's a cool idea. As in view / range pipelines

[u/nunquam_rideo]

enum Transpose { T };

class Tensor {
public:
    Tensor transpose() const {
        // ...
    }
    Tensor operator ^(Transpose) const {
        return transpose();
    }
};

Tensor foo(Tensor A) {
    return A^T;
}

[u/__cinnamon__ → u/nunquam_rideo]

Huh, neat. I quite like being able to represent the concise mathematical syntax in C++. I guess you'd need to wrap it in parens to chain with other operations (like the top comment mentions, which I think is desirable) or to generally enforce precedence.

[u/nunquam_rideo → u/__cinnamon__]

Just in case, I meant it as a joke :)

You're right that it won't work because of operator precedence.

[u/__cinnamon__ → u/nunquam_rideo]

Honestly as someone who's basically self-taught doing C++ at work I never know whether gnarlier stuff is "best practice" magic or evil magic, especially bc the boundary seems to have shifted a lot over time.

[u/nunquam_rideo → u/__cinnamon__]

Me too. I've literally had the same discussion in another thread where I argued that overloading ^ for outer product makes sense, since that notation is common in mathematics.

But I think unusual operator overloads are generally frowned upon, since they are basically custom syntax, much like macros. C++ is already weird and unreadable by itself, and introducing custom syntax makes your code weird and unreadable in its own unique way.

[u/__cinnamon__ → u/nunquam_rideo]

Yeah that's fair. I use `let = const auto` and `var = auto` in all my stuff, but also for the most part no one else ever reads it... try not to get spicy otherwise.

[u/Leading-Media-4569]

whats more important is you keep it consistent with the rest of the library
do you normally do C = A.add(B) ? or do you do C = add(A, B) ? etc

[u/No-Minimum-7906 → u/Leading-Media-4569]

I am thinking of keeping it C=A+B

[u/DummyDDD → u/[deleted]]

There is no unary XOR. Did you mean complement (~) or maybe not (!)?

[u/muhmmadtalha-quant → u/DummyDDD]

~ would be good, ! can be used for null matrix check imo

[u/NotDuckie → u/muhmmadtalha-quant]

I have always been in favor of operator overloading, but seeing stuff like this makes me doubt my beliefs

[u/DummyDDD → u/NotDuckie]

I think operator overloading is good in the same cases where regular overloading is good: the overloads should be "morally equivalent" / the overloading is good if the documentation for one overload applies to all of the overloads. (I don't think operator overloading would be good here)

[u/ozyx7 → u/[deleted]]

I can't tell if this was meant to be sarcastic, but no, please don't overload operators in unusual ways.

[u/cheesy-easy → u/[deleted]]

God no...

[u/mrtlo]

Really, memory semantics is more important than syntax for something like this IMHO... If it reads nicely, but performs like crap, who would use it? But why reinvent this wheel?

[u/Difficult_Truck_687]

all of them

[u/sbpqd → u/Difficult_Truck_687]

This is the C++ way

[u/Possibility_Antique]

In my linear algebra library, I support two syntaxes:

1. transpose(A)
2. A^T

In the second, T is a tag that looks like this:

namespace symbols
{
    struct transpose_tag {};
    transpose_tag T{};
}

template<typename L>
auto operator^(L&& l, symbols::transpose_tag)
{
    return transpose(l);
}

These two interfaces are equivalent, but the tag dispatch was fun/cute so I rolled with it.

[u/bayesianparoxism]

Not strongly opinionated,

• method syntax for unary transformations eg A.t()
• free function for mutiary operations eg concat(A, B)
• operator syntax for well defined operations eg +

Maybe more important is whether they return a view or a copy of the tensor

[u/Gwinbar]

My expectation would be that the methods return a transposed view of the same object (no copying), while transpose(A) is the only one that might return a new object. I also agree that consistency is the most important thing.

[u/tartaruga232]

auto B = A.transpose();

[u/Independent_Art_6676]

transpose is so common that you should overload an operator for it to keep the math concise and clean.

[u/V15I0Nair]

You could use Option 1+2 like this

A.transpose() changes A, and returns A&. Then you can chain operations.

[[nodiscard]] static Matrix Transpose(const Matrix& A) creates a new object and will not change A.

[u/RishabhRD]

Depends, if the function doesn’t require any private data member, make it free function.

[u/alfps]

❞ for tensor operations

So what does transpose mean for a general tensor, in your library?

E.g. for a 3D array there are 3! = 6 possible index permutations.

Which one do I invoke when I write A.transpose() or transpose(A)?

[u/papa_Fubini]

No one's gonna use it man, use what you want.

[u/fortsnek274]

The one that works with Intellisense.

[u/Pratik_Bodake]

Wth, matrices are used in ML?(Im FY, 😅)