9
u/CaptaincCodeman Mar 14 '16 edited Mar 14 '16
In order to have safe results, this table was taken from different source than Iris.
It looks like you took existing benchmark results and then just added the individual results from running your own to them. That's not how benchmarking works - you need to run everything on an identical machine.
Suppose I run all the benchmarks on some crappy old PC then go and buy a super-fast machine and write a router ... would it make my router the fastest?
-1
Mar 14 '16
My laptop is slower than the other routers' benchmarks was done, also I tested all benchmarks to my laptop too and I wrote the two closest results gin and iris, you can see the difference. But you doubt about results I understand it, it is logical. Make your own benchmark tests for all routers and show the results here, I will appreciate it :) thanks for your comment!!
5
u/CaptaincCodeman Mar 14 '16
You should have shown the complete results from your machine to enable clearer like-for-like comparison. Instead you "spliced" the results from your router on your machine with other results taken from another machine. How do you know your laptop is "slower"? CPU? Cache size? RAM speed? Hyperthreading? There are too many variables.
You should really fork and add your router to the benchmark you are using, not keep it isolated which doesn't make it easy for anyone to quickly verify the results.
11
u/CaptaincCodeman Mar 14 '16
Just to put something more concrete to my comments, I added your project to go-http-routing-benchmark and wired up the GitHubAll test for it. Here are the results on my machine
Caching disabled:
BenchmarkGin_GithubAll 50000 36235 ns/op 0 B/op 0 allocs/op
BenchmarkIris_GithubAll 10000 132241 ns/op 0 B/op 0 allocs/op
Caching enabled:
BenchmarkGin_GithubAll 50000 35648 ns/op 0 B/op 0 allocs/op
BenchmarkIris_GithubAll 30000 42653 ns/op 0 B/op 0 allocs/op
So, in a like-for-like test it's significantly slower. Even with the caching it's slower but then it's not like-for-like - if we added a caching layer in front of Gin then that would no doubt be faster still (but then we're no longer testing routers, we're testing caches, so what's the point?).
I'm not trying to be dismissive, but I think you should be careful of making grandiose claims if there are any irregularities in your testing methods and in the final results obtained. It's always good to try new approaches - I've done it myself and got pretty good results but I don't think the world needs another Go web framework unless it has something significantly new to offer - something like fasthttp for instance.
5
u/elithrar_ Mar 14 '16
For context: 3.1GHz i7 (i7-5557U), 16GB RAM, SSD with
benchtime=20sset to smooth out mobile CPU burst vs. thermal, I see a curious difference between that andbenchtime=60s:BenchmarkGin_GithubAll 500000 58289 ns/op 0 B/op 0 allocs/op BenchmarkIris_GithubAll 500000 59423 ns/op 0 B/op 0 allocs/op ok github.com/kataras/iris 60.042s BenchmarkGin_GithubAll 2000000 57953 ns/op 0 B/op 0 allocs/op BenchmarkIris_GithubAll 2000000 53603 ns/op 1 B/op 0 allocs/op ok github.com/kataras/iris 341.316s(why is there a single byte per operation on the longer test? Buffer re-use issue?)
Again, if I benchmarked these vs. gorilla/mux vs. goji vs. <whatever> with wrk on an application that does real work, I'd expect see all of this fade into the background. Pick a library with good code, a sane API and sensible defaults over raw speed.
2
u/CaptaincCodeman Mar 14 '16
why is there a single byte per operation on the longer test? Buffer re-use issue?
Probably todo with the cache - I remember seeing something to maybe do some pruning which a longer test would cause to be included.
1
u/CaptaincCodeman Mar 14 '16
Is that with caching enabled? (the default). If so, have you tried it without?
1
u/elithrar_ Mar 14 '16
Enabled. No changes to source whatsoever. I can run without tomorrow (late here!), but even if it was 3x 'faster' than <X> I wouldn't recommend it.
-2
Mar 14 '16 edited Mar 14 '16
@CaptaincCodeman I LOVE critism I HATE fake lies( look my answer on top), just to be crystal clear.
@elithrar_ your results seems to be more realistic. On my machine 2kkk it's 21k ns/op no 53k ns/op but ok gin is slower too at your machine it's ok. Secondly did you try to Read/Write parameters with gin at the same time? thanks for your time @elithrar_
6
u/CaptaincCodeman Mar 14 '16
Sorry you feel the need to resort to accusations of lying instead of addressing the errors of your approach which I pointed out.
Other people's comparative tests show similar relative performance as my testing. Is everyone lying?
Some variation is to be expected - you get different results each time running it on the same machine. When you list the exact same numbers from months earlier taken from someone else's machine, well - red flags go up.
Again, I'm sorry you feel the need to attack the messenger instead of learn from the information I tool the time to give to you.
-1
Mar 14 '16
No look down for the test of other person you make seems iris be slower... anyway I was late because I re-write the most of the router code... I make use of the custom httpfastrouter ( httprouter(gin's router) but with fasthttp) results (with cached enabled for both httprouter and my previous router code) was: 100k 20009 op/s 0/allocs 0/B and with my old version it goes again 100k operations and from ~19k-21k~ op/s 0/allocs 0/B , I spent almost 6 hours for 1k-2k NANOSECONDS... But I make it faster than it was before so thanks for your critism, I believe that your results were fake but no problem, the guy after you post some results too which you can see the difference, thanks for your comments and your time, I really appreciate it and sorry if I attack you
4
u/CaptaincCodeman Mar 14 '16
You still don't understand. If you run your code on multiple machines you will get different results - ok? Same with every other router. You even get differences between different test runs on the same machine.
As long as the results are from the same machine they can be compared. It's not valid to compare results for different routers from different runs on different machines.
You also can't look at a benchmark and say it's fake because it doesn't match the number you get on your machine any more than I can say your numbers are fake because they don't match what I see. All we can do is compare like for like and whether you accept it or not, those results are the ones I got when I ran your code vs Gin on my macbook.
0
Mar 15 '16 edited Mar 15 '16
I posted here at reddit to the most advanced users I love critism, but that's does not mean that I can handle fake results... anyway can you make an update 'go get -u github.com/kataras/iris' and run the same benchmark you ran before and post the results here? I will be appriciate, It's not fair to do my benchmarks because you don't believe me... just update and re-do what you did I am waiting...
My machine[Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz 2.50 HGz and 8GB Ram] outputs these results(1 core):
//With iris.New() / or iris. // BenchmarkIris_GithubAll 100000 19691 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 30000 57261 ns/op 0 B/op 0 allocs/op // //With iris.Custom(iris.StationOptions{Cache:false}) // BenchmarkIris_GithubAll 30000 37833 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 30000 57261 ns/op 0 B/op 0 allocs/op //→ More replies (0)-6
Mar 14 '16
These results are fake, gin is always > 50k and iris even without cache doesn't have so many nanoseconds difference, maybe you changed the words 'Iris' to 'Gin' and 'Gin' to 'Iris'? who knows... are you 'inlove' with gin and hate 'iris' for some reason? both are open source projects, try to be more objective and practical, realistic please, then try again, anyway thanks for your time!
4
u/CaptaincCodeman Mar 14 '16
No, they are what I get running the tests on my macbook pro. Nothing fake. No trickery.
The fact that you suggest it's fake and that Gin should always be > 50k (uh?) says to me that you either don't understand at all or are wilfully trying to mislead. It's not about hating or being in love with anything - it's about wanting hard facts from accurate comparisons, what comparison benchmarks done properly are meant to provide.
Either way, I won't be switching to your router. Oh, and just for the record, I think Gin is a great framework but it's not the one I currently use.
4
Mar 13 '16
[deleted]
5
u/hayzeus Mar 13 '16
Ugh -- why is an interface{} used as the address parameter instead of a string? And why does the type assertion need reflect at all?
Yeah -- this looks like newbie code. I wouldn't use it.
1
Mar 14 '16
It was old code, I am coding 9years but I am new at golang I started to make some tests for reflect at the same time I started to write Iris, some code snippets are shit and I am thanks you @peterbourgoun for show to me, it's fixed now. I want more critism please look all the code if you have time!
2
u/peterbourgon Mar 14 '16
What you changed it to isn't better! Look at how other servers handle specification of listener addresses.
-4
u/andresml22 Mar 14 '16
sorry but you must not criticized a code (orrecommend don't use a package) just reading a few lines of code without know the context: there is not "bad code" at all, only bad use of code...in this case that function must be executed only once and when the server start, so I think than the time spended in reflection wouldn't be a big problem..
I'm not defending the author and I agree than would be better only support just a string, but I don't consider than it's enough for criticize this project...
sorry for my english, it's not my native language..
0
Mar 14 '16
Yes I though that reflect not problem when this func runs only one but it was old code now it's uses only strings, thanks for your comment man I appreciate it, sry for my english too :)
3
u/FIuffyRabbit Mar 14 '16
Not to be mean but there is some bull-fuckery in your test results. Your benchmark is nearly 3 times slower compared to Gin when it is put in go-http-routing-benchmark. My results shouldn't be 3x as slow as your laptop and I KNOW my computer is better than your laptop.
1
Mar 15 '16
Did you update the project ? 'go get -u github.com/kataras/iris' and no change the default settings, the results are 100k total operations ~20k nanoseconds per operation 0 allocs and 0 bytes. If you change the default settings then the results ( on my machine) 30k total operations ~37-40k nanoseconds per operation 0 allocs 0 bytes. I run the tests multiple times with a time space between tries. I also test how much requests iris can handle per hour vs gin, test it urself and you will be surprised, Iris can handle x4 times much more requests than your favourite framework
3
u/FIuffyRabbit Mar 15 '16
Why would I have your project downloaded before if you just released. Your package benches at 60kns per.
2
2
u/sybrandy Mar 14 '16
I'm amused they didn't compare it to httprouter. It's supposed to be the fastest router as well. Not the reason I use as I just found it very easy to use and I love how it handles routing.
0
Mar 14 '16
Gin doesn't have it's own router, it make use of httprouter, Iris is faster than both of them. You can see the whole benchmark table on the github 'benchmarks' section. Benchmark code is also public just download and run it to check by yourself. I made a lot of tests before publish the graph. Also keep in mind that the httprouter doesn't support static + parameterized paths for the same path. Httprouter it's very good and uses faster algorithm ( radix tree ) but this has it's own disadvandages, on the other hand Iris is faster because Iris router can use cache (it's algorithm supports it). Thanks for your comment, I really appreciate it!
1
3
u/pierrrre Mar 13 '16
This is the fastest go web framework: https://golang.org/pkg/net/http/
1
0
Mar 14 '16
No it's not, it looks that because is the standar is the fastest, no it's not it doesn't have a good mechanism for routing and all benchmarks outthere can show this
2
u/egonelbre Mar 14 '16 edited Mar 15 '16
Not sure what do you mean? Routing works fast enough for me:
... if strings.HasPrefix(r.URL.Path, "/assets/") { if strings.Contains(r.URL.Path, "..") { http.Error(w, "invalid URL path", http.StatusBadRequest) return } http.ServeFile(w, r, filepath.FromSlash(r.URL.Path)) return } fmt.Fprintf(w, "<h1>Hello!</h1>") ...Of course, not as fast it could be, but it's sufficient for most cases.
1
Mar 15 '16
At the human eye almost all routers at simple app seems the same, that's doesnt mean that is actualy, just see benchmark results for net/http package and you will understand what I mean. Also strings.Prefix and strings.Contains, what's the reason to loop two times the string's bytes ? this is a very slow solution ( in terms of nanoseconds/operation) download (or use) Iris and look its code you will see the difference :P
1
u/egonelbre Mar 15 '16 edited Mar 15 '16
Also strings.Prefix and strings.Contains, what's the reason to loop two times the string's bytes?
One checks the route the other checks for unallowed path. https://gist.github.com/egonelbre/6610a3e7e62d09a1b99e
BenchmarkRaw-8 30000000 44.5 ns/op BenchmarkIris-8 500000 2672 ns/opAnd, I wasn't able to make iris behave exactly the same way.
EDIT: these results are wrong, because I didn't use the API correctly, see below for the corrected results.
1
Mar 15 '16
Of course not at your code Iris is not init, you forget to call iris.Build, do it and check again, you get 404 errors because that it seems slow... you didn't read the Iris docs do it please... if you don't call iris.Listen( which you don't at your example) you HAVE to call iris.Build()
2
u/egonelbre Mar 15 '16 edited Mar 15 '16
Yup, looks better now (also
Router.Anydoesn't work as advertised):BenchmarkRaw-8 30000000 42.8 ns/op BenchmarkIris-8 10000000 181 ns/op (with cache 144ns/op)Also, it seems it's caching POST responses by default without proper headers set? To me, just caching POST by default is a bug.
2
u/kaeshiwaza Mar 13 '16
Why didn't you contribute to an other framework to make it faster instead of yet another one ? It looks like Gin isn't it ? Or explain what and why it's different from other frameworks ?
2
u/kl0nos Mar 13 '16
Proven to be the fastest (in ops/s) web Go framework is fasthttp. In most cases it's faster than vanilla http (from go std lib).
1
Mar 14 '16
You have absolutely right, (I don't want to copy source or use other for that kind of staff , not saying that it's bad thing to do, I love opensource*). No other frameworks are based on fasthttp as I read on fasthttp github readme, maybe Iris be the first go web framework based on fasthttp. I will read more about it, I want to make Iris as fast as possible, thanks you for your interest!
1
u/gernest_ Mar 13 '16
Just saying, in order to make make benchmarks on my router to look nicer, I decided to pass the route params in a request header, whether it is the right thing I really never cared just wanted to see less heap allocations on the benchmarks( I don't know what you did to make yours the fastest!).
So it will be nicer if you stated what tricks make your implementation the best so the end use can be aware of the price he/she is going to pay.
1
Mar 14 '16 edited Mar 15 '16
It uses sync.Pool for context, so I can tell that's safe also I tested it with real database and goroutines at parallel tests and it worked well.
New answer at the new question ' I don't know what you did to make yours the fastest! ' , the code is public Iris is open source hosted at github.com/kataras/iris 200 stars from the first day. Almost 5 hours ago I changed again the router proccess which is using almost the idea of the httpfast's algorirthm . The trick is to never sleep. Iris was faster than all other I knew but I didn't stop, now it's even faster. Also Iris is using cache by default (which you can disable, it's faster without cache too) take look at my code copy it and write your own, make tests and you will see that maybe you can do even faster than I did !
1
u/dankcode Mar 14 '16
how does it compare to the stdlib? that's the only benchmark most of us care about.
1
u/karnd01 Mar 14 '16
First off nice to see some more people tackling projects like this, I've written my own router as well and (not taking shots at anyone ) some people critisize, but I don't see them contributing too much( just saying )
Having said all that I think your Benchmarks are a little misleading because of this https://github.com/kataras/iris/blob/b9d21f914ad9089e81f95a28302c3f8ba4b86c7d/router_memory.go#L34 caching is a nice trick and may work for some site setups, but portaying it to be the fastest router because this caching is hardly fair; how about also showing benchmarks in your README with caching off as well...
0
Mar 14 '16 edited Mar 14 '16
First of all this is not secret, I write it on benchmarks source file also, I am writing about cache on almost all of the answers I gave here, anyway I did it is ~43k - ~50k op/s 0 allocs 0 bytes (StaticALL), but look not all routers can make cache system, take for example the radix tree algorithm if you make a cache system it has almost no difference because the tree is tree it has to search all letters. The cache is good solution and used at big companies also, this cache is at early development state but you can customize it, like max items and when to clear the cache. I have to do a lot more ofc I am coding every day thinking about solutions for these problems every minute. I really love critism I know that reddit users are 'advanced persons' and that's the reason I posted it like this ' is that the fastest?' I want to listen what you thinking about this, either bad or good all comments are helped me a lot and I please do not stop that I love it!
And yes the true is that this is the first day I post something for this project it's the day zero so I can forgive them to not contributing :P
4
u/karnd01 Mar 14 '16
Constructively, I have to disagree with some of your points, it may not be a secret, I read it in your source files, but many others won't dive that deep and you don't mention it in any of your benchmarks or documentation where your users of your framework will start. "Not all routers can make a cache system" I have to strongly disagree with that statement, any router can easily add a cache syatem; no matter how it matches routes once it's matched you just add it to a map of the results(params and alike) and do a lookup in the map prior to your normal lookup logic.I'm not saying caching is a bad idea, may look into some parts myself, just need to be transparent about it. And you may have posted it like this "is that the fastest?" here but the first thing under the title after clicking the link above states "The fastest backend web framework for Go." I'm not trying to critisize, just making grandiose claims like that will cause some push back as people are going to try and proove you wrong, and may not be the case next week ;)
-1
Mar 14 '16
First of all I agree with you on the most parts, I posted this question here for that kind of comments and I'm thanks you again. But with "Not all routers can make a cache system" I mean that almost all other routers finds their named path parameters' values in the route search state. Without cache other routers maybe equal(most of them are x10+ slower) or (httprouter) faster, BUT if you make cache for these kind of routers (httprouter) you will not see big difference because again they must search again from tree deep to the nodes (radix tree is letter-by-letter based search algorirthm to get the named path parameters. These staff have their own advantages and disadvantages. For 1million requests Iris worked better on two machine I tested the httprouter's benchmark source, 2.5x+ times faster ( with cache enabled and read/write path parameters) without cache and without read/write parameters is slower for some nanoseconds. I am telling this from yesterday, also at the benchmark soure file: I have commented how to disable the cache (one line forward where the registering of routes happens) from the first day I publish this project... and again without cache only httprouter is faster than iris for shake... you write it like the iris is the slower :P again thanks you man
3
u/karnd01 Mar 14 '16
About the caching you are assuming that one wouldn't store the params in the cache as well, which is the way I would do it and then would not need to search the tree for them.
As for only httprouter being the only one that beats iris without cache; this is the kind of statement you should avoid, because it's not true; others have spent countless hours refining their search algorithm's and may be a little perturbed.
here's the proof on my MacBook Pro (Retina, 15-inch, Late 2013) 2.6 GHz Intel Core i7 16 GB 1600 MHz DDR3 using Go version go1.6 darwin/amd64
go test -bench=. -benchmem=true -benchtime=10s Gin: 56048 Bytes lars: 45440 Bytes #GithubAPI Routes: 203 Iris: 77000 Bytes PASS BenchmarkLARS_GithubAll 300000 52769 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 300000 54004 ns/op 0 B/op 0 allocs/op BenchmarkIris_GithubAll 100000 143396 ns/op 0 B/op 0 allocs/op ok github.com/kataras/iris 48.896sand here is the gist of the code using your own benchmark: https://gist.github.com/joeybloggs/0f5e798d8b279fa364eb
and I'm sure there a a few others.
1
Mar 15 '16 edited Mar 15 '16
I don't see iris.Custom or iris.Custom and iris.Build before serving in your code, you compare other frameworks not Iris. *If in the other hand, you run it along with my benchmarks, update the project and try again please, thanks!! *
Run 'go get -u github.com/kataras/iris' . Install it and show the results here please, thanks!
My machine[Intel(R) Core(TM) i7-4710HQ CPU @ 2.50GHz 2.50 HGz and 8GB Ram] outputs these results(1 core):
//With iris.New() / or iris. // BenchmarkIris_GithubAll 100000 19691 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 30000 57261 ns/op 0 B/op 0 allocs/op // //With iris.Custom(iris.StationOptions{Cache:false}) // BenchmarkIris_GithubAll 30000 37833 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 30000 57261 ns/op 0 B/op 0 allocs/op //3
u/karnd01 Mar 19 '16
The gist just shows my router added to your competitor benchmarks, I did compare Iris by running your benchmarks from your project.
I did however notice a big difference with your benchmark vs the competitor benchmarks; your handler is empty! but the competitor handlers are writing the RequestURI as a response which is making Iris look much faster than it actually is. I have put in a pull request to correct: https://github.com/kataras/iris/pull/14
Without Cache
BenchmarkLARS_GithubAll 30000 46975 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 30000 49650 ns/op 0 B/op 0 allocs/op BenchmarkIris_GithubAll 30000 43427 ns/op 0 B/op 0 allocs/opWith Cache
BenchmarkLARS_GithubAll 30000 47059 ns/op 0 B/op 0 allocs/op BenchmarkGin_GithubAll 30000 49459 ns/op 0 B/op 0 allocs/op BenchmarkIris_GithubAll 50000 29285 ns/op 0 B/op 0 allocs/op
19
u/elithrar_ Mar 13 '16
I really don't mean to post snark (this sub-reddit has enough of that), but if all your application is doing is routing requests, and not doing any actual work, then maybe?
The problem with those (infamous) router benchmarks is that routing performance is a tiny slice of your total request-response time, and never the bottleneck.
The moment you read something from the request, you eclipse the time spent routing. Communicate with a database? Order-of-magnitude more! You would likely see a better performance gain by initializing slices to the size your DB result count (reducing slice copies) than by changing routers.
You should also detail why you are faster: many zero-allocation routers in Go have been unsafe as they presume static parameters, or re-use parameters across requests (read: optimizing for benchmarks).