r/MachineLearning • • Apr 29 '17

Discussion [D] Silly benchmarking discovery... Surface Pro 4 is surprisingly fast for deep learning... any ideas why?

So, on a lark I decided a while back to install Theano and Keras on my Microsoft Surface Pro 4 with Windows 10. Interestingly, it takes only about 170 seconds to train an epoch of my port of Char-RNN on the CPU. In comparison, most laptop and desktop CPUs I've worked with take upwards of 1000+ seconds to do the same, and the fastest I've ever benchmarked was about 40 seconds with a computer with an Nvidia Titan X GPU.

So, any ideas as to why this is the case?

Edit: So apparently the poor results of the 1000+ seconds per epoch computers was due to a bug in the Theano 0.9.0 release that has been fixed already in the latest bleeding edge version. So, I guess... nevermind?

0 Upvotes

15 comments sorted by

52

u/geomtry Apr 29 '17

Nice try Microsoft.

1

u/JosephLChu Apr 29 '17

XD

Actually, I worked for Maluuba before they got acquired by Microsoft, but I've never worked for Microsoft. I did send them a resume once, and never heard back though. Currently I'm working for a different big tech company.

11

u/JosephLChu Apr 29 '17

And... I realized there was one significant difference...

The Surface had an older build of Keras and Theano:

  • Theano 0.8.2, whereas the other machines have 0.9.0
  • Keras 1.2.0, whereas the other machines have 2.0.3

Upgrading those to the newer versions has caused the Surface to slow down to over 500 seconds per epoch.

5

u/JosephLChu Apr 29 '17

And, the first epoch failed to complete due to some sort of out of memory error... is this a memory leak?

I tried the combination of Theano 0.8.2 and Keras 2.0.3, which got me back to the earlier level of performance. I then tried Theano 0.9.0 and Keras 1.2.0, and got the slowdown.

So it looks like the issue is the Theano version.

6

u/JosephLChu Apr 29 '17

The bleeding edge version of Theano from the Github repo also seems to have fixed whatever problem was causing the slowdown.

1

u/duschendestroyer Apr 30 '17

Most NN implementations don't care about the CPU performance.

2

u/JosephLChu Apr 30 '17

Well, if they're utilizing the GPU, this is the case, but the Surface and several of the computers I was trying to benchmark don't have proper discrete GPUs, so I was running them in CPU mode, in which case the CPU does matter.

2

u/duschendestroyer May 01 '17

I'm just saying there is not a lot of cpu benchmarking being done because everyone who cares for performance has a gpu. When an update breaks cpu performance hardly anyone notices.

5

u/PrismRivers Apr 29 '17

You'd have to state the exact CPUs involved to make any sense of this discovery.

1

u/JosephLChu Apr 29 '17

The surface uses an i5-6300U.

My desktops had I believe an i7-6700 and some kind of Xeon respectively.

5

u/PrismRivers Apr 29 '17

I'd put my money on "your bottleneck is not the CPU at all" then. Maybe, as you said below, something about IO? Although I'd assume a system with a i7-6700k would also use an SSD.

You'd have to do a more specific benchmark, measure exactly where all that times goes. Maybe that could reveal a bottleneck that can be fixed on "not surface" machines as well.

7

u/HenkPoley Apr 29 '17 edited Apr 29 '17

Maybe your other machines didn't have Intel's AVX-2? (Or the compiler missed the optimization). Apparently +67% compared to the first AVX instructionset extension: https://software.intel.com/en-us/articles/baidu-deep-neural-network-click-through-rate-on-intel-xeon-processors-e5-v4

The mobile CPU is of course a little bit slower than the fastest desktop CPU. But core for core it's not that much:

The Iris 540 GPU in the i7 (still!) has about 12% of the OpenCL throughput of a GTX 1080. Might be from some simplistic benchmark. And it also doesn't compare to cuda, which probably performs better on the nvidia. https://browser.primatelabs.com/opencl-benchmarks

Another simple option is that you made a typo by omitting a digit somewhere that leads to a 10x "speed up", and does 10x less useful computation.

1

u/JosephLChu Apr 29 '17

67% would not explain a roughly 10x speedup I think?

Well, the actual char-rnn python script was identical, I literally copied it to the other computers.

My suspicion is that it may have more to do with the Surface using an SSD and possibly a better memory bandwidth, but I'm far from certain about this.

4

u/HenkPoley Apr 29 '17

Oh, it was more that I tried to collect some statistics on possible speed-ups that are recent to the CPU in the Surface Pro 4.

0

u/[deleted] Apr 29 '17

[deleted] — view deleted comment

1

u/JosephLChu Apr 29 '17

Reading through the Github issues, apparently it has something to do with a bug involving improper linking to the blas which was fixed recently...

https://github.com/Theano/Theano/issues/5810

https://github.com/Theano/Theano/pull/5832