As we get new requests for AI Lab and FPGA Expansion Pack in more geographical regions like China, we are receiving feedback (and complaints) about latency and responsiveness. In anticipation of an upcoming event involving AI Lab, we ran a series of simple networking tests to determine the latency, bandwidth and in general, real-life responsiveness of the interaction via web VNC.
TL;DR version: If you want a responsive VNC session to a cloud server, low latency seems more important than bandwidth.
How it is conducted
We used a simple speed test tool called iperf. Iperf allows you to do a simple bandwidth test between two computers. Using the same client computer, which is a laptop in our Chengdu, China office, we targeted five AWS Regions. They were:
- Singapore
- Beijing
- Tokyo
- Seoul
- US East
We ran four tests for jitter with the command (udp)
iperf -c <ip> -u
Three tests for bandwidth with the command (tcp)
iperf -c <ip>
For UDP, we were particularly interested in the jitter(delay) as that seemed to be the main cause of lags in VNC.
For VNC, we used a web-based VNC client called novnc. For some reason (someone maybe please explain this to me) that it is more responsive than conventional VNC tools like RealVNC or TightVNC.
Results
Here are the jitter results. We averaged them and ranked them based on how good they were. (The lower the better)
| Jitter (ms) UDP | ||||||
| Region | Test 1 | Test 2 | Test 3 | Test4 | Avg | Rank (A) |
| Seoul | 6.135 | 6.11 | 3.342 | 2.248 | 4.45875 | 2 |
| Japan | 6.503 | 16.502 | 2.752 | 2.779 | 7.134 | 4 |
| China | 2.153 | 0 | 0 | 1.056 | 0.80225 | 1 |
| Singapore | 5.415 | 37.342 | 3.328 | 3.963 | 12.512 | 5 |
| US | 1.102 | 11.148 | 6.285 | 3.004 | 5.38475 | 3 |
Here are the results for bandwidth. A final score was derived by adding up rankings A and B. So the lower the score the better.
| TCP Bandwidth (Mbits/sec) | ||||||
| Region | Test 1 | Test 2 | Test 3 | Avg | Rank (B) | *Score (A+B) |
| Seoul | 12.3 | 14.5 | 12 | 12.93 | 3 | 5 |
| Japan | 16 | 18.8 | 16.7 | 17.17 | 2 | 6 |
| China | 30.1 | 25.9 | 19.1 | 25.03 | 1 | 2 |
| Singapore | 5.81 | 4.93 | 5.34 | 5.36 | 5 | 10 |
| US | 6.29 | 6.4 | 6.61 | 6.43 | 4 | 7 |
Responsiveness Test
While the numbers gave an idea of relative speed/latency, we wanted to experience the latency and responsiveness first-hand. So we fired up VNC, logged on to the servers and recorded videos of the VNC session. You can gauge for yourself if Seoul was indeed faster than US East, etc.
Takeaways and Observations
- Our experience (which can be seen from the videos) is that interactivity is highly related to jitter rate. While Singapore and US servers had the similar bandwidth values, the responsiveness of the Singapore server was much poorer compared to the US servers.
- There were actually not much differences between the Seoul and Tokyo servers in terms of responsiveness. But it seemed faster to upload/download files to the Tokyo servers as the bandwidth seemed higher. When comparing Tokyo to US servers, the Tokyo server felt more responsive than the US servers, perhaps due to higher bandwidth.
- AWS seemed to throttle or cap the bandwidth as you hit a certain limit. We experienced a notable slowdown in responsiveness as we were doing bandwidth tests with the US server.
- Latency matters a lot when you are using an interactive VNC session on AI Lab. Plunify Cloud uses a browser-based VNC client to access the server and the bandwidth requirements are pretty low. While monitoring using "Network" on Chrome's devtools, it uses between 1 to 10Kb for each packet.
Conclusion
We hope these give our AI Lab users a better understanding of what affects the latency. If you would like to try out AI Lab, feel free to register an account.