Vivado 2016.4 and Vivado 2017.1 comparison

In this blog post we will compare the performance of Vivado 2016.4 and Vivado 2017.1 using InTime.

In summary, 3 experiments were conducted to test the performance of Vivado 2016.4 against Vivado 2017.1. Overall, Vivado 2017.1 achieves better results than Vivado 2016.4. Although the results are somewhat similar in experiment 1, the best TNS and Worst Slack acquired from Vivado 2017.1 was much better than its predecessor in experiments 2 and 3.

Methods

For this performance test, we will be using a design named “cpu” with a target device of "xc7k70tfbg676-2".

Comparison between the 2 toolchains will be done using the following experiments:
Settings used for compilations will be generated using InTime with Vivado 2016.4. The same set of settings will be used by Vivado 2017.1 for compilation.
Settings used for compilations will be generated using InTime with Vivado 2017.1. The same set of settings will be used by Vivado 2016.4 for compilation.

For each experiment, a total of 150 compilations were made for each toolchain. InTime learns from past results and this has to be taken into account when testing the performance of these toolchains. By running the same experiment twice, one can then analyze how well or bad InTime learns from different toolchains.

Experiment 1

The results of experiment 1 can be found below. The table list some basic statistics of the results while the graphs show the TNS and Worst Slack values of each compilation for each toolchain. An analysis of the table shows that the best TNS for both toolchains are actually quite similar. This is a different story for Worst Slack where Vivado 2017.1 performed better. The distribution of TNS and Worst Slack for Vivado 2017.1 is lower and less varied as compared to Vivado 2016.4, indicating that the newer Vivado toolchain is more stable and produces better, though just slightly, better results than its previous version.

Looking at the graph, we can tell that the bad results in Vivado 2016.4 (red line) has actually improved in Vivado 2017.1 (blue line). On the other hand, the good results in Vivado 2016.4 actually became worse in the newer version of Vivado. Overall, it seems that the performance of Vivado 2017.1 is just slightly better than 2016.4.

Experiment 2

The results of experiment 2 is reported in a similar format as experiment 1. In this experiment, Vivado 2017.1 out performs Vivado 2016.4. Notice how the results of Vivado 2016.4 (red line) tends to lie below the Vivado 2017.1 results (blue line). This indicates that the newer toolchain generally produces better results.

One might notice that the number of successfully compilations for experiment 2 is much higher than that of experiment 1. This indicates that the results produced by Vivado 2017.1 has a higher correlation with the settings and this has allowed InTime to learn and avoid unsuccessful compilations.

Comparison against eight_bit_uc

We should of course compare our results against another design before coming to any conclusion. In this 3rd experiment, we repeated experiment one using a design named “eight_bit_uc” design (target device “xc7k70tfbg484-2”). The results are reported below. The story here is the same as the previous 2 experiments - Vivado 2017.1 out performs Vivado 2016.4 in almost all aspects.

Conclusion

In conclusion, Vivado 2017.1 achieves better results than Vivado 2016.4. Although the results are somewhat similar in experiment 1, the best TNS and Worst Slack acquired from Vivado 2017.1 was much better than its predecessor in experiments 2 and 3.