{"id":3302,"date":"2018-04-17T07:18:13","date_gmt":"2018-04-17T07:18:13","guid":{"rendered":"https:\/\/support.plunify.com\/en\/?p=3302"},"modified":"2018-08-28T03:22:46","modified_gmt":"2018-08-28T03:22:46","slug":"compare-timing-performance-of-seed-sweep","status":"publish","type":"post","link":"https:\/\/support.plunify.com\/en\/2018\/04\/17\/compare-timing-performance-of-seed-sweep\/","title":{"rendered":"How Much Do You Really Know About Placement Seed Sweep?"},"content":{"rendered":"
This whitepaper compares the effectiveness of two timing optimization methods: The InTime Default recipe provided by the InTime FPGA design optimization tool and another one commonly known as a \u201cSeed Sweep\u201d.<\/p>\n
InTime Default is a machine learning approach that finds good synthesis and place-and-route setting combinations for a design. It shares data insights across different designs and produces predictable effects.<\/p>\n
A Seed Sweep varies the Quartus Fitter SEED value which affects the initial placement of a design. Changing the seed modifies the conditions of a design at the start of place-and-route and leads to fluctuations in the Fitter results\u00a0. This is a well-known approach used by design teams worldwide to optimize their designs. However, the effects are random and there is no \u201cgolden\u201d seed value that applies to all designs.<\/p>\n
The experiment described here is performed using a Stratix V design and an Arria 10 design, each compiling 200 data points for the InTime Default recipe and 200 data points for a Seed Sweep for each design. The best Worst Slack (WS), Total Negative Slack (TNS), Fmax and runtimes are then compared.<\/p>\n
For the Stratix V design, InTime Default improved the Worst Slack by 57.38%<\/u> while Seed Sweep only had 13.46%<\/u> improvement.<\/p>\n
For the Arria 10 design, InTime Default recipe improved the Worst Slack by 34.98%<\/u> while Seed Sweep only had 11.44%<\/u> improvement.<\/p>\n
In summary, the InTime Default recipe performed better than a Seed Sweep with respect to timing performance.<\/p>\n
Design Details<\/h2>\n
The two designs used for this experiment are listed below.<\/p>\n
Table 2.1: Stratix V design<\/p>\n
\n\n
\n
Design Info\u00a0 <\/strong><\/td>\n
<\/td>\n<\/tr>\n
\n
\u00a0Quartus Prime Version<\/td>\n
\u00a017.1.0 Build 590 10\/25\/2017 SJ Standard Edition<\/td>\n<\/tr>\n
The experiment was performed via the following steps:<\/p>\n
\n
\u00a0Open Stratix V design in InTime.<\/li>\n
\u00a0Run InTime Default recipe for 20 rounds. Each round will run 20 compilations.<\/li>\n
\u00a0Run Seed Sweep for 200 compilations, each with a different seed value.<\/li>\n
\u00a0Compare the Worst Slack (WS) ,Total Negative Slack (TNS) and Fmax.<\/li>\n
\u00a0Repeat Steps 1 to 4 for the Arria 10 design.<\/li>\n<\/ol>\n
Results<\/h2>\n
Tables 4.1 ,4.2 and 4.3 below compare the timing results between the Stratix V and Arria 10 designs using WS ,TNS values and Fmax.<\/p>\n
Table 4.1: Worst Slack (WS) comparison<\/p>\n
\n\n
\n
\n
Best Worst Slack, WS (ns)<\/p>\n<\/td>\n<\/tr>\n
\n
<\/td>\n
Original (ns)<\/td>\n
Seed Sweep<\/td>\n
InTime Default<\/td>\n<\/tr>\n
\n
Slack(ns)<\/td>\n
%<\/td>\n
Slack(ns)<\/td>\n
%<\/td>\n<\/tr>\n
\n
Stratix V design<\/td>\n
-0.765<\/td>\n
-0.662<\/td>\n
13.46<\/td>\n
-0.326<\/td>\n
57.38<\/td>\n<\/tr>\n
\n
Arria 10 design<\/td>\n
-0.769<\/td>\n
-0.681<\/td>\n
11.44<\/td>\n
-0.5<\/td>\n
34.98<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Table 4.2: Total Negative Slack (TNS) comparison<\/p>\n
\n\n
\n
\n
Best Total Negative Slack, TNS (ns)<\/p>\n<\/td>\n<\/tr>\n
\n
<\/td>\n
Original (ns)<\/td>\n
Seed Sweep<\/td>\n
InTime Default<\/td>\n<\/tr>\n
\n
Slack(ns)<\/td>\n
%<\/td>\n
Slack(ns)<\/td>\n
%<\/td>\n<\/tr>\n
\n
Stratix V design<\/td>\n
-897.292<\/td>\n
-745.228<\/td>\n
16.95<\/td>\n
-39.736<\/td>\n
95.57<\/td>\n<\/tr>\n
\n
Arria 10 design<\/td>\n
-977.742<\/td>\n
-635.613<\/td>\n
34.99<\/td>\n
-75.44<\/td>\n
92.28<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Table 4.3:Fmax comparison<\/p>\n
\n\n
\n
\n
Fmax\u00a0 (MHz)<\/p>\n<\/td>\n<\/tr>\n
\n
<\/td>\n
Original (MHz)<\/td>\n
Seed Sweep<\/td>\n
InTime Default<\/td>\n<\/tr>\n
\n
Fmax(MHz)<\/td>\n
%<\/td>\n
Fmax(MHz)<\/td>\n
%<\/td>\n<\/tr>\n
\n
Stratix V design<\/td>\n
306.28<\/td>\n
316.25<\/td>\n
3.26<\/td>\n
353.89<\/td>\n
15.54<\/td>\n<\/tr>\n
\n
Arria 10 design<\/td>\n
305.90<\/td>\n
314.37<\/td>\n
2.76<\/td>\n
333.33<\/td>\n
8.97<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
Optimization Process and Run Time<\/h2>\n
Running a Seed Sweep is a simple process of specifying \u201cFitter Seed\u201d values from 1 to 200. The default value is 1. The InTime Default recipe involves running 10 rounds of 20 compilations each. Each round starts after the previous round has ended. Therefore, there are differences in the overall runtime of the two approaches.<\/p>\n
Table 5.1 below shows design runtime differences. It is worth noting that the average runtime for InTime Default is higher than that for Seed Sweep. In addition, there are larger variations in the compilation times for InTime Default as the settings attempted are more varied than seeds. The total runtime is also longer for InTime Default due to the need for analyzing results at the end of each round to generate settings for the next.<\/p>\n
Table 5.1: Run Time comparison based on different concurrent runs and rounds<\/p>\n
\n\n
\n
\n
Run Time (h)<\/p>\n<\/td>\n<\/tr>\n
\n
<\/td>\n
Original Run Time (h)<\/td>\n
Concurrent Runs<\/td>\n
Seed Sweep \n(1 round of 200 compilations)<\/td>\n
InTime Default \n(10 rounds of 20 compilations)<\/td>\n<\/tr>\n
\n
Total Run Time (h)<\/td>\n
Avg Run Time (h)<\/td>\n
Total Run Time (h)<\/td>\n
Avg Run Time (h)<\/td>\n<\/tr>\n
\n
Stratix V design<\/td>\n
0.5<\/td>\n
5<\/td>\n
26<\/td>\n
0.6<\/td>\n
37<\/td>\n
0.7<\/td>\n<\/tr>\n
\n
Arria 10 design<\/td>\n
0.5<\/td>\n
3<\/td>\n
37<\/td>\n
0.8<\/td>\n
38<\/td>\n
1.0<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n
5.1 Stratix V design results<\/h3>\n
Here are the results for the Stratix V design.<\/p>\n
5.1.1 Seed Sweep<\/strong><\/p>\n
<\/p>\n
Figure 5.1.1: TNS(ns) values for 200 seeds<\/p>\n
<\/p>\n
Figure 5.1.2: WS(ns) values for 200 seeds<\/p>\n
5.1.2 InTime Default<\/strong><\/p>\n
(Note: The Y-axis is in a logarithmic scale due to larger fluctuations)<\/p>\n
<\/p>\n
Figure 5.1.3: TNS values for InTime Default<\/p>\n
<\/p>\n
<\/p>\n
Figure 5.1.4: WS values for InTime Default<\/p>\n
The chart below shows how the results improved across rounds. The X-axis displays job numbers. Each column represents the timing results for a job. The green line is the best result in each job and the red line is the worst.<\/p>\n
<\/p>\n
Figure 5.1.5: TNS values for InTime Default<\/p>\n
<\/p>\n
<\/p>\n
Figure 5.1.6: WS values for InTime Default<\/p>\n
5.2 Arria 10 design results<\/h3>\n
Here are the results for the Arria 10 design.<\/p>\n
5.2.1 Seed Sweep<\/strong> \nThe following chart shows the results for 1 to 200 seed values. Green represents results that are better than the original result and blue indicates results that are worse.<\/p>\n
<\/p>\n
Figure 5.2.1: TNS (ns) values for 200 seeds<\/p>\n
<\/p>\n
Figure 5.2.2: WS(ns) values for 200 seeds<\/p>\n
5.2.2 InTime Default<\/strong> \nInTime runs 20 compilations each time for 10 rounds. When each round completes, InTime analyzes and learns from the data to generate the parameters for the next round.<\/p>\n
The following charts show the results for 200 compilations of InTime. Note that the Y-axis is in a logarithmic scale due to larger fluctuations in timing results.<\/p>\n
<\/p>\n
Figure 5.2.3: TNS values for InTime Default<\/p>\n
<\/p>\n
Figure 5.2.4: WS values for InTime Default<\/p>\n
The chart below shows how the results improve across rounds. The X-axis displays job numbers. Each column represents a job. The green line is the best result in each job and the red line is the worst.<\/p>\n
<\/p>\n
Figure 5.2.5: TNS improvements for InTime Default (10 rounds)<\/p>\n
<\/p>\n
Figure 5.2.6: WS values for InTime Default (10 rounds)<\/p>\n
Conclusion<\/h2>\n
InTime is more effective at delivering higher timing performance. In the same number of compilations runs (200), InTime Default is able to produce 30% to 50% performance improvements as compared to only 11% to 14% gains for a Seed Sweep.<\/p>\n
Furthermore, there are even greater performance improvements if both methods are used in conjunction \u2013 namely, by doing a Seed Sweep based on good InTime Default results. This approach is used by the InTime tool to enhance timing performance and accelerate time-to-market. The key is to first get a sufficiently good result before using seeds to get one over the finishing line.<\/p>\n
![\"5.1.1\"](\"https:\/\/i1.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.1.1.png?resize=849%2C496&ssl=1\")
<\/p>\n![\"5.1.2\"](\"https:\/\/i0.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.1.2.png?resize=850%2C494&ssl=1\")
<\/p>\n![\"5.1.3\"](\"https:\/\/i0.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.1.3.png?resize=960%2C528&ssl=1\")
<\/p>\n![\"5.1.4\"](\"https:\/\/i0.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.1.4.png?resize=960%2C520&ssl=1\")
<\/p>\n![\"5.1.5\"](\"https:\/\/i2.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.1.5.png?resize=960%2C611&ssl=1\")
<\/p>\n![\"5.1.6\"](\"https:\/\/i0.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.1.6.png?resize=960%2C613&ssl=1\")
<\/p>\n![\"5.2.1\"](\"https:\/\/i0.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.2.1.png?resize=884%2C543&ssl=1\")
<\/p>\n![\"5.2.2\"](\"https:\/\/i0.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.2.2.png?resize=944%2C540&ssl=1\")
<\/p>\n![\"5.2.3\"](\"https:\/\/i1.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.2.3.png?resize=854%2C489&ssl=1\")
<\/p>\n![\"5.2.4\"](\"https:\/\/i1.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.2.4.png?resize=853%2C491&ssl=1\")
<\/p>\n![\"5.2.5\"](\"https:\/\/i1.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.2.5.png?resize=840%2C545&ssl=1\")
<\/p>\n![\"5.2.6\"](\"https:\/\/i2.wp.com\/support.plunify.com\/en\/wp-content\/uploads\/sites\/5\/2018\/04\/5.2.6.png?resize=843%2C542&ssl=1\")
<\/p>\n