Background:
Compilation runs generated and executed in InTime seem to result in the server machine(s) hanging or running out of system resources like memory.
InTime can issue compilations in parallel based on the user-specified Concurrent Runs property. This has the benefit of exploring more compilation strategies and shortening the amount of time required to get an optimized design. However, it is vital to ensure that following conditions are met:
- Compilation servers have sufficient system resources, especially memory.
For example, if one compilation usually consumes 64GB of memory, then for Concurrent Runs = 10, InTime needs at least 64GB x 10 = 640GB of memory. - There is a way to control how jobs are launched by all users of the servers.
Before launching a job, check how much memory, CPU cores and disk space are available.
More Information:
Concurrent Runs (user-configurable) is a property that the user can set in InTime. This value limits the number of concurrent compilations for a particular job. For example if a user sets Concurrent Runs to 2, this means that the maximum number of compilations in parallel is capped at 2.
The Run Target (user-configurable) property specifies where to run the compilations. Options include the local machine ("Local"), a server farm ("Private Cloud") or in the Amazon Web Services cloud ("Plunify Cloud").
Peak memory usage: All FPGA software output the amount of peak memory used during compilation. Refer to your compilation logs to determine how much memory each compilation typically uses.
Scenario #1: Run Target = Local, shared server
- A single shared server has 256GB RAM. Each compilation requires approximately 64GB of RAM.
- InTime generates nine strategies to compile.
- Concurrent Runs = (256 / 64) - 1 = 3, so that the server has spare memory left.
- Ensure that other users do not start intensive processes on the same server.
Outcome: On a shared server, leave some resources for the Operating System and for other users.
However, if other users start intensive processes on the same server, memory and other system resources will be affected. In that situation, set the Concurrent Runs to 1.
Scenario #2: Run Target = Local, dedicated server
- A dedicated server has 512GB RAM. Each compilation requires approximately 100GB of RAM.
- InTime generates nine strategies to compile.
- Concurrent Runs = 512 / 100 = 5
Outcome: On a dedicated server, leave some resources for the Operating System.
Scenario #3: LSF Server Farm
- A server farm contains 10 server machines with 512GB RAM each. Each compilation requires approximately 100GB of RAM.
- InTime generates nine strategies to compile.
- Set up InTime Private Cloud settings according to the steps outlined in https://docs.plunify.com/intime/latest/configuration_lsf.html
- Concurrent Runs = 1000
InTime will automatically adjust the Concurrent Runs to match the number of strategies.
Outcome: In a server farm, the grid management software e.g. LSF / SGE / SLURM etc. typically limits the number of processes that can be launched in the farm. InTime uses the same server farm settings to support the number of compilations in parallel.
Applies to:
- Linux Operating System
- Windows Operating System
Knowledge Base ID: 71300 - Last Review: Aug 14, 2022 - Revision: 1.0