If the scan messages are queued for more than one hour, the scanning process may time out. When the file uploading speed is higher than that amount, the scan messages are queued in the SQS ScannerQueue. If the latency is 1 second, the maximum number of scan messages that can be handled in 1 second is bound to 1000. The polling latency determines the actual amount of scan messages that can be handled simultaneously. In other words, ScannerLambda can poll at most 1000 scan messages from the SQS ScannerQueue simultaneously, even if there are still allowable concurrent executions. The maximum number of batches that an event source mapping can process simultaneously is 1000, and the batch size for the ScannerLambda is 1. The ScannerLambda receives scan messages by setting up an event source mapping to the SQS ScannerQueue. Additionally, the concurrent executions will be further split with any other Lambdas deployed under the same AWS account in the same region. Thus, if two File Storage Security scanners are deployed under the same AWS account in the same region, the 1000 concurrent executions will be shared among the scanners (they won't each get 1000). The AWS Lambda service has a default setting of 1000 total allowable concurrent executions, and File Storage Security's ScannerLambda function follows this configuration. How many files can be scanned concurrently? When a burst of traffic occurs in an architecture with one S3 bucket and one scanner, the BucketListener Lambda needs more concurrency. In addition, there are three Lambda concurrencies invoked in the scan process, namely, the BucketListener, Scanner, and PostScanActionTag Lambda concurrencies. When the shared or reserved concurrency is lower than the requirement for the used concurrency, delays can be expected due to throttles.
The data in the table was collected in a deployment where there was one storage stack and one scanner in the same region. You can leave this setting at its default (set by AWS) and make sure the Lambda concurrency in this AWS account is sufficient for the File Storage Security scan.īelow are some estimates of how long a scan takes and how much concurrency was used for various numbers of zipped 10 MB files.
The Lambda concurrency setting number includes not only the Lambdas that File Storage Security uses, but also all the other Lambdas used in the same AWS account. The exact scan time depends on the AWS concurrency settings of the Lambda functions, and well as how many other Lambdas you have running at the same time in your AWS account. If a large number of scanning requests hit File Storage Security all at once, multiple instances of the ScannerLambda will be invoked to process the requests in parallel. How are bursts in load handled? How do I estimate the scan time and how much concurrency is used when a burst of scanning occurs? The estimates assume that 512 MB of memory (the default) has been allocated to the File Storage Security scanner: The data was collected by uploading individual sample files to an S3 bucket that was in the same region as the scanner. AWS performance and scaling Performance metrics (scan times)īelow are some estimates of how long a scan takes for various file types.
0 kommentar(er)
