The following table displays the hardware specifications for Processing workers. These requirements are determined by the number of workers needed to achieve a specified range of throughput (# of GB/day).
Worker Specifications
CPU 8 core (physical)
RAM 16 GB
Network 1 Gbps
Storage Windows Temp SSD recommended
Server Physical recommended *
Expected throughput 100-150 GB/day **
* The primary reason for using physical workers is for performance. The overhead from virtualization can cause degradation in performance, particularly during text extraction and other CPU-intensive operations. If you do choose to virtualize your workers, beware of over-committing resources on the host. If you have hyper-threading enabled on the host, you may need to allocate 16 vCPUs to each worker to achieve results similar to those in the Processing performance baselines.
The reason why computers has the need to periodically check
for the DNS of websites that an individual has already visited because certain
websites can change its IP address and that the individual using it can be
directed to a different website that he or she does not know of.
