Resource usage monitoring for web systems using real-time statistical analysis of log data

For Web-based systems accessible from the Internet, it is difficult to estimate workloads precisely. Precise estimation of resources necessary for the system is important for effective utilization of resources in a datacenter. Therefore, capacity planning to forecast the amount of IT resources necessary for a system is important. In capacity planning, the amount of resources necessary for a system is calculated based upon numbers determined by the architecture and business requirements of the system. An example of a number determined by the architecture is the amount of memory required by a single user. An example of a number determined by business requirements is the estimated maximum number of simultaneous users. By multiplying these two numbers, a maximum memory requirement can be calculated. Usually, system memory consumption and the number of simultaneous users are monitored during operation, and if either value exceeds a threshold, an alarm is sent to operators. The authors propose a method to monitor memory consumption per user from memory consumption data and the number of users, and perform statistical significance testing in real time by applying a stream database. The window size used in a CQL statement for the test affects the precision of the test and memory consumption of the stream database. Through experimentation, the authors propose an optimal window size.