Web Control report generation error.

[Sarvesh Singh 1]

I have created rule in Web Control to log all allowed websites visited by the user and also created template to generate report for the same.

 

When the visited website data was less (200 URL's) then I was able to fetch the details easily.

 

Now on the user's PC, visited websites are around 65k and when I fetch the data I get message ERROR DISPLAYING REPORT in ERAC. 

 

What I understood in this scenario is that the agent fetch the data when we give command from the ERAC. 

 

If the client ERA agent is not communicating to the server agent or the client workstation is not available then it is hard to get the report.

Server agent should fetch all the logs time to time and store it to the server database so we can easily fetch the logs when needed..

 

 

 

 

Report Templates export 2016-01-18 10-32-30.txt

[MartinK 378]

Does it take some time (60 or more seconds) before it fails? If this is the case, it is most probably performance problem with too many data. Please try to add some filter to your report template that will decrease number of processed data -> not sure if available in your version, but check other pre-defined report templates for "Range Report" in theirs filter section for report data paging.

 

In case it fails immediately after execution, please provide full error message from SERVER trace.log.

[Sarvesh Singh 1]

After clicking on generate now button it takes some time and then gives the error. If it is a performance issue then it should be resolved by developer end.

 

I have tried creating filter by workstation and particular Website it works because the data was minimal (450 times visit).

 

I am unable to get complete Web Control log even for particular client, as logged URL's are around 65k.

 

So what I can do in this scenario?

[MartinK 378]

Please provide some details about your configuration, especially database type, database version,  memory and CPU.

 

Also it would help us if you could:

• enable "debug" verbosity of trace log in SERVER configuration
• try to generate problematic report and wait until it fails
• revert trace log verbosity in SERVER configuration

and search SERVER trace.log for something like this:

2016-01-18 08:53:04 Information: CReportsModule [Thread 7f79f7fff700]: 2 MessageProcessorThread starting report generation 327
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports: ID: 137 name: Fetchwebcontrolagregated_eventReportMapper
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports: CALL usp_load_server_info();
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports: CALL usp_create_table_computers_ids();
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports SQL: SELECT TL00120.SourceUuid,TL00120.Severity,TL00120.Occurred,TL00120.Account,TL00120.Url,TL00120.MatchingUrl,TL00120.CategoryId,TL00120.ActionId,TL00120.Count,TL00120.CSN,TL00120.sequence_no FROM tbl_computers_ids AS CID JOIN tbl_computers AS C ON C.computer_id=CID.id JOIN tbl_log_webcontrolagregated_event AS TL00120 ON TL00120.SourceUuid_id=C.computer_id WHERE (C.removed = 0) ORDER BY TL00120.Occurred DESC
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports time: total: 3ms (cpu)     total: 9ms     from mem: 0ms     to db: 0ms     from db: 0ms     guard: 0ms     sql exec: 0ms
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports rows: read: 0     to db: 3     from db: 0
2016-01-18 08:53:04 Information: CDatabaseModule [Thread 7f79f7fff700]: Reports: protobuf size: 0.00 MB
2016-01-18 08:53:04 Information: CReportsModule [Thread 7f79f7fff700]: 2 MessageProcessorThread ended report generation 327
2016-01-18 08:53:04 Information: CReportsModule [Thread 7f79f7fff700]: 2 CReportGenerator::MessageProcessorThread(): report localization takes 0 ms
2016-01-18 08:53:04 Information: CReportsModule [Thread 7f79f7fff700]: 2 CReportGenerator::MessageProcessorThread(): report size (before,after) localization: (0.00,0.00) MB
2016-01-18 08:53:04 Information: CReportsModule [Thread 7f79f7fff700]: 2 CReportGenerator::MessageProcessorThread(): report sorting takes 0 ms
2016-01-18 08:53:04 Information: CReportsModule [Thread 7f79f7fff700]: GenerateReport copying ended 327 time: 0 ms

It may help us to locate source of your performance problem.

[Sarvesh Singh 1]

Database type: We have installed the same SQL comes in all-in-one setup.

Database version:  Microsoft SQL Server 2008 R2 10.52.4000.0

Memory: 1.5 GB (Using ERAS in VM.)

CPU: Intel Xeon CPU 3.3 Ghz.

 

 

• I enabled "debug" verbosity of trace log in SERVER configuration...
• Tried generating WebControl report until it fails. And the result is attached here.

WebControl Server Debug Log..txt

[MartinK 378]

 

Database type: We have installed the same SQL comes in all-in-one setup.

Database version:  Microsoft SQL Server 2008 R2 10.52.4000.0

Memory: 1.5 GB (Using ERAS in VM.)

CPU: Intel Xeon CPU 3.3 Ghz.

 

 

• I enabled "debug" verbosity of trace log in SERVER configuration...
• Tried generating WebControl report until it fails. And the result is attached here.

 

 

Thanks, relevant output is:

2016-01-18 12:05:59 Information: CDatabaseModule [Thread c9c]: Reports time: total: 56646ms     from mem: 0ms     to db: 16ms     from db: 14572ms     guard: 0ms     sql exec: 41948ms
2016-01-18 12:05:59 Information: CDatabaseModule [Thread c9c]: Reports rows: read: 137967     to db: 13     from db: 137967

where we can see that it took more then 56 seconds to fetch data (137967 rows) from database.

 

Could you please try to increase RAM? We are recommending at least 4GB for this type of all-in-one installation scenarios - especially database server should benefit from it.

[jimwillsher 65]

Just for comparison, we have a VM for ERA (4GB) and a separate VM for SQL (16GB) and performance is fine. So as Martin suggests, 1.5GB is woefully low.

 

 

Jim