TeaTimer CPU load after startup thoughts

Since many people ask about it, we gave the issue some thoughts as to what could be done about this:

1. We could display it's acitivity; if it is showing which process it is scanning, that might make it more understandable.
2. We could remember which files have already been scanned (hash of files) and thus "skip" known ones; would have to clear this cache after each update though since new updates might lead to different results.

Thoughts please

PepiMK (Patrick):

To start with, perhaps just an explanation of what TeaTimer is doing when it first starts would help.

For example:

Right now I have 53 processes running and if my Avira Antivir's "Active Processes" scan count is correct those 53 processes are using 2144 files (exe, dll, etc.).

If I run my cursor over the TeaTimer system tray icon I see:

• 88292 processes blacklisted.
• 283740 known rating available.

If I were to restart TeaTimer would it compare the 2144 files against the list if 88292 blacklisted processes, the 283740 rated processes, both or are there other things TeaTimer is doing?

You're right, a quick overview might be helpful
Those 283740 ratings are those entries from the RunAlyzer database that are of use here (e.g. LSPs are not monitored by TeaTimer, so they're not in the list). These are loaded from TTLassh.sbs.

For each new, changed or deleted registry item, the lassh (hash sum of various of its properties) is calculated, and looked up in this list.

While this includes the MD5 of an associated file, that is very quick, and lookup of the created lassh in a hash table returns a result immediately as well. So this part doesn't take user-noticable time.

The 88292 processes are gathered from the regular detection databases, read from all .sbi files found (if you create your own, that would be more or less those File and DownloadFile rules as described on the OpenSBI wiki, though file paths etc. would be disregarded).

TeaTimer tests every new process it finds against this database of 88292 items, and when you start TeaTimer, it will regard every running process as such a new process. In your case, that would mean those 53 processes, not those 2144 which probably includes both loaded modules and file handles.

As for the speed, this is very similar to what thw new SDFiles.exe is doing, which is slow I have to admit. The changes appearing in 1.6 were mostly to reduce memory usage. By pre-organizing the data in memory, scans per file would be faster, but memory usage would increase. Keeping the in-memory database means slower per-file scans (I'm also thinking on various ideas how this could be improved).

hello,

yes, please do so - the more visible information the user gets, the more are we able to find out, report & repair (system) errors ... (if any)


---
in my case: (that's why I came here today )
my system resets / windows crashes without warning whenever teatimer starts - I don't know why yet, but it's everytime I start teatimer manually (it's already taken out of autostart / system start for observation)

once I saw some kind of message: "you have more then 1083 files in temp" or something - then a count down (pretty fast) then my system crashes / resets again ...

I do NOT say that teatime is the result of my system crashes - I had teatimer running before without any probs - but this time (only) it's different ...
it could be some systemfile or virus and teatimer causing some kind of chain-reaction with the anti-virus software or something like this - anyways:

it would be nice to see what files are getting checked by the teatimer so I can find out what file is causing that trouble ...
(spybot 1.6 - winxp sp3)

Millus
you seem to have a different problem
start a new thread
also post what AV and firewall you run
do you or did you have any older AV or anti spyware installed
did your machine come pre installed with a suite or AV?

run CCleaner to clean up those temp files

appreciate your comments on this topic

PepiMK (Patrick):

Thank you for the explanation.

Both proposed enhancements have merit. The first would allow people to see that TeaTimer was actively scanning during startup and the second should reduce the startup time/utilization except immediately after updates.

I can't think of anything else except possibly ordering the blacklisted processes and doing a binary lookup if it is not already being done.

Is anyone surprised that T-timer is actually doing its job?

There may be other ways to slice the baloney but for now
you want resident protection- you get protection

you want to clean up afterward then turn it off
ps
other Antispyware programs with resident protection take even longer

you could also run Win Patrol (or similar) which would monitor and tell you something has already infected your system and maybe able to reverse the action
does not have the "lag"

(I run Win Patrol ALSO, IN ADDITION TO A real time Anti Spyware protector)

you could also run System Safety Monitor, a virtual machine, Process Guard or similar

All trade offs and all approaches open to discussion

md usa spybot fan said:

PepiMK (Patrick):

To start with, perhaps just an explanation of what TeaTimer is doing when it first starts would help.....

Click to expand...

Ya, how about a notification balloon, like when you install avast antiviruis, it shows the balloon and does a box pop up that explains what stuff is, so that you know what the icon is and what it does. I think that would be the best option so that people would understand teatimer better upon first install of spybot. If md spybot fan meant something else, then oops, but I think something like i suggested would be a good way to understand teatimer.

wow

I feel really dumb. I read that thread so fast haha, i misunderstood what milius.net meant. I read that thread and that got me onto showing more teatimer information. Disregard my above reply lol. :funny:

@wyrmrider: Perhaps you should fully familiarize yourself with the problem reports and history before making any assumptions. This is a case of obvious abnormalities, not a need for trade-offs or alternatives and most certainly not some baloney case of TeaTimer doing its normal job. This is a case of TT needing a fix for a real and verifiable problem.

First, on some systems, the TeaTimer process NEVER recovers. It jumps to 99% on start/restart and stays there. Period. Not for a few minutes. Not for a few hours. Permanently - at least until forced closed the hard way.

On other systems, it jumps to 99% usage at boot and stays there for [at least] a few minutes, but then it does recover. But such a long spike shouldn't even be needed for truly normal activity in the first place.

And unfortunately, until it recovers or is killed, an affected system is unusable because of the CPU flood.

Other programs with resident protection DO NOT take even longer. On systems where this problem exists, every other real-time solution works fine - only TT is showing spikes.

Of further significance is that these problems were introduced during the beta period. These problems were not always present. I know because I originally reported a similar (and likely related) TT crash problem here:

http://forums.spybot.info/showthread.php?t=30453#7

and it was only after that got fixed that the CPU spike issues began. In fact, as I said then, there were no similar TT problems up until RC2 (or maybe RC1). The original problem was not present in beta 1 and 2 and the spikes didn't occur until a fix was put in place for the reported TT crash.

This all seems to be of rather critical importance to any potential resolution...

So what changed in TT between beta 2, RC1, and RC2?

And perhaps since this also never occurred before 1.6, what is TT doing now that it didn't do in 1.5.x?

Once we define what changed, the logical test is to then disable whatever changed and see if the problems persists.

At the very least, a way of displaying - or preferably logging - all TT activity would also be useful (as Patrick suggested). Trying to do that with something like Process Monitor just got too out of hand (generating 50+ Megs of data every few seconds).

I am somewhat skeptical that this will come down to "normal" behavior, however, since on nearly identical systems with the same software and configuration different levels of the problem can occur - plus, like I said, the problem didn't occur with the betas.

FYI, when TT does do something it is supposed to do, it does not spike - even on affected systems. It has a brief but normal resource increase, but that's it. I can only make a permanent spike [at will] by fully exiting and then later restarting it (at least on some systems). Further FYI, I have no TEMP files (auto-scrubbers in place) and usually <40 processes running. So anything TT does should be unnoticeable and instant (which truly normal "doing its job" TT behavior is).

Anyway, thanks very much for continuing to look into this PepiMK. If you need me to help test anything or provide any other details, please just let me know. I may not be available much now though.

Sorry if I posted any redundant info here, but evidently while I was away some people decided to start multiple separate threads for the same related issue...

Fully agree with Zero voltage

Zer0 Voltage said:

@wyrmrider: Perhaps you should fully familiarize yourself with the problem reports and history before making any assumptions. This is a case of obvious abnormalities, not a need for trade-offs or alternatives and most certainly not some baloney case of TeaTimer doing its normal job. This is a case of TT needing a fix for a real and verifiable problem..........................................

...............
I am somewhat skeptical that this will come down to "normal" behavior, however, since on nearly identical systems with the same software and configuration different levels of the problem can occur - plus, like I said, the problem didn't occur with the betas.

FYI, when TT does do something it is supposed to do, it does not spike - even on affected systems. It has a brief but normal resource increase, but that's it. I can only make a permanent spike [at will] by fully exiting and then later restarting it (at least on some systems). Further FYI, I have no TEMP files (auto-scrubbers in place) and usually <40 processes running. So anything TT does should be unnoticeable and instant (which truly normal "doing its job" TT behavior is).

Anyway, thanks very much for continuing to look into this PepiMK. If you need me to help test anything or provide any other details, please just let me know. I may not be available much now though.

Sorry if I posted any redundant info here, but evidently while I was away some people decided to start multiple separate threads for the same related issue...

Click to expand...

It is absolutely clear that there is some terrible bug with TT that makes it spike the CPU.As Zero voltage suggests any normal working need not consume that resource.I hope the spybot team quickly put their heads into action and correct it or the reputation of the software will get affected.In fact I was among the first who posted a thread on the CPU spike but a pity no proper response from the forum.I have for the moment uninstalled Spybot ,and watching for the correction.

Please, try to keep two things separated

1. A long time for scanning each process. For this, see the discussion about SDFiles.exe before the 1.6 release. That is a trade-off between reducing the time for loading the database to a third (and using only a quarter of memory speaking of resources) while the time for scanning each file has about doubled, which, for single files, is still an improvement.
That is not a bug, just but a discussable shift of preferences.

2. If TeaTimer never recovers as Zer0 Voltage described, that is a bug.

@129260: a Windows standard notification balloon is indeed what I tried this morning While the advantage here is that it is a system standard, it's downside is that the time it is shown has to be specified before showing it; hiding it when the file scanning has finished is not possible right now, so the timeout the balloon will have has to be guessed.

@Zer0 Voltage: the change at RC 2 was the shifting between database loading and scanning time.
I think I'll make that toggleable by a registry tweak to allow easier testing of both methods.

@md usa spybot fan: a proper binary lookup only works with a clear set of data. If we would check every file by MD5 or some other hash that we would have for each file, that would do. But if you take a look at the wiki, there are dozens of methods; most time-consuming are cached using hash lists, but they still need to be calculated.

@topic: what we do have now:

• balloon hint on each scanned process, shown for average scanning time of last scanned file.
• registry tweak ("ConserveMemory") to toggle between both modes for easier debugging.
• registry tweak ("WriteProcessLog") and command line switch ("/verbose") to enable more detailed process scanning logging.
• improved the low-memory profile structure to speed up that scan, even though it will never reach the speed up the full structure.

what we need to still do:

• possibly port all TeaTimer messages to new balloon method.
• cache scan results between seasons (except for scan after update).
• improve the low-memory-usage data structure some more.

I even tend to get the more memory consuming variant back into place and keep the slow-but-conservative method as a more or less undocumented feature for users of very old systems that would prefer it, will have to discuss that with the team tomorrow.

One file per second doesn't sound too much or much slower than two files per second, but summed up for 50 files...

(btw, this is why we didn't put 1.6 into the updates today, in case anyone wonders )

PepiMK,

The concept of being able to chose by registry tweak or other method between the two modes (less memory use, slower TT scan/More memory use, faster TT scan) at least enables TT to be optimised for the type of PC it is being used on.

I am less enthusiastic about the balloon notification. Sure, those who like to watch utilities work and who try to understand what the software is doing may like it. There is however arguably more users who simply want what ever protection software they install to do its job as unobtrusively as possible. They don't have the time or inclination to worry about how the protection works and they don't want it to interrupt them unless it is for something critical. They expect it to get on with its task and leave them to get on with theirs (what ever they are using the computer for). Of a group of 6 people I provide support to, 5 fall into this catagory, they are computer users, not enthusiasts. Perhaps you would consider also making the balloon a diagnostic option that can be turned on or off.

Well I agree with everyone
Greyfox baloon toggle is a good idea

but if there is the hang that Zero finds- well that requires some additional investigation and a fix
60 seconds for start up with big cpu spikes is not a Bug, it's a feature, and that is being addressed
Nagan writes
"It is absolutely clear that there is some terrible bug with TT that makes it spike the CPU."
Read PepiMKs post which was made after yours-
are you finding any of the hanging issues Zero finds?
Nag
What AV are you running, firewall, anything else?

Ok agreed that it(cpu spike) is not a bug and a intended function as the people who are in close contact with spybot say.
My query is when the real time alert is present it will always prevent a wrong file from acting.What is the use in scanning files at the start when a detailed scan option is already available.
I used ESET trial and presently Avira.They go about their jobs without the need for a high CPU usage.
Whatever be the reason people do get paranoid on seeing a consistently higher usage of CPU ,because such a symptom preceeds a crash.Would not know whether it is an improvement.
The best tradeoff should be a scan if need be with the minimum of resource used.

good points Nagan
It appears that the scan of running processes on first access is what's taking the t-timer-time
As opposed to a system wide scan I do not kow how you can do any useful work till the system is secure
I have confidence in Spybot team getting a grip on this

have you had any issues with sd-helper as others have reported.

I mentioned AV as there have been several posts about alleged interference with AVG 8 or AVG8 anti-spyware additions. I'v seen no such reports for Avast or Antivir

We've moved the other regular hints to standard balloons now as well, which should, as a side effect, make those constant forth and back with resilient software that doesn't accept someone is not accepting their changes less desktop cluttering.
Thanks for the hint that many users don't really want to know about these things as well - I went ahead and added an option to either show them or not to give the user the choice here. The point of it was to show some progress while processes are scanned on startup, so we decided to have it enabled by default.

Caching between seasons works fine now as well, making a rescan only necessary if either the database has been updated, or a file has been changed.

As for choosing between the two methods, we decided that it might be best to automatically switch depending on the memory a machine has available.

We'll be looking at possible improvements to the low-mem method again tomorrow, but after that, we should have a new TeaTimer for testing available here (and I hope the new verbose mode has enough output to even find that permanent stall)

PepiMK said:

We've moved the other regular hints to standard balloons now as well, which should, as a side effect, make those constant forth..................
As for choosing between the two methods, we decided that it might be best to automatically switch depending on the memory a machine has available.

We'll be looking at possible improvements to the low-mem method again tomorrow, but after that, we should have a new TeaTimer for testing available here (and I hope the new verbose mode has enough output to even find that permanent stall)

Click to expand...

Thanks for the reply and giving the thoughts a look into.But whatever you say people would not like to see a spike of around 50% (in dual core) or more in single core systems.The pertinent question (ofcourse from a totally non-technical angle) is when an antivirus does not need such an extended scan why should spybot do?Scanning memory hardly takes time like any Task Manager will prove.I think Spybot is doing a little more (you are the best judge) in the initial scans.Could you not make them low priority ones or change the method of scan?I have been using spybot and am very impressed with it.This 1.6 is proving a little difficult which I am sure would be sorted out soon.:bigthumb:

nagan:

One point that has not been arisen during any discussions concerning the CPU utilization of TeaTimer is the fact it runs with a priority of "Low". Although it is using a lot of CPU time during startup, I personally have not observed any adverse affects in the performance of other processes during that time.

Incidentally, after the 2008-07-16 updates there appears to be 146536 blacklisted processes vs. the 88292 I cited above. TeaTimer is using more CPU time and yet I have still not seen any adverse affects on other processes during TeaTimer startup.

__________

Side note:

In most PCs the processor (CPU) is the single most expensive component and is generally priced by its speed. Theoretically if it is sitting idle you are not getting the bang for your buck.

md usa spybot fan said:

nagan:

One point that has not been arisen during any discussions concerning the CPU utilization of TeaTimer is the fact it runs with a priority of "Low". Although it is using a lot of CPU time during startup, I personally have not observed any adverse affects in the performance of other processes during that time.

Incidentally, after the 2008-07-16 updates there appears to be 146536 blacklisted processes vs. the 88292 I cited above. TeaTimer is using more CPU time and yet I have still not seen any adverse affects on other processes during TeaTimer startup.

__________

Side note:

In most PCs the processor (CPU) is the single most expensive component and is generally priced by its speed. Theoretically if it is sitting idle you are not getting the bang for your buck.

Click to expand...

Agreed.But the issue is to what extent of time it can possess the cpu.I see some program whose spike is limited to few seconds like eset or avira.Well it would only make sure if there were balloons on what exactly spybot is doing during that time.