Well, the main problem is not much the amount of memory used in the working set, even if I think that it is high, but the amount of dedicated memory, which is the sum of the working set (in physical memory) and the paged out memory:Originally Posted by Coronamaker
The working set is effectively quite high during Windows startup (along with a quite high CPU usage which is a clear sign of lot of missing optimisations for performance, and one of the main cause why the Windows boot time is quite long (SpyBotSD's TeatTimer can consume more than half of the total CPU time, when all the other processes and services are starting), but after some time, it decreases to about 660KB no my system (I think it is still too high, even that this represents more than 20% of the remaining unused memory). Then this workign set restarts to grow, progressively over a period of about 24 hours, up to nearly 40MB, even when there's absolutely no interactive application running (with only a reduced set of the Windows services, from which I have disabled or about one fourth of all the services of Vista Integral Edition that I don't need, and configured nearly one third of the other services for Manual startup for running them only on demand).
It's true that Vista uses a lot of memory for the "netsvcs" services group and I don't know how to reduce it significantly (most probably, I should tweak some unknown parameters to reduce their internal buffering which is clearly excessive as well).
Howeer the main problem is the dedicated memory which constantly fills the paging file, up to the point that it can fill up. In order to avoid exhaustion of the paing file, I need to set it 3 times the total of physical memory, only because of SpyBotSD's TeaTimer. You may think that the fact that the working set correctly shrinks after some time is enough, but really the constantly growing dedicated memory in the paging file is the clear sign of memory leaks in TeaTimer, and it can explain why the dedicated memory also grows slowly after some time due to page fragmentation in the working set which is mostprobably full of very small memory fragments that will only be paged out after a long time.
After about 48 hours, the system performances start to be very quite slow, with slow displays of directory contents in the Widnows Explorer for example when scrolling long directories. I've tried to increase the throughput by flushing the icon/thumbnail cache or by disabling it, but this has no significant impact on the performance: the GUI becomes very sluggish, and some actions in Wnidows explorer take more and more time (displaying the content of the windows\system32 directory and scrolling it may then require more than noe minute, selecting an element with right-click can take up to one minute before it displays the contextual menu, sometimes not for the selected element. Deleting a file by selecting it and pressing the Delete key or using the Delete command in the context menu can become quite dangerous as it may unexpectedly operate on the wrong element (not the one that was selected first).
All this degradation of performances becomes unsupportable, just because of TeatTime filling most the remaining space in the pagnig file. Other effects include slow start of applications, incorrect default fonts displayed in web pages (I start loosing Chinese characters, then complex Asian scripts like Devanagri, then Arabic fonts...), or other CSS properties ignored (like background colors or images...).
I'm running Vista with 2GB RAM (however my processor is single core at 2GHz, something that should still be enough for Vista, even with the Ultimate Edition). Disabling Vista Aero or even all themes, or the transparent browsers has a very small impact which is insignificant for this case. All I can do is effectivel to exit TeaTime and restart it.
Given that I have an Antivirus that also includes its own antispyware, I wonder if it is only useful to run TeaTimer (given that I frequently get 2 or 3 alerts for the same modification of system settings: Windows Defender, My antivirus, and TeaTimer: this is excessive, and in fact I suspect that TeaTimer's memory management is not very clean if it cannot access immediately to some system information that is being accessed also by other antimalware tools running concurrently: in the case where TeatTime acnnot get an exclusive lock on the object to control, it seems that it yields and retries after waiting some time, but it forgets to release some of the termporary ressources used, such as internal buffers or structures needed to parse the content modified system item).
Given the fact that the memory usage is growing very slowly, I suspected that this could be handles to registry keys that are left open but leaked, or this could be handles to asynchronous IO event handlers or some other interprocess communciation handles (used between the SpybotSD TeaTimer's background service and the TeaTimer UI process that filters events and displays them when appropriate)
When I see just after boot time that the dedicaetd memory for TeaTimer alone is alerady one third of the total physical memory, TeaTimer really has a problem.
Antoher related problem is the initial time needed just after the first intallation or upgrade to perform the first full scan: on my system it took more than 1 hours and half ! Durnig that time, I could not even go to the preferences panel to check its settings (there's not even any way to dismiss this first scan before changing the settings, and if you attempt to cancel this initial scan run at program start, the program will freeze indefinitely and there's no other way to unfreeze SpyBotSD than killing it from the Tasks Manager: here there's clearly a bug in SpyBotSD: you shuold be able to cancel the initial scan performed before the SBSD UI is displayed without freezing the program indefinitely and without exiting immediately if I click the button to not perform a full system scan).
In terms of usability, SpyBotSD has become more and more memory hungry, and now it takes really too much time just for scanning one family of spywares that contains more than 100,000 signatures (apparently only in memory because there's no disk activity during that long time). This suggests that the signatures in SpyotSD are not compressed efficiently and SpyBotSD is constantly rescanning the same registry key too many times. This also suggests that the old scanning memory is now obsolete: SpyBotSD inspects each known Spsyware individually and is unable to merge the many similar signatures for a fast scan. This constributes to the constant degratation of SpyBot performances. Other antispyware tools are apparenty using a smarter compressed format for their database in order to avoid verifying the same places too many times.
My conclusion is that SpyBotSD is now facing a major problem due to its internal representation of its database: factorizing common signatures to checking only once the same registry entry is now an emergency, because the number of spywares to check is exploding. This cannot continue this way. There was a time when Spybot could perform a full system scan in about 5-10 minutes using its database. Now the database is so big with more than 440,000 malwares to inspect, than it has become unmanageable.
This system where each malware is scanned independantly of the others must find an end. You need a new database format, for reducing the full scanning time (consider inspecting the registry linearily, using a snigle index pointing to the possible list of malwares, then find the complete list of infection points by looking into an external database not permanently in memory (or only present in memory within a cache of reasonnable sizen, according to an MRU strategy). There is certainly now much less than 400 000 total points of infections in the registry, given that most malwares are trying to invade the same few locations.
There are tradeoffs to find to optimize it and this is a problem exactly similar to the problem of performing a join in a SQL database between two tables whose one is much larger than the other: even if this is logically equivalent, the loop that fullscans the greatest number of locations must be the inner loop, not the outer loop. Now it seems that this optimal order of loops is reversed: the outer loop is on the full list of malwares (440,000 and growing) and the inner loop is on the few locations that each malware is possibly trying to infect. Swap the loops so that SpyBot will scan the first the few registry locations that all malwares are trying to infect, then read the associated data from the registry of filesystem to get its value, then perform a lookup for the complete of possible malwares that may infect this location: this loop is probably much longer in terms of unique enumerated values than the previous loop on locations.
For inspecting the ActiveX classes, it seems that you need a separate index of the GUIDs used there: this index will point to the data associated to the small list of malwares that use this GUID, get this list from an external file source (on disk, not necerssarily in RAM) then inspect its contained elements, that are all possible infection locations in the registry or filesystem, filter out the items that have already been scanned (this requires that this list of locations is fully ordered in some way) and check the rest of the possible locations. To make this possible, you need a new format for the database index that must be ordered completely differently, to make the maximum benefit of its data compression.
SpyBotSd was a great program in the past, now it cannot sustain the exploding growth of known malwares. For TeatTimer itself, it does not need to perform a full check of all possible locations with indection data: TeaTimer should just be able to detect the few locations that the malwares need to indect in order to be functional, but it does not need to inspect all locations in the registry or on the filesystem, unless the critical locations needed by the malwares are found to be infected.
So:
(1) consider changing the database and indexing it in a more efficient way. This is exactly the same problem as optimizing the schema and queyres in a SQL relational database.
(2) investigate the memory leakages (notably the leakage of system handles that are left open if a critical section cannot get the necessary lock immediately). I think that most of these leaked handles are handles to registry keys (this is apparently confirmed by the total number of open handles which whose growth is nearly propertional to the growth of total dedicated memory (and most of them get pages out and never reused later). Too many handles also means that the dedicated memory for the System process is also growing in size for storing the system structures allocated in memory for storeing the attributes or data of an open handle (and this is happening as well).
(3) prioritize the scanning loop
(4) correct the program freeze when canceling a scan in progress performed when the program is started (but the main GUI window interface still not displayed): the scan should stop immediately but should not forbid opening the GUI interface to change some settings
(5) consider placing the SpyBotSD configuration panel in another program run separately from the rest of the interface. SpyBotSD must be fully configurable even if no scan has been performed still.
For now, all we can do is effectively exiting TeaTimer and restarting it.
Note that TeaTimer is considered by some serious antimalware tools (including Norton or AVG) to be itself a malware (not much because of what it performs or installs, but because of the uninformed choice that Spybot gives to users in case of a possible infection: delete key/file, heal file, or ignore the issue, a chioce that many users cannot do safely without errors, where the user may inadvertenty choose to ignore the issue, with no way to revert his decision to ignore the issue later: most of these chioces should be automatic ior should have reasonnable defaults)
Reply With Quote
