Spybot-S&D 1.6, Release Candidate 1

@PepiMK:

BTW, if interested in why those Copy and Move To dialogs pop-up like that, here is a response from Microsoft:

http://blogs.msdn.com/oldnewthing/ar.../02/66160.aspx

I also found the following page today:

http://windowsxp.mvps.org/movetocontext.htm

which has an alleged work-around posted at the bottom. Supposedly just using HKEY_CLASSES_ROOT\*\shellex\ContextMenuHandlers only lets it work without causing those unwanted dialogs. I haven't tried it yet though.

Also BTW, I apologize for asking about enabling a detailed log yesterday. That was pretty stupid of me. Had I thought about it properly I would have known better. I'll use yesterday having been a long day and my having been spaced out on a large volume of pain killers as an excuse... :p:

Quote:

---

Originally Posted by PepiMK

Thank you for the details :)

Earlier beta versions didn't support Firefox 3 yet. But SQLite is a database format, contrary to the old HTML format, so it should be faster instead of slower than Firefox 2 support.

Yes, 14 minutes are unacceptable. I'll have to check where exactly the longest delay takes place; while SQLite is a database format designed for quicker data access, it's not like a full server-sided database, so it might help to do some of the heavier operations not on database engine side (I already optimized a bit there, but there might be more).

Thanks again for very useful information, I feel better now ;)

---

Question: When you wrote that earlier beta versions did not support Firefox 3, does that mean that the Spybot status bar indication of Spybot's checking the Firefox bookmarks was false information?

I ask as, when running Betas 1 & 2, I would always see, at the end of the process, information on the status bar that the bookmarks in the Firefox profile were being checked--and the name of the profile that Spybot indicated was, in fact, the name of my FF 3 profile. (I have two profiles, but the bookmarks file in the second one is so very small that Spybot might have stated that is was checking it, and that check could have been so quick that I did not notice.

I did not notice, during my testing this morning, if Spybot ever checked the bookmarks file in the second profile. (By the time the checking process finally finished, I was having a hard time even seeing the monitor screen...).

[EDIT] After posting this message, it occurred to me that I might still have a bookmarks.html file in my Firefox 3 profile, a "left-over" from the FF beta-test process. I just looked at my profile, and, yes, there is such a file, dated 6Feb08, which is likely about the time that I first installed Minefield (the FF3 beta-test code-named version.) Was that what Spybot betas were checking? If so, that would certainly explain why betas 1 & 2 worked so much faster than RC1.

I've been weeding out no-longer needed files from FF2 in my FF3 profile, and seemed to have overlooked the bookmarks.html one.

Beta 1 and beta 2 were scanning bookmarks in the form of bookmarks.html. RC 1 looks at the profile folder, and if cookies.sqlite exists, it scans cookies.sqlite for cookies and places.sqlite for bookmarks, otherwise it defaults to cookies.txt and bookmarks.html for scanning.

It won't scan both since that may be confusing to the user ("help, where does that bookmark come from").

My places.sqlite file is about 5 MB large and takes 4 minutes to scan (I just enabled and improved a mini benchmarking function) - less than 14, but still too much for scanning 7660 items in the cache.

Which reminds me: I do not have 7660 bookmarks I would guess, so it might be I'm scanning too much (history plus bookmarks maybe - the new database structure in there includes a dozen tables... hmmm... I should add a SQLite browsing tab to FileAlyzer to make it easier to search that file, I haven't found a single SQLite browser app yet that fully satisfies my needs ;) ).

Update: a combined query on moz_places and moz_bookmarks is most probably better than the current one on moz_places, which includes the history and might thus hit a huge multiple of the pure bookmark list. Having results from the history might be nice, but not at this price. Looking at the proper query now :)

Update 2: wrote a small SQLiteAlyzer to access the files faster than through command line ;)
Reading smaller amount of "places" now (only bookmarked ones), tests successful, Spybot-S&D now takes from previously 6:30 to around 1:00 on Firefox bookmarks now. Still not perfect, but getting better ^^

And I have the similar problem like user Always Confused but it all works ok when ff3 is started while scanning( http://forums.spybot.info/showpost.p...6&postcount=11 )

Filesizes are:
places.sqlite - 3.48MB
bookmarks.html - 258KB
bookmarks-2008-07-01.json in bookmarks backup folder is 47KB
cookies.sqlite - 38KB

As if I were not "Always Confused", now I am always confused....

1. I set the FF config option to force FF to export my bookmarks as an html file; that resulted in bookmarks.html, 914 KB.

2. I then ran AM-Deadlink against bookmarks.html, which indicated that I have 951 bookmarks.

3. My places.sqlite file is shown as 15,392 KB.

4. I next deleted places.sqlite, bookmarkbackups and localstore.rdf. Doing that, plus having created a new bookmarks.html file, forces FF 3 to create a new places.sqlite file when FF is restarted.

5. Restarted FF; the new places.sqlite file is 15,392 KB, but there is only one .json file in the new bookmarkbackups folder of course.

6. I then read Ivan1981's comment about running Spybot with FF3 also running, so I started Spybot, told it to check. This time, when Spybot got to Firefox bookmarks, the Spybot process ended almost instanteously.

[In the line of being Always Confused, as I was writing this brilliant missive, I realized that I might not have deleted bookmarks.html. I'm going to do that, restart FF, rerun Spybot, report the result. (This saves me from copying this, closing FF, restarting, etc.; easier to just confuse myself further...).

The forum software would not allow me to post the wonderful verbiage I had added to my previous missive, so herewith enclosed please find whatever it is that is enclosed:

A. I deleted the bookmarks.html file, restarted FF, and reran S&D. When the S&D process neared the end, I watched the S&D status bar carefully, and determined that the reason for the nearly instantaneous completion was due to S&D completely skipping the in-use FF3 profile. S&D did scan the places.sqlite file in my other profile; that places file is only 144 KB, which would account for S&D scanning it so quickly.

B. I then closed FF and reran S&D; when it got to the places.sqlite file for my normally-used profile, S&D again became so slow that I terminated the process.

My conclusion is that S&D has a problem scanning .sqlite files, that problem being directly (and understandably) related to the size of such files.

@Always Confused(but now not :) ): Maybe you are right.
But if I use FF3 when scanning everything is ok like I said, but if FF3 is not started scanning firefox-default-bookmarks is slow and sometimes(not every single time) Spybot found two or one bookmark(s) something like this:

Code:

---

CN.wAQdN188: Bookmark (Firefox: default) (Bookmark, nothing done)
 
CN.wAQdN188: Bookmark (Firefox: default) (Bookmark, nothing done)

---

Spybot(1.5.2 and 1.6 beta 1 and 2) with same definition file never reported any of this like RC1 does now.

The SQLite database Firefox uses isn't intended as a multi-user database; it's up to SQLite3.dll to allow or not allow access to the database. If it's in use, it's most probably not available. We could make a copy, modify that copy, replace back etc., but that could cause inconsistencies depending on what exactly Firefox keeps in memory.

As I wrote above, places.sqlite contains a lot of information, it has a huge list of "places", which in the beginning of my Firefox 3 usage wasn't that large, but quickly grows to be much larger through history, rss feeds and many other places stored in there (probably even a cache list).
If there's interest, I could add more details about the various Firefox 3 databases to the Spybot wiki I guess, I've already written down a lot of structure anyway.

Also, I've limited the query from this full places list to one just scanning bookmarks now, see above how that got faster.

I can upload an updated SpybotSD.exe in the next hour, I think it'll improve things similarly to my experience ;)

Small update: sqlitalyz-0.1.zip

Just a very basic SQLite database browser that'll show you the content of Firefox' databases to better understand their content :)

SpybotSD.exe test version including the reduced scan set coming next ;)

Sorry if it took nearly the full hour, but doing a few full scans for testing takes some time, and the public build environment always compiles at least 30 files, which is slooow ;)

SpybotSD.exe-ff3-improvements.zip

Just the main executable with Firefox 3 bookmarks support that does work faster by using only the data from moz_bookmarks with titles and urls from moz_places, similar to what is seen in the sqlite demo above (Templates menu).