Collector Rebuild

Discuss issues related to the MLNetflix plugin.
User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Collector Rebuild

Postby ronsatter » Sat Jan 29, 2011 3:10 pm

Hi Dave,

I have yet to get through a week of smooth movie updates using the collector running in the background. It either seems to get hung on some type of mismatch error and stops completely or just keeps running for days without completion. Not good!

I could deal with the background task running until it finishes, but while the collector is running, the plugin shows no movies at all. You can't use the plugin until the update is completely done. Really not good!

Is there a way to leave the existing database file accessible while the collector is grinding away (12,000 movies takes several hours)?

At some point, after letting the collector run endlessly, I shutdown ML and run the collector.exe with logging, so I can see the process. On my system, with 16 GIG of bandwidth internet connection, it takes 3 to 5 secs per movie item to update the database file. That seems very slow to me. I assume the collector is supposed to just go online, compare any new listings against the downloaded database, and add the new additions. How many movies could that be, 100 or less per week? Shouldn't the collector handle the job in minutes instead of hours?

Maybe I'm not understanding this process correctly, and I'm surprised others are not having similar issues. Any ideas for improving this? I really think streaming is the ultimate answer for home viewing. Is it possible to improve this plugin? Or should I be looking for other streaming methods?

Ron
If it ain't broke ... don't fix it!

User avatar
CinemarDave
Site Admin
Site Admin
Posts: 10535
Joined: Fri Feb 07, 2003 8:56 am
Location: Planet Earth
Contact:

Re: Collector Rebuild

Postby CinemarDave » Sat Jan 29, 2011 3:59 pm

First make sure you are running the latest version of the collector. 4.0.27 as of today. There was an issue in the old version that would erase the database on a very specific error.

When the collector runs it first marks all the movies in the database as available for deletion. As it parses the online catalog it resets the delete flag for every movie it discovers. When it has parsed the entire catalog it will then delete the entries whose delete flag has not been reset. So at no point in time should you ever see an empty database after the initial load. An empty database means a bug that needs fixing.

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Sat Jan 29, 2011 4:53 pm

OK. I'm currently running the collector. Should I wait until the database is updated before installing the latest version of the plugin, or will I need to run the collector again as soon as the plugin updates?

Ron
If it ain't broke ... don't fix it!

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Sat Jan 29, 2011 8:25 pm

CinemarDave wrote:First make sure you are running the latest version of the collector. 4.0.27 as of today.

I don't see it in the installer. Boo Boo?
NetFlix Installer Version.jpg

Also, as I watch the collector log, I have had two more stalls during the current download. Both times I was able to resume by simply selecting "Catalog" from the drop-down menu in the collector app. What surprised me was that the collector did not immediately start up again where it found the error, but returned to the very first page of movies in the first genre and proceeded to re-load every page again (not scan every movie again) until it reached the point of the error. Depending on where the error was in the database index time-line, that rescan can take several minutes to an hour just to get back to the point of the error.

Now it's finally scanning individual movies again ... until it finds another error and stops. BTW, I would have wasted another few hours of collector chugging, if I had not made a point to babysit this process. It's a good thing I'm semi-retired. #-o

Ron
You do not have the required permissions to view the files attached to this post.
If it ain't broke ... don't fix it!

User avatar
CinemarDave
Site Admin
Site Admin
Posts: 10535
Joined: Fri Feb 07, 2003 8:56 am
Location: Planet Earth
Contact:

Re: Collector Rebuild

Postby CinemarDave » Sat Jan 29, 2011 9:04 pm

why don't you just simply send me the log it is generating? The latest collector is always bundled with the plugin.

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Sun Jan 30, 2011 6:57 pm

PM'd you with log.

Ron
If it ain't broke ... don't fix it!

User avatar
DavidL
Simply Incredible
Simply Incredible
Posts: 11071
Joined: Sat Feb 08, 2003 9:39 am
Location: Metamora, Michigan
Contact:

Re: Collector Rebuild

Postby DavidL » Sun Jan 30, 2011 7:06 pm

Your screen shot shows you haven't updated to the latest Agent.
Download that and install it.

All MainLobby components need to be latest and greatest across the board.
That is the easiest way of resolving hard to fix problems and the by far most common issue.

I haven't read your issue but running out of date plugins will surely bite at some point.

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Sun Jan 30, 2011 7:49 pm

Thanks for that catch, David.

I don't think the agent has anything to do with the collector app. I may be wrong, but I believe it is mainly responsible for stream playback.

I do agree having all the plugin pieces updated to the correct versions is a good idea. My oversight :roll:

Ron
If it ain't broke ... don't fix it!

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Mon Feb 07, 2011 11:30 pm

Dave,

The collector has been stalling out for the last 3 days. Log entries show a problem finding the last page of every genre type. Look at the log for 2/5/11. For some reason the collector decided to delete (retire) all the movies from the database again. So again, I have no working database and a never-ending loop of collector scanning. Is there a way to disable the next scheduled scan, if we notice problems in the logs?

Here are the logs for those days.
NetFlix Logs.zip


PS - During this time, I have not added or deleted any movies from my queue, or initiated a Netflix stream. I was wondering if over-night virus scanning when the collector is running would cause issues? I think it would be good if we could define collector run times to avoid colliding with some other overnight maintenance services.

Ron
You do not have the required permissions to view the files attached to this post.
If it ain't broke ... don't fix it!

User avatar
CinemarDave
Site Admin
Site Admin
Posts: 10535
Joined: Fri Feb 07, 2003 8:56 am
Location: Planet Earth
Contact:

Re: Collector Rebuild

Postby CinemarDave » Tue Feb 08, 2011 11:13 am

This is an issue where every time the collector went to Netflix to browse the catalog it got vectored to the login page. As if the cookie did not take hold. Saw this yesterday on another users PC. I seem to have some magical Internet connection where I never see these types of problems. What I am going to do. (At least for the US customers) is cache the Netflix catalog on our server and have all the collectors go out to our server for the refresh. That will put more traffic on our server but will end up being much more reliable for our customers. Then when Netflix changes the site I'll only have one place to worry about.

I sent out an update yesterday to address the cookie issue.

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Tue Feb 08, 2011 2:38 pm

CinemarDave wrote:I seem to have some magical Internet connection where I never see these types of problems.


Who is your service provider ... Homeland Security? :D

I'm all for a different approach to collection from NetFlix. If you cache to the Cinemar server, will the update time be improved? If the database file update is done on Cinemar's end, can you write a script which compares our current database file with the updated version on Cinemar's server, and overwrites our current file with the changes?

I think a plugin option for when the file gets updated would be very useful. Even though the collector app is running as a background process, I still notice subtle performance slowdowns in ML when it is running. It spikes the CPU usage just enough sometimes to cause scene changes with animations to freeze in mid-transition. Being able to set an update time in the early morning hours when ML is not being used, would solve this issue.

Ron
If it ain't broke ... don't fix it!

User avatar
ronsatter
Is there life beyond Cinemar?
Is there life beyond Cinemar?
Posts: 1304
Joined: Mon Dec 04, 2006 1:40 am
Location: San Leandro, CA
Contact:

Re: Collector Rebuild

Postby ronsatter » Wed Mar 23, 2011 8:11 pm

Hi Dave,

Just thought I would report back on the collector app. The latest version is running very smoothly and I always have the database available. The updates happen without a hitch, logs show normal entries. The new movie preview feature is my favorite addition. I just wish the classics had previews. Still very cool 8)

Ron
If it ain't broke ... don't fix it!

User avatar
CinemarDave
Site Admin
Site Admin
Posts: 10535
Joined: Fri Feb 07, 2003 8:56 am
Location: Planet Earth
Contact:

Re: Collector Rebuild

Postby CinemarDave » Wed Mar 23, 2011 8:17 pm

Thanks for the feedback. We like to hear good stories...


Return to “MLNetflix”

Who is online

Users browsing this forum: No registered users and 1 guest