Processing times

Last update to this page: 21 July 2008

This is a summary of the current processing times for newly submitted data, updated at least weekly. For an automated daily update, see this page, but note that lists can be updated on that page for a wide variety of reasons, such as new title or name references.

General rules

In general, most lists (areas of the database) are updated at least weekly; many are processed daily throughout the working week. Most lists usually have less than a week's data waiting to be processed. Exceptions are detailed below.

Please note that all dates below refer to when data entered our system. This can be the day after you send it (depending on your time zone). In addition, data for new titles does not enter our system until the title itself is approved, and data for names new to our system is similarly delayed. See the relevant sections below for details.

Data is not necessarily processed in the order it is received. Sometimes data for more prominent titles is given priority; sometimes all pending data for a given title is processed at once; sometimes data that is easier to process is handled more quickly. Therefore, it's possible for two items sent to the same list at the same time to appear at different times.

Once data has been processed, it usually needs to go through a daily database build. Therefore, it can take another 24 to 48 hours before data in our master files goes live on the site. The information below reflects the state of our master files.

Highlights for the last week (15-21 July)

Due to an unexpected staffing issue, some lists may be processed on a reduced basis until early August. During this time, data that is very easy to process or that pertains to the most popular names/titles will be processed, but other data may have to wait until things are back to normal. The affected lists are casting directors, cinematographers, editors, miscellaneous crew, thanks, technical data, and the animation, art, camera, casting, costume, editorial, sound, and transportation departments. Details of which lists are in partial backlog will appear in the backlog section. No lists are currently affected.

The purchasing links are up to date, except for a daily batch from 25 June that was exceptionally large. That batch is only partially processed.

All lists that have reported in so far are up to date.

It has become clear recently that new keywords and title corrections are not tracked as closely as some thought. Until these dates are tracked accurately, these areas will be noted as untracked in the list of areas that are up to date.

Repeating the previous non-backlog news, messages about new titles are now sent as help desk messages. We are aware that many of our emails were not getting through due to spam filters; these are now guaranteed to get through (though the automatic emails notifying you of their availability may not). You can always look at your help desk messages under your account, at this location. We believe this now applies to all such messages; certain categories were converted a week or two after other messages.

Also, there has been a policy change to the writers list: A job title (attribute) is now required. Where there was none present in the existing list, the job title (writer) has been inserted; this should generally be viewed as a placeholder until someone can supply the correct on-screen job title.

New/alternate titles

Most new titles are examined within a day or two (except weekends/holidays); the master list is usually updated daily. However, some new titles, depending on their type, may take a few more days to be processed. Please note that all titles received by Friday are cleared every week; those that cannot be examined in time, usually about 10% with the least supporting evidence, receive a form letter (usually sent the following Sunday or Monday). At the moment, the oldest title in the system waiting for processing is from 12 July.

Be aware that once a title is approved, it can take up to two days for that title to appear on the live site. In addition, data will appear gradually for newly approved titles over the following week or two.

Closely related to this, processing of alternate titles is complete through 14 July.

New/alternate names

When a name is new to the database, the data associated with it has to wait for the name to be approved, to ensure it is truly a new name and not just a typo, or a mistaken variant of an existing name. The same delay applies to the first credit in a list (section) for an existing name. New names, other than those associated with new titles, have been processed through 18 July.

Corrections are complete through 18 July (note that unformatted corrections are discarded unless they cannot be handled through the normal interface).

Please note that once a new name has been processed, only then can the actual processing of the credits associated with the name begin. Therefore, you should expect any credits associated with new names to take at least another week once the name has been approved. Also, new names sent in conjunction with a new title do not enter the pipeline until the title is approved; for those names, the effective "received" date is the date the title went live.

Some new names are approved much more quickly, due to their lack of resemblance to other names in the database and other factors that allow them to be processed more easily. It's entirely possible for two new names submitted at the exact same time to go live at different times.

Alternate names created automatically, from (as ...) attributes, are updated when the associated lists are updated, or roughly daily. Alternate names sent in separately (the need for this should be rare) have been processed through 17 July.

Other lists

The following lists have more than a week's data waiting to be processed.

• Purchasing links: Up to date, except for data from 25 June.
• Awards: New events were processed through 24 July 2006 and are, for now, not being accepted. All other awards data is up to date.
• Laserdiscs: Have not been updated since 2000; no current plans to do so.
• Title recommendations: Not being updated until the automated recommendation system is rewritten.

The lists of which titles have complete casts/crews are routinely delayed an extra week or two, so that data sent to complete a list can be processed. These lists have currently been processed through 29 June.

The following non-list areas are not yet tracked accurately: Title corrections, new keywords (as opposed to adding an existing keyword to a new title).

All other lists are on their normal schedule, with a week or less of data waiting. Among the lists that are up to date (processed-through dates of 11 July to today):

• Non-cast filmography lists: special effects department, visual effects department, directors, set decorators, makeup/hair, costume designers, animation department, thanks, composers, casting department, casting directors, costume department, editorial department, transportation department, art directors, production designers, cinematographers, editors, costume department, editorial department, music department, miscellaneous (other) crew, producers, production managers, writers, sound department, camera department, art department, assistant directors, camera department, sound department.
• Cast-related filmography lists: actors, actreesses, stunts, gender.
• Non-filmography title data: certificates, running times, sound mix, MPAA rating reasons, release dates, in-production status, plot summaries, tag lines, business, crazy credits, alternate versions, countries, languages, genres, keywords, locations, color info, soundtracks, movie connections, trivia, goofs, quotes, taglines, title URLs, literature, movie connections, plot outlines.
• Biographies: articles, books, biographical films, covers, birth/death dates, height, interviews, nicknames, other work, portrayed in, pictorial, personal quotes, birth name, salary, spouse, trademarks, bio trivia, where are they now, mini-bios.
• Other name data: direct contact information, name URLs, image comments/corrections.
• Company data: company corrections, special effects companies, contact information, distributors, production companies, miscellaneous companies, new company names.
• Technical data: camera, lab, film length, negative format, process, print format, image ratio.
• Awards, except for new events.

Impending delays

This section will highlight those lists where planned absences or known problems may cause noticeable delays in normal processing.

The following lists were minimally processed for the last couple of weeks: locations, business, non-TV cast. While they never fell into backlog, expect a few weeks of slight delays as we catch up.

The plot summary list is currently in transition between list managers. There may be a slight delay; the list is currently being processed in segments by alphabetical order of the title, and some segments may be more up to date than others. Plot outlines are not affected.

Several other lists are currently being realigned among different team members. There may be some delays as we adjust workloads. See the weekly summary for details.

Open bugs that can cause data loss

This section describes those bugs in our additions system that have been identified and can cause data to be lost. We apologize for these problems, and are working to fix them, but the fixes may require fairly extensive changes.

See also this post for information on other known bugs that do not cause data loss, but may make contributing more difficult.

Duplicate title except for accent

Problem: If a new title is the same as one in the database except for accent marks, all data submitted with that new title gets incorrectly sent to the existing title and is ultimately lost.

Workaround: None available at this time.

Prognosis: We've just identified this problem and are investigating. Should be a relatively simple fix once we know why it's happening. Damage is likely limited to the time since we changed the rules about Roman numerals (about August 2006). (internal bug number 2207). We've just uploaded a fix that should handle many (most?) of the affected cases, though it appears there may still be a related problem with titles that have articles at the beginning (especially "A").

New names/titles

Problem: When a new title goes live on the site, there's a window of a few hours to a day where the additions backend doesn't recognize the title. Data sent for that title is silently rejected. This also holds true for new names created via the resume service.

Workaround: Wait at least a day after a name or title goes live before sending any data for it. Re-send any affected data.

Prognosis: A proper fix would require major changes to the way the additions system works, but it might be possible to fairly easily change the sequencing/timing of some processes so that the window doesn't exist - in theory, the additions system could be updated before the web site. (That could help the title problem, but the name problem will require some rethinking.)

Multiple connected version-of links lost

Problem: If two or more related version-of movie connections are processed in the same batch, they get mangled into an incorrect form. This means that some or all of them will get rejected.

Workaround: Submit one connection from the group; wait for it to appear online before submitting the next.

Prognosis: There is apparently a design flaw in this aspect of the version-of processing. Analysis/debate is continuing on the best way to fix it. (internal bug number 1116)

Resolved bugs that have caused data loss in the past

The bugs in this section have been fixed, so are not causing problems for new data; however, they may have caused some data loss in the past.

Filmography data lost with new titles

Problem: All name-related (filmography) data submitted with a new title did not get sent into our additions system, beginning 23 November. This was an unexpected side effect of some other site changes.

Status: Fixed (as far as we know) mid-day on 28 November. The missing data was recovered on 1 and 2 December. There were 1149 titles affected; their submission date will be treated as the date their data was recovered. Net effect, then: Some titles were/will be delayed a few days.

Quote migration with embedded stage directions

Problem: When a migrated quote includes a stage direction (phrase in square brackets) at the end of a line of dialogue, or embedded with the line of dialogue, the migration does not happen. The delete and add do not match up due to some internal reformatting of the stage direction, and the entire change is discarded. The quote remains at the series level. (Note that you may not be able to determine from your update history whether a stage direction is at the end of a line, since before this was fixed, the update and display system always split these off.)

Status: Fixed for some cases (stage directions at the end of a line) as of late October 2006. Remaining known cases (stage directions in the middle of a line) in mid-June 2007. Unfortunately, it doesn't appear to be practical to recover data from before that time.

Name replacement with Roman numerals

Problem: When a name is replaced in a filmography (either from a title or by checking the "show name" box from a name page), if the replacement name is new to the database and differs from existing names only by having a new Roman numeral, the new name will not be created.

Status: Fixed. Data from before September 2005 will have to be resubmitted. (Definitely fixed this time, and there's an alert in place in case it somehow breaks again.)

Cover image deletes

Problem: If a title is associated with the wrong cover image, an attempt to delete that cover image may work for a while, but the wrong image can come back later.

Workaround: If you can find a replacement image, rather than a delete, it will stay updated.

Status: Fixed as of 30 May 2006. If you have a delete that came back, please resubmit it, though if there's a proper replacement, that's still a better approach.

Simultaneous attribute and order change

Problem: If a sequence number (cast or writer lists) and some other attribute are changed on the same entry at the same time, the sequence number can get lost.

Status: Believed to be fixed for most cases, though we believe there are some obscure cases we haven't been able to reproduce yet. Data before about July 2005 that fits this description will need to be resubmitted.

NAMECORRECTAS

Problem: When a name correction is made with the box ticked to retain the credited name in the form of an "(as ...)" attribute, the "(as ...)" was getting lost, beginning some time around January 2006. (Because of the backlog in name corrections at that time, it would have affected corrections sent up to about 2 months earlier.)

Status: We believe this has been fixed as of about 1 May 2006 for corrections going forward. However, it appears to be impractical to recover the lost data.

Discarded data

Problem: Data sent during 2005 in the following areas that had formatting or other processing problems was discarded silently: Awards, alternate titles. Also, some DVD data was discarded rather than being incorporated into Amazon.com's catalog. Unprocessed data in those same lists from the first part of 2006 was also discarded, with email notification to its contributors.

Status: We believe that changes in tools, processes, and staffing will avoid any repeat of this.

Migration to another list

Problem: Data sent to the wrong list is supposed to get automatically re-routed to the proper list. However, from about February 2006 to about August 2006, the data was getting put off to one side, where it may or may not have been processed. This may also have affected credits that moved from one list to another due to attribute changes (for example, moving out of misc due to a change in job title).

Status: We believe we have made process changes to prevent this from being a problem in the future. Some missing data was recovered when the bug was discovered, but unfortunately, most of the affected data will probably need to be resubmitted.

Trivia qv corrections lost

Problem: In some cases, correcting an incorrect qv'ed name in a trivia item gets lost. The specific example we have is correcting 'Bruce Willis (I)' (qv) to 'Bruce Willis' (qv); it's not yet know whether this affects other lists, or corrections other than removing roman numerals, or corrections other than changing a name from an incorrect one to a correct one. Arguably, such invalid qv-links shouldn't exist at all, but that's a larger bug (and not one that's relevant to this list).

Status: Believed to be fixed in April 2007. Affects data sent from about September 2006; we do not expect to be able to recover the lost data. This also affected goofs. (internal bug number 2404)

Language, country reordering

Problem: Changing the order in which languages or countries are listed actually deletes the relevant entries instead of just reordering them.

Status: Believed to be fixed as of about 2 March 2007. Unfortunately, any data from before that date is lost and will have to be resubmitted.

Free text (miscellaneous) comments/corrections lost

Problem: Some miscellaneous comments/corrections were lost during the processing cycle for at least the early months of 2007.

Status: Fixed around the beginning of May. Unfortunately, any miscellaneous comments/corrections from before that time that have not been processed will need to be resubmitted.

Biography typographical updates

Problem: If an item in the biography list (e.g., date/place of birth/death, other works, publicity, mini-bio) is changed, and the only difference between the old and new is upper/lower case, accented characters, or qv-linkage, the change is lost.

Status: This fix required replacing the tool we used to manage biographies. The tool was replaced in August 2007; however, the problem was not corrected until mid-September. It is now believed to be corrected.

Biography deletes discarded

Problem: A valid delete (e.g., of a duplicated trivia item) gets lost. (Please note that many kinds of deletes people want to make are not valid.) This is particularly likely to happen if there is any kind of odd punctuation (e.g., multiple quote marks).

Status: See above. The tool replacement has taken place, and the problem is now believed to be solved.