tag:blogger.com,1999:blog-8799846414755150351.post6885448897730164407..comments2016-12-02T19:34:18.813+01:00Comments on The Jabbithole blog: The anatomy of an importKrisztian Pifkohttp://www.blogger.com/profile/14968087027411562816noreply@blogger.comBlogger2125tag:blogger.com,1999:blog-8799846414755150351.post-86637914699008726392014-07-18T12:11:10.065+02:002014-07-18T12:11:10.065+02:00It would be unwise to run the import directly agai...It would be unwise to run the import directly against the live database. First, processing XMLs this large requires a lot of resources and that would affect the performance of the website in an uncontrolled manner, which would be bad. Second, data coming from the outside world shall not be trusted right away. There might be issues with the XML that need to be caught, like inconsistencies due to a client crash, or an unnoticed change in the game API. Once the XMLs are found to be good they are imported into a test database to see that the import process executes without an error, as there still might be obscure issues with individual XML files. The process logs all the changes which goes through a manual review to make sure that everything's gone as planned. If things are fine then the same import process is executed against an identical copy of the live database which is again reviewed. If this last step is done without a glitch then the data is ready to go live. The live import itself takes about 10-15 minutes, the majority of the downtime is the waiting for the disk cache to be purged (which will not be an issue anymore after next week). So it all comes down to 15 minutes of weekly downtime vs a constantly overloaded server and potentially compromised data integrity.Krisztian Pifkohttps://www.blogger.com/profile/14968087027411562816noreply@blogger.comtag:blogger.com,1999:blog-8799846414755150351.post-672044046468914762014-07-18T04:27:28.059+02:002014-07-18T04:27:28.059+02:00Thanks for sharing; it's always interesting to...Thanks for sharing; it's always interesting to hear the story behind the improvements. <br /><br />Question: Why not import each XML file when it's uploaded? This prevents a backlog requiring downtime from forming, and gets the item DB updated more regularly. Dickie Cricketshttps://www.blogger.com/profile/14384295548577093467noreply@blogger.com