sql - How should data be provided to a web server using a data warehouse? -


we have data stored in data warehouse follows:

  • price
  • date
  • product name (varchar(25))

we have 4 products. changes infrequently (on average once every 10 years). once every business day, 4 new data points added representing day's price each product.

on website, user can request information entering date range , selecting 1 or more products names. analytics shows feature not heavily used (about 10 users requests per week).

it suggested data warehouse should daily push (sftp) csv file containing data (currently 6718 rows of data , growing 4 each day) web server. then, web server read data file , display data whenever user made request.

usually, push once day, more 1 push possible communicate (infrequent) price corrections. in price correction scenario, data delivered in file. problems approach?

would better have web server make request data warehouse per user request? or have issues such greater chance network errors or performance issues?

would better have web server make request data warehouse per user request?

yes would. have little data, there no need try , 'cache' in way. (apart fact csv might not best way this). there nothing stopping doing these requests webserver database server. little information not find performance issue, if when grows, there lot gained on database-side (indexes etc) survive next 100 years in fashion.

the amount of requests users (also extremely small) not need special treatment, again, direct query best.

or have issues such greater chance network errors or performance issues?

well, might, not justify csv method. examples , why need not worry, be

  • the connection databaseserver down.
    issue both methods, 1 connection per day change of 1-in-10000 failures might seem better once-a-day methods. these issues should not come often, , if do, should able handle them. (retry request, give message user). enourmous amounts of websites do, trust me if not issue. also, think of mean if daily update failed? present bigger problem!
  • performance issues
    said, due amount of data , requests, not problem. , if becomes one, problem should able catch @ different level. use caching system (non csv) on database server. use caching system on webserver. fix indexes stop performance being problem.

but:

it far strange want data-warehouse separated web system. if requirement, , surely be, best thing can re-create warehouse-database (the 1 defended being enough query directly) on machine. might results doing master-slave system

  • your datawarehouse master-database: sends changes slave inexcessible otherwise
  • your 2nd database (on webserver even) gets updates master, , read-only. can query data
  • your webserver cannot connect datawarehouse, can connect slave read information. if there injection hack, doesn't matter, read-only.

now don't have single moment update queried database (the master-slave replication keep updated always), no chance queries webserver put warehouse in danger. profit!


Comments

Popular posts from this blog

Unable to remove the www from url on https using .htaccess -