mongodb - Performance issue with JSON input -

-

i loading mysql table mongodb source through kettle. mongodb table has more 4 million records , when run kettle job takes 17 hours finish first time load. incremental load takes more hour.i tried increasing commit size , giving more memory job, still performance not improving. think json input step takes long time parse data , hence slow. have these steps in transformation

  1. mongodb input step
  2. json input
  3. strings cut
  4. if field value null
  5. concat fields
  6. select values
  7. table output.

same 4 million records when extracted postgre way more fast mongodb. there way can improve performance? please me.

thanks, deepthi

run multiple copies of step. sounds have mongo input json input step parse json results right? use 4 or 8 copies of json input step ( or more depending on cpu's) , it'll speed up.

alternatively need parse full json, maybe can extract data via regex or something.


Comments

Post a Comment

Popular posts from this blog

design - Custom Styling Qt Quick Controls -

-
Image
i have custom design qt quick controls. example, change background colour of tool bar, since hate default design. how can that? in qt quick controls, there limited styling available via qt quick control styles items, buttonstyle , checkboxstyle , etc. at moment, other styles require delving qt sources , messing internal details. here complete example of how 1 might modify toolbar's style. main.qml import qtquick 2.1 import qtquick.layouts 1.0 import qtquick.controls 1.0 applicationwindow { toolbar: toolbar { id: toolbar component.oncompleted: toolbar.data[0].item.children = [newrectangle]; property item _newrectangle: rectangle { // rectangle within toolbarstyle's panel // gleaned from: // http://qt.gitorious.org/qt/qtquickcontrols/source/ // c304d741a27b5822a35d1fb83f8f5e65719907ce:src/styles/base/toolbarstyle.qml id: newrectangle anchors.fill: parent...
Read more

Unable to remove the www from url on https using .htaccess -

-
i want remove www site url. ie. https://www.domainname.com.au https://domainname.com.au . i have tried various .htacces code nothing work me. here present code remove www http not work https. rewriteengine on rewritebase / rewritecond %{http_host} ^www\.(.*)$ [nc] rewriterule ^/(.*)$ http://%1/$1 [r=301,l] rewritecond %{https} off rewriterule ^(.*)$ https://domainname.com.au/$1 [r=301,l] rewriterule ^code-of-conducts/(.*)/$ code-of-conducts.php?type=$1 rewriterule ^parents/post-a-job/(.*)/edit/$ parents/post-a-job.php?action=edit&id=$1 rewriterule ^parents/post-a-job/(.*)/$ parents/post-a-job.php?job_type=$1 rewriterule ^parents/open-jobs/(.*)/delete/$ parents/open-jobs.php?action=delete&id=$1 rewriterule ^parent-profile/(.*)/$ parent-profile.php?id=$1 rewriterule ^parents/job-details/(.*)$ parents/job-details.php?applied_id=$1 rewriterule ^sitterprofile/(.*)/$ sitterprofile.php?id=$1 rewriterule ^sitter/job-board/state/(.*)/page/(.*)/$ sitter/job-board.ph...
Read more