Timeline for Speed inserts in database with a table with +47 million data per each year, containing more than 10 years?
Current License: CC BY-SA 3.0
12 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| Jan 25, 2015 at 8:27 | history | tweeted | twitter.com/#!/StackDBAs/status/559266264569638913 | ||
| S Jan 9, 2015 at 23:40 | history | suggested | Jon of All Trades | The previous tags did not apply at all | |
| Jan 9, 2015 at 22:12 | review | Suggested edits | |||
| S Jan 9, 2015 at 23:40 | |||||
| Nov 16, 2011 at 12:35 | comment | added | Gaius | Can you write a tool to process your proprietary format into CSV, and use that as an intermediate step? | |
| Nov 16, 2011 at 10:49 | comment | added | Lucia | OK, I'm sorry... it is indeed a bit confusing. First of all, what I'm trying to do (because it's simpler) is to transfer the data form our custom database (which is filesystem-based, with custom binary files) to a standard SQL database. The real source of the info comes from our propietary software and it is the data alocated in the memory:the program has a large set of objects in which we store vectors of date/value which can be huge depending on the processing (because we can choose any startDate and endDate).We want to store hourly data.This would be the ultimate goal of the SQL database. | |
| Nov 16, 2011 at 10:41 | answer | added | Simon Hughes | timeline score: 1 | |
| Nov 16, 2011 at 10:32 | comment | added | Gaius | @lucia there's quite a nice trick in one of the comments for CSV files. But I am confused by your last comment - are you selecting from MySQL then inserting back into it? | |
| Nov 16, 2011 at 10:20 | comment | added | Lucia | I'm using MyIsam! | |
| Nov 16, 2011 at 10:19 | comment | added | Lucia | I'm trying to insert one year in each try. The source of the data is the actual data base, but this part don't take too much. I'm using VB.NET. I will try now to insert from file instead of programming language to see how this works. Do you think this would help? | |
| Nov 16, 2011 at 10:16 | comment | added | Gaius | Which storage engine are you using? It can make a big difference | |
| Nov 16, 2011 at 10:13 | comment | added | gbn | Are you trying to insert in one go? What is the source of the data? What .net code are you using? Also see dba.stackexchange.com/questions/7848/… for inspiration | |
| Nov 16, 2011 at 10:02 | history | asked | Lucia | CC BY-SA 3.0 |