1

In order to insert my data fast in MySQL, I have decided to take the 6200 JSON files I have (which have ~200K rows each) and convert them to CSV. Then, I want to use LOAD DATA INFILE to quickly insert all CSVs in MySQL. This is what I am doing so far:

myList = [] path = *some path to my files* for filename inglob.glob(os.path.join(path, '*.JSON')): with open(filename) as json_data: j = json.load(json_data) for i in j["rows"]: user_id = i["values"][0][0] birthday = int(float(i["values"][0][1])/1000) gender = str(epoch_timestamp) age = i["values"][0][2] eye_color = i["values"][0][3] data = (user_id, birthday, gender, age, eye_color) myList.append(data) fn = '/Users/srayan/Desktop/myCSV.csv' with open(fn,'w') as out: writer = csv.writer(out) writer.writerows(myList)

My problem is, going through each, individual row out of 1 billion rows of JSON is very inefficient - is there any way that I can either grab row data from JSON without having to iterate every single row? Or is anyone aware of a faster method than going from JSON -> CSV -> LOAD INFILE to MySQL? Thanks in advance.

Improve this question
4
  • Hopefully this will help. stackoverflow.com/questions/14330314/bulk-insert-in-mysql Commented Aug 3, 2017 at 18:27
  • @WEI_DBA I am planning on using LOAD DATA INFILE - but I am looking more for a solution on how to convert JSON to CSV fast Commented Aug 3, 2017 at 18:35
  • why its a billion row , its 200k row is it? . why your combining , its a loop of 6200 !! Commented Aug 3, 2017 at 19:08
  • No, 200k row per JSON file. 6200 JSON files. If you do the math... 1.2 billion rows total. Commented Aug 3, 2017 at 19:35
Load 7 more related questions Show fewer related questions

0

Reset to default

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.