0

I am having an almost identical problem to this one, posted in '09. It doesn't appear to be successfully solved, and I have been poring over related questions all day & implementing suggestions I have found.

I am attempting to pull dynamic (changes daily) data from an XML source (not a file) and insert into a mySQL database. When I attempt to pull the data without fwrite, the data returns without tags/keys in the output. Thus, I cannot properly assign the data to specific database fields.

I decided to write the information to a file and go from there, but the file aborts/"completes" at 334kb every single time. Unfortunately, I am a cURL novice and don't have the chops to see exactly what the issue is. Also, I am using vqMod for OpenCart to do this, otherwise I'd be using straight up PHP.

What am I missing?

Also, is there a way to pull the xml as an array instead of as a string (thereby potentially bypassing the fwrite step)? Or, should I be writing to some other file type?

Any suggestions or a redirect to a solved question I've missed are appreciated.

This is the relevant code, commented portions are fixes I've attempted: 

$curl = curl_init(); $fp = fopen('dir/file.xml' , "w" ); //w+ will not download any information from the url - file is created but empty. //AS IS, downloads first 334KB of file then lands on a blank page //and a 500 error when any of these other options are implemented. curl_setopt($curl, CURLOPT_URL, 'http://www.url.com'); curl_setopt($curl, CURLOPT_FILE, $fp); curl_setopt($curl, CURLOPT_FOLLOWLOCATION, true); // curl_setopt($curl, CURLOPT_RETURNTRANSFER, true); // curl_setopt($curl, CURLOPT_TIMEOUT, 300); // curl_setopt($curl, CURLOPT_NOPROGRESS, false); // curl_setopt($curl, CURLOPT_RANGE, '0-1000'); // $data = array(); $data = curl_exec($curl); fwrite($fp, $data); curl_close($curl); fclose($fp);

Update: Attempted to use simplexml_load_string instead of fwrite() to pull one product's information, but am still having limited success. Example of XML I am using:

 <?xml version="1.0"?> <response> <root> <part> <![CDATA[PARTNUM]]> </part> <errorcode>0</errorcode> <errormsg></errormsg> <special>N</special> <description> <![CDATA[]]> </description> <price>75</price> <weight>1.02</weight> <webpurch>Y</webpurch> <altnum> <![CDATA[ALT-NUM]]> </altnum> <active>Y</active> <redo> <![CDATA[]]> </redo> <codes> <code> <fieldname> <![CDATA[Mfr Part No]]> </fieldname> <fieldvalue> <![CDATA[PARTNUM]]> </fieldvalue> </code> <code> <fieldname> <![CDATA[Special Code 1]]> </fieldname> <fieldvalue> <![CDATA[XYZ123]]> </fieldvalue> </code> </codes> <customtag>N</customtag> <onhand>0</onhand> <notes> <![CDATA[PRODUCT-SPECIFIC NOTE]]> </notes> <mfr> <mfr_name> <![CDATA[MFR]]> </mfr_name> </mfr> <altpartnums> <altnum> <![CDATA[PARTNUM.12]]> </altnum> </altpartnums> <gtrue>N</gtrue> <group> <![CDATA[GROUP NAME]]> </group> <categories> <cat>294</cat> <cat>475</cat> </categories> </root> </response>

This is an example of the $data return for multiple products when I do not use fwrite(): 0 N 75 1.02 Y Y N 0 N 294 475 0 N 288 12 Y Y Y 18 Y 222 456 3786

Improve this question
6
  • how about simplexml_load_file / simplexml_load_string, php.net/manual/en/function.simplexml-load-file.php, it might do it, it doesn't give you an array per se but it does gives you a pretty easy to handle object, how big is the file? Commented Mar 1, 2013 at 19:54
  • Would file_get_contents not also work here? Commented Mar 1, 2013 at 19:55
  • @NathanMann, it would but it still needs parsing, simplexml_load_file and the likes of it in PHP solve it. Commented Mar 1, 2013 at 19:56
  • @Guy: The file varies in length - it is a list of active (30,000+) products with some related data coming from a Unix server. Every day, the list is slightly different. On the old system I'm converting from, I have a similarly created .csv that is usually about 23MB. Commented Mar 1, 2013 at 20:03
  • 1
    @Guy: No, I don't control the actual data that is included in the list. It is always the same detail set about each product that comes through (which is why I'd normally use the tags for allocation), but the products themselves are input/edited by other people and there is no way to know when someone has changed a minor detail. Essentially, I'd be updating manually all day every day if I had to make incremental changes to products in the DB. I'm testing the simplexml_load_file, will add results to initial post. Commented Mar 1, 2013 at 20:18

2 Answers 2

Reset to default
0

Remove this line:

fwrite($fp, $data);

I'm quite sure $data is empty because you've specified not to return any string data, instead write it to a file. Not so surprising the the file is empty too.

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

That's the thing - the file is not always empty. The information returned for $data is a list of characters (e.g. '1 Y N N 1333.4 55.45 MFR') without fwrite. There are at least 30,000 products to parse, and not all of them have a value in all fields of my xml. Thus, it is imperative that I have the matching xml tags to be able to properly put my product information from my server directly to my database. As I stated initially, I had tried it without fwrite but then decided to go that route to see if I could pull intact xml.
What do you mean by "matching tags"? Why are the contents not the same when you choose to return it as a string and not save to file? Doesn't make sense to me...
I am pulling from xml with various trees. I need information for more than one tree AND need to know exactly which tree the info came from. The info is returned as an unbroken string, with no indicators as to what information is being presented. <tag>1</tag><tag2>y</tag2> vs. '1 Y' ---- There are further problems with nested trees, empty trees, etc.
^^ I meant "roots" not 'trees'. Either way, I've got roots, parents, children, sub-children, even sibling elements (tags) that may or may not contain data for each specific product.
Your problem right now is failing to download the file right? One step at a time, you can't parse the file before you've succeeded to download it completely. Could you perhaps send a link to it unless it's private?
|
0

I got it to work sans fwrite() using this code:

 if(extension_loaded('curl')){ $curl = curl_init(); curl_setopt($curl, CURLOPT_URL, 'http://www.url.com'); curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1); curl_setopt($curl, CURLOPT_TIMEOUT, 30); curl_setopt($curl, CURLOPT_NOPROGRESS, FALSE); curl_setopt($curl, CURLOPT_MAXCONNECTS, 1); curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 0); $newparts = curl_exec($curl); $xml = simplexml_load_string($newparts); foreach($xml->modpart as $item){ if(($item->active == 'Y') || ($item->active == 'YES')){ $status = '1'; } else { $status = '0'; } if(!empty($item->mfr)){ $rs = $this->db->query("SELECT manufacturer_id FROM ".DB_PREFIX."manufacturer WHERE name = '".$item->mfr->mfr_name."'"); $mfr_id = $rs->row['manufacturer_id']; } if(!empty($item->codes)){ if($item->codes->code->fieldname == 'Mfr Part No'){ $mpn = $item->codes->code->fieldvalue; } } $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product (model, sku, mpn, quantity, date_available, sort_order, manufacturer_id, price, weight, status, date_added) VALUES ( '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', '".mysql_real_escape_string($item->sku)."', '".$mpn."', '".mysql_real_escape_string($item->mpn)."', NOW(), 1000, '".$mfr_id."', '".mysql_real_escape_string($item->price)."', '".mysql_real_escape_string($item->weight)."', '".$status."', NOW() )"); $pr = $this->db->query("SELECT product_id FROM ".DB_PREFIX."product WHERE sku = '".$item->part."'"); $product_id = $pr->row['product_id']; if(isset($product_id)){ $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_description (product_id, language_id, name, meta_keyword, meta_description, description, tag) VALUES( '" . (int)$product_id . "', '1', '" . $item->part. "', '" . $item->part. "', '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."', '".mysql_real_escape_string($item->description=htmlspecialchars(trim($item->description)))."' )"); $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_store SET product_id = '" . (int)$product_id . "', store_id = '0'"); $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "url_alias SET query = 'product_id=" . (int)$product_id . "', keyword = '" .$item->part . "'"); foreach($item->cats as $cats){ $category = $cats->cat; foreach($category as $category_id){ $this->db->query("INSERT IGNORE INTO " . DB_PREFIX . "product_to_category SET product_id = '" . (int)$product_id . "', category_id = '" . (int)$category_id . "'"); } } $this->cache->delete('product'); } } curl_close($curl); }

I still have had no luck with the timeout issue, but at least I can get my products into the database by some means. I will post the timeout solution if/when I find it.

Improve this answer

Comments

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.