Before opening a new ticket, please verify SureDone status at https://status.suredone.com and also review our helpful articles below.

Start a new topic

Automation from public HTTPS


We need to import inventory data from a URL:


https://mywebsite.com/amfeed/feed/download?id=10&file=mywebsite_inventory_feeds.csv


The automation does downloads the file, but it fails, I believe, because we do not define connection: payload. 


Questions:


1. And how to set up the connection:payload?

2. Can you give us a working example of what we else we need apart from the payload?

 

See attachment for the error message and the file structure.


See also, the full code:


"connection":{

"type":"http",

"method":GET,

"address":"mywebsite.com/amfeed/feed/download?id=10&file=mywebsite_inventory_feeds.csv",

"paginate":1

}


"file_configs":

[{

"name":"mywebsite_inventory_feeds.csv",

"identifier":"guid",

"identifier_prefix_vendor":"FTNJ",

"search":"guid:FTNJ",

"diff_update": 1,

"diff_fields":["stock"],

"field_map":{

 "guid":"Sku",

 "stock":"Stock Qty"

},

"field_run":{

 "Sku":"Sku",

 "Stock Qty":"Stock Qty"

},

"field_format":{

 "guid":{

  "combine":{

   "fields":[

    "Sku",

    ""

   ],

  "delimiter":""

  }

 }

},


"stock_field":"stock",

"stock_negative":false,

"update":"edit"

}]



Hey Erno, after investigating this, we determined it was related to a bug with certain paginated requests in Automation Engine. This bug should now be fixed and you should see this automation behave as expected, with no configuration changes needed on your end!


1 person likes this

this does work now , I had to delete the "paginate":1 thought

Hi, I have another issue with anther file  I can not see why. ( Note: I run the same file from an FTP server without any issue)


this does not work


"connection":{

 "type":"http",

 "method":"GET",

 "address":"b2b.holley.com/data/atp/?api_key=0.jo47graux5",

 "pagination":{

  "page_iterator":25000,

  "page_parameter":"page"

 }

}


"file_configs":

{

"name":"atp.csv",

"headers":[

 "mpn",

 "brand",

 "description",

 "weight",

 "length",

 "width",

 "height",

 "quantity",

 "map_price"

],


"identifier":"guid",

"diff_update": 1,

"diff_fields":["stock"],

"field_map":{

 "stock":"quantity"

},

"field_run":{

"mpn":"mpn",

"brand":"brand"

},


"value_map": {

 "ACCEL": "BDDP",

 "APR": "HJQF",

 "B&M": "BBFT",

 "Bright Earth": "HMQT",

 "DiabloSport": "CLNS",

 "Dinan": "GFBV",

 "Flowmaster": "BDXN",

 "Flowtech": "CJWH",

 "Frostbite": "GSKG",

 "Hays": "BCCR",

 "Holley": "BBVL",

 "Holley EFI": "GVMR",

 "Hooker": "BBVM",

 "Hurst": "BCCS",

 "Lakewood": "BCCT",

 "Mallory": "BFKR",

 "MSD": "BCCX",

 "Quick Time": "GCHT",

 "Racepak": "GXNC",

 "SuperChips": "BGND",

 "Weiand": "BHTK",

 "XDR": "HGHP"

},

"field_format":{

 "guid":{

  "combine":{

   "fields":[

    "brand",

    "mpn"

   ],

  "delimiter":""

  }

 }

},


"stock_field":"stock",

"stock_negative":false,

"update":"edit"

}

Hey Erno,

Looking at the logs for this automation run, it looks like the response is HTML, as opposed to the CSV file you're expecting.

You can verify the response by using an API Tester (such as Postman) to send the GET request to that address and you can see the raw response.
Pasting the URL into a browser, it seems to show a login page first, so you may be able to use connection.login_address, connection.username, connection.password in order to pass in the proper credentials, which will allow the automation to log into the site before sending a follow-up request to download the file.

Let us know if this works, and if you have any further issues!

Hi,

 

I changed the connection to, but I can not tell if the file is read/generated.

What I can see in the logs is that the guid can not be generated, even-though when I read the file from a regular FTP with the same "file_configs", it creates the guid just fine....


Anyway see below my latest try ( does not work....). Please take a look at this and let me know if you can work out a working example for me. Regards


"connection":{

 "type":"http",

 "login_address":"https://b2b.holley.com",

 "username":"my user name",

 "password":"my password",

 "method":"GET",

 "address":"b2b.holley.com/data/atp/?api_key=my_apikey",

 "pagination":{

  "page_iterator":25000,

  "page_parameter":"page"

 }

}

Hey Erno,

A few observations to debug this:

  • The logs show that the response from the connection is still a jumble of HTML which indicates the connection is still not configured properly.

  • Looking at your config, I see the following: "address":"b2b.holley.com/data/atp/?api_key=your_apikey/atp.csv", but generally URL query parameters (?some_field=some_value) are at the very end of a URL, indicating this is probably a malformed URL

  • If there's an API key being used, it's actually unlikely that a login config is needed (contrary to my previous post -- I mentioned trying that because when I tried to paste the URL you have into the browser, it showed a login page, so Automation Engine can usually handle that via those configs, but again if we have an API key on hand, that should be enough actually)

  • It seems we're just trying to download a CSV file, so there should be no need for pagination in this case. Pagination is used for calling an API multiple times and getting different data for each page (usually in the form of JSON), but if we're downloading a .csv file here, we're almost certainly getting the full file in one single call.

  • "https://" should probably be prepended to the address


So with all that said, let's give it another shot! Try this connection config:


"connection": {

  "type":"http",

  "method":"GET",

  "address":"https://b2b.holley.com/data/atp/?api_key=your_apikey"

}

Let us know if it works!


Login or Signup to post a comment