Mailing List Archive: 49091 messages
  • Home
  • Script library
  • AltME Archive
  • Mailing list
  • Articles Index
  • Site search
 

POST examples ...

 [1/10] from: petr::krenzelok::trz::cz at: 1-Jun-2001 19:13


Hi, at my work we are downloading CICS (IBM3270) files using some custom company apps, which simply wraps and parses screen data and stores them in the file. However, - nowadays more modern solutions exist, one of them being TCP/IP based solutions. As our company is moving to beast called SAP R3, we've got new mainframe. Part of it contains installed Unix, and one of the services is Rexx based ftp server from IBM (or so I think :-) Our admins prepared one solution for us - they told us to use IE, or Netscpape, so http interface. I asked them for simpler solution using ftp protocol, but they don't want to prepare special solution for us. But we need a little bit of automatition ... So I tried to wrap the solution usign Rebol, but I've got only some 30 min to play with it, as I had to leave my office. I got thru authentization, as it is simple http://user:[pass--server]..... Looking at html source however, I was not able to prepare POST part of http request, so I have few questions here: 1) are there any examples available using POST method? 2) what should I concentrate one? There seem to be plenty of various fields, and also javascript code. 3) how do I recognize which parameters should be enclosed in "", and which don't? 4) is there any way of how to split download into several parts? Some of files are more than 40 MB in size. I think that 'read reads whole file into memory. I remember some discussion using read/custom [start 0 end 100] or something like that but I am not sure if it is oficially supported. Should I use 'open instead of 'read? And in case I use 'open http:// ... - are all necessary http info send to server? Why do I want to use rebol for that?: 1) I want to use rebol in real life ... 2) The more tools I produce in rebol and show my friends it can pretty much help us to make some of our processes more automatic, the more chance we will go and buy /Command (I think it is set in stone already - I am chief of our small department, but I just want to be fair to my colleagues and don't push them to use some tool, before they come and tell me - hey, we can't solve it in a simple way - write something in Rebol for us :-) Thanks, -pekr-

 [2/10] from: gjones05:mail:orion at: 1-Jun-2001 16:55


From: "Petr Krenzelok"
> However, - nowadays more modern solutions exist, one of them being
TCP/IP
> based solutions. As our company is moving to beast called SAP R3,
we've got
> new mainframe. Part of it contains installed Unix, and one of the
services
> is Rexx based ftp server from IBM (or so I think :-)
SAP: :Lots-Of-Money ;-)
> Our admins prepared one solution for us - they told us to use IE, or > Netscpape, so http interface. I asked them for simpler solution using
ftp
> protocol, but they don't want to prepare special solution for us. But
we
> need a little bit of automatition ... > > So I tried to wrap the solution usign Rebol, but I've got only some 30
min
> to play with it, as I had to leave my office. I got thru
authentization, as
> it is simple http://user:[pass--server]..... > > Looking at html source however, I was not able to prepare POST part of
http
> request, so I have few questions here: > > 1) are there any examples available using POST method?
http://www.rebol.com/docs/core23/rebolcore-13.html#pgfId-956524 Look for "Posting CGI Data"
> 2) what should I concentrate one? There seem to be plenty of various
fields,
> and also javascript code.
Search for the <form ***> tags. There may be more than one. Find the one that seems relevant to your query. The find the bracketing </form>. If there are lots of fields, I usually edit out all the non-tag stuff, except just enough to know what the input field is about. With all the inputs, etc, between the <form />. You've got your base information. Generally, JavaScript code can be ignored, as it is used to validate fields, update other fields, etc. Keep an eye out for hidden fields (that are not visible in the browser) as they frequently contain critical data.
> 3) how do I recognize which parameters should be enclosed in "", and
which
> don't?
It is all "stringified" ultimately and url encoded. If there are spaces in data, then enclose.
> 4) is there any way of how to split download into several parts? Some
of
> files are more than 40 MB in size. I think that 'read reads whole file
into
> memory. I remember some discussion using read/custom [start 0 end 100]
or
> something like that but I am not sure if it is oficially supported.
Should I
> use 'open instead of 'read? And in case I use 'open http:// ... - are
all
> necessary http info send to server?
http://www.rebol.com/docs/core23/rebolcore-14.html#pgfId-539576 under direct port access
> Why do I want to use rebol for that?: > > 1) I want to use rebol in real life ... > > 2) The more tools I produce in rebol and show my friends it can pretty
much
> help us to make some of our processes more automatic, the more chance
we
> will go and buy /Command (I think it is set in stone already - I am
chief of
> our small department, but I just want to be fair to my colleagues and
don't
> push them to use some tool, before they come and tell me - hey, we
can't
> solve it in a simple way - write something in Rebol for us :-)
Great strategy!
> Thanks, > -pekr-
Good luck. --Scott Jones

 [3/10] from: petr:krenzelok:trz:cz at: 2-Jun-2001 0:30


Hi Scott. thanks for the reply!
> From: "Petr Krenzelok" > > However, - nowadays more modern solutions exist, one of them being
<<quoted lines omitted: 6>>
> SAP: :Lots-Of-Money > ;-)
I know :-) What is more - you have still to do most things in the most complicated manner - yourself :-)
> > 2) what should I concentrate one? There seem to be plenty of various > fields,
<<quoted lines omitted: 8>>
> (that are not visible in the browser) as they frequently contain > critical data.
Excelent description. There are two forms, and even some hidden fields ...
> > 3) how do I recognize which parameters should be enclosed in "", and > which > > don't? > > It is all "stringified" ultimately and url encoded. If there are spaces > in data, then enclose.
Ah, I don't think so. I remember one Sterling's email where he suggested someone to remove enclosing "" for one of parameters. I also tried the same with one of them, and it seemed to help (no more complaints about it, but the server seemed still to lack some info ...)
> > 4) is there any way of how to split download into several parts? Some > of
<<quoted lines omitted: 9>>
> http://www.rebol.com/docs/core23/rebolcore-14.html#pgfId-539576 > under direct port access
hmm, is there a way of how to get file length? using POST methog? -pekr-

 [4/10] from: gjones05:mail:orion at: 1-Jun-2001 19:14


Hi, Petr, I didn't have time earlier to find a good sample form. Here is the source for a test form taken from a cgi explanation site: (url to original page-form is half way down) http://www.cc.ukans.edu/~acs/docs/other/cgi-with-perl.shtml #####source for example form##### <html> <form method="post" action="http://www.cc.ukans.edu/cgiwrap/grobe/send-veggi-info.pl"> <P> If you would like more information about vegetarianism, please enter your name and e-mail address below.<P> Please enter your name:<br> <input type="text" name="name" size="35"><br> Please enter your e-mail address:<br> <input type="text" name="address" size="35"><p> <input type="submit" value="send address"> <input type="reset" value="start over"> </form> </html> ###### When I tried the way posted in the manula, it did not work. Sorry. I thought I had used that way before. However, maybe that is why I started using a different method. It took a bit to find this info, but I have translated a rebol post using my method: name: "My Name" address: [myname--mydom--dom] page: read/custom http://www.cc.ukans.edu/cgiwrap/grobe/send-veggi-info.pl reduce [ 'post rejoin [{name=} to-url name {&address=} to-url address] ] Maybe this example and real post form will let you experiment a bit before you return to work. --Scott Jones

 [5/10] from: petr:krenzelok:trz:cz at: 4-Jun-2001 11:55


GS Jones wrote:
> Hi, Petr, > > I didn't have time earlier to find a good sample form. >
OK, thanks for the tip. I made it - it works. I just had to remove enclosing "" for each parameter and take care of hidden newlines ... Now its easy to download the stuff using read/custom url [post {string-here}] The question is, I would like to do partial downloads, as some files are > 40 MB at the end of the year. I found Martin Johannesson's script at http://www.rebol.org/web/http-post.html, but I can't get it to work. Isn't there any more easy way? Some kind of 'rate even for 'read function would be handy :-) Anyway - thanks a lot ... Cheers, -pekr-

 [6/10] from: petr:krenzelok:trz:cz at: 4-Jun-2001 15:28


GS Jones wrote:
> From: "Petr Krenzelok" > > OK, thanks for the tip. I made it - it works. I just had to remove
<<quoted lines omitted: 8>>
> use: > 'post to-url rejoin [{name=} name {&address=} address]
I use more general aproach :-) Website source contains following string: #T1107_0003*MVS*DOWNLOAD**MZ7090S.TRA*P$PZ.MZ7090S.R$31007.CNTL**ASCII*172.26.80.11*MVS*NO*NO ... it contains some 22 or more such strings, each for one file .... I defined post-object: post-obj: context [ function: "TRASO_ACTION" ACTION_TYPE: "DOWNLOAD" FILE_DEF: FILE_TYPE: FTP_CODE_PAGE: ZIP: PC_FILE_NAME: SERVER_FILE_NAME: TARGET_SERVER: WHAT: "DOWNLOAD" ] I thought I will kill someone for putting "function" into website in lowercase. I spent nearly one hour figuring out what is the problem .... ... then I fill in object using above mentioned definition string: fill-post-obj: func [def-str [string!]][ tmp: parse def-str "*" post-obj/FILE_DEF: tmp/1 post-obj/TARGET_SERVER: tmp/2 post-obj/FILE_TYPE: tmp/3 post-obj/FTP_CODE_PAGE: tmp/8 post-obj/ZIP: tmp/11 post-obj/PC_FILE_NAME: tmp/5 post-obj/SERVER_FILE_NAME: tmp/6 ] and now let's produce post string from object definition. Look how powerfull Rebol is for series handling :-) post-obj-str: does [ str: copy "" foreach [word] next first post-obj [append str join "&" [word "=" get in post-obj word]] remove at str 1 ; remove first "&" at string ... return str ] and then: file-string: read/custom hostc-downloads reduce ['POST post-obj-str] ... one problem which bothers me is - I can't know the size of file :-) You know - it's all MVC (CICS3270 file :-)), you can know only some block-size, but it will not help you ... bloody historical things :-)
> I have no way to exactly duplicate your conditions, but I simulated a > situation where I wanted to skip the first 800k of a 1600k file, then > download in 1k chunks and the following worked for me. > > from-port: open/direct ftp://user:[pass--www--mydom--dom]/logs/my-access-log >
yes, but I am not using ftp. I am using http post method, so I don't know if such aproach is even possible ... One crazy thought: would it be possible to use http port 'awake field for calling the fucntion, which would count current buffer size and could update the progress bar? ... once you use read/part/custom .... /part refinement seems to be ignored .... the same will go for /skip imo .... So - there seems to be no simple solution. Probably the only one way is to use 'open and construct all http headers myself, but that's something I have no experience with yet ... -pekr-

 [7/10] from: gjones05:mail:orion at: 4-Jun-2001 10:48


From: "Petr Krenzelok" ...
> I thought I will kill someone for putting "function" into website in > lowercase. I spent nearly one hour figuring out what is the problem
.... I think we've all had that happen. Frustrating.
> yes, but I am not using ftp. I am using http post method, so I don't
know if
> such aproach is even possible ...
OK, not trying to beat a dead horse here, but what the heck. The same principle can be applied to http get requests. This example grabs the second half of Microsoft's home page. from-port: open/direct http://www.microsoft.com to-port: open/direct %//windows/desktop/log.txt from-port: skip from-port 8000 while [data: copy/part from-port 1000][append to-port data] close from-port close to-port However, you need to make the request with a 'post request, so I shamelessly hacked Martin Johannesson's http-post.r (the original url is no longer correct, so I also updated it). I commented the major changes so that you could see what I've done compared to the original. You will see that this method could be readily cleaned up and made more generic and encapsulated. Watch for wrapped lines!!! ;============================= REBOL [ Title: "Simple HTTP POST" File: %http-post.r Author: "Martin Johannesson" Email: [d95-mjo--nada--kth--se] Date: 30-Jun-1999 Purpose: { This script sends a "form" to a webserver using the POST method. The included example translates a string in English to German by posting the data to AltaVista's translation web page and then parsing the reply. } Category: [web util net 4] Comment: {Shamelessly hacked by Scott Jones for demonstrating tight http control for my good, friend Petr Krenzelok ;-} ] url-encode: func [ {URL-encode a string} data "String to encode" /local new-data ][ new-data: make string! "" normal-char: charset [ #"A" - #"Z" #"a" - #"z" #"@" #"." #"*" #"-" #"_" #"0" - #"9" ] if not string? data [return new-data] forall data [ append new-data either find normal-char first data [ first data ][ rejoin ["%" to-string skip tail (to-hex to-integer first data) -2] ] ] new-data ] http-post-form: func [ {Post a form to a web server} url "The URL to post to" data [block!] "A block of name/value pairs to represent the form" /local encoded-data port-spec HTTP-Post-Header http-request buffer tmp-buffer ][ port-spec: make port! [ scheme: 'tcp port-id: 80 timeout: 0:10 ] net-utils/url-parser/parse-url port-spec url encoded-data: make string! "" foreach [name value] data [ append encoded-data rejoin [ url-encode name "=" url-encode value "&" ] ] remove back tail encoded-data HTTP-Post-Header: make object! [ Accept: "*/*" User-Agent: reform ["REBOL" system/version] Host: port-spec/host Content-Type: "application/x-www-form-urlencoded" Content-Length: length? encoded-data ] http-request: rejoin [ "POST /" either found? port-spec/path [port-spec/path][""] either found? port-spec/target [port-spec/target][""] " HTTP/1.0^/" net-utils/export HTTP-Post-Header "^/" encoded-data ] ;dropped lines refinement and added direct http-port: open/direct [ scheme: 'tcp port-id: port-spec/port-id timeout: port-spec/timeout host: port-spec/host user: port-spec/user pass: port-spec/pass ] ;inserts 'post request insert http-port http-request ;now grabbing header stuff 1 byte at a time ;so that we can get content length ;look for two newlines in a row data: make string! 1000 while [not parse/all data [to "^/^/" to end]][ append data copy/part http-port 1 ] ;parses out the content length into ret-length ;which is left as a string for now parse data [thru "Content-Length: " copy ret-length to "^/" to end] ;print ret-length ;now get the rest of the file in what ever proportions you want data: make string! 1000 l-port: open/direct %//windows/desktop/log.txt ;uncomment the next line if you wish to skip some amount of info ;http-port: skip http-port 2000 while [data: copy/part http-port 1000][append l-port data] close http-port close l-port comment { buffer: make string! 10000 tmp-buffer: reform ["HTTP-Response:" pick http-port 1] while [not none? tmp-buffer] [ append buffer rejoin [tmp-buffer "^/"] tmp-buffer: pick http-port 1 ] close http-port } HTTP-Header: make object! [ HTTP-Response: Date: Server: Last-Modified: none Accept-Ranges: Content-Encoding: Content-Type: none Content-Length: Location: Expires: Referer: Connection: none ] ; parse-header HTTP-Header buffer ] english-to-german: func [ {Translates a string in English to German, using babelfish.altavista.com} english-text "String in english" ][ not-lt-gt: complement charset [#"<" #">"] tag-rule: ["<" some not-lt-gt ">"] tmp: http-post-form http://babelfish.altavista.com/tr reduce [ "doit" "done" "urltext" english-text "lp" "en_de" ] comment { if none? find tmp/HTTP-Response "200" [return join "Error: " tmp/HTTP-Response] either parse tmp/content [ thru "Auf Deutsch:" to "<font" thru ">" copy trans to "<" to end ][ return trans ][ return "Error: Unable to parse Babelfish HTML" ] } ] english-to-german "hello, how are you?" ;===============================
> One crazy thought: would it be possible to use http port 'awake field
for
> calling the fucntion, which would count current buffer size and could
update
> the progress bar?
It seems like there should be some way, knowing the total length and the chunk lengths. Hope this one helps *and* solve the problem. --Scott Jones

 [8/10] from: petr:krenzelok:trz:cz at: 4-Jun-2001 19:07


Hi,
> insert http-port http-request > ;now grabbing header stuff 1 byte at a time
<<quoted lines omitted: 4>>
> append data copy/part http-port 1 > ]
.. pressed for the time. I can understand most parts of the script, but are above lines correct? So you run 'while loop containing 'parse. What do you do if string like "asdfasf^/^/ something else" arrives to your port? Are you sure that by reading out 1 byte from the port you will get whole string till ^/^/ is reached? Thanks anyway - will try it at my work tomorrow ...
> Hope this one helps *and* solve the problem.
I already solved it :-) Well, the only one problem is 'read needs lots of memory, but it works :-) Right now I just want to get more familiar with POST method, so I am asking questions you are answering in a great manner. Appreciated! -pekr-

 [9/10] from: gjones05:mail:orion at: 4-Jun-2001 13:06


From "Scott Jones" .. SJ> > insert http-port http-request SJ> > ;now grabbing header stuff 1 byte at a time SJ> > ;so that we can get content length SJ> > ;look for two newlines in a row SJ> > data: make string! 1000 SJ> > while [not parse/all data [to "^/^/" to end]][ SJ> > append data copy/part http-port 1 SJ> > ] From: "Petr Krenzelok"
> .. pressed for the time. I can understand most parts of the script,
but are
> above lines correct? So you run 'while loop containing 'parse. What do
you
> do if string like "asdfasf^/^/ something else" arrives to your port?
Are
> you sure that by reading out 1 byte from the port you will get whole
string
> till ^/^/ is reached?
The logic I used was based on the fact that the header info is separated from the content by (a minimum of) 2 CRLFs (by http specification as I recall). What my hacked example does not take into account are error conditions, which could well occur. I guess the loop could check for excessive content header length (meaning it missed the two newline characters for what ever reason). It's hard to imagine a header being longer than a few hundred bytes, so I guess it could check to be sure the length doesn't exceed some reasonable default. I just realized that the "^/^/" could represent different things on different platforms, but the specification uses the CRLFCRLF sequence as the separator of header from content. I only let it read one byte at a time so as to not overshoot the sequence. I guess one could just read an arbitrary length the first time through, parse this data for header, then let the rest (the first part of the content) be parsed off to the content.
> Thanks anyway - will try it at my work tomorrow ... > > > Hope this one helps *and* solve the problem. > > I already solved it :-) Well, the only one problem is 'read needs lots
of
> memory, but it works :-) Right now I just want to get more familiar
with
> POST method, so I am asking questions you are answering in a great
manner.
> Appreciated!
I am sorry that I misunderstood that you *had* already solved it. I thought the code you posted showed the nice way that you constructed the parameters for a POST operation. I thought you were still wishing for a way to know the length and download either in portions and/or download only a specific portion. I used Martin J's code and method, mainly for speed, because all I had to do was change the parts that managed the port and getting header/length stuff. I posted this most recent code as a way of showing how to manage the port. I meant no slight to you, of course. Sorry. The way I outlined should avoid any significant memory problems, but performance could be improved for large files by picking a larger length. The size of the minimal hard disk storage unit would be the minimal ideal (is it a sector or a cluster; I have forgotten). Best wishes, --Scott Jones

 [10/10] from: gjones05:mail:orion at: 4-Jun-2001 7:56


From: "Petr Krenzelok"
> OK, thanks for the tip. I made it - it works. I just had to remove > enclosing "" for each parameter and take care of hidden newlines ... > > Now its easy to download the stuff using read/custom url [post > {string-here}]
Good. Of course, later I realized that a single 'to-url could have more easily been placed before 'rejoin. In other words, instead of: 'post rejoin [{name=} to-url name {&address=} to-url address] use: 'post to-url rejoin [{name=} name {&address=} address] which would save a lot of redundant calls on long forms.
> The question is, I would like to do partial downloads, > as some files are 40 MB at the end of the year. I found > Martin Johannesson's script at > http://www.rebol.org/web/http-post.html, but I can't > get it to work. Isn't there any more easy way? Some > kind of 'rate even for 'read function would be handy :-)
I have no way to exactly duplicate your conditions, but I simulated a situation where I wanted to skip the first 800k of a 1600k file, then download in 1k chunks and the following worked for me. from-port: open/direct ftp://user:[pass--www--mydom--dom]/logs/my-access-log to-port: open/direct %/local/folder/my-access-log.txt from-port: skip from-port 800000 ;skip the first 800k ;for fun download in small chunks, to prove that tight control is possible while [data: copy/part from-port 1000][append to-port data] close from-port close to-port Maybe this method will be useful. --Scott Jones

Notes
  • Quoted lines have been omitted from some messages.
    View the message alone to see the lines that have been omitted