POST examples ...

TCP/IP

we've got

services

SAP: :Lots-Of-Money ;-)

ftp

we

min

authentization, as

http

http://www.rebol.com/docs/core23/rebolcore-13.html#pgfId-956524 Look for "Posting CGI Data"

fields,

Search for the <form ***> tags. There may be more than one. Find the one that seems relevant to your query. The find the bracketing </form>. If there are lots of fields, I usually edit out all the non-tag stuff, except just enough to know what the input field is about. With all the inputs, etc, between the <form />. You've got your base information. Generally, JavaScript code can be ignored, as it is used to validate fields, update other fields, etc. Keep an eye out for hidden fields (that are not visible in the browser) as they frequently contain critical data.

which

It is all "stringified" ultimately and url encoded. If there are spaces in data, then enclose.

of

into

or

Should I

all

http://www.rebol.com/docs/core23/rebolcore-14.html#pgfId-539576 under direct port access

much

we

chief of

don't

can't

Great strategy!

Good luck. --Scott Jones

I know :-) What is more - you have still to do most things in the most complicated manner - yourself :-)

Excelent description. There are two forms, and even some hidden fields ...

Ah, I don't think so. I remember one Sterling's email where he suggested someone to remove enclosing "" for one of parameters. I also tried the same with one of them, and it seemed to help (no more complaints about it, but the server seemed still to lack some info ...)

hmm, is there a way of how to get file length? using POST methog? -pekr-

OK, thanks for the tip. I made it - it works. I just had to remove enclosing "" for each parameter and take care of hidden newlines ... Now its easy to download the stuff using read/custom url [post {string-here}] The question is, I would like to do partial downloads, as some files are > 40 MB at the end of the year. I found Martin Johannesson's script at http://www.rebol.org/web/http-post.html, but I can't get it to work. Isn't there any more easy way? Some kind of 'rate even for 'read function would be handy :-) Anyway - thanks a lot ... Cheers, -pekr-

I use more general aproach :-) Website source contains following string: #T1107_0003*MVS*DOWNLOAD**MZ7090S.TRA*P$PZ.MZ7090S.R$31007.CNTL**ASCII*172.26.80.11*MVS*NO*NO ... it contains some 22 or more such strings, each for one file .... I defined post-object: post-obj: context [ function: "TRASO_ACTION" ACTION_TYPE: "DOWNLOAD" FILE_DEF: FILE_TYPE: FTP_CODE_PAGE: ZIP: PC_FILE_NAME: SERVER_FILE_NAME: TARGET_SERVER: WHAT: "DOWNLOAD" ] I thought I will kill someone for putting "function" into website in lowercase. I spent nearly one hour figuring out what is the problem .... ... then I fill in object using above mentioned definition string: fill-post-obj: func [def-str [string!]][ tmp: parse def-str "*" post-obj/FILE_DEF: tmp/1 post-obj/TARGET_SERVER: tmp/2 post-obj/FILE_TYPE: tmp/3 post-obj/FTP_CODE_PAGE: tmp/8 post-obj/ZIP: tmp/11 post-obj/PC_FILE_NAME: tmp/5 post-obj/SERVER_FILE_NAME: tmp/6 ] and now let's produce post string from object definition. Look how powerfull Rebol is for series handling :-) post-obj-str: does [ str: copy "" foreach [word] next first post-obj [append str join "&" [word "=" get in post-obj word]] remove at str 1 ; remove first "&" at string ... return str ] and then: file-string: read/custom hostc-downloads reduce ['POST post-obj-str] ... one problem which bothers me is - I can't know the size of file :-) You know - it's all MVC (CICS3270 file :-)), you can know only some block-size, but it will not help you ... bloody historical things :-)

yes, but I am not using ftp. I am using http post method, so I don't know if such aproach is even possible ... One crazy thought: would it be possible to use http port 'awake field for calling the fucntion, which would count current buffer size and could update the progress bar? ... once you use read/part/custom .... /part refinement seems to be ignored .... the same will go for /skip imo .... So - there seems to be no simple solution. Probably the only one way is to use 'open and construct all http headers myself, but that's something I have no experience with yet ... -pekr-

.... I think we've all had that happen. Frustrating.

know if

OK, not trying to beat a dead horse here, but what the heck. The same principle can be applied to http get requests. This example grabs the second half of Microsoft's home page. from-port: open/direct http://www.microsoft.com to-port: open/direct %//windows/desktop/log.txt from-port: skip from-port 8000 while [data: copy/part from-port 1000][append to-port data] close from-port close to-port However, you need to make the request with a 'post request, so I shamelessly hacked Martin Johannesson's http-post.r (the original url is no longer correct, so I also updated it). I commented the major changes so that you could see what I've done compared to the original. You will see that this method could be readily cleaned up and made more generic and encapsulated. Watch for wrapped lines!!! ;============================= REBOL [ Title: "Simple HTTP POST" File: %http-post.r Author: "Martin Johannesson" Email: [d95-mjo--nada--kth--se] Date: 30-Jun-1999 Purpose: { This script sends a "form" to a webserver using the POST method. The included example translates a string in English to German by posting the data to AltaVista's translation web page and then parsing the reply. } Category: [web util net 4] Comment: {Shamelessly hacked by Scott Jones for demonstrating tight http control for my good, friend Petr Krenzelok ;-} ] url-encode: func [ {URL-encode a string} data "String to encode" /local new-data ][ new-data: make string! "" normal-char: charset [ #"A" - #"Z" #"a" - #"z" #"@" #"." #"*" #"-" #"_" #"0" - #"9" ] if not string? data [return new-data] forall data [ append new-data either find normal-char first data [ first data ][ rejoin ["%" to-string skip tail (to-hex to-integer first data) -2] ] ] new-data ] http-post-form: func [ {Post a form to a web server} url "The URL to post to" data [block!] "A block of name/value pairs to represent the form" /local encoded-data port-spec HTTP-Post-Header http-request buffer tmp-buffer ][ port-spec: make port! [ scheme: 'tcp port-id: 80 timeout: 0:10 ] net-utils/url-parser/parse-url port-spec url encoded-data: make string! "" foreach [name value] data [ append encoded-data rejoin [ url-encode name "=" url-encode value "&" ] ] remove back tail encoded-data HTTP-Post-Header: make object! [ Accept: "*/*" User-Agent: reform ["REBOL" system/version] Host: port-spec/host Content-Type: "application/x-www-form-urlencoded" Content-Length: length? encoded-data ] http-request: rejoin [ "POST /" either found? port-spec/path [port-spec/path][""] either found? port-spec/target [port-spec/target][""] " HTTP/1.0^/" net-utils/export HTTP-Post-Header "^/" encoded-data ] ;dropped lines refinement and added direct http-port: open/direct [ scheme: 'tcp port-id: port-spec/port-id timeout: port-spec/timeout host: port-spec/host user: port-spec/user pass: port-spec/pass ] ;inserts 'post request insert http-port http-request ;now grabbing header stuff 1 byte at a time ;so that we can get content length ;look for two newlines in a row data: make string! 1000 while [not parse/all data [to "^/^/" to end]][ append data copy/part http-port 1 ] ;parses out the content length into ret-length ;which is left as a string for now parse data [thru "Content-Length: " copy ret-length to "^/" to end] ;print ret-length ;now get the rest of the file in what ever proportions you want data: make string! 1000 l-port: open/direct %//windows/desktop/log.txt ;uncomment the next line if you wish to skip some amount of info ;http-port: skip http-port 2000 while [data: copy/part http-port 1000][append l-port data] close http-port close l-port comment { buffer: make string! 10000 tmp-buffer: reform ["HTTP-Response:" pick http-port 1] while [not none? tmp-buffer] [ append buffer rejoin [tmp-buffer "^/"] tmp-buffer: pick http-port 1 ] close http-port } HTTP-Header: make object! [ HTTP-Response: Date: Server: Last-Modified: none Accept-Ranges: Content-Encoding: Content-Type: none Content-Length: Location: Expires: Referer: Connection: none ] ; parse-header HTTP-Header buffer ] english-to-german: func [ {Translates a string in English to German, using babelfish.altavista.com} english-text "String in english" ][ not-lt-gt: complement charset [#"<" #">"] tag-rule: ["<" some not-lt-gt ">"] tmp: http-post-form http://babelfish.altavista.com/tr reduce [ "doit" "done" "urltext" english-text "lp" "en_de" ] comment { if none? find tmp/HTTP-Response "200" [return join "Error: " tmp/HTTP-Response] either parse tmp/content [ thru "Auf Deutsch:" to "<font" thru ">" copy trans to "<" to end ][ return trans ][ return "Error: Unable to parse Babelfish HTML" ] } ] english-to-german "hello, how are you?" ;===============================

for

update

It seems like there should be some way, knowing the total length and the chunk lengths. Hope this one helps *and* solve the problem. --Scott Jones

.. pressed for the time. I can understand most parts of the script, but are above lines correct? So you run 'while loop containing 'parse. What do you do if string like "asdfasf^/^/ something else" arrives to your port? Are you sure that by reading out 1 byte from the port you will get whole string till ^/^/ is reached? Thanks anyway - will try it at my work tomorrow ...

I already solved it :-) Well, the only one problem is 'read needs lots of memory, but it works :-) Right now I just want to get more familiar with POST method, so I am asking questions you are answering in a great manner. Appreciated! -pekr-

but are

you

Are

string

The logic I used was based on the fact that the header info is separated from the content by (a minimum of) 2 CRLFs (by http specification as I recall). What my hacked example does not take into account are error conditions, which could well occur. I guess the loop could check for excessive content header length (meaning it missed the two newline characters for what ever reason). It's hard to imagine a header being longer than a few hundred bytes, so I guess it could check to be sure the length doesn't exceed some reasonable default. I just realized that the "^/^/" could represent different things on different platforms, but the specification uses the CRLFCRLF sequence as the separator of header from content. I only let it read one byte at a time so as to not overshoot the sequence. I guess one could just read an arbitrary length the first time through, parse this data for header, then let the rest (the first part of the content) be parsed off to the content.

of

with

manner.

I am sorry that I misunderstood that you *had* already solved it. I thought the code you posted showed the nice way that you constructed the parameters for a POST operation. I thought you were still wishing for a way to know the length and download either in portions and/or download only a specific portion. I used Martin J's code and method, mainly for speed, because all I had to do was change the parts that managed the port and getting header/length stuff. I posted this most recent code as a way of showing how to manage the port. I meant no slight to you, of course. Sorry. The way I outlined should avoid any significant memory problems, but performance could be improved for large files by picking a larger length. The size of the minimal hard disk storage unit would be the minimal ideal (is it a sector or a cluster; I have forgotten). Best wishes, --Scott Jones

Good. Of course, later I realized that a single 'to-url could have more easily been placed before 'rejoin. In other words, instead of: 'post rejoin [{name=} to-url name {&address=} to-url address] use: 'post to-url rejoin [{name=} name {&address=} address] which would save a lot of redundant calls on long forms.

I have no way to exactly duplicate your conditions, but I simulated a situation where I wanted to skip the first 800k of a 1600k file, then download in 1k chunks and the following worked for me. from-port: open/direct ftp://user:[pass--www--mydom--dom]/logs/my-access-log to-port: open/direct %/local/folder/my-access-log.txt from-port: skip from-port 800000 ;skip the first 800k ;for fun download in small chunks, to prove that tight control is possible while [data: copy/part from-port 1000][append to-port data] close from-port close to-port Maybe this method will be useful. --Scott Jones