Apache log analysis
[1/3] from: jason::cunliffe::verizon::net at: 18-Jun-2002 12:10
We need to analyze Apache logs by subdirectory. Preferably using REBOL. Our server has lots of sub-directories named by artists work: Works/groundzero/ Works/mythichybrid/ Works/trees/ etc.. Q1: Does anyone know how to configure Apache to generate log files per directory? I googled around a bit and found only examples for Virtual Hosts, which we've already done. I am using Webalizer for our main HTML presentation. [Linux RedHat 7.2] Q2: Examples of Apache log file analysis using REBOL? Our main access_log file typically reaches 10Mb before logrotate rolls it over and starts a fresh one. There are currently about 50 subfolders I need to generate individual stats for. If REBOL is fast enough, I don't mind just using it to crank out 50+ stat files with any lines including the named subdirectory. Ideally this would be run daily but can also be called on the fly on a parameterized basis. Webalizer is good for big domain level analysis, but I'd love to have configurable REBOL script I could pipe to each artists index page, creative visualization using Oldes' %make-swf.r ... you get the idea. thanks ./Jason
[2/3] from: ethanak:interclub:pl at: 19-Jun-2002 8:44
On Jun 18 at 12:10 Jason Cunliffe wrote:
> We need to analyze Apache logs by subdirectory. Preferably using REBOL.
Of course :)
> Q1: Does anyone know how to configure Apache to generate log files per > directory?
It's not possible. TransferLog directive is permitted only in main server configuration and virtual hosts (see http://localhost/manual ;) You can use Rewrite module to perform any action (f.ex. private logs). But be careful! I don't know whether REBOL can use non-buffered I/O on standard input/output! Or - you can specify action for html/text/images/whatever_you_want types (see documentation for mod_action). It will be safer but will consume more resources. One stupid question: why don't you want to analyse PATH part from Apache transfer log file? It would be quite simple and fast! Regards ethanak
[3/3] from: jason:cunliffe:verizon at: 19-Jun-2002 5:25
> > Q1: Does anyone know how to configure Apache to generate log files per > > directory? > > It's not possible. TransferLog directive is permitted only in main server > configuration and virtual hosts (see http://localhost/manual ;)
Thanks very much.. I checked my Apache books and googled around, but I could not find anything anyhere about TransferLog for directories. But am/was not sure if I had missed something.
> You can use Rewrite module to perform any action (f.ex. private > logs). But be careful! I don't know whether REBOL can use > non-buffered I/O on standard input/output!
Rewrite is some magic I recently discovered. It seems adepts can make it do almost anything. Not me yet though..
> Or - you can specify action for html/text/images/whatever_you_want > types (see documentation for mod_action). It will be safer but > will consume more resources.
hmm.. those ideas could be worth testing.
> One stupid question: why don't you want to analyse PATH part > from Apache transfer log file? It would be quite simple > and fast!
Right. That's what I started writing last night ;-) Used a 5Mb current access_log file to test and started building a tool in /View to search for string [folder name] and by date. I was very impressed by how quick it is :-) Fast to run and to write. ...except for the lack of better /View docs >:-(( One problem I see now is that for remote client reblet use, I still need to pre-process my log file on the server, else the download data size is too big. Another is that searching the Apache request paths could be unerailable. Hopefully c\seom common sense rules-of-thumb can offset that. Anyway, I found some starter code in 'REBOL for Dummies', which I've come to appreciate more and more. It's the only book in that series I own. Clear simple with some useful stuff sometimes not covered in the other REBOL books. Here's the main function: log-file: %access_log_sample.txt search-log: func [keyword ][ result: copy "" accesslog: open/direct/read/lines log-file while [(line: pick accesslog 1) <> none ][ if (find line keyword) [ append result rejoin [line newline] ] ] close accesslog return result ] and a little tool for the dates so they can be passed as keywords also ;; convert dates to format used in Apache logs so we can search them also: 18/Jun/2002 to-logdate: func [adate [date!]][ monthname: pick ["Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct" Nov "Dec"] adate/month d: copy "" append d rejoin [adate/day "/" monthname "/" adate/year] return d ] I'll post the reblet when it stabilizes and gets some more mature features. ./Jason