Apache log analysis
[1/3] from: jason::cunliffe::verizon::net at: 18-Jun-2002 12:10
We need to analyze Apache logs by subdirectory. Preferably using REBOL.
Our server has lots of sub-directories named by artists work:
Works/groundzero/
Works/mythichybrid/
Works/trees/
etc..
Q1: Does anyone know how to configure Apache to generate log files per
directory?
I googled around a bit and found only examples for Virtual Hosts, which we've
already done.
I am using Webalizer for our main HTML presentation. [Linux RedHat 7.2]
Q2: Examples of Apache log file analysis using REBOL?
Our main access_log file typically reaches 10Mb before logrotate rolls it over
and starts a fresh one.
There are currently about 50 subfolders I need to generate individual stats for.
If REBOL is fast enough, I don't mind just using it to crank out 50+ stat files
with any lines including the named subdirectory. Ideally this would be run daily
but can also be called on the fly on a parameterized basis.
Webalizer is good for big domain level analysis, but I'd love to have
configurable REBOL script I could pipe to each artists index page, creative
visualization using Oldes' %make-swf.r ... you get the idea.
thanks
./Jason
[2/3] from: ethanak:interclub:pl at: 19-Jun-2002 8:44
On Jun 18 at 12:10 Jason Cunliffe wrote:
> We need to analyze Apache logs by subdirectory. Preferably using REBOL.
Of course :)
> Q1: Does anyone know how to configure Apache to generate log files per
> directory?
It's not possible. TransferLog directive is permitted only in main server
configuration and virtual hosts (see http://localhost/manual ;)
You can use Rewrite module to perform any action (f.ex. private
logs). But be careful! I don't know whether REBOL can use
non-buffered I/O on standard input/output!
Or - you can specify action for html/text/images/whatever_you_want
types (see documentation for mod_action). It will be safer but
will consume more resources.
One stupid question: why don't you want to analyse PATH part
from Apache transfer log file? It would be quite simple
and fast!
Regards
ethanak
[3/3] from: jason:cunliffe:verizon at: 19-Jun-2002 5:25
> > Q1: Does anyone know how to configure Apache to generate log files per
> > directory?
>
> It's not possible. TransferLog directive is permitted only in main server
> configuration and virtual hosts (see http://localhost/manual ;)
Thanks very much..
I checked my Apache books and googled around, but I could not find anything
anyhere about TransferLog for directories. But am/was not sure if I had missed
something.
> You can use Rewrite module to perform any action (f.ex. private
> logs). But be careful! I don't know whether REBOL can use
> non-buffered I/O on standard input/output!
Rewrite is some magic I recently discovered. It seems adepts can make it do
almost anything.
Not me yet though..
> Or - you can specify action for html/text/images/whatever_you_want
> types (see documentation for mod_action). It will be safer but
> will consume more resources.
hmm.. those ideas could be worth testing.
> One stupid question: why don't you want to analyse PATH part
> from Apache transfer log file? It would be quite simple
> and fast!
Right. That's what I started writing last night ;-) Used a 5Mb current
access_log file to test and started building a tool in /View to search for
string [folder name] and by date.
I was very impressed by how quick it is :-)
Fast to run and to write.
...except for the lack of better /View docs >:-((
One problem I see now is that for remote client reblet use, I still need to
pre-process my log file on the server, else the download data size is too big.
Another is that searching the Apache request paths could be unerailable.
Hopefully c\seom common sense rules-of-thumb can offset that.
Anyway, I found some starter code in 'REBOL for Dummies', which I've come to
appreciate more and more. It's the only book in that series I own. Clear simple
with some useful stuff sometimes not covered in the other REBOL books. Here's
the main function:
log-file: %access_log_sample.txt
search-log: func [keyword ][
result: copy ""
accesslog: open/direct/read/lines log-file
while [(line: pick accesslog 1) <> none ][
if (find line keyword) [
append result rejoin [line newline]
]
]
close accesslog
return result
]
and a little tool for the dates so they can be passed as keywords also
;; convert dates to format used in Apache logs so we can search them also:
18/Jun/2002
to-logdate: func [adate [date!]][
monthname: pick ["Jan" "Feb" "Mar" "Apr" "May" "Jun" "Jul" "Aug" "Sep" "Oct"
Nov
"Dec"] adate/month
d: copy ""
append d rejoin [adate/day "/" monthname "/" adate/year]
return d
]
I'll post the reblet when it stabilizes and gets some more mature features.
./Jason