Bureaucrats, developer, Administrators
9,842
edits
Talk:Link transit: Difference between revisions
I don't currently have C++-compiler installed so I can't run that old code that Tim Starling supplied us with
Tim Starling (talk | contribs) (clarification) |
(I don't currently have C++-compiler installed so I can't run that old code that Tim Starling supplied us with) |
||
| (9 intermediate revisions by 5 users not shown) | |||
| Line 2: | Line 2: | ||
:This tends to be expensive software run by [[ad server]] companies. But it is certainly in use in all [[publicly traded search engine]]s like [[Yahoo]] and [[Google]], in fact, you can see the "imgurl" they use to track say which queries led to which image lookups. | :This tends to be expensive software run by [[ad server]] companies. But it is certainly in use in all [[publicly traded search engine]]s like [[Yahoo]] and [[Google]], in fact, you can see the "imgurl" they use to track say which queries led to which image lookups. | ||
: start with logging the HTTP referrer. Uncomment this line in apache's httpd.conf: '''#CustomLog logs/referer_log referer''' then rotate this file and publish it. | |||
---- | ---- | ||
| Line 10: | Line 12: | ||
::You say both are useful for the same reasons. What reasons are those? -- [[User:Tim Starling|Tim Starling]] 07:13, 4 Sep 2004 (EEST) | ::You say both are useful for the same reasons. What reasons are those? -- [[User:Tim Starling|Tim Starling]] 07:13, 4 Sep 2004 (EEST) | ||
:::A serious [[encyclopedia]] or any [[journal]] would care which pages were reviewed, and which were reviewed from which others, and how often, and what connections were of interest to readers. It's kind of embarassing to have to say that out loud. | |||
::::Of course we care about review, | |||
:::::[[Trolls]] doubt that very much. | |||
::::but wouldn't that be better served by popularity data than link transit data? | |||
:::::Restoring the information about page popularity is also quite useful. But more useful is information about paths between, i.e. that the single most popular path was for instance [[Wikipedia]] -> [[GFDL corpus]] -> [[sysop vandalism]] -> [[Wikimedia]] -> [[tabloid journalism]] -> [[libel suit]] -> [[easy street]] would demonstrate that people actually had understood the subjects correctly and went to the next most logical page to learn the next most important thing and [[ending Wikimedia|ended up in the most logical place where most reasonable people would want to go]] given that starting point, while a path from [[Zionism]] -> [[Robert Kaiser]] -> [[Nazipedia]] -> [[Anti-Anti-Anti-Anti-Zionism]] would perhaps indicate a quite confused person who was pretty much being subjected to a pile of [[propaganda]] and would come away with a quite different impression of the subject than what a serious editor would want. | |||
::::What would you do with link transit information? How do you "elaborate" a link? The best use of it I can think of is to pick a small set of related articles, and draw pretty graph pictures. A noble goal, to be sure, but it would require a change to the program below to generate such data efficiently. -- [[User:Tim Starling|Tim Starling]] 06:30, 6 Sep 2004 (EEST) | |||
:::::Pictures aren't really the point. What if one were able to simply see the number of times that a link was transited in the past month, as a superscripted number ("exponent") on that link? Then it would just fit in the regular page display. It would give some idea of the typical paths followed through pages. | |||
This matters because I need to know what the output format should be, and I need to have some way to justify using server resources to generate such data. | This matters because I need to know what the output format should be, and I need to have some way to justify using server resources to generate such data. | ||
| Line 18: | Line 30: | ||
:It looks like C to me and it looks like that the Main() takes standard httpd.log as input. I'll run this on our logs sometime when I have the time. Kinda busy now. --[[User:Juxo|Juxo]] 13:02, 3 Sep 2004 (EEST) | :It looks like C to me and it looks like that the Main() takes standard httpd.log as input. I'll run this on our logs sometime when I have the time. Kinda busy now. --[[User:Juxo|Juxo]] 13:02, 3 Sep 2004 (EEST) | ||
::Juxo, did you ever run this on the logs? If so then please put the output at [[Consumerium:attention]]. The word attention is used for compatibility with [[attention.xml]] which should include this type of information eventually for all [[web service]]s that support it. | |||
:::No, I haven't run the code and there is this little problem that I don't have a C++ compiler installed I tried once to install it but the installation said that there were some dependencies that were unmet or something similar. Right now I'm I don't have the installation disks with me so I can't install the compiler. --[[User:Juxo|Juxo]] 10:34, 9 Jun 2005 (GMT) | |||
::It's C++. It outputs two sections separated by a double linefeed. The first is an indexed list of URLs. The second has three values on each line: index from, index to and the transit count. The idea is that you would read all this into a relational database with an index on all three columns, then perform whatever analysis you need to perform. -- [[User:Tim Starling|Tim Starling]] 07:13, 4 Sep 2004 (EEST) | ::It's C++. It outputs two sections separated by a double linefeed. The first is an indexed list of URLs. The second has three values on each line: index from, index to and the transit count. The idea is that you would read all this into a relational database with an index on all three columns, then perform whatever analysis you need to perform. -- [[User:Tim Starling|Tim Starling]] 07:13, 4 Sep 2004 (EEST) | ||
| Line 29: | Line 45: | ||
#define LINE_BUF_SIZE 1000 | #define LINE_BUF_SIZE 1000 | ||
#define REPORTING_INTERVAL 10000 | #define REPORTING_INTERVAL 10000 | ||
| Line 171: | Line 188: | ||
} | } | ||
</pre> | </pre> | ||
---------------- | |||
''from [http://en.wikipedia.org/w/wiki.phtml?title=Wikipedia:Village_pump&oldid=5582713#Downloading_log_files.3F Wikipedia Village pump discussions 28 August to 1 September 2004 ]'' | |||
Does anybody know where I can download a few days worth of Wikipedia hit log files? I'd like to do some analysis on them, but I can't figure out where they're kept. Thanks, --William Pietri 00:30, 28 Aug 2004 (UTC) | |||
We don't have daily log files. We have log files for the current month, updated nightly. If you take the difference from one nightly log file to the next, you can calculate the day's hits. →Raul654 00:37, Aug 28, 2004 (UTC) | |||
Ah, I was looking for the raw log files, with Referer information. I was curious do some analysis on Wikipedia usage patterns, including how people come in via search engines and what they do after that point. But I gather you're saying that raw hit logs just aren't collected on a regular basis? In that case, are there older ones around, perhaps collected during an optimization run? --William Pietri 00:44, 28 Aug 2004 (UTC) | |||
The raw logs are available only to wikimedia's developers. Publishing them would be a major issue, as in general we view people's IPs, the sites and pages they visit, and what they do here to be private information. A lot of people would be very pissed off if the raw logs were available to the casual visitor. I suppose if there were such a thing as a decent unified logfile sanitiser (which removed/abstracted IP information) then publishing that would be an option. But if I were running such a sanitiser, I'd surely have it purge the referrer information too, as (occasionally) there's juicy info in the referring URL (particularly when the referrer is an email in a webmail service, and sometimes a search query). That said, there doesn't seem to be such a program anyway (I guess I don't know what it would be called, so it's rather hard to google for it). The problem of examining the log files is compounded by wikimedia's server setup - most pages are served ex-cache by one of the frontline squid servers; so we'd need to publish a number of the (huge) squid logs too. It would indeed be a very interesting exercise to run some analyses of the logs, as there's all kinds of things we don't know about how visitors and search engines enter, move around, and leave the site. If we could figure out an acceptable way to do things then there is much value to be mined. -- Finlay McWalter | Talk 00:58, 28 Aug 2004 (UTC) | |||
Right, logs are private. You can find some statistics extracted from log files, including a list of referrers, on the Webalizer stats page (http://wikimedia.org/stats/en.wikipedia.org/). -- Tim Starling 01:28, Aug 28, 2004 (UTC) | |||
Interesting. I hadn't thought about the privacy issues, but they make complete sense to me. If I get enough time to do the research properly, I'll contact y'all with a proposal to either A) write a log sanitizer that meets your standards, or B) write an analysis program that you guys run, so I never see the log data. Also, thanks for the link to the Webalizer pages; I had missed those the first time through. Thanks, --William Pietri 20:54, 30 Aug 2004 (UTC) | |||
When you write your log file analysis program, keep in mind that the squid logs for each day come to about five gigabytes. Also, depending on how much spare CPU time there is, running the analyzer might be done only intermittently, as has been the case with the Webalizer statistics. It will be interesting to see the results of this. Until now, everybody wanting to do this has been all talk and no code. -- Jeronim 05:26, 1 Sep 2004 (UTC) | |||