FS#2808 Popularity data submission doesn't seem to work in many cases since Rincewind

Backend

• 

  2013-06-19 Michitux

  Some analysis of the timestamps of popularity data submissions (see https://github.com/michitux/dokuwiki-popularity-analysis for the code and https://github.com/michitux/dokuwiki-popularity-analysis/raw/master/avg_submissions.pdf for the resulting PDF) shows that there was a huge drop of submissions after the release of DokuWiki 2011-05-25 "Rincewind". Rincewind is the release where automatic popularity data submission was introduced and the actual submission of the data was moved to the server. Somehow during this change something must have been broken. The relation between the release and the drop of submissions gives the impression that the problem is not dokuwiki.org but the popularity plugin.
  
  I suggest the following steps:
  1. find out if there were any other changes that might lead to the problem (including design changes)
  2. test if server-side manual and automatic submission and the fallback to client-side submission works
  
  If we can't find a real source I suggest to separate the actual popularity data submission and the enabling of automatic popularity data submission and to make the actual popularity data submission from the client side. Furthermore we should include in this popularity data the result of some test upload in order to find out what's going wrong on those systems where the server-side submission doesn't work (if that should be the issue).
• 

  2013-06-21 Klap-in

  Had where a Adore Belle installation which didn't auto submit popularity info.
  - Submitting works, since the first time i perform it manually. (It was sent without error, and there was also generated no error file. Of course i don't know how it received. I can share my id so you check)
  (last submit time 05 February 2013 17:23:03 CET)
  - In February, after manual submit i enable autosubmit.
  - In June the last submit was the manual submit of February.
  
  When reading code and trying https://www.dokuwiki.org/indexer#debugging i observed:
  - Firstly, i added some print statements to the popularity plugin action component.
  - Next running [wiki]/lib/exe/indexer.php?id=hoofdpagina&debug=1 shows:
  runIndexer(): started
  Indexer: index for hoofdpagina up to date
  runSitemapper(): started
  runSitemapper(): finished
  sendDigest(): started
  sendDigest(): finished
  
  Sometimes for other pages:
  runIndexer(): started
  Indexer: finished
  But when the Indexer passes, it stucks always at sendDigest().
  
  Looking at the code reveals:
  This code is included in Adore Belle, when i right.
  https://github.com/splitbrain/dokuwiki/blob/cd2f903b3e35c080dc9c1aa0c9f7383573dcae31/lib/exe/indexer.php#L168
  It returns always true, so other tasks aren't executed.
  
  Fixed by this commit later:
  https://github.com/splitbrain/dokuwiki/commit/46a853c3756aa4abe582ccb0de6f4e1f4b2035c4
  
  Function is refactored again later, so todays:
  https://github.com/splitbrain/dokuwiki/blob/master/lib/exe/indexer.php#L172
  
  Testing by letting always return false by sentDigest(), performs directly sending the popularity data. Autosumbit time: 19 June 2013 16:41:46 CEST
  
  So: is the problem now solved?? Or is the chain at another place broken too?
• 

  2013-06-22 Michitux

  I've just had a look at the differences in lib/exe/indexer.php between the releases before Adora Belle and from what I saw the "return true" was only introduced in release 2012-10-13 so the automatic popularity data submission should have worked before. Furthermore this only affected installations with enabled subscriptions (though afaik we don't have any data how many installations that are).
  
  Apart from that this has only affected automatic submissions. All submissions before Rincewind were manual submissions so this doesn't really explain why we see so much less manual submissions. Even if the explanation was that the manual submissions aren't used anymore after enabling automatic submissions we should at least see a last wave of manual submissions after the release of Rincewind when everybody has enabled automatic submissions and thus executed a last manual submission. Furthermore we should see a huge wave of automatic submissions after the update to Weatherwax as the automatic submission should be executed during the first page views after the update. However according to the data I have there is no such wave.
  
  I don't know exactly how the data is saved and what's happening with the ids, i.e. how multiple submissions from the same installation are handled etc. so I can't exactly say what kind of timestamps I have used for my data. But even if these are not individual submissions but just the last submission from each id there should rather be the opposite trend in the data. Even if the data from before Rincewind contains individual submissions while the data after Rincewind contains just one timestamp per installation (unlikely, but the extreme case) I can't imagine that we had so many submissions from the same installations that this huge difference can be explained.
  
  It would really be interesting to plot other data over time, e.g. DokuWiki versions over time. That way we might get a better idea of the changes between the releases. Afaik I currently don't have access to this data.
• 

  2013-07-29 Michitux

  Fixed in DokuWiki in 7d8a6abb and also on the server so a lot more popularity data submissions should show up now.