x/stackalytics
Code Issues Proposed changes

Be more careful crawling the stuff

Browse Source 
Specify credible user agent while retrieving data from URLs.
Also change default member report update to 30 days instead of week.

Partial bug 1361927

Change-Id: I7f956912c2cc3aa2765a20ed193e1e34ecbf7f91
This commit is contained in:
Ilya Shakhat
2014-10-02 18:24:52 +04:00
parent 9f8d82d801
commit 8d08624711
4 changed files with 18 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
stackalytics/processor/mps.py
View File

@@ -12,6 +12,7 @@
# implied.
# See the License for the specific language governing permissions and
# limitations under the License.
import random
import re
import time
@@ -106,5 +107,7 @@ def log(uri, runtime_storage_inst, days_to_update_members, members_look_ahead):
LOG.debug('New member: %s', member['member_id'])
yield member
time.sleep(random.random() * 5)
LOG.debug('Last_member_index: %s', last_member_index)
runtime_storage_inst.set_by_key('last_member_index', last_member_index)