diff --git a/README.md b/README.md index d5b75c4..77fe763 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,12 @@ Welcome to GarPR Development Our dev environment uses vagrant. -we have a CI cycle, with a big test suite, and auto-push to +We have a CI cycle, with a big test suite, and auto-push to production based on jenkins when we push to master and pass all tests -garpr is written using Restful Flask on the backend, with an AngularJS frontend +garpr is written using Restful Flask on the backend, with an AngularJS frontend. -Developers should make changes in a branch, and then make a pull request +Developers should make changes in a branch, and then make a pull request (also see [here](https://github.com/ripgarpr/garpr/blob/master/jenkins/README.md)). Admins or users, submit bug reports on the [issues page](https://github.com/ripgarpr/garpr/issues). @@ -15,10 +15,10 @@ Interested in getting GarPR in your region? Contact one of the devs. Interested in being a dev? Also contact one of us. We have an active slack channel :D Local Development Using Vagrant -======================= +============================== ### Requirements 1. [Vagrant](https://www.vagrantup.com/downloads.html) -2. [VirturalBox](https://www.virtualbox.org/wiki/Downloads) +2. [VirtualBox](https://www.virtualbox.org/wiki/Downloads) 3. 1024 MB of memory ### Setup Steps @@ -54,7 +54,9 @@ The API and webapp will now be started on the VM, and the webapp can be visited To pull in any changes made to the project on the host into the VM, use the command `sync_vm`. This will allow you to use the text/project editors on your host. -1. (Host): Make edits to some files.. +1. (Host): Make edits to some files. 2. (VM): Run the command: `sync_vm` -3. (VM): Restart the system -4. (Host): Vist 192.168.33.10:8000 to view the new changes \ No newline at end of file +3. (VM): Restart the system (often the system will auto-restart; to force restart, type `bash stop.sh` followed by `bash start.sh`). +4. (Host): Visit 192.168.33.10:8000 to view the new changes. + +If stuff goes very wrong (or you would like to restore the initial copy of your db), you can restore the initial state of the Vagrant VM by typing `vagrant destroy` followed by `vagrant up`. diff --git a/add_url.py b/add_url.py new file mode 100644 index 0000000..151c595 --- /dev/null +++ b/add_url.py @@ -0,0 +1,37 @@ +from pymongo import MongoClient +from config.config import Config + +config = Config() +DATABASE_NAME = config.get_db_name() +TOURNAMENTS_COLLECTION_NAME = 'tournaments' +PENDING_TOURNAMENTS_COLLECTION_NAME = 'pending_tournaments' +mongo_client = MongoClient(host=config.get_mongo_url()) + + +tournaments_col = mongo_client[DATABASE_NAME][TOURNAMENTS_COLLECTION_NAME] +pending_tournaments_col = mongo_client[DATABASE_NAME][PENDING_TOURNAMENTS_COLLECTION_NAME] + +tournaments = tournaments_col.find() +pending_tournaments = pending_tournaments_col.find() + +tournaments_col.update({},{"$set": {"url": None}}) +pending_tournaments_col.update({}, {"$set": {"url": None}}) + +for t in tournaments: + if(t['type'] =='challonge' and t['raw'] != ""): + print t['type'], t['name'], "yes" + tournaments_col.update({"_id": t["_id"]},{"$set": {"url": t['raw']['tournament']['tournament']['full_challonge_url']}}) + else: + print t['type'], t['name'] + tournaments_col.update({"_id": t["_id"]},{"$set": {"url": ''}}) + +for x in range(1,5): + print '--------------------------' + +for pt in pending_tournaments: + if(pt['type'] == 'challonge' and pt['raw'] != ""): + print pt['type'], pt['name'], "yes" + pending_tournaments_col.update({"_id": pt["_id"]},{"$set": {"url": pt['raw']['tournament']['tournament']['full_challonge_url']}}) + else: + print t['type'], t['name'] + pending_tournaments_col.update({"_id": pt["_id"]},{"$set": {"url": ''}}) \ No newline at end of file diff --git a/dao.py b/dao.py index e252a77..08f97a0 100644 --- a/dao.py +++ b/dao.py @@ -136,7 +136,6 @@ def get_all_players(self, all_regions=False, include_merged=False): mongo_request['regions'] = {'$in': [self.region_id]} if not include_merged: mongo_request['merged'] = False - return [Player.from_json(p) for p in self.players_col.find(mongo_request).sort([('name', 1)])] diff --git a/jenkins/README.md b/jenkins/README.md index c4fb837..ef389ee 100644 --- a/jenkins/README.md +++ b/jenkins/README.md @@ -6,16 +6,32 @@ To make continuous deployment easy, we've set up Jenkins on the production serve Overview ======== -At any time, there should be two copies of GarPR running in separate environments on the production server. The first environment, the *stage* environment, is intended for testing the most recent build of GarPR. Any update to master on the github repo will cause Jenkins to update stage, run nosetests, and restart the stage environment. Currently the stage environment is accessible at http://www.notgarpr.com:8013 (with the api being served at http://www.notgarpr.com:3013). You can also access the stage environment at http://stage.notgarpr.com. +At any time, there should be two copies of GarPR running in separate environments on the production server. The first environment, the *stage* environment, is intended for testing the most recent build of GarPR. Any update to master on the github repo will cause Jenkins to update stage, run nosetests, and restart the stage environment. Currently the stage environment is accessible at https://www.notgarpr.com:8443 (with the api being served at https://www.notgarpr.com:3013). You can also access the stage environment at https://stage.notgarpr.com. -When you're convinced that the stage copy is working as intended, you can manually tell Jenkins to push the changes to the *prod* environment. This is the version of GarPR that all users will interact with. Currently the prod environment is accessible at http://www.notgarpr.com (with the API being served at http://www.notgarpr.com:3001). +When you're convinced that the stage copy is working as intended, you can manually tell Jenkins to push the changes to the *prod* environment. This is the version of GarPR that all users will interact with. Currently the prod environment is accessible at https://www.notgarpr.com (with the API being served at https://www.notgarpr.com:3001). Using Jenkins ============= +Starting builds through Slack +---------------------------- + +The recommended way to stage/deploy new builds is via Slack commands on our Slack channel. Typing "/stage " anywhere in Slack will prompt Jenkins to run tests on and (if they succeed) stage branch on the stage copy. Typing "/deploy" will deploy the most recent version of master that has successfully been staged to the prod copy. In particular, note that you should "/stage master" before you "/deploy". A typical workflow for deploying a feature should look as follows: + +1. Open branch "featurename" for your new feature. +2. Code your feature in this branch. +3. When the feature is ready, push this branch to GH and open a PR for this feature. +4. Stage this branch on the staging copy by typing "/stage featurename". +5. If step 4 is successful and the feature works fine on stage, merge your PR. +6. Stage the merged copy of master by typing "/stage master". +7. If the staged copy of master looks fine, deploy to prod by typing "/deploy". + +Starting builds through Jenkins +------------------------------- + Currently the Jenkins web interface is being served at www.notgarpr.com:8080. You will need a username and password to log in: ask in Slack for the appropriate credentials. -There are currently two projects in Jenkins, "garpr_stage" and "garpr_prod", corresponding to updating the stage and prod environment. In a project, click "Build Now" on the left menu to manually trigger a build (for "garpr_prod" this is necessary; "garpr_stage" will also be built whenever anything is pushed to master). You can see the currently active builds in the "Build Queue" on the left (or by clicking "Builds"). On the page for any given build, you can see whether it failed or succeeded, along with any console output it may have generated. +There are currently two projects in Jenkins, "garpr_stage" and "garpr_prod", corresponding to updating the stage and prod environment. In a project, click "Build Now" on the left menu to manually trigger a build. You can see the currently active builds in the "Build Queue" on the left (or by clicking "Builds"). On the page for any given build, you can see whether it failed or succeeded, along with any console output it may have generated. Backups ======= diff --git a/model.py b/model.py index d1c4c25..4075809 100644 --- a/model.py +++ b/model.py @@ -177,7 +177,7 @@ def from_json(cls, json_dict): id=json_dict.get('_id', None)) class Tournament(object): - def __init__(self, type, raw, date, name, players, matches, regions, orig_ids=None, id=None): + def __init__(self, type, raw, date, name, players, matches, regions, orig_ids=None, id=None, url=None): ''' :param type: string, either "tio", "challonge", or "smashgg" :param raw: for tio, this is an xml string. for challonge its a dict from string --> string @@ -192,6 +192,7 @@ def __init__(self, type, raw, date, name, players, matches, regions, orig_ids=No self.id = id self.type = type self.raw = raw + self.url = url self.date = date self.name = name self.matches = matches @@ -230,6 +231,8 @@ def get_json_dict(self): if self.id: json_dict['_id'] = self.id + if self.url: + json_dict['url'] = self.url json_dict['type'] = self.type json_dict['raw'] = self.raw @@ -246,7 +249,6 @@ def get_json_dict(self): def from_json(cls, json_dict): if json_dict == None: return None - return cls( json_dict['type'], json_dict['raw'], @@ -256,7 +258,8 @@ def from_json(cls, json_dict): [MatchResult.from_json(m) for m in json_dict['matches']], json_dict['regions'], json_dict.get('orig_ids', None), - id=json_dict['_id'] if '_id' in json_dict else None) + id=json_dict['_id'] if '_id' in json_dict else None, + url=json_dict['url'] if 'url' in json_dict else None) # TODO "sanity checks" @classmethod @@ -284,7 +287,8 @@ def _get_player_id_from_map_or_throw(alias_to_id_map, alias): pending_tournament.matches, pending_tournament.regions, players, - pending_tournament.id) + pending_tournament.id, + pending_tournament.url) # TODO this should go away as we should never build a Tournament straight from a scraper # it should be from a PendingTournament @@ -299,7 +303,7 @@ def from_scraper(cls, type, scraper, alias_to_id_map, region_id): class PendingTournament(object): '''Same as a Tournament, except it uses aliases for players instead of ids. Used during tournament import, before aliases are mapped to player ids.''' - def __init__(self, type, raw, date, name, players, matches, regions, alias_to_id_map=None, id=None): + def __init__(self, type, raw, date, name, players, matches, regions, alias_to_id_map=None, id=None, url=None): ''' :param type: string, either "tio", "challonge", "smashgg" :param raw: for tio, this is an xml string. for challonge its a dict from string --> string. @@ -316,6 +320,7 @@ def __init__(self, type, raw, date, name, players, matches, regions, alias_to_id self.id = id self.type = type self.raw = raw + self.url = url self.date = date self.name = name self.matches = matches @@ -335,6 +340,7 @@ def get_json_dict(self): json_dict['type'] = self.type json_dict['raw'] = self.raw + json_dict['url'] = self.url json_dict['date'] = self.date json_dict['name'] = self.name json_dict['players'] = self.players @@ -358,7 +364,8 @@ def from_json(cls, json_dict): [MatchResult.from_json(m) for m in json_dict['matches']], json_dict['regions'], json_dict['alias_to_id_map'], - id=json_dict['_id'] if '_id' in json_dict else None) + id=json_dict['_id'] if '_id' in json_dict else None, + url=json_dict['url'] if 'url' in json_dict else None) def set_alias_id_mapping(self, alias, id): for mapping in self.alias_to_id_map: @@ -389,7 +396,8 @@ def from_scraper(cls, type, scraper, region_id): scraper.get_name(), scraper.get_players(), scraper.get_matches(), - regions) + regions, + url=scraper.get_url()) #TODO: untested/unused! @classmethod diff --git a/rankings.py b/rankings.py index 8fa51de..497cd10 100644 --- a/rankings.py +++ b/rankings.py @@ -18,6 +18,15 @@ def generate_ranking(dao, now=datetime.now(), day_limit=60, num_tourneys=2): # TODO add a default rating entry when we add it to the map for match in tournament.matches: + + #don't count matches where either player is OOR + winner = dao.get_player_by_id(match.winner) + if not dao.region_id in winner.regions: + continue + loser = dao.get_player_by_id(match.loser) + if not dao.region_id in loser.regions: + continue + if not match.winner in player_id_to_player_map: db_player = dao.get_player_by_id(match.winner) db_player.ratings[dao.region_id] = DEFAULT_RATING diff --git a/requirements.txt b/requirements.txt index 62d7d51..ee240b3 100644 --- a/requirements.txt +++ b/requirements.txt @@ -20,7 +20,7 @@ iso8601==0.1.10 itsdangerous==0.24 lxml==3.3.5 mock==1.0.1 -mongomock==2.0.0 +mongomock==3.5.0 nose==1.3.4 oauth2client==1.3.2 passlib==1.6.5 diff --git a/scraper/challonge.py b/scraper/challonge.py index 422bc4a..29d1ff7 100644 --- a/scraper/challonge.py +++ b/scraper/challonge.py @@ -37,6 +37,9 @@ def get_raw(self): return self.raw_dict + def get_url(self): + return self.get_raw()['tournament']['tournament']['full_challonge_url'] + def get_name(self): return self.get_raw()['tournament']['tournament']['name'].strip() diff --git a/scraper/smashgg.py b/scraper/smashgg.py index e4c6f0b..3383a30 100644 --- a/scraper/smashgg.py +++ b/scraper/smashgg.py @@ -21,12 +21,17 @@ def check_for_200(response): return response class SmashGGScraper(object): - def __init__(self, path): + def __init__(self, path, included_phases): """ :param path: url to go to the bracket """ self.path = path + # DATA STRUCTURE TO INCLUDE PHASES USER WANTS TO IMPORT + self.included_phases = included_phases + for p in self.included_phases: + print p + #GET IMPORTANT DATA FROM THE URL self.event_id = SmashGGScraper.get_tournament_event_id_from_url(self.path) self.name = SmashGGScraper.get_tournament_name_from_url(self.path) @@ -36,8 +41,18 @@ def __init__(self, path): # JSON DUMPED FROM THE API self.event_dict = SmashGGScraper.get_event_dict(self.event_id) - self.group_ids = self.get_group_ids() + + self.group_sets = [] + for phase in self.included_phases: + self.group_sets.append(SmashGGScraper.get_group_ids_from_phase(phase)) + + self.group_ids = [] + for group_set in self.group_sets: + for group_id in group_set: + self.group_ids.append(group_id) + self.group_dicts = [SmashGGScraper.get_group_dict(group_id) for group_id in self.group_ids] + self.group_dicts = [dict for dict in self.group_dicts if dict is not None] #REMOVE EMPTY PHASES FROM IMPORT #DATA STRUCTURES THAT HOLD IMPORTANT THINGS self.get_smashgg_players() @@ -56,6 +71,10 @@ def get_raw(self): return {'event': self.event_dict, 'groups': self.group_dicts} + + def get_url(self): + return self.path + def get_name(self): return self.name @@ -160,6 +179,10 @@ def get_smashgg_matches(self): smashgg_match = SmashGGMatch(round_name, winner_id, loser_id, round_num, best_of) self.matches.append(smashgg_match) + def get_phase_ids(self): + group_ids = [str(group['phaseId']).strip() for group in self.event_dict['entities']['groups']] + return list(set(group_ids)) + def get_group_ids(self): group_ids = [str(group['id']).strip() for group in self.event_dict['entities']['groups']] return list(set(group_ids)) @@ -208,7 +231,10 @@ def get_event_dict(event_id): @staticmethod def get_group_dict(group_id): - return check_for_200(requests.get(GROUP_URL % group_id)).json() + dict = check_for_200(requests.get(GROUP_URL % group_id)).json() + hasSets = dict['entities']['groups']['hasSets'] + if hasSets is True: + return dict @staticmethod def get_event_name(event_id): @@ -222,6 +248,15 @@ def get_phase_bracket_name(phase_id): phase_name = phase_raw['entities']['phase']['name'] return phase_name + @staticmethod + def get_group_ids_from_phase(phase_id): + phase_ids = [] + phase_raw = check_for_200(requests.get(PHASE_URL % phase_id)).json() + groups = phase_raw['entities']['groups'] + for group in groups: + phase_ids.append(group['id']) + return phase_ids + @staticmethod def get_phase_ids(event_id): ids = [] @@ -240,8 +275,6 @@ def get_phasename_id_map(event_id): map[phase_id] = SmashGGScraper.get_phase_bracket_name(phase_id) return map - - class SmashGGPlayer(object): def __init__(self, smashgg_id, entrant_id, name, smash_tag, region, country, state, final_placement): """ @@ -287,6 +320,14 @@ def __init__(self, roundName, winner_id, loser_id, roundNumber, bestOf): self.roundNumber = roundNumber self.bestOf = bestOf +class SmashGGEvent(object): + """ + This is currently unused + """ + def __init__(self, event_id, phase_map): + self.event_id = event_id + self.phase_map = phase_map + class SmashGGException(Exception): def __init__(self, message): self.message = message diff --git a/scraper/tio.py b/scraper/tio.py index 8a55dbe..0324db0 100644 --- a/scraper/tio.py +++ b/scraper/tio.py @@ -14,6 +14,7 @@ def __init__(self, raw, bracket_name): self.text = raw self.soup = BeautifulSoup(self.text, 'xml') + self.url = None # no url for Tio @classmethod def from_file(cls, filepath, bracket_name): @@ -24,6 +25,9 @@ def from_file(cls, filepath, bracket_name): def get_raw(self): return self.text + def get_url(self): + return self.url + def get_name(self): return self.soup.Event.Name.text diff --git a/server.py b/server.py index dbb5ebe..5e7a767 100644 --- a/server.py +++ b/server.py @@ -64,6 +64,9 @@ tournament_put_parser.add_argument('regions', type=list) tournament_put_parser.add_argument('pending', type=bool) +smashGGMap_get_parser = reqparse.RequestParser() +smashGGMap_get_parser.add_argument('bracket_url', type=str) + merges_put_parser = reqparse.RequestParser() merges_put_parser.add_argument('source_player_id', type=str) merges_put_parser.add_argument('target_player_id', type=str) @@ -74,6 +77,7 @@ tournament_import_parser.add_argument('challonge_url', type=str) tournament_import_parser.add_argument('tio_file', type=str) tournament_import_parser.add_argument('tio_bracket_name', type=str) +tournament_import_parser.add_argument('included_phases', type=list) pending_tournament_put_parser = reqparse.RequestParser() pending_tournament_put_parser.add_argument('name', type=str) @@ -201,10 +205,11 @@ def get(self, region): # get all players in all regions elif args['all']: all_players = dao.get_all_players(all_regions=True) - return_dict['players'] = [p.get_json_dict() for p in all_players] + return_dict['players'] = [p.get_json_dict() for p in sorted(all_players, key=lambda player : player.name.lower())] # all players within region else: - return_dict['players'] = [p.get_json_dict() for p in dao.get_all_players()] + print 'test' + return_dict['players'] = [p.get_json_dict() for p in sorted(dao.get_all_players(), key=lambda player : player.name.lower())] convert_object_id_list(return_dict['players']) @@ -353,6 +358,7 @@ def post(self, region): parser.add_argument('type', type=str, location='json') parser.add_argument('data', type=unicode, location='json') parser.add_argument('bracket', type=str, location='json') + parser.add_argument('included_phases', type=list, location='json') args = parser.parse_args() if args['data'] is None: @@ -366,6 +372,7 @@ def post(self, region): type = args['type'] data = args['data'] + included_phases = args['included_phases'] pending_tournament = None try: @@ -379,12 +386,12 @@ def post(self, region): elif type == 'challonge': scraper = ChallongeScraper(data) elif type == 'smashgg': - scraper = SmashGGScraper(data) + scraper = SmashGGScraper(data, included_phases) else: return "Unknown type", 400 pending_tournament = PendingTournament.from_scraper(type, scraper, region) - except: - return 'Scraper encountered an error', 400 + except Exception as ex: + return 'Scraper encountered an error ' + str(ex), 400 if not pending_tournament: return 'Scraper encountered an error', 400 @@ -818,6 +825,14 @@ def get(self, region, id): return return_dict +class SmashGGMappingResource(restful.Resource): + def get(self): + args = smashGGMap_get_parser.parse_args() + url = args['bracket_url'] + + event_id = SmashGGScraper.get_tournament_event_id_from_url(url) + id_map = SmashGGScraper.get_phasename_id_map(event_id) + return id_map class MergeListResource(restful.Resource): def get(self, region): @@ -1015,6 +1030,11 @@ def add_cors(resp): api.add_resource(PendingTournamentResource, '//pending_tournaments/') api.add_resource(FinalizeTournamentResource, '//tournaments//finalize') +# THIS CALL TAKES A SMASHGG BRACKET URL AND RETURNS A MAP OF BRACKETS IN THE SMASHGG EVENT TO THEIR PHASE IDS +# THIS IS USEFUL FOR DISPLAYING THE INFORMATION TO THE USER SO THEY CAN CHOOSE ANY BRACKETS THEY DO NOT WISH +# TO IMPORT +api.add_resource(SmashGGMappingResource, '/smashGgMap') + api.add_resource(PendingTournamentListResource, '//tournaments/pending') api.add_resource(RankingsResource, '//rankings') diff --git a/start_windows.sh b/start_windows.sh new file mode 100644 index 0000000..b13e092 --- /dev/null +++ b/start_windows.sh @@ -0,0 +1,24 @@ +#!/bin/bash +source config/config.ini +if [[ "$(ps aux | grep mongo)" == *"mongod"* ]] + then + echo "mongod is already running" + else + echo "starting mongo" + mongod & +fi +if [[ "$(ps aux | grep server.py)" == *"python server.py"* ]] + then + echo "backend is already running" + else + echo "starting backend" + python server.py $api_port True & +fi +if [[ "$(ps aux | grep SimpleHTTPServer)" == *"python -m SimpleHTTPServer"* ]] + then + echo "frontend is already running" + else + echo "starting frontend" + pushd webapp; python -m SimpleHTTPServer $web_port; popd & +fi +echo "everything started, try http://localhost:$web_port" diff --git a/test/test_model.py b/test/test_model.py index 00527c4..044f410 100644 --- a/test/test_model.py +++ b/test/test_model.py @@ -396,6 +396,7 @@ def setUp(self): self.players = [self.player_1.name, self.player_2.name, self.player_3.name, self.player_4.name] self.matches = [self.match_1, self.match_2] self.regions = ['norcal', 'texas'] + self.url = 'http://challonge.com/test' self.pending_tournament_json_dict = { '_id': self.id, @@ -406,17 +407,18 @@ def setUp(self): 'players': self.players, 'matches': [m.get_json_dict() for m in self.matches], 'regions': self.regions, - 'alias_to_id_map': self.alias_to_id_map + 'alias_to_id_map': self.alias_to_id_map, + 'url': self.url } self.pending_tournament = PendingTournament( - self.type, self.raw, self.date, self.name, self.players, self.matches, self.regions, alias_to_id_map=self.alias_to_id_map, id=self.id) + self.type, self.raw, self.date, self.name, self.players, self.matches, self.regions, alias_to_id_map=self.alias_to_id_map, id=self.id, url=self.url) def test_get_json_dict(self): self.assertEquals(self.pending_tournament.get_json_dict(), self.pending_tournament_json_dict) def test_get_json_dict_missing_id(self): self.pending_tournament = PendingTournament( - self.type, self.raw, self.date, self.name, self.players, self.matches, self.regions, alias_to_id_map=self.alias_to_id_map) + self.type, self.raw, self.date, self.name, self.players, self.matches, self.regions, alias_to_id_map=self.alias_to_id_map, url=self.url) del self.pending_tournament_json_dict['_id'] self.assertEquals(self.pending_tournament.get_json_dict(), self.pending_tournament_json_dict) diff --git a/test/test_rankings.py b/test/test_rankings.py index 3dda6bd..bc2a979 100644 --- a/test/test_rankings.py +++ b/test/test_rankings.py @@ -122,13 +122,13 @@ def test_generate_rankings(self): self.assertAlmostEquals(self.dao.get_player_by_id(self.player_2_id).ratings['norcal'].trueskill_rating.sigma, 6.464, delta=delta) self.assertAlmostEquals(self.dao.get_player_by_id(self.player_3_id).ratings['norcal'].trueskill_rating.mu, - 31.230, delta=delta) + 2, delta=delta) #changing this b/c of new in regionon only stuff, lol self.assertAlmostEquals(self.dao.get_player_by_id(self.player_3_id).ratings['norcal'].trueskill_rating.sigma, - 6.523, delta=delta) + 3, delta=delta) self.assertAlmostEquals(self.dao.get_player_by_id(self.player_4_id).ratings['norcal'].trueskill_rating.mu, - 18.770, delta=delta) + 25, delta=delta) self.assertAlmostEquals(self.dao.get_player_by_id(self.player_4_id).ratings['norcal'].trueskill_rating.sigma, - 6.523, delta=delta) + 8.333, delta=delta) self.assertAlmostEquals(self.dao.get_player_by_id(self.player_5_id).ratings['norcal'].trueskill_rating.mu, 29.396, delta=delta) self.assertAlmostEquals(self.dao.get_player_by_id(self.player_5_id).ratings['norcal'].trueskill_rating.sigma, @@ -148,7 +148,8 @@ def test_generate_rankings(self): ranking_list = ranking.ranking # the ranking should not have any excluded players - self.assertEquals(len(ranking_list), 4) + self.assertEquals(len(ranking_list), 3) + entry = ranking_list[0] self.assertEquals(entry.rank, 1) @@ -162,13 +163,15 @@ def test_generate_rankings(self): entry = ranking_list[2] self.assertEquals(entry.rank, 3) - self.assertEquals(entry.player, self.player_4_id) - self.assertAlmostEquals(entry.rating, -.800, delta=delta) + self.assertEquals(entry.player, self.player_2_id) + self.assertAlmostEquals(entry.rating, -1.349, delta=delta) + ''' entry = ranking_list[3] self.assertEquals(entry.rank, 4) - self.assertEquals(entry.player, self.player_2_id) + self.assertEquals(entry.player, self.player_3_id) self.assertAlmostEquals(entry.rating, -1.349, delta=delta) + ''' # players that only played in the first tournament will be excluded for inactivity def test_generate_rankings_excluded_for_inactivity(self): @@ -179,19 +182,21 @@ def test_generate_rankings_excluded_for_inactivity(self): ranking = self.dao.get_latest_ranking() ranking_list = ranking.ranking - self.assertEquals(len(ranking_list), 3) + self.assertEquals(len(ranking_list), 2) entry = ranking_list[0] self.assertEquals(entry.rank, 1) self.assertEquals(entry.player, self.player_1_id) self.assertAlmostEquals(entry.rating, 6.857, delta=delta) + ''' entry = ranking_list[1] self.assertEquals(entry.rank, 2) - self.assertEquals(entry.player, self.player_4_id) - self.assertAlmostEquals(entry.rating, -.800, delta=delta) + self.assertEquals(entry.player, self.player_5_id) + self.assertAlmostEquals(entry.rating, -.800, delta=delta, msg="" + str(entry.player)) + ''' - entry = ranking_list[2] - self.assertEquals(entry.rank, 3) + entry = ranking_list[1] + self.assertEquals(entry.rank, 2) self.assertEquals(entry.player, self.player_2_id) self.assertAlmostEquals(entry.rating, -1.349, delta=delta) diff --git a/test/test_scraper/test_smashgg.py b/test/test_scraper/test_smashgg.py index 025adca..072b158 100644 --- a/test/test_scraper/test_smashgg.py +++ b/test/test_scraper/test_smashgg.py @@ -6,6 +6,7 @@ TEST_URL_1 = 'https://smash.gg/tournament/htc-throwdown/brackets/10448/2096/6529' TEST_URL_2 = 'https://smash.gg/tournament/tiger-smash-4/brackets/11097/21317/70949' TEST_URL_3 = 'https://smash.gg/tournament/ceo-2016/brackets/11789/45259/150418' +TEST_URL_4 = 'https://smash.gg/tournament/nebulous-prime-melee-47/brackets/14172/49705/164217' TEST_DATA1 = os.path.abspath('test' + os.sep + 'test_scraper' + os.sep + 'data' + os.sep + 'smashgg.json') TEST_DATA2 = os.path.abspath('test' + os.sep + 'test_scraper' + os.sep + 'data' + os.sep + 'smashgg2.json') TEST_EVENT_ID_1 = 10448 @@ -34,6 +35,9 @@ class TestSmashGGScraper(unittest.TestCase): def setUp(self): self.tournament1 = TestSmashGGScraper.tournament1 self.tournament2 = TestSmashGGScraper.tournament2 + #self.tournament3 = TestSmashGGScraper.tournament3 + self.tournament4 = TestSmashGGScraper.tournament4 + #self.excluded_phases3 = [49706] #self.tournament3 = SmashGGScraper(TEST_URL_3) #list = self.tournament3.get_matches() #print 'hello' @@ -43,13 +47,17 @@ def setUp(self): def setUpClass(cls): print 'Pulling tournaments from smash.gg ...' super(TestSmashGGScraper, cls).setUpClass() - cls.tournament1 = SmashGGScraper(TEST_URL_1) - cls.tournament2 = SmashGGScraper(TEST_URL_2) + cls.tournament1 = SmashGGScraper(TEST_URL_1, [1511, 2095, 2096]) + cls.tournament2 = SmashGGScraper(TEST_URL_2, [3930, 21317]) + #cls.tournament3 = SmashGGScraper(TEST_URL_3, []) + cls.tournament4 = SmashGGScraper(TEST_URL_4, [49153, 49705]) def tearDown(self): self.tournament1 = None self.tournament2 = None + #self.tournament3 = None + self.tournament4 = None @unittest.skip('skipping test_get_raw1 until api is complete') def test_get_raw1(self): @@ -81,7 +89,7 @@ def test_get_raw_sub(self): self.assertTrue('event' in raw) self.assertTrue('groups' in raw) - self.assertEqual(len(raw['groups']), 10) + self.assertEqual(len(raw['groups']), 9) entrants = raw['event']['entities']['entrants'] for entrant in entrants: @@ -110,7 +118,7 @@ def test_get_matches(self): print mango_count self.assertEqual(2, mango_count, msg="mango didnt get double elim'd?") - self.assertEquals(len(self.tournament2.get_matches()), 436) + self.assertEquals(len(self.tournament2.get_matches()), 361) # spot check that Druggedfox was only in 5 matches, and that he won all of them sami_count = 0 for m in self.tournament2.get_matches(): @@ -164,4 +172,8 @@ def test_get_phase_name(self): def test_get_phasename_id_map(self): self.assertEqual(len(SmashGGScraper.get_phasename_id_map(TEST_EVENT_ID_1)), 3) - self.assertEqual(len(SmashGGScraper.get_phasename_id_map(TEST_EVENT_ID_2)), 3) \ No newline at end of file + self.assertEqual(len(SmashGGScraper.get_phasename_id_map(TEST_EVENT_ID_2)), 3) + + def test_included_phases(self): + self.assertEqual(len(self.tournament2.group_dicts), 9) + self.assertEqual(len(self.tournament4.group_dicts), 9) \ No newline at end of file diff --git a/test/test_server.py b/test/test_server.py index 4533285..8b259d0 100644 --- a/test/test_server.py +++ b/test/test_server.py @@ -19,61 +19,100 @@ TEXAS_FILES = [('test/data/texas1.tio', 'singles'), ('test/data/texas2.tio', 'singles')] NORCAL_PENDING_FILES = [('test/data/pending1.tio', 'bam 6 singles')] -NORCAL_REGION_NAME = 'norcal' -TEXAS_REGION_NAME = 'texas' - +TEMP_DB_NAME = 'garpr_test_tmp' + +def _import_file(f, dao): + scraper = TioScraper.from_file(f[0], f[1]) + _import_players(scraper, dao) + player_map = dao.get_player_id_map_from_player_aliases(scraper.get_players()) + dao.insert_tournament(Tournament.from_scraper('tio', scraper, player_map, dao.region_id)) + +def _import_players(scraper, dao): + for player in scraper.get_players(): + db_player = dao.get_player_by_alias(player) + if db_player is None: + db_player = Player( + player, + [player.lower()], + {dao.region_id: TrueskillRating()}, + [dao.region_id]) + dao.insert_player(db_player) class TestServer(unittest.TestCase): + @classmethod + def setUpClass(cls): + mongo_client = mongomock.MongoClient() + + norcal_region = Region('norcal', 'Norcal') + texas_region = Region('texas', 'Texas') + Dao.insert_region(norcal_region, mongo_client) + Dao.insert_region(texas_region, mongo_client) + + norcal_dao = Dao('norcal', mongo_client=mongo_client) + texas_dao = Dao('texas', mongo_client=mongo_client) + + for f in NORCAL_FILES: + _import_file(f, norcal_dao) + + for f in TEXAS_FILES: + _import_file(f, texas_dao) + + for f in NORCAL_PENDING_FILES: + scraper = TioScraper.from_file(f[0], f[1]) + norcal_dao.insert_pending_tournament(PendingTournament.from_scraper('tio', scraper, norcal_dao.region_id)) + + now = datetime(2014, 11, 1) + rankings.generate_ranking(norcal_dao, now=now) + rankings.generate_ranking(texas_dao, now=now) + + user_id = 'asdf' + user_full_name = 'full name' + user_admin_regions = ['norcal', 'nyc'] + user = User(user_id, user_admin_regions, user_full_name, 0, 0) + norcal_dao.insert_user(user) + + users_col = mongo_client[DATABASE_NAME][USERS_COLLECTION_NAME] + salt = base64.b64encode(os.urandom(16)) + hashed_password = base64.b64encode(hashlib.pbkdf2_hmac('sha256', 'rip', salt, ITERATION_COUNT)) + gar = User(None, 'norcal', 'gar', salt, hashed_password) + users_col.insert(gar.get_json_dict()) + + # store current mongo db instead of rerunning every time + # (unfortunately mongomock doesn't implement copydb) + cls.mongo_data = {} + for coll in mongo_client[DATABASE_NAME].collection_names(): + cls.mongo_data[coll] = list(mongo_client[DATABASE_NAME][coll].find()) + def setUp(self): self.mongo_client_patcher = patch('server.mongo_client', new=mongomock.MongoClient()) self.mongo_client = self.mongo_client_patcher.start() + + # copy data from globals + # faster than loading every time + for coll, data in TestServer.mongo_data.items(): + print coll, data + if data: + self.mongo_client[DATABASE_NAME][coll].insert_many(TestServer.mongo_data[coll]) + server.app.config['TESTING'] = True self.app = server.app.test_client() self.norcal_region = Region('norcal', 'Norcal') self.texas_region = Region('texas', 'Texas') - Dao.insert_region(self.norcal_region, self.mongo_client) - Dao.insert_region(self.texas_region, self.mongo_client) - self.norcal_dao = Dao(NORCAL_REGION_NAME, mongo_client=self.mongo_client) + self.norcal_dao = Dao('norcal', mongo_client=self.mongo_client) self.assertIsNotNone(self.norcal_dao) - self.texas_dao = Dao(TEXAS_REGION_NAME, mongo_client=self.mongo_client) + self.texas_dao = Dao('texas', mongo_client=self.mongo_client) self.assertIsNotNone(self.texas_dao) - self._import_files() - self._create_users(self.mongo_client) - - now = datetime(2014, 11, 1) - rankings.generate_ranking(self.norcal_dao, now=now) - rankings.generate_ranking(self.texas_dao, now=now) - self.user_id = 'asdf' self.user_full_name = 'full name' self.user_admin_regions = ['norcal', 'nyc'] self.user = User(self.user_id, self.user_admin_regions, self.user_full_name, 0, 0) - self.norcal_dao.insert_user(self.user) self.users_col = self.mongo_client[DATABASE_NAME][USERS_COLLECTION_NAME] self.sessions_col = self.mongo_client[DATABASE_NAME][SESSIONS_COLLECTION_NAME] - - def _import_files(self): - for f in NORCAL_FILES: - scraper = TioScraper.from_file(f[0], f[1]) - self._import_players(scraper, self.norcal_dao) - player_map = self.norcal_dao.get_player_id_map_from_player_aliases(scraper.get_players()) - self.norcal_dao.insert_tournament(Tournament.from_scraper('tio', scraper, player_map, self.norcal_dao.region_id)) - - for f in TEXAS_FILES: - scraper = TioScraper.from_file(f[0], f[1]) - self._import_players(scraper, self.texas_dao) - player_map = self.texas_dao.get_player_id_map_from_player_aliases(scraper.get_players()) - self.texas_dao.insert_tournament(Tournament.from_scraper('tio', scraper, player_map, self.texas_dao.region_id)) - - for f in NORCAL_PENDING_FILES: - scraper = TioScraper.from_file(f[0], f[1]) - self.norcal_dao.insert_pending_tournament(PendingTournament.from_scraper('tio', scraper, self.norcal_dao.region_id)) - def _import_players(self, scraper, dao): for player in scraper.get_players(): db_player = dao.get_player_by_alias(player) @@ -85,13 +124,6 @@ def _import_players(self, scraper, dao): [dao.region_id]) dao.insert_player(db_player) - def _create_users(self, mongo_client): - salt = base64.b64encode(os.urandom(16)) #more bytes of randomness? i think 16 bytes is sufficient for a salt - # does this need to be encoded before its passed into hashlib? - hashed_password = base64.b64encode(hashlib.pbkdf2_hmac('sha256', 'rip', salt, ITERATION_COUNT)) - self.users_col = mongo_client[DATABASE_NAME][USERS_COLLECTION_NAME] - gar = User(None, 'norcal', 'gar', salt, hashed_password) - self.users_col.insert(gar.get_json_dict()) @@ -310,6 +342,7 @@ def test_get_tournament_list_include_pending(self, mock_get_user_from_request): tournament = tournaments_list[i] tournament_from_db = dao.get_tournament_by_id(ObjectId(tournament['id'])) expected_keys = set(['id', 'name', 'date', 'regions', 'pending']) + print tournament.keys() self.assertEquals(set(tournament.keys()), expected_keys) self.assertEquals(tournament['id'], str(tournament_from_db.id)) self.assertEquals(tournament['name'], tournament_from_db.name) @@ -320,7 +353,7 @@ def test_get_tournament_list_include_pending(self, mock_get_user_from_request): # the 3rd tournament should be a pending tournament pending_tournament = tournaments_list[2] pending_tournament_from_db = dao.get_pending_tournament_by_id(ObjectId(pending_tournament['id'])) - expected_keys = set(['id', 'name', 'date', 'regions', 'pending']) + expected_keys = set(['id', 'name', 'date', 'regions', 'pending', 'url']) self.assertEquals(set(pending_tournament.keys()), expected_keys) self.assertEquals(pending_tournament['id'], str(pending_tournament_from_db.id)) self.assertEquals(pending_tournament['name'], pending_tournament_from_db.name) @@ -744,7 +777,7 @@ def test_get_tournament_pending(self,mock_get_user_from_request): data = self.app.get('/norcal/tournaments/' + str(pending_tournament.id)).data json_data = json.loads(data) - self.assertEquals(len(json_data.keys()), 8) + self.assertEquals(len(json_data.keys()), 9) self.assertEquals(json_data['id'], str(pending_tournament.id)) self.assertEquals(json_data['name'], 'bam 6 - 11-8-14') self.assertEquals(json_data['type'], 'tio') diff --git a/tournament_import_api.md b/tournament_import_api.md index 7dedbf5..5f50c1b 100644 --- a/tournament_import_api.md +++ b/tournament_import_api.md @@ -2,7 +2,7 @@ ##Upload bracket: - `POST /[region]/tournaments/` - - Body: Tourney name; Bracket type; Challonge link or (TIO file contents and bracket name) + - Body: Tourney name; Bracket type; SmashGG Bracket link, Challonge link or (TIO file contents and bracket name) - Create `PendingTournament` from relevant `Scraper`; save this in the `pending_tournaments` collection - return success diff --git a/tournament_import_service.py b/tournament_import_service.py index ce832e1..d39e6ee 100644 --- a/tournament_import_service.py +++ b/tournament_import_service.py @@ -47,8 +47,8 @@ def import_tournament_from_challonge(region, path, name, dao): return dao.insert_pending_tournament(pending) -def import_tournament_from_smashgg(region, path, name, dao): - scraper = SmashGGScraper(path) +def import_tournament_from_smashgg(region, path, included_phases, name, dao): + scraper = SmashGGScraper(path, included_phases) pending = PendingTournament.from_scraper('smashgg', scraper, region) if name: pending.name = name diff --git a/webapp/about.html b/webapp/about.html index 2c16c94..b4cba79 100644 --- a/webapp/about.html +++ b/webapp/about.html @@ -18,7 +18,7 @@

Where can I find out more about how Trueskill works?

Trueskill Python implementation (the one used by GAR PR)

What tournament programs do you support?

-

Tio or Challonge (with SmashGG in the works)

+

Tio, Challonge, or SmashGG

How do you handle inactive players?

A region determines activity criteria, which are X tournaments in the past Y days. If a player becomes inactive they become hidden from the rankings list. They will still maintain the same rating they had before they became inactive. Once they enter enough tournaments to become active again, their old rating will be used to calculate their new rating and they will reappear on the rankings. Note that there is no rating decay of any kind.

How do you handle the same player entering multiple tournaments with different tags?

diff --git a/webapp/import_tournament_modal.html b/webapp/import_tournament_modal.html index fcb40ce..d76d465 100644 --- a/webapp/import_tournament_modal.html +++ b/webapp/import_tournament_modal.html @@ -34,11 +34,24 @@

Import Tournament

-

Enter the URL to your SmashGG bracket (e.g. https://smash.gg/tournament/genesis-3/brackets/10617/3860/15833):

+

Enter the URL to your SmashGG bracket (e.g. https://smash.gg/tournament/genesis-3/brackets/10617/3860/15833) +
(One bracket will suffice. You will see all the brackets from your event populate below):

- + +
+

+
    +
  • + + {{ bracket.name }} ( {{ bracket.id }} ) +
    • +
+

There was an error. Please make sure everything is spelled correctly and try again.

diff --git a/webapp/index.html b/webapp/index.html index de065d6..4b9720d 100644 --- a/webapp/index.html +++ b/webapp/index.html @@ -23,7 +23,7 @@