Tactic Testing Process at MrlTactics
Many of you have asked about our testing process, and although I have answered some of you before, I felt that it would be good to write a short article about it, so that I have an URL I can refer to whenever questions about it pop up.
First of all, let us talk about the database we use for testing. I have made the database myself, and I want to explain the reasons behind the choices I have made here.
There are two divisions in the database. The reason for having two divisions is that it helps AI managers from getting fired in the test league.
We have made a test template savefile, so that each test gets the same schedule. This also guarantees that everything in the test is 100% equal for all tactics tested (No room for human error).
The test league consists of 46 teams, ranging from the likes of Barcelona, Real Madrid and Bayern to Los Angeles, Rosenborg and Basel.
We have two test teams in our league, one with CA120 players and one with CA145 players.
These players are real players with adjusted CA (Current Ability). There are enough players in both test teams to fill 8+ first teams, so that every tactic we test has access to the right type of players for that tactic.
Training is set to Balanced, and Match preparation is set to "Tactics Only".
We use the opposition instructions the tactic creator has specified.
All tactics are tested in a plug and play manner. This means that some of the tactics we test could perform better/worse if you follow the instructions from the creator. For example, we do not test home/away tactic sets, or tactics that are based on changing the approach during matches. We only test standalone tactics. Neither do we set manual marking.
Changes in the editor:
- Removed all injuries
- Removed all loans
- Removed all bans
- Removed all media stuff
- Occurrence ratio for all injuries set to 0.000001
- Removed every other competition, team and person in the whole world to increase testing speed
- Removed the need to register players in the league
- Added Transfer Embargo for all clubs
- Modified all players in the league to have 10 in "Important Matches"
- Modified all players in the league to have 20 in "Loyalty"
- Modified all players in the league to have 20 in "Professionalism"
- Modified all players in the league to have 20 in "Adaptability"
- Modified all players in the league to have 20 in "Consistency"
- Modified all players in the league to have 1 in "Injury Proneness"
- All staff in the two test teams have 200 Current Ability
- Removed all PPM's (Player Preferred Moves) for all players in the two test teams.
- Added the German language on all players (so that they can understand each other)
- Made all players retire after season, to avoid unhappiness
- Set all players nationality to the same as the league to avoid homesickness/unhappiness
- Set test teams' "Tactical Attributes" to 10
- Set test teams' reputation to match their expected position in the league.
To sum up the database, it's a custom league with real teams and real players from all over Europe, sort of like Champions League in a normal league setup. The reason I have used real players and real teams is that I want it to resemble a personal save as much as possible, but with added test accuracy (the changes in some player attributes).
Now, onto the testing method. Over the years, I have used every single test method possible, and learnt from my experiences every year. To me, the most important thing about a test league is the ratio of accuracy:time. This means that we must look at the bigger picture rather than simply trying to maximize accuracy for every single match. Every tester knows that testing a tactic for 50 matches is not enough, as the results would vary too much between every test. This is due to the random factors in the game that are unremovable. The only way to increase accuracy in this regard is thus to increase the number of matches played. Think of it like rolling a dice 10 times vs 10.000 times. If you roll a dice 10 times you could be lucky/unlucky, but if you roll it 10.000 times the luck part is gone. This is how casinoes make money!
Onto another problem in most other tactic tests. Due to time limitations, many testers use "instant result" or "simulate match" to test tactics. This is a problem, because the assistant manager often changes the tactic during match (around 30% of the time). We have made a system which uses neither, but that can test tactics in the same amount of time and without the assistant manager interfering with the tactic.
For a long time, we tested manually with this setup, but now, our programmer Luigi has written a software which tests tactics for us automatically, with its own dedicated computer. This has made it possible for us to keep testing tactics even when we have busy workdays etc.
Here is some information about how the software works:
We put the tactic file and a spreadsheet including OI's for that tactic into a google disk folder. The software has auto-search for new files in this folder. If there is a new tactic here, it loads the savegame, loads the chosen tactic from the google disk folder, maximizes the familiarity with the in-game Editor, sets OI's (from the spreadsheet), changes manager, does the same thing for the other test team, starts FM Fitness Coach, loads the list of teams, starts freezing, and then it starts testing. When it is done testing, it saves the game, exports the league table from the game and publishes the results automatically to our site (makes thread, adding test values etc.)
We do not wish to discuss the software in more detail than this, due to it being core to our website's function.
I am sure there are some things I have forgotten to write down.. For FM18, I will write this down WHILE I am making the database! I also try to improve the database every year, so feel free to make suggestions for our FM18 database.
Best wishes to all,