42 replies to this topic

[AdrianK]

Latest Update (13-Sept): Lorp v1.3 Released - WoT 9.10 Compatible

 

Download Lorp at Codeplex: https://lorp.codeplex.com/

 

Support content *might* be updated & extended in due course.

 

 

Intro 

 

I've built an application (that runs on your local machine) which will extract useful information from your replay files, provide some limited reporting, and let you export them to CSV (from which you can easily dump into Excel to do your own custom reporting.

 

Keep an eye on this OP for latest updates; then next few posts will contain supporting information (documentation, planned feature list, etc). 

 

Download Lorp at Codeplex: https://lorp.codeplex.com/

 

About

 

This is a tool which runs locally on your PC (it doesn't need an internet connection to run).

It allows you to:

1. Locate a notable replay by scanning your replay files and reporting back useful data.
2. Let you export this data to CSV (and then into a tool like Excel where you can analyse the data.
3. Provides some basic (but hopefully interesting & useful) reporting functionality.

Currently a decent range of information is extracted from the replay file but not everything.

 

Crawl Screen: 

 

Spoiler - Screen Shots

Vehicle Performance Report UI (new in v0.2) showing the 1 / 7 / 30 / 60 / 120 Day Vie both as tabular data and charts (shown here): 

 

 

Some of the data provided in the Data View UI:

 

Exporting from the Data View screen  (just select one or more rows and press "e":

 

You can also apply some basic filters over the data: 

 

 

Manage Data Screen:

 

 

 

 

Known Issues

 

• Fields not populated: "Battle Tier", "Platooned"
• Credits earnt not populated for replay version ??? (December 2013)
• Error sometimes occurs when crawling (usually against the current day).  Try crawling without any filters or date limits, cancel after a few files have been crawled and try again; otherwise close and restart.

 

Performance

 

• Lorp uses parallel processing to crawl multiple replay files at once - based on the number of cores available, the more cores you have the faster Lorp will crawl the replays.
• Performance is also dependent on disk-speed.  Running on an Lenovo X1 Carbon laptop (with 4 cores) I can process just over 30 replays a second - but that's with an SSD harddrive (no moving parts).  Based on (admittedly limited) information from others I estimate that using a more common spindle drive might lower that to around 10 replays a second.
• As of Lorp v0.8 the Crawl screen includes both a Cancel button and provides an estimated time to complete; if you cancel the data already crawled will still be available to you.

 

Acknowledgements

 

Gort for use of his icon, see: http://www.forrestwalter.com/icons
Shout out to Stormtroop3r    amade    Maho__Nishizumi_ for providing old replays for testing.

The WoT Asia community, especially those active on the forums.

Edited by AdrianK, 13 September 2015 - 10:27 AM.

[AdrianK]

Content (mostly) current for Lorp version 1.0, schema 0.8.

How to Use 

See the User Guide (part of the application download).

 

Lorp should be intuitive enough to get started with on its own, but here's some help if you need it.

 

 

Spoiler - Installation Guide

Lorp is basically just an exe file (Lorp.exe) which you run.  In order to run successfully it also needs a few other files and a couple of sub-folders.

To install Lorp just download the application zip (usually called Lorp v0.5 Beta - Application.zip) and unzip; double-click on the Lorp.exe.

 

Need to upgrade? see Lorp User Guide (MS Word Doc).

 

 

Spoiler - Basic Running Instructions

Getting Started  

1. Install Lorp.
2. Run Lorp.exe
3. Click on the light blue Crawl Replays button (top left of the main screen).
4. The Crawl Replay Files screen should have opened; in the Replay Folder field type (or preferably copy-and-paste) in the address of the directory that contains your replay files (e.g. C:\Games\WorldofTanks\replays).
5. Click Crawl. 
   • Lorp will provided progress updates as it crawls your replays, this is visible just above the Show Crawl Summary button. 
   • Once the crawl is complete Lorp will tell you via a message in the empty space below the Show Crawl Summary button. E.g.:

Started @ 10/02/2015 6:14:57 p.m.

Complete @ 10/02/2015 6:14:57 p.m.

Duration: 00:00:00.2496004

6. You can now view and/or save the replay data collected via the crawl.
   1. Save replay data by clicking Save Data, it will appear listed in the main (Manage Data) screen.
   2. Data can also be viewed immediately by clicking Show in Data Grid, a new screen will open and display your data.  From this screen you can also export the data to CSV file.  To export to CSV file simply select one or more rows (Ctrl+A will select all) and press e; a save file dialog will appear and you can save the file.

 

 

About Crawl Options

 

Lorp breaks the parsing process into two stages; the first extracts out the Lorp readable data and puts it into a more consistent format (the RawReplayInfo object), Lorp then performs the second stage where it reads these raw replay objects and pulls out the specific data Lorp is looking for.  You can use the Crawl options to save the raw replay objects to disk (as an XML file - stored in the "Raw Data" folder, located in the folder where Lorp is running from).

 

Exporting

 

You can export data to a CSV file anywhere you see a data grid (like the one shown below).

 

Spoiler - Exporting

 

 

1. Crawl your files and open the Data Grid or Vehicle Performance screens.
2. Select all rows by clicking in the empty top-left square (next to "Filename" column), or select any number of rows by clicking on the empty space on the far left of the row.
3. Press the "e" key.

If no rows are selected a error message will be displayed, otherwise the Save File dialog will launch and allow you to save the data selected in CSV format.

 

Using the Show Crawl Summary & Replay Detail functions

 

This feature is more for development support than practical use.

 

Spoiler

 

1. Crawl your files and click on "Show Crawl Summary" - this will list all the replay files parsed, including whether or not the replay file is complete (there's lots of data which does not get saved into the replay file if you exit the game early).
2. Note the #number of a replay that you want to know more about, enter that number into the number input and click "Show Replay Detail (Debug)". A new window will launch which shows the semi-raw replay data broken down into a number of sections, each listing all the data contained in a "[key]": "[value]" pair.

 

 

WoT Replay File Compatibility

 

The tool was initially developed and tested against WoT v9.4 & v9.5 replay files; recent work now provides proven support for version as old as 0.8.9, however the amount of test data available was a little limited so your mileage may vary.

Edited by AdrianK, 19 March 2015 - 11:49 AM.

[AdrianK]

Content (mostly) current for Lorp version 1.0, schema 0.8.

 

Current Features

Spoiler

1. Extraction of game data from World of Tanks replay files; includes options to restrict file crawl by file-name or date created.

2. Displaying extracted data in a sort-able grid, and export of extracted data to CSV files.  data view includes ability to filter by vehicle, map, game-mode and battle-type.

3. Ability to save replay data so you don't have to always crawl it every time.

4. New screen for managing saved replay data, including the ability to merge data-sets together.

5. Ability to save the (semi) raw replay data as an xml file (one file per replay); this feature is to provide a degree of future proofing so that data-items not currently extracted by Lorp (during second stage parsing) can be extracted by later releases (meaning you could potentially delete your actual replay files if you wanted to).  ** Warning - I take no responsibility if you delete your replay files and then discover later that you wish you hadn't, whether it be for Lorp related reasons or not.

6. Added ability for user to define & save replay folder locations (as well as typing in ad-hoc folder addresses).

7. Added basic filters to the Data View screen.

8. Vehicle Performance Reports (with CSV export), see individual vehicle performance by map, battle-type and game-mode, includes damage ratio, credit profit and more.  This allows you to see common key stats for a given vehicle over different time slices - e.g. 1, 7, 30, 60 and 120 days (similar to forum signatures).

9. Map Performance Reports (with CSV export), - similar to the vehicle performance reports, includes breakdown by game-mode and spawn.

10. Support for older replay versions including v0.8.9.

11. SPG Impact Reports - looks at the impact SPGs have on game duration, includes breakdown by SPG count sand tier.

12. WN8 support.

13. WN8 sandpit.

 

Data Collected (v1.0) & Reported On

 

Spoiler

See the Lorp User Guide (part of the application download) which includes a data dictionary.

 

1. Filename
2. Is Complete
3. When Played
4. Vehicle
5. Map
6. Game Mode
7. Battle Type
8. Region
9. WoT Version
10. Player
11. Players Team
12. WinningTeam
13. Result
14. Vehicle ID
15. Class
16. Nation
17. Tier
18. Relative Tier
19. Top Tier
20. WN8
21. Indicative Dmg Rating
22. Current MoE Rating (%)
23. Current MoE Average Damage
24. Damage Dealt
25. Assisted Damage (via Radio)
26. Assisted Damage (via Tracking)
27. Total Assisted Damage
28. Total Damage
29. Damage Received
30. Damage Blocked by Armor
31. Damage Profit
32. Enemies Spotted
33. Enemies Damaged
34. Enemies Killed
35. Survived?
36. Cap Points Captured
37. Cap Points Defended
38. Life Time (Seconds)
39. Battle Duration (Seconds)
40. Survival Duration (%)
41. Raw XP
42. Mastery Level
43. Platooned?
44. Shots Fired
45. Shots Hit
46. Shots Penetrated
47. Hit Ratio
48. Hit / Pen Ratio
49. Overall Pen Ratio
50. Is Premium Account
51. Credits Earnt
52. Compensation Received
53. Total Credits Received
54. Fine Paid
55. Repair Cost
56. Consumable Cost
57. Ammo Cost
58. Total Cost
59. Profit
60. Allied Light Tank
61. Enemy Light Tank
62. Allied Medium Tank
63. Enemy Medium Tank
64. Allied Heavy Tank
65. Enemy Heavy Tank
66. Allied Tank Destroyer
67. Enemy Tank Destroyer
68. Allied SPG
69. Enemy SPG
70. Team 1 Player List
71. Team 2 Player List
72. Data Quality Index (VRD)
73. Personal Missions
74. Arena ID
75. Processing Duration
76. Lorp Source-Data Schema Version

 

Spoiler - Old data schema (v0.7)

 

New / altered fields (as of v0.7) in purple.

 

1. Filename
2. Processing Duration
3. Is Complete
4. When Played
5. Vehicle
6. Map
7. Game Mode
8. Battle Type
9. Battle Tier
10. WoT Version
11. Player
12. Players Team
13. WinningTeam
14. Result
15. Current MoE Rating (%)
16. Current MoE Average Damage
17. Damage Dealt
18. Assisted Damage (via Radio)
19. Assisted Damage (via Tracking)
20. Total Assisted Damage
21. Total Damage
22. Damage Received
23. Damage Blocked by Armor
24. Damage Profit
25. Enemies Spotted
26. Enemies Damaged
27. Enemies Killed
28. Survived?
29. Cap Points Captured
30. Cap Points Defended
31. Life Time (Seconds)
32. Battle Duration (Seconds)
33. Survival Duration (%)
34. Mastery Level
35. Raw XP
36. Platooned?
37. Shots Fired
38. Shots Hit
39. Shots Penetrated
40. Hit Ratio
41. Hit / Pen Ratio
42. Overall Pen Ratio
43. Is Premium Account
44. Credits Earnt
45. Compensation Received
46. Total Credits Received
47. Fine Paid
48. Repair Cost
49. Consumable Cost
50. Ammo Cost
51. Total Cost
52. Profit
53. Vehicle ID
54. Tier
55. Class
56. Nation
57. Top Tier
58. Allied Light Tank
59. Enemy Light Tank
60. Allied Medium Tank
61. Enemy Medium Tank
62. Allied Heavy Tank
63. Enemy Heavy Tank
64. Allied Tank Destroyer
65. Enemy Tank Destroyer
66. Allied SPG
67. Enemy SPG
68. Team 1 Player List
69. Team 2 Player List
70. Data Quality Index (VRD)
71. Lorp Source-Data Schema Version

 

 

Planned Features

Spoiler

Not including general polishing & defect fixing: 

 

• Ability to have Lorp "auto-save" replay data by vehicle, making it easier to manage data and see progress faster.
• Ability to save (not just export) filtered data (in data view screen).
• Ability to crawl raw-replay data.
• Ability to remove (or not crawl) incomplete replays.
• Ability to calculate WN8

 

* Features struck through = delivered in latest version.

* Highlighted Features = planned for next release.

 

 

Change Log

 

(removed - see thread, I post new features in a post when the release is put up on CodePlex).

 

 

 

Edited by AdrianK, 19 March 2015 - 11:49 AM.

[Ezz]

Which replay versions does it support?

[AdrianK]

Ezz, on 06 January 2015 - 04:11 PM, said:

Which replay versions does it support?

 

I'm still posting the content - (wife's calling me, so might be a delay...).

The tool was developed and tested against WoT v9.4 & v9.5 replay files, and appears at first glance to work with v9.3 replay files as well.

 

[thommo_nz]

Nice work...

 

 

1. You might have to exempt temp.wotreplay from processes because of buggers like me who run it when WoT is open. WoT hooks that file.. :|

2. I have a number in there that show a status of 'Replay file incomplete'..  How are those 'counted'?

 

 

[AdrianK]

thommo_nz, on 06 January 2015 - 04:59 PM, said:

Nice work...

 

1. You might have to exempt temp.wotreplay from processes because of buggers like me who run it when WoT is open. WoT hooks that file.. :|

2. I have a number in there that show a status of 'Replay file incomplete'..  How are those 'counted'?

 

Cheers.  yes, the tool will attempt to parse all files.  I plan to add functionality to help filter them out when it comes time to do analytics.

[AdrianK]

Does anyone have a Stronghold, Company and Team battle replay they can give me for testing?  If so please PM me.  Cheers.

 

Thanks Karl.

Edited by AdrianK, 07 January 2015 - 10:52 AM.

[AdrianK]

New Features (v0.2) 

 

1. Added basic filters to the Data View screen.
2. Vehicle Performance Report (with CSV export), see individual vehicle performance by map, battle-type and game-mode, includes damage ratio, credit profit and more.  This allows you to see common key stats for a given vehicle over different time slices - e.g. 1, 7, 30, 60 and 120 days (similar to forum signatures).
3. Simple crawl options (get recent files by name or date filter).
4. Properly skips temp.wotreplay files.

 

The Vehicle Performance UI:

 

 

 

 

[AdrianK]

New Features (v0.3)

 

1. Fixed issue where the crawl attempts to parse the temp.wotreplay file still locked by the WoT client application.
2. Added ability for user to define & save replay folder locations (as well as typing in ad-hoc folder addresses as per v0.1 & 0.2).
3. Added ability to open the currently selected target replay folder from the UI.

Edited by AdrianK, 14 January 2015 - 06:31 PM.

[AdrianK]

New Features (v0.4)

 

Express release (my wifes back! I'll update the above content later).

 

1. New data items:
   • Life Time (in seconds; how long you survived in the game)
   • Battle Duration (in seconds)
   • Survival Duration (as a percentage)
   • Lorp data schema version (schema = definition of what data is collected during the crawl).
2. Ability to save replay data so you don't have to always crawl it every time.
3. New screen for managing saved replay data, including the ability to merge data-sets together.
4. Ability to save the (semi) raw replay data; this feature is to provide a degree of future proofing so that data-items not currently extracted by Lorp can be extracted by later releases (meaning you can delete your replay files if you wanted to).

Edited by AdrianK, 17 January 2015 - 04:32 AM.

[AdrianK]

I'm just working on trying to support older replay versions with the current functionality; so far I've got versions

• 0.9.5 & 0.9.4 covered; 0.9.2 looks consistent with those (so that and 0.9.3 should be ok too).
• 0.8.11 looks consistent with 0.9.5 as well (in terms of the file layout, so they should be ok too.
• 0.8.10 & 0.8.9 is a bit murky though as the way the values are written is different (see below).  I'll be able to process these but will take a bit more work.

 

Block Quote - 0.9.5 format

 "achievementXP": 0, "deathReason": 0, "capturePoints": 0

 

Block Quote - 0.8.10 format

 'crits'p39I0sS'pierced'p40I1sg16I0sS'he_hits'p41I0ss

 

 

 

 

[AdrianK]

"Beta" vs Version 1.0

 

What's the difference? Well whilst still in beta expect that:

1. The schema of the data extracted and saved may change (more than usual, and with less warning).
2. The RawReplayInfo format to "future proof" things may change (it already is in v0.5 as I get to grips with older replay versions).  So don't delete your replay files just yet.

 

[AdrianK]

Fyi, parsing data from all versions going really well, just a matter of finding the time to finish that off.  I've made some adjustments to how I parse all files so need to test I haven't broken coverage of the newer versions of the game.  O hope to have an ETA (or trial version) ready this weekend.

[AdrianK]

Lorp v0.5 Beta Released

 

New features:

1. Now supports older WoT replay file versions* including:
   • 0.9.5
   • 0.9.4
   • 0.9.3
   • 0.9.2
   • 0.9.1
   • 0.9.0
   • 0.8.11
   • 0.8.10
   • 0.8.9
   • Mystery version from February 2013 (version not recorded in replay file - as currently parsed).
2. User Settings screen so you can add / remove favorite replay folder locations for easy crawling.
3. Lorp now opens to the "Manage Data" screen where you can do stuff like merge sets of replay data into one file, for sharing, export or further analysis.
4. Parsing process significantly refactored; you can now parse replays to (semi)-raw state and save them - this is partly to provide future-proofing.

 

Big shout-out to Stormtroop3r, Amade and Maho__Nishizumi_ for providing replays for testing. 

 

* Please note that due to limited test data (and time) your mileage in parsing old replay files may vary - let me know how you get on.

Edited by AdrianK, 26 January 2015 - 12:12 PM.

[AdrianK]

Freebie & Sneak Preview...

 

Q: "How can I find team mates (or adversaries) by name?"

 

Currently (v0.5) Lorp won't show you who all they players were in the replays, however, if you use one of the "save semi-raw replay data" crawl options you can then use Windows Explorer to find those games for you.  For example, below I've located all the replays that had PBKAC players in them; you can just as easily search for players by name.

 

 

 

Sneak Preview: Map Comparison Summary

 

One of the questions that been burning me for a while is knowing more about which maps I need to get better on.  Below is a screenie of a report currently in alpha; this will be more useful once I can factor in the tier of the vehicle played.  I also want to look at the break-down by spawn, but that'll most likely be in a later release.

 

 

[AdrianK]

Lorp v0.6 Beta Released

 

New Features:

1. New Map Performance Reports - includes breakdown by game-mode and spawn.
2. Vehicle & map performance reports now allow multiple selection.
3. Made CSV export filenames more descriptive (when exporting off the vehicle & map performance reports).

 

Spoiler - Map Performance Comparison Report: By game-mode & spawn 

 

 

 

Next release will include adding vehicle tier & class to the core data set allowing you to do analysis on scenarios like "Are games with arty shorter"?

[AdrianK]

Lorp v0.7 Preview

 

Update - I forgot to mention, I've introduced some performance buffs - the crawl speed is almost doubled (even with the additional team & reference data stuff - see below).  Processing of 1200 replays now takes about 35 seconds, was over a minute (that's approx 1 replay every 30th of a second, assuring I'm not too bad@maths).  For the geeks, I'm now using .net 4.5's  Parallel.ForEach.

 

I've pulled in vehicle reference data now, so you'll (eventually) be able to see performance by tier (etc).  It also means I can provide team break-down by vehicle type, etc.  You'll also be able to search (in excel, once you export) for players by name (I haven't included clan names but could add that).

 

Here's what the new data looks like in the data view at the moment. 

 

 

Here's what the team list's look like - tier, class, common vehicle name & player name, sorted by tier: 

 

8 TD JgTig.8 SB360 / 
8 TD JPanther II AdrianK / 
8 Hvy T34 Arex_Strong / 
8 SPG G.W. Tiger P artworkcafe / 
8 Med Panther II SiIenT_WarriOR / 
8 Hvy FCM 50 t PewBooMpeW / 
7 TD SU-152 Ghilgamesh / 
7 Hvy T29 Mnoplive / 
7 Hvy Tiger I 93gyer / 
7 Lt AMX 13 75 SFqymsun / 
7 TD JPanther rae_sia / 
6 TD SU-100 kpj / 
6 Hvy ARL 44 KerryTank / 
-1 Not found uk-GB75_AT7 han77718 / 
-1 Not found uk-GB21_Cromwell NiteTribe / 

 

 

You'll note my reference data collection isn't complete (the "-1's" - I'm working on that) but the good news is that the reference data will be stored as a CSV file with the application so it'll be very easy for you to update as needed.

 

Edited by AdrianK, 05 February 2015 - 06:28 AM.

[AdrianK]

Lorp v0.7 Released

 

• Faster crawl processing (performance on my dev machine has effectively doubled, now processing approx 30 replays a second).
• Added SPG Impact reports (2) to new "Misc Reports" screen.
• Added quick visual to Vehicle Performance screen.
• General polishing & usability enhancements.

 

And other stuff I've forgotten.

[ouchthathurts]

Well, this is frustrating. While trying to debug some problems with my WoT installation last month, I now notice that I accidentally deleted almost three years of replay files going back to 0.8.3. :-(

 

If you need data from 8.4 through 8.7 for testing, however, I do have my NA server replays from those versions (about 2 GB worth) on my DropBox account; PM me if you'd like access to download some or all of it.

Edited by ouchthathurts, 10 February 2015 - 04:31 PM.