Skip to content

Commit 7d434e5

Browse files
author
Josh Freeman
committed
Big clean up of structure and contents
1 parent 74795df commit 7d434e5

25 files changed

+182
-137
lines changed

‎.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
.idea

‎README.md

+32-30
Original file line numberDiff line numberDiff line change
@@ -2,13 +2,21 @@
22

33
This repository is to serve as a place to share data mining information related to Final Fantasy XIV. All findings and knowledge that each member discovers will be provided here for all in the FFXIV Community to read, learn and use.
44

5-
Huge thanks to the community team members involved:
5+
# Documents of Knowledge
66

7-
- [Ioncannon](https://www.reddit.com/user/Ioncannon) from [FFXIV Explorer](http://ffxivexplorer.fragmenterworks.com)
8-
- [Clorifex](https://www.reddit.com/user/Clorifex) from [GarlandTools](http://www.garlandtools.org/db/)
9-
- [Hezkezl](https://www.reddit.com/user/Hezkezl) for being awesome
7+
- [Click here to view Documents on various FFXIV systems](docs/README.md)
8+
9+
# Research and misc files
10+
11+
- [Click here to view various research and resource files](research/README.md)
12+
13+
# CSVs
14+
15+
This repository keeps a record of CSV's extracted via SaintCoinach using the command `rawexd`, this is so they can be easily linked and referenced when datamining, so we have a git history of changes and just to make life easier in some cases :)
1016

11-
## Updating CSV files:
17+
- [Click here to view CSV files](csv/)
18+
19+
### Updating CSV files:
1220

1321
- Download and open **SaintCoinach.cmd**
1422
- Run: `rawexd`
@@ -19,44 +27,38 @@ The idea is to maintain an easy diffing view of what has changed during a patch
1927

2028
It would also be very useful to keep a history of `ex.json` files from SaintCoinach for each patch as the Korean and Chinese versions are on different patches than the Official live client.
2129

30+
### whats the "ex" folder?
2231

23-
## Helping out
32+
Ex contains an archive list of the [SaintCoinach ex.json](https://github.com/ufx/SaintCoinach/blob/master/SaintCoinach/ex.json) file, ths main point was to keep a `ex.json` file per Patch, then if you wanted to extract data from an older patch (such as Korean or Chinese clients) you can hot-swap the `ex.json` file and extract data in the correct format.
2433

25-
Getting started in datamining is quite easy. If you enjoy digging around CSV database files then you've come to the right place! The best place to start would be:
34+
This kind of hot swap does not always work however as new structures are detected that are implemented into SaintCoinach, or you may have systems that expect data in a specific way (eg the great "Tooltip Descriptions Changes" of Patch 4.4)** ...
2635

27-
- Download this repository and start looking at the files in the `/csv` folder
28-
- Download **SaintCoinach** and use either the `SaintCoinach.Cmd` or `Godbert` tools.
29-
- Saint provides releases: https://github.com/ufx/SaintCoinach/releases
36+
You can just grab an old version from git history, which I may switch to linking to instead of keeping copies.
3037

31-
If you find a connection between files or an identification of data, throw up an issue on the repository and it will be sorted :)
3238

33-
- Connection between files could be an number in 1 column that matches the numbers in another column or even another file
34-
- Identification of data could be values in a column that have a meaning and a pattern can be observed (eg ClassJob ID, or Craft Level)
39+
### whats the "uld" folder?
3540

36-
# The tools of the trade
41+
It's an extract of FFXIV UI Elements (images) extracted from SaintCoinach.
3742

38-
### Saint Coinach
43+
---
3944

40-
- Written by: Clorifex and Rogueadyn
41-
- GitHub: https://github.com/ufx/SaintCoinach
42-
- Downloads: https://github.com/ufx/SaintCoinach/releases
43-
- SaintCoinach.Cmd is a tool to rip data (csv, music, icons, maps, etc)
44-
- Godbert is a *viewer* tool to browse data.
45+
### Huge thanks to:
4546

46-
A great tool showing really impressive results, it will allow you to view and export a lot of data. The source is all open for your luxery.
47+
- [Ioncannon](https://www.reddit.com/user/Ioncannon) from [FFXIV Explorer](http://ffxivexplorer.fragmenterworks.com)
48+
- [Clorifex](https://www.reddit.com/user/Clorifex) from [GarlandTools](http://www.garlandtools.org/db/)
49+
- [Hezkezl](https://www.reddit.com/user/Hezkezl) for being awesome
4750

4851

49-
---
52+
### Becoming a dataminer
5053

54+
Getting started in datamining is quite easy. If you enjoy digging around CSV database files then you've come to the right place! The best place to start would be:
5155

52-
**Other cool projects**
56+
- Download this repository and start looking at the files in the `/csv` folder
57+
- Download **SaintCoinach** and use either the `SaintCoinach.Cmd` or `Godbert` tools.
58+
- Saint provides releases: https://github.com/ufx/SaintCoinach/releases
5359

54-
- REST API: http://xivapi.com/
55-
- (PHP) Lodestone Parser: https://github.com/viion/lodestone-php
56-
- (PHP CLI) Libra exporter: https://github.com/viion/libra-php
57-
- (PHP CLI) Lodestone Icon Ripper: https://github.com/viion/lodestone-icons
58-
- (PHP CLI) SaintCoinach CSV Parser: https://github.com/viion/saint-csv-parser
59-
- (C#) Memory Reader: https://github.com/Icehunter/sharlayan
60-
- (NodeJS) Dead Lodestone Parser: https://github.com/viion/lodestone-nodejs
60+
If you find a connection between files or an identification of data, throw up an issue on the repository and it will be sorted :)
6161

62+
- Connection between files could be an number in 1 column that matches the numbers in another column or even another file
63+
- Identification of data could be values in a column that have a meaning and a pattern can be observed (eg ClassJob ID, or Craft Level)
6264

‎research/achievements_requirement_type.md ‎docs/AchievementsRequirementTypes.md

+5-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,8 @@
1-
Achievements link to a "pre-requisite type" that defines it as completed. The list of Types are:
1+
# Achievements Requirement Types
2+
3+
Achievements link to a "pre-requisite type" that defines it as completed.
4+
5+
**This is a known list of types as of: 3rd October, 2018**
26

37
```
48
0: complete specific Legacy thing

‎research/collectable_reward_info.txt ‎docs/CollectibleRewardInfo.md

+3-1
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Collectible Reward Info
2+
13
MasterpieceSupplyDuty is where all of the Collectable information is kept. Unknown how scrips are calculated for base value,
24
but Bonus / High Bonus is calculated very easily, for both EXP and Scrips.
35

@@ -30,7 +32,7 @@ rewards across all classes for the exact same level (assuming an item isn't in d
3032
you can divide it by 1.2 and round up to get the info!) Gathering collectables I'd presume share a similar EXP systme, but I
3133
haven't mapped out those values yet.
3234

33-
Level = EXP
35+
Level = EXP
3436
------------------
3537
Level 51 = 52920
3638
Level 52 = 63360

‎research/companion-app.md ‎docs/CompanionAppApi.md

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
1-
# FFXIV Companion App
1+
# Companion App API
2+
23
- https://eu.finalfantasyxiv.com/companion/
34

45
The FINAL FANTASY XIV companion app has the ability to query the game servers for a specific resource to pull information (market info, retainer items, player items, chat messages, etc). This document will list research and provide logic for obtaining data out of the app as well as quering the companion API.

‎research/emotes.md ‎docs/EmoteLogMessage.md

+6
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,9 @@
1+
# Emote Log Messages
2+
3+
Attempting to debug the emote LogMessages
4+
5+
---
6+
17
100 Standard Emotes 1 0 False
28

39

‎research/fate_exp.md ‎docs/FateExpPoints.md

+2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# FATE Experience Points
2+
13
Fate EXP is still a mystery, research so far:
24

35
Equation: ` fate offset 22 * paramgrow offset 8 + paramgrow offset 8 = exp`

‎research/fate_levels.md ‎docs/FateLevels.md

+2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# FATE Levels
2+
13
The sync level for fates is always +5 of the fate level.
24

35
## Test Data

‎research/icon_paths.md ‎docs/IconPaths.md

File renamed without changes.

‎research/item_actions.md ‎docs/ItemActions.md

File renamed without changes.

‎research/item_stats.md ‎docs/ItemStats.md

+2
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,5 @@
1+
# Item Stats
2+
13
Most of these are kind of obvious, but will add here for safe keeping!
24

35
# Attack speed

‎research/leve_to_npc.md ‎docs/LeveToNpc.md

File renamed without changes.

‎research/map_coordinates.md ‎docs/MapCoordinates.md

+30-42
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,24 @@
11
# Conversions to 2D Map
22

3-
*Code samples in C#*
43

5-
Conversion of a Level coordinate into a 2D map coordinate:
4+
### Version of an in-game coordinate to a 2D map coordinate
5+
6+
**By: Clorifex, slightly modified by: Vekien - In C#**
7+
https://github.com/xivapi/xivapi-mappy/blob/master/Mappy/Helpers/MapHelper.cs
8+
9+
```csharp
10+
public static double ConvertCoordinatesIntoMapPosition(double scale, double offset, double val)
11+
{
12+
val = Math.Round(val, 3);
13+
val = (val + offset) * scale;
14+
return ((41.0 / scale) * ((val + 1024.0) / 2048.0)) + 1;
15+
}
16+
```
17+
18+
### Conversion of a Level coordinate into a 2D map coordinate:
19+
20+
**By: Clorifex - In C#**
21+
622
```csharp
723
private double ToMapCoordinate(double val) {
824
var c = Map.SizeFactor / 100.0;
@@ -12,7 +28,10 @@ private double ToMapCoordinate(double val) {
1228
}
1329
```
1430

15-
Conversion of a FishingSpot coordinate into a 2D map coordinate:
31+
### Conversion of a FishingSpot coordinate into a 2D map coordinate:
32+
33+
**By: Clorifex - In C#**
34+
1635
```csharp
1736
private double ToMapCoordinate(double val) {
1837
var c = TerritoryType.Map.SizeFactor / 100.0;
@@ -21,51 +40,20 @@ private double ToMapCoordinate(double val) {
2140
}
2241
```
2342

24-
# Conversions to Long/Lat (eg Leaflets/Google Maps)
43+
### Conversion of MapPosition to Pixels
2544

26-
*Code samples in PHP*
45+
**By: Vekien - In C#**
46+
https://github.com/xivapi/xivapi-mappy/blob/master/Mappy/Helpers/MapHelper.cs
2747

28-
You can translate the `levels.exd` values to Long/Lat using the `x`, `y` and **scale** would be the maps [size_factor](https://github.com/viion/XIV-Datamining/blob/master/offsets/3.1_list.txt#L679)
29-
30-
```php
31-
//
32-
// Used due to spherical maps, world is round irl :P
33-
//
34-
function mapsRadiansToDegrees($rad)
48+
```csharp
49+
public static int ConvertMapPositionToPixels(double value, double scale)
3550
{
36-
return $rad / (pi() / 180);
37-
}
38-
39-
//
40-
// This code will take the X, Y and map scale from the levels CSV and convert it into a LAT/LONG
41-
// which can be used on leaflet or Google maps or just about any map tool
42-
//
43-
public function mapsTranslateXYZtoLatLong($x, $y, $scale = 100)
44-
{
45-
if ($x == 0 || $y == 0 || $scale == 0) {
46-
return false;
47-
}
48-
49-
$scale = $scale / 100;
50-
51-
$tilesize = 2048 / $scale;
52-
53-
$pixelsPerLonDegree = $tilesize / 360;
54-
$pixelsPerLonRadian = $tilesize / (2 * pi());
55-
56-
$lng = $x / $pixelsPerLonDegree;
57-
$latRadians = $y / -$pixelsPerLonRadian;
58-
$lat = $this->mapsRadiansToDegrees(2 * atan(exp($latRadians)) - pi() / 2);
59-
60-
$coords = [
61-
'x' => $lng,
62-
'y' => $lat,
63-
];
64-
65-
return $coords;
51+
return Convert.ToInt32((value - 1) * 50 * scale);
6652
}
6753
```
6854

55+
----
56+
6957
To translate a `levels.exd` position to an in-game X/Y:
7058

7159
```php

‎docs/MapScales.md

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
# Maps and various scales
2+
3+
> Do not remember what purpose this had...
4+
5+
ID: 31 = 100 = Lower La Noscea = 41.9
6+
ID: 52 = 200 = New Gridania = 21.4
7+
ID: 58 = 300 = The Tam-Tara Deepcroft = 14.6
8+
ID: 359 = 400 = The Navel = 8.6
9+
ID: 733 = 800 = Mizzenmast Inn = 6.0
10+
ID: 2200 = 95 = Coerthas Western Highlands = 44

‎research/quest_exp_reward.md ‎docs/QuestExpReward.md

+4-1
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,7 @@
1-
Quest experience points are generated using a small equation.
1+
# Quest Exp Reward
2+
3+
Quest experience points are generated using a small equation.
4+
25
Why this is done I am not sure as the values always seem static.
36

47
### Formula updated: 28th May, 2018

‎docs/README.md

+66
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
# FFXIV Data-Mining
2+
3+
- Research
4+
- [Achievements Requirement Types](docs/AchievementsRequirementTypes.md)
5+
- [Collectable Reward Info](docs/CollectibleRewardInfo.md)
6+
- [CompanionApp API](docs/CompanionAppApi.md)
7+
- [Emote Log Messages](docs/EmoteLogMessage.md)
8+
- [FATE Experience Points](docs/FateExpPoints.md)
9+
- [FATE Levels](docs/FateLevels.md)
10+
- [Icon Paths](docs/IconPaths.md)
11+
- [Item Actions](docs/ItemActions.md)
12+
- [Item Stats](docs/ItemStats.md)
13+
- [LeveToNpc](docs/LeveToNpc.md)
14+
- [MapCoordinates](docs/MapCoordinates.md)
15+
- [MapScales](docs/MapScales.md)
16+
- [QuestExpReward](docs/QuestExpReward.md)
17+
- [UIColor](docs/UIColor.md)
18+
- [Weather](docs/Weather.md)
19+
20+
- Tools of the trade
21+
- [SaintCoinach](https://github.com/ufx/SaintCoinach) - The #1 datamining tool.
22+
- [TextTools2](https://github.com/liinko/FFXIV_TexTools2) - Extract data, import game mods.
23+
- [Sapphire Server Emulation](https://github.com/SapphireMordred/Sapphire) - FFXIV Server Emulation
24+
- [Machina](https://github.com/ravahn/machina) - Network Packet Reader
25+
- [Sharlayan](https://github.com/FFXIVAPP/sharlayan) - In-Game memory reader.
26+
- [FFXIV MemoryReader](https://github.com/TamanegiMage/FFXIV_MemoryReader) - In-Game memory reader.
27+
- [XIVAPI.com Source Code](https://github.com/xivapi/xivapi.com)
28+
- [LibPomPom](https://github.com/Minoost/libpompom-sharp) - C# code that authenticates with the FFXIV Companion App.
29+
- [3D Web Model Viewer](https://github.com/viion/ffxiv-3d)
30+
- [Sapphire Ubuntu Setup Script](https://github.com/viion-misc/ffxiv-sapphire-setup) - a script to install Sapphire on ubuntu.
31+
- [Custom FFXIV Launcher](https://github.com/xivapi/ffxiv-launcher)
32+
- [FFXIV Mapping Tool](https://github.com/xivapi/xivapi-mappy) - Map Eorzea by grabbing position data of NPCs, Enemies and EObjs.
33+
- [Lodestone Parser](https://github.com/xivapi/lodestone-parser) - PHP library that parses lodestone.
34+
- [ClassJob Icons](https://github.com/xivapi/xivapi-classjobs) - A collection of HQ job class icons.
35+
- [FFXIV Icons](http://xivapi.com/docs/Icons) - Download all sorts of FFXIV related icons.
36+
- [XIVAPI Quick Search](https://chrome.google.com/webstore/detail/xivapi-quick-search/lgefpgdbbmcahllbifniibndmoignmfg) - Search XIVAPI via a chrome extension!
37+
- [REST API](http://xivapi.com/) - A little REST API that has a few bits and bobs.
38+
39+
- Archives
40+
- [Extract of every FFXIV Patch since 2.0](https://github.com/viion/ffxiv-datamining-patches)
41+
42+
- Dead
43+
- [Data Tool](https://github.com/viion-misc/xivdb-data-tool) - A python library that extracts game data.
44+
- [Libra PHP](https://github.com/viion-misc/libra-php) - PHP library that extracts data from the FFXIV Libra app.
45+
46+
- Discords:
47+
- [XIVAPI](https://discord.gg/MFFVHWC)
48+
- [GarlandTools](https://discord.gg/)
49+
- [TeamCraft](https://discord.gg/493Z6pQ)
50+
- [Sapphire](https://discord.gg/xxcdCER)
51+
- [Apkallu Falls](https://discord.gg/RNcV4eY)
52+
53+
Do you have a discord for your FFXIV project? Provide a PR with your invite link added!
54+
55+
---
56+
57+
# Community Members
58+
59+
Notable members that have contributed information in some way, If you're part of the datamining and FFXIV discovery community be sure to add your credit!
60+
61+
- Vekien (xivapi)
62+
- Clorifex (GarlandTools)
63+
- Hezkezl (GamerEscape)
64+
- Icarus Twine (GamerEscape & Blue Mage overlord)
65+
- Miu (TeamCraft)
66+
- Ioncannon (FFXIV Explorer)

‎research/UIColor.md ‎docs/UIColor.md

+9
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,12 @@
1+
# UI Color
2+
3+
### Update 3rd October, 2018
4+
5+
It is believed that:
6+
- 72 = Foreground
7+
- 73 = Glow (not background)
8+
9+
110
In Patch 4.4, SE changed how colours work in tooltips in preperation of new UI "Skins". The logic for displaying colours changed quite drastically but was figured out quickly, here is the process and how it works now:
211

312
## (ID: 174) Bane tooltip

‎research/weather.md ‎docs/Weather.md

+6-1
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,11 @@
1+
# Weather
2+
13
The current weather of a zone is based on the current unix epoch, and a zone weather table (WeatherRate).
4+
25
A forecast target integer is first calculated. This value to the frequency of the zone's weather.
6+
37
When the number is less than the cumulative rate at which a weather pattern occurs, that weather will appear in the zone.
8+
49
For example, in Gridania it's raining 20% of the time, and foggy 10%. It rains when the number is < 20, and foggy when it's >= 20 and < 30, and so on.
510

611
````js
@@ -29,4 +34,4 @@ calculateForecastTarget: function(lDate) {
2934
// 0x64 = 100
3035
return step2 % 100;
3136
}
32-
````
37+
````

‎schema_history/ex.411.json ‎ex/ex.411.json

File renamed without changes.

‎schema_history/ex.420.json ‎ex/ex.420.json

File renamed without changes.

‎schema_history/ex.430.json ‎ex/ex.430.json

File renamed without changes.

‎schema_history/ex.435.json ‎ex/ex.435.json

File renamed without changes.

‎schema_history/ex.436.json ‎ex/ex.436.json

File renamed without changes.

0 commit comments

Comments
 (0)