Skip to content

Commit 01b5e5a

Browse files
vekienvekien
committed
2 parents 68896b2 + 09d9d77 commit 01b5e5a

10 files changed

+468
-56
lines changed

‎README.md

+11
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,17 @@ Huge thanks to the community team members involved:
88
- [Clorifex](https://www.reddit.com/user/Clorifex) from [GarlandTools](http://www.garlandtools.org/db/)
99
- [Hezkezl](https://www.reddit.com/user/Hezkezl) for being awesome
1010

11+
## Updating CSV files:
12+
13+
- Download and open **SaintCoinach.cmd**
14+
- Run: `rawexd`
15+
- Copy all the contents in the folder: `<date>/rawexd/**` to the `/csv` folder
16+
- Push a PR
17+
18+
The idea is to maintain an easy diffing view of what has changed during a patch
19+
20+
It would also be very useful to keep a history of `ex.json` files from SaintCoinach for each patch as the Korean and Chinese versions are on different patches than the Official live client.
21+
1122

1223
## Helping out
1324

‎research/README.md

+67
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,67 @@
1+
# Research
2+
3+
This folder contains various research on the FFXIV database structure, file structure and various formulas and the craziness that comes with FFXIV :)
4+
5+
### Achievements
6+
7+
- Requirement Types: https://github.com/viion/ffxiv-datamining/blob/master/research/achievements_requirement_type.md
8+
9+
### Collectables
10+
11+
- https://github.com/viion/ffxiv-datamining/blob/master/research/collectable_reward_info.txt
12+
13+
### Emotes
14+
15+
- https://github.com/viion/ffxiv-datamining/blob/master/research/emotes.md
16+
17+
### DevNPCs
18+
19+
This really needs maintaining, there is a bunch of NPC's which are "dev" specific, you do not find them in game
20+
21+
- https://github.com/viion/ffxiv-datamining/blob/master/research/devnpcs.csv
22+
23+
24+
### F.A.T.Es
25+
26+
- EXP: https://github.com/viion/ffxiv-datamining/blob/master/research/fate_exp.md
27+
- Levels: https://github.com/viion/ffxiv-datamining/blob/master/research/fate_levels.md
28+
29+
### Icons
30+
31+
- https://github.com/viion/ffxiv-datamining/blob/master/research/icon_paths.md
32+
33+
### Items
34+
35+
- Stats: https://github.com/viion/ffxiv-datamining/blob/master/research/item_stats.md
36+
37+
### Leves
38+
39+
- Leve to NPC: https://github.com/viion/ffxiv-datamining/blob/master/research/leve_to_npc.md
40+
41+
### Maps
42+
43+
- Scales: https://github.com/viion/ffxiv-datamining/blob/master/research/map_scales.md
44+
- Coordinate conversion: https://github.com/viion/ffxiv-datamining/blob/master/research/map_coordinates.md
45+
46+
### Quests
47+
48+
- EXP Formula: https://github.com/viion/ffxiv-datamining/blob/master/research/quest_exp_reward.md
49+
50+
### Weather
51+
52+
- https://github.com/viion/ffxiv-datamining/blob/master/research/weather.md
53+
54+
55+
---
56+
57+
58+
# File structure research
59+
60+
**Huge thank you to FFXIV Explorer Ioncannon for providing these**
61+
62+
- EXDF: https://github.com/viion/ffxiv-datamining/blob/master/research/explorer_exdf_files
63+
- EXHF: https://github.com/viion/ffxiv-datamining/blob/master/research/explorer_exhf_files
64+
- Macro Dat: https://github.com/viion/ffxiv-datamining/blob/master/research/explorer_macro_dat_file
65+
- SCD: https://github.com/viion/ffxiv-datamining/blob/master/research/explorer_scd_files
66+
- SqPack Dat: https://github.com/viion/ffxiv-datamining/blob/master/research/explorer_sqpack_dat_files.txt
67+
- SqPack Index: https://github.com/viion/ffxiv-datamining/blob/master/research/explorer_sqpack_index_files.txt

‎research/explorer_exdf_files

+70
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
#### EXDF RESEARCH #### By Ioncannon
2+
3+
Current research on EXDF files
4+
5+
!!!NOTE: FILE IS IN BIG ENDIAN FORM!!!
6+
7+
===HEADER=== (Header is 32 bytes, padded with 0s)
8+
0x000: Signature Int32; "EXDF"
9+
0x004: Version Short; Seems to be always 0x0002
10+
0x006: Unknown Short;
11+
0x008: Offset Table Size Int32; Size of the offset table (below)
12+
0x00C: Data Section Size Int32; Size of the string data section
13+
0x00F: Padding ;Up to 0x1F
14+
15+
===OFFSET TABLE=== (starts 0x1F(32) bytes from 0x00)
16+
0x000: Index Int32; These continue on from page to page, not starting at 0 in every file.
17+
0x004: Offset Int32; Points to a string segment in the data section
18+
19+
===DATA SECTION=== (starts after [Offset Table] or [Offset Table Size] * 0x08 + 0x1F) THIS IS MOST LIKELY DEFINED IN EXH file.
20+
0x000: Size of chunk Int32; Starting after this header
21+
0x004: Unknown Short;
22+
0x006: Data Chunk
23+
String Chunk ;After Data Chunk, look in EXH for size of data chunk to skip
24+
25+
~~~~~DEFINED IN EXH FILE~~~~~~~~~~~~
26+
27+
------Data Types-----
28+
29+
>=0x19: bitflags *
30+
31+
0xb: 4 ints packed in 64 bits
32+
0x9: float
33+
0x7: uint
34+
0x6: int
35+
0x5: ushort
36+
0x4: short
37+
0x3: ubyte
38+
0x2: byte
39+
0x1: bool
40+
0x0: string
41+
42+
*If greater or equal to 0x19, subtract datatype by 0x19. This value is the position of the bit in the data
43+
at the given offset. If bit == 1, true else false.
44+
45+
------Variables------
46+
47+
Special variables can be within strings. The follow the format:
48+
49+
0x00: Start Marker, always 0x02
50+
0x01: Type
51+
0x02: Size of payload including End Marker
52+
0x03: Payload
53+
0x--: End marker, always 0x03
54+
55+
Known Types:
56+
57+
0x13: Color Change
58+
0x20: Value Amount?
59+
0x27: Name Start, (0x01 0x01 0x01 0x01)
60+
0x27: Name End, (0xCF 0x01 0x01 0x01)
61+
0x28: Icon?
62+
0x2E: Autotranslate
63+
64+
65+
Examples:
66+
67+
<Player First Name>: 02 2C 0D FF 07 02 29 03 EB 02 03 FF 02 20 02 03 (File 0x67A9C0A, 0x1A79)
68+
<Player Last Name>: 02 2C 0D FF 07 02 29 03 EB 02 03 FF 02 20 03 03
69+
<HQ ICON> 02 28 0A FF 06 41 64 64 6F 6E 0A 01 03
70+
<Gil Amount> 02 20 03 E8 03 03

‎research/explorer_exhf_files

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
#### EXHF RESEARCH #### By Ioncannon
2+
3+
Updated: 7/14/2015
4+
5+
Current research on EXHF files
6+
7+
!!!NOTE: FILE IS IN BIG ENDIAN FORM!!!
8+
9+
===EXHF Header===
10+
0x000 Signature Int32; EXHF
11+
0x004 Version Short; Always 0x03
12+
0x006 Size of Dataset Chunk Short; How large the data set chunk will be.
13+
0x008 Number of Datasets Short; Size of Dataset Definition table;
14+
0x00A Number of Pages Short; The size of the Page table below.
15+
0x00C Number of Lang Codes Short; Size of the language table at the end of the file.
16+
0x00E Unknown Short;
17+
0x010 Unknown Int32; Always 0x010000.
18+
0x014 Number of entries Int32; How many total entries are in this file (spanning multiple EXDFs)
19+
0x018 Padding ; Padded to 0x1F
20+
21+
===Dataset Definition Table=== (Size of Number of Datasets)
22+
0x000 Data Type Short; Most likely the data type. 2 - Byte, 3 - Byte, 4 - Short, 19 - String
23+
0x002 Data Offset Short; Where in the data chunk this byte is (offset is after 01 and the chunk size variables).
24+
25+
===Page Table=== (Size of Number of Pages)
26+
0x000 Page Entry Int32; Page Entry Name. See Items Page entry names.
27+
0x004 Page Entry Size Int32; Number of entries in a page. For example Item_0.exd has 500 entries... 0x1F4, with item_500.exd being the next file.
28+
29+
===Languge Table==== (Size of Number of Lang Codes)
30+
31+
0x000 Language Code Short;
32+
33+
The language codes are:
34+
35+
0x0 - n/a (No language)
36+
0x1 - ja (Japanese)
37+
0x2 - en (English)
38+
0x3 - de (German)
39+
0x4 - fr (French)
40+
0x5 - chs (Chinese - Singapore)
41+
0x6 - cht (Chinese - Traditional)
42+
0x7 - ko (Korean)
43+
44+
-----------------------NOTES---------------------
45+
46+
-The items.exd files span multiple pages. Start at items_0, then items__500, then items__1000, etc.

‎research/explorer_macro_dat_file

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
MACRO DAT FILE By Ioncannon
2+
3+
0x00: Unknown
4+
0x02: Unknown
5+
0x04: Size of file minux 0x20?
6+
0x08: Size of macro book from 0x10.
7+
8+
After a 0x11 byte header, the macro entries begin. All macro entries
9+
are XOR encoded with 0x73.
10+
11+
Each macro entry follows this format:
12+
13+
-Title
14+
-Icon
15+
-Key
16+
-Line List
17+
18+
Each section begins with a marker (T, I, K, L), followed by two bytes (little endian) telling the size, and then the data.
19+
Data seems to be always null terminated. For example:
20+
21+
A title: T 06 00 Greet\0
22+
A icon: I 08 00 001024B\0
23+
A key: K 04 00 005\0
24+
A line: L 0A 00 Well met!\0
25+
26+
Each macro entry contains 1 title, 1 icon, 1 key, and 15 lines. All three of these entries must exist even if not used,
27+
with data size being 0x01 for Titles and Lines, containing an empty string ("\0").
28+
29+
-Max line size is 0xB5.
30+
-Max lines is 15.
31+
-99 macros must exist, regardless if used (values are empty as above).
32+
-Icons are hex values to the icon table. There is a exh file called "macroicon.exh" which contains all usable icons. The key MUST equal the row number (hex) of the icon. This is how SE fixed the
33+
issue of allowing icons outside this set to be used.
34+
35+
After 99 macros, the data is padded with 0s to 0x46000. This is most likely to allow the max amount of data to be filled
36+
for all macros, lines, and titles.

‎research/explorer_scd_files

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
#### SCD FILE RESEARCH #### By Ioncannon
2+
3+
!Some information was based on VGMStream!
4+
5+
Current research on SCD music/sound files. Sorry it's a mess.
6+
7+
Structure (based on 0c0000.win32.index):
8+
9+
===SEDB SSCF HEADER===
10+
0x00 Signature Int64; SEDBSSCF
11+
0x08 Version Int32; For FFXIV should be 3;
12+
0x0C ~~Unknown~~ Short; Seems to be always 0x0400
13+
0x0E Header Size~~ Short; Points to the address where the next header is (also size of this header). Seems to be always 0x30.
14+
0x10 File Size Int32; Total File size
15+
~~~~~~~~Padding~~~~~~~~ Padded to fill 48 bytes (0x2F
16+
17+
===OFFSETS HEADER===
18+
0x00 Size of Offset Table 0 Short;
19+
0x02 Size of Sound Entry Offset Table Short; This is also the number of sounds in this file.
20+
0x04 Size of Offset Table 2 Short;
21+
0x06 ~~Unknown~~ Short; In one file: 0x0001, in another 0x270F
22+
0x08 Offset Table 0 Offset Int32;
23+
0x0C Sound Entry Offset Table Offset Int32;
24+
0x10 Offset to Offset Table 2 Int32;
25+
0x14 ~~Unknown/Null~~~ Int32;
26+
0x18 Offset To ??? Int32;
27+
28+
--Table 0: 4 * [Size of Table 1]
29+
--Table 1: 4 * [Size of Table 2]
30+
--Table 2: 4 * [Size of Table 3] ;Offsets to Sound Entries
31+
--Table 3: 4 * [Size of Table 1]
32+
33+
--Data 1 [Pointed from Table 3]
34+
--Data 2 [Pointed from Table 0]
35+
--Data 3 [Pointed from Table 1]
36+
2027296
37+
---------------MUSIC IS HERE----------------------------
38+
39+
===SOUND ENTRY HEADER=== (32 byte header)
40+
0x00 Music File Length Int32;
41+
0x04 Num Channels Int32; 0x01: Mono, 0x02: Stereo, FFXIV is usually 0x02 channels
42+
0x08 Frequency Int32; FFXIV is usually 44100HZ (AC440000)
43+
0x0C Data Type Int32; 0x0C: MS-ADPCM, 0x06: OGG. FFXIV seems to use OGG for music, MS-ADPCM for sound.
44+
0x10 Loop Start Int32; In Bytes. Calculation: (filesize/amount of samples)*sample
45+
0x14 Loop End Int32; Ditto, if you wanted to loop a whole song, set this to the file's size.
46+
0x18 First Frame Pos Int32; First OggS after (possibly, not always) encrypted header.
47+
0x1C Aux Chunk Count Short; Number of Aux Chunks
48+
49+
===AUX CHUNKS===
50+
-----MARK----
51+
0x00 ID Int32; ASCII MARK Id
52+
0x04 Chunk Size Int32; Size of this chunk, from start to end.
53+
0x08 Mark Table ;Chunk Size / 4 entries.
54+
55+
===IF MS-ADPCM===
56+
WAVEFORMATEX See: http://msdn.microsoft.com/en-us/library/windows/desktop/dd390970%28v=vs.85%29.aspx
57+
ADPCMCOEFSET Int32; * 7
58+
59+
===IF OGG SEEK TABLE HEADER=== (32 bytes)
60+
0x00 Encode Type Short; Seems to tell what encoding is being used. 02,20 is XOR. 03,20 is Heavensward one.
61+
0x02 Encode Byte Byte; If encoding is 02,20, XOR the header with this byte
62+
0x04
63+
0x08
64+
0x0c
65+
0x10 Size of Seek Table Int32; How many seek table entries there are.
66+
0x14 Vorbis Header Size Int32; Size of the Ogg Vorbis header (for decoding purpose).
67+
0x18
68+
~~~~~~~~~Padding~~~~~~~~ Padded to fill 32bytes
69+
70+
===SEEK TABLE=== (Size is 4 * [Num Seek Table])
71+
0x00 Seek Entry Int32; An offset from the Vorbis header to some Ogg page.
72+
73+
===VORBIS HEADER== (Size is [Vorbis Header Size])
74+
-A standard Ogg Vorbis header. Check the encode byte in the seek table header to see if this header is encoded. If it is,
75+
XOR all header bytes with the encode byte.
76+
-Many songs have a LOOPSTART and LOOPEND comment tag defining where music should begin and end after the first loop.
77+
78+
===DATA===
79+
Ogg Vorbis data is here
80+
81+
--------------Duplicate Header-----------------------------
82+
It seems that the first header is 3 chunks in size, each 0x120 bytes long. It's the Vorbis
83+
Indentification, Comment, and Setup headers. This is repeated after.
84+
85+
-------------Heavensward Encoding--------------------------
86+
Square Enix had encrypted the Heavensward music using a strange algorithm. Before decoding, you need the XOR table which can be found in the .exe if you search for:
87+
88+
3A 32 32 32 03 7E 12 F7 B2 E2 A2 67 32 32 22 32
89+
90+
The XOR table is 0xFF byte long.
91+
92+
Example code to decode (note dataLength is [Music File Length] in the entry header):
93+
94+
private void xorDecodeFromTable(byte[] dataFile, int dataLength) {
95+
int byte1 = dataLength & 0xFF & 0x7F;
96+
int byte2 = byte1 & 0x3F;
97+
for (int i = 0; i < dataFile.length; i++)
98+
{
99+
int xorByte = XORTABLE[(byte2 + i) & 0xFF];
100+
xorByte &= 0xFF;
101+
xorByte ^= (dataFile[i]&0xFF);
102+
xorByte ^= byte1;
103+
dataFile[i] = (byte) xorByte;
104+
}
105+
}
106+
107+
--------------To Inject OGG-------------------------------
108+
109+
-Change total file size
110+
-Change single file size
111+
-Change First Frame Position value
112+
-Change Header length value
113+
-Add data to end
114+
115+
--------------Putting MS-ADPCM into WAV--------------------
116+
-Write "RIFF"
117+
-Write 36 + [Size of first frame to end of file]
118+
-Write "WAVE"
119+
-Write "fmt "
120+
-Write 0x10
121+
-Write wave header
122+
-Write "data"
123+
-Write [Size of first frame to end of file]
124+
-Write data from [Size of first frame to end of file]
125+
126+
--------------Ogg Vorbis format info for understanding the encrypted header-----------------
127+
128+
===OGG PAGE===
129+
0x00 Signature Int32; Value is OggS
130+
0x04 Version Byte; Seems to be 0x00 for FFXIV
131+
0x05 Type Byte; Type of header: 0x01: Continuation, 0x02: Beginning, 0x03: End
132+
0x06 Granule Position Int64;
133+
0x0E Serial Number Int32; Seems to be 0x00 for FFXIV
134+
0x12 Sequence Number Int32;
135+
0x16 Checksum Int32;
136+
0x1E Num Page Segs Byte; Seems to be 0x01 for FFXIV
137+
0x1F Segment table Byte; Each segment is a byte long, * 0x1E's value. However, FFXIV's music only has 1 segment it seems, thus 1 byte size.
138+
139+
===VORBIS PAGE===
140+
0x00 Header Type Byte; Type of vorbis header: 0x01: ID Header, 0x03: Comment Header, 0x05: Setup Header. If has leading 0, audio header.
141+
0x01 Signature 6Bytes; VORBIS
142+
143+
--If ID Header--
144+
1 [vorbis_version] = read 32 bits as unsigned integer
145+
2 [audio_channels] = read 8 bit integer as unsigned
146+
3 [audio_sample_rate] = read 32 bits as unsigned integer
147+
4 [bitrate_maximum] = read 32 bits as signed integer
148+
5 [bitrate_nominal] = read 32 bits as signed integer
149+
6 [bitrate_minimum] = read 32 bits as signed integer
150+
7 [blocksize_0] = 2 exponent (read 4 bits as unsigned integer)
151+
8 [blocksize_1] = 2 exponent (read 4 bits as unsigned integer)
152+
9 [framing_flag] = read one bit
153+
154+
--If Comment Header---
155+
1 [vendor length] = read an unsigned integer of 32 bits
156+
2 [vendor string] = size of vendor length; Seems to be always �Xiph.Org libVorbis I 20040717� with a varying year.
157+
3 [fields] = read an unsigned integer of 32 bits; FFXIV only uses the LoopStart and LoopEnd fields

0 commit comments

Comments
 (0)