Tibia: Data File Structure

[jo3bingham]

I've already written a thread on the Dat file structure, but I thought I'd post a new, updated, official one.

Note: As of typing this, the structure I'll be posting works with the current client version, 9.44, and should work back until the isSprite flag (8.54?). When I get time I'll update this to support Dat file structures as far back as I can.

[size=x-small]*All the names I use are what CipSoft uses in their flash client. But I also commented beside what may be confusing.[/size]
- The first 4 bytes of the Dat file, which I call the DatVersion, is unique for each client version so that the client knows that the correct Dat file is being used.
- Next we have 4 shorts (2 bytes each) that contain the number of items, outfits, effects, and projectiles (respectively) in the Dat file.
- Now starts the actual data of these objects. The first thing you'll encounter are what we call Flags. These Flags let the client know if the item can be walked on, if it can be picked up, if it gives off light, etc. etc.
[code=vb.net]
Dim Flag_Bank As Byte = 0
Dim Flag_Clip As Byte = 1
Dim Flag_Bottom As Byte = 2
Dim Flag_Top As Byte = 3
Dim Flag_Container As Byte = 4
Dim Flag_Cumulative As Byte = 5
Dim Flag_ForceUse As Byte = 6
Dim Flag_MultiUse As Byte = 7
Dim Flag_Write As Byte = 8
Dim Flag_WriteOnce As Byte = 9
Dim Flag_LiquidContainer As Byte = 10
Dim Flag_LiquidPool As Byte = 11
Dim Flag_Unpass As Byte = 12
Dim Flag_Unmove As Byte = 13
Dim Flag_Unsight As Byte = 14
Dim Flag_Avoid As Byte = 15
Dim Flag_Take As Byte = 16
Dim Flag_Hang As Byte = 17
Dim Flag_HookSouth As Byte = 18
Dim Flag_HookEast As Byte = 19
Dim Flag_Rotate As Byte = 20
Dim Flag_Light As Byte = 21
Dim Flag_DontHide As Byte = 22
Dim Flag_Translucent As Byte = 23
Dim Flag_Shift As Byte = 24
Dim Flag_Height As Byte = 25
Dim Flag_LyingObject As Byte = 26
Dim Flag_AnimateAlways As Byte = 27
Dim Flag_Automap As Byte = 28
Dim Flag_LensHelp As Byte = 29
Dim Flag_FullBank As Byte = 30
Dim Flag_IgnoreLook As Byte = 31
Dim Flag_Clothes As Byte = 32
Dim Flag_Market As Byte = 33
[/code]
- As you can see above we have a new Market flag. This was introduced with the new Market system in the 9.41 client. Here are the market categories associated with it:
[code=vb.net]
Private Enum MarketCategory
Armors = 1
Amulets = 2
Boots = 3
Containers = 4
Decoration = 5
Food = 6
Helmets_Hats = 7
Legs = 8
Others = 9
Potions = 10
Rings = 11
Runes = 12
Shields = 13
Tools = 14
Valuables = 15
Ammunition = 16
Axes = 17
Clubs = 18
DistanceWeapons = 19
Swords = 20
Wands_Rods = 21
MetaWeapons = 22
End Enum
[/code]
- After we go through all the bytes we come across the information needed for the sprites of the object. (height, width, layers, phases, etc.)
- Then we loop through all the available sprites associated with the item.
- And that's it. We hope on to the next object and do it again.

Now, here's code used to read the Dat file that you can throw in to VB.Net and run. If you have any questions feel free to ask:

[code=vb.net]
Public Class Form1
Dim dataArray As ItemData()

Dim Flag_Bank As Byte = 0 'ground tile
Dim Flag_Clip As Byte = 1
Dim Flag_Bottom As Byte = 2
Dim Flag_Top As Byte = 3
Dim Flag_Container As Byte = 4
Dim Flag_Cumulative As Byte = 5 'stackable
Dim Flag_ForceUse As Byte = 6 'always used
Dim Flag_MultiUse As Byte = 7 'usable
Dim Flag_Write As Byte = 8 'writeable
Dim Flag_WriteOnce As Byte = 9 'readable
Dim Flag_LiquidContainer As Byte = 10
Dim Flag_LiquidPool As Byte = 11 'splashes
Dim Flag_Unpass As Byte = 12
Dim Flag_Unmove As Byte = 13
Dim Flag_Unsight As Byte = 14 'blocks projectiles
Dim Flag_Avoid As Byte = 15 'blocks creature movements
Dim Flag_Take As Byte = 16 'pickupable
Dim Flag_Hang As Byte = 17
Dim Flag_HookSouth As Byte = 18
Dim Flag_HookEast As Byte = 19
Dim Flag_Rotate As Byte = 20
Dim Flag_Light As Byte = 21
Dim Flag_DontHide As Byte = 22
Dim Flag_Translucent As Byte = 23
Dim Flag_Shift As Byte = 24
Dim Flag_Height As Byte = 25
Dim Flag_LyingObject As Byte = 26
Dim Flag_AnimateAlways As Byte = 27
Dim Flag_Automap As Byte = 28
Dim Flag_LensHelp As Byte = 29
Dim Flag_FullBank As Byte = 30
Dim Flag_IgnoreLook As Byte = 31
Dim Flag_Clothes As Byte = 32
Dim Flag_Market As Byte = 33

Private Enum MarketCategory
Armors = 1
Amulets = 2
Boots = 3
Containers = 4
Decoration = 5
Food = 6
Helmets_Hats = 7
Legs = 8
Others = 9
Potions = 10
Rings = 11
Runes = 12
Shields = 13
Tools = 14
Valuables = 15
Ammunition = 16
Axes = 17
Clubs = 18
DistanceWeapons = 19
Swords = 20
Wands_Rods = 21
MetaWeapons = 22 'all weapons
End Enum

Private Structure ItemData
Dim ID As Integer

Dim isBank As Boolean
Dim Waypoints As Integer
Dim isClip As Boolean
Dim isBottom As Boolean
Dim isTop As Boolean
Dim isContainer As Boolean
Dim isCumulative As Boolean
Dim isForceUse As Boolean
Dim isMultiUse As Boolean
Dim isWriteable As Boolean
Dim MaxTextLength As Integer
Dim isWriteableOnce As Boolean
Dim isLiquidContainer As Boolean
Dim isLiquidPool As Boolean
Dim isUnpassable As Boolean
Dim isUnmoveable As Boolean
Dim isUnsight As Boolean
Dim isAvoid As Boolean
Dim isTakeable As Boolean
Dim isHangable As Boolean
Dim isHookSouth As Boolean
Dim isHookEast As Boolean
Dim isRotateable As Boolean
Dim isLight As Boolean
Dim Brightness As Integer
Dim LightColor As Integer
Dim isDontHide As Boolean
Dim isTranslucent As Boolean
Dim isDisplaced As Boolean
Dim DisplacementX As Integer
Dim DisplacementY As Integer
Dim isHeight As Boolean
Dim Elevation As Integer
Dim isLyingObject As Boolean
Dim isAnimateAlways As Boolean
Dim isAutomap As Boolean
Dim isAutomapColor As Integer
Dim isLensHelp As Boolean
Dim LensHelp As Integer
Dim isFullBank As Boolean
Dim isIgnoreLook As Boolean
Dim isCloth As Boolean
Dim ClothSlot As Integer
Dim isMarket As Boolean
Dim MarketCategory As MarketCategory
Dim MarketTradeAs As Integer
Dim MarketShowAs As Integer
Dim MarketName As String
Dim MarketRestrictProfession As Integer
Dim MarketRestrictLevel As Integer

Dim Width As Integer
Dim Height As Integer
Dim ExactSize As Integer
Dim Layers As Integer
Dim PatternWidth As Integer
Dim PatternHeight As Integer
Dim PatternDepth As Integer
Dim Phases As Integer
Dim NumberOfSprites As Integer

Dim Sprites As Integer()
End Structure

Private Sub Form1_Load(ByVal sender As System.Object, ByVal e As System.EventArgs) Handles MyBase.Load
Dim FileName As String = vbNullString
MessageBox.Show("Please locate and open your Tibia.dat file", ".dat Reader")
With OpenFileDialog1
.Filter = "Dat file (*.dat)|*.dat|" & "All files|*.*"
If .ShowDialog = Windows.Forms.DialogResult.OK Then
FileName = .FileName
Else
Application.Exit()
End If
End With

Using reader As New IO.BinaryReader(IO.File.OpenRead(FileName))
Dim DatVersion As Integer = reader.ReadUInt32()
Dim Items As Integer = reader.ReadUInt16()
Dim Outfits As Integer = reader.ReadUInt16()
Dim Effects As Integer = reader.ReadUInt16()
Dim Projectiles As Integer = reader.ReadUInt16()

Dim maxID As Integer = Items + Outfits + Effects + Projectiles
Dim ID As Integer = 100

dataArray = Array.CreateInstance(GetType(ItemData), maxID - ID)

Do While (ID < maxID)
Dim itemData As New ItemData
itemData.ID = ID
Do
Dim optByte As Byte
optByte = reader.ReadByte()
Select Case optByte
Case Flag_Bank
itemData.isBank = True
itemData.Waypoints = reader.ReadUInt16() 'ground speed
Case Flag_Clip
itemData.isClip = True
Case Flag_Bottom
itemData.isBottom = True
Case Flag_Top
itemData.isTop = True
Case Flag_Container
itemData.isContainer = True
Case Flag_Cumulative
itemData.isCumulative = True
Case Flag_ForceUse
itemData.isForceUse = True
Case Flag_MultiUse
itemData.isMultiUse = True
Case Flag_Write
itemData.isWriteable = True
itemData.MaxTextLength = reader.ReadUInt16()
Case Flag_WriteOnce
itemData.isWriteableOnce = True
itemData.MaxTextLength = reader.ReadUInt16()
Case Flag_LiquidContainer
itemData.isLiquidContainer = True
Case Flag_LiquidPool
itemData.isLiquidPool = True
Case Flag_Unpass
itemData.isUnpassable = True
Case Flag_Unmove
itemData.isUnmoveable = True
Case Flag_Unsight
itemData.isUnsight = True
Case Flag_Avoid
itemData.isAvoid = True
Case Flag_Take
itemData.isTakeable = True
Case Flag_Hang
itemData.isHangable = True
Case Flag_HookSouth
itemData.isHookSouth = True
Case Flag_HookEast
itemData.isHookEast = True
Case Flag_Rotate
itemData.isRotateable = True
Case Flag_Light
itemData.isLight = True
itemData.Brightness = reader.ReadUInt16()
itemData.LightColor = reader.ReadUInt16()
Case Flag_DontHide
itemData.isDontHide = True
Case Flag_Translucent
itemData.isTranslucent = True
Case Flag_Shift
itemData.isDisplaced = True
itemData.DisplacementX = reader.ReadUInt16()
itemData.DisplacementY = reader.ReadUInt16()
Case Flag_Height
itemData.isHeight = True
itemData.Elevation = reader.ReadUInt16()
Case Flag_LyingObject
itemData.isLyingObject = True
Case Flag_AnimateAlways
itemData.isAnimateAlways = True
Case Flag_Automap
itemData.isAutomap = True
itemData.isAutomapColor = reader.ReadUInt16()
Case Flag_LensHelp
itemData.isLensHelp = True
itemData.LensHelp = reader.ReadUInt16()
Case Flag_FullBank
itemData.isFullBank = True
Case Flag_IgnoreLook
itemData.isIgnoreLook = True
Case Flag_Clothes
itemData.isCloth = True
itemData.ClothSlot = reader.ReadUInt16() 'head, torso, legs, feet, etc.
Case Flag_Market
itemData.isMarket = True
itemData.MarketCategory = reader.ReadUInt16() 'see the MarketCategory enum above
itemData.MarketTradeAs = reader.ReadUInt16()
itemData.MarketShowAs = reader.ReadUInt16()

'using the binary reader's ReadString() function returned the correctly lengthed string, but started a character early
'eg. "gold coin" would say " gold coi"
'to bypass this I just read the string length (first 2 bytes) then read that many bytes
'but the string is iso-8859-1 encoded, so you have to decode it to get the actual string
Dim MarketNameLength As Integer = reader.ReadUInt16()
itemData.MarketName = System.Text.Encoding.GetEncoding("iso-8859-1").GetString(reader.ReadBytes(MarketNameLength ))

itemData.MarketRestrictProfession = reader.ReadUInt16()
itemData.MarketRestrictLevel = reader.ReadUInt16()
Case 255
Exit Do
Case Else
MessageBox.Show("Invalid flag!: " & optByte.ToString)
End Select
Loop

itemData.Width = reader.ReadByte()
itemData.Height = reader.ReadByte()
If itemData.Width > 1 Or itemData.Height > 1 Then
itemData.ExactSize = reader.ReadByte()
End If

itemData.Layers = reader.ReadByte()
itemData.PatternWidth = reader.ReadByte()
itemData.PatternHeight = reader.ReadByte()
itemData.PatternDepth = reader.ReadByte()
itemData.Phases = reader.ReadByte()

Dim numSpr As Integer = itemData.Width * itemData.Height
numSpr *= itemData.Layers * itemData.PatternWidth
numSpr *= itemData.PatternHeight * itemData.PatternDepth
numSpr *= itemData.Phases

itemData.NumberOfSprites = numSpr

itemData.Sprites = Array.CreateInstance(GetType(Integer), itemData.NumberOfSprites)
For i As Integer = 0 To itemData.NumberOfSprites - 1
itemData.Sprites(i) = reader.ReadUInt16()
Next

dataArray(ID - 100) = itemData
ID += 1
Loop
End Using
End Sub
End Class
[/code]

All objects are stored in dataArray and you can access objects by ID. Here's an example:
[code=vb.net]
Dim GoldCoin As ItemData = dataArray(3031 - 100) '3031 is the ID of gold coin; we have to subtract 100 because the array is 0-based
[/code]

[rtoip]

Structure in Ruby 1.9 with BinData:

require 'bindata'

class TibiaString < BinData::Record
	endian :little
	
	uint16 :len, value: -> { s.length }
	string :s, read_length: :len
	
	def get; s; end
	def set v; s = v; end
end

Bank, Clip, Bottom, Top, Container, Cumulative, ForceUse,
MultiUse, Write, WriteOnce, LiquidContainer, LiquidPool, Unpass, Unmove,
Unsight, Avoid, Take, Hang, HookSouth, HookEast, Rotate,
Light, DontHide, Translucent, Shift, Height, LyingObject, AnimateAlways,
Automap, LensHelp, FullBank, IgnoreLook, Clothes, Market = *(0 .. 33)

Armors, Amulets, Boots, Containers, Decoration, Food, Helmets_Hats,
Legs, Others, Potions, Rings, Runes, Shields, Tools,
Valuables, Ammunition, Axes, Clubs, DistanceWeapons, 
Swords, Wands_Rods, MetaWeapons = *(1 .. 22)

class DatObject < BinData::Record
	endian :little
	
	array :flags, read_until: -> { element.flag == 0xFF } do
		uint8 :flag, check_value: -> { value <= 33 or value == 0xFF }
		
		uint8 :waypoints, onlyif: -> { flag == Bank }
		uint16 :maxtextlength, onlyif: -> { flag == Write or flag == WriteOnce }
		uint16 :brightness, onlyif: -> { flag == Light }
		uint16 :lightcolor, onlyif: -> { flag == Light }
		uint16 :displ_x, onlyif: -> { flag == Shift }
		uint16 :displ_y, onlyif: -> { flag == Shift }
		uint16 :elevation, onlyif: -> { flag == Height }
		uint16 :automapcolor, onlyif: -> { flag == Automap }
		uint16 :lenshelp, onlyif: -> { flag == LensHelp }
		uint16 :clothslot, onlyif: -> { flag == Clothes }
		uint16 :marketcategory, onlyif: -> { flag == Market }
		uint16 :markettradeas, onlyif: -> { flag == Market }
		uint16 :marketshowas, onlyif: -> { flag == Market }
		tibia_string :marketname, onlyif: -> { flag == Market }
		uint16 :marketrestrictprofession, onlyif: -> { flag == Market }
		uint16 :marketrestrictlevel, onlyif: -> { flag == Market }
	end
	
	uint8 :width
	uint8 :height
	uint8 :exactsize, onlyif: -> { width > 1 or height > 1 }
	
	uint8 :layers
	uint8 :patternwidth
	uint8 :patternheight
	uint8 :patterndepth
	uint8 :phases
	
	array :sprites, type: :uint16, initial_length: -> { width * height * layers * patternwidth * patternheight * patterndepth * phases }
end

class TibiaDat < BinData::Record
	endian :little
	
	uint32 :version
	
	uint16 :numitems
	uint16 :numoutfits
	uint16 :numeffects
	uint16 :numprojectiles
	
	array :items, type: :dat_object, initial_length: -> { numitems - 100 }
	array :outfits, type: :dat_object, initial_length: -> { numoutfits }
	array :effects, type: :dat_object, initial_length: -> { numeffects }
	array :projectiles, type: :dat_object, initial_length: -> { numprojectiles }
end

BinData::trace_reading do
	struct = File.open('Tibia.dat') {|f| TibiaDat.read(f) }
end

[Sketchy]

[code=vb]
'using the binary reader's ReadString() function returned the correctly lengthed string, but started a character early
'eg. "gold coin" would say " gold coi"
'to bypass this I just read the string length (first 2 bytes) then read that many bytes
'but the string is iso-8859-1 encoded, so you have to decode it to get the actual string
[/code]

Just wanted to explain that the reason why BinaryReader.ReadString does this is because it uses a variable-length quantity scheme for the string length, so the number of bytes it uses for the length adjusts according to the size of the string. It uses the classic "7-bit encoding" scheme where each length byte contains 7 bits of the integer, the remaining bit (most significant) is used as a flag whether the next byte is part of the integer. So basically it reads the length like so (actually this is exactly what it does):

int value = 0;
int shift = 0;
byte current;

do
{
	current = Reader.ReadByte();
	value |= (current & 0x7F) << shift;
	shift += 7;
} while ((current & 0x80) != 0);

So basically in the gold coin example ReadString would only read the first byte of the length as the length and then start reading the string from the second length byte. The string encoding would have been another issue that would've required giving the reader the correct text encoding which can only be done through its constructor. Anyway knowing that BinaryReader/Writer stores the string length like this would useful to know when you must manipulate files/memory which have such strings but you do not have access to either of the classes (or simply don't want to use them for whatever reason).


Anyway nice work Jo3.

[klusbert]

is there any flag for istackable?

Anyhow, great work!

[jo3bingham]

[code=vb]
'using the binary reader's ReadString() function returned the correctly lengthed string, but started a character early
'eg. "gold coin" would say " gold coi"
'to bypass this I just read the string length (first 2 bytes) then read that many bytes
'but the string is iso-8859-1 encoded, so you have to decode it to get the actual string
[/code]

Just wanted to explain that the reason why BinaryReader.ReadString does this is because it uses a variable-length quantity scheme for the string length, so the number of bytes it uses for the length adjusts according to the size of the string. It uses the classic "7-bit encoding" scheme where each length byte contains 7 bits of the integer, the remaining bit (most significant) is used as a flag whether the next byte is part of the integer. So basically it reads the length like so (actually this is exactly what it does):

int value = 0;
int shift = 0;
byte current;

do
{
	current = Reader.ReadByte();
	value |= (current & 0x7F) << shift;
	shift += 7;
} while ((current & 0x80) != 0);

So basically in the gold coin example ReadString would only read the first byte of the length as the length and then start reading the string from the second length byte. The string encoding would have been another issue that would've required giving the reader the correct text encoding which can only be done through its constructor. Anyway knowing that BinaryReader/Writer stores the string length like this would useful to know when you must manipulate files/memory which have such strings but you do not have access to either of the classes (or simply don't want to use them for whatever reason).


Anyway nice work Jo3.

Ah, that makes sense. As always, thanks for the information.

is there any flag for istackable?

Anyhow, great work!

Yeah, I commented it.

Dim Flag_Cumulative As Byte = 5 'stackable

[klusbert]

sry totaly missed it xD

but it return false on gold coin :S

[maozao]

Great work Jo3 =)

It will help us a lot.

Thanks.

[Wesker]

ty for linking me, appreciated and added to favorites for sure

[jo3bingham]

sry totaly missed it xD

but it return false on gold coin :S

Which client's dat file were you using this with? Because the code I gave works for 8.60+, anything before that won't work with those flags. However, I did manage to write up code for all the different flags back to 7.10, which I'll post soon.

Actually, while typing this, I think I found the problem. In my example code I said to use this for the gold coin example:
[code=vb.net]
Dim GoldCoin As ItemData = dataArray(3031) '3031 is the ID of gold coin
[/code]

When in fact you have to subtract 100, because of how I stored the data:
[code=vb.net]
Dim GoldCoin As ItemData = dataArray(3031 - 100) '3031 is the ID of gold coin
[/code]

[jo3bingham]

I'll update the original post to better explain the structure. I seemed to have just skimmed through it with lack of detail.