пятница, 14 ноября 2008 г.

Cisco Raw Data Record format

PRELUDE:

Recently the company I work at bought SCE which reports data throught Netflow v9. Though, in the other docs the SCE is said not to support Netflow v9. So we decided to catch RDRs and convert it to Netflow v5.
I was disappointed that the Raw Data Records: Formats and Field Contents document didn't contain full format description of RDR. It misses the main point: it doesn't tell you how the fields are layed out after you meet certain TAG.
Hopefully, we found a man who have already done RDR to Netflow v5 converter and asked him to help us out with RDR format. He sent us his RDR to Netflow v5 converter which I was observing. When I found out the RDR layout I was once again disappointed, because the first time I tried to catch and read RDRs, I used plain C structures.
For example, Transaction usage RDR contains almost all we need for a Netflow v5 packet. So I used
struct TUR {
char *subscriberID;
unsigned short packageID;
int serviceID;
- - - -
};

When thinking how they could send strings I assumed they simply send zero terminated strings like C does. That was false.

TOPIC:

GCC typedefs I use:
//! Typedef for signed 8-bit integer
typedef signed char s8;
//! Typedef for signed 16-bit integer
typedef signed short s16;
//! Typedef for signed 32-bit integer
typedef signed int s32;
//! Typedef for unsigned 8-bit integer
typedef unsigned char u8;
//! Typedef for unsigned 16-bit integer
typedef unsigned short u16;
//! Typedef for unsigned 32-bit integer
typedef unsigned int u32;


Every RDR field has the following format:
struct RDRField {
u8 type; // Field type
u32 size; // Field size
void *data;
};


Every RDR packet has the following format (RDRHeader format is unknown to me):
struct RDRPacket {
RDRHeader header; // sizeof(RDRHeader) == 20?
u32 tag; // Transaction usage, Subscriber, etc.
u8 fieldsNb; // Number of fields following (25 for TUR, 12 for SUR, etc.)
RDRField *fields;
};


For example, STRING field format is this:
struct STRING {
u8 type; // 0x29
u32 length;
char *str;
};

PACKAGE_ID:
struct PACKAGE_ID {
u8 type;
u32 size;
u16 value;
};

SERVICE_ID:
struct SERVICE_ID {
u8 type;
u32 size;
s32 value;
};

And so on.
Type and size are useful probably only for STRING fields to determine their lengths. Other fields don't need them really.
Note that most signed and unsigned 32-bit integers are in big endian (for example, a field's u32 size).

пятница, 26 сентября 2008 г.

Filesystem's system space size: ReiserFS > NTFS > Ext3





PRELUDE:

Recently I bought the second 750G hard drive Seagate with 32M cache. The first one was 750G Western digital with 16M cache. I bought it because I was out of space because I kept all I have on hard drive, rather than kept it on DVDs, because I share data with other users (local P2P atm).
So I decided to use the old Western digital as data one to keep music, movies, install files and all that is mostly read-only accessed thus giving my old hard drive a break after running for over half a year mostly 24h a day. So I went on copying data from old hard drive to the new one. First I found out that the disk with 32M cache is about 2-4 times faster on writing! So that's only sane it is used for system read-write. When I backed up everything I decided I will have single partition on the old hard drive for data. I was wondering which filesystem to choose. As a Linux user, I like ReiserFS as it is faster than Ext3 (while Ext4 is not yet out atm). But as a sane person I know I need to access the data from Windows too (playing games). First, I thought to use tools like Ext2FS for Windows and ReiserFS tool to access either volume from Windows. Both sucked too much. Ext2 one couldn't deal with Russian language, though I tried cp866, cp1251, utf-8, nothing helped. And ReiserFS one only works in cmd.exe. So back to old good NTFS, thanks ntfs-3g driver is now shipped by default with Slackware (distro I use since 10.0).

TOPIC:

Choosing among the three filesystems gave me some interesting results on the amount of system data they require to work. At the top you can see images with the results. It was really disgusting to know Ext3 eats about 1.5% of your space (which results in 11G of literally wasted space here) just to keep files. While NTFS and ReiserFS both eat under 100M of space, about 0.12% and 0.08% respectively. Notice, that ReiserFS eats about 1.6 times less than NTFS. I didn't know ReiserFS also rules in the space too :)

Though, the ReiserFS4 was sabotaged: http://linuxhelp.150m.com/jews/saboteurs.htm (in flavor of Ext4?)

вторник, 22 июля 2008 г.

Theading, Multitheaded game engine

Multithreaded Game Engine Architectures:

http://www.gamasutra.com/features/20060906/monkkonen_01.shtml

Threading 3D Game Engine Basics:

http://www.gamasutra.com/features/20051117/gabb_01.shtml

POSIX Threads Programming (good article to get understanding what thread is):

https://computing.llnl.gov/tutorials/pthreads/

четверг, 3 апреля 2008 г.

Shortcut tags I use

WIR = What I've Read

Links, books, articles, etc. which were useful to me.

CBF = Can Be Found (in Google)

The book is free for download if you bother to search for it. No direct links.