java - Dynamic recognition and handling of protocol data units in bytestream -

i implemented protocol little multiplayer game. based on bytes, deserialize received messages had iterate on byte stream , parse bit bit. after had bytes , knew message type, threw bytes in reverse constructor constructed protocol data unit raw bytes.

this whole process ugly, not oo , had unreadable if/else code. had implement reverseconstructor(byte[] bytes) every protocol data unit (pdu) added. approach kind of schema defined per pdu (e. g. schema = [1 byte int (id = x), x bytes ascii string, 4 bytes double]), , handling of bytes done schema, more elegant.

i got hint here on use google's protobufs (apparently not fitting needs, since have change protocol adhere protobuf standards).

info

i can't change protocol. there two different scenarios (i don't want support them @ same time or in same program):

• the protocol data units have length field encoded in header
• the protocol data units have no length field, 1 can derive message type when/where message ends.

i fan of length fields. have adhere protocol else designed. protocols fix. have header contains protocol id, unique message id, , in first scenario length field.

question

can give me small example 2 simple protocol data units parsed efficient, generic receive method? examples found in protobuf tutorials of type: user sends message x, user b expects message x , can deserialize without problem.

but if user b has prepared message x, y , z. how can 1 handle situation without code duplication in intelligent way.

i appreciate hints design principles enable me achieve greater code here without use of extern library.

---

edit

i think sth way go. can find more of code here. bytes read dynamically till object found, , position of buffer reset.

                while (true) {                         if (buffer.remaining() < framelength) {                                 buffer.reset();                                 break;                         }                         if (framelength > 0) {                                 object resultobj = prototype.newbuilderfortype().mergefrom(buffer.array(), buffer.arrayoffset() + buffer.position(), framelength).build();                                 client.firemessagereceived(resultobj);                                 buffer.position(buffer.position() + framelength);                                 buffer.mark();                         }                         if (buffer.remaining() > fieldsize) {                                 framelength = getframelength(buffer);                         } else {                                 break;                         }                 } 

javadoc - mergefrom

parse data message of type , merge message being built. small wrapper around messagelite.builder.mergefrom(codedinputstream). ^{https://developers.google.com/protocol-buffers/docs/reference/java/com/google/protobuf/message.builder#mergefrom(byte[])}

the problem part message of type, should possible address issue generic approach.

---

sample

here sample protocol data unit. has length field. there scenario pdus have no length field. pdu of variable size. there pdus of fixed size.

for completeness' sake. here representation of strings in protocol data units.

1. protocol design

frankly, common mistake create first protocol implementation without further changes in mind. exercise, let's try design flexible protocol.

basically, idea have several frames encapsulated each other. please note, have payload id available, easy identify next frame in sequence.

you can use wireshark in order see real-life protocols follow same principle.

such approach simplifies packet dissection lot, still possible deal other protocols.

2. protocol decoding(dissection)

i spent quite lot of time developing next generation network analyzer previous company.

can't expose details, 1 of key features flexible protocol stack, capable of identifying protocol frames. rtp example, because there no hint on lower layer (usually udp) next frame rtp frame. special vm developed execute dissectors , control process.

the news have smaller personal projects java-based dissectors (i'll skip javadocs in order save several lines).

/**  * high-level dissector contract definition. dissector meant simple  * protocol decoder, analyzes protocol binary image , produces number  * of fields.  *  * @author renat.gilmanov  */ public interface dissector {      /**      * returns dissector type.      */     dissectortype gettype();      /**      * verifies packet data belongs protocol represented dissector.      */     boolean isprotocol(datainput input, dissection dissection);      /**      * performs dissection.      */     dissection dissect(datainput input, dissection dissection);      /**      * returns protocol corresponds current dissector.      *      * @return protocol instance      */     protocol getprotocol(); } 

protocol knows upper layer protocol, when there no direct hint available possible iterate through known protocols , use isprotocol method in order identify next frame.

public interface protocol {      // ...      list<protocol> getupperprotocols(); } 

as said rtp protocol bit tricky handle:

so let's check implementation details. verification based on several known facts protocol:

/**  * verifies current frame belongs rtp protocol.  *  * @param input data input  * @param dissection initial dissection  * @return true if protocol frame rtp  */ @override public final boolean isprotocol(final datainput input, final dissection dissection) {     int available = input.available();     byte octet    = input.getbyte();     byte version  = getversion(octet);     byte octet2   = input.getbyte(1);     byte pt       = (byte) (octet2 & 0x7f);      return ((pt < 0x47) & (rtp_version == version)); } 

dissection set of basic operations:

public final dissection dissect(datainput input, dissection d) {

    // --- protocol header --------------------------------     final byte octet1 = input.getbyte(0);     final byte version = getversion(octet1);     final byte p = (byte) ((octet1 & 0x20) >> 5);     final byte x = (byte) ((octet1 & 0x10) >> 4);     final byte cc = (byte) ((octet1 & 0x0f));      //...      // --- seq --------------------------------------------     final int seq = (input.getint() & 0x0000ffff);     final int timestamp = input.getint();     final int ssrc = input.getint(); 

finally can define protocol stack:

public interface protocolstack {      string getname();      protocol getrootprotocol();      dissection dissect(datainput input, dissection dissection, dissectoptions options); } 

under hood handles complexity , decodes packet, frame frame. biggest challenge make dissection process bullet-proof , stable. using such or similar approach you'll able organize protocol decoding code. proper implementation of isprotocol let handle different version , on. anyway, not approach simple, provides lot of flexibility , control.

3. there universal solution?

yes, there asn.1:

abstract syntax notation 1 (asn.1) standard , notation describes rules , structures representing, encoding, transmitting, , decoding data in telecommunications , computer networking. formal rules enable representation of objects independent of machine-specific encoding techniques. formal notation makes possible automate task of validating whether specific instance of data representation abides specifications. in other words, software tools can used validation.

here example of protocol defined using asn.1:

fooprotocol definitions ::= begin      fooquestion ::= sequence {         trackingnumber integer,         question       ia5string     }      fooanswer ::= sequence {         questionnumber integer,         answer         boolean     }  end 

btw, there java asn.1 compiler available:

jac (java asn1 compiler) tool if want (1)parse asn1 file (2)create .java classes , (3)encode/decode instances of classes. forget asn1 byte streams, , take advantage of oop! ber, cer , der supported.

finally

i recommend several simple pocs in order find best possible solution. decided not use asn.1 in order reduce complexity , have room optimization, might you.

anyway, try can , let know results :)

you can check following topic: efficient decoding of binary , text structures (packets)

4. update: bidirectional approach

i'm sorry quite long answer. want have enough options find best possible solution. answering question regarding bidirectional approach:

• option 1: can use symmetrical serialization approach: define dataoutput, write serialization logic -- done. recommend looks through berkeleydb api , tuplebinding. solve same problem providing full control on store/restore procedure.

this class takes care of converting entries to/from tupleinput , tupleoutput objects. 2 abstract methods must implemented concrete subclass convert between tuples , key or data objects.

entrytoobject(tupleinput) objecttoentry(object,tupleoutput) 

• option 2: universal way define structure containing set of fields. every field requires following information:
  • name
  • type
  • size (bits)

for example, rtp following:

version:          byte (2 bits) padding:          bool (1 bit) extension:        bool (1 bit) csrc count:       byte (4 bits)  marker:           bool (1 bit) payload type:     byte (7 bits) sequence number:  int  (16 bits) 

having can define generic way of reading/writing such structures. closest working example know javolution struct. please through, have examples:

class clock extends struct { // hardware clock mapped memory.      unsigned16 seconds  = new unsigned16(5); // unsigned short seconds:5 bits      unsigned16 minutes  = new unsigned16(5); // unsigned short minutes:5 bits      unsigned16 hours    = new unsigned16(4); // unsigned short hours:4 bits      ...  }