Issues in Custom Message Encoder

When implementing custom message encoder there are few hidden things which are different than in an implementation of the custom channel.
First issue you could have is for example in implementation of BuildChannelListener in the class which derives from MessageEncodingBindingElement.
Usually this method either is not implemented or it creates the inner channel:

public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
{
  TChannel innerChannel = base.BuildChannelListener<TChannel>(context);
  return new MyChannel(innerChannel);

}

This works perfectly as long you are working with usual bindings or transport elements. Unfortinatelly, one MessageEncoder is a special type of binding with special status on the stack. In a case of encoder, yoi do not have to put it in the channel stack. Instead, register the encoder by adding it to binding parameter list and WCF will do the rest:

public override IChannelListener<TChannel> BuildChannelListener<TChannel>(BindingContext context)
{
  context.BindingParameters.Add(this);
  return base.BuildChannelListener<TChannel>(context);
}

If this property is not registered following override will never be invoked by WCF:

public override MessageEncoderFactory CreateMessageEncoderFactory()

Second problem you can get is following error:

None of the binding elements in binding 'CustomBinding' define a message version. At least one binding element must define a message version and return it from the GetProperty<MessageVersion> method.

To solve it, implement a property GetProperty<T> in the class which derives from MessageEncodingBindingElement.
Here is exact code which show how to do this. Please also note the cast (T)(object).

public override T GetProperty<T>(BindingContext context)
{
  if (typeof(T) == typeof(MessageVersion))
  {
    return (T)(object)m_MessageVersion;
  }
  else
  {
     return base.GetProperty<T>(context);
  }
}