Charset encoding other than UTF-8 (as ISO-8859-1) not working (CXF 2.7.8)

[Jose María Zaragoza]

Hello:

I'm using CXF 2.7.8
I've created a JAX-RS service with JSON databinding provider (Jettison)
Service is running on Tomcat server in a Windows machine

I've annotated my service class as

@Consumes("*/*")
@Produces("application/json;charset=ISO-8859-1")

but it always returns

Content-Type=[application/json] , not "application/json;charset=ISO-8859-1"

If I read returned payload with

BufferedReader br = new BufferedReader(new
InputStreamReader(connection.getInputStream(),"ISO-8859-1"));

prints (bad)

"agent_name":"LÃ-­a"


but with

BufferedReader br = new BufferedReader(new
InputStreamReader(connection.getInputStream(),"UTF-8"));

prints ( right )

"agent_name":"Lí­a"


What am I doing wrong ?
Why Content-Type isn't application/json;charset=ISO-8859-1 ?


Thanks and regards

[Jose María Zaragoza]

I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding

I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));

For example:

CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8

Why is not encoding in iso-8859-1 ?

Regards

[Sergey Beryozkin]

Looks like it is something related to Jettison...XML works OK.
Need to investigate more.

Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

[Sergey Beryozkin]

Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

[Jose María Zaragoza]

Post by Sergey Beryozkin
Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Thank you

How can I get this JSONUtils fixed ?
I would like to backport this patch to CXF 2.7.8 , could you help me
to know the line(s) fixed ?


Regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

[Sergey Beryozkin]

Hi, sure, see changes at
https://issues.apache.org/jira/browse/CXF-5619

Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Thank you
How can I get this JSONUtils fixed ?
I would like to backport this patch to CXF 2.7.8 , could you help me
to know the line(s) fixed ?
Regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

[Jose María Zaragoza]

Post by Sergey Beryozkin
Hi, sure, see changes at
https://issues.apache.org/jira/browse/CXF-5619

Hi:

I was having a look at this bug and I've seen that you fixed it for
createStreamWriter() method( I tested it and it works fine )
but you didn't it for createStreamReader() method, any reason ?

I think the solution is in JSONUtils.java source class, in the line with

new JettisonReader(namespaceMap, factory.createXMLStreamReader(is));

Should it be factory.createXMLStreamReader(is, enc) ?


So I guess it should be a

String enc = HttpUtils.getSetEncoding(m, headers, "UTF-8");

in

T readFrom(Class<T> type, Type genericType, Annotation[] anns,
MediaType mt, MultivaluedMap<String, String> headers, InputStream is)




And another question :

what about these methods

public static XMLStreamWriter createBadgerFishWriter(OutputStream os)
throws XMLStreamException
public static XMLStreamReader createBadgerFishReader(InputStream is)
throws XMLStreamException

?

Should they have a encoding parameter ?


Thanks and regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Thank you
How can I get this JSONUtils fixed ?
I would like to backport this patch to CXF 2.7.8 , could you help me
to know the line(s) fixed ?
Regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

[Sergey Beryozkin]

I'm going to check headers ( Content-Type really )
Content-Type:application/json means payload is encoded with ISO-8859-1
I've seen that you use
String enc = HttpUtils.getEncoding(mt, "UTF-8");
but the default value, according HTTP spec , should be ISO-8859-1

well, it is not that simple, I guess ISO-8859-1 was treated as a default
because when HTML 1.0 was created many years ago the question of supporting
all sort of languages was not a high priority; and again when we deal with
JSON we may say that that defaulting to ISO-8859-1 is wrong as per the JSON
spec

Well, I don't agree, because if you use MediaType ( HTTP header ) to
retrieve encoding, you must follow HTTP spec

No, it is not that simple. Besides this text:

"The "charset" parameter is used with some media types to define the
character set (section 3.4) of the data. When no explicit charset
parameter is provided by the sender, media subtypes of the "text" type
are defined to have a default charset value of "ISO-8859-1" when
received via HTTP"

seems very outdated, applies to text/* only, seems to be talking about
the senders only and is not available in HTTPBIS specs (I can't see it
at least), it does not make sense to implement it

Sergey

Anyway, with your permission, I going to make a local fork of
JSONProvider.java . Indeed, default encoding could be a good candidate
to be a set property
Thanks and regards

[Sergey Beryozkin]

Post by Sergey Beryozkin

I'm going to check headers ( Content-Type really )
Content-Type:application/json means payload is encoded with ISO-8859-1
I've seen that you use
String enc = HttpUtils.getEncoding(mt, "UTF-8");
but the default value, according HTTP spec , should be ISO-8859-1

well, it is not that simple, I guess ISO-8859-1 was treated as a default
because when HTML 1.0 was created many years ago the question of supporting
all sort of languages was not a high priority; and again when we deal with
JSON we may say that that defaulting to ISO-8859-1 is wrong as per the JSON
spec

Well, I don't agree, because if you use MediaType ( HTTP header ) to
retrieve encoding, you must follow HTTP spec

"The "charset" parameter is used with some media types to define the
character set (section 3.4) of the data. When no explicit charset
parameter is provided by the sender, media subtypes of the "text" type
are defined to have a default charset value of "ISO-8859-1" when
received via HTTP"
seems very outdated, applies to text/* only, seems to be talking about
the senders only and is not available in HTTPBIS specs (I can't see it
at least), it does not make sense to implement it

Specifically see
http://tools.ietf.org/html/draft-ietf-httpbis-p2-semantics-26#section-3.1.1.1
and then follow to Accept.

Even we say that HTTP 2.0 has not been released, we can still be sure
that text/* is not related to application/json

Thanks, Sergey

Post by Sergey Beryozkin
Sergey

Anyway, with your permission, I going to make a local fork of
JSONProvider.java . Indeed, default encoding could be a good candidate
to be a set property
Thanks and regards

[Jose María Zaragoza]

Post by Sergey Beryozkin
"The "charset" parameter is used with some media types to define the
character set (section 3.4) of the data. When no explicit charset parameter
is provided by the sender, media subtypes of the "text" type are defined to
have a default charset value of "ISO-8859-1" when received via HTTP"
seems very outdated, applies to text/* only, seems to be talking about the
senders only and is not available in HTTPBIS specs (I can't see it at
least), it does not make sense to implement it

Yes, you're right .
But i need to face different interpretations by my clients about this
rule . Damn realpolitik ... :-)

Post by Sergey Beryozkin
Sergey

Anyway, with your permission, I going to make a local fork of
JSONProvider.java . Indeed, default encoding could be a good candidate
to be a set property
Thanks and regards

[Sergey Beryozkin]

Post by Jose María Zaragoza

Post by Sergey Beryozkin
"The "charset" parameter is used with some media types to define the
character set (section 3.4) of the data. When no explicit charset parameter
is provided by the sender, media subtypes of the "text" type are defined to
have a default charset value of "ISO-8859-1" when received via HTTP"
seems very outdated, applies to text/* only, seems to be talking about the
senders only and is not available in HTTPBIS specs (I can't see it at
least), it does not make sense to implement it

Yes, you're right .
But i need to face different interpretations by my clients about this
rule . Damn realpolitik ... :-)

I think it is OK telling clients gently that may be their interpretation
is not technically correct; but in the end of the day you can simply add
a filter that will add "charset=ISO-8859-1" to Content-Type if it is
missing a charset

Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Sergey

Anyway, with your permission, I going to make a local fork of
JSONProvider.java . Indeed, default encoding could be a good candidate
to be a set property
Thanks and regards

[Jose María Zaragoza]

Hi,
I've updated JSONProvider to pass the custom encoding if any to the readers.
But only on the trunk as I had to update few protected methods and I think
there may be few users customizing the jettison reader...
By the way, I checked
http://www.ietf.org/rfc/rfc4627.txt

You're right but my clients don't think like you ( and me )

I'm going to check headers ( Content-Type really )
Content-Type:application/json means payload is encoded with ISO-8859-1

I've seen that you use

String enc = HttpUtils.getEncoding(mt, "UTF-8");


but the default value, according HTTP spec , should be ISO-8859-1


Regards

It says only UTF-8 and other UTF-* encoding are supported.
Not sure if that is too limiting or not, I guess if the client or server
uses some other encoding with a charset parameter then we can support it...
Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Hi, sure, see changes at
https://issues.apache.org/jira/browse/CXF-5619

I was having a look at this bug and I've seen that you fixed it for
createStreamWriter() method( I tested it and it works fine )
but you didn't it for createStreamReader() method, any reason ?
I think the solution is in JSONUtils.java source class, in the line with
new JettisonReader(namespaceMap, factory.createXMLStreamReader(is));
Should it be factory.createXMLStreamReader(is, enc) ?
So I guess it should be a
String enc = HttpUtils.getSetEncoding(m, headers, "UTF-8");
in
T readFrom(Class<T> type, Type genericType, Annotation[] anns,
MediaType mt, MultivaluedMap<String, String> headers, InputStream is)
what about these methods
public static XMLStreamWriter createBadgerFishWriter(OutputStream os)
throws XMLStreamException
public static XMLStreamReader createBadgerFishReader(InputStream is)
throws XMLStreamException
?
Should they have a encoding parameter ?
Thanks and regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Thank you
How can I get this JSONUtils fixed ?
I would like to backport this patch to CXF 2.7.8 , could you help me
to know the line(s) fixed ?
Regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

[Jose María Zaragoza]

I'm going to check headers ( Content-Type really )
Content-Type:application/json means payload is encoded with ISO-8859-1
I've seen that you use
String enc = HttpUtils.getEncoding(mt, "UTF-8");
but the default value, according HTTP spec , should be ISO-8859-1

well, it is not that simple, I guess ISO-8859-1 was treated as a default
because when HTML 1.0 was created many years ago the question of supporting
all sort of languages was not a high priority; and again when we deal with
JSON we may say that that defaulting to ISO-8859-1 is wrong as per the JSON
spec

Well, I don't agree, because if you use MediaType ( HTTP header ) to
retrieve encoding, you must follow HTTP spec
Anyway, with your permission, I going to make a local fork of
JSONProvider.java . Indeed, default encoding could be a good candidate
to be a set property

Thanks and regards

[Sergey Beryozkin]

Post by Jose María Zaragoza

Hi,
I've updated JSONProvider to pass the custom encoding if any to the readers.
But only on the trunk as I had to update few protected methods and I think
there may be few users customizing the jettison reader...
By the way, I checked
http://www.ietf.org/rfc/rfc4627.txt

You're right but my clients don't think like you ( and me )

But your clients are not my clients :-), lol

Post by Jose María Zaragoza
I'm going to check headers ( Content-Type really )
Content-Type:application/json means payload is encoded with ISO-8859-1
I've seen that you use
String enc = HttpUtils.getEncoding(mt, "UTF-8");
but the default value, according HTTP spec , should be ISO-8859-1

well, it is not that simple, I guess ISO-8859-1 was treated as a default
because when HTML 1.0 was created many years ago the question of
supporting all sort of languages was not a high priority; and again when
we deal with JSON we may say that that defaulting to ISO-8859-1 is wrong
as per the JSON spec

Cheers, Sergey

Post by Jose María Zaragoza
Regards

It says only UTF-8 and other UTF-* encoding are supported.
Not sure if that is too limiting or not, I guess if the client or server
uses some other encoding with a charset parameter then we can support it...
Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Hi, sure, see changes at
https://issues.apache.org/jira/browse/CXF-5619

I was having a look at this bug and I've seen that you fixed it for
createStreamWriter() method( I tested it and it works fine )
but you didn't it for createStreamReader() method, any reason ?
I think the solution is in JSONUtils.java source class, in the line with
new JettisonReader(namespaceMap, factory.createXMLStreamReader(is));
Should it be factory.createXMLStreamReader(is, enc) ?
So I guess it should be a
String enc = HttpUtils.getSetEncoding(m, headers, "UTF-8");
in
T readFrom(Class<T> type, Type genericType, Annotation[] anns,
MediaType mt, MultivaluedMap<String, String> headers, InputStream is)
what about these methods
public static XMLStreamWriter createBadgerFishWriter(OutputStream os)
throws XMLStreamException
public static XMLStreamReader createBadgerFishReader(InputStream is)
throws XMLStreamException
?
Should they have a encoding parameter ?
Thanks and regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Thank you
How can I get this JSONUtils fixed ?
I would like to backport this patch to CXF 2.7.8 , could you help me
to know the line(s) fixed ?
Regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

[Sergey Beryozkin]

Hi,
I've updated JSONProvider to pass the custom encoding if any to the
readers. But only on the trunk as I had to update few protected methods
and I think there may be few users customizing the jettison reader...

By the way, I checked
http://www.ietf.org/rfc/rfc4627.txt

It says only UTF-8 and other UTF-* encoding are supported.
Not sure if that is too limiting or not, I guess if the client or server
uses some other encoding with a charset parameter then we can support it...

Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Hi, sure, see changes at
https://issues.apache.org/jira/browse/CXF-5619

I was having a look at this bug and I've seen that you fixed it for
createStreamWriter() method( I tested it and it works fine )
but you didn't it for createStreamReader() method, any reason ?
I think the solution is in JSONUtils.java source class, in the line with
new JettisonReader(namespaceMap, factory.createXMLStreamReader(is));
Should it be factory.createXMLStreamReader(is, enc) ?
So I guess it should be a
String enc = HttpUtils.getSetEncoding(m, headers, "UTF-8");
in
T readFrom(Class<T> type, Type genericType, Annotation[] anns,
MediaType mt, MultivaluedMap<String, String> headers, InputStream is)
what about these methods
public static XMLStreamWriter createBadgerFishWriter(OutputStream os)
throws XMLStreamException
public static XMLStreamReader createBadgerFishReader(InputStream is)
throws XMLStreamException
?
Should they have a encoding parameter ?
Thanks and regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Jose María Zaragoza

Post by Sergey Beryozkin
Fortunately the bug is in CXF JSONUtils, whcih always passes UTF-8 to
Jettison. So it's now fixed, thanks for spotting it

Thank you
How can I get this JSONUtils fixed ?
I would like to backport this patch to CXF 2.7.8 , could you help me
to know the line(s) fixed ?
Regards

Post by Sergey Beryozkin
Cheers, Sergey

Post by Sergey Beryozkin
Looks like it is something related to Jettison...XML works OK.
Need to investigate more.
Thanks, Sergey

Post by Jose María Zaragoza
I've got to return "application/json;charset=iso-8859-1" but
i've checked that it always returns UTF-8 encoding
I tested with universalchardet by Mozilla , with Firefox browser (
changing View -> Encoding ) and with
InputStreamReader(connection.getInputStream(),"iso-8859-1"));
CODE:200
HEADER:{[HTTP/1.1 200 OK], Date=[Fri, 14 Mar 2014 12:26:13 GMT],
Transfer-Encoding=[chunked],
Content-Type=[application/json;charset=iso-8859-1],
Server=[Apache-Coyote/1.1]}
Detected encoding = UTF-8
Why is not encoding in iso-8859-1 ?
Regards

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com

--
Sergey Beryozkin
Talend Community Coders
http://coders.talend.com/
Blog: http://sberyozkin.blogspot.com