The character encoding was not declared proceeding using windows 1252

When you use Validate by URI, the server is supposed to announce the character encoding in HTTP headers, more exactly in the charset parameter of the Content-Type header value. In this case, this apparently does not happen. You can check the situation, e.g., using Rex Swain’s HTTP Viewer.

According to clause 4.2.5.5 Specifying the document’s character encoding in HTML5 CR, “If an HTML document does not start with a BOM, and its encoding is not explicitly given by Content-Type metadata, and the document is not an iframe srcdoc document, then the character encoding used must be an ASCII-compatible character encoding, and the encoding must be specified using a meta element with a charset attribute or a meta element with an http-equiv attribute in the Encoding declaration state.” This is a bit complicated, but the bottom line is: there are several ways to declare the encoding, but if none of them is used, the document is non-conforming.

Why it specifies so is somewhat speculative, but the general idea is that such rules promote reliability and robustness. When the rule is not obeyed, different browsers may use different defaults or guesswork.

The validator assumes Windows-1252, because that’s what HTML5 rules lead to. The processing rules are in 8.2.2.1 Determining the character encoding. They are fairly complicated, but they largely reflect the way modern browsers do (and aims at making it a standard). The rules there are meant to deal with non-conforming documents, too, but this does not make those documents conforming; error processing rules are not really “fallbacks” and should not be relied on, especially since old browsers do not always play by the rules.

The error rules get somewhat loose when it comes to a situation where everything else fails and an “implementation-defined or user-specified default character encoding” is to be used. There are just “suggestions” on what browsers might do (again, reflecting what modern browsers generally do), and this may involve using the “user’s locale”, an obscure concept. The validator uses Windows-1252 then, perhaps because that’s the default for English and the validator “speaks” English, or maybe just because it’s the guess that is expected to be correct more often than any other single alternative.

Why is this HTML5 document invalid?

I’m getting pretty confused about an error message I’m getting when I try to validate any simple HTML document without a meta encoding like this:

The W3C validator http://validator.w3.org reluctantly accepts the document as valid with just a few warnings when it is pasted into the direct input form, but when the document is uploaded or loaded by URI, validation fails with this error message

The character encoding was not declared. Proceeding using windows-1252.

There are two things I don’t understand about this error:

Can someone explain these two points please? I’m pretty new to this stuff, so please bear with me.

4 Answers 4

Well, it depends on what you are using.

If you don’t want the validator to guess, and use UTF-8, you can add the following line

It is the «Direct Input» mode of the validator that defaults to UTF-8. User-agents (browsers) will default to other encodings based on a number of things:

If a user agent reads a document with no character encoding information, it can fall back to using some other information. For example, it can rely on the user’s settings, either browser-wide or specific for a given document, or it can pick a default encoding based on the user’s language. For Western European languages, it is typical and fairly safe to assume Windows-1252, which is similar to ISO-8859-1 but has printable characters in place of some control codes.

W3C validator said:

The validator checked your document with an experimental feature: HTML5 Conformance Checker. This feature has been made available for your convenience, but be aware that it may be unreliable, or not perfectly up to date with the latest development of some cutting-edge technologies.

So take some results with a pinch of salt.

Also, there is no useful ‘fall back’, the validator just needs to pick something/anything so it can try to validate for you. W3C can’t determine/decide what encoding you want/need to use. You must declare it yourself based on what characters you need to serve on your web page(s), and then ask W3C to validate your document based on that.

What editor/WYSIWYG are you using to make web pages? Can we have the URL you are trying to validate?

When you use Validate by URI, the server is supposed to announce the character encoding in HTTP headers, more exactly in the charset parameter of the Content-Type header value. In this case, this apparently does not happen. You can check the situation e.g. using Rex Swain’s HTTP Viewer.

According to clause 4.2.5.5 Specifying the document’s character encoding in HTML5 CR, “If an HTML document does not start with a BOM, and its encoding is not explicitly given by Content-Type metadata, and the document is not an iframe srcdoc document, then the character encoding used must be an ASCII-compatible character encoding, and the encoding must be specified using a meta element with a charset attribute or a meta element with an http-equiv attribute in the Encoding declaration state.” This is a bit complicated, but the bottom line is: there are several ways to declare the encoding, but if none of them is used, the document is non-conforming.

Why it specifies so is somewhat speculative, but the general idea is that such rules promote reliability and robustness. When the rule is not obeyed, different browsers may use different defaults or guesswork.

The validator assumes windows-1252, because that’s what HTML5 rules lead to. The processing rules are in 8.2.2.1 Determining the character encoding. They are fairly complicated, but they largely reflect the way modern browsers do (and aims at making it a standard). The rules there are meant to deal with non-conforming documents, too, but this does not make those documents conforming; error processing rules are not really “fallbacks” and should not be relied on, especially since old browsers do not always play by the rules.

The error rules get somewhat loose when it comes to a situation where everything else fails and an “implementation-defined or user-specified default character encoding” is to be used. There are just “suggestions” on what browsers might do (again, reflecting what modern browsers generally do), and this may involve using the “user’s locale”, an obscure concept. The validator uses windows-1252 then, perhaps because that’s the default for English and the validator “speaks” English, or maybe just because it’s the guess that is expected to be correct more often than any other single alternative.

Источник

I just noticed that there is a warning message pops up when I view my mootool.js script on FireFox browser.

The warning message is

The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.

does that mean I have to add a Charset or something? but it is a script!!

Is there a solution for this?

12 Answers 12

In your HTML it is a good pratice to provide the encoding like using the following meta like this for example:

But your warning that you see may be trigged by one of multiple files. it might not be your HTML document. It might be something in a javascript file or css file. if you page is made of up multiples php files included together it may be only 1 of those files.

I dont think this error has anything to do with mootools. you see this message in your firefox console window. not mootools script.

maybe you simply need to re-save your html pages using a code editor that lets you specify the correct character encoding.

What web server are you using? Are you sure you are not requesting a non-existing page (404) that responds poorly?

Check your URL’s protocol.

You will also see this error if you host an encrypted page (https) and open it as plain text (http) in Firefox.

Источник

Declaring character encodings in HTML

Intended audience: HTML authors (using editors or scripting), script developers (PHP, JSP, etc.), Web project managers, and anyone who needs an introduction to how to declare the character encoding of their HTML file.

Question

How should I declare the encoding of my HTML file?

You should always specify the encoding used for an HTML or XML page. If you don’t, you risk that characters in your content are incorrectly interpreted. This is not just an issue of human readability, increasingly machines need to understand your data too. A character encoding declaration is also needed to process non-ASCII characters entered by the user in forms, in URLs generated by scripts, and so forth. This article describes how to do this for an HTML file.

Quick answer

Always declare the encoding of your document using a meta element with a charset attribute, or using the http-equiv and content attributes (called a pragma directive). The declaration should fit completely within the first 1024 bytes at the start of the file, so it’s best to put it immediately after the opening head tag.

You should always use the UTF-8 character encoding. (Remember that this means you also need to save your content as UTF-8.) See what you should consider if you really cannot use UTF-8.

If you have access to the server settings, you should also consider whether it makes sense to use the HTTP header. Note however that, since the HTTP header has a higher precedence than the in-document meta declarations, content authors should always take into account whether the character encoding is already declared in the HTTP header. If it is, the meta element must be set to declare the same encoding.

You can detect any encodings sent by the HTTP header using the Internationalization Checker.

Details

What about the byte-order mark?

If you have a UTF-8 byte-order mark (BOM) at the start of your file then recent browser versions other than Internet Explorer 10 or 11 will use that to determine that the encoding of your page is UTF-8. It has a higher precedence than any other declaration, including the HTTP header.

You could skip the meta encoding declaration if you have a BOM, but we recommend that you keep it, since it helps people looking at the source code to ascertain what the encoding of the page is.

Should I declare the encoding in the HTTP header?

Use character encoding declarations in HTTP headers if it makes sense, and if you are able, for any type of content, but in conjunction with an in-document declaration.

Content authors should always ensure that HTTP declarations are consistent with the in-document declarations.

Pros and cons of using the HTTP header

One advantage of using the HTTP header is that user agents can find the character encoding information sooner when it is sent in the HTTP header.

The HTTP header information has the highest priority when it conflicts with in-document declarations other than the byte-order mark. Intermediate servers that transcode the data (ie. convert to a different encoding) could take advantage of this to change the encoding of a document before sending it on to small devices that only recognize a few encodings. It is not clear that this transcoding is much used nowadays. If it is, and it is converting content to non-UTF-8 encodings, it runs a high risk of loss of data, and so is not good practice.

On the other hand, there are a number of potential disadvantages:

It may be difficult for content authors to change the encoding information for static files on the server – especially when dealing with an ISP. Authors will need knowledge of and access to the server settings.

Server settings may get out of synchronization with the document for one reason or another. This may happen, for example, if you rely on the server default, and that default is changed. This is a very bad situation, since the higher precedence of the HTTP information versus the in-document declaration may cause the document to become unreadable.

There are potential problems for both static and dynamic documents if they are not read from a server; for example, if they are saved to a location such as a CD or hard disk. In these cases any encoding information from an HTTP header is not available.

Similarly, if the character encoding is only declared in the HTTP header, this information is no longer available for files during editing, or when they are processed by such things as XSLT or scripts, or when they are sent for translation, etc.

So should I use this method?

If serving files via HTTP from a server, it is never a problem to send information about the character encoding of the document in the HTTP header, as long as that information is correct.

On the other hand, because of the disadvantages listed above we recommend that you should always declare the encoding information inside the document as well. An in-document declaration also helps developers, testers, or translation production managers who want to visually check the encoding of a document.

(Some people would argue that it is rarely appropriate to declare the encoding in the HTTP header if you are going to repeat it in the content of the document. In this case, they are proposing that the HTTP header say nothing about the document encoding. Note that this would usually mean taking action to disable any server defaults.)

Working with polyglot and XML formats

XHTML5: An XHTML5 document is served as XML and has XML syntax. XML parsers do not recognise the encoding declarations in meta elements. They only recognise the XML declaration. Here is an example:

The XML declaration is only required if the page is not being served as UTF-8 (or UTF-16), but it can be useful to include it so that developers, testers, or translation production managers can visually check the encoding of a document by looking at the source.

Since a polyglot document must be in UTF-8, you don’t need to, and indeed must not, use the XML declaration. On the other hand, if the file is to be read as HTML you will need to declare the encoding using a meta element, the byte-order mark or the HTTP header.

If you use the meta element with a charset attribute this is not something you need to consider.

Additional information

The information in this section relates to things you should not normally need to know, but which are included here for completeness.

Working with non-UTF-8 encodings

Using UTF-8 not only simplifies authoring of pages, it avoids unexpected results on form submission and URL encodings, which use the document’s character encoding by default. If you really can’t avoid using a non-UTF-8 character encoding you will need to choose from a limited set of encoding names to ensure maximum interoperability and the longest possible term of readability for your content.

Although these are normally called names, in reality they refer to the encodings, not the character sets. For example, the Unicode character set or ‘repertoire’ can be encoded in three different encoding schemes.

Until recently the IANA registry was the place to find names for encodings. The IANA registry commonly includes multiple names for the same encoding. In this case you should use the name designated as ‘preferred’.

The new Encoding specification now provides a list that has been tested against actual browser implementations. You can find the list in the table in the section called Encodings. It is best to use the names in the left column of that table.

Working with legacy HTML formats

HTML 4.01 doesn’t specify the use of the charset attribute with the meta element, but any recent major browser will still detect it and use it, even if the page is declared to be HTML4 rather than HTML5. This section is only relevant if you have some other reason than serving to a browser for conforming to an older format of HTML. It describes any differences from the Answer section above.

HTML4: As mentioned just above, you need to use the pragma directive for full conformance with HTML 4.01, rather than the charset attribute.

XHTML 1.x served as text/html: Also needs the pragma directive for full conformance with HTML 4.01, rather than the charset attribute. You do not need to use the XML declaration, since the file is being served as HTML.

XHTML 1.x served as XML: Use the encoding declaration of the XML declaration on the first line of the page. Ensure there is nothing before it, including spaces (although a byte-order mark is OK).

The charset attribute on a link

It was intended for use on an embedded link element like this:

The idea was that the browser would be able to apply the right encoding to the document it retrieves if no encoding is specified for the document in any other way.

There were always issues with the use of this attribute. Firstly, it is not well supported by major browsers. One reason not to support this attribute is that if browsers do so without special additional rules it would be an XSS attack vector. Secondly, it is hard to ensure that the information is correct at any given time. The author of the document pointed to may well change the encoding of the document without you knowing. If the author still hasn’t specified the encoding of their document, you will now be asking the browser to apply an incorrect encoding. And thirdly, it shouldn’t be necessary anyway if people follow the guidelines in this article and mark up their documents properly. That is a much better approach.

This way of indicating the encoding of a document has the lowest precedence (ie. if the encoding is declared in any other way, this will be ignored). This means that you couldn’t use this to correct incorrect declarations either.

Working with UTF-16

According to the results of a Google sample of several billion pages, less than 0.01% of pages on the Web are encoded in UTF-16. UTF-8 accounted for over 80% of all Web pages, if you include its subset, ASCII, and over 60% if you don’t. You are strongly discouraged from using UTF-16 as your page encoding.

If, for some reason, you have no choice, here are some rules for declaring the encoding. They are different from those for other encodings.

The HTML5 specification forbids the use of the meta element to declare UTF-16, because the values must be ASCII-compatible. Instead you should ensure that you always have a byte-order mark at the very start of a UTF-16 encoded file. In effect, this is the in-document declaration.

The XML declaration

This is significant, because if you decide to omit the XML declaration you must choose either UTF-8 or UTF-16 as the encoding for the page if it is to be used without HTTP!

You should use an XML declaration to specify the encoding of any XHTML 1.x document served as XML.

It can be useful to use an XML declaration for web pages served as XML, even if the encoding is UTF-8 or UTF-16, because an in-document declaration of this kind also helps developers, testers, or translation production managers ascertain the encoding of the file with a visual check of the source code.

Using the XML declaration for XHTML served as HTML. XHTML served as HTML is parsed as HTML, even though it is based on XML syntax, and therefore an XML declaration should not be recognized by the browser. It is for this reason that you should use a pragma directive to specify the encoding when serving XHTML in this way*.

* Conversely, the pragma directive, though valid, is not recognized as an encoding declaration by XML parsers.

On the other hand, the file may also be used at some point as input to other processes that do use XML parsers. This includes such things as XML editors, XSLT transformations, AJAX, etc. In addition, sometimes people use server-side logic to determine whether to serve the file as HTML or XML. For these reasons, if you aren’t using UTF-8 or UTF-16 you should add an XML declaration at the beginning of the markup, even if it is served to the browser as HTML. This would make the top of a file look like this:

If you are using UTF-8 or UTF-16, however, there is no need for the XML declaration, especially as the meta element provides for visual inspection of the encoding by people processing the file.

Catering for older browsers. If anything appears before the DOCTYPE declaration in Internet Explorer 6, the page is rendered in quirks mode. If you are using UTF-8 or UTF-16 you can omit the XML declaration, and you will have no problem.

If, however, you are not using these encodings and Internet Explorer 6 users still count for a significant proportion of your readers, and if your document contains constructs that are affected by the difference between standards mode vs. quirks mode, then this may be an issue. If you want to ensure that your pages are rendered in the same way on all standards-compliant browsers, you will have to add workarounds to your CSS to overcome the differences.

There may also be some other rendering issues associated with an XML declaration, though these are probably only an issue for much older browsers. The XHTML specification warns that processing instructions are rendered on some user agents. Also, some user agents interpret the XML declaration to mean that the document is unrecognized XML rather than HTML, and therefore may not render the document as expected. You should do testing on appropriate user agents to decide whether this will be an issue for you.

Of course, as mentioned above, if you use UTF-8 or UTF-16 you can omit the XML declaration and the file will still work as XML or HTML. This is probably the simplest solution.

Источник

Character with encoding UTF8 has no equivalent in WIN1252

I am getting the following exception:

Is there a way to eradicate such characters, either via SQL or programmatically?
(SQL solution should be preferred).

I was thinking of connecting to the DB using WIN1252, but it will give the same problem.

9 Answers 9

More info is available on the PostgreSQL wiki under Character Set Support (devel docs).

Try to open this file in for example Notepad, save-as it in ANSI encoding and add (or replace similar) set client_encoding to ‘WIN1252’ line in your file.

Don’t eridicate the characters, they’re real and used for good reasons. Instead, eridicate Win1252.

I had a very similar issue. I had a linked server from SQL Server to a PostgreSQL database. Some data I had in the table I was selecting from using an openquery statement had some character that didn’t have an equivalent in Win1252. The problem was that the System DSN entry (to be found under the ODBC Data Source Administrator) I had used for the connection was configured to use PostgreSQL ANSI(x64) rather than PostgreSQL Unicode(x64). Creating a new data source with the Unicode support and creating a new modified linked server and refernecing the new linked server in your openquery resolved the issue for me. Happy days.

That looks like the byte sequence 0xBD, 0xBF, 0xEF as a little-endian integer. This is the UTF8-encoded form of the Unicode byte-order-mark (BOM) character 0xFEFF.

I’m not sure what Postgre’s normal behaviour is, but the BOM is normally used only for encoding detection at the beginning of an input stream, and is usually not returned as part of the result.

In any case, your exception is due to this code point not having a mapping in the Win1252 code page. This will occur with most other non-Latin characters too, such as those used in Asian scripts.

Can you change the database encoding to be UTF8 instead of 1252? This will allow your columns to contain almost any character.

Источник

C++ Visual Studio character encoding issues

Not being able to wrap my head around this one is a real source of shame.

I’m working with a French version of Visual Studio (2008), in a French Windows (XP). French accents put in strings sent to the output window get corrupted. Ditto input from the output window. Typical character encoding issue, I enter ANSI, get UTF-8 in return, or something to that effect. What setting can ensure that the characters remain in ANSI when showing a «hardcoded» string to the output window?

Will show in the output:

(here encoded as HTML for your viewing pleasure)

I would really like it to show:

8 Answers 8

Before I go any further, I should mention that what you are doing is not c/c++ compliant. The specification states in 2.2 what character sets are valid in source code. It ain’t much in there, and all the characters used are in ascii. So. Everything below is about a specific implementation (as it happens, VC2008 on a US locale machine).

To start with, you have 4 chars on your cout line, and 4 glyphs on the output. So the issue is not one of UTF8 encoding, as it would combine multiple source chars to less glyphs.

From you source string to the display on the console, all those things play a part:

1 and 2 are fairly easy ones. It looks like the compiler guesses what format the source file is in, and decodes it to its internal representation. It generates the string literal corresponding data chunk in the current codepage no matter what the source encoding was. I have failed to find explicit details/control on this.

3 is even easier. Except for control codes, just passes the data down for char *.

5 is a funny one. I banged my head to figure out why I could not get the é to show up properly, using CP1252 (western european, windows). It turns out that my system font does not have the glyph for that character, and helpfully uses the glyph of my standard codepage (capital Theta, the same I would get if I did not call SetConsoleOutputCP). To fix it, I had to change the font I use on consoles to Lucida Console (a true type font).

Some interesting things I learned looking at this:

BTW, if what you got was «ÓÚÛ¨» rather than what you pasted, then it looks like your 4 bytes are interpreted somewhere as CP850.

Because I was requested to, I’ll do some necromancy. The other answers were from 2009, but this article still came up on a search I did in 2018. The situation today is very different. Also, the accepted answer was incomplete even back in 2009.

The Source Character Set

Every compiler (including Microsoft’s Visual Studio 2008 and later, gcc, clang and icc) will read UTF-8 source files that start with BOM without a problem, and clang will not read anything but UTF-8, so UTF-8 with a BOM is the lowest common denominator for C and C++ source files.

The language standard doesn’t say what source character sets the compiler needs to support. Some real-world source files are even saved in a character set incompatible with ASCII. Microsoft Visual C++ in 2008 supported UTF-8 source files with a byte order mark, as well as both forms of UTF-16. Without a byte order mark, it would assume the file was encoded in the current 8-bit code page, which was always a superset of ASCII.

The Execution Character Sets

The way Visual C and C++ violate the language standard is by making their wchar_t UTF-16, which can only represent some characters as surrogate pairs, when the standard says wchar_t must be a fixed-width encoding. This is because Microsoft defined wchar_t as 16 bits wide back in the 1990s, before the Unicode committee figured out that 16 bits were not going to be enough for the entire world, and Microsoft was not going to break the Windows API. It does support the standard char32_t type as well.

UTF-8 String Literals

The third issue this question raises is how to get the compiler to encode a string literal as UTF-8 in memory. You’ve been able to write something like this since C++11:

If it would be too inconvenient to type a character in, or if you want to distinguish between superficially-identical characters such as space and non-breaking space or precomposed and combining characters, you also have universal character escapes:

You can use these regardless of the source character set and regardless of whether you’re storing the literal as UTF-8, UTF-16 or UCS-4. They were originally added in C99, but Microsoft supported them in Visual Studio 2015.

There is another way to do this that worked in Visual C or C++ 2008, however: octal and hexadecimal escape codes. You would have encoded UTF-8 literals in that version of the compiler with:

Источник

Welcome to the Treehouse Community

The Treehouse Community is a meeting place for developers, designers, and programmers of all backgrounds and skill levels to get support. Collaborate here on code errors or bugs that you need feedback on, or asking for an extra set of eyes on your latest project. Join thousands of Treehouse students and alumni in the community today. (Note: Only Treehouse students can comment or ask questions, but non-students are welcome to browse our conversations.)

Looking to learn something new?

Treehouse offers a seven day free trial for new students. Get access to thousands of hours of content and a supportive community. Start your free trial today.

seal-mask

Hello! I recently submitted a project after running it through the linked W3C HTML validator as well as the W3C CSS validator and both showed no errors.

I just got my review results back though, and they’ve claimed the following

---

Unfortunately, there is an error when running the html through the validator. :

«Error: The character encoding was not declared. Proceeding using windows-1252.»

No errors in the CSS. If you fix this, it will be perfect.

I ran it through the validators again and I’m not getting this error. :-/

Is my reviewer using some other validator than the one linked in the project description or what am I doing wrong?

3 Answers

The validator you used probably guessed what character encoding was being used. Where your reviewers did not guess correctly. If you add the following in your <head> tag you should be fine:

This is based off what your reviewer said.

seal-mask

Thank you so much! I’ve added that line and am now resubmitting.

This should really be taught in the early parts of the course or at the very least the reviewers should be using the same validator that is linked in the project description. That line wasn’t in any of my previous projects and there wasn’t a problem. I hope this «Needs work» doesn’t go against my final results!

Thanks again Matthew! Much appreciated

it is better to just copy your codes and past in W3 web consortium that can be very helpful .