Javascript smart quotes database replacement

Javascript smart quotes database replacement how to#

Unfortunately, you probably found this problem because a bunch of files or database records had badly encoded data in it. Then, use force_encoding to force that mis-encoded Windows-1252 string to be read as UTF-8: irb(main):006:0> 'theyГўв‚¬в„ўre'. Use encode to convert the UTF-8 string back into a Windows-1252 string. To fix it, you just have to follow those backwards steps. Whose bytes should have been read as UTF-8 (theyвЂ™re) How do you get things back to normal? LetвЂ™s think about the problem backwards:Д¬onverted from a Windows-1252 string, (theyГўв‚¬в„ўre) And if you didnвЂ™t know how it happened, itвЂ™d be almost impossible to untangle it. Those broken characters now look like theyвЂ™re supposed to be there. Your string has been badly-encoded twice. Ruby will helpfully convert it to UTF-8 for you, so youвЂ™ll end up with: irb(main):004:0> 'theyвЂ™re'. But youвЂ™ll probably save that data to a database, or a file, as UTF-8. The data is supposed to be UTF-8, but is being misread as Windows-1252. encode('UTF-8') to display the string in the console.) You can check this in irb: irb(main):003:0> 'theyвЂ™re'. So, which encoding would represent as Гўв‚¬в„ў? If you look at a few tables of popular encodings, youвЂ™ll see itвЂ™s Windows-1252. Instead, each byte is showing up as a different character. It seems like those three bytes should be read as UTF-8, where theyвЂ™d represent a curly quote. That seems like more than a coincidence, right? And our messed up string, theyГўв‚¬в„ўre, has three characters where it should just have one. When you focus on just the curly quote: irb(main):002:0> 'вЂ™'. bytes => Д®ven though the string only has 7 characters, theyвЂ™re represented by 9 bytes! So some characters, like the curly quote вЂ™, are represented with more than one byte: irb(main):001:0> 'theyвЂ™re'. Not every character can be represented in a single byte, because there are more than 256 possible characters. Last week, you learned that an encoding is just a way to turn groups of meaningless bytes into displayable characters. You could create a giant table, so you could find bad characters and replace them with good ones: Д«ut thereвЂ™s an easier, more reliable way to fix those broken characters.

$javascript smart quotes database replacement$

(Or maybe youГўв‚¬в„ўve seen it?) ItвЂ™s when a curly quote turns into Гўв‚¬в„ў, or an em-dash turns into Гўв‚¬вЂќ. But thereвЂ™s one encoding problem thatвЂ™s much harder to solve.

When youвЂ™ve concluded that the changes are correct (do a diff File.txt to see the difference), you can delete all the backup files with rm *.bk.In last weekвЂ™s article, you learned a short process that solves most encoding problems. To fix the smart quotes in all the text files in a directory, do the following:ДЇor i in *.txt do sed -i '.bk' s//'''/g $i doneДЄt the conclusion of the command, you will have double the number of text files in the directory, due to all the backup files. This tells the sed command to make the change вЂњin placeвЂќ, while backing up the original file to in case anything goes wrong. If you wish to save the files in place, overwriting the original contents, you would do Sed s//'''/g File.txt > WithoutSmartQuotes.txt This wonвЂ™t actually change the contents of the File, but you can save the results to a new file Recall that you can do global find/replace by using sed.

Javascript smart quotes database replacement how to#

IвЂ™ll show you how to replace these curly quotes with the traditional straight quote. You can turn them off in Word but if youвЂ™re trying to remedy the problem after the fact, sed is your old friend. Some programs donвЂ™t handle these characters very well. If youвЂ™ve copied and pasted text from Microsoft Word, chances are there will be the so-called smart quotes in that text.