Length of (exotic) string

Open Discussions Summit Questions
1

Hello everybody,

I want to show the name and size of files, with columns of regular width.

This is what I tried:

Get-ChildItem -Path C:\Ant -File -ea SilentlyContinue | Select-Object Directory, @{n="Name";e={($_.Name.Substring(0,30))}}, @{n="."; e="."}, @{n="Size".PadLeft(30); e={($_.Length.ToString("N0").PadLeft(30))}}

This works pretty good for most files, but some are written with far-east characters, or have graphic characters in their names, such as for instance :rabbit:

For these, the string obtained by

 @{n="Name";e={($_.Name.Substring(0,30))}}

gives a longer output on screen.

I imagine those characters are coded on more bytes each, and with Substring I select bytes, and not characters.

How could I select characters properly?

And I tried to insert a separator column to better identify in what column was the problem, but only the title of it appeared. Any suggestion?

2

I don’t think that PowerShell is the problem here. The name of the file is in Unicode (stored in UTF-16, or two cells) and the .Substring retrieves 30 “characters”, some of which contain characters that use more than 8-bits to be displayed correctly. So your 30 “characters” (of 16 bits each) may be interpreted by the console as being longer that 30 “characters”.

The solution is to use a Windows Terminal or a console character set that fully supports CJK coverage (SimSun-ExtB, MS Gothic, MS Mincho, or Noto Sans CJK (which I think Google can supply but you have to install manually)).

You can also try setting the console to use UTF8encoding, but that probably won’t help with the inability to display the characters correctly (because of the font).

3

substring is already characters, not bytes. the issue is display width, wide and emoji characters take 2 columns on screen, so 30 chars still prints wider.

4

Oh, so you mean the terminal has to be replaced by another product for this?

Well, thank you.

5

They only occupy two “columns” on screen if the interpretation of the UTF-16 character is treated as if it were two 8-bit characters.

$a = "ABC鿿"
$a | format-hex -encoding unicode

$a.ToCharArray() | ForEach-Object {
    "{0}  U+{1:X4}" -f $_, [int]$_
}

The Format-Hex treats the output as single characters and shows the last multi-byte character as if it were two ASCII characters.

The snippet of PowerShell code treats the multi-byte character properly (depending on the font). Both examples were run on VS Code and directly from PowerShell.

EDIT: Remove extraneous “\” character from the previous “[int\]”.

6

If you mean “does the physical device” need to be replaced, the answer is “no”.

It would be helpful if you could provide an example of one of those “exotic strings”. The code below would do that. You just need to replace the “some-file-name” with the name of one of the files that’s causing the problem.

(Get-ChildItem C:\Ant\some-file-name -File).Name | Format-Hex
7

I obtain this:

A  U+0041
B  U+0042
C  U+0043
鿿  U+9FFF

Oh, interesting indeed. I then have to treat the display, the night can bring good advice …

Thank you.

8

You do not use Windows, do you?

In Windows the Terminal is a piece of software, that proposes a few display modes with corresponding functionalities, including the command line, and PowerShell. If I understood properly you proposed me to replace it by SimSun-ExtB, MS Gothic, MS Mincho, or Noto Sans CJK.

What was said recently here gives me an idea, to test after spending some time on it.

9

Well, “Windows Terminal” is software “the terminal” usually refers to a hardware device – especially if you’ve been working with computers since the 1960s!

10

Those are the names of fonts that are used to map the bits to a character on the display. The entire set of Unicode is rarely needed (and is quite large). Each font usually concentrates on a particular set of symbols for different alphabets. E.g., Crylic, Greek, etc.

11

On the separator column, the expression in a calculated property has to be a script block, so use e={‘.’} instead of e=‘.’. That is why only the header showed up. For the width problem, Substring counts UTF-16 code units, and wide or emoji characters can still take two screen cells, so manual padding will never line up perfectly. If you only need aligned columns, let PowerShell do the layout with something like:

```

Get-ChildItem C:\Ant -File | Format-Table Name, @{n=‘Size’;e={$_.Length.ToString(‘N0’)};a=‘right’} -AutoSize

```

That avoids cutting names at a fixed length, and it copes better with wide characters than PadLeft does.

12

Hello,

As perhaps you saw, I limited the width of the Name column because some files have very long names and the other columns did not fit in the width of the screen.

Some work is needed on this for strings containing characters longer than a byte, I am working on this.

Very good answer about the separator column, thank you very much.

13

If you’re interested, you can install the PowerShell module “PSWriteColorEX” and use the method “Measure-DisplayWidth” to find the number of positions occupied by the Unicode character on the display.

Install-Module PSWriteColorEX
Measure-DisplayWidth "ABC鿿"

As you’re discovering, Unicode is a complex topic! For example, there are codepoints that occupy zero positions (zero-width joiners), rules such as “Unicode East Asian Width”, grapheme clusters, and more.

14

Thank you. Oh, for sure it is quicker than what I was about to do, measure each character and calculate the total.

I still have to browse the whole list to evaluate the one that needs the more spaces, and then for each deduce the number of extra spaces used.

But it is already a good thing to avoid browsing each character.

15

It is not really that the topic is very complicated indeed, but if you have to keep an index for the characters and another for the lines, it is good to avoid confusing between them.

16

Hello,

I should like to know what you think of this.

$a = (Get-ChildItem "C:\Ant\videos-mdr.com - Un toboggan vraiment flippant _ il propulse à la verticale ! - Vidéos MDR.mp4").Name.Substring(0,30);

$b = (Get-ChildItem "C:\Ant\1080p - ❤️🍑🍆 Cœur, Pêche, Aubergine  Ados, Vie Sexuelle & Affective*.*").Name.Substring(0,30);

$a;$b;Measure-DisplayWidth($a);Measure-DisplayWidth($b);

#Answer:

videos-mdr.com - Un toboggan v
1080p - ❤️🍑🍆 Cœur, Pêche, Au
30
30

Not really vertically aligned … Although the displaywidth is the same.

The characters that show fruits take the width of three letters, whether they are attributed a displaywidth of two. Should I use a more appropriate font?

In ‘Courier New’ $b ends at the g of $a. The length comparison is reversed, and with a bigger difference. Unexpected, no?

With MsGothic $b still has “Au” to display where $a has finished.

I obtain something nearly aligned with MsGothic if I give a character width of 20pt for the letters and figures, and 16pt for the characters that show fruits.

I thought I could avoid to examine each character separately, but if this is the solution to target, I have to take the hexadecimal representation in account, if the upper code is different from 0 I must give a more little width. And I did not have a look at the Korean characters yet.

I wonder whether getting the position of the keyboard cursor after displaying each character would be a good idea. It is theoretically what Measure-DisplayWidth intended to avoid.

In the Windows API there is something to know the display width, that takes the font in account. It is sad to call the Windows API if we have a DisplayWidth function in PowerShell, but if it gives better results … I have to precise that I did not try it in this particular case yet.

17

For the provided sample, it works quite good with this:

Get-ChildItem C:\Ant -File | Format-Table @{n=‘Size’;e={$_.Length.ToString(‘N0’)};width=10;a=‘right’},@{n='Dir'; e={$_.Directory}}, @{n='!';e={'.'}}, @{n='Name'; e={$_.Name.Substring(0,28)};width=30},@{n='!';e={'.'}} -AutoSize

Well, next I talk to you about the Asiatic characters …

18

After telling I should talk about the Asiatic characters, I realized they are coded on 4 bytes each. Initially the width of the name column exceeded the other lines of about 10 characters, so I fixed a column width of 4 times the number of bytes. This time, the column width only exceeds the width of the other lines of 1.

Get-ChildItem C:\Ant -File | Format-Table @{n=‘Size’;e={$_.Length.ToString(‘N0’)};width=10;a=‘right’},@{n='Dir'; e={$_.Directory};w=10;}, @{n='!';e={'.'}}, @{n='Name'; e={$_.Name.Substring(0,15)};width=60},@{n='!';e={'.'}}

By putting the name at the end of the line, I see the alignment problem only with the ‘.’ at the end that is shifted of one position, which is less a nuisance than seeing the size not aligned.

If you think we can still improve the results …

***

I realize that it was not logical to give a width of 4 times the number of bytes, for the number of characters is the fourth of the number of bytes. It is not logical, but on the screen it gave a better results. So, I arrived at it by accident, thanks to a logical error. The reasoning was based on less rational arguments: I saw that line showing the name on a much longer width, without having really understood why.