Wikipedia:Reference desk/Archives/Computing/2013 March 6#Video codecs for future storage

Part 1

I'm getting several hours of super 8 film digitized. I want to keep it in a format that strikes a good balance between keeping the amount of storage space needed at a managable level, and preserving as much as possible of the information in the video. I've had tests done at several companies who specialize in this, and the company so far that produces the best digitizations, uses [http://www.mwa-nova.com/downloads/product_sheets/flashscan_HD.pdf this hardware], and a codec that VideoLan describes thusly:

Stream: 0

Type: Video

Codec: Packed YUV 4:2:2, U:Y:V:V (2vuy)

Language: English

Resolution: 1280x720

Display resolution: 1280x720

Frame rate: 24

edit: YUV 4:4:2 corrected to YUV 4:2:2

When I calculate the total number of pixels encoded, the file size is about 2.7 bytes per pixel.

• Do we have an article about this codec? I've read Chroma subsampling, but anything more specific?
• Is this a codec that can be expected to be readable a long time into the future?
• In this particular codec, are all frames represented equally faithful to the original, or are there key frames that are represented more faithfully, with diffs in between which use some kind of lossy compression with respect to the key frames? To clarify: The details in the encoding scheme, i.e. whether key frames and diffs are used or whether information is stored a sequence of entire images, does not matter to me. If the diffs are exact, it's fine.
• I'm happy with a file size of up to about 4 bytes per pixel. Are there codecs that are widely used, can be expected to be readable a long time in the future, and which can be manipulated with ffmpeg (Windows version from http://ffmpeg.zeranoe.com/builds/), that would be a better choice than the one described above?

Part 2

• For some reason, the company mentioned above only delivers the files in .MOV containers. Can I convert to .AVI losslessly with ffmpeg, and safely delete the original?
• Will ffmpeg be able to split longer files into shorter sequences losslessly?

Thanks, --NorwegianBlue ^talk 07:42, 6 March 2013 (UTC)

:1a. I can't find a specific article but YUV has more information. 1b. Yes. 1c. It is uncompressed; you lose nothing except half the horizontal color resolution, which is normally not noticeable and which virtually all other codecs lose as well. Every frame is a key frame. 1d. 8-bit YUV 4:2:2 is two bytes per pixel. You could ask about 24-bit RGB, which would be 3 bytes per pixel, but it's possible that YUV 4:2:2 is the raw output format of their digitizer and converting to anything else would lose information. You could also ask about more than 8 bits per channel, but I think you would have more compatibility problems. 2a. I think ffmpeg -i input.mov -vcodec copy output.avi will work. 2b. Probably. At any rate there are tools that can. -- BenRG (talk) 18:47, 6 March 2013 (UTC)

::Thanks a lot, BenRG. I wonder what the extra 0.7 bytes per pixel are? I just discovered Mediainfo, which reports the format like this:

{{collapse top}}

General
Complete name                            : MyFile.mov
Format                                   : MPEG-4
Format profile                           : QuickTime
Codec ID                                 : qt
File size                                : 8.41 GiB
Duration                                 : 3mn 23s
Overall bit rate                         : 355 Mbps
Encoded date                             : UTC 2013-01-24 16:33:18
Tagged date                              : UTC 2013-01-24 16:36:06
Writing library                          : Apple QuickTime
©TIM                                     : 00:00:00:00
©TSC                                     : 24
©TSZ                                     : 1
Video
ID                                       : 1
Format                                   : YUV
Codec ID                                 : 2vuy
Duration                                 : 3mn 23s
Bit rate mode                            : Constant
Bit rate                                 : 354 Mbps
Width                                    : 1 280 pixels
Height                                   : 720 pixels
Display aspect ratio                     : 16:9
Frame rate mode                          : Constant
Frame rate                               : 24.000 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Compression mode                         : Lossless
Bits/(Pixel*Frame)                       : 16.000
Stream size                              : 8.37 GiB (100%)
Title                                    : Apple aliasdatahåndterer
Language                                 : English
Encoded date                             : UTC 2013-01-24 16:33:18
Tagged date                              : UTC 2013-01-24 16:36:06
Audio
ID                                       : 2
Format                                   : PCM
Format settings, Endianness              : Little
Format settings, Sign                    : Signed
Codec ID                                 : sowt
Duration                                 : 3mn 23s
Bit rate mode                            : Constant
Bit rate                                 : 1 536 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Stream size                              : 37.2 MiB (0%)
Title                                    : Apple lydmediahåndterer / Apple aliasdatahåndterer
Language                                 : English
Encoded date                             : UTC 2013-01-24 16:33:18
Tagged date                              : UTC 2013-01-24 16:36:06
Other
ID                                       : 3
Type                                     : Time code
Format                                   : QuickTime TC
Duration                                 : 3mn 23s
Time code of first frame                 : 00:00:00:00
Time code settings                       : Striped
Title                                    : Tidskodemediehåndterer / Apple aliasdatahåndterer
Language                                 : English
Encoded date                             : UTC 2013-01-24 16:36:06
Tagged date                              : UTC 2013-01-24 16:36:06

{{collapse bottom}}

::Could a silent audio stream consume so much space? I interpret the "Bits/(Pixel*Frame) : 16.000" output as a confirmation of your statement that the yuv compression should result in 2 bytes per pixel.

::The conversion to .AVI that you suggested worked, sort of. The file played fine in VideoLan, I compared a PNG snapshot with the corresponding frame in the original, and found the the files to be identical (binary diff). However, this particular content in an AVI container crashed MediaInfo (but not ffprobe). --NorwegianBlue ^talk 23:27, 6 March 2013 (UTC)

:::You don't need a silent audio track whether it consumes a lot of space or not, include -an in your ffmpeg command. ¦ Reisio (talk) 00:09, 7 March 2013 (UTC)

::::I know I don't need it, but I'm puzzled by the size of the files as delivered by the company who did the digitization. I don't want to mess with the original files until I'm 110% sure what I'm doing, but I'll try stripping it as you suggested on a copy of one of the files, to see if that's the explanation. Thanks! --NorwegianBlue ^talk 08:41, 7 March 2013 (UTC)

:::::1280×720×24×16 = 353894400, which is consistent with the 354 Mbps rate quoted above, and 203s×354Mbps = 8.37 GB, so I think either you miscalculated or the file with the problem is a different one. Note that 1 Mbps = 10⁶ bps, but 1 GB = 2³⁰ B. The audio track accounts for less than 1% of the file size, as shown by Mediainfo. It's possible they include it because some playback software wasn't designed to handle files without an audio track. You could at least downsample it to 8 KHz or something. -- BenRG (talk) 18:31, 7 March 2013 (UTC)

::::::I indeed miscalculated, and was about to come here and say, slightly embarrassed, "mystery solved". I used the frame rate of the source material (18 fps), which was part of the file names, and completely ignored the blindingly obvious fact that they had inserted extra frames, to adjust to 24 fps instead of 18 fps. But it's a good thing you beat me to it, since in doing so, you also spelled out how other relevant calculations, which I was unsure about, are done. Thanks a lot! --NorwegianBlue ^talk 20:23, 7 March 2013 (UTC)

{{unresolved}}

I use Windows7 and Firefox v19 and/or Microsoft Internet Explorer v9.
I need to copy the text of all the available options inn a Combo box ("drop-down list") and paste them into my local text editor.
For example I would like to simultaneously copy all the options:

Volvo
Saab
Opel
Audi

from the combo box in the Result field on the web page: http://www.w3schools.com/tags/tryit.asp?filename=tryhtml_select (Link found and visited: 2013-03-06)
(Of course: In most cases there are tens or sometimes maybe hundreds of options in the combo boxes that I need to copy).
How may I do this?
--46.15.136.227 (talk) 17:20, 6 March 2013 (UTC)

:::(OP) Adding to my question at 23:21, 6 March 2013 (UTC) :
The example list (Volvo Saab Opel Audi), referred to above, only uses the HTML-tag: …, but of course I also sometimes need to copy a combo box-list that is given in a script, like the one on the web page http://www.oft.gov.uk/ (visited: 2013-03-06) under: "I am looking for..." (with the options: "Anti-money laundering", "OFT Board", "Current Campaigns", …).
The solutions suggested, below, so far (Thank you all!), namely:
1. Screenshot and OCR.
2. Viewing the page source, searching, cuting and pasting (with or without the Firefox Firebug-Add-on).
they both do work, but have prooven to be far too time consuming and cumbersome in most cases.
I guess, what I wish for is the ability to just riht-click the combo box-arrow and choose: "Copy the whole selection list to the clipboard".
-- (OP) 46.15.136.227 (talk) 23:21, 6 March 2013 (UTC)

:If all you have access to is the finished app, try this: It's messy, but a screen grab (or multiples, if it doesn't fit on one page) followed to OCR could do it. If you have access to the source code, then better methods are available. StuRat (talk) 18:06, 6 March 2013 (UTC)

: You could just view the page source and copy the options and their surrounding HTML, which you would then of course have to edit out, either by some suitable searching-and-replacing in a text editor, or perhaps a spreadsheet if there are extra complications. AndrewWTaylor (talk) 19:41, 6 March 2013 (UTC)

:I do stuff like that pretty often. Usually (? always? depends what browser I guess) there's a "view source" via a menu or rightclick or something. When you've got the HTML source, if you're lucky, you can search for one of the combo box entries and find the place where they're all specified. If you're not lucky, they might be elsewhere in some CSS file or something that's transcluded into the HTML, in which case I'm out of my depth and somebody else might be able to continue the advice. Gzuckier (talk) 19:46, 6 March 2013 (UTC)

::If you use Firebug (a Firefox addon) it's a simple matter of rightclicking the select box, "Inspect with FireBug", copy-pasting the source code there, and getting rid off the tags using search-and-replace. The inspect will work whether or not the dropdown was in the original source code, or generated by JavaScript. 82.75.247.59 (talk) 20:01, 6 March 2013 (UTC)

:I have expanded my original question a little (see above, just below my first entry).
-- (OP) 46.15.136.227 (talk) 23:21, 6 March 2013 (UTC)

:On a page with a lot of OPTION tags like that, you can paste something like this into the address bar of your browser:

::javascript:var o=document.getElementsByTagName("option"); var r=""; for(var i=0;i

:In theory, this will then give you a pop-up box with all of the text between the OPTION tags. You should then (on a modern browser, like Chrome) be able to copy those directly and paste them into your text editor. Note that this won't work exactly on that w3schools page, because those OPTION tags are embedded in an IFRAME. --Mr.98 (talk) 23:55, 6 March 2013 (UTC)

::If you did need to go inside of an IFRAME like the one on that page — or if you just want to experiment with it — you could use this code instead:

:::javascript:var o=document.getElementsByTagName("iframe")[1].contentWindow.document.getElementsByTagName("option"); var r=""; for(var i=0;i

::You would have to modify the number after the "iframe" (1 in this case) to refer to the right one (there are at least two IFRAMEs on that page, so in this case it is accessing the second one, because these are ordered starting from 0). --Mr.98 (talk) 00:03, 7 March 2013 (UTC)

openadserving.com malware

had this infesting me for a while now, but norton (as supplied by my isp and automatically updated) doesn't seem to do anything with it. I got it temporarily bypassed with the hosts file but would sure like to eradicate it from the system, but a quick googling comes up with a whole mess of different lists of files, registry keys, startup menu entries to delete, i.e. a different list from every source. So, does anyone know when any of the antivirus guys are going to get a handle on this, or why they can't? Gzuckier (talk) 21:28, 6 March 2013 (UTC)

:Have you tried [http://malwarebytes.org Malwarebytes] in safe mode? It's saved me from reinstalling Windows many times. I don't know if it will work in your case or not, but it's worth a shot. And from my experience/reading, Norton really isn't all that great - I would recommend [http://windows.microsoft.com/en-ca/windows/security-essentials-download Microsoft Security Essentials] for real-time protection. Both AV's I've linked are free. ProtossPylon 00:00, 7 March 2013 (UTC)

::This is just my personal opinion, but I'm not sure I'd call MalwareBytes an antivirus program.--Rockfang (talk) 18:33, 7 March 2013 (UTC)

:::What would you call it, then? That's the only description I can think of for it. ProtossPylon 23:10, 7 March 2013 (UTC)

Unix question

Hi. I have to survey a network of computers; there are Windows boxes, Mac boxes, various flavours of linux, Ubuntu and Kubuntu. One thing I need to do is establish how many hard disks (not partitons, actual physical drives) are inside each box. This is easy peasy for the Windows and Mac ones, but is there a simple fast and dirty way of doing it via a terminal on the others? I have some (but admittedly not a lot) experience of using bash. Cheers  pablo 23:40, 6 March 2013 (UTC)

:fdisk -l | grep 'Disk /' (as root / via sudo), /sbin/blkid, million other ways. :) ¦ Reisio (talk) 23:53, 6 March 2013 (UTC)

::And the answer is fdisk: illegal option -- l ... maybe I have less experience than I thought, or need a more foolproof solution! pablo 00:31, 7 March 2013 (UTC)

::: That's odd. /sbin/fdisk is part of util-linux, so it should work on any desktop or server Linux. Embedded Linux distributions may use the Busybox version instead, but even [http://busybox.net/downloads/BusyBox.html#fdisk has the -l option too]. -- Finlay McWalterჷTalk 00:42, 7 March 2013 (UTC)

::The "fdisk -l" one is questionable, as it identifies 24 out of my fileserver's 11 hard disks: the 11 physical drives, the three software RAID arrays built from those drives, and the 10 LVM logical volumes sliced out of the RAID arrays. --Carnildo (talk) 01:50, 7 March 2013 (UTC)

:::You need admin access to run "fdisk -l" in most cases. You can try "sudo fdisk -l" if you're setup for it. Shadowjams (talk) 02:18, 7 March 2013 (UTC)

:I seem to recall some oddity about Unix that you had to mount a disk to see it. Is that still the case ? StuRat (talk) 02:33, 7 March 2013 (UTC)

::Not to see that it exists, or to see its partitions. Shadowjams (talk) 03:02, 7 March 2013 (UTC)

::No. In order to mount an HDD or partition, the OS has to know it exists. You can type cat /proc/partitions and see both mounted and unmounted hard drives and partitions. Hard disks in Linux are listed in /dev. For example, if I had two SATA HDDs in my computer, I would have /dev/sda and /dev/sdb. Each disk is divided into one or more partitions. So if sda were divided into two partitions, I would have /dev/sda1 and /dev/sda2. If they were IDE hard drives, they would be /dev/hda and /dev/hdb. However, as Carnildo noted, a RAID controller can be used to group multiple disks into one and lead Linux to believe that the array is a single disk. For example, the RAID may have three disks, each 500 GB in size. But they're in a RAID 0, so Linux thinks they're one 1.5 TB disk. You may think that's an academic exception, but it's actually common in server environments where Linux is often used. In such a case, you should use the command-line utility that came with your RAID card to determine how many disks are in the array. For example, if your RAID is made by 3ware, you use tw_cli. If it's made by LSI, you use megamgr. If you're not using a RAID (which most home users aren't) then you can just ignore that part about RAIDs and simply count the number of lettered disks (e.g., sda, sdb, etc.) to get the number of disks in your system. Reading the contents of /proc/partitions is my preferred way to do so.—Best Dog Ever (talk) 03:21, 7 March 2013 (UTC)

{{od}}Thanks all. Reading /proc/partitions has sorted things out in most cases, but on a few machines /proc/partitions does not exist, don't know why. fdisk does not run with the -l parameter on these machines either. /dev contains many, many entries of the form /sd??, for example

sd10q

sd10r

sd10s

sd10t

sd129i

sd129j

sd129k

sd129l

sd129m

sd129n

sd129o

sd129p

sd130q

sd130r

sd130s

sd130t

sd130u

sd136a

sd136b

sd136c

sd136d

sd136e

sd136f

sd136g

sd136h

sd137i

sd137j

sd137k

sd137l

sd137m

sd137n

sd137o

sd137p

sd138q

sd138r

sd138s

sd138t

sd138u

sd144a

sd144b

sd144c

sd144d

sd144e

sd144f

sd144g

sd144h

sd145i

sd145j

sd145k

sd145l

sd145m

sd145n

sd145o

sd145p

sd146q

sd146r

sd146s

sd146t

sd146u

sd16a

sd16b

sd16c

sd16d

sd16e

sd16f

sd16g

sd16h

sd17i

sd17j

sd17k

sd17l

sd17m

sd17n

sd17o

sd17p

sd18q

sd18r

sd18s

sd18t

sd18u

sd1i

sd1j

sd1k

sd1l

sd1m

sd1n

sd1o

sd1p

sd2q

sd2r

sd2s

sd2t

sd2u

sd3a

sd3b

sd3c

sd3d

sd3e

sd3f

sd3g

sd3h

sd8a

sd8b

sd8c

sd8d

sd8e

sd8f

sd8g

sd8h

sd9i

sd9j

sd9k

sd9l

sd9m

sd9n

sd9o

sd9p

from this it seems as though these partitions all exist on 18 physical hard disks. Am I interpreting this correctly? It does't seem likely as the box doesn't look big enough! pablo 16:27, 7 March 2013 (UTC)

:If you type df does it list any physical devices in /dev under filesystem? What kind of box is it and what version of UNIX or Linux is it running? You can check by typing cat /proc/version or uname -a. I know BSD uses /dev/ada0, /dev/ada1, etc. Macs use /dev/disk0, /dev/disk1, etc.--Best Dog Ever (talk) 01:12, 8 March 2013 (UTC)

::Under Linux, "df" lists filesystems, not drives, and only those filesystems that are mounted. Further confounding the issue is that it lists loopback filesystems, which have a "/dev" entry but which correspond to a file. --Carnildo (talk) 02:38, 8 March 2013 (UTC)

:::He can still use it to deduce which devices are being mounted as disks. In the left-column, it will list physical partitions currently mounted and he can then deduce what types of devices are being partitioned.—Best Dog Ever (talk) 03:24, 8 March 2013 (UTC)

:::: Aha! Mystery boxes are running SunOS. Entries in dev/dsk are of the format c0t0d0s0. I have d0, d1 and d2 = 3 disks which seems more feasible. Apologies if I made this more complicated than it is, I now have a solution for each of the different operating systems. Thanks! pablo 09:27, 8 March 2013 (UTC)