Add an option to disable parallel downloading of streams

pandlouk

Please add an option to disable the parallel downloading of the streams (introduced in versions 5054 and later).
When the "temp" folder is placed on an ssd parallel downloading is fine.
But in hdds with Shingled magnetic recording (SMR) technology (all? hdds manufactured after 2015) causes constant 100% disk activity during the download and processing.

thank you

Dog

While not related to the parallel download issue, I thought this might be useful information. SMR (shingled magnetic recording) is NOT used on ALL drives - it's typically only used for the lower cost drives. Unless you have a non-SMR drive (e.g. CMR), with equivalent specs, that doesn't hit 100% under the same conditions, I wouldn't conclude that the 100% drive activity issue is related to SMR. The fastest spindle drive (HDD) is an absolute dog compared to the slowest SSD (at least that I've ever seen) due to the mechanical nature of HDDs. If you're trying to write multiple, incoming streams of data to the drive at the same time, the physical head assembly has to move back and forth constantly in order to write each stream to its respective area on the HDD (not to mention also having to wait until the appropriate sector rotates under the head before the data can actually be written, allowing the head to move ot the next location) - I imagine this would probably be enough to push any HDD to 100% once its cache is exhausted.

Having said all that, I agree that the option to disable such a feature would be a big boon for those recording to spindle drives.

pandlouk

Nah
CMR hdds (at least those in my possession) do not suffer from 100% disk activity.

I also run some tests with 5053 and 6001 and found some interesting results:
5053:
when downloading a file of 1.5 gb it writes 1.5 x 2 = 3 gb (is normal since first downloads and the decrypts the streams) on the disk but and then
ffmpeg during the remux of the streams writes another 1.5 x 2 = 3 gb (this is not normal... first is remuxing 1.5 gb then copies the file in the desired location and the deletes the original remuxed file) when it should write only 1.5 gb (remux the file and then move it to the desired location instead of copying it.

5054, 5055, 5057, 5058, 6000 and 6001:
when downloading a file of 1.5gb write 1.5 x 3 = 4.5 gb => somewaht normal but I don't like it,.. (1.5 gb are downloaded temp chunks, then 1.5 gb are those parts getting decrypted and finally the last 1.5gb are those chunks getting merged into 1 temp file for each streams)
then ffmpeg during the remux writes another 1.5 x 2 = 3 gb (same as the version 5053 which is not normal as stated above).

so v5053 for every file, writes 4x times on the disk
and v5054 to v6001 for every file, write 5x times on the disk

Personally, I prefer the way 5053 downloads and decrypts even when using my ssds (although they need to fix the remux and move file to the correct folder process)

pandlouk

@Mona
Please also ask the developer to add an option to enable/disable the ffmpeg command "faststart".
When enabled should remain as "-movflags faststart+disable_chpl" (the way it is now)
When disabled should be "-movflags disable_chpl".

Moving the index (moov atom) at the beginning is useful if one wants to upload a video file on streaming servers (YouTube, Dailymotion, etc. ) and I do not understand why is enabled by default (since it forces ffmpeg to write twice the same file on the disk).

Dog

@Mona
Please also ask the developer to add an option to enable/disable the ffmpeg command "faststart".
When enabled should remain as "-movflags faststart+disable_chpl" (the way it is now)
When disabled should be "-movflags disable_chpl".

Moving the index (moov atom) at the beginning is useful if one wants to upload a video file on streaming servers (YouTube, Dailymotion, etc. ) and I do not understand why is enabled by default (since it forces ffmpeg to write twice the same file on the disk).
pandlouk

I second this request as well as the first.

Cats4U

I don't agree. No more options. It has become abundantly clear that a great many of the StreamFab users aren't ready to make such technical decisions. Perhaps only 1 in 10 know what the Turbo switch is for. Prior to that, the GPU option, which didn't work at all, was being sworn as to either the fix for a problem or the cause of another. Finally, StreamFab got smart and removed that option since it did absolutely nothing.
StreamFab doesn't need to be super fast or efficient. It doesn't need to be pretty or even totally logical in its design. It just needs to do one thing (for at least awhile) and that is to play the cat and mouse games with the world's streamers to bypass their efforts and the laws that protect them to allow users to retain possession on their hardware of videos that they paid to watch. Don't complicate things. If a user gets the show he wants on his hard drive, he is happy. He doesn't care whether it got written somewhere twice or it took 50 milliseconds more than it should of. The Developers needs to devote all his/her time to making the user happy by making all the shows that the user wants winds up on his hard drive. K.I.S.S.

pandlouk

I don't agree. No more options. It has become abundantly clear that a great many of the StreamFab users aren't ready to make such technical decisions. Perhaps only 1 in 10 know what the Turbo switch is for. Prior to that, the GPU option, which didn't work at all, was being sworn as to either the fix for a problem or the cause of another. Finally, StreamFab got smart and removed that option since it did absolutely nothing.
StreamFab doesn't need to be super fast or efficient. It doesn't need to be pretty or even totally logical in its design. It just needs to do one thing (for at least awhile) and that is to play the cat and mouse games with the world's streamers to bypass their efforts and the laws that protect them to allow users to retain possession on their hardware of videos that they paid to watch. Don't complicate things. If a user gets the show he wants on his hard drive, he is happy. He doesn't care whether it got written somewhere twice or it took 50 milliseconds more than it should of. The Developers needs to devote all his/her time to making the user happy by making all the shows that the user wants winds up on his hard drive. K.I.S.S.
Cats4U

I do not care if it is made in as option enable/disable, or through registry or whatever.

If they do not won't to add at least the first as an option then, they should either revert the change or refund me...
(since when I bought the program in august worked as it should and now causes all kinds of problems).
And nowhere in the manual or the requirements or anywhere they stated information that is or will become incompatible with hdds.

And I doubt that all their customers do not care, if the product is an ssd killer since it reduces the lifespan of their ssd's.

Especially when **** achieves the same downloading/decrypting/remuxing by writing only once on the disk
For a file of 10gb
StreamFab 5053 writes 40gb
StreamFab 6001 writes 50gb
Anystream writes 10gb

Jimd6500

I do not care if it is made in as option enable/disable, or through registry or whatever.

If they do not won't to add at least the first as an option then, they should either revert the change or refund me...
(since when I bought the program in august worked as it should and now causes all kinds of problems).
And nowhere in the manual or the requirements or anywhere they stated information that is or will become incompatible with hdds.

And I doubt that all their customers do not care, if the product is an ssd killer since it reduces the lifespan of their ssd's.

Especially when AnyStream achieves the same downloading/decrypting/remuxing by writing only once on the disk
For a file of 10gb
StreamFab 5053 writes 40gb
StreamFab 6001 writes 50gb
Anystream writes 10gb
pandlouk

I am a bit confused with some of the options being requested, but looking at the above that you have stated on files size is very concerning to me. I am limited on my server disk space so if I have to compress files when I don't need to because they are larger than they should be, that is a problem. I also loose the MOV_TEXT in files when I compress movies or TV shows so that is a problem too. If the files sizes don't need to be compressed, I am happy.

Mike_M

I am a bit confused with some of the options being requested, but looking at the above that you have stated on files size is very concerning to me. I am limited on my server disk space so if I have to compress files when I don't need to because they are larger than they should be, that is a problem. I also loose the MOV_TEXT in files when I compress movies or TV shows so that is a problem too. If the files sizes don't need to be compressed, I am happy.
Jimd6500

Jimd6500,

Not sure, but I *think* there's a distinction here and that the writes that pandlouk is talking about is not related closely to the final size of the file.

To me I interpret this as "bad" programming that causes the SDD to "utilize" writing/reading? 4x to 5x less efficiently and therefore is making the SDD jump all around and is depreciating the life of the SDD.

I've found that the actual size of a file that should be exactly the same between StreamFab and the Competitor is pretty close with the difference that StreamFab is slightly larger because it is including information about your machine, your user id, etc... in every file you get via StreamFab so you can be tracked if you upload it for sharing and including that information in the file also. (See last paragraph of the attachment that was from StreamFab's website that I think has since been modified after it was questioned on this forum, but I think *not* really responded too here)

StreamFab keeps saying *all* they do is get the stream, but I'd like to know why the re-muxing/processing stages that seem to take longer than the competitor are ran and how much of that is doing things like the above for tracking and other unnecessary server communications/steps that could be in the program.

The Competitor made a recent post (not going to provide the link here) that specifically referenced that StreamFab is doing the highlighted action above (and in attachment) and that the Competitor does not do this for tracking you and your file, therefore using it as a selling point.

It would be nice to get an official response from DVDFab/StreamFab on this. :rolleyes:

Mike

US - CST, No Vpn, Latest version of Streamfab/DVDFab All-in-one full products (for now) lifetime licenses
Primary Machine: Win 10/64 Pro & Win 11/64 Pro, RTX 3080Ti, 32gb Ram, I7 12700k
Secondary Machine: Win 10/64 Pro, RTX 2070 Super, 16gb Ram, Skylake processor

pandlouk

I am a bit confused with some of the options being requested, but looking at the above that you have stated on files size is very concerning to me. I am limited on my server disk space so if I have to compress files when I don't need to because they are larger than they should be, that is a problem. I also loose the MOV_TEXT in files when I compress movies or TV shows so that is a problem too. If the files sizes don't need to be compressed, I am happy.
Jimd6500

The final size is the same of the file is the same 10gb so you do not need to worry about that.

As for the option to disable the parallel downloading/processing (and why it causes failures and buffer overruns) I'll make an example with v6001.
Lets say you download a 10gb video with 2 audio streams. StreamFab 6001 downloads and decrypts all the streams in parallel and to achieve that it has to do it by splitting them in small chunks:
Video stream : (Header.tmp + Header decrypted) + (1.temp + 1.temp.dec) + (2.temp + 2.temp.dec) + (3.temp + 3.temp.dec) + .... (N.temp + N.temp.dec)
Audio stream 1 : (Header.tmp + Header decrypted) + (1.temp + 1.temp.dec) + (2.temp + 2.temp.dec) + (3.temp + 3.temp.dec) + .... (N.temp + N.temp.dec)
Audio stream 2 : (Header.tmp + Header decrypted) + (1.temp + 1.temp.dec) + (2.temp + 2.temp.dec) + (3.temp + 3.temp.dec) + .... (N.temp + N.temp.dec)

then it merges/concatenates the temp files in the 3 big files.

But (and here lies the problem in version 6001) issues a command join/add "n.temp.dec" to the temp file stream, and then a delete command "n.temp + n.temp.dec".
So the disk constantly writes and deletes (or at least tries to delete) those small pieces..
Result=> SMR hdd's cannot keep up.

What I ask is to bring back the previous 5053 setting:
Download
Video stream : in one big temp file -> decrypt it
Audio stream 1 : in one big temp file -> decrypt it
Audio stream 2 : in one big temp file -> decrypt it

Or if they want to go with small chunks as it is now... At least do not issue a delete command for every chunk added (during the merge) but issue only the delete folder command after the merge is finished.

pandlouk

@Mike_M
I do not consider it as bad programming... more like bad implementation.
StreamFab uses renpy & ffmpeg to create the files while AnyStream uses Java MP4 parser.
StreamFab with ffmpeg can merge unlimited streams in a single file but needs to write 3x times on the disk (download+decrypt+merge).
Anystream with the Java MP4 parser needs to write only once since it downloads+decrypts+merges during the download.... but supports only H264 and cannot merge multiple audio streams.

As for the customer info included in the streams I do not see any such activity in v505x and later. Maybe was true in previous versions when instead of downloading, it was recording the stream.

Mike_M

@Mike_M
I do not consider it as bad programming... more like bad implementation.
StreamFab uses renpy & ffmpeg to create the files while AnyStream uses Java MP4 parser.
StreamFab with ffmpeg can merge unlimited streams in a single file but needs to write 3x times on the disk (download+decrypt+merge).
Anystream with the Java MP4 parser needs to write only once since it downloads+decrypts+merges during the download.... but supports only H264 and cannot merge multiple audio streams.

As for the customer info included in the streams I do not see any such activity in v505x and later. Maybe was true in previous versions when instead of downloading, it was recording the stream.
pandlouk

Hi pandlouk,

The current statement from StreamFab does not say it applies to specific versions and is at:
https://streamfab.com/statement.htm

This statement from StreamFab previously (see how it was attached) had the sentence "We've included the customer/account id in the metadata of the files extracted from streaming platforms" that was removed when it was questioned on this forum.

DVDFab/StreamFab didn't say whether or not they did this, they just removed the sentence about what exactly they said they were doing.

This came up again when we saw some unencrypted files named 65518339-F04A-4B70-839E-FD168961FDV1 showing up in the StreamFab & DVDFab directories for users after a release. The file included a "Token Id", our email address, IP address, machine id, etc...

After it was pointed out these files were unencrypted, they became encrypted again and I haven't seen them for a while now.

I don't think the file 65518339-F04A-4B70-839E-FD168961FDV1 was really ever meant to be seen by the end user - unencrypted or encrypted.

StreamFab still has the statement at their website and has not addressed whether or not this is the practice in this forum when asked yet has a statement about it.

BluRayfan said he found the instances of it in some files and JPP72 said he thought it was BS to prevent users from sharing software.

I'm not a programmer, just going by what StreamFab said/says they are doing.

Mike

US-CST, No VPN, current version of StreamFab, Win 10/64 Pro & Win 11/64 Pro

pandlouk

@Mike_M
Hi,
actually you can still find the file 65518339-F04A-4B70-839E-FD168961FDV1 in the C:\Users\[Username]\AppData\Roaming\DVDFab directory.

ps. on my previous post I wrote "renpy + ffmpeg", instead of writing "Python + ffmpeg" ... I was coding in renpy (is based in python) at the moment and mistyped without realizing it...

Cats4U

ps. on my previous post I wrote "renpy + ffmpeg", instead of writing "Python + ffmpeg" ... I was coding in renpy (is based in python) at the moment and mistyped without realizing it...
pandlouk

Please also ask the developer to add an option to enable/disable the ffmpeg command "faststart".
When enabled should remain as "-movflags faststart+disable_chpl" (the way it is now)
When disabled should be "-movflags disable_chpl".

StreamFab uses renpy & ffmpeg to create the files while AnyStream uses Java MP4 parser.
StreamFab with ffmpeg can merge unlimited streams in a single file but needs to write 3x times on the disk (download+decrypt+merge).
Anystream with the Java MP4 parser needs to write only once since it downloads+decrypts+merges during the download.... but supports only H264 and cannot merge multiple audio streams.

So, tell me, how do you happen to know all this stuff? None of that is indicated in any of the logs.

Mike_M

@Mike_M
Hi,
actually you can still find the file 65518339-F04A-4B70-839E-FD168961FDV1 in the C:\Users\[Username]\AppData\Roaming\DVDFab directory.

ps. on my previous post I wrote "renpy + ffmpeg", instead of writing "Python + ffmpeg" ... I was coding in renpy (is based in python) at the moment and mistyped without realizing it...
pandlouk

Pandlouk,

Yep,

It's in the DVDFab path in an encrypted format on mine too right now, but no longer in the \DvdFab\Streamfab path also.

For some reason this file was also in C:\Users\[Username]\AppData\Roaming\DVDFab\StreamFab directory in mid August also and they were unencrypted formats that you could simply view in notepad.

Mike

pandlouk

@Cats4U
Why should any of these be displayed in the logs?
For streamfab you can use Process Explorer to view/monitor how it operates.
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer

Mike_M

[USER="82454"]
For streamfab you can use Process Explorer to view/monitor how it operates.
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
pandlouk

Padlouk,

Thanks!

Will definitely look at this soon!

Have downloaded it, but working on some stuff right now.

Mike

US CST, No Vpn, Win 10/64 Pro, Win 11/64 Pro, Some version of StreamFab 6.0.0.2 ;)

weavermovies

I keep seeing references to this file named "65518339-F04A-4B70-839E-FD168961FDV1" regarding its purpose and location.....

For what it's worth..... and without necessitating any hierarchical deep dive..... I have two of these things I see every day sitting in the StreamFab folder.....

I've never deleted them because I just assumed they were there holding login information for the various sites from which I download.....

Maybe someone can tell me if they are serving any useful purpose and if not..... I'll delete them.....

Also..... for the record..... regarding this read/write thing pandlouk has brought up..... common sense dictates that processing..... any processing..... should be accomplished requiring the least amount of read/write instructions no matter where the destination of said read/write instructions are to be performed.....

And please pardon my frankness..... however..... only a fool attempting to smokescreen their ignorance would think otherwise.

Cats4U

@Cats4U
For streamfab you can use Process Explorer to view/monitor how it operates.
https://learn.microsoft.com/en-us/sysinternals/downloads/process-explorer
pandlouk

Thanks, but I have it already. I have all the Sysinternals. I prefer to use Process Lasso for the few times that I need to get down to the process level, certainly not for the reason you use a process explorer.

Mona

"65518339-F04A-4B70-839E-FD168961FDV1"
weavermovies

It won't affect the use of the software, you may delete it directly.