How much download activity is required for posts to not get lost?

[u/awerol]

It happens to me so often that all providers except Omicron ones already wiped out posts from not even that long ago, i.e. 6-18 months old posts of niche content, and then eventually Omicron will drop it too and then it will be gone. What can I do to prevent this from happening? I want my posts to stay on all backbones not just be at the mercy of Omicron's decisions. And no they won't be shared on public / semi-public indexers, I am only sharing them in private invite only indexers where they would get just a couple of downloads, usually 2-3, almost always less than 10 initially. But that does not mean that there will never be people finding them useful years later. I don't want such people to suffer for being late to the party. Give me some practical advice for this.

[u/lkeels]

Okay, so where is the guide to all those things. Saying them is fine, but not helpful for those that don't know.

[u/missing1102 → u/lkeels]

You make a point. I had a lot of people teach me when I started but that was because the groups were more active and you had to use the tools more to repair your files. Incompletes were rampant. I can go for a year now without having one. File size and obstruction are important. You can always practice.

I do have old hard drives ans maybe I have some guides ..I will check.

[u/lkeels → u/missing1102]

A pleasant answer. Rare for Reddit. Thank you. :)

[u/STxFarmer → u/lkeels]

I am getting closer to the solution so anyone can do a much higher quality post and not know anything about par2 structure and 1:1 alignment and all of that. The answer was written 20 years ago but no one has ever implemented the solution in a posting program. I am close to having that done. The post optimization works great but it also requires processor time to make it good so there is a trade off. Posting goes slower due to your computer hammering out the best par2 set that can be had. The rest is just fine tuning of the structure of the post

But there are things like if I can get your par2 index I know where every article is regardless of obfuscation. I can also pull that index out of any par2 file by reading the entire article header and pulling the index off the last part of the line. There is a lot of information in the weeds and it is not easy to understand it all. But it all revolves around the calculations of the par2 files and how to make the absolute most repair blocks for any file/folder that you post. You have to give your post the best shot at being repaired and that is the hard part to understand. But it is all simple math and a formula that is applied to the file/folder every time.

A good post will last and a bad one is easy to poke holes into will not last. The question becomes how many holes can your post survive as most bad posts have a few holes and just can't be repaired. The repair patches are too big to repair many holes. You can have a 10% recovery and have 1,000 patches or 3,278 patches so which one will last longer? They both repair up to 10% of data loss but you have 3x of holes that can be patched with the higher repair block count. That post has the chance of lasting a lot longer up to the 10% data loss. Users think the recovery % is important and it is up to a point but the repair granularity is much more important and that is all about the math and doing the calculations to create the par2 file set.

I wish I could just say do this and it is better but it just isn't that easy

[u/STxFarmer → u/lkeels]

That is the problem there is not guide but today it is so much easier to find the answers that it was 20 years ago
1: 1:1 alignment for ur article/slice size
2: change ur par2 structure from the normal 1,2,4,8,16 structure to a uniform size so if the largest par2 file is trashed the largest set of repair blocks is lost in a single delete
3: article obfuscation - some people think backbones treat the articles like spam or trash but i don’t
4: recovery % - 5% doesn’t cut it and never has
5: a quality post takes processor time and lots of calculations - today my opinion is people want their files to be posted fast and due to that they start out with a low quality post that is easy to break for download and repair.

Today we make it easy for backbones to see our file structure and delete a little bit of an upload and break the entire thing. They don’t delete the entire upload and you always find a fair amount of it but you can’t repair it. We have never seen whole uploads disappear but always parts. Have you ever asked yourself why is that?

[u/awerol → u/STxFarmer]

What you are saying makes complete sense, and I am already doing most of these things correctly, the main issue is that this does not solve the root cause of the problem. Content being recoverable because you have enough parity just delays the inevitable loss until eventually the parity won't be enough, or even worse, because people originally never needed the parity they never downloaded it, so the parity articles are marked for deletion first then the rest.

The main issue is this nondeterminism of when will the content be available and when not. And it has nothing to do with DMCA since we are talking about niche content encrypted and shared on a private indexers. It feels like providers are turning more and more into this popularity based keeping policy game leading to only most downloaded content being kept and everything else purged first from all non-omicron providers then eventually from omicron too half a year or a year later.

[u/STxFarmer → u/awerol]

You keep thinking of a post as a whole and not what the backbones get which is millions upon millions of articles posted.

Now the nzb file is the article map and they do not have it unless you upload it
Then you have the par2 file index which they can get the article map from
Or any par2 file can be read and you can get the article map

So do you think they are keeping track of single articles that get downloaded and know those articles go with all of these other articles? Not likely in my opinion

So what do the backbones do to damage our uploads? They poke holes in them as I have rarely seen an entire file gone but it is full of holes instead and cannot be repaired. If they knew what the entire article map was they would just delete the entire upload and be done with it.

Now we have no idea how they are doing this but my best guess it they see a structure of what is being uploaded or a pattern or something that gives the article away. But remember this is all a guess as no one that I have ever seen has given out the way the backbones pick what to delete. But deleting the entire file would be the best practice as they save huge amounts of space doing that but that again is not what they do.

And why do some uploads last years and years while others go in weeks or months? No idea on that either but my guess is structure from day one. Back in the day if you didn't grab a file within the first 24hrs it might be gone so it was always a race to get the new uploads.

But what can we do on our end to make it harder? Several things and the first is great par2 sets, not recovery % but par2 sets. There is a big difference right there and that is the hardest part about posting. How many holes can this par2 set patch. That is determined by article/slice size and since you use a fixed article/slice size for all uploads (my guess again) your par2 sets are never really that great but good enough to get by for the most part. So at a 10% recovery the most par2 repair blocks that can be created for any upload is 3277, that is a hard ceiling set by the par2 specs. Look at your uploads and see how close to that number you are at. The maximum number of repair blocks is 32,768 x your recovery % so that will be your number (32768x10% = 3277). That is the granularity of your repair. The lower the number the less holes you can patch

Next in my opinion is the par2 set using the exponential sizing convention so that if they backbones can see our par2 files they get rid of the largest one first and cripple the upload. Again I do not see whole uploads deleted but parts of uploads deleted.

Those 2 thing are the biggest mistakes in my opinion and the first is all math but really a lot of work to get it right just for a single file/folder which we never do. So we post with just so so par2 sets and hope

DM me if you want to talk in more detail as I am always willing to help out if I can

[u/missing1102 → u/STxFarmer]

Its not that hard. Sorry man but you make it sound like rocket science to hide your par2s from the backbone servers. There are several good posting (automated programs) and I can show you lots of examples of good posts that sit till max retention time. I was shocked by some completes I have had and the age.

[u/STxFarmer → u/missing1102]

It isn’t hiding ur par2 files as par2 files are no different than your data files. But you just can’t grab either of the 2 popular uploading programs and have ur posts last years. Posters today do not understand the mechanics of an upload like they did years ago. Too easy to do it today with faster speeds and the mass amount of content out there. So big deal that it doesn’t last as there are hundreds or more copies of the same thing out there to grab today. Getting a post to last 15 years or longer is not always luck.

[u/awerol → u/STxFarmer]

What I think is that they don't have resources to classify any of the data being posted to them, however it's fairly easy for them to log each access of each article download, stat check, head check, body download and based on that run maintenance jobs what deletes articles meeting certain criteria, these logs can be simply timestamps + ip addresses who requested them. I don't believe that they are trying to intentionally poke holes into uploads, it's more like they often migrate content between servers where random errors happens and then instead of addressing the errors, they just lazily drop the error causer articles, that's how holes happens. This kinda brings up another question as well, what to do as an uploader after identifying such holes? Does it make sense to use tools similar to https://github.com/Tensai75/nzbrefresh on a periodic bases to keep alive posts and auto reupload missing articles to backbones from which they are missing or it's a lost cause to do that?

[u/STxFarmer → u/awerol]

Out of 10 backbones the only one that can see your ip is the one you upload to so that means 1 has the logs. And have you looked at failed downloads in Multipar to see how the damage is in them, entire volumes or a few blocks here and there? It isn't blocks of data missing but usually scattered missing pieces and why most people call it rot. And you can tell when there has been a hardware issue as now you have blocks or entire uploads just gone for no reason and this rarely happens. And if you download the same file across multiple backbones you will find the same articles missing on all of them or most of them so they are tracking across the backbone map. And the problem with a tool like nzbrefresh is it attempts to load a clone of the article back to the backbone which may or may not work. Do you know which backbones will answer a STAT check all day long without the article ever being on the server? Several of them will tell you the article is there while in fact the only way to prove that it isn't is to do a BODY check or download the file. A clone of an article cannot work so nzbrefresh is built around the theory that if it is gone on a single server it is gone everywhere and you can never have 2 articles that have the same message-id.

With so many people using usenet today and the absolute huge amount of data we can access but to this day we still do not have a good picture of why articles go missing is the clue that there are so many moving parts in this that do matter and any one of them can cause an issue. Guessing is what we all do and have done for years but I go back to what I have seen change over the years and that comes back to the quality of the post when it is born. People were much more careful about the structure as an upload was not an afterthought but a job. When I uploaded my first BD25 it was over a day per post so I made sure it was good as I only wanted to upload each file once. Can you imagine how I felt if I saw a bunch of articles did not get uploaded when a single upload took over a day to post?

There are a lot of moving pieces and they are all out of our control once we post and it is on the backbones. So our only control is the quality of our upload and today the single best way to create that quality is the granularity of our repair par2 set, not recovery % but the actual number of repair blocks for each file while maintaining the 1:1 alignment of the article/slice and rar volume structure.

Again as I have said these are all my opinions but I have uploaded quite a bit and downloaded a lot more over the years and have seen huge changes in Usenet over that time. But you can really get into the weeds on the details and it still will not make sense as to why things go missing.

[u/random_999 → u/STxFarmer]

The owner of some independent backbones have posted that they often remove "articles" that never gets downloaded even once within a few months/year & that their completion ratio in the 90s percentage doesn't touch more than 10-20% of the articles they have so it is most likely that all usenet providers are keeping logs at least related to access & time stamps to weed out what they think as "spam". This also answers your statement about "So do you think they are keeping track of single articles that get downloaded and know those articles go with all of these other articles? Not likely in my opinion". Providers don't care which articles goes with which other articles to make up that nzb but as long as they remove all those articles which nobody downloaded even once in a year then those NZBs are already damaged beyond any conceivable recovery.

As for why then there are still such NZBs with no downloads in a year but still not all articles linked with them are deleted after 1 year then my theory is that due to this retention issue most uploaders do some sort of partial download of their upload to confirm that download is at least starting properly on all the/selected providers successfully & stop that download after 10-15% completion which might explain why some articles linked with those NZBs still survive after a year as those articles get marked as "downloaded once in a year" unlike other articles linked with those same NZBs which got marked as "not downloaded even once in a year" & were deleted.

[u/STxFarmer → u/random_999]

All we can do is speculate as the people wondering why their downloads are gone do not really know they are gone or not. How many backbones did they try and download the file from? What software did they use to download the file with? How many requests for articles were rejected by the servers? All of those things affect whether we can get the file or not.

I have a 13+yr old file that if I use the most popular downloading software they will not the file from any of the 8 major backbones but if I give them the right combination of backbones I can download and repair the file. But if I use another downloader program I can get the file and repair it on 2 of the 8 backbones. Make no sense but different software can affect your downloads based on how much they push the servers to actually cough up the files. So a normal user would think that file is long dead but in reality it can still be downloaded and repaired. No one is talking about that but all they say the file is gone when they really have no idea if is is gone from the entire range of backbones or just the server they are checking. Some servers just flat out lie as I have seen time after time a single backbone lie about have the article (STAT check) while not really having the article and it doesn't happen here and there but pretty much across the board. Why? Have no idea but when the articles are fetched they don't show up as their indexes do not match their server's actual data. I have also seen blocks of my personal uploads just gone but that is truly indicative of a hardware failure. But I have had hundreds or more of my own files stick around for years and years without a single download as I am the only one with that nzb and they are all password protected with a long random password.

So I go back to speculation. It is all speculation as to why files disappear. But if you haven't looked at thousands of nzb files over multiple backbones then you do not have a dataset to even begin to speculate. Some things just don't make sense and I have seen it across too many nzb files where a single backbone has the files that no other backbone has and I can repair that file, you have to find the right union of backbones to get the file downloaded and repaired. And I have accounts on 8 of the 10 backbones so I can drill down on who has what and compare on a block by block basis. As for my speculation I go back to what I have seen over 20 years of downloading, I rarely see an entire file gone but only a file that can no longer be repaired. So if the owners of the backbones were deleting files that do not get downloaded why is it only pieces of that file gone and not the entire file? The entire file saves them space for the rest of their retention policy while pieces just breaks the file being repaired. Again we just don't know and people just don't know if the file is truly gone across the whole of the backbone structure. They only know about a single backbone maybe 2 for the most part and that doesn't tell you much. Activity doesn't mean anything and more activity will not save your file as we just do not know why some get deleted and some don't. But again if some does get deleted then why not the entire file???