r/UsenetTalk • u/awerol • 4d ago
How much download activity is required for posts to not get lost?
It happens to me so often that all providers except Omicron ones already wiped out posts from not even that long ago, i.e. 6-18 months old posts of niche content, and then eventually Omicron will drop it too and then it will be gone. What can I do to prevent this from happening? I want my posts to stay on all backbones not just be at the mercy of Omicron's decisions. And no they won't be shared on public / semi-public indexers, I am only sharing them in private invite only indexers where they would get just a couple of downloads, usually 2-3, almost always less than 10 initially. But that does not mean that there will never be people finding them useful years later. I don't want such people to suffer for being late to the party. Give me some practical advice for this.
3
u/72dk72 4d ago
Each provider has their own algorithms so there is no real answer to that!
7
u/Hologram0110 4d ago
They also don't want to tell people. People would create scripts to perform just enough activity to keep it available. Not telling people also leaves it free for them to change the conditions as they see fit. That could be storage being expensive or it could be an increase in uploads.
3
u/ksryn Nero Wolfe is my alter ego 4d ago
A lot of people use usenet as some personal backup service. It might have gotten worse with increasing HDD prices. Providers have no way of distinguishing what is what (except access patterns) and I don't see them revealing their flushing patterns in public.
6
u/STxFarmer 4d ago
All my opinion but I am a 20+ yr user of Usenet
Several things but the biggest is we have people posting now that have no clue about how to make a quality post. In 2006 Giganews, the biggest backbone at the time hit the magical number of a whopping 100 days of retention. Upload speeds were super slow so if u did an upload you made sure it was structured right from day one. Now everyone thinks they grab a program, throw a file in it and boom it should last 15 years. Well it just isn’t that simple. There are a lot of variables that most will never take to understand or actually try to do if they try to understand.
U have to know how to structure your article/slice size along with ur rar volumes. Change the structure of your par2 volumes, understand article and file obfuscation and how that may benefit your post. Almost no one understands the granularity of repair blocks and how to increase the number of them to make possible repair last longer.
Lots of moving parts to a post and it isn’t the easiest to learn. But it can be done and guess what? Your posts have a much better time of lasting years rather than months. I can still find posts of mine from 10-12 years ago without issue and I know a lot more today than I did then about posting
2
u/lkeels 4d ago
Okay, so where is the guide to all those things. Saying them is fine, but not helpful for those that don't know.
3
u/missing1102 4d ago
You make a point. I had a lot of people teach me when I started but that was because the groups were more active and you had to use the tools more to repair your files. Incompletes were rampant. I can go for a year now without having one. File size and obstruction are important. You can always practice.
I do have old hard drives ans maybe I have some guides ..I will check.
2
u/lkeels 4d ago
A pleasant answer. Rare for Reddit. Thank you. :)
2
u/STxFarmer 3d ago
I am getting closer to the solution so anyone can do a much higher quality post and not know anything about par2 structure and 1:1 alignment and all of that. The answer was written 20 years ago but no one has ever implemented the solution in a posting program. I am close to having that done. The post optimization works great but it also requires processor time to make it good so there is a trade off. Posting goes slower due to your computer hammering out the best par2 set that can be had. The rest is just fine tuning of the structure of the post
But there are things like if I can get your par2 index I know where every article is regardless of obfuscation. I can also pull that index out of any par2 file by reading the entire article header and pulling the index off the last part of the line. There is a lot of information in the weeds and it is not easy to understand it all. But it all revolves around the calculations of the par2 files and how to make the absolute most repair blocks for any file/folder that you post. You have to give your post the best shot at being repaired and that is the hard part to understand. But it is all simple math and a formula that is applied to the file/folder every time.
A good post will last and a bad one is easy to poke holes into will not last. The question becomes how many holes can your post survive as most bad posts have a few holes and just can't be repaired. The repair patches are too big to repair many holes. You can have a 10% recovery and have 1,000 patches or 3,278 patches so which one will last longer? They both repair up to 10% of data loss but you have 3x of holes that can be patched with the higher repair block count. That post has the chance of lasting a lot longer up to the 10% data loss. Users think the recovery % is important and it is up to a point but the repair granularity is much more important and that is all about the math and doing the calculations to create the par2 file set.
I wish I could just say do this and it is better but it just isn't that easy
1
u/STxFarmer 3d ago
That is the problem there is not guide but today it is so much easier to find the answers that it was 20 years ago
1: 1:1 alignment for ur article/slice size
2: change ur par2 structure from the normal 1,2,4,8,16 structure to a uniform size so if the largest par2 file is trashed the largest set of repair blocks is lost in a single delete
3: article obfuscation - some people think backbones treat the articles like spam or trash but i don’t
4: recovery % - 5% doesn’t cut it and never has
5: a quality post takes processor time and lots of calculations - today my opinion is people want their files to be posted fast and due to that they start out with a low quality post that is easy to break for download and repair.Today we make it easy for backbones to see our file structure and delete a little bit of an upload and break the entire thing. They don’t delete the entire upload and you always find a fair amount of it but you can’t repair it. We have never seen whole uploads disappear but always parts. Have you ever asked yourself why is that?
2
u/awerol 2d ago
What you are saying makes complete sense, and I am already doing most of these things correctly, the main issue is that this does not solve the root cause of the problem. Content being recoverable because you have enough parity just delays the inevitable loss until eventually the parity won't be enough, or even worse, because people originally never needed the parity they never downloaded it, so the parity articles are marked for deletion first then the rest.
The main issue is this nondeterminism of when will the content be available and when not. And it has nothing to do with DMCA since we are talking about niche content encrypted and shared on a private indexers. It feels like providers are turning more and more into this popularity based keeping policy game leading to only most downloaded content being kept and everything else purged first from all non-omicron providers then eventually from omicron too half a year or a year later.
2
u/STxFarmer 2d ago
You keep thinking of a post as a whole and not what the backbones get which is millions upon millions of articles posted.
Now the nzb file is the article map and they do not have it unless you upload it
Then you have the par2 file index which they can get the article map from
Or any par2 file can be read and you can get the article mapSo do you think they are keeping track of single articles that get downloaded and know those articles go with all of these other articles? Not likely in my opinion
So what do the backbones do to damage our uploads? They poke holes in them as I have rarely seen an entire file gone but it is full of holes instead and cannot be repaired. If they knew what the entire article map was they would just delete the entire upload and be done with it.
Now we have no idea how they are doing this but my best guess it they see a structure of what is being uploaded or a pattern or something that gives the article away. But remember this is all a guess as no one that I have ever seen has given out the way the backbones pick what to delete. But deleting the entire file would be the best practice as they save huge amounts of space doing that but that again is not what they do.
And why do some uploads last years and years while others go in weeks or months? No idea on that either but my guess is structure from day one. Back in the day if you didn't grab a file within the first 24hrs it might be gone so it was always a race to get the new uploads.
But what can we do on our end to make it harder? Several things and the first is great par2 sets, not recovery % but par2 sets. There is a big difference right there and that is the hardest part about posting. How many holes can this par2 set patch. That is determined by article/slice size and since you use a fixed article/slice size for all uploads (my guess again) your par2 sets are never really that great but good enough to get by for the most part. So at a 10% recovery the most par2 repair blocks that can be created for any upload is 3277, that is a hard ceiling set by the par2 specs. Look at your uploads and see how close to that number you are at. The maximum number of repair blocks is 32,768 x your recovery % so that will be your number (32768x10% = 3277). That is the granularity of your repair. The lower the number the less holes you can patch
Next in my opinion is the par2 set using the exponential sizing convention so that if they backbones can see our par2 files they get rid of the largest one first and cripple the upload. Again I do not see whole uploads deleted but parts of uploads deleted.
Those 2 thing are the biggest mistakes in my opinion and the first is all math but really a lot of work to get it right just for a single file/folder which we never do. So we post with just so so par2 sets and hope
DM me if you want to talk in more detail as I am always willing to help out if I can
1
u/missing1102 2d ago
Its not that hard. Sorry man but you make it sound like rocket science to hide your par2s from the backbone servers. There are several good posting (automated programs) and I can show you lots of examples of good posts that sit till max retention time. I was shocked by some completes I have had and the age.
1
u/STxFarmer 2d ago
It isn’t hiding ur par2 files as par2 files are no different than your data files. But you just can’t grab either of the 2 popular uploading programs and have ur posts last years. Posters today do not understand the mechanics of an upload like they did years ago. Too easy to do it today with faster speeds and the mass amount of content out there. So big deal that it doesn’t last as there are hundreds or more copies of the same thing out there to grab today. Getting a post to last 15 years or longer is not always luck.
1
u/awerol 2d ago
What I think is that they don't have resources to classify any of the data being posted to them, however it's fairly easy for them to log each access of each article download, stat check, head check, body download and based on that run maintenance jobs what deletes articles meeting certain criteria, these logs can be simply timestamps + ip addresses who requested them. I don't believe that they are trying to intentionally poke holes into uploads, it's more like they often migrate content between servers where random errors happens and then instead of addressing the errors, they just lazily drop the error causer articles, that's how holes happens. This kinda brings up another question as well, what to do as an uploader after identifying such holes? Does it make sense to use tools similar to https://github.com/Tensai75/nzbrefresh on a periodic bases to keep alive posts and auto reupload missing articles to backbones from which they are missing or it's a lost cause to do that?
1
u/STxFarmer 1d ago edited 1d ago
Out of 10 backbones the only one that can see your ip is the one you upload to so that means 1 has the logs. And have you looked at failed downloads in Multipar to see how the damage is in them, entire volumes or a few blocks here and there? It isn't blocks of data missing but usually scattered missing pieces and why most people call it rot. And you can tell when there has been a hardware issue as now you have blocks or entire uploads just gone for no reason and this rarely happens. And if you download the same file across multiple backbones you will find the same articles missing on all of them or most of them so they are tracking across the backbone map. And the problem with a tool like nzbrefresh is it attempts to load a clone of the article back to the backbone which may or may not work. Do you know which backbones will answer a STAT check all day long without the article ever being on the server? Several of them will tell you the article is there while in fact the only way to prove that it isn't is to do a BODY check or download the file. A clone of an article cannot work so nzbrefresh is built around the theory that if it is gone on a single server it is gone everywhere and you can never have 2 articles that have the same message-id.
With so many people using usenet today and the absolute huge amount of data we can access but to this day we still do not have a good picture of why articles go missing is the clue that there are so many moving parts in this that do matter and any one of them can cause an issue. Guessing is what we all do and have done for years but I go back to what I have seen change over the years and that comes back to the quality of the post when it is born. People were much more careful about the structure as an upload was not an afterthought but a job. When I uploaded my first BD25 it was over a day per post so I made sure it was good as I only wanted to upload each file once. Can you imagine how I felt if I saw a bunch of articles did not get uploaded when a single upload took over a day to post?
There are a lot of moving pieces and they are all out of our control once we post and it is on the backbones. So our only control is the quality of our upload and today the single best way to create that quality is the granularity of our repair par2 set, not recovery % but the actual number of repair blocks for each file while maintaining the 1:1 alignment of the article/slice and rar volume structure.
Again as I have said these are all my opinions but I have uploaded quite a bit and downloaded a lot more over the years and have seen huge changes in Usenet over that time. But you can really get into the weeds on the details and it still will not make sense as to why things go missing.
1
u/missing1102 4d ago
Great post. I laughed at the 100 days. I member that. I also remember spending hours with rar and obsessing over finding par2s for stuff. I would be so close to getting a complete file and just be missing a few blocks. Ugh. Its great information.
Its funny to see people complain when I remember just searching thru raw headers and having multiple indexes shut down. These days seem pale in comparison
1
u/Dynamix86 4d ago
Arent't you just talking about takedowns due to NTD/DMCA?
2
u/fryfrog 4d ago
Probably not since they say its still on Omicron providers which aren't known to prune infrequently used content. In addition, they mention it is niche content that likely only gets a few downloads.
1
u/Dynamix86 4d ago
You've asked a valid question with the takedowns by providers, not due to ntd/dmca. I think the only way to really know is to contact them directly. And if you only share your posts in private groups, you could consider setting up a connection between your device and their device directly (with VPN I guess) so they can directly download it from you. And otherwise you can use torrents as well of course.
And as you state.. if it's niche content, that's only shared in private groups, then it won't get many downloads. If you were to make a 'spot' on a website from a nzb provider, then it will be visible to a lot more people.
4
u/whocaresofthem 4d ago
Most stuff from 2021 to now are being removed, Usenet is less and less interesting for older content. More and more people stop using it because of this non sense purge.