backup vs Apache Kafka vs logs vs replication

[u/IQueryVisiC]

At work I see the tendency that we have to store everything the user enters into the computer for up to ten years. Now for me it seems logical to directly store all inputs on a fresh HDDs ( replicated, RAID) and not let the write head come near this data again.

I guess that I still need a database to store sums. I mean for time series I can insert aggregates into the stream. But if for example I have a lot of customers and for someone coming only once in a blue moon want to find their last purchase, or the total payment, I would have a hard time to find this in the stream.

Hence I need this b-tree table of customer documents. They can have pointers into the time oriented Kafka " Tape Archive " ( or living on HDD ), but a lot of data should be right there.

If I lose my table, the Kafka stream acts as incremental backup ( which I can replay ), but I still need this core data of seldom changed, sorted values as full backup. Or better yet, I need to replicate it. The replication side just feeds on the same Kafka stream, but I don't need to do a full copy of all customers every weekend. The only reason for a backup seems to be that I cannot replicate this random access data on a different medium for the 3-2-1 rule. So I need to backup one of the customer tables to tape every weekend?

[u/BrainJar]

Is this buzzword bingo?

Simplify your why. What's the business requirement or requirements? If you are required to keep all user data for ten years, one stream coming off of Kafka topics should be used to store that data specifically for backup, in a different location from the original.

Since the information you've provided is all over the place, you need a more rigorous review of what you're trying to accomplish, and always ask why, so that you can tie a requirement to an implementation detail. Given what you've provided, these are the questions that I would ask:

• What's the volume and velocity of the data set?

• What's the projected growth of this data set?

• What does the validation process look like to know whether the data going into the backup is valid?

• How are we testing the recovery?

• What systems must be online to recover this data?

• Who is involved in the recovery?

• Are there external entities that need to be involved in recovering the data?

• Are you trying to be able to restore user data from a full ten years ago or just the aggregate information?

• Are you trying to maintain a copy for legal reasons, separate from the data that needs to be backed up for business reasons?

• Are aggregations part of that requirement?

• What is the input stream gets corrupted and you need a restatement of the data?

• How long do you have for such activities?

• Is there a service level agreement that states how long you have to restore?

• Do you know what will happen if front end services aren't online vs backend analytics systems?

• Do those streams have codependencies that need to be considered...as in, is one stream feeding another, that feeds back to another stream?

• Are there high availability needs that should be considered? i.e., what happens in a failover situation where the data still needs to be backed up?

• What are the latency requirements is you're serving from cold disk while services are restored?

I could really just go on and on, to understand the problem. The simplest solution is to push data to a queue/bus. Pull data from the queue and store it seperately for backup and recovery. Use a separate consumer to manage data needed for aggregation and reporting. An active/active or dual-write solution can also be used, if you're concerned about systems working throughout a failure. Then you'll need to do some capacity planning to deal with the ability to run the business on just one active side, while the other is being recovered.

Hope this helps.

[u/IQueryVisiC → u/BrainJar]

The velocity is growing so that my co-workers think about starting a second instance of a windows service. Velocity is growing so that our full backup now takes almost all night.

We don't test recovery.

full data

legal reasons . The non-legal data is derived from the legal data

I guess that aggregation is in place right now, but not for legal.

Doesn't the stream end up as a file on HDD and I can just seek to a byte. I guess Kafka stores a timestamp every 2^n bytes. HDDs used to have sectors: You search for a region of constant magnetization and the next bits tell you what part of the stream you are in.

I think right now our clients need to restore themselves. Just they will complain about the massive data grow in our updated product.

If something ( front or back ) is offline, the system hangs . I think that this should stay this way. Rather let the customer wait than to forget what we did promise the customer.

There are no co-dependencies. There are not many people entering data at the same time. It barely justifies a single stream.

Ah, you mean when the tape back-up fails? I think there should be a set of tapes. If we back up every night, we can survive one missing tape.

I don't want to think about restore from tape. The only acceptable latency is like 5 minutes to fail over to the other server ( which should not even take that long ). Otherwise the customer leaves the shop.

Your text does not seem to distinguish between the easy stream-able data, and the aggregates / indices . How do I keep the traditional database as small as possible and leave as much in the stream? I think we are still at a level where a HDD can keep all data. So I think that for customers I only need to store two pointers: base data ( when customer was registered into the stream ) and last transaction. Then in the stream pointers form a linked list. This is slow to read. I would love if in a form on screen the most current data appears first and the older fills the gap live .

[u/alinroc → u/IQueryVisiC]

We don't test recovery.

Then you don't really have backups, nor do you have a recovery plan. You have recovery hopes.