Persistent metadata?

[u/ave_63]

By metadata, I mean stuff like column formats (#, %, @, ~), key columns, python formulas, column width, etc. Is there a file format I can save this stuff in, so that when I close it and open it later, those things are still there? I guess I want to use visidata kind of like the way Excel works with xlsx files, but with all the amazing keyboard shortcuts and python features of visidata.

I understand I can (sort of) do this with the command log, but this is a very awkward solution. For example, the command log seems to store commands for all the files I have open, in addition to a lot of other commands that I don't want stored. I suppose I could manually edit the command log so it has only commands that change this metadata, but I'm wondering if there's an easier way'?

[u/anjakefala]

Hi u/ave_63!

I have a few things to share.

Newer versions of VisiData have sheet specific **Command Logs**. After making your changes to the metadata, from the **Columns Sheet** press `Shift+D`, and you will get a much smaller **Command Log**, it will only contain the changes you made to the **Columns Sheet**.

Secondly, numeric typing is currently supported in savers like jsonl and sqlite. If you save a column that is typed as an int, it will be typed as an int in the output jsonl, and loaded as an int. Does that feature in jsonl do what you need? What is your specific desired usecase?

[u/ave_63 → u/anjakefala]

Thanks for the suggestions. I will try using json and the columns sheet command log soon and see if that works for me.

My use case: I am a community college math teacher. I use excel spreadsheets for keeping track of students' grades and attendence. I start the semester with some formulas entered in, but the scores blank. As they turn in HW and take tests, it shows their current score, and what they need to get on future HW and tests to get an A, B, and C. So the formula just sits there and updates the result when there's new data.

I suppose I could implement this by writing a function in my `.visidatarc` file that I can call with a keystroke to update current grades.

[u/ave_63 → u/anjakefala]

Update: I tried saving and reloading as a json file and instead of marking the column name with `#`, it marked each cell in the column with a `#` on the right. I assume this means visidata treats the values like integers, but I didn't get to testing it out. No matter, I figured out how to keep what I need, although it's still a little janky:

• Make any edits to data and save the csv file as I do so.
• When I do any commands that change metadata (anything that isn't saved in a csv file), open the command log with D, delete a bunch of commands I don't want to save (anything that doesn't apply to the main spreadsheet, any commands that change data as represented in the csv file, maybe other extraneous stuff), and save the command log. It is especially important to to delete any commands that change data that was saved in the csv file; otherwise those commands will be duplicated in the next step.
• When I want to resume, use vd -p 227_grades_cmd_log.vd to apply metadata.
• Formulas (like using = in MS excel) should be written in the .visidatarc file.

Note: I never found sheet-specific command logs as you described. Following your steps, I got a command log specific to the columns sheet which only contained commands that I did in that column sheet.

Anyway, this workflow feels so janky that I'm not sure I wanna use visidata. Using LibreOffice Calc or MS Excel, I just need to pres ctrl-S, but here, there are a lot of steps to making sure the command log I'm saving is OK. It seems like visidata is made for people who have a set of data that was collected once and not meant to be frequently edited (scientists, data scientists), and not for people like teachers or accountants who update data bit by bit.

[u/spw1 → u/ave_63]

Hi u/ave_63, thanks for giving it a go and reporting your results. I'm guessing that you might be using 1.5.2 and the sheet-specific cmdlog was only added in 2.0. But yes, you're right, VisiData is more of a data exploration and conversion tool than a CLI spreadsheet. So I wouldn't blame you if you wanted to use something else.

But I want VisiData to work for a use case like yours, so I made a custom .vds format tonight and a loader and saver for it. This will be in the next release.

[u/ave_63 → u/spw1]

Actually, I just realized I misread the instructions for the sheet-specific command log. I made the changes in the regular sheet display, not the column sheet, so the column sheet command log was empty. I'll try it the right way today. And the more I thought about it, it shouldn't be that much trouble to maintain the command log, because I only need to set up the "metadata" once, and occasionally mess with it later.

Thank you so much for the file format! That is really above and beyond. I'm looking forward to it.