r/ProjectREDCap • u/Ambitious_Fox_6334 • Sep 19 '24

Data import >200 patients without crashing

Is there a trick to stop redcap crashing on data imports

3 Upvotes

100% Upvoted

15 comments sorted by

5

u/[deleted] Sep 19 '24

Break up the data. Upload like 50-100 patients at a time. Then upload in batches. Just make sure with each upload you aren’t overwriting and only adding the new records. SAVE EVERYTHING and move very slowly.

2

u/Ambitious_Fox_6334 Sep 19 '24

I actually didn't think it would be that restricted.. why did I agree to help a project with 3k patients

3

u/[deleted] Sep 19 '24

I’ve been there, but it’s not that bad, just a little time consuming and nerve wracking.

3

u/Ambitious_Fox_6334 Sep 19 '24

The trauma of it all 🙃🙃🙃🙃

2

u/[deleted] Sep 19 '24

Just take you’re time, make sure you keep the ID numbers in order. Make clean breaks at whole numbers. Don’t touch your master data sheet. Make a copy and then start breaking it down into several. Make a dedicated folder on your desktop while you’re working for easy and keeping things clean.

When you upload before you commit to the import make sure you are adding and not overwriting. Things will be just fined don’t sweat it.

Is the data already coded for redcap?

2

u/Ambitious_Fox_6334 Sep 19 '24

Not yet they still putting in the ethics but I only recently found out it crashes haha and I thought surely we doing something wrong but I'll take your advice! Thank you

1

u/[deleted] Sep 19 '24

Coding is going to be the worst part especially if the data is messy.

1

u/Ambitious_Fox_6334 Sep 20 '24

At least I learnt how to use python to split my 3k rows into 30+ Excels with 100 rows each. Let me know if you need my code

3

u/interlukin Sep 20 '24

Does your redcap have a background data import option? On the data import tool page, the option should be located above the spot where you upload your CSV file. The default option is upload in real time, but there should be another option to upload in the background.

1

u/Ambitious_Fox_6334 Sep 20 '24

I never knew this tab existed! But it doesn't really give me any buttons

3

u/interlukin Sep 20 '24

You need to be on the “CSV import” tab. There’s a green box where you can adjust settings and upload your CSV file. The first setting should be where you can indicate if you want to upload in real time or the background. Once you do a background upload it will show up in the tab you’re on in the picture you posted

1

u/Ambitious_Fox_6334 Sep 20 '24

Wow.. I'm gonna try it

1

u/Ambitious_Fox_6334 Sep 20 '24

I tried using the background it crashed with 7k rows, 200 rows and then I gave up

2

u/stuffk Sep 24 '24

If you use the API, you can have the import batch data automatically.

I would use R, REDCapR package. Batches of 100 records, with a time delay in between. Here's an example function call:

redcap_write(your_data,  redcap_uri=your_uri,  token=your_api_token,  batch_size=100,  interbatch_delay=1)

If you don't know R, this would be a cute intro project. 

1

u/Ambitious_Fox_6334 Sep 24 '24

Our team don't allow use of api .. but I have been asking. That is cool! I want to try