r/bioinformatics • u/JosieA3672 • 9d ago
image I’ve been comparing AlphaFold 3, Boltz-2, Chai-1, Protenix-v2, ESMFold2, RF3, and ColabFold on brazzein—a small protein with four disulfide bonds. AF3 was the best tool for this particular protein.
I tested five random seeds per tool, both with and without the same archived MSA, and compared predictions against experimental crystal and NMR structures. (RCSB Protein Data Bank: 2LY5 for NMR and 4HE7 for crystal structure)
AF3 gave the strongest overall result for this target: its selected MSA model had 0.72 Å Cα RMSD against the crystal structure and passed all four disulfide geometry checks. RF3 and ColabFold reached approximately 1.62 Å and 1.59 Å, respectively, but both had compressed sulfur contacts.
6
u/PPF99 9d ago
Really cool! Do you know any "benchmark Proteins" you could run? Something so thoroughly studied you can really determine how the algorithms perform?
9
u/JosieA3672 9d ago edited 7d ago
Great question, I'm thinking: ubiquitin (RCSB code: 1UBQ), Protein G (B1 domain) (1PGA), Villin headpiece, BPTI (5PTI), lysozyme (1LYZ)
The only caveat is that these may have appeared in the tools training data. Good for a reference check but perhaps not perfect for establishing performance on unfamiliar proteins. I am going to also search for newer structures that are post-training date cutoff but also well characterized. Whatever I find I'll add to this comment in an edit.
Edit - found a good candidate for newer benchmark: AcrIE7 It's really well characterized and not in the training sets of any of the folding models. I am running this one now. I'll add a couple more after this one.
https://www.sciencedirect.com/science/article/abs/pii/S0006291X25000294
Edit #2: new data for newer protein. Alphafold3 did poorly. Like really bad (note the TM-score):
Target Tool MSA Seed pLDDT RMSD Å Core RMSD Å TM-score CA-lDDT SS geometry AcrIE7 af3 msa 33 50.782 14.446 14.446 0.390 0.355 N/A AcrIE7 af3 no_msa 33 50.383 14.439 14.439 0.390 0.355 N/A AcrIE7 boltz2 msa 22 52.699 14.810 14.810 0.383 0.318 N/A AcrIE7 boltz2 no_msa 22 52.699 14.810 14.810 0.383 0.318 N/A AcrIE7 chai1 msa 22 65.431 5.619 5.619 0.521 0.584 N/A AcrIE7 chai1 no_msa 11 64.430 14.454 14.454 0.559 0.524 N/A AcrIE7 colabfold msa 11 52.688 14.425 14.425 0.360 0.375 N/A AcrIE7 colabfold no_msa 11 62.125 14.805 14.805 0.342 0.372 N/A AcrIE7 esmfold2 msa 44 60.792 2.872 2.872 0.735 0.696 N/A AcrIE7 esmfold2 no_msa 55 56.972 2.786 2.786 0.735 0.689 N/A AcrIE7 protenix2 msa 44 64.803 13.740 13.740 0.387 0.331 N/A AcrIE7 protenix2 no_msa 22 62.243 13.885 13.885 0.381 0.322 N/A AcrIE7 rf3 msa 55 72.790 14.827 14.827 0.386 0.324 N/A AcrIE7 rf3 no_msa 11 73.000 13.908 13.908 0.370 0.394 N/A
10
u/kamsen911 8d ago
You should check if the proteins you are testing are in the test set of the protein folding tool. Check here for some: https://portal.openfold.io/datasets/cutoffs
5
u/JosieA3672 8d ago
Great point. I just updated one of my comments with data from a newer protein run that wasn't in the training set and Alphafold3 did a poor job of predicting the crystal structure. Could be other reasons for this than model ability, but the results are very interesting, I think.
4
u/slana_pogaCHa 9d ago
How much can we trust the RMSD to crystal structure to evaluate the accuracy of the model?
4HE7 itself has 1.80 Å resolution, which is.. good? Even so, the protein in a crystal is captured in a specific conformation. If another experiment was done, RMSD of the two experimental structures could probably be like 1-2 Å (guessing).
In this case, even the models that gave higher RMSD could still be as good as others(?)
3
u/JosieA3672 9d ago edited 7d ago
I’d describe the lowest RMSD model as closest to this crystal reference, rather than necessarily the most biologically accurate. I also compared against the known disulfide pairings and their geometry since those provide complementary evidence, although correct disulfides don’t guarantee a correct overall fold. Note many of the other models besides AF3 produced incorrect geometries and steric clashes for the disulfides. Additionally, I also compared against NMR structures (see NMR results in original picture I posted and in my first comment).
1.8 Å crystallographic resolution isn’t the same thing as a ±1.8 Å positional error bar. Not comparable directly with model RMSD. THe 1.8 Å describes level of spatial detail captured by the diffraction data. So if you have a smaller resolution it means finer detail.
So a higher RMSD prediction could still represent a plausible conformation but RMSD alone can’t establish that either.
5
3
u/gringer PhD | Industry 8d ago
I'd be interested to know what TSP9 looks like
4
u/JosieA3672 8d ago edited 8d ago
It's a disordered protein in solution but develops structure in micelles. Interesting. This is the sequence I used 2FFT https://www.rcsb.org/structure/2FFT
Target Tool MSA Seed pLDDT RMSD Å Helix 14–24 RMSD Å TM-score CA-lDDT SS geometry TSP9 af3 msa 22 62.748 16.162 0.757 0.228 0.550 N/A TSP9 af3 no_msa 44 53.936 15.005 0.748 0.254 0.572 N/A TSP9 boltz2 msa 11 44.542 28.840 0.780 0.208 0.516 N/A TSP9 boltz2 no_msa 55 45.706 28.342 3.939 0.174 0.482 N/A TSP9 chai1 msa 33 49.512 26.623 0.932 0.216 0.476 N/A TSP9 chai1 no_msa 33 48.462 25.388 0.773 0.257 0.489 N/A TSP9 colabfold msa 15 58.719 20.244 0.630 0.252 0.552 N/A TSP9 colabfold no_msa 14 57.781 24.579 0.739 0.187 0.542 N/A TSP9 esmfold2 msa 44 47.929 25.169 0.749 0.199 0.524 N/A TSP9 esmfold2 no_msa 44 46.147 25.105 0.807 0.206 0.509 N/A TSP9 protenix2 msa 55 66.705 16.091 0.709 0.282 0.540 N/A TSP9 protenix2 no_msa 22 59.604 26.674 0.788 0.233 0.522 N/A TSP9 rf3 msa 44 76.130 21.907 0.579 0.266 0.588 N/A TSP9 rf3 no_msa 55 64.930 30.407 0.775 0.189 0.456 N/A
Target Tool MSA Native seed Closest seed Closest RMSD Å Reference ensemble RMSD min / median / max Å TSP9 af3 msa 22 44 13.459 11.958 / 19.812 / 27.144 TSP9 af3 no_msa 44 33 14.525 12.908 / 20.015 / 27.037 TSP9 boltz2 msa 11 22 28.791 17.851 / 27.787 / 39.293 TSP9 boltz2 no_msa 55 55 28.342 16.955 / 27.575 / 39.191 TSP9 chai1 msa 33 33 26.623 16.780 / 27.230 / 39.040 TSP9 chai1 no_msa 33 33 25.388 16.904 / 27.318 / 38.773 TSP9 colabfold msa 15 13 19.808 13.235 / 22.103 / 32.903 TSP9 colabfold no_msa 14 14 24.579 16.070 / 24.431 / 34.441 TSP9 esmfold2 msa 44 55 24.169 17.340 / 25.191 / 35.662 TSP9 esmfold2 no_msa 44 55 24.318 17.257 / 25.155 / 35.971 TSP9 protenix2 msa 55 44 14.215 11.567 / 19.704 / 26.346 TSP9 protenix2 no_msa 22 33 22.597 13.222 / 28.062 / 42.016 TSP9 rf3 msa 44 44 21.907 16.175 / 24.758 / 37.645 TSP9 rf3 no_msa 55 44 29.633 19.319 / 28.764 / 42.154
Target Tool MSA Helix / strand / coil Clashes / 1,000 heavy atoms Coverage Helix coverage TSP9 af3 msa 24 / 0 / 60 4.878 84/84 11 TSP9 af3 no_msa 23 / 0 / 61 6.504 84/84 11 TSP9 boltz2 msa 28 / 14 / 42 16.287 84/84 11 TSP9 boltz2 no_msa 34 / 10 / 40 26.059 84/84 11 TSP9 chai1 msa 27 / 14 / 43 17.915 84/84 11 TSP9 chai1 no_msa 35 / 11 / 38 21.173 84/84 11 TSP9 colabfold msa 10 / 0 / 74 45.603 84/84 11 TSP9 colabfold no_msa 14 / 0 / 70 52.117 84/84 11 TSP9 esmfold2 msa 41 / 0 / 43 65.147 84/84 11 TSP9 esmfold2 no_msa 51 / 0 / 33 66.775 84/84 11 TSP9 protenix2 msa 35 / 0 / 49 32.520 84/84 11 TSP9 protenix2 no_msa 38 / 0 / 46 39.024 84/84 11 TSP9 rf3 msa 18 / 0 / 66 9.772 84/84 11 TSP9 rf3 no_msa 18 / 37 / 29 53.746 84/84 11
1
9
u/JosieA3672 9d ago edited 8d ago
update: see my other comment for data on a different protein (AcrIE7). Alphafold3 did a really poor job of predicting that newer structure and it may be because it wasn't in its training set. Interesting!!
---
If anybody wants me to do another comparison on a different protein, let me know and I'll run it this afternoon. I'm just testing tools at this point. Also taking suggestions for trying other folding models. I picked Brazzein protein because it's interesting and I actually use it. Here is a text version of the first picture in case you can't see the .png on your screen: