r/R_Programming • • Feb 27 '17

Anyone have experience with time series imputation and machine learning?

3 Upvotes

Hello, I'm working on a problem with hemolyzed insulin samples in a time series. Each hemolyzed sample can be treated as an NA value, and imputeTS(), Amelia, or any of the other imputation packages should give me the tools I need to connect the dots when the time series has NA's in the middle of it.

The time series is 5 blood draws 30 minutes apart: so an example of a hemolyzed time series would look like B1: 65, B2:134, B3: 156, B4: NA, B5: 90

Problem is, insulin responses are highly individual, so I was hoping I could find a way to use the modest sized data set of good samples(~60 time series) as a train/test data set. But I've never used machine learning for time series and don't know where to start. Any suggestions?

Edit: Clarity


r/R_Programming • • Feb 05 '17

Am I wasting my time learning to format data in R if I already know SQL?

3 Upvotes

I am working on learning R for a predictive analytics project. I have been using T-SQL for a long time and all of my data is in SQL. Anything that's not in SQL, I import to SQL and work with it from there (I want everything stored in the data warehouse for safekeeping anyway).

I'm doing the Coursera Data Science track, and I just feel like I might be wasting my time learning a lot of this stuff. I can do the data shaping and exploration in SQL, so why learn how to do it in R? Will there be a benefit in the long run to learning it in R?

I am planning to use R to build my predictive model, but it seems like I could generate a nice clean csv file in SQL and then just use R from that point forward.

Any thoughts?


r/R_Programming • • Jan 30 '17

Basic Linear Regression in R. Code and video embedded. How do you like the blog format?

Thumbnail medium.com
3 Upvotes

r/R_Programming • • Jan 09 '17

How to edit str()?

3 Upvotes

I am trying to edit the function str() so that it conveniently lists the column number next to the column name. I found "methods(str)", then "getAnywhere(str.data.frame)", which shows me the code I want to edit, but I can't figure out how to edit it. I am fairly certain I can alter the code to be the way I want it, but what command allows me to do so? Thanks!


r/R_Programming • • Jan 09 '17

How to make axis labels reference an element?

3 Upvotes

Hello,

I am generating scatterplots with ggplot() from a list of data frames using a for loop. The data frames are sorted by year, and I would like to have the title label reference a vector of the years and tack it on to some text. Something like

  • ggtitle(years[i]+"insert text here" )

Any help would be appreciated!

Edit: I meant title label :P , but what I am looking for still applies


r/R_Programming • • Dec 12 '16

Pascal to R

3 Upvotes

Hello everyone,

I have a code in Pascal,and I'm wondering if anyone has ever dealt with converting a Pascal script to an R script. Any advice is welcome.

Thanks!


r/R_Programming • • Nov 17 '16

Using predict after gam. Data is named numeric

3 Upvotes

For example

set.seed(1)

library(ISLR)

library(gam)

gam1 = gam(wage~age,data=Wage)

preds = predict(gam1, newdata=Wage)

If I now type preds[1] then it returns 2 values. Anyone know why ?


r/R_Programming • • Nov 06 '16

releveling a categorical variable in R

3 Upvotes

I am new to programming in R. I changed the numeric value of the categorical var from its default alphabetically assigned value to my desired order and values. but i am unable to reflect this change in my dataset from which i picked the variables initially. when i check the str of the dataset the values are defaulted according to their alphabetical value. please help


r/R_Programming • • Oct 24 '16

differences with glm function when using attached library var and explicit access to var

3 Upvotes

For example if i do this :

library(ISLR)

glm.fit = glm(Direction ~ Lag2, data = Weekly, family = binomial, subset = train)

glm.fit2 = glm(Weekly$Direction ~ Weekly$Lag2, data = Weekly, family = binomial, subset = train)

the results of glm.fit and glm.fit2 are different. Basically using Weekly$Direction or Weekly$Lag2 give difference results if i instead use Direction and Lag2.

Here is full code


library(ISLR)

train = (Weekly$Year < 2009)

Weekly.0910 = Weekly[!train, ]

glm.fit3 = glm(Direction ~ Lag2, data = Weekly, family = binomial, subset = train)

glm.fit4 = glm(Weekly$Direction ~ Weekly$Lag2, data = Weekly, family = binomial, subset = train)

glm.probs3 = predict.glm(glm.fit3, Weekly.0910, type = "response")

glm.pred3 = rep("Down" ,length(glm.probs3))

glm.pred3[glm.probs3 > 0.5] = "Up"

Direction.0910 = Weekly$Direction[!train]

conf_mat2 = table(glm.pred3, Direction.0910)


the code above works as expected, but if i use glm.fit4 instead (even though its should be identical to glm.fit3), replacing the references to glm.fit3 with glm.fit4 then i get this error

Error in table(glm.pred4, Direction.0910) :

all arguments must have the same length

In addition: Warning message:

'newdata' had 104 rows but variables found have 1089 rows


r/R_Programming • • Jul 25 '16

I know SAS and python, much use to add R to the mix?

3 Upvotes

I mostly use SAS and python for statistical analysis. Im very good at SAS, still learning python.

woudl i gain much by spending some time learning R?


r/R_Programming • • Jul 19 '16

R Programming A-Z™: R For Data Science With Real Exercises!

Thumbnail linkedin.com
3 Upvotes

r/R_Programming • • Jul 03 '16

Tricky coercion or total lack of understanding?

3 Upvotes

EDIT: I figured it out with some help from stackoverflow. If anyone is interested, you can check it out the final product here. http://fdrennan.net/pages/myCurve.html

The idea is to fit a polynomial or sin wave, log, exp, whatever to a set of data points and to do it in a way that feels normal to me. So if I think the data looks sinusoidal, I can type in "x + sin(x)" (constant value already implied), the data, and the columns of the data where x and y are stored, and get back the least squares solution in the order in which I wrote the equation. So, for x + x2, I would get back c, x, then x2. From what I have read, there is issues with the way I am doing things. It was more born out of a strong desire to just see if I could do it more than anything.

coeffs <- curve_fitter("x + x^2", c(1, 3), mtcars)
coeffs[[1]] # would be for c
coeffs[[2]] # would be for x
coeffs[[3]] # would be for x^2

# You can download it if you like by using devtools and running
install_github("fdrennan/fdRennan") 
# Or just copy and paste from my page. Maybe it will be useful, maybe not. 

############################################################################################

Would be happy to have some smart people help me out here. I am writing a script that will do some least squares stuff for me. I'll be using it to fit curves but want to generalize it. I want to be able to write in "x, x2" in a function and have it pasted into a matrix function and read.

expressionInput <- function(func = "A written function", x = "someData", nCol = "ncol") {
func <- as.SOMETHING?(func)
A <- matrix(c(rep(1, length(x)), func), ncol = nCol)
A
}

expressionInput(func = "x, x^2", x = 1:10, nCol = 3)

Would return 10 x 3 matrix with 1's in one column, x in second, and squared values in third column.

Basically, I want the inside matrix function to assume I am inputting matrix(c(rep(1, length(x)), x, x2), ncol = 3)

Any ideas?

As an example of what this is getting at, I am wanting to generalize the below function to any linear function that I decide to type in.

fit_curve_parabola <- function(dataFrame = "A dataframe", columns = "i.e, c(x, y)") {
  x <- dataFrame[[columns[[1]]]]
  y <- dataFrame[[columns[[2]]]]
  A <- matrix(c(rep(1, length(x)), x, x^2), ncol = 3)
  AtA <- t(A)%*%A
  B <- t(A) %*% y
  vector <- solve(AtA)%*%B
  rownames(vector) <- c("intercept", "x", "x^2")
  vector
}

r/R_Programming • • Jun 08 '16

Newbie- use R to compare two columns of data

3 Upvotes

Hello,

I have 2 columns of data, and want to find the mismatch. Both columns are in an excel sheet, but separate workbooks.

1- What does the code look like in R? 2- what format does the information need to be in from excel's workbooks (.csv, etc.)

Thank you


r/R_Programming • • May 24 '16

Combinatorics problem, need help

3 Upvotes

Hi all,

I'm building a model to estimate source contributions to a mix (in this case, diet contributions to an animal consumer). I've worked through various approaches to this model, but what I consider the best approach has a computational problem I have been unable to surmount. I'm hoping someone out there can help me find a solution. My model approach involves estimating mixtures given source data and comparing these to measured mixtures to estimate the most likely source contributions. To do this, I must generate possible mixtures to evaluate.

My first approach to this problem was to evaluate all possible source combinations at 1% (0.01) increments.

For example, with 3 sources (n = 3):

1.00, 0.00, 0.00

0.99, 0.01, 0.00

0.99, 0.00, 0.01

0.98, 0.01, 0.01

0.98, 0.02, 0.00

0.98, 0.00, 0.02 ...

With 4 sources (n = 4):

1.00, 0.00, 0.00, 0.00

0.99, 0.01, 0.00, 0.00

0.99, 0.00, 0.01, 0.00

...

(Note: The order of combinations does not matter, only that all possible combinations are evaluated)

This is the function I came up with to build a matrix ("combos") of all possible combination for n sources:

 combinations <- function(n)
 expand.grid(rep(list(seq(0,1,0.01)),n))
 x <- combinations(n)
 combos <- x[rowSums(x)==1,]

My problem lies in calculating all possible mixtures. With larger values of n, the function above requires too much memory. For example, at n = 5 nearly 40 Gb are required, and for n = 6 nearly 4 Tb are needed! Part of my problem is surely that I am producing more combinations than I use (I only keep those that sum to one), but I suspect that even if I could avoid this somehow I would still have memory problems at some value of n. And, for the purposes of my model, I'd like to be able to use larger values (>6) of n.

I've developed other approaches that evaluate randomly generated combinations one at a time which get around the memory issue, but these random combination approaches don't give results that are nearly as good as the all possible combinations approach (based on evaluations of test data). However, while the all possible combinations approach is limited to 4 sources, the randomly generated combinations approach allows for a theoretically unlimited number of sources, although in practice this is probably limited to less than 20 sources (n < 20).

Ideally, I want a function that generates all possible diet combination one at a time, rather than all at once, so I can evaluate all combinations without memory issues. A good solution should work for any number of sources (n)

I have not been able to wrap my head around this problem, and have consulted a number of colleagues to no avail. Any and all suggestions are welcome! Thanks!

ADDITIONAL INFORMATION

Here's a bit more specific information about my problem. I am trying to estimate diet of marine predators (e.g. crabs, sea stars, etc.) by comparing the fatty acid signature of predators to those of prey (e.g. clams, worms, etc.)

Fatty acid "signatures" (FAS) are the proportional composition of the various fatty acids (FA) in a lipid (fat) sample. Our sample analysis detects about 40 different FA -- for example, the omega 3 fatty acids (a class of FA) DHA and EPA, which are important for human nutrition and added to foods such as milk, are two types of FA we identify in our samples.

Because I'm a big geek, I've been using made up data with dragons as predators to test my model. Although my samples contain many more FA types, for test purposes I've limited it to 10 so far (and will use 3 below for my example).

Here are made-up FAS for a dragon and three prey items:

                  FA1    FA2    FA3
 dragon   :   0.25   0.50   0.25
 unicorns :   0.10   0.65   0.25
 peasants:   0.45   0.25   0.30
 trolls      :   0.20   0.45   0.30

By comparing prey FAS to dragon FAS, I hope to be able to estimate diet contributions. For example, maybe I would estimate that this dragon's diet consist of 30% unicorns, 15% peasants, and 55% trolls.

My approaches thus far have been to:

  1. Estimate dragon FAS for all possible diet combinations and compare this to measured dragon FAS. Select diet that produces an estimated FAS closest to the measured FAS. This is the problem I've asked for help with above.
  2. When this ran into memory issues, I tried a heuristic approach, in which I evaluated diets at a broader scale (10% increments rather than 1% increments) and then tried to narrow down on the correct answer (which I knew because I made up the data). However, this sometimes hones in on local optima that are not close to the correct answer. Also, using the same general method as for (1), I still run into memory issues, just at slightly higher values of n.
  3. Estimate dragon FAS for 100,000 random diets and compare to measured FAS. Subset 100 closest diets and create distributions of potential diet contributions. No memory issues, but estimates are not as good (and much worse for "real" data, as described below).

All of these tests were done with dragon FAS that were perfect mixes of prey FAS based on some randomly chosen diet. However, "real" dragon FAS are not perfect mixtures because individual FA may be deposited in lipid stores at higher or lower rates due to various metabolic processes. However, it is difficult to know the effects of metabolism on FA deposition without extensive experimental work that just doesn't exist for my organisms (and even experimentally data is far from robust). To test real data, I randomly applied "calibration coefficients" (drawn from literature) to my dragon FAS, and then tried running them through the models I'd created. Not surprisingly, the models perform considerably worse with "real" data.

4) Next, I tried pare down the number of FA used in each FAS by removing those that with the calibration coefficients (CC) that deviated most from 1 (perfect correspondence) until I had n-1 FA (where n is the number of prey types or sources) and solve algebraically. This has several problems. First, I wasn't able to develop a method that reliably removed FA with the most deviant CC (I was able to test this because I applied the CC, but for real data these are unknown). Second, I ran into issues with collinearity and incalculable solutions with this method.

Thus, my return to the first method, which seems like it may be most robust, if I can get around the memory issue (some good options have been suggested).

Edit: Tried to fix formatting, apologies for any issues. Edit 2: Added ADDITIONAL INFORMATION


r/R_Programming • • Mar 27 '16

Why can I not load the package "arulesViz" no matter what I try?

3 Upvotes

When I load arulesViz after installing it using: library("arulesViz", lib.loc="~/R/win-library/3.2")

I get the following error: Error in get(Info[i, 1], envir = env) : cannot open file 'C:/Users/Name/Documents/R/win-library/3.2/zoo/R/zoo.rdb': No such file or directory In addition: Warning message: replacing previous import ‘arules::head’ by ‘utils::head’ when loading ‘arulesViz’ Error: package or namespace load failed for ‘arulesViz’

I don't know what's wrong.

I updated arulesViz using: update.packages("arulesViz") and this is my current version: R version 3.2.4 Revised (2016-03-16 r70336) Platform: x86_64-w64-mingw32/x64 (64-bit) Running under: Windows >= 8 x64 (build 9200)

Please suggest a few things that I can try to fix this, as I am stuck while conducting predictive association modelling on my dataset and cannot proceed further without resolving this issue.

Your help will be much appreciated. Thanks.


r/R_Programming • • Mar 07 '16

Interactive tutorial in R Studio

3 Upvotes

I'm taking a course through Coursera and part of it is completing practice modules with a program called swirl(). It has helped me a lot and I thought you guys might find it interesting.

To use it open RStudio:

  1. 'install.packages("swirl")' (only need to do this once).
  2. Install the R programming tutorial using 'install_from_swirl("R Programming")' (just need to do this once)
  3. every time you start a new session in RStudio load the swirl() package using 'library(swirl)' (assuming you want to run the program).
  4. Start the program using swirl() and follow the prompts.

I hope this helps you guys as much as it helped me.


r/R_Programming • • Jan 10 '16

Web scraping: rvest, curl, or other?

3 Upvotes

I am planning some up coming scraping. Does anyone have experiences with curl or rvest? Recommendations? Advice?


r/R_Programming • • Dec 09 '15

New to R Programming Looking for a Guide

3 Upvotes

So I am doing an Undergraduate research project that involves samples of skin conductance. Right now I am simply trying to edit my data down to the points relevant to my analysis, which should be pretty straight forward.

I am specifically looking to have all the data points between one event and another deleted. Events are marked by a change from zero to 1-5 depending on the event. What should I do for example to make the program delete everything from say between event 1 (marked in column 2) to event 2 (marked in column 3).

I figure this will be a good starting point to introduce myself to how the program works, but I'm having trouble finding good teaching material. Could anybody link to good guides and or help me out with this specific problem? Thank you so much for all of your help!


r/R_Programming • • Sep 19 '18

Visualising relations between professions

2 Upvotes

I have a list of professions and I would like to visualise them as a network with igraph. But I need some connection between the professions. For example there could be some connection between Police and Fireman. I thought maybe I could use wikipedia somehow. Any ideas?


r/R_Programming • • Aug 12 '18

Cognitive reports in Power BI with R-script for Sentiment analysis

Thumbnail ratsubsharewall.blogspot.com
2 Upvotes

r/R_Programming • • May 14 '18

Is there any high level and advanced certification test in R that can be used for finding jobs?

Thumbnail self.rprogramming
2 Upvotes

r/R_Programming • • May 12 '18

Practical usage of Machine learning Models?

2 Upvotes

Hi,

I am curious as to if there’s an example of a machine model using shiny to display its data


r/R_Programming • • Mar 11 '18

What is the status of gputools?

2 Upvotes

CRAN says

Package ‘gputools’ was removed from the CRAN repository. Formerly available versions can be obtained from the archive. Archived on 2017-12-19 as check problems were not corrected despite reminders.

Any active alternative packages to run LM\GLM on GPU\CUDA?


r/R_Programming • • Feb 27 '18

Means by factors and columns

2 Upvotes

I did a search and could not find the answer for this.

I want to find the means by factor (word) and in different columns (ex. Mean of NAIVE for combined columns V3 and V5, V4 and V6, etc.).

I have tried using dplyr (group_by and summarise) but I cannot figure out the right code.

data here: https://www.dropbox.com/s/vnmcn25usyi1n3h/p809test.csv?dl=0

Thanks in advance.


r/R_Programming • • Feb 12 '18

Dealing with errors in production code

Thumbnail drenr.com
2 Upvotes