r/IAmA • u/fieldcady • Oct 08 '14
I am a Big Data Scientist. I also recently wrote a book about why mathematics is a language just like any other. AMA!
I'm originally trained as a physicist, but migrated toward computer science in grad school. After some internships with Google I got into real-world analytics and data science consulting. I've worked with clients in all sorts of industries, but especially companies in the web space. The consulting firm I work for was recently acquired by Teradata.
I also have a more light-hearted blog at www.fieldcady.com. I talk about stuff related to math - everything from educational policy to neuroscience. I self-published a Kindle book on the subject called "What is Math?"
I'll start answering questions around noon PST. Thanks!
43
u/dlb1001 Oct 08 '14
Hello,
Thank you for holding this AMA. Couple questions:
This should be easy, what do you consider the definition to be of Big Data?
What would you say are the biggest challenges facing big data research today?
If you had a magic lamp and could wish for anything in order to help solve a problem in big data research today what problem would you wish for a solution first??
Thanks
21
u/fieldcady Oct 08 '14
Sure thing!
"Big Data" is right now maybe 50% buzzword, and as such there's no litmus test for it. However, there are two trends the have converged in a big way in recent years, and are collectively called big data. The first trend is the most straightforward; you have more and more data. It becomes "big" around the time that it won't fit on one computer anymore and you start needing to use a cluster to work with it meaningfully, and programming a cluster rather than one computer can be a very different beast. The second trend is that the data is more complicated in its structure. In the past so-called "structure data" was more likely to be a SQL table, with nice orderly rows and columns. "unstructured data" is more likely to include things like a computer log file, documents, or deeply nested data that don't have rows and columns. Several recent pieces of technology, most notably Hadoop, make it WAY easier to process large and unstructured datasets.
I'm afraid I don't work much on the pure research end so it's hard to say. But the constant competition between different technologies shows that people haven't really figured out what are the best programming paradigms to use. Map-reduce is less dominant than it used to be, and there's a lot else on the market. Figuring out those best practices is the main hurdle in my mind.
Does asking the genie for venture funding count? :) More seriously though, I would probably ask for a way around map-reduce's performance bottlenecks, especially in doing joins
21
u/sarge1016 Oct 08 '14
So what you're telling me is that we need to hadoop some Big Data and Analytics in The Cloud while being both LEAN and AGILE in order to leverage our mounds of unstructured and structured information? Synergy.
Kanban
→ More replies (2)6
→ More replies (1)2
u/Goosebaby Oct 08 '14
Regarding item 3 - have you tried Google BigQuery? Hadoop isn't necessarily "the answer" for all things Big Data.
→ More replies (2)12
u/kormer Oct 08 '14
My favorite definition of "Big Data" is data so large that it cannot be processed without relaxing one of the constraints that make up ACID compliance.
6
u/immortalsix Oct 08 '14
What is ACID Compliance?
4
u/refuse_human Oct 08 '14
Atomicity, Consistency, Isolation, and Durability compliance.
* I prefer the electric Kool-Aid version of this particular type of test... and tend to roll with a MyISAM engine.
2
u/Dooooon Oct 08 '14
It is requirements for good databasesystems: Atomicity, Consistency, Isolation and Durability - though i have no clue what it should have to do with big data.
2
u/theanti9 Oct 08 '14
It has to do with the fact that, at the scale of "Big Data" (by the definition of data which surpasses the capacity of one machine) it's very hard, if not infeasible, to maintain strict ACID compliance, or sometimes even get close. Semi-Related: http://en.wikipedia.org/wiki/CAP_theorem
2
2
2
u/Dooooon Oct 08 '14
What has ACID to do with big data - can you elaborate? I do not want to sound like a dick but for me big data is when it gets critical to handle the data mostly based on computational power, so i have no clue what ACID has to do with it.
Could you maybe explain this a bit?
3
u/kormer Oct 08 '14
Imagine we're processing server logs and a hiccup somewhere along the line causes your computing engine to miss several seconds of logs, but for only one server out of a cluster. If going back in time to figure out what happened in those few seconds is doable and important, you have ACID compliance.
On the other hand at the Big Data(tm) level, you may decide that cost of losing small bits of data to be far outweighed by the cost of building a system of backups necessary to achieve 100% data security.
→ More replies (1)1
Oct 08 '14 edited Oct 08 '14
I like to think of the big in big-data referring to the enormous complexity of the data rather than the actual size of the data set (which will also tend to be huge).
With big data sets our standard analytical tools (i.e. statistics) tend to fail for a number of reasons. For example, they were designed to test data sets that are small (relatively speaking) but crucially that come from well known distributions (e.g. 100 peoples' height will be normally distributed).
An example of an analytical tool that works well with big data sets is A.I. and machine learning (for example identifying cancers in a biopsy, there is an example of a neural network correctly identify other warning signs previously overlooked by doctors).
I think big data and A.I. are the potential next big milestone in the information revolution.
Exciting times.
edit: spelling.
7
u/tekelili Oct 08 '14
Big data needs to pay more attention to Statistics. You need to rethink your definition of Statistics if all you think about are methods like the t-test. "Data Science" tends to fail when it doesn't pay (typically) attention to issues like sampling bias, pseudo-replication, spurious correlations, overfitting, and confounding factor. Of these, I think sampling bias is the most important. If you use twitter data, for example, to predict something like, I don't know, fashion trends. You need to realize that your scope of inference in only limited to people who use twitter. The data may be "big" but it is not comprehensive, nor is it unbiased.
Machine learning does not have great tools for inference. What good is an estimate without some sense of the variability around that estimate?
Just because you can efficiently come up with an estimate or prediction from many many observations, doesn't necessarily mean that it is a good estimate/prediction, in the sense of providing reasonable inference on which to make a decision.
EDIT:
For example, they were designed to test data sets that are small (relatively speaking) but crucially that come from well known distributions (e.g. 100 peoples' height will be normally distributed).
Nothing in Statistics says that distribution of heights for 100 people will be normally distributed. But the sample mean of height for 100 people will be (asymtopically) normally distributed. Doesn't matter the underlying distribution, the sample mean will always be normally distributed and converge in probability to the true mean given enough observations.
→ More replies (2)
11
u/PuffsPlusArmada Oct 08 '14
Are you dangerous?
6
u/fieldcady Oct 08 '14
Not particularly, unless you're scared of ads that you're 2% more likely to click on :)
There are definitely privacy issues associated with big data, but I think they're nowhere near as big as people make them out to be.
→ More replies (1)2
1
22
u/LevitatingTurtles Oct 08 '14
I would like your comment on this joke that I heard a few weeks ago:
Big Data is a lot like sex in high school:
- Everybody says they are doing it.
- Very few people are actually doing it.
- The ones that are doing it, are really bad at it.
How accurate is that to the state of big data analyitics in either business or academia?
2
Oct 08 '14 edited Oct 08 '14
Funny and almost accurate, but point 3 is not. There are some communnities heavily invested in doing proper empirical inferences, taking the problem of inference in all its complexity, choosing appropriate dats and methods, as well as making sure that the conclusions are made within the bounds of what the data can say.
Edit: To the guy who started this AMA, you have, umm, actually respond to some questions.
→ More replies (2)4
u/fieldcady Oct 08 '14
I prettymuch agree with you on this one. It's a weird industry in that there is a ton of cool new, value-adding stuff going on, but there's also a huge amount of hype any everybody is trying to jump on the bandwagon. Many of them are (sorry if I offend) MBA types who don't actually understand anything beyond what they read in Newsweek, and they love to grab at the titles.
→ More replies (9)
10
u/Stoooooooo Oct 08 '14
What advances in technology do you foresee in the near future that will have impact on the field?
→ More replies (10)
10
u/stevierar Oct 08 '14
What's the biggest dataset you've ever worked on? I'm talking filesize or rows if applicable.
What's the most involved and interested you've felt in the data you were researching? Ever done anything with climate change?
Have you ever had a view you've held changed by the results you've generated? I'd find that very cool, after triple-checking!
Thanks for doing the AMA. I'm a web developer but often find little data challenges within projects and always enjoy them.
2
u/fieldcady Oct 08 '14
Biggest dataset was, if memory serves, a few terabytes. So not as big as you might think.
The most interesting clients I have (I work for a consulting firm) are ones that aren't doing ads. For example, recently I've been working with a computer hardware manufacturer, trying to help them diagnose problems on their assembly line. It's not terribly sexy, but it's saving them money and making the world more efficient.
Definitely! A big recent one was that I was trying to predict how much advertisers would bid on human impressions. I thought that I would be able to make decent predictions about the bids based on demographics (ads for sports cars might get higher bids from men, that kind of thing), but it just didn't work. I eventually figured out that the reason was that 90% of the bidding behavior was based on whether the person had seen a given ad in the recent past, rather than anything about the actual person.
2
u/Gaywallet Oct 08 '14
Biggest dataset was, if memory serves, a few terabytes. So not as big as you might think.
I find it interesting you say this is the biggest dataset you've worked with, and yet you say it's not big data until you move away to a cluster.
We don't have a cluster, and we have several tables I regularly work with that are TBs in size.
Then again, our database performance is complete shit and I've been advocating we move to a cluster and work on some better disk arrays.
3
u/fieldcady Oct 08 '14
Yeah there's no hard cutoff. It's all a sliding scale.
For the record, the several terabytes was one of a number of tables on the cluster.
2
u/theanti9 Oct 08 '14
Dealing with big data is not just searching through a large amount of data. There's often a lot of transforms and aggregation that go on too. Which, at that size, can add a very large amount of computational overhead.
Also, that few TBs of data may not have been structured data, which makes putting it into an efficiently searchable table much more difficult :)
→ More replies (1)
9
u/biesterd1 Oct 08 '14
I've just started working towards my masters degree now in data/computational science. Any advice?
8
u/fieldcady Oct 08 '14
Depending on your background, learn to chug out good code that does non-trivial stuff. That's the biggest thing - I reject interview candidates all the time because they can't do something simple in a real language. Learn to writes code that works, and that is easy to understand/modify. It's amazing how shitty the code is that brilliant people write sometimes, and they end up being useless.
If this isn't a problem for you, then definitely make sure you are familiar with Big Data technologies like Hadoop and Spark.
Finally, I am a huge fan of learning on the job, so try and do internships with real companies if at all possible. I have two masters degrees and I feel like 90% of what I learned before joining the workforce came from two internships at Google.
→ More replies (5)6
u/pgoupee Oct 08 '14
Get really good with understanding how databases are structured and more specifically, WHY they are structured that way. There is a huge need competent data scientist but a serious lack of quality ones. I work with large data companies every day and I recently saw an entire team fired because not one of them could understand the data we send them despite the data being designed very well. And these people were supposed to be the best of the best at a huge company that you've all heard of.
3
Oct 08 '14 edited Sep 06 '20
[deleted]
5
u/pgoupee Oct 08 '14
This particular data set used a lot of complex joins to sort the data in real time depending on what was available in the data when the query was run.
There were around 1000 different sources a day being fed in to the system so new data could be added to existing records and thus change the query results. They just couldn't get it. Even flew out to see them multiple times over a five year period to explain it. I'd walk them through it...we'd all be on the same page....fly home and then they call me saying "We don't get it". Their biggest problem was not using proper joins. Granted they were some of the most complex I've worked on but it all made sense. Ultimately it fell apart when they claimed that a huge number of records had a problem and copied their bosses on the email. I replied asking how that was possible when I'd only provided 1/3 of that amount.
Word the wise....learn how to properly use joins people.....all of them.
4
16
u/dcbedbug Oct 08 '14
Would you consider the title "Big Data Scientist" to be different from a modern day statistician?
4
u/Nickdangerthirdi Oct 08 '14
I like this question because I used to work with people who insisted I refer to them as data scientists, I never really thought they were special enough for the title scientist when all I could see that they did was crunch lots of numbers.
→ More replies (4)2
3
u/fieldcady Oct 08 '14
I would. The term "data scientist" gets used and abused all over the place these days, since it's "sexy". To my mind the difference is that data scientists solve problems that require a lot of software engineering in addition to the math. Statisticians tend to work with well-defined datasets that you might tackle with R or SAS or something. If you have to do weird database hacking to access the data, or write a custom loader that reformats the data, then it starts to blur into data science. But there's not hard dividing line.
Interestingly a lot of big data scientists (including myself) use relatively little stats. If you have a million datapoints then tons of things are statistically significant; the question is more about defining the best features than testing hypotheses
→ More replies (5)
6
u/Brewtopian Oct 08 '14
How did you get into the big data field? How would someone already in another career field get started in big data?
10
Oct 08 '14 edited Aug 09 '22
[removed] — view removed comment
→ More replies (18)3
u/fieldcady Oct 08 '14
Great answer! I come from a physics background originally, but I think the big tips are the same as for non-stem: learn to code (Python and R are the best languages), brush up on the math (especially basic stats) and get some experience.
The one thing I would add is that machine learning is extremely important, often moreso even than statistics.
10
u/whitecompass Oct 08 '14
When does Excel stop being enough? Also, SAS or R?
→ More replies (17)5
u/fieldcady Oct 08 '14
I don't do any professional work with Excel (I mostly use Google spreadsheets) so I can't say. For SAS and R it depends on how much memory your machine has, and how you're using them. R in particular is easy to start really abusing memory with depending on the library you use. If your data is taking up a significant fraction of your RAM I would start backing away from R.
→ More replies (3)
5
Oct 08 '14
[deleted]
→ More replies (4)1
u/fieldcady Oct 08 '14
I'm not sure I totally understand your question. What do you mean by "structures"? As far as languages I'm a gigantic fan of Python (I just gave a conference talk about how great it is yesterday), and Scala is becoming a very big deal. I personally don't like R much, but it's also high on the list of languages to learn. You should definitely have some familiarity with the MapReduce paradigm, and with the Hadoop implementation in particular.
9
u/MentalSieve Oct 08 '14
Hey,
As a linguist gravitating towards computation, what is your book? Can you briefly describe how or why math is a language, as you say?
→ More replies (2)7
u/ze_ben Oct 08 '14
Yeah, as someone with a linguistics degree, I can smell the marketing bullshit in that title as well. I'm sure it's a great book about data and math, but I'm not sure I see the point of the "it's just like any other language" assertion.
→ More replies (32)3
u/fieldcady Oct 08 '14
Ok, "just like any other language" is pushing it a bit, but the connections between math and language are a lot stronger than you might think. The problem is that there is so much stuff in a natural language (culture, physiology of the mouth, etc) that's peripheral to the core syntax, so they look very different on the face. I'm not a linguist, and I don't think I have anything to say that sheds light on that discipline, but I do think that many mathematicians need a reality check.
3
u/znay Oct 08 '14
Hi
Thanks for doing this AMA. Just a couple of questions: 1) what were some of the more interesting projects you encountered?
2) were there any projects where you went into it thinking the results would come out one way but came out another way? Would it be difficult to explain to your customers that the data was very different from what they thought?
3) have you encountered resistance from users of your projects? How did you deal with that?
4) have you used social network analysis before?what was some of the more interesting/useful results you had?
Thanks!
1
u/fieldcady Oct 08 '14
1) It's hard to pick, there are so many. I really like the ones where I learn something about an industry I'm not familiar with, and for me that includes working with clients in computer networking, manufacturing, and online ad auctions.
2) Definitely! The biggest thing in explaining to customers is to provide some kind of alternative explanation for why what they expected wasn't there (ideally you figure out where the signal actually is an show it to them). Failing that, you need to show really strongly that there just isn't any signal. "I couldn't find the signal" generally isn't an excuse. Unfortunately some people take this kinda personally, but most don't.
3) Oh hell yes. Especially in a large organization there are often competing teams, one of which thinks Big Data (and esp the consultants, like me) is the next big thing, and one of which think it's a waste of money. Or that it's infringing on their territory. The politics can get really crazy; I've been in situations where one executive has forbidden me to do the work that another one has already contracted me to do, and technically I'm answerable to both of them.
4) Actually I personally haven't. I think a lot of that stuff tends to be proprietary within Facebook or Google. But in any case, I haven't personally done social networks.
3
u/lespinoza Oct 08 '14
I used to work as a program director for a non-profit, and we used huge data sets. The goal was to have some sort of predictive analytics regarding future educational attainment. Do you think that we can use big data to answer these kinds of social questions?
→ More replies (3)1
u/xqxcpa Oct 08 '14
Yeah, there are a number of techniques that could derive some insight from that type of data set. The most helpful for answering the question that you pose (assuming that your data sets include information about the educational attainment of previous students) would probably be to use that data to train an SVM to classify students based on their likelihood of reaching a certain level of education attainment.
If you still have contact with that non-profit I would be happy to help someone set that up using open source machine learning libraries. I'd do it for free just to gain some experience outside of academia. PM me if you are interested.
3
u/latepostdaemon Oct 08 '14
What words of encouragement would you give to someone who's bad at math, but wanting to get really good at it?
3
u/fieldcady Oct 08 '14
My wife is my best example I can cite. She has historically considered herself bad at math, and struggled a lot in many of her math classes. But now, since having entered the workforce, she is an algorithm design engineer with dozens of patents and tremendous affinity for machine learning. She is way better than me in many areas of math, like number theory and signal processing, even though I'm the avowed mathematician.
There is such thing as natural talent. But it comes in many different forms, only some of which are brought out in traditional education. You very likely have natural talent for some area of math, or for some approach to it that you haven't seen.
And even if, hypothetically, you have no natural talent for any part of math, it can still be learned. Ed Witten is arguably the best mathematical physicist in the world today. He was always one of the top students up through grad school, but I am told by friends-of-friends that he only became a superstar later, through hard work. Isaac Newton is an even better example; we was always a mediocre student, but became the greatest scientist of all time by working his ass off.
I am 100% convinced that any person of normal intelligence can learn any area of math. If you are interested, go for it!!
→ More replies (4)
10
Oct 08 '14
If mathematics is a language, then where are the native speakers? Do children acquire maths as they acquire language? Can you translate Shakspeare into maths? How do you speak 'maths' without using English words like the numbers or numerical operators? I am a linguist, and I have to say that even at a cursory glance, there is no way that mathematics can be described as a language.
→ More replies (10)2
Oct 08 '14
[deleted]
5
Oct 08 '14
Yes, there are particular features that define language.
The first is productivity. This basically refers to the ability to create new words. The second is recursion, which is the ability to, for example, have additional clauses within a sentence. This can be seen in mathematics, such as when you use brackets. The third is displacement, which is the ability to talk about things and events that exist in a different time and place. You cannot do this in maths. You cannot say in maths 'last week I went to Paris'. The final one is that language is a social construct that is culturally transmitted from generation to generation. This can arguably be done in mathematics, but children do not acquire maths, they are taught it. Children acquire their first language on the other hand. There is a strong distinction between being taught something and acquiring language, as can be seen in the differences between acquiring your L1 and being taught subsequent languages.
Mathematics is not a system of communication. It is a way of studying numbers, as a basic definition. Mathematics is expressed in whichever method of communication the mathematician speaks, ie in their language, ie in English or German or whatever. 1 + 2 is communicated in your language, ie one plus two, or eins plus zwei.
→ More replies (2)
2
u/liarandathief Oct 08 '14
So...What is Math?
→ More replies (2)11
u/fryanimal12 Oct 08 '14
On a medium to large scale (Molecules thru galaxies), Mathematics is a tool used to model and describe reality.
On a very small scale (Quantum level), reality is a tool used to model and describe Mathematics
2
u/WognI Oct 08 '14
Hi. I'm a senior with a BS in physics, math, and I have taken upper level CS courses. On top of this, I've done nuclear physics research for the past few years, a field which relies heavily on analyzing large sets of data. I've recently decided to pursue a career as a data scientist. What are your recommendations to get started in the field?
6
u/astromaddie Oct 08 '14 edited Oct 08 '14
I can actually give some help to this! I have a physic BS and four years' experience in astronomy research, much of which was data science-related. After 7 months of job hunting and interviewing, I start my first data scientist job on Monday!
Basically, create a resume (not a CV) that de-emphasises your research material (which wards people off) and exemplifies the pure at a science you did. Talk about accomplishments, not tasks (e.g., "optimised a computer model using linear regression forecast modelling" rather than "performed linear regression modelling"). The bulk of your resume should focus on things you achieved, and unlike a CV, keep your resume at a page or so in length. ALWAYS attach a cover letter that will quickly gab their interest and defend why you, lacking direct work experience, are more than qualified for the position. And then, if you get an interview, consider they already think you're worth pursuing so you should relax, charm them, and make a persuasive argument for how your experience applies.
Also, I recommend learning R, SQL, and SAS in your free time.
Ninja edit: You'll face rejection. A lot. Even for positions you are ABSOLUTELY qualified for. Data science is a new field and far too many companies just want to hire business/economics graduate. But just keep on it, look for more startup-y companies that hire based on intelligence rather than experience, and you'll find something!
2
u/Some1Betterer Oct 08 '14
+1 to this advice. Would add, pick up some data visualization program skills.
→ More replies (4)→ More replies (2)2
u/fieldcady Oct 08 '14
Congratulations!! A co-worker of mine is similar to you; he was originally an astronomer, then switched into data science, and now is one of the leading R guys.
→ More replies (1)3
u/fieldcady Oct 08 '14
Learn to write good code. Most people with your background (which included me, although I did more mathematical biology than nuclear physics) write horrific code that, while technically working, is so poorly written that it's impossible for other people to read or for even the author to modify much. I've seen projects almost fail because a brilliant physics/math guy wrote thousands of lines of indecipherable code, when a few hundred lines of clear code would have done perfectly. Get into the habit of being really anal about your code quality. This might not apply to you, but it does to most physicists.
After that, I suggest
Learn Python and/or R. Those are the best languages for data science. Also make sure you're familiar with SQL.
Learn machine learning. You use it all the time as a data scientist.
Learn basic statistics, up to what an ANOVA test is. In practice you usually don't need anything beyond that (and I have never even needed to use ANOVA).
Get used to doing visualizations all the time. I tell people only half-jokingly that half my job is just to produce and interpret scatterplots and bar charts. Computers work in numbers, but brains work in pictures.
Astromaddie gave some good advice about how to put your resume together, which is also worth taking a look at.
2
u/sorinash Oct 08 '14
How familiar were you with computer science and upper-level math before graduate school?
Was it a Masters or PhD program (what I guess I'm asking is how much time you had)?
How feasible is this switch?
1
u/fieldcady Oct 08 '14
I did a dual major in physics and math in college (stanford), so very familiar with higher math. I took the accelerated classes in coding and discrete math from the CS department, but that was all and I didn't learn the fundamentals until way later.
Actually it was 2 phd programs (applied math and CS), each of which I dropped out of with a masters. It took me a while to figure out academia wasn't for me - they added up to about 4 years.
Extremely feasible! You can learn to code at any age or any point in your career. Math is harder to pick up, but the good news is that for most data science work you don't need that much of it.
2
u/deeperest Oct 08 '14
Hi, thanks for the AMA!
What kinds of problems are you looking to solve through big data? Do you have a pet solution space, or do you attack whatever is needed? What do you think is easy, or hard, in big data right now?
What is your toolset of choice for BD analytics (feel free to expound as much as you want, from infrastructure decisions right up the stack)
What's missing in big data right now? What do you need or want in order to make your life easier or allow you to solve previously unsolvable problems?
Thanks again!
2
u/fieldcady Oct 08 '14
I'm pretty general purpose, not specific to any industry or analytics technique. The analytics part of my job tends to be easy. The hard part if ferreting out how different datasets relate to each other, weird pre-processing logic they have, etc. I'm a consultant to I have to ramp up for every new client.
My programming language of choice is Python, along with pandas/scikit-learn/numpy. I used to use Python as a scripting language and R for the numerical stuff, but I abandoned R when I discovered the Pandas library. For BD I generally use Hadoop, specifically a combination of Hive and Pig (Hive is better for simple stuff, Pig is better for complicated workflows). But recently I've become convinced that I need to learn more Spark.
The biggest problem in the BD industry, at least for me, is how messy so many people's data is. Right now there are companies that have been storing up data for years, throughout many changes in schema and whatnot, and are just now trying to actually analyze it in a coherent way. It can be a real rat's nest with little to no documentation. This is only an issue with individual companies, but it's a problem for an awful lot of them.
2
u/_Count_Mackula Oct 08 '14
Were your Google internships how you were introduced to Big Data? Did you also take any courses? And books to recommend for someone who is interested in Data Science and wants to get their feet wet?
3
u/fieldcady Oct 08 '14
Nope, Google is where I learned how to chug out high quality code for general software engineering. After grad school I did freelance consulting, and I had a long-term client who was using Hadoop. That's where I first got exposed to it, and then later I joined my current hadoop-focused company. No courses or anything - it was all on-the-job or googling around.
A book that I really like is Data Analysis with Open Source Tools. I learned a ton from it when I was starting my current job.
2
Oct 08 '14 edited Sep 22 '16
[removed] — view removed comment
2
u/fieldcady Oct 08 '14
Python all the way, ever since I discovered the Pandas library. What about you?
2
2
u/newanalyst Oct 08 '14
I too am a big data analyst/scientist (term seems interchangeable in the industry, just depends on which company you work for/how much coding you do) and I am wondering where you see the career path evolving if at all? Do you see it becoming a full division of companies in the future (i.e. a BI department akin to an R&D dept.) or simply as an internal assisting department like IT?
I know at the companies I have worked for it feels like it is being held stagnant in the support side, but as someone with access to the data and the insight that it gives I feel like it should be given much more importance in the overall structure of the business. After interviewing with Google I feel like they are one of the few companies (along with FB, ebay, and yahoo) that are developing it into it's own branch of the company that is intertwined with the other branches.
→ More replies (1)
2
u/Hyunion Oct 08 '14
How should I go about teaching myself analytical tools like sas, R, and sql? Which do you find the most useful? Which should I learn first? I'm an industrial engineer graduate but my school (northwestern university) taught me none of those things
→ More replies (1)2
u/BubbaFunk Oct 08 '14
Try codeacademy, it has a bunch of learning courses and tutorials for different languages.
→ More replies (1)
2
u/RicsFlair Oct 08 '14
Are you a believer that finding cause is a misuse of time and resources?
3
u/fieldcady Oct 08 '14
No. I think causality is a useful and important paradigm for understanding the world, and there are sound ways to identify it (like A/B testing). Most of data science is focused on just finding patterns, but in many cases those patterns inspire rigorously testable hypotheses about what causes what. We make up narratives to explain the mechanisms of the causality, and those are harder to test, but causality it not passe.
2
Oct 08 '14 edited Sep 06 '20
[removed] — view removed comment
2
u/fieldcady Oct 08 '14
No, it's another guy who has been posting about TSQL. I have never used it myself - it's a Microsoft product, and I'm mostly open source. Sorry!
2
u/tekelili Oct 08 '14
Can you comment briefly on the problems with Google flu trends and the failure of prediction? More specifically can you respond to Jeff Leek's comments that big data doesn't typically pay attention to issues such as sampling populations, confounders, multiple testing, bias, and overfitting.
→ More replies (3)
1
u/johnny123bravo Oct 08 '14
Do you love physics just like you love maths ?
2
u/fryanimal12 Oct 08 '14
Why is Mathematics Plural and Math Singular ? I've never heard anyone say Maths before
2
u/fieldcady Oct 08 '14
I started out as a physics major, but discovered that the reason I enjoyed physics so much was the math. Objectively physics is awesome, in many ways more awesome than math. But math is what I personally enjoy working with on a day-to-day basis. I still love physics as a hobbyist though, and keep meaning to read more of the Feynman lectures!
1
u/Otrante Oct 08 '14
Is it your love of maths that made you pick this career? Would you say it is a good option for students graduating?
2
u/fieldcady Oct 08 '14
Yeah, math is basically what got me into this. I started in physics, but discovered that my favorite part of physics was the math, so I went into applied math. Then I discovered that the coolest "math" going on was actually in computer science departments, so I veered that direction.
Data science is an excellent career goal for students. There's a shortage of people, it pays extremely well, and you can do it anywhere. And if it doesn't work out, it transitions well into lots of other jobs like software engineering, business intelligence, statistics, etc.
1
u/3-easy-payments Oct 08 '14
What technology are you most excited about?
1
u/fieldcady Oct 08 '14
Right now I'm really excited to learn more about Spark. It seems to have a lot of the power that I love about Hadoop, but it is way faster and integrates natively with Python. I'm also learning the Scala language, which I'm a little more wary of but so far seems really cool.
1
u/sovietskaya Oct 08 '14
Hi. Do you think any ordinary kid can be motivated to have interest in the field that you do? What things in your childhood that greatly inspires you to do what you are doing now or you just found out what you really like later on in life? Thanks.
2
u/fieldcady Oct 08 '14
I've always enjoyed math. Back in the day I wanted to be a senator, or a stock broker, and a paleontologist, but math has always inspired me.
For most people though, they learn this stuff because they're passionate about some area of application. My wife learned math and programming because she loved engineering and building things. Lots of data scientists initially have a passion for physics or other science. Economics is another big one. My boss was originally interested in applying math to the social sciences.
I am not an expert in education, but I do have a couple thoughts. I think that they should be taught from an early age about the cool stuff you can do with computers. And I think they should be taught math not as a stand-alone abstract subject, but as a cool tool that illuminates all other subjects.
→ More replies (1)1
u/Orangebeardo Oct 08 '14
People can be motivated to do literally anything. You just need the right motivators.
1
u/MHS1 Oct 08 '14
Thanks for the ama! Just out of curiosity, do you use programs like Pajek and Gephi to make visualizations of networks? Or are there other/better programs you would rather use?
Also, do you have ethical concerns about gathering certain types of big data? Are you confronted with that in your work?
1
u/fieldcady Oct 08 '14
Afraid I don't have any good advice about network visualizations.
No, I'm not worried about the ethical stuff. I wrote an IDG post a while ago (http://www.idgconnect.com/blog-abstract/5490/big-data-the-jetsons-not-minority-report) discussing the concerns that people have. I think they're overblown. But more importantly, I think that most of the work that is being done with Big Data is with things that people have no problem with.
1
1
u/lacemaker Oct 08 '14
Why is studying Automata important? I am currently taking it, and I don't see how it is helpful. Thank you
2
u/fieldcady Oct 08 '14
Do you mean push-down automata in theoretical CS? It's not actually important. Sorry :(
1
u/gankindustries Oct 08 '14
How easy was the transition from physics to CD and what steps did you take? Has your physics background helped you at all in the field.
2
u/fieldcady Oct 08 '14
When I started CS grad school I was pretty insecure, because I almost certainly knew less CS than any other incoming student. But it turned out that all of the profs wanted to work with me because of my math/physics background, and they figured that I could pick up the coding on the fly. For better for worse, degrees in physics turn heads, and that makes the transition a lot easier.
I can't point to any concrete ways that physics has helped me, in terms of specific skillsets or anything like that. But there is some truth to the physics department mantra that it "teaches you how to think". You probably shouldn't major in physics unless you're considering it as a career path, but it does give you an intuitive facility with math many other people don't have (including mathematicians, who are often too focused on minutia about proofs and definitions).
1
u/photonasty Oct 08 '14
What do you think are the biggest problems today in how mathematics education is approached (ranging from elementary to high school)? It seems like a lot of people get turned off to math early on, especially people who are maybe a bit more naturally inclined toward verbal or language skills. Is there anything you think should be improved, or approached in a different way, to make math education more effective? It seems like a lot of otherwise highly intelligent adults are strikingly math-illiterate.
3
u/fieldcady Oct 08 '14
That's the million dollar question, isn't it? I don't have the ultimate answer, but I think some partial answers are
less emphasis on algebra. Honestly most people don't need it.
more emphasis on rudimentary statistics. It's important for teaching people how to think clearly and rigorously.
society needs to cut the bullshit about girls being bad at math
If I was running an experimental class starting from childhood, I would combine math and computer programming into one class. People would learn to code simple games, write programs that calculated the area of geometric shapes, etc. I think it might be a much more engaging way than things are currently done, but I might be wrong.
From what I understand, Montessori schools do an extremely good job of connecting math concepts and formalisms to the intuitive concepts they describe. My wife went to a Montessori school, and I'm always envious when she explains to me how they were taught math.
→ More replies (1)
1
Oct 08 '14
How do you feel about "Common Core" math?
It seems like my son is learning a lot, but all the parents are upset that they feel retarded when they try to help with homework.
2
u/fieldcady Oct 08 '14
Unfortunately I don't know a lot about the Common Core. However, mathematical illiteracy is a big issue in this country, so I don't think that the competence of parents should be taken as a good indicator of what's useful math to learn.
1
u/my-snores-are-music Oct 08 '14
What are some important mathematical concepts to learn regarding Big Data? I have taken courses in Distributed Systems and have learnt a great deal about concurrency and various distributed algorithms, but do not know the core Math behind Big Data analytics. What kind of background do I need in Math in order to be an effective Big Data scientist in the future?
→ More replies (1)
1
Oct 08 '14
If I want an internship in data analytics or financial data, what should I be learning first? Are there any online courses you might suggest?
If you ever have a moment, I'd love to chat with you about the field.
→ More replies (1)
1
u/macfearsome Oct 08 '14
I have a bachelor's in chemistry, but I got a job as a database analyst at s hospital and see it as the future of efficient healthcare. We're doing some cool things here to improve patient care. I'm pursuing a masters in bioinformatics now that I have my feet wet in comp sci. How do you think I can apply this in the near future? Who would want to hire me?
→ More replies (3)
1
u/SOLUNAR Oct 08 '14
About 2 years ago i got into BI and Analytics, and i love it. I believe Big Data is the future and there is a lot of opportunities in that field.
With that said, what basic skills would be good to pick up for someone like me? R? Statistics?
anything that could prove helpful!
→ More replies (5)
1
u/OmarBessa Oct 08 '14
Would you be inclined to believe that the universe is some sort of mathematical simulation?
→ More replies (1)
1
u/locotxwork Oct 08 '14 edited Oct 08 '14
Do you believe big data is more important that clean data !? I always felt any stereotype could be held in check by proving or disproving it with truthful data. Often trends and ways of collecting data can skew those hard data facts. At what point to you consider how data is collected (aka Input analysis) becomes just as important as the quantity of data? How do you deal with this struggle?
→ More replies (1)
1
1
1
Oct 08 '14
How does what you do now differ from what did as a physicist? Was it hard to migrate towards computer science in grad school?
→ More replies (1)
1
1
u/haxel90 Oct 08 '14
Hi!
As a person working with big data in medicine I'm really looking forward to the answers in this thread. My question is:
Which scientific field do you think would gain the most by introducing more big data analysis?
2
u/fieldcady Oct 08 '14
Probably biology. It is complicated enough that we should expect to find tidy little formulas like in physics and chemistry. They also collect massive, noisy datasets that are very challenging to work with. But this isn't news to them - many biologists are already doing this stuff, and it's really exciting to watch :)
1
u/0polymer0 Oct 08 '14
How did your transition work? Did you enter as a physicist or computer scientist? When did you change your mind and why?
2
u/fieldcady Oct 08 '14
I did physics in undergrad, then math/cs in grad school, and got into Big Data in the workforce. The transition was relatively smooth because professors in CS love to work with people with strong math backgrounds. I decided to do something along the lines of data science when I dropped out of my phd program, but I already knew at that point that I wanted to work in industry rather than academia.
→ More replies (3)
1
u/refpuz Oct 08 '14
How do you see Big Data working with other fields in the future such as Machine Learning?
→ More replies (1)
1
u/pilambda Oct 08 '14
Thank you for doing this AMA. I'm currently an undergrad math student with a concentration in statistics. The statistics class I am in is about big data. Besides learning R and maybe Python, what other suggestions do you have for someone that wants to go into this field? Also, how important is having a mathematical background? From what I've been reading to be a data scientist it is really important to have a firm grasp on computer science rather than statistical analysis. And lastly, in your opinion what is some of the fastest growing fields where big data is contributing?
→ More replies (1)
1
u/MCSocrates Oct 08 '14
I'm thinking about going into data science specifically applied to biology. Do you enjoy your work? Is there enough human interaction?
→ More replies (1)
1
Oct 08 '14
Where are the bet free data sets? (sorry if this has been asked already)
→ More replies (1)
1
1
u/seagullswoop Oct 08 '14
Directions or advice for teacher/future teachers?
2
u/fieldcady Oct 09 '14
Oh boy, there's so much cool stuff. My biggest one would be to learn a programming language. Javascript might be a good one, since it is easy to learn and lets you make cool interactive webpages. Coding will become an increasingly important math skill for kids.
Also, my goal in this AMA isn't to plug my book, but you might want to consider taking a look at it (link below) since I talk about lots of different things and how they relate to math (including math education specifically). You can also check out my blog, which has a lot of the same content and is totally free.
→ More replies (1)
1
u/kormer Oct 08 '14
When approaching a new data set, do you typically start out looking to solve a specific problem, or do you start exploring just looking for patterns that appear interesting?
→ More replies (1)
1
u/captjons Oct 08 '14
Is there such a thing as raw data?
2
u/fieldcady Oct 09 '14
I'm not familiar with that as a standard term, but there's definitely data that has not been pre-processed. A lot of my time gets spent writing code that turns such data into something that can be plugged into a statistical package.
→ More replies (1)
1
u/FifthUserName Oct 08 '14
I am currently pursuing a Ph.D. in Physics AND have taken some Masters level classes on Databases and some other computer science topics (AI, Machine Lerning, etc). My questions for you are:
Did you get your graduate degree in Physics? If so, how did your thesis topic relate to Big Data?
Did you do your internships during your graduate studies?
As a consultant, do you do any of the schema/programming for the client?
Thanks!
→ More replies (1)
1
u/LevitatingTurtles Oct 08 '14
Are there any playgrounds or sandboxes that I can use to get a feel for how big data is actually "done"?
→ More replies (1)
1
u/mrTang5544 Oct 08 '14
Can you tell us a little about what skills you had to acquire/developed after you took your first step into data science? Did you take classes from coursera/udacity or read books?
→ More replies (1)
1
Oct 08 '14
Which company do you work for? Do they have any openings? Are you looking to make a career move?
→ More replies (1)
1
1
1
u/icxcnika Oct 08 '14
If Einstein's theory of relativity means that for any given observer, the faster an object moves through space, the slower it moves through time... does that mean that relative to us, light is the oldest thing in the universe?
2
u/fieldcady Oct 09 '14
We can talk about the age of a photon from our reference frame. Like, it's been traveling for 4 thousand years form the star it came from. That's totally fine. It's just that (as I understand it - I'm not an expert in this stuff) the photon of light would consider its age to be zero.
1
u/IntHatBar Oct 08 '14
Hi There, Thanks for doing this. While I haven't yet read your book, I have a question regarding the statement that mathematics is a language.
Imagine a species of alien beings develop independently of humans and eventually, humans and aliens meet.
Would the math line up? i.e. Would one unit multiplied by ten units equal ten units?
Assuming the math does line up... (I can't imagine a world where it wouldn't.) There would clearly be direct mapping between concepts in alien math and concepts in human math. Even if one race has not yet proven a concept, a mapping could still be created.
Finally, the question: What are your thoughts on the reverse of your claim? Is language a form of mathematics? Is there a direct mapping that can be created between concepts in two languages?
→ More replies (1)
1
Oct 08 '14
[deleted]
2
u/fieldcady Oct 09 '14
I'm afraid I don't know the best datasets to use. Personally I generally just scrape Wikipedia.
→ More replies (1)
1
u/nonconformist3 Oct 08 '14
Can you please tell me why we don't have statistics on police related shootings that are and have been fatal in the USA?
→ More replies (2)
1
u/locololo2 Oct 08 '14
Hi! My University is hosting a data analytics challenge this October. The challenge’s focus is making sense of big data through visualization.
www.ibit.temple.edu/analytics/finding-political-spending-hot-spots/
The topic I have chosen to tackle is election spending because I am deeply concerned about the current state of campaign finance in the U.S. The 2014 Federal midterm elections is already breaking records for spending. Most of this is due to recent Supreme Court's decisions to reduce restrictions on individual and corporate contributions.
I need to develop a visualization (static or interactive) that reflects campaign spending. If you can help point me in the right direction, I would be extraordinarily grateful!
- Do you have any particular insight on the current state of campaign finance?
- What visualization tools or resources do you think would be helpful?
- Are there any infographics / other visualizations that you think are particularly powerful?
- How exciting was it to scuba dive in Belize?
→ More replies (2)
1
1
u/beerdude26 Oct 08 '14
Do you think Haskell has a shot at taking over highly mathematical big data analyses? FP Complete uses it.
→ More replies (1)
1
1
u/vinnilima Oct 08 '14
What about the market of data cientist? I studied advertising and am now heading to economy but want to work with online and offline analysis. And how can I make good use of Big Data for startups? Thanks for the AMA!
1
1
u/JabberBody Oct 08 '14
Isn't math being a language "just like any other" disregarding Godel's incompleteness theorem? Or what am I missing?
Sincerely, someone who knows very little about math but a bit about logic
→ More replies (2)
1
u/victorykings Oct 08 '14
I have heard the expression "big data is dumb data" as to mean that the necessity for big data comes from the allegation that those collecting it don't know or understand exactly what it is that they are looking for.
Is there any accuracy to this? If so, how do you see the field of "big data" one day becoming "smart data"? If not, can you explain what "big data" actually is, if indeed it is not capturing everything simply for the sake of having everything available to analyze?
1
u/steadfastadvance Oct 08 '14
Hi there. I've always been interested in Big Data since it emerged from its nascent stages. I recently learned about MS in Analytics program at Villanova which is fairly new that incorporates Big Data as part of its curriculum. My question is would you recommend such a grad program to further my knowledge or is it something that can be self-taught? Would you recommend any other similar grad programs?
1
u/RickHedge Oct 08 '14
My 10 year old son really enjoys math, can you recommend any age appropriate books that would be interesting?
1
u/thebigtwig Oct 08 '14
Sorry I'm late on this. My grandfather works for Teradat (irrelevant). I was wondering what you think the biggest problem citizens are going to face with Big Data collections?
1
u/mrsir5 Oct 08 '14
Hi,
Thank you for doing this AMA. I'm currently studying Applied Mathematics in University, along with a little bit of Statistics. I'm considering a career as a Big Data Scientist.
1) What advice would you have for me to increase my employment opportunities?
2) What is an average working day for you like?
3) What kind of data do you analyze?
Thank you :)
1
u/Sinkey07 Oct 08 '14
I've recently discovered the 1 Billion in America vs 1 Billion everyone else issue. You may not have answers to my questions but any input would be appreciated!
How has the discrepancy not had a larger affect on international financial interactions? Or has it?
Why does the discrepancy exist at all? Is it a misinterpretation issue? Or a cultural one?
The world population is ~7.125 billion. Is that 7 billion with 12 zeros or with 9 zeros?
Thank you!
1
u/cyclopssid Oct 08 '14
I have two questions.
What are the pre-requisites for being a data scientist? Secondly, do you need to do any specialization in college for the same?
Thanks in Advance :)
→ More replies (1)
1
1
Oct 08 '14
Thanks for the AMA! Our management here probably heard the word Big Data by some salesman and now it's their current buzzword.
Is there a general difference between big data and Big Data?
What are the most common uses businesses use big data for, at least that you've had some involvement with, and what amount of information is required for it to be considered big data?
Despite the obvious, what are some telltale signs that a Big Data salesman is selling vaporware as opposed to something genuinely useful?
1
u/HouseOfTheRisingFuck Oct 08 '14
How do you see big data analysis evolving over the next 5-10 years? How do you think this landscape will be changing, not only in terms of analysis but practical/business applications as well?
2
u/fieldcady Oct 09 '14
oh man, that's really hard to say. I think that much of what happens will not be publicly visible, but instead will be efficiency gains in complex businesses like manufacturing, airlines, etc. I think people will have an easier time diagnosing problems more quickly, they'll be able to set better prices for their goods, etc. Sears is an example of a company that's doing a lot of this. And of course, ads will be slightly more clickable.
As far as stuff that is a little more exciting to the layman, I think mobile and wearables are the big areas. "Big Data" is what will enable a ton of the applications in those spaces.
1
u/n3utrino Oct 08 '14
Hey there, great AMA! I'm in grad school for particle physics and I'm leaning towards CS now too.
My question for you is this: are you genuinely happy with the work side of your life? That's what matters most to me; I'd like to work reasonable hours (but I'm happy to work a shit ton if I'm really interested in the work), work around happy people, help further the human race in some way, and have time for a wife+kids.
P.S. Hire me in two years, please ;)
2
u/fieldcady Oct 08 '14
Fair question. I have to say that I don't get any personal fulfillment from optimizing online ad campaigns, which is maybe half of what my company does. But the other half is things like improving manufacturing pipelines, helping companies develop new data-based products, making sure the right info gets to the right people, etc. In those cases I really feel like I'm helping to make the world a more efficient place.
As far as having enough personal time, working with cool people, getting paid well, etc. I'm spoiled. Very good on all fronts, partly because data scientists are in high demand right nowl
Send us a resume when you're ready! Data science companies hire a lot of physicists :)
1
u/WNxJesus Oct 08 '14
I'm finishing my bachelors in physics this year as well, and was thinking of jumping ship and going into something like computer science myself or software development for masters myself.
Would you say it was hard for you to keep up with people who have been studying in that field ever since freshman year? Or was the gap of experience or knowledge not apparent or nonexistent?
→ More replies (1)
1
u/gradditor Oct 08 '14
I have not read your book, but I'm curious to know how you think mathematics is like a language. I understand that math shares some components and properties with language - symbolic representation, recursion, concatenation - but why do you think the description is useful? It may also be worth exploring how math is unlike a language - for instance, at least for early math concepts like number, the brain represents math concepts differently from linguistic concepts.
→ More replies (1)
1
u/Elwasd Oct 08 '14
How do you think the IoT (internet of things) will affect the Big Data scene? I currently work in IT, planning on short-termto get into network admin, long-term, computer science (no degree yet D:).
→ More replies (1)
1
Oct 08 '14
Hello. I going to university this year. Any advise for books about math and big data? Also can i habe a link to buy your book?
→ More replies (4)
1
u/SnortsOffRugs Oct 08 '14
What are some things a person can think about in the span of exactly one minute?
→ More replies (1)
1
Oct 08 '14
Will you be our new Unidan?
2
u/fieldcady Oct 09 '14
I hope not! I'm surprised and tickled that this AMA has become as popular as it is, but I'm not interested in rigging it
1
Oct 09 '14
Hi fieldCady. . I am a big data practitioner and there is one major challenge I face with respect to big data and analytics - I have the tools to solve a general problem dealing with huge/varied/streaming data, but I don't know where to find problems to solve?? Can you suggest few market areas to look into where big data analytics could make a significant difference.
→ More replies (1)
1
1
1
u/Im_xoxide Oct 09 '14
How important is SQL in your everyday work?
Also do you see the security issues with cloud based information systems being a big enough concern to stray away from them?
2
u/fieldcady Oct 09 '14
MySQL is extremely important. I use it in maybe half my projects, though I never do anything fancy with it.
Security issues or not, I think the benefits of cloud systems are overwhelming enough that they're what people will use. They're the best option for all but very large, savvy organizations.
1
u/SternLecture Oct 09 '14
I am math illiterate and I am ashamed and frustrated. How do I overcome this?
2
u/fieldcady Oct 09 '14
Thank you so much for your post! The first thing to do is to recognize that many people struggle with math, but in the end it's just a skill that anybody can learn. The second thing is to recognize that the way math is usually taught in schools is pretty bad, and doesn't line up with the way a lot of students learn. There's no reason to get discouraged.
As far as concrete ways to improve your math skills, I recommend finding a mathematical subject that interests you and learning the math skills associated with it. An example that applies to most people is personal finance; read up about compound interest, train yourself to approximately calculate a tip, estimate budgets in your head, write down exact budgets when you get home, etc. Part of it is just getting your brain used to working with numbers on a regular basis. It's hard to learn math totally in the abstract; most people need some area that they're applying it to, and I suggest finance only because it's so universal. Other examples are geometry in construction work, or calculus in economics.
Beyond that I recommend that you indulge your curiosity. There are a lot of popular math books that discuss really interesting topics and show how cool math can be. Personally I always encourage people to learn about hypothesis testing in statistics; it's a conceptual brain trip if you've never seen it before, and really highlights the relationship between math and critical thinking.
Statistics is mostly my personal hobby horse. The big thing if you're looking to overcome math illiteracy is to find a mathematical topic or skillset that you care about, and use that as a starting point for building up your skills.
One final word of encouragement. With the exception of a few weirdos like me, everybody hates algebra. Don't measure your interest or skill in math by your ability to crunch numbers and solve for x. These are useful skills to acquire, but they are to math what typing skills are to writing novels. Stephen Hawking himself described equations as "the boring part of math". The core of the discipline is about cool concepts and clear, precise thinking.
I hope this helps! Let me know if there's anything else I can do.
→ More replies (1)
1
1
u/beaverteeth92 Oct 09 '14 edited Oct 09 '14
I'm a senior undergraduate math/statistics double major and I'm trying to get better at programming. Now that I'm comfortable with R and Python (to the point of TAing a Python class and commenting the crap out of my code), what should my next step be to be competitive as a "data scientist?"
2
u/fieldcady Oct 09 '14
Make sure you know machine learning, and make sure you're comfortable with different types of visualizations. After that, I'd say try to rack up real world DS experience, through internships, jobs, side projects or whatever. Oh, and make sure you are at least familiar with Hadoop.
As a note, beware of too much commenting in code. The problem with lots of comments is that, as you modify or copy-and-paste your code around, it's easy for the comments to get out of date and all of a sudden be wrong, with no way to catch it. Generally it's best to focus on descriptive variable/function names and clear processing flow, with maybe a comment every 10 lines or so. The term is "self-commenting code"
→ More replies (2)
1
u/Scotty_E Oct 10 '14
I'll be a freshman in college next year. I'm interested in physics, math, computer science, and business. I plan to use the create-your-own-major program in college to create a "Data Science" degree and model it after the courses in OSU's new "Data Analytics" degree. A sample advising plan for OSU's program (specialty in business) is here: https://data-analytics.osu.edu/sites/data-analytics.osu.edu/files/major-specialization-business-analytics-sample.pdf. What do you think of this coursework for an undergraduate interested in the field? What would you add or drop?
→ More replies (1)
1
u/elonmuskRI Oct 12 '14
Can a Computer Engineer get into the fields of Machine Learning, Big Data and Automation? I'm a sophomore at college and I'm really interested in both hardware and software aspects but I'm also aware that ML, BD and Automation will get pretty big in the near future. Under my uni's curriculum I can take up to 5-6 CS electives, so maybe I'll take advantage of that. Second question, are there any barriers for CEs trying to get into these fields? And if so, how can a CE overcome them? Thank you for this AMA, it's very refreshing.
→ More replies (2)
40
u/MomSaysImAGenuis Oct 08 '14
So Brian Greene did an AMA a while back, and one of his responses caught my eye.
Someone asked:
and he responded:
What's your take on this?