Nicole Levin’s ST558 Blog

Fifth St558 Blog Post

2022-12-01T00:00:00+00:00

I have learned many things in ST558 that will be valuable in the future and will impact how I do things in practice. I learned a lot about R, RStudio, and R Markdown that will help me program better in the future and will improve my documentation when I do program in the future. I learned about the Tidyverse and ggplot which will impact the ways that I manipulate data and create plots using R in the future. I’ll also put more thought into exploratory data analysis and variable selection methods next time I am presented with that task.

When I look to how I may use data science and statistics in the future, there are some unknowns related to whether I will choose to make a career pivot into a role that is directly related to my new education or if I choose to instead continue on my chemical engineering path using my new education as an extra set of tools at my disposal. Either way, I expect to use the data science skills of manipulating, visualizing, and analyzing data wherever the future takes me. I’d like to learn more about other predictive model options in the future. Seeing how many options are available in the caret package makes it very clear that there is so much out there that I don’t know about.

Project 3

2022-11-13T00:00:00+00:00

For this project, I am analyzing a dataset of features about articles published by Mashable over a period of two years. My overall goal is predicting the number of shares an article will get based on the attributes provided in the dataset. I looked at the data split out by data channels. There are six data channels in the complete dataset: lifestyle, entertainment, business, social media, technology, and world. My report for each data channel contains some summary statistics and plots, model-fitting for a linear regression model and a boosted tree, and a comparison of the predictive abilities of the two models. The full dataset contains 61 attributes for each article, but I focused on shares as the response variable and the following six predictor variables for summarizing and modeling:

num_hrefs: Number of links
n_tokens_title: Number of words in the title
kw_avg_avg: Average keyword
average_token_length: Average length of the words in the content
num_imgs: Number of images
n_non_stop_unique_tokens: Rate of unique non-stop words in the content

A lot of the challenge in this project was related to the automation. One thing that worked well for me on this project was to start the automation pieces of the project with enough time available to do some stumbling around and thinking about how to get them to work. I also ran into the challenge of not having a partner for this project, which meant there was not a teammate to turn to when things were not working. If I could do something differently, I would have gotten an overall earlier start on the project to spread the work out even more and allow more opportunities to look at things with fresh eyes.

My biggest takeaways from this project are that automating reports in RStudio with R Markdown is a very powerful tool and predicting the number of shares that articles will get is pretty difficult. None of my models worked very well for predicting the number of shares.

If you would like to see the results of my analysis, the rendered Github Pages site can be viewed here and the repository is located here.

Fourth St558 Blog Post

2022-10-30T00:00:00+00:00

One important part of building a regression model is determining what variables to use. Three resources that I read to inform my plan going forward for selecting variables for a model are listed at the end of this blog post. One key takeaway I found from those resources is that variable selection can be complicated and subjective, and there is no one-size-fits-all answer for it.

The steps, thoughts, and ideas I plan to take with me for variable selection in the future are:

Talk with a subject matter expert (SME) if one is available. Get their explanations on what the variables are and how they may relate to the response variable of interest and each other. Understand what their goals are. Is prediction an important use for their model or are there other priorities? Are there clearly established relationships between any of the predictors and the response that can allow us to skip those variables in the selection process and just include them in the final model?
Start with exploratory data analysis (EDA.) Get a feel for trends in graphs and simple number summaries. Possibly identify variables that could benefit from a transformation to make a more linear trend of it with the response, Look for highly correlated variables that could create an unstable model.
Make any transformations that come to light as appropriate based on the EDA, experience, or conversations with the SME.
Pick a couple of methods that make sense for the situation, and consider not just sticking with the classic forward, backward, or best subset selection. Look at penalized options like LASSO as well. My preference from the stepwise options is backward selection. My preference of the penalized options is the LASSO, as it is one I have at least a small amount of experience with. Among the tree-based methods and others more commonly used for big data sets with lots of predictors, I will for the time-being need to rely on the recommendations from coursework and try multiple options to see what provides the best results in different situations.
Take a good look at the best performing sets of variables and do some interpretation of what those combinations are saying. Does one make more sense logically? Is one more cost effective to get measurements and data for in future data sets? Given all of the information gathered from experts, experience, and the variable selection process what makes sense to proceed with?

In summary, my main take away from delving a little into variable selection is to be more thoughtful and conscious of every step and decision along the way. Think critically when I need to, expansively when I need to, and above all else, understand what I’m doing and why all along the way.

Resources that helped me question whether I know anything about variable selection at all:

Exploratory Data Analysis

2022-10-15T00:00:00+00:00

The goal of exploratory data analysis (EDA) is to analyze and investigate the main characteristics of datasets, often using data visualization methods like plots. My first step for EDA is to look at the dataset and understand it at a high level. How large is it and what are the variables? Are there missing values? Are there a lot of missing values of specific variables? Are each of the variables continuous, discrete, or categorical? After getting this basic understanding, I want to look at some plots. If the data is taken at different times, I would look at plots of variables vs. time to look for signs of seasonality or a general positive or negative trend over time. I also will look at basic summary statistics, like mean and variance, and plots that allow me to see the shape of the distribution of variables, like histograms and bar plots. Then, I can look for significant correlation between variables using a correlation matrix. Finally, I can pull together and summarize what I have learned from the data. The learnings from the EDA can inform whether assumptions are reasonably being met for statistical analysis techniques I may want to use for further analysis, and the EDA also helps me identify interesting relationships between variables that I may want to investigate further.

Project 2

2022-10-12T00:00:00+00:00

The goal of this project was to create a vignette about contacting an API using functions that my groupmate and I created to query, parse, and return well-structured data. Then, the functions were to be used to obtain data from the API to do some exploratory data analysis (EDA.) My groupmate and I initially elected to use a brewery API and did some initial work with querying and parsing data from that API. What became apparent after a while was that the brewery API lacked numerical data that would be needed for creating the plots needed for our EDA. We chose to abandon the brewery API, shifted our focus from beer to food, and started working with the Spoonacular food API. Losing out on that work wasn’t particularly pleasant, but it did give me some practice with generating functions to perform API queries, so the time wasn’t all wasted. Our final vignette can be viewed in it’s rendered form here and the associated Github repository is here.

The Spoonacular API is full of data on many ingredients and recipes, allowing for searches based on everything from calorie limits, to ingredients, to cuisine categories, to levels of specific micronutrients. One of the things I was interested in looking for was whether other variables seemed to have a relationship with the number of aggregate likes a recipe had. My thoughts were that maybe recipes with more calories would have more likes since they would likely have more fat or sugar to make them tastier, or possibly recipes with higher health scores would get more likes based on a perceived health halo. I was not successful with finding any strong trends there. Most of the recipes had low levels of likes, and then there were a few outliers with many more likes. I may need to look up some of the outliers to try out cooking for myself. Additionally, we saw in our numeric summaries a decrease in average calories when moving from the low to medium to high carb categories. That result was unexpected.

The largest frustrations and difficulties with the project for me were on the technical side of getting Github and R Studio properly connected and rendering output to display properly. Hours were wasted on getting to the point where I could successfully pull down the project repo to R Studio, make changes, commit, and push the updated files to the repo. Those are hours I would have much preferred to have spent working on the meat of the project. The main thing I would change if presented with a similar project in the future is to get started with the technical setup aspects as quickly after the project is assigned as possible. What looks like half an hour of work can turn into a lot more. When I think about the coding portion of the project, the main challenges that I went through were thinking about how I could get from search options that would be easy for a user to work with to the data that I wanted to pull. The logic of pulling the item numbers from an initial query and turning it into one long string of comma separated item numbers that could be used with the “bulk search” option in the API was critical there. Overall, this was an interesting project that I enjoyed during the periods where I wasn’t wildly frustrated with the technology aspects.

Second St558 Blog Post

2022-09-14T00:00:00+00:00

I completed my first project for ST558 earlier this week, and you can check out that project here. The purpose of the project was to write functions that manipulate, process, and plot data sets that come in a certain form. The data sets we worked with came from the census bureau. Through this project, I learned about different functions that can be used to parse and manipulate data. I also gained a better understanding of some of the video material around creating functions by putting it into practice. One thing I did that worked well for me was spreading my time working on the project out across many days. This allowed me to have lots (and lots and lots) of hours available to work on the project, which is important, as I am a slow, novice programmer. I also find that I run into a lot of roadblocks when working on programming assignments that tend to benefit from spending some time not actively thinking about them, so having time in my schedule to be able to step away tends to be pretty beneficial for me. One thing I would do differently in the future is to include more narrative text sections in my R Markdown file from the beginning. I misunderstood the requirement of having a narrative throughout the document, so I originally used a combination of text blocks and lots of code comments to form that narrative. Spending a chunk of time at the end of the project just to go back and split up code chunks to add narrative sections to rewrite things that I had originally written as comments was not a particularly enjoyable experience. I will now at least understand that expectation for the future, and it did make for a nicer looking, easier to read, final product.

First St558 Blog Post

2022-08-28T00:00:00+00:00

At the beginning of a course called “Data Science for Statisticians” it is useful to take a step back and spend some time thinking and reading about what a data scientist actually is. The best, simple summary seems to be that a data scientist is someone who is able to take a knowledge of statistics and a knowledge of computer science and combine the skills of both fields to process data and make predictions. Data scientists and statisticians use some different techniques and focuses, and they can complement each other and often need each other’s help in solving problems. A data scientist is typically dealing with very large datasets, so they need to have substantial knowledge in dealing with and cleansing massive amounts of data. Data scientists also tend to be more interested in the predictive ability of a model and less interested in quantifying uncertainty than a statistician may be. To be a good data scientist, a person needs to have strong coding abilities and a good applied statistics background. I view myself as a statistician who can gain a lot by having the knowledge of some of the tools and methods used by data scientists. I see developing an understanding of the way data scientists approach data and models as not just a way to add skills for if and/or when I need to deal with a big dataset, but as a way to expand my ability to think about problems in different ways.